Testing a function that loads an RDS file in R

I was wondering how you would go to test a function with testthat that loads inst/extdata rds files to do computation on ?

my_function = function() {

  file = paste(find.package("xyz"), "/extdata/my_file.rds", sep = "")

  if (!file.exists(file)) {
    return(stop("File could not be found."))
  }

  population = readRDS(file)

  ... and then some computation
}

When I do a test like this...

test_that("my function can read file", {
  expect_equal(my_function(), "xyz")
})

I get this error...

Error (test-xyz.R:63:3): my function can read file
Error: File could not be found.

Thanks for your inputs !