How to clean up downloaded files in Django view testing?

I have a Django view that returns a file to download (response['Content-Disposition'] = 'attachment; filename=' + filename) in response to a POST as the contents of the file are dependent on data submitted by a form. In setting up my tests for this view, I'm having a hard time finding a clean way to download the file and make sure it is cleaned up afterwards as this doesn't come for free with the TestCase behavior. Currently I have the following:

with tempfile.TemporaryDirectory(dir='.') as temp_dir:
    os.chdir(temp_dir)
    response = self.client.post(...)
    # Do asserts of the downloaded file/response object here
    os.chdir('..')

This has the benefit of creating a temporary directory where the file is downloaded to, and the whole directory is deleted when the test completes. Also, if any of the asserts fail the temp directory (and contents) will stick around, but this specifically feels fragile and like it could break other tests because of the directory change.

Is there a better way/pattern to accomplish this?