Django unit test remove Files created inside the tests

I'm using DRF view tests. Inside my tests I create files which I want to remove after the tests were running. I'm using addCleanup inside my setUp method to remove the directories. It basically works, but when I run my complete set of unit tests one dir won't be deleted (I think it's the first one created).

Here is the relevant code:

class BaseViewTest(APITestCase):
    client = APIClient()
    base_media_root = settings.MEDIA_ROOT

    def setUp(self):
        settings.MEDIA_ROOT = tempfile.mkdtemp(dir=self.base_media_root)

        # add test data
  
        self.addCleanup(shutil.rmtree, settings.MEDIA_ROOT)