Testing Laravel Passport

I have a problem - I am using Laravel Passport for my api. I need to write tests. Whenever I use the WithoutMiddleware trait in my tests it disables the Implicit route model binding feature, which I use. Whenever I don't use that trait, I need to authenticate directly from my test. To do that, I need to create an API token via Passport.

Passport, however, needs to be installed before testing via artisan passport:install, because my tests are using DatabaseTransactions and DatabaseMigrations traits. When I do this, tests take enormous amount of time to run and I feel like it is not the right way to go about it. Isn't there a way to disable only the auth middleware? Or any other ideas to go about this?