Multiple Tests in the same file eat up resources and cause failures

I have several tests in the same file that do nearly identical computation. When I run them together the first two pass and run in about 5 seconds each and the last two fail and take longer (the fourth test takes over twice as long). I can only assume this is because the tests that are run first are hoarding resources. They all are self contained and I figured everything would de-allocate once the test finished.

They all run in about 5 seconds and pass if I comment out all the other tests so on their own they are fine.

I tried running go test -p 1. But that had no effect.