Pact testing using random port with jUnit5 and SpringBoot

I can't get the pact provider tests to run on any other port than 8080 when using jUnit5. I have the following code:

@ExtendWith(SpringExtension.class)
@Provider(PROVIDER)
@PactFolder("pacts")
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
public class PactProviderTest {

  @LocalServerPort private int serverPort;

  @Autowired Result<IServiceManagementFacade2> serviceManagementFacade;

  @Autowired private RestServiceFactory restServiceFactory;

  @TestTemplate
  @ExtendWith(PactVerificationInvocationContextProvider.class)
  void pactVerificationTestTemplate(PactVerificationContext context) {
    context.setTarget(new HttpTestTarget("localhost", serverPort));
    context.verifyInteraction();
  }

When I run this I get the following error:

Request Failed - Connect to localhost:8080 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

If I change

@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)

to

@SpringBootTest(webEnvironment = WebEnvironment.DEFINED_PORT)

the tests pass. But then no other app can be running on that port during the tests which is not acceptable in the test environment. Any ideas how I can solve this?