Add an DelegationHandler (outgoing request middleware) to TestServer's HttpClient

Is it possible add a custom DelegationHandler (outgoing request middleware) to the HttpClient built by the asp.net core TestServer?

I'm trying to combine the customization possibilities of using HttpClientFactory to influence HttpClients with the in-memory testing utility TestServer:

public class ExternalProxySslDowngradeSimulator : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(
            HttpRequestMessage request,
            CancellationToken token)
     {
        if (request.RequestUri.Scheme == "https")
                request.RequestUri = new Uri(
                     request.RequestUri.OriginalString.Replace("https:", "http:"));

         return base.SendAsync(request, token);
     }
}

[TestFixture
public class TestClass
{
   [Test]
   public async Task CallApi()
   {
      var builder = new WebHostBuilder()
           .ConfigureAppConfiguration((ctx, config) => { })
           .UseUrl("http://customDomain.com")
           .UseStartup<CustomApi.Startup>();

      var testServer = new Microsoft.AspNetCore.TestHost.TestServer(builder);

      var httpClient = testServer.CreateClient();

      var apiResult = await httpClient.GetAsync("https://customDomain");
   }
}

I can't find a way to plug my DelegatingHandler into the HttpClient returned from testServer.CreateClient()

---

What I've tried so far:

1. Registering a custom HttpClientBuilder (did not work):

// did not work:
var builder = new WebHostBuilder()
   .UseStartup<CustomApi.Startup>()
   .ConfigureServices(services =>
   {
        services
          .AddHttpClient("")
          .AddHttpMessageHandler<ExternalProxySslDowngradeSimulator>();

Looking through the code for TestServer it is not hooking into that pipeline.

2. Custom HttpClientFactoryOptions (did not work)

Thought I could bind a custom HttpClientFactoryOptions and add a custom action to HttpClientFactoryOptions.HttpMessageHandlerBuilderActions. But the TestServer doesn't really consume this way a default Http Client Factory does.

3. Customize the setup of TestServer (did not work)

Looking through the documentation / source didn't see anywhere to hook in. There's an extension method to further control DI, but I didn't see how that could get me a hook into the custom HttpMessangeHandler that TestServer is using.