How to add client certificate in ASP.NET Web API in-memory testing?

I want to test my Web API service using in-memory HttpServer.

The current setup looks the following:

 var httpConfig = CreateTestHttpConfiguration();
 var server = new HttpServer(httpConfig);

 var handler = new WebRequestHandler();
 var cert = new X509Certificate2("filename", "password");
 handler.ClientCertificates.Add(cert);
 var client = HttpClientFactory.Create(handler, server);

I can make requests to the server using these client and everything works except that certificate is not added to the request.

As I understand that happens since server executes before handler (I can't rearrange them since they implement different interfaces) and since server immediately responses handler is not even executed (I've tested this assumption using HttpClientHandler subclass instead of handler).

So my question is: How can I add the client certificate for in-memory testing?