Why is Castle Windsor trying to load dependencies from installers I'm excluding?

We have this ASP.NET Web API project which configures a Castle Windsor container on startup using a series of implementations of IWindsorInstaller.

The method in the API which does this looks like:

public static IWindsorContainer RegisterContainer(IWindsorInstaller settingsInstaller, InstallerFactory skipSettings,
    params Action[] onRegisterCallbacks)
{
    Container = new WindsorContainer();
    Container.AddFacility<StartableFacility>(x => x.DeferredStart());

    Container.Install(settingsInstaller); //Must be registered first!!!
    Container.Install(FromAssembly.InThisApplication(skipSettings));

    DependencyResolverService.Initialise(Container);

    onRegisterCallbacks
        .ToList()
        .ForEach(x => x());

    return Container;
}

I'm now trying to introduce a component test framework which uses Microsoft.Owin.Testing.TestServer to call the API but treat it as a black box, mocking any dependency which relies on external services.

I thought I'd be able to wire up my dependencies in my component test project by calling the above RegisterContainer method, but pass in an InstallerFactory that filtered out any installers with external dependencies. I'd register these later with mocks.

However, having done this I was getting System.IO.FileLoadException exceptions on the line Container.Install(FromAssembly.InThisApplication(skipSettings)) looking for a Microsoft.WindowsAzure.Storage assembly which is only required by one of the filtered installers.

So, trying to backtrack to see if I'd made a mistake, I changed my InstallerFactory to this:

public class SkipTestSettingsInstallerFactory : InstallerFactory
{
    public override IEnumerable<Type> Select(IEnumerable<Type> installerTypes)
    {
        return new List<Type>();
    }
}

By my understanding this should filter out all installers and thus cause the line Container.Install(FromAssembly.InThisApplication(skipSettings)) to do nothing. But no, same exception.

Now, of course I could reference Microsoft.WindowsAzure.Storage in my component test project, but I don't want to because to me that would undermine its purpose as an API black box. So what would be the best approach to essentially allowing two projects to share an IoC configuration whilst providing the scope to override specific registrations?