compile_time_purge_matching doesn't work during testing

The following config.exs

config :logger,
  compile_time_purge_matching: [
    # [application: :foo],
    [module: SomeModule, level_lower_than: :error]
  ]

Above code works when running from shell but not when doing mix test

The config.exs is clearly parsed and i can't find that the configuration is overridden anywhere.