Integration tests when WebConfig is dependency

I've written a good number of unit tests for a new feature I'm implementing and now I'm getting around to writing integration tests, only, I'm now questioning the justification for them in this case.

The feature includes creating an MVC action filter attribute. My unit tests check the paths of logic within the attribute and also checks that they're applied to the controller actions we expect.

The feature is driven by the WebConfig, which decides whether an organisation is enabled or not.

I could use the ConfigurationManager to set the value before each test is run, but this seems like a test smell to me, or is this common when integration testing when a config file is a dependency?