Should configuration files be part of unit tests?

I believe this question to be general in nature, even though I mention the Spring framework here.

I've been working with the Spring framework, and one of the things I've seen a lot of is code like this:

The spring framework will map these values in a config.yaml file...

server:
    url: http://someUrl
    port: 8080
    user: user
    password: secret

to the following class...

@ConfigurationProperties("server")
public class ServerConfig {
    private String url;
    private Integer port;
    private String user;
    private String password;
    //getters, setters...
}

My point is, is that objects get instantiated at run time based on a configuration file.

My question is, when unit testing, should I incorporate configuration files in my tests, or should all setup and configuration happen in code?

I seem to think that unit tests should be deterministic and not rely on dependencies like a configuration file for setup. But if the configuration file is part of the tests, then maybe that's OK?