Do not build all projects in solution while testing

I have solution with a lot of projects. Building the whole solution takes about 20 minutes. I need to debug only one class in project so I made one testing project, dealt with referencing the project to test project, prepare my test and run debug test. It builds the whole solution again. Even if I just modify one symbol in my test and run debug test, it again builds whole solution. When I run debug test without modifying it builds only a few projects and I have to wait too.

I understand that if I'm making changes to both test and regular project that they need to rebuild but why are they rebuilding referenced projects when they were no changes? Can't it just use already compiled projects?

Is there some way to let it build only that one test project and use already compiled assemblies of other projects without waiting 20 minutes for build? It is really annoying and I don't seem to find a way to bypass this process.

I am using Visual Studio 2013 Professional.