C# Performance benchmark

To keep track of performance in our software we measure the duration of calls we are interested in.

for example:

using(var performanceTrack = new PerformanceTracker("pt-1"))
{
    // do some stuff

    CallAnotherMethod();

    using(var anotherPerformanceTrack = new PerformanceTracker("pt-1a"))
    {
       // do stuff

       // .. do something
    }

    using(var anotherPerformanceTrackb = new PerformanceTracker("pt-1b"))
    {
       // do stuff

       // .. do something
    }
    // do more stuff
}

This will result in something like:

pt-1  [----------------------------] 28ms

      [--]                            2ms from another method

pt-1a   [-----------]                11ms

pt-1b                [-------------] 13ms

In the constructor of PerformanceTracker I start a stopwatch. (As far as I know it's the most reliable way to measure a duration.) In the dispose method I stop the stopwatch and save the results to application insights.

I have noticed a lot of fluctation between the results. To solve this I've already done the following:

• Run in release built, outside of visual studio.
• Warm up call first, not included in to the statistics.
• Before every call (total 75 calls) I call the garbage collector.

After this the fluctation is less, but still not very accurate. For example I have run my test set twice. Both times

See here the results in milliseconds.

Avg: 782.946666666667 981.68
Min: 489 vs 513
Max: 2600 vs 4875
stdev: 305.854933523003 vs 652.343471128764
sampleSize: 75 vs 75

Why is the performance measurement with the stopwatch still giving a lot of variation in the results? I found on SO (https://stackoverflow.com/a/16157458/1408786) that I should maybe add the following to my code:

//prevent the JIT Compiler from optimizing Fkt calls away
long seed = Environment.TickCount;

//use the second Core/Processor for the test
Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2);

//prevent "Normal" Processes from interrupting Threads
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;

//prevent "Normal" Threads from interrupting this thread
Thread.CurrentThread.Priority = ThreadPriority.Highest;

But the problem is, we have a lot of async code. How can I get a reliable performance track in the code? My aim is to discover performance degradation when for example after a check in a method is 10ms slower than before...