How to Benchmark Your .NET Code Using BenchmarkDotNet

Sometimes as developers we’re faced with making a certain piece of code perform faster. We often need to decide which piece of code performs the best. Benchmarking provides us concrete measurements between different pieces of code, so we can make verifiably correct decisions based on performance. In this post, I’ll introduce the BenchmarkDotNet library. This library simplifies the process of benchmarking our .NET code, and provides a bunch of cool features out of the box.

Getting Started

To begin performing our benchmarks, we’ll start by creating a new C# .NET Core Console app. If you wanted to test code for other runtimes such as .NET Framework, BenchmarkDotNet supports other runtimes. For the full documentation, see BenchmarkDotNet's website.

Creating the Project

Using Visual Studio 2019, we’ll start by creating a new console app using the .NET Core runtime. With the project created, we’re greeted by the standard “Hello World” console program.

using System;

namespace BenchmarkingIntro
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Importing the BenchmarkDotNet Library

Next, we need to install the latest BenchmarkDotNet Nuget package. This is easily done via the Nuget package manager or the dotnet add package BenchmarkDotNet command.

Setting Up the Benchmarking Class

Now we need to design a benchmark that we’re eventually going to measure. We can start with a simple benchmark that compares the lookup speeds of two types of collections, a List and a HashSet. I’ve designed a simple class that that creates a List and a HashSet that contain the numbers 0-999. I’ve created two methods, both decorated with the [Benchmark] attribute, that perform a lookup using Contains. The [Benchmark] attribute tells BenchmarkDotNet to measure the method as part of the benchmark. I’ve also used the [MinColumn, MaxColumn] attributes on the LookupSpeed class to include both min and max values for test on the benchmark summary. Below is the code:

[MinColumn]
[MaxColumn]
public class LookupSpeed
{
    private const int N = 1000;
    private const int LOOKUP = 250;

    private readonly IList _list;
    private readonly ISet _set;

    public LookupSpeed()
    {
        _list = new List();
        _set = new HashSet();

        for (int i = 0; i < N; i++)
        {
            _list.Add(i);
            _set.Add(i);
        }
    }

    [Benchmark]
    public bool ListLookup() => _list.Contains(LOOKUP);

    [Benchmark]
    public bool SetLookup() => _set.Contains(LOOKUP);
}

Running the Benchmark

In our program’s Main method, we need to add BenchmarkRunner.Run<LookupSpeed>() to tell BenchmarkDotNet to run a benchmark on the LookupSpeed class we created.

static void Main(string[] args)
{
    BenchmarkRunner.Run<LookupSpeed>();
}

Running this code now generates the following summary output in a console window.

// * Summary *

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.17763.678 (1809/October2018Update/Redstone5)
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview3-010431
  [Host]     : .NET Core 2.2.2 (CoreCLR 4.6.27317.07, CoreFX 4.6.27318.02), 64bit RyuJIT  [AttachedDebugger]
  DefaultJob : .NET Core 2.2.2 (CoreCLR 4.6.27317.07, CoreFX 4.6.27318.02), 64bit RyuJIT


|     Method |       Mean |     Error |    StdDev |        Min |       Max |
|----------- |-----------:|----------:|----------:|-----------:|----------:|
| ListLookup | 147.012 ns | 1.1715 ns | 1.0958 ns | 145.836 ns | 149.28 ns |
|  SetLookup |   9.967 ns | 0.2299 ns | 0.2647 ns |   9.777 ns |  10.57 ns |

// * Warnings *
Environment
  Summary -> Benchmark was executed with attached debugger

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  Min    : Minimum
  Max    : Maximum
  1 ns   : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:00:40 (40.3 sec), executed benchmarks: 2

Important Note About Debug Mode

Make sure your console app is set to run in Release mode, not Debug mode. BenchmarkDotNet will complain if you attempt to run in Debug mode as the results are unlikely to be accurate.

Conclusion

While this is kind of a silly example, and a lot of us probably already know that a HashSet has better lookup performance than a List, benchmarking allows us to get a more definitive, clearer picture of this. Oftentimes we’re forced to make a decision to use one piece of code over another based on performance, and benchmarking our code via BenchmarkDotNet provides a simple way to make an informed decision.

Happy coding!