Benchmarking .NET Applications with BenchmarkDotNet: A Step-by-Step Guide

Optimizing for performance is a very important part of making software, especially when it comes to improving .NET apps. Benchmarking is a great way to get an accurate picture of performance, whether you're fine-tuning algorithms or looking at multiple implementations.

In this blog post, we'll talk about how to utilize BenchmarkDotNet, a well-known and robust framework for testing .NET code. We'll go over setting up a console program, writing benchmarks, and understanding the results.

What is BenchmarkDotNet?

BenchmarkDotNet is an open-source .NET library designed to make performance benchmarking easy, reliable, and repeatable. In addition to managing warm-up stages and mitigating external variables (such as JIT compilation), it automates benchmark execution and gives comprehensive statistical statistics. The.NET runtime team and other members of the.NET community utilize it extensively.

Key features of BenchmarkDotNet include:

• Automated Warm-up and Iteration: Ensures consistent results by warming up the JIT compiler.
• Statistical Analysis: Provides mean, median, standard deviation, and other metrics.
• Cross-Platform Support: Works on Windows, Linux, and macOS.
• Export Options: Generates reports in various formats (e.g., markdown, CSV, HTML).
• Diagnostics: Supports memory usage analysis and advanced profiling.

Setting Up a Console Application for Benchmarking

Let's build a console application to compare two distinct string concatenation methods. We'll compare the performance of StringBuilder against string concatenation with the + operator.

Step 1: Create a New Console Application

Add the BenchmarkDotNet NuGet package to your project:

dotnet add package BenchmarkDotNet

Alternatively, add it to your .csproj file:

<ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.12" />
</ItemGroup>

Open your terminal or IDE and create a new .NET console application:

dotnet new console -n BenchmarkDemo
cd BenchmarkDemo

Step 2: Write the Benchmark Code

Replace the content of Program.cs with the following code. This example benchmarks two methods for concatenating a list of strings: one using StringBuilder and another using the + operator.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text;

namespace BenchmarkDemo
{
    public class Program
    {
        static void Main(string[] args)
        {
            // Run the benchmarks
            var summary = BenchmarkRunner.Run<StringConcatBenchmark>();
        }
    }

    [MemoryDiagnoser] // Tracks memory allocations
    [Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)] // Orders results by performance
    [RankColumn] // Adds a rank column to the results
    public class StringConcatBenchmark
    {
        private readonly List<string> words = new List<string> { "Hello", "World", "Benchmark", "DotNet" };

        // Marks this as the baseline for comparison.
        [Benchmark(Baseline = true)] 
        public string ConcatUsingString()
        {
            string result = "";
            foreach (var word in words)
            {
                result += word;
            }
            return result;
        }

        [Benchmark]
        public string ConcatUsingStringBuilder()
        {
            StringBuilder sb = new StringBuilder();
            foreach (var word in words)
            {
                sb.Append(word);
            }
            return sb.ToString();
        }
    }
}

Code Explanation

• Benchmark Class: The StringConcatBenchmark class contains the methods to benchmark. It’s decorated with attributes to configure BenchmarkDotNet’s behavior:
• [MemoryDiagnoser]: Tracks memory allocations for each benchmark.
  • [Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]: Sorts results from fastest to slowest.
  • [RankColumn]: Adds a column to rank methods by performance.
• Benchmark Methods: Each method to benchmark is marked with [Benchmark]. The Baseline = true attribute on ConcatUsingString makes it the reference point for relative performance comparisons.
• Test Data: A simple List<string> with four words is used as input for both methods.
• Main Method: BenchmarkRunner.Run<StringConcatBenchmark>() runs the benchmarks and generates results.

Step 3: Run the Benchmarks

1. BenchmarkDotNet will execute the benchmarks, performing warm-up iterations, actual runs, and statistical analysis. The output will be displayed in the console and saved to files in the BenchmarkDotNet.Artifacts folder.

Build and run the console application in Release mode (BenchmarkDotNet requires Release mode for accurate results):

dotnet run -c Release

Understanding the Results

After running the benchmarks, BenchmarkDotNet generates a detailed report. Here’s an example of what the console output might look like:

BenchmarkDotNet=v0.13.12, OS=Windows 11
Intel Core i7-12700H, 1 CPU, 20 logical and 14 physical cores
.NET SDK=8.0.100
  [Host]     : .NET 8.0.0, X64 RyuJIT
  DefaultJob : .NET 8.0.0, X64 RyuJIT

| Method                | Mean      | Error     | StdDev    | Median    | Rank | Allocated |
|-----------------------|-----------|-----------|-----------|-----------|------|-----------|
| ConcatUsingStringBuilder |  85.23 ns |  1.672 ns |  2.345 ns |  84.91 ns |    1 |      88 B |
| ConcatUsingString        | 235.67 ns |  4.891 ns |  6.123 ns | 234.22 ns |    2 |     336 B |

Interpreting the Output

• Mean: The average execution time (in nanoseconds). StringBuilder is significantly faster (85.23 ns) than string concatenation (235.67 ns).
• Error and StdDev: Indicate the precision and variability of the measurements. Lower values mean more consistent results.
• Median: A robust measure of central tendency, less affected by outliers.
• Rank: Shows StringBuilder is ranked 1 (faster) and string concatenation is ranked 2 (slower).
• Allocated: Memory allocations. StringBuilder allocates less memory (88 bytes) compared to string concatenation (336 bytes), which is expected due to string immutability.

The results confirm that StringBuilder is both faster and more memory-efficient, especially for larger datasets.

Exporting Results

The BenchmarkDotNet.Artifacts folder contains the markdown, CSV, and HTML reports that BenchmarkDotNet automatically creates. Exports can be modified or incorporated into CI/CD pipelines.

Best Practices for Benchmarking

1. Run in Release Mode: Debug mode includes optimizations that skew results.
2. Avoid External Interference: Run benchmarks on a quiet system without heavy background processes.
3. Use Representative Data: Test with data sizes and types that reflect real-world scenarios.
4. Repeat Benchmarks: Run multiple times to ensure consistency.
5. Analyze Memory: Use [MemoryDiagnoser] to monitor memory usage, especially for high-performance applications.

Conclusion

BenchmarkDotNet is an essential tool for .NET developers looking to measure and optimize performance. By automating the difficulties of benchmarking, you can focus on writing and comparing code. In our console application example, we found that StringBuilder outperformed string concatenation in terms of execution speed and memory utilization.

To learn more, read the BenchmarkDotNet documentation and experiment with various scenarios in your projects. Happy benchmarking!