# Network Benchmark .NET *Low Level .NET 6 Networking libraries benchmarked for UDP socket performance* ![Screenshot](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/screenshot.png) ## Table of Contents 1. [Description](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#description) 2. [Benchmark Setup](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#benchmark-setup) 3. [Benchmark Results](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#benchmark-results) 4. [Installation](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#installation) 5. [Usage](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#usage) 6. [Contributions](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#contributions) ## Description NBN is a benchmark for low level networking libraries using UDP and can be used with [Unity](https://unity3d.com) and for [.Net Core](https://en.wikipedia.org/wiki/.NET_Core) standalone server applications. The benchmark focuses on latency, performance and scalability. ### Supported Libraries * [ENet-CSharp](https://github.com/nxrighthere/ENet-CSharp) (v 2.4.8) * Wrapper for [ENet](https://github.com/lsalzman/enet), building a reliable sequenced protocol on top of UDP * Max concurrent connections are limited to 4095 due to the protocol * Packetsize overhead: 10 bytes * [Unity Client Example](https://github.com/JohannesDeml/ENetUnityMobile) * [LiteNetLib](https://github.com/RevenantX/LiteNetLib) (v 1.0.1) * Very feature-rich library * Packetsize overhead: 1 byte for unreliable, 4 bytes for reliable * [Unity Client Example](https://github.com/RevenantX/NetGameExample) * [Kcp2k](https://github.com/vis2k/kcp2k) (v 1.34.0) * Port of KCP with 100% C# Code, Future Technology for [Mirror-NG](https://github.com/MirrorNG/MirrorNG) * Packetsize overhead: 24 byte * [Unity Example](https://github.com/vis2k/kcp2k) * [NetCoreServer](https://github.com/chronoxor/NetCoreServer) (v 6.7.0) * Pure C# / .Net library for TCP/UDP/SSL with no additional protocols on top * Packetsize overhead: 0 bytes, but you have to invent the wheel yourself * [Unity Client Example](https://github.com/JohannesDeml/Unity-Net-Core-Networking-Sockets) ## Benchmark Setup ### Hardware * Ubuntu VPS * Virtual private server with dedicated CPU's running - [Hardware](https://www.netcup.eu/bestellen/produkt.php?produkt=2624) * Ubuntu 22.04.2 LTS x86-64 Kernel 5.15.0-48-generic ``` $> hostnamectl Chassis: vm Virtualization: kvm Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-48-generic Architecture: x86-64 Hardware Vendor: netcup Hardware Model: KVM Server ``` * Ubuntu Desktop / Windows Desktop * Desktop PC from 2020 - [Hardware](https://pcpartpicker.com/user/JohannesDeml/saved/zz7yK8) * Windows 11 Pro x86-64 Build 22621,1485 (22H2) * Ubuntu 22.04.2 LTS x86-64 Kernel 5.19.0-38-generic ### Software * [.NET](https://dotnet.microsoft.com/download/dotnet) 6.0.13 on Ubuntu, 6.0.15 on Windows * [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet) 0.13.5 ### Procedure For the two desktop setups, the benchmarks are run on a restarted system with 5 minutes idle time before starting the benchmark. They are run with admin privileges and all unnecessary other processes are killed before running the benchmarks. For Ubuntu VPS, the benchmarks are run through continuous integration on a typical indie server setup with other processes running as well. After the benchmarks are run, a list of all running processes to make them more reproducible. To reproduce the benchmarks, run `sudo sh linux-benchmark.sh` or `win-benchmark.bat` . If you want to execute directly from the compiled program, run `./NetworkBenchmarkDotNet -b Essential`. ## Benchmark Results The raw data and additional files can be downloaded from the [release section](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/../../releases). ### Benchmark [PingPongUnreliable](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs) Runs the benchmark with **500** clients, which pingpong **1 message** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**. This test is for getting an idea of an average roundtrip time (lower is better). ![PingPong Unreliable .NET Benchmark chart](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/nbn-pingpongunreliable-1.1.0.png) ### Benchmark [PingPongReliable](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/NetworkBenchmarkDotNet/PredefinedBenchmarks/ReliablePerformanceBenchmark.cs) Runs the benchmark with **500** clients, which pingpong **1 message** each with the server with **reliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**. This test is for getting an idea of an average roundtrip time (lower is better). ![PingPong Reliable .NET Benchmark chart](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/nbn-pingpongreliable-1.1.0.png) ### Benchmark [PingPongBatchedUnreliable](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/NetworkBenchmarkDotNet/PredefinedBenchmarks/UnreliablePerformanceBenchmark.cs) Runs the benchmark with **500** clients, which pingpong **10 messages** each with the server with **unreliable** transmission. The benchmark runs until a total of **500,000** messages are sent to the server and back to the clients. Message size is **32 bytes**. This test is for multiplexing / message merging performance (higher is better). ![PingPong Batched Unreliable .NET Benchmark chart](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/nbn-pingpongbatchedunreliable-1.1.0.png) ### Benchmark [SampleEchoSimple](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/NetworkBenchmarkDotNet/PredefinedBenchmarks/SamplingBenchmark.cs) Runs the benchmark with **1** client, which pingpong **10 messages** each with the server. The benchmark runs until a total of **100,000** messages are sent to the server and back to the clients. Message size is **128 bytes**. This test collects information about generated garbage and CPU times while running the benchmark. Those results can be analyzed with [PerfView](https://github.com/microsoft/perfview) on Windows. ### Overview This is a comparison between all tests with their message throughput (higher is better). ![Network Benchmark .NET results overview](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/nbn-overview-1.1.0.png) ### Notes * Ubuntu outperforms windows in almost all tests and libraries. * The benchmarks are run locally and therefor a lot of processing power is used for simulating the clients. Therefore, this benchmark does not include a CCU comparison, but is sufficient to compare different libraries for the same setup. * Also the network latency results do not show any costs of the network itself, but only the cost of the application layer in the round trip time. Since multiple clients are run at the same time, this time with be smaller than presented here. * Creation, Connection and Disconnection and Disposal of the Server and Clients is not included in the performance benchmarks, but is included in the .nettrace files from the Sampling benchmark. * To access the Sampling results, you can use [PerfView](https://github.com/microsoft/perfview) to open the `.nettrace` files. * Kcp2k has been recently added and might have some room for improvements. Especially using `Thread.Sleep` on Windows creates [noticeable delays](https://social.msdn.microsoft.com/Forums/vstudio/en-US/facc2b57-9a27-4049-bb32-ef093fbf4c29/threadsleep1-sleeps-for-156-ms?forum=clr). For now it is excluded of the predefined benchmarks, until its execution and cleanup are improved. * This is not a general purpose benchmark for every and any game type. Different games have different requirements. I hope these benchmarks help you as a smoke test and give you the possibility to quickly test how your hardware handles the different libraries. ## Installation Make sure you have [.Net 6 SDK](https://dotnet.microsoft.com/download) installed. Then just open the solution file with Visual Studio/Rider/Visual Studio Code and build it. Note that results of the benchmarks can be very different with a different operating system and hardware. ## Usage In general there are two different types of benchmarks: Custom and predefined benchmarks. Custom benchmarks and be defined through the [command-line options](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#command-line-options). Predefined Benchmarks are set in code and are executed through [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet). They are used for the statistics presented here and are more accurate and better reproducible then Custom benchmarks. However, they also take a longer time to finish. You can also run the clients and server on different machines to test the libraries over your local network or remotely (see [remote benchmarks](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/#remote-benchmarks)). ### Custom Benchmarks You can run custom benchmarks through the command-line. Use can test multiple settings and its combinations in an easy and quick way. The tests will just run once and are not as accurate as running a predefined benchmark. An example for running a benchmark might be `./NetworkBenchmarkDotNet --library ENet --transmission Unreliable --clients 100 --duration 10 ` #### Command-line Options (`./NetworkBenchmarkDotNet --help`) ``` Usage: NetworkBenchmarkDotNet [options] Options: -b, --benchmark Run predefined benchmarks [default: -m, --execution-mode Control what parts to run [default: Complete] -t, --test Test type [default: PingPong] --transmission Transmission type [default: Unreliable] -l, --library Library target [default: ENet] -d, --duration Test duration in seconds (-1 for manual stopping) [default: 10] --address

IP Address, can be ipv4 (e.g. 127.0.0.1) or ipv6 (e.g. ::1) [default: ::1] --port Socket Port [default: 3330] --clients # Simultaneous clients [default: 500] --parallel-messages #Parallel messages per client [default: 1] --message-byte-size Message byte size sent by clients [default: 32] --message-payload Message load sent by clients [default: Random] --verbose Verbose output of test steps and errors [default: True] --client-tick-rate Client ticks per second if supported [default: 60] --server-tick-rate Server ticks per second if supported [default: 60] --use-native-sockets Use native Sockets (LiteNetLib only) [default: True] --version Show version information -?, -h, --help Show help and usage information ``` ### Predefined Benchmarks Predefined benchmarks take some time to run, but generate reproducible numbers. The easiest way to run a predefined benchmark is to run `win-benchmark.bat` on windows or `sh linux-benchmark.sh` on windows. #### Types * **Quick** (<1min): Runs a quick benchmark with whatever is set in [QuickBenchmark.cs](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/../../blob/master/NetworkBenchmarkDotNet/PredefinedBenchmarks/QuickBenchmark.cs) * **Performance** (>15min): High Performance statistical test with all included libraries * **Sampling** (<1min): Test with all included libraries using cpu sampling and memory allocation statistics * **Essential** (>15min): Running Performance + Garbage Benchmark ![Run Predefined Benchmark windows command-line screenshot](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/Docs/run-predefined-benchmark.png) ### Remote Benchmarks To test a library remotely, you can use the parameter `--execution-mode Server` and `--execution-mode Client` respectively. This setup requires to first start the server with the correct library (and probably an indefinite execution duration) on your target server, and then the client process. Here is an example: Server: `./NetworkBenchmarkDotNet --library ENet --transmission Reliable --execution-mode Server --address YOUR_ADDRESS -d -1` Client: `./NetworkBenchmarkDotNet --library ENet --transmission Reliable --execution-mode Client --address YOUR_ADDRESS --clients 100 -d 10` If you change the address in `QuickBenchmark.cs`, you can also run a more sophisticated remote benchmark this way. ## Contributions Your favorite library is missing, or you feel like the benchmarks are not testing everything relevant? Let's evolve the benchmark together! Either hit me up via [E-mail](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/mailto:public@deml.io) to discuss your idea, or [open an issue](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/../../issues), or make a pull request directly. There are a few rules in order to not make the benchmark too cluttered. ### Adding a Library Your new proposed library ... * works with Unity as a Client * works with .NET 6 for the server * uses UDP (additional RUDP would be nice) * is Open Source (can still be commercial) * is stable enough not to break in the benchmarks * is in active development * adds value to the benchmarks #### How to add a library 1. Add a new folder inside the NetworkBenchmarkDotNet solution with the name of your library 2. Add a script called `YourLibraryBenchmark.cs` which implements [ANetworkBenchmark](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/../../blob/master/NetworkBenchmarkDotNet/Libraries/ANetworkBenchmark.cs) 3. Add your library name to the [NetworkLibrary](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/../../blob/master/NetworkBenchmarkDotNet/Libraries/NetworkLibrary.cs) enum 4. Add your Implementation Constructor to `INetworkBenchmark.CreateNetworkBenchmark()` 5. Use the `-l ` argument (or `BenchmarkSetup.Library`) to test your library and if everything works as expected. 6. Change `[Params(NetworkLibrary.Kcp2k)]` in `QuickBenchmark.cs` to your library and run `./NetworkBenchmarkDotNet -b Quick` to see, if your library works with high CCU and looping benchmarks with BenchmarkDotNet 8. Create a PR including your benchmark results 🎉 ## License [MIT](https://github.com/JohannesDeml/NetworkBenchmarkDotNet/raw/master/LICENSE)