Utf8Json - Fast JSON Serializer for C# === Definitely Fastest and Zero Allocation JSON Serializer for C#(.NET, .NET Core, Unity and Xamarin), this serializer write/read directly to UTF8 binary so boostup performance. And I adopt the same architecture as the fastest binary serializer, [MessagePack for C#](https://github.com/neuecc/MessagePack-CSharp) that I've developed. ![image](https://user-images.githubusercontent.com/46207/30883721-11e0526e-a348-11e7-86f8-efff85a9afe0.png) > This benchmark is convert object to UTF8 and UTF8 to object benchmark. It is not to string(.NET UTF16), so [Jil](https://github.com/kevin-montrose/Jil/), [NetJSON](https://github.com/rpgmaker/NetJSON/) and [Json.NET](https://github.com/JamesNK/Newtonsoft.Json) contains additional UTF8.GetBytes/UTF8.GetString call. **Definitely** means does not exists encoding/decoding cost. Benchmark code is in [sandbox/PerfBenchmark](https://github.com/neuecc/Utf8Json/tree/master/sandbox/PerfBenchmark) by [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet). > I've tested more benchmark - [Benchmark of Jil vs Utf8Json](https://github.com/neuecc/Jil) for test many dataset patterns(borrwed from Jil Benchmark) and three input/output compare(`Object <-> byte[](https://github.com/neuecc/Utf8Json/raw/master/Utf8), Object <-> Stream(Utf8) and Object <-> String(UTF16)`). If target is UTF8(both `byte[]` and `Stream`), Utf8Json wins and memory allocated is extremely small. Utf8Json does not beat MessagePack for C#(binary), but shows a similar memory consumption(there is no additional memory allocation) and achieves higher performance than other JSON serializers. The crucial difference is that read and write directly to UTF8 binaries means that there is no overhead. Normaly serialization requires serialize to `Stream` or `byte[]`, it requires additional UTF8.GetBytes cost or StreamReader/Writer overhead(it is very slow!). ```csharp TargetClass obj1; // Object to UTF8 byte[] [Benchmark] public byte[] Utf8JsonSerializer() { return Utf8Json.JsonSerializer.Serialize(obj1, jsonresolver); } // Object to String to UTF8 byte[] [Benchmark] public byte[] Jil() { return utf8.GetBytes(global::Jil.JSON.Serialize(obj1)); } // Object to Stream with StreamWriter [Benchmark] public void JilTextWriter() { using (var ms = new MemoryStream()) using (var sw = new StreamWriter(ms, utf8)) { global::Jil.JSON.Serialize(obj1, sw); } } ``` For example, the `OutputFormatter` of [ASP.NET Core](https://github.com/aspnet/Home) needs to write to Body(`Stream`), but using [Jil](https://github.com/kevin-montrose/Jil)'s TextWriter overload is slow. (This not means Jil is slow, for example `StreamWriter` allocate many memory(`char[1024]` and `byte[3075]`) on constructor ([streamwriter.cs#L203-L204](https://github.com/Microsoft/referencesource/raw/master/mscorlib/system/io/streamwriter.cs#L203-L204)) and other slow features unfortunately). ```csharp // ASP.NET Core, OutputFormatter public class JsonOutputFormatter : IOutputFormatter //, IApiResponseTypeMetadataProvider { const string ContentType = "application/json"; static readonly string[] SupportedContentTypes = new[] { ContentType }; public Task WriteAsync(OutputFormatterWriteContext context) { context.HttpContext.Response.ContentType = ContentType; // Jil, normaly JSON Serializer requires serialize to Stream or byte[]. using (var writer = new StreamWriter(context.HttpContext.Response.Body)) { Jil.JSON.Serialize(context.Object, writer, _options); writer.Flush(); return Task.CompletedTask; } // Utf8Json // Utf8Json.JsonSerializer.NonGeneric.Serialize(context.ObjectType, context.HttpContext.Response.Body, context.Object, resolver); } } ``` The approach of directly write/read from JSON binary is similar to [corefxlab/System.Text.Json](https://github.com/dotnet/corefxlab/tree/master/src/System.Text.Json/System/Text/Json) and [corefxlab/System.Text.Formatting](https://github.com/dotnet/corefxlab/wiki/System.Text.Formatting). But it is not yet finished and not be general serializer. Corefxlab has [UTF8String](https://github.com/dotnet/corefxlab/issues/1751) and C# discussing [UTF8String Constants](https://github.com/dotnet/csharplang/issues/909) but maybe it is far future. Install and QuickStart --- The library provides in NuGet except for Unity. Standard library availables for .NET Framework 4.5 and .NET Standard 2.0. ``` Install-Package Utf8Json ``` And official Extension Packages for support other library(ImmutableCollection) or binding for framework(ASP.NET Core MVC). ``` Install-Package Utf8Json.ImmutableCollection Install-Package Utf8Json.UnityShims Install-Package Utf8Json.AspNetCoreMvcFormatter ``` NuGet page links - [Utf8Json](https://www.nuget.org/packages/Utf8Json), [Utf8Json.ImmutableCollection](https://www.nuget.org/packages/Utf8Json.ImmutableCollection), [Utf8Json.UnityShims](https://www.nuget.org/packages/Utf8Json.UnityShims), [Utf8Json.AspNetCoreMvcFormatter](https://www.nuget.org/packages/Utf8Json.AspNetCoreMvcFormatter) for Unity, you can download from [releases](https://github.com/neuecc/Utf8Json/releases) page. There providing .unitypackage. Unity support details, see [Unity section](https://github.com/neuecc/Utf8Json#for-unity). You can find third-party extension package like [Utf8Json.FSharpExtensions](https://github.com/pocketberserker/Utf8Json.FSharpExtensions) for F# types, [NServiceBus.Utf8Json](https://www.nuget.org/packages/NServiceBus.Utf8Json/) or others. QuickStart, you can call `Utf8Json.JsonSerializer`.`Serialize/Deserialize`. ```csharp var p = new Person { Age = 99, Name = "foobar" }; // Object -> byte[] (UTF8) byte[] result = JsonSerializer.Serialize(p); // byte[] -> Object var p2 = JsonSerializer.Deserialize(result); // Object -> String var json = JsonSerializer.ToJsonString(p2); // Write to Stream JsonSerializer.Serialize(stream, p2); ``` In default, you can serialize all public members. You can customize serialize to private, exclude null, change DateTime format(default is ISO8601), enum handling, etc. see the [Resolver](https://github.com/neuecc/Utf8Json#resolver) section. Performance of Serialize --- This image is what code is generated when object serializing. ![image](https://user-images.githubusercontent.com/46207/30877807-c7f264d8-a335-11e7-91d8-ad1029d4ae86.png) ```csharp // Disassemble generated serializer code. public sealed class PersonFormatter : IJsonFormatter { private readonly byte[][] stringByteKeys; public PersonFormatter() { // pre-encoded escaped string byte with "{", ":" and ",". this.stringByteKeys = new byte[][] { JsonWriter.GetEncodedPropertyNameWithBeginObject("Age"), // {\"Age\": JsonWriter.GetEncodedPropertyNameWithPrefixValueSeparator("Name") // ,\"Name\": }; } public sealed Serialize(ref JsonWriter writer, Person person, IJsonFormatterResolver jsonFormatterResolver) { if (person == null) { writer.WriteNull(); return; } // WriteRawX is optimize byte->byte copy when we know src size. UnsafeMemory64.WriteRaw7(ref writer, this.stringByteKeys[0]); writer.WriteInt32(person.Age); // itoa write directly to avoid ToString + UTF8 encode UnsafeMemory64.WriteRaw8(ref writer, this.stringByteKeys[1]); writer.WriteString(person.Name); writer.WriteEndObject(); } // public unsafe Person Deserialize(ref JsonReader reader, IJsonFormatterResolver jsonFormatterResolver) } ``` Object to JSON's main serialization cost is write property name. Utf8Json create cache at first and after that only do memory copy. Optimize part1, concatenate "{", ":" and "." to cached propertyname. Optimize part2, use optimized custom memory copy method(see: [UnsafeMemory.cs](https://github.com/neuecc/Utf8Json/raw/f724c83986d7c919a336c63e55f5a5886cca3575/src/Utf8Json/Internal/UnsafeMemory.cs)). Normally memory copy is used `Buffer.BlockCopy` but it has some overhead when target binary is small enough, releated to [dotnet/coreclr - issue #9786 Optimize Buffer.MemoryCopy](https://github.com/dotnet/coreclr/pull/9786) and [dotnet/coreclr - Add a fast path for byte[] to Buffer.BlockCopy #3118](https://github.com/dotnet/coreclr/pull/3118). Utf8Json don't use `Buffer.BlockCopy` and generates length specialized copy code that can reduce branch cost. Number conversion is often high cost. If target encoding is UTF8 only, we can use `itoa` algorithm so avoid `int.ToString` and UTF8 encode cost. Especialy double-conversion, Utf8Json ported [ google/double-conversion](https://github.com/google/double-conversion) algorithm, it is fast `dtoa` and `atod` works. Other optimize techniques. * High-level API uses internal memory pool, don't allocate working memory under 64K * Struct JsonWriter does not allocate any more and write underlying byte[] directly, don't use TextWriter * Avoid boxing all codes, all platforms(include Unity/IL2CPP) * Heavyly tuned dynamic IL code generation, it generates per option so reduce option check: see:[DynamicObjectResolver.cs](https://github.com/neuecc/Utf8Json/raw/a8203cb4cfd81a8462105dac5babf5e26fa01ac5/src/Utf8Json/Resolvers/DynamicObjectResolver.cs#L729-L1292) * Call Primitive API directly when IL code generation knows target is primitive * Getting cached generated formatter on static generic field(don't use dictionary-cache because lookup is overhead) * Don't use `IEnumerable` abstraction on iterate collection, specialized each collection types, see:[CollectionFormatter.cs](https://github.com/neuecc/Utf8Json/raw/f724c83986d7c919a336c63e55f5a5886cca3575/src/Utf8Json/Formatters/CollectionFormatters.cs) * Uses optimized type key dictionary for non-generic methods, see: [ThreadsafeTypeKeyHashTable.cs](https://github.com/neuecc/Utf8Json/raw/f724c83986d7c919a336c63e55f5a5886cca3575/src/Utf8Json/Internal/ThreadsafeTypeKeyHashTable.cs) Performance of Deserialize --- When deserializing, requires property name to target member name matching. Utf8Json avoid string key decode for matching, generate [automata](https://en.wikipedia.org/wiki/Automata_theory) based IL inlining code. ![image](https://user-images.githubusercontent.com/46207/29754771-216b40e2-8bc7-11e7-8310-1c3602e80a08.png) use raw byte[] slice and try to match each `ulong type` (per 8 character, if it is not enough, pad with 0). ```csharp // Disassemble generated serializer code. public sealed class PersonFormatter : IJsonFormatter { // public sealed Serialize(ref JsonWriter writer, Person person, IJsonFormatterResolver jsonFormatterResolver) public unsafe Person Deserialize(ref JsonReader reader, IJsonFormatterResolver jsonFormatterResolver) { if (reader.ReadIsNull()) return null; reader.ReadIsBeginObjectWithVerify(); // "{" byte[] bufferUnsafe = reader.GetBufferUnsafe(); int age; string name; fixed (byte* ptr2 = &bufferUnsafe[0]) { int num; while (!reader.ReadIsEndObjectWithSkipValueSeparator(ref num)) // "}" or "," { // don't decode string, get raw slice ArraySegment arraySegment = reader.ReadPropertyNameSegmentRaw(); byte* ptr3 = ptr2 + arraySegment.Offset; int count = arraySegment.Count; if (count != 0) { // match automata per long ulong key = AutomataKeyGen.GetKey(ref ptr3, ref count); if (count == 0) { if (key == 6645569uL) { age = reader.ReadInt32(); // atoi read directly to avoid GetString + int.Parse continue; } if (key == 1701667150uL) { name = reader.ReadString(); continue; } } } reader.ReadNextBlock(); } } return new PersonSample { Age = age, Name = name }; } } ``` Of course number conversion(decode to string -> try parse) is high cost. Utf8Json directly convert byte[] to number by atoi/atod algorithm. Built-in support types --- These types can serialize by default. Primitives(`int`, `string`, etc...), `Enum`, `Nullable<>`, `TimeSpan`, `DateTime`, `DateTimeOffset`, `Guid`, `Uri`, `Version`, `StringBuilder`, `BitArray`, `Type`, `ArraySegment<>`, `BigInteger`, `Complext`, `ExpandoObject `, `Task`, `Array[]`, `Array[,]`, `Array[,,]`, `Array[,,,]`, `KeyValuePair<,>`, `Tuple<,...>`, `ValueTuple<,...>`, `List<>`, `LinkedList<>`, `Queue<>`, `Stack<>`, `HashSet<>`, `ReadOnlyCollection<>`, `IList<>`, `ICollection<>`, `IEnumerable<>`, `Dictionary<,>`, `IDictionary<,>`, `SortedDictionary<,>`, `SortedList<,>`, `ILookup<,>`, `IGrouping<,>`, `ObservableCollection<>`, `ReadOnlyOnservableCollection<>`, `IReadOnlyList<>`, `IReadOnlyCollection<>`, `ISet<>`, `ConcurrentBag<>`, `ConcurrentQueue<>`, `ConcurrentStack<>`, `ReadOnlyDictionary<,>`, `IReadOnlyDictionary<,>`, `ConcurrentDictionary<,>`, `Lazy<>`, `Task<>`, custom inherited `ICollection<>` or `IDictionary<,>` with paramterless constructor, `IEnumerable`, `ICollection`, `IList`, `IDictionary` and custom inherited `ICollection` or `IDictionary` with paramterless constructor(includes `ArrayList` and `Hashtable`), `Exception` and inherited exception types(serialize only) and your own class or struct(includes anonymous type). Utf8Json has sufficient extensiblity. You can add custom type support and has some official/third-party extension package. for example ImmutableCollections(`ImmutableList<>`, etc), [Utf8Json.FSharpExtensions](https://github.com/pocketberserker/Utf8Json.FSharpExtensions)(FSharpOption, FSharpList, etc...). Please see [extensions section](https://github.com/neuecc/Utf8Json#extensions). Object Serialization --- Utf8Json can serialze your own public `Class` or `Struct`. In default, serializer search all public instance member(field or property) and uses there member name as json property name. If you want to avoid serialization target, you can use `[IgnoreDataMember]` attribute of `System.Runtime.Serialization` to target member. If you want to change property name, you can use `[DataMember(Name = string)]` attribute of `System.Runtime.Serialization`. ```csharp // JsonSerializer.Serialize(new FooBar { FooProperty = 99, BarProperty = "BAR" }); // Result : {"foo":99} public class FooBar { [DataMember(Name = "foo")] public int FooProperty { get; set; } [IgnoreDataMember] public string BarProperty { get; set; } } ``` Utf8Json has other option, allows private/internal member serialization, convert property name to camelCalse/snake_case, if value is null does not create property. Or you can use a different DateTime format(default is ISO8601). The details, please read [Resolver](https://github.com/neuecc/Utf8Json#resolver) section. Here is sample. ```csharp // default serializer change to allow private/exclude null/snake_case serializer. JsonSerializer.SetDefaultResolver(StandardResolver.AllowPrivateExcludeNullSnakeCase); var json = JsonSerializer.ToJsonString(new Person { Age = 23, FirstName = null, LastName = "Foo" }); // {"age":23,"last_name":"Foo"} Console.WriteLine(json); ``` Serialize ImmutableObject(SerializationConstructor) --- Utf8Json can deserialize immutable object like this. ``` public struct CustomPoint { public readonly int X; public readonly int Y; public CustomPoint(int x, int y) { this.X = x; this.Y = y; } } ``` Utf8Json choose constructor with the most matched argument by name(ignore case). > MessagePack for C# choose `least` matched argument, please be aware of the opposite. This is design miss of MessagePack for C#. If can not match automatically, you can specify to use constructor manually by `[SerializationConstructorAttribute]`. ```csharp public class CustomPoint { public readonly int X; public readonly int Y; public CustomPoint(int x, int y) { this.X = x; this.Y = y; } // used this constructor. [SerializationConstructor] public CustomPoint(int x) { this.X = x; } } ``` ShouldSerializeXXX pattern --- UtfJson supports [ShouldSerialize feature of Json.NET](https://www.newtonsoft.com/json/help/html/ConditionalProperties.htm). If defined `public bool ShouldMemberName()` method, call method before serialize member value and if false does not output member. ```csharp public class MyPerson { public string Name { get; set; } public string[] Addresses { get; set; } // ShouldSerialize*membername** // method must be `public` and return `bool` and parameter less. public bool ShouldSerializeAddresses() { if (Addresses != null && Addresses.Length != 0) { return true; } else { return false; } } } -- // {"Name":"foo"} JsonSerializer.ToJsonString(new MyPerson { Name = "foo", Addresses = new string[0] }); // {"Name":"bar","Addresses":["tokyo","kyoto"]} JsonSerializer.ToJsonString(new MyPerson { Name = "bar", Addresses = new[] { "tokyo", "kyoto" } }); ``` Dynamic Deserialization --- If use JsonSerializer.Deserialize