Introducing JsonSrcGen 1.1

JsonSrcGen 1.1 has been released. This is the first release that includes UTF8 support.

Why UTF8?

UTF8 support is a big deal for a Json serializer. In c# strings are UTF16, but the vast majority of real JSON usage is UTF8, files, rest API's all use UTF8. This means a JSON serializer that uses .net strings has to do a double conversion, first from UTF8 to a .Net string (UTF16) then from the .net string to the json object. If a JSON serializer can work with UTF16 directly then a large amount of time can be saved.

Features

• UTF8 Support
• Decimal type support
• Struct and read only struct support

Code Examples

// To Json UTF8
ReadOnlySpan<byte> json = convert.ToJsonUtf8(new MyType(){MyProperty = "Some value"});

// From Json UTF8
ReadOnlySpan<byte> utf8Json = Encoding.Utf8.GetBytes("{\"MyProperty\:\"Some value\"}");
convert.FromJson(utf8Json, myType);

Benchmarks

The code for the following benchmarks can be found in github. The following class is used for these benchmarks:

public class JsonTestClass
{
    public string FirstName{get;set;}
    public string LastName{get;set;}
    public int Age{get;set;}
    public bool Registered {get;set;}
}

How to get it

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

JsonSrcGen is available as a nuget package:

https://www.nuget.org/packages/JsonSrcGen

Thanks to...

A special thanks goes out to the people who have contributed to JsonSrcGen for this release.

@trampster - Daniel Hughes

@hugobritobh - Hugo de Brito V. R. Alves

@sirh3e - Marvin Huber

@Youssef1313 - Youssef Victor

How to contribute

• Testing/bug reports
• Submit pull requests
• Github sponsorship

 

 Introducing JsonSrcGen 1.0

JsonSrcGen 1.0.3 has been released. JsonSrcGen 1.0 is a production ready stable release. 

Features

• Compile time code generation using c# Source Generators
• Serialization to and from strings
• Custom serializers
• Support for all common .net types including Lists, Arrays and Dictionarys
• High performance in both startup and runtime

Whats Next?

JsonSrcGen 1.0 will only get bug fixes going forward. Work will start on JsonSrcGen 1.1 which will add the ability to serialise to an from utf8 data. In real world applications it is more likely that you will need to work with utf8 than with strings, so JsonSrcGen 1.1 should provide a significant boost for those use cases.

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

JsonSrcGen is available as a nuget package:

https://www.nuget.org/packages/JsonSrcGen

 

JsonSrcGen 1.0.1 RC 2 Released

JsonSrcGen 1.0.1 RC 2 has been released, This is the second release candidate leading up to the first production ready release. All going well this will be the last release candidate before the first stable release.

New Features

• Added JsonOptionalAttribute to specify a property should be set to default if not present in the JSON during deserialisation.

Bug Fixes

• Fixed deserialising empty objects

JsonOptionalAttribute

Added JsonOptionalAttribute to specify a property should be set to default if not present in the JSON during deserialisation. This is only necessary if you are reusing an object for multiple deserialisations.

public class MyJsonType
{
    [JsonOptional]
    string MyProperty{get;set;}
}

Real Json Testing program

If you would like to ensure your Json API works with JsonSrcGen then you can submit your tests to be included in our RealJsonTests folder via a merge request. The JsonSrcGen developers will ensure that any tests included here pass for each new release.

Tests must meet the following criteria:

• Use nunit
• Use local json data (no external API calls from the tests)

API's are eligible for free inclusion under any of the following conditions:

• The API is available to the public free of charge.
• The server or client code is available under an OSI approved opensource licence.

If your API does not meet the above conditions please contacts the maintainers to discus how you can support the development in exchange for having your tests included.

Call to testing

Please test JsonSrcGen against your Json and raise bug reports for any problem you find.

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

JsonSrcGen is available as a nuget package:

https://www.nuget.org/packages/JsonSrcGen

JsonSrcGen 1.0.0.1 RC 1 Released

JsonSrcGen 1.0.0.1 RC 1 has been released, This is the first release candidate leading up to the first production ready release. JsonSrcGen is now feature complete.

Changes

• Serialises from ReadOnlySpan<char> instead of from string

New Features

• Support skipping null values when serialising
• Support DateTimeOffset

Serialises from ReadOnlySpan<char> instead of from string

From Json conversions now serialize from ReadOnlySpan<char> instead of string. Strings can be converted to ReadOnlySpan<char> very cheaply but converting a ReadOnlySpan<char> to a string is expensive because it requires allocating memory. c# will automatically convert string to ReadOnlySpan<char> so you can continue to use strings.

Support skipping null values when serialising

The attribute [JsonIgnoreNull] can be added to a class to instruct JsonSrcGen to skip serializing null values. 

Real Json Testing program

If you would like to ensure your Json API works with JsonSrcGen then you can submit your tests to be included in our RealJsonTests folder via a merge request. The JsonSrcGen developers will ensure that any tests included here pass for each new release.

Tests must meet the following criteria:

• Use nunit
• Use local json data (no external API calls from the tests)

API's are eligible for free inclusion under any of the following conditions:

• The API is available to the public free of charge.
• The server or client code is available under an OSI approved opensource licence.

If your API does not meet the above conditions please contacts the maintainers to discus how you can support the development in exchange for having your tests included.

Call to testing

Please test JsonSrcGen against your Json and raise bug reports for any problem you find.

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

JsonSrcGen is available as a nuget package:

https://www.nuget.org/packages/JsonSrcGen

JsonSrcGen 0.2.0 Alpha Released

JsonSrcGen 0.2.0 alpha has been published. This release contains the following changes:

Breaking Changes

• Serialise to ReadOnlySpan<char> instead of string
• JsonSrcGenConvert renamed to JsonConverter

New Features

• Support for Json Values
• Support for Custom Converters
• Support for char

Serialise to ReadOnlySpan<char> instead of string

To Json conversions now produce ReadOnlySpan<char> instead of string, is produces a significant speedup as it allows us to avoid the memory allocation required to create the String. The ReadOnlySpan reuses the same memory for each ToJson conversion within a thread. Because of this the ReadOnlySpan must be consumed before calling ToJson again or the data it points at will change.

Support for Json Values

Simple Json Values can not be convertered to and from Json. To do this you must specify a JsonValue attribute at the solution level as follows:

[assembly: JsonValue(typeof(int))]

...

var converter = new JsonConverter

ReadOnlySpan<char> json = converter.ToJson(1456);

int value = converter.FromJson(0, "1456"); 

Support for Custom Converters

Custom Converters allow you to provide a custom conversion code for a type. To do so you must implement ICustomConverter<T> and add the CustomConverter attribute to your class.

[CustomConverter(typeof(int))]
public class CustomCaseStringConverter : ICustomConverter<int>
{
    public void ToJson(IJsonBuilder builder, int value)
    {
        // Write your json to the builder here
    }

    public ReadOnlySpan<char> FromJson(ReadOnlySpan<char> json, ref int value)
    {
        // Read the Json from the json span here
    }
}

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

JsonSrcGen is available as a nuget package:

https://www.nuget.org/packages/JsonSrcGen

JsonSrcGen + CoreRT = Pure Magic

In my previous post I talked about how using a Source Generator allows us to make a reflection free Json generator. This has serious advantages when it comes deployment size and startup time.

These advantages become even bigger when paired with an AOT or Ahead Of Time compiler like CoreRT. CoreRT can produce impressively small binaries but cannot do Reflection.Emit and has only limited support for reflection. But seeing as JsonSrcGen doesn't do any reflection the two are a match made in heaven.

For me this produces a binary that is only 2.1 MB and will startup and serialise a simple Json class in only 5 ms.

In order to see just how good this is here a comparison to what you can do with the officially supported .NET Json and self contained publishing options. Which included the new Trimming and Ready To Run options.

How to use JsonSrcGen with CoreRT

1. Create a new .NET 5 console project

dotnet new console

2. Add the CoreRT Package Source

CoreRT requires a custom package source to be added to your nuget.config. I didn't have one so added one using: 

dotnet new nuget

Then add the dotnet-core Package Source to your nuget config so it looks as follows:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="dotnet-core" value="https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json" />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
</configuration>

3. Add the CoreRT and JsonSrcGen nuget packages

dotnet add package JsonSrcGen

dotnet add package Microsoft.DotNet.ILCompiler

4. Add the following lines to you .csproj file

<LangVersion>preview</LangVersion>
<IlcDisableReflection>true</IlcDisableReflection>
<IlcInvariantGlobalization>true</IlcInvariantGlobalization>

The first line is required because Source Generators are currently in preview, this will not be required once they are released.

The Second line tells CoreRT that we do not need reflection, this is a significant saving in binary size.

The final line tells CoreRT that we do not need localisation support.

5. Install prerequisites

What you need here depends on your platform, but for me on linux I needed the following:

sudo apt-get install clang zlib1g-dev libkrb5-dev libtinfo5

6. Then build the project to get your output

dotnet publish -r linux-x64 -c Release

The -r linux-x64 tells CoreRT to build for linux 64 bit, if you are on a different platform you will need to change this.

7. If you are on linux use the strip command to get rid of debug information

This step is only required on linux, because the debug information ends up in the binary on linux.

strip bin/Release/net5.0/linux-x64/publish/CoreRTSample

Please checkout the JsonSrcGen project on github: https://github.com/trampster/JsonSrcGen

You can find a JsonSrcGen CoreRT sample project in our samples folder

Introducing JsonSrcGen

Over the last month I've been working on a new Json Library for .net that takes advantage of the new c# Source Generator feature that is being previewed in .NET 5. I've called this JsonSrcGen which is short for Json Source Generator.

Why yet another Json library?

Source Generators allow us to generate optimal Json serialization code at compile time. Up until now Json Libraries have made heavy use of Reflection and Reflection.Emit to produce serialization code at runtime. But this approach has some draw backs.

• Startup times are slow because of the reflection and time to emit the serialization code.
• AOT (Ahead of Time) platforms like Xamarin.iOS, Blazor and CoreRT struggle with Reflection and cannot do Reflection.Emit. Even if they can do reflection they can get significant size savings if they exclude it
• They require slow runtime lookups to match Types to generated Serializers

How do I use it?

Source Generators currently require .NET 5 and LangVersion set to 'preview'

<TargetFramework>net5.0</TargetFramework>
<LangVersion>preview</LangVersion>

Add a nuget reference to our nuget package

Annotate your c# class with JsonSrcGen Attributes:

[Json]
public class MyType
{
    public int Age {get;set}

    [JsonName("my_property")]
    public string MyProperty {get;set}   

    [JsonIgnore]
    public string IgnoredProperty {get;set;}
}

Then use JsonSrcGenConvert instance to convert to and from Json strings.

var converter = new JsonSrcGenConvert();

//convert from Json
MyType myType = new MyType();
converter.FromJson(myType, jsonString);

//convert to json
jsonString = converter.ToJson(new MyType());

Notice that when converting from Json you supply an already instantiated instance of the type, this allows you to reuse type instances and thus reduce memory allocations. Which adds up to a significant performance boost while deserializing.

How fast is it?

I am not going to make any claims about the performance of JsonSrcGen. Many .net serializers have made claims about being the fastest only to have their claims age very poorly. Instead I will just say that JsonSrcGen has a strong focus on performance. Below is a benchmark of serialization and deserialization of a simple class. However I strongly advise you to benchmark against your own types and your own use case to determine which is best for you.

How do I get it?

JsonSrcGen is available on Nuget in here

The code is available on github here under the MIT license.

JsonSrcGen is currently in alpha quality and should not be used for production code. However feel free to try it out and report any issues you have with it.

Jsonics 0.2.0-alpha Released

Jsonics 0.2.0-alpha Release

Jsonics 0.2.0-alpha contains a bunch of new features and gets us a lot closer to being ready for our first release.

New Features

The following new features are present in Jsonics 0.2.0-alpha

Ignore Support

.net properties and fields can be excluded from Json serialisation or deserialisation using the new Ignore attribute. Simply decorate the property with the Ignore attribute as follows:

public Person
{
    [Ignore]
    public int Age
    {
        get;
        set;
    }
}

Property Name Support

.net properties and fields can now have different names from their json counterparts to achieve this use the new Name attribute as follows:

public Person
{
    [Name("experience"]
    public int Age
    {
        get;
        set;
    }
}

.net Standard 2.0

Jsonics has been moved to .net Standard 2.0. This allows for a much broader api space and will be supported everywhere .net is.

.net Field support

Jsonics now supports .net fields as well as .net properties.

public Person
{
    public int Age;
}

Char support

Char support was missing in the last release and has now been added.

Decimal support

Decimal support was missing in the last release and has now been added.

Nuget Changes

The Jsonics.StrongName packages has been unlisted and will not get any further updates. The Jsonics package is now StrongNamed. However the assembly version will be frozen to the major version in order to remove the need for Binding redirects. The File Version and Package version will now follow SemVer 2.0 rules. It was necessary to remove the Jsonics.StrongName package to avoid the situation where two libraries have a dependency on on Jsonics one using the StrongName version and one using the standard version.

Bug Fixes

• Fixed support from primitive list and array types other than int and string

Whats Next?

The road to 1.0 will continue with the addition of the following features which will be included in the 0.3 release:

• Custom converter support
• IList support

How do I get it?

Jsonics is available on Nuget in here

The code is available on github here under the MIT license.

Jsonics is currently in alpha quality and should not be used for production code. However feel free to try it out and report any issues you have with it.

Introducing Jsonics

Over the past months I have been working on a new opensource Json library for .net called Jsonics.
I am now pleased to announce its first release 0.1.0-alpha.

What is Jsonics?

Jsonics is a high performance Json Library for c#. Jsonics stands for JSON In C Sharp. Jsonics aims to be as fast as is possible in scenarios where the same type will be serialised or deserialised many times. Jsonics uses runtime code generation and other advanced techniques to create an optimal serialiser and deserialiser based on the supplied type.

How do I use it?

//create an optimised Json converter for type Person
var jsonConverter = JsonicFactory.Compile();

//serilize a person instance 
string jsonstring = jsonCoverter.ToJson(new Person(){"FirstName"="Luke", "LastName"="Skywalker"});

//deserialise a person json string
Person person = jsonCoverter.FromJson(jsonString);

How fast is it?

Nearlly all Json parsers claim to be the fast. Here are some claims by popular .net Json libraries.

Newtonsoft (Json.net):

• high-performance JSON framework for .NET
• 50% faster than DataContractJsonSerializer, and 250% faster than JavaScriptSerializer.

NetJson

• Faster than Any Binary?

JIL

• Jil aims to be the fastest general purpose JSON (de)serializer for .NET.

I don't want to make any claims about speed. These tend to either be proven wrong or to fail to age well. However Jsonics does have a focus on speed and currently performs well in relation to other .net Json libraries. The following is a benchmark created using the awesome BenchmarkDotNet  library.

I have uploaded the benchmark code to github here if you are interested in details.

Please bare in mind that performance is very dependent on your use case, so it is best to benchmark with your own workload before choosing a library.

Features

Jsonics has support for most commonly used .net types. If you find a type you need is missing please raise an issue and I will add it.

Limitations

Jsonics is a pre 1.0 alpha release and as such is missing some features, these will be added before Jsonics will be deemed production ready. These include:

• Ignore support for properties
• Custom ToJson/FromJson converters
• Decimal and char type support

How do I get it?

Jsonics is available on Nuget in both Signed and Unsigned versions.

The code is available on github here under the MIT license.

Jsonics is currently in alpha quality and should not be used for production code. However feel free to try it out and report any issues you have with it.

Wide Margin 1.2.17 Release Candidate

I am pleased to announce the release of Wide Margin 1.2.17 Release Candidate 

New Features include:

• New history bible book category and missing books from poetry section.
• Added bible module copyright to about info.
• Improved error handling and retry on bible module install.

Right now we need lots of testing done to ensure that this release candidate is ready for release. I want to say a big thank you to all the people who contributed to this release to make it possible. Wide Margin is free and opensource and is produced entirely by volunteers. Wide Margin is available for Ubuntu and Windows destkop.

To download or help out Wide Margin visit our website widemargin.org

Wide Margin 1.2.13 Beta

I am pleased to announce the release of Wide Margin 1.2.13 Beta 

New Features include:

• Browse button for passage navigation added to the top bar.
• Menu icon updated to be more recognizable.
• Fixed menu placement on dual monitor when maximized.

 
Right now we need lots of testing done to ensure that it is ready for release. I want to say a big thank you to all the people who contributed to this release to make it possible.

Wide Margin is free and opensource and is produced entirely by volunteers. Wide Margin is available for Ubuntu and Windows destkop.

To download or help out Wide Margin visit our website widemargin.org

Wide Margin 1.2.11 Beta

I am pleased to announce the release of Wide Margin 1.2.11 Beta the free and opensource bible application for Ubuntu and Windows.

Wide Margin 1.2.11 Beta include bugs fixes as well as a 'Open Link in New Tab' menu item for those times you don't have a mouse.
New Features include:

•  'Open Link in New Tab' menu item.
•  Improved auto complete list.

Right now we need lots of testing done to ensure that it is ready for release. I will be organizing a testing evening in the next few weeks. I want to say a big thank you to all the people who contributed to this release to make it possible.

Wide Margin is free and opensource and is produced entirely by volunteers. If you would like to help or contribute please visit our contribute page:

Wide Margin 1.2.9 Beta

I am pleased to announce the release of Wide Margin 1.2.9 Beta 

Wide Margin 1.2.9 Beta includes search and passage rendering improvements as well as bug fixes for the windows version. 
New Features include:

•  Show search results for book names which also match passage content.
•  Highlight search terms in search results.
•  Better reference rendering
•  Improved poetry layout

Right now we need lots of testing done to ensure that it is ready for release. I will be organizing a testing evening in the next few weeks. I want to say a big thank you to all the people who contributed to this release to make it possible.

Wide Margin is free and opensource and is produced entirely by volunteers. If you would like to try it or help contribute please visit our website at http://widemargin.org

Wide Margin 1.2.7 Alpha 2

I am pleased to announce the release of Wide Margin 1.2.7 Alpha 2 

Wide Margin 1.2.7 Alpha 2 includes a much improved passage text layout with headings, module references and note support. On windows the GTK UI has been replaced with a native windows user interface using WPF. 

New Features include:

•  Section headings
•  Improved text layout.
•  References/notes in bible passages
•  Native UI on Windows

I want to say a big thank you to all the people who contributed to this release to make it possible.

Wide Margin is free and opensource and is produced entirely by volunteers. New contributions are welcome. Even if you can't code you can contribute by testing, raising bug reports, contributing artwork and mock-ups.

You can download it from our website widemargin.org

Wide Margin 1.2.2 Alpha 1

I am please to announce the release of Wide Margin 1.2.2 Alpha 1

Wide Margin 1.2.2 Alpha 1 is the first release to support a Sword module. Currently only one sword module is supported, the ESV (English Standard Version).
The ESV was chosen because it it is modern and accurate. Support for the KJV will be coming soon.

New Features include:

•  ESV sword module support.
•  Paragraph layout instead of verse per line.
•  First searching using lucene.net

This was made possible through our new SharpSword backend. Which I will blog about later.

I want to say a big thank you to all the people who contributed to this release to make it possible.

Wide Margin is free and opensource and is produced entirely by volunteers. If you would like to help or contribute please visit our contribute page:

Wide margin 1.2 Alpha 1 can be downloaded from our website at http://widemargin.org/Download

Subscript and Superscript in GTK#

In Wide Margin I had a need to use superscript for verse numbers. This is quite common in printed bibles because it reduces the impact of the verse number on the readability of the text. In Wide Margin I use a Gtk TextView to display my text, formatting in a Gtk TextView is achieved using TextTags which specify formatting for a range in the text.

A TextTag does not have a property for subscript or superscript rather it has a property call Rise. Setting this to a positive value moves the text up and setting it to negative moves the text below the line.

In GTK3 the value of Rise is in pixels, in GTK2 the value of Rise is in Pango Units, There are 1024 pango units in a Pixel. This confused me for quite a while because I was reading the GTK3 docs and assuming a unit of Pixels, but I was using GTK2. I was trying to Rise my text by 5 units which in GTK2 is smaller then what can be rendered.

 Setting Rise will move the character up or down but will leave the size unaffected. To get superscript or subscript you will also need to reduce the font size.

Superscript (GTK2) Subscript (GTK2)
The exact amount you want to move change the size and move the the text up or down will depend on how you want it to look. So have a play until you get it right.

Wide Margin 1.1 Released

I am please to announce the release of Wide Margin 1.1

Wide Margin 1.1 comes with a load of usability improvements and bug fixes.

New Features include:

•  Mouse driven passage lookup
•  Filtered Searches
•  Continuous Scrolling for search results
•  Target verse highlighting

I want to say a big thank you to all the people who contributed to this release to make it possible.

The application can be downloaded here and supports both ubuntu and windows.

Wide Margin is free and opensource and is produced entirely by volunteers. If you would like to help or contribute please visit our website:

http://widemargin.org

Invoking on the UI thread in GTK#

Not Threadsafe
GTK like most GUI toolkits is not thread safe. This means that you must only ever update it form the UI thread. If we want our applications to be responsive however we must run any long running processing on a different thread so that the UI thread is free to respond to user input. This posses a problem because we need to be able to update the UI to reflect the progress and outcome of the processing, but we can't do that directly because the UI is not threadsafe.

How to update from another thread without updating from another thread
Fortunitely GTK# has a built in mechanizm for running code on the UI thread from another thread. This mechanism is call Application.Invoke and you can call it like this.


Application.Invoke((_,__) =>
{
    // update UI here
});


This method however is suboptimal, firstly because Application.Invoke takes two parameters (object sender, EventArg args). In most cases we don't need these and so in my example I have used _ for the sender and __ for the args. The second problem is that we only really need to call this if we are not already on the UI thread. We don't want to take on the penalty of the overhead if we are already on the UI thread.

There must be a better way
In winforms useful methods are provided which solve the problems presented by Application.Invoke.

The first is Control.InvokeRequired this will return true if we are not on the UI thread or false if we are.

The second is Control.Invoke this is like Application.Invoke but without the overhead of having to specify a sender and args and can be called like this:

Control.Invoke(()  =>
{
    //update UI here
});

By combining these you can write code that only invokes when it needs to. The details of this can be found here:
http://msdn.microsoft.com/en-us/library/ms171728(v=vs.80).aspx

Can we do that in GTK?
Yes we can. We can achive exactly that using Extension Methods. Actually we can improve on it by making the Invoke method only call Application.Invoke if we are not already on the UI thread. The following code is provided under the Unicorn licence.

Doing this means that what was:

Application.Invoke((_,__) =>
{
    // update UI here
});

Becomes:

Invoke(()  =>
{
    //update UI here
});

Which is cleaner and takes care of figuring out what thread your are on for you.

Wide Margin 1.1 hits Beta

I am please to announce that the 1.1 release of Wide Margin has hit Beta.

Changes since 1.0 include:

• Mouse friendly passage navigation and filtering.
• Search Filtering.
• Continuous scrolling on search results.
• Windows support.

How ca you get it?

The windows version can be downloaded from here:

https://bitbucket.org/trampster/widemargin/downloads

The ubuntu version can be installed from our PPA:

ppa:trampster/widemargin

How can you help?

At this stage we need testers. We need people to install and use Wide Margin. If you find any problem please raise them on our bug tracker here:

https://bitbucket.org/trampster/widemargin/issues?status=new&status=open

Wide Margin's first full release.

Wide Margin has reached a major milestone, it has had it's first full release, Wide Margin 1.0.9.

Wide Margin is a simple and quick bible application for ubuntu and has the following features:

• As you type searching and passage navigation
• Out of the box King James Version support
• Full navigation history
• Familiar browser inspired interface
• Daily reading planner
• Ubuntu PPA

You can install it now using our Ubuntu PPA ppa:trampster/widemargin

We welcome bug reports, patches, UI mockup and any other way you might want to contribute. Please visit our website https://bitbucket.org/trampster/widemargin/wiki/Home to get started

Going forward we plan to start development on release 2 which will bring Wide Margin to windows. Release 1 will continue to get bug fixes and regular updates until Release 2 is complete.