simdjson : Parsing gigabytes of JSON per second

JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms to parse JSON 2.5x faster than anything else out there.

• Fast: Over 2.5x faster than other production-grade JSON parsers.
• Easy: First-class, easy to use API.
• Strict: Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises.
• Automatic: Selects a CPU-tailored parser at runtime. No configuration needed.
• Reliable: From memory allocation to error handling, simdjson's design avoids surprises.

This library is part of the Awesome Modern C++ list.

Table of Contents

• Quick Start
• Documentation
• Performance results
• Real-world usage
• Bindings and Ports of simdjson
• About simdjson
• Funding
• Contributing to simdjson
• License

Quick Start

The simdjson library is easily consumable with a single .h and .cpp file.

0. Prerequisites: g++ or clang++.

1. Pull simdjson.h and simdjson.cpp into a directory, along with the sample file twitter.json.

   wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json
   

2. Create parser.cpp:

   #include "simdjson.h"
   int main(void) {
     simdjson::document::parser parser;
     simdjson::document& tweets = parser.load("twitter.json");
     std::cout << tweets["search_metadata"]["count"] << " results." << std::endl;
   }
   

3. c++ -o parser parser.cpp simdjson.cpp -std=c++17

4. ./parser

   100 results.
   

Documentation

Usage documentation is available:

• Basics is an overview of how to use simdjson and its APIs.
• Performance shows some more advanced scenarios and how to tune for them.
• Implementation Selection describes runtime CPU detection and how you can work with it.

Performance results

The simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON and fifty percent less than sajson. To our knowledge, simdjson is the first fully-validating JSON parser to run at gigabytes per second on commodity processors.

On a Skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows.

parser	GB/s
simdjson	2.2
RapidJSON encoding-validation	0.51
RapidJSON encoding-validation, insitu	0.71
sajson (insitu, dynamic)	0.70
sajson (insitu, static)	0.97
dropbox	0.14
fastjson	0.26
gason	0.85
ultrajson	0.42
jsmn	0.28
cJSON	0.34
JSON for Modern C++ (nlohmann/json)	0.10

Real-world usage

• Microsoft FishStore
• Yandex ClickHouse
• Clang Build Analyzer

If you are planning to use simdjson in a product, please work from one of our releases.

Bindings and Ports of simdjson

We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).

• ZippyJSON: Swift bindings for the simdjson project.
• pysimdjson: Python bindings for the simdjson project.
• simdjson-rs: Rust port.
• simdjson-rust: Rust wrapper (bindings).
• SimdJsonSharp: C# version for .NET Core (bindings and full port).
• simdjson_nodejs: Node.js bindings for the simdjson project.
• simdjson_php: PHP bindings for the simdjson project.
• simdjson_ruby: Ruby bindings for the simdjson project.
• simdjson-go: Go port using Golang assembly.
• rcppsimdjson: R bindings.

About simdjson

The simdjson library takes advantage of modern microarchitectures, parallelizing with SIMD vector instructions, reducing branch misprediction, and reducing data dependency to take advantage of each CPU's multiple execution cores.

Some people enjoy reading our paper: A description of the design and implementation of simdjson is in our research article in VLDB journal: Geoff Langdale, Daniel Lemire, Parsing Gigabytes of JSON per Second, VLDB Journal 28 (6), 2019appear)

We also have an informal blog post providing some background and context.

For the video inclined, (it was the best voted talk, we're kinda proud of it).

Funding

The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant number RGPIN-2017-03910.

Contributing to simdjson

Head over to CONTRIBUTING.md for information on contributing to simdjson, and HACKING.md for information on source, building, and architecture/design.

License

This code is made available under the Apache License 2.0.

Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.