2020-03-25 00:33:12 +08:00
[](https://cloud.drone.io/simdjson/simdjson)
[](https://circleci.com/gh/simdjson/simdjson)
[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened& q=proj%3Asimdjson& can=2)
[](https://ci.appveyor.com/project/lemire/simdjson-jmmti/branch/master)
[![][license img]][license]
2020-03-26 01:53:24 +08:00
simdjson : Parsing gigabytes of JSON per second
===============================================
2020-03-21 10:07:52 +08:00
< img src = "images/logo.png" width = "10%" style = "float: right" >
2020-03-25 00:33:12 +08:00
JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh
2020-03-26 01:53:24 +08:00
approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms
2020-03-25 00:33:12 +08:00
to parse JSON 2.5x faster than anything else out there.
2020-03-21 10:07:52 +08:00
2020-03-25 00:33:12 +08:00
* **Fast:** Over 2.5x faster than other production-grade JSON parsers.
* **Easy:** First-class, easy to use API.
* **Strict:** Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises.
* **Automatic:** Selects a CPU-tailored parser at runtime. No configuration needed.
* **Reliable:** From memory allocation to error handling, simdjson's design avoids surprises.
2020-03-21 10:07:52 +08:00
This library is part of the [Awesome Modern C++ ](https://awesomecpp.com ) list.
2020-03-26 01:53:24 +08:00
Table of Contents
-----------------
2020-03-25 00:33:12 +08:00
* [Quick Start ](#quick-start )
2020-03-26 01:53:24 +08:00
* [Documentation ](#documentation )
* [Performance results ](#performance-results )
* [Real-world usage ](#real-world-usage )
* [Bindings and Ports of simdjson ](#bindings-and-ports-of-simdjson )
2020-03-25 00:33:12 +08:00
* [About simdjson ](#about-simdjson )
2020-03-26 01:53:24 +08:00
* [Funding ](#funding )
* [Contributing to simdjson ](#contributing-to-simdjson )
* [License ](#license )
Quick Start
-----------
The simdjson library is easily consumable with a single .h and .cpp file.
2020-03-21 10:07:52 +08:00
0. Prerequisites: `g++` or `clang++` .
2020-03-22 13:14:10 +08:00
1. Pull [simdjson.h ](singleheader/simdjson.h ) and [simdjson.cpp ](singleheader/simdjson.cpp ) into a directory, along with the sample file [twitter.json ](jsonexamples/twitter.json ).
2020-03-21 10:07:52 +08:00
```
wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json
```
2. Create `parser.cpp` :
```c++
#include "simdjson.h"
int main(void) {
2020-03-29 02:43:41 +08:00
simdjson::dom::parser parser;
simdjson::dom::element tweets = parser.load("twitter.json");
2020-03-21 10:07:52 +08:00
std::cout < < tweets [ " search_metadata " ] [ " count " ] < < " results . " < < std::endl ;
}
```
2020-03-24 01:00:48 +08:00
3. `c++ -o parser parser.cpp simdjson.cpp -std=c++17`
2020-03-21 10:07:52 +08:00
4. `./parser`
```
100 results.
```
2019-03-13 03:13:25 +08:00
2020-03-26 01:53:24 +08:00
Documentation
-------------
2019-08-26 22:24:31 +08:00
2020-03-26 01:53:24 +08:00
Usage documentation is available:
2019-08-26 22:24:31 +08:00
2020-03-26 01:53:24 +08:00
* [Basics ](doc/basics.md ) is an overview of how to use simdjson and its APIs.
* [Performance ](doc/performance.md ) shows some more advanced scenarios and how to tune for them.
* [Implementation Selection ](doc/implementation-selection.md ) describes runtime CPU detection and
how you can work with it.
2018-12-15 10:50:55 +08:00
2020-03-26 01:53:24 +08:00
Performance results
-------------------
2019-03-03 06:18:45 +08:00
2020-03-26 01:53:24 +08:00
The simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON and
2020-03-25 00:33:12 +08:00
fifty percent less than sajson. To our knowledge, simdjson is the first fully-validating JSON parser
to run at gigabytes per second on commodity processors.
2018-12-28 09:33:18 +08:00
2020-03-28 02:11:02 +08:00
The following figure represents parsing speed in GB/s for parsing various files
on an Intel Skylake processor (3.4 GHz) using the GNU GCC 9 compiler (with the -O3 flag).
We compare against the best and fastest C++ libraries.
The simdjson library offers full unicode (UTF-8) validation and exact
number parsing. The RapidJSON library is tested in two modes: fast and
exact number parsing. The sajson library offers fast (but not exact)
number parsing and partial unicode validation. In this data set, the file
sizes range from 65KB (github_events) all the way to 3.3GB (gsoc-2018).
Many files are mostly made of numbers: canada, mesh.pretty, mesh, random
and numbers: in such instances, we see lower JSON parsing speeds due to the
2020-03-28 02:12:30 +08:00
high cost of number parsing. The simdjson library uses exact number parsing which
is particular taxing.
2020-03-28 02:11:02 +08:00
2019-01-02 05:23:56 +08:00
< img src = "doc/gbps.png" width = "90%" >
2018-12-28 09:33:18 +08:00
2020-03-28 02:11:02 +08:00
On a Skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows, using again GNU GCC 9.1 (with the -O3 flag). The popular JSON for Modern C++ library is particularly slow: it obviously trades parsing speed for other desirable features.
2019-01-19 00:58:08 +08:00
2019-03-03 06:18:45 +08:00
| parser | GB/s |
| ------------------------------------- | ---- |
2020-03-28 02:11:02 +08:00
| simdjson | 2.5 |
| RapidJSON UTF8-validation | 0.29 |
| RapidJSON UTF8-valid., exact numbers | 0.28 |
| RapidJSON insitu, UTF8-validation | 0.41 |
| RapidJSON insitu, UTF8-valid., exact | 0.39 |
| sajson (insitu, dynamic) | 0.62 |
| sajson (insitu, static) | 0.88 |
| dropbox | 0.13 |
| fastjson | 0.27 |
| gason | 0.59 |
| ultrajson | 0.34 |
| jsmn | 0.25 |
| cJSON | 0.31 |
| JSON for Modern C++ (nlohmann/json) | 0.11 |
The simdjson library offer high speed whether it processes tiny files (e.g., 300 bytes)
or larger files (e.g., 3MB). The following plot presents parsing
speed for [synthetic files over various sizes generated with a script ](https://github.com/simdjson/simdjson_experiments_vldb2019/blob/master/experiments/growing/gen.py ) on a 3.4 GHz Skylake processor (GNU GCC 9, -O3).
< img src = "doc/growing.png" width = "90%" >
[All our experiments are reproducible ](https://github.com/simdjson/simdjson_experiments_vldb2019 ).
2018-12-19 14:04:13 +08:00
2020-03-26 01:53:24 +08:00
Real-world usage
----------------
2018-12-28 09:41:38 +08:00
2020-03-25 00:33:12 +08:00
- [Microsoft FishStore ](https://github.com/microsoft/FishStore )
- [Yandex ClickHouse ](https://github.com/yandex/ClickHouse )
- [Clang Build Analyzer ](https://github.com/aras-p/ClangBuildAnalyzer )
2018-12-28 09:48:21 +08:00
2020-03-25 00:33:12 +08:00
If you are planning to use simdjson in a product, please work from one of our releases.
2018-12-28 09:48:21 +08:00
2020-03-26 01:53:24 +08:00
Bindings and Ports of simdjson
------------------------------
2019-11-27 03:13:17 +08:00
2020-03-25 00:33:12 +08:00
We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).
2018-12-15 10:32:42 +08:00
2020-03-25 00:33:12 +08:00
- [ZippyJSON ](https://github.com/michaeleisel/zippyjson ): Swift bindings for the simdjson project.
- [pysimdjson ](https://github.com/TkTech/pysimdjson ): Python bindings for the simdjson project.
- [simdjson-rs ](https://github.com/Licenser/simdjson-rs ): Rust port.
- [simdjson-rust ](https://github.com/SunDoge/simdjson-rust ): Rust wrapper (bindings).
- [SimdJsonSharp ](https://github.com/EgorBo/SimdJsonSharp ): C# version for .NET Core (bindings and full port).
- [simdjson_nodejs ](https://github.com/luizperes/simdjson_nodejs ): Node.js bindings for the simdjson project.
- [simdjson_php ](https://github.com/crazyxman/simdjson_php ): PHP bindings for the simdjson project.
- [simdjson_ruby ](https://github.com/saka1/simdjson_ruby ): Ruby bindings for the simdjson project.
- [simdjson-go ](https://github.com/minio/simdjson-go ): Go port using Golang assembly.
- [rcppsimdjson ](https://github.com/eddelbuettel/rcppsimdjson ): R bindings.
2018-12-15 10:32:42 +08:00
2020-03-26 01:53:24 +08:00
About simdjson
--------------
The simdjson library takes advantage of modern microarchitectures, parallelizing with SIMD vector
instructions, reducing branch misprediction, and reducing data dependency to take advantage of each
CPU's multiple execution cores.
Some people [enjoy reading our paper ](https://arxiv.org/abs/1902.08318 ): A description of the design
and implementation of simdjson is in our research article in VLDB journal: Geoff Langdale, Daniel
Lemire, [Parsing Gigabytes of JSON per Second ](https://arxiv.org/abs/1902.08318 ), VLDB Journal 28 (6), 2019appear)
We also have an informal [blog post providing some background and context ](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/ ).
2020-03-29 04:18:57 +08:00
For the video inclined, < br / >
[](http://www.youtube.com/watch?v=wlvKAT7SZIQ)< br />
2020-03-26 01:53:24 +08:00
(it was the best voted talk, we're kinda proud of it).
Funding
-------
2019-06-27 02:16:32 +08:00
2019-10-19 05:32:54 +08:00
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant number RGPIN-2017-03910.
2019-06-27 02:16:32 +08:00
2019-03-03 06:18:45 +08:00
[license]: LICENSE
[license img]: https://img.shields.io/badge/License-Apache%202-blue.svg
2020-03-25 00:33:12 +08:00
2020-03-26 01:53:24 +08:00
Contributing to simdjson
------------------------
Head over to [CONTRIBUTING.md ](CONTRIBUTING.md ) for information on contributing to simdjson, and
[HACKING.md ](HACKING.md ) for information on source, building, and architecture/design.
License
-------
2020-03-25 00:33:12 +08:00
This code is made available under the Apache License 2.0.
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.
