2020-03-25 00:33:12 +08:00
[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened& q=proj%3Asimdjson& can=2)
2020-07-22 00:02:30 +08:00
/badge.svg)
2020-08-05 07:44:52 +08:00
[/badge.svg)](https://simdjson.org/plots.html)
2020-07-22 00:02:30 +08:00
2020-06-30 09:43:07 +08:00
2021-03-17 23:08:44 +08:00
[![][license img]][license] [](https://simdjson.org/api/0.9.0/index.html)
2020-03-25 00:33:12 +08:00
2020-03-26 01:53:24 +08:00
simdjson : Parsing gigabytes of JSON per second
===============================================
2020-03-21 10:07:52 +08:00
< img src = "images/logo.png" width = "10%" style = "float: right" >
2020-03-25 00:33:12 +08:00
JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh
2020-03-26 01:53:24 +08:00
approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms
2021-03-13 04:01:12 +08:00
to parse JSON 4x faster than RapidJSON and 25x faster than JSON for Modern C++.
2020-03-21 10:07:52 +08:00
2021-03-13 04:01:12 +08:00
* **Fast:** Over 4x faster than commonly used production-grade JSON parsers.
2020-10-23 21:15:04 +08:00
* **Record Breaking Features:** Minify JSON at 6 GB/s, validate UTF-8 at 13 GB/s, NDJSON at 3.5 GB/s.
* **Easy:** First-class, easy to use and carefully documented APIs.
2020-03-25 00:33:12 +08:00
* **Strict:** Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises.
* **Automatic:** Selects a CPU-tailored parser at runtime. No configuration needed.
* **Reliable:** From memory allocation to error handling, simdjson's design avoids surprises.
2020-10-23 21:18:47 +08:00
* **Peer Reviewed:** Our research appears in venues like VLDB Journal, Software: Practice and Experience.
2020-03-21 10:07:52 +08:00
This library is part of the [Awesome Modern C++ ](https://awesomecpp.com ) list.
2020-03-26 01:53:24 +08:00
Table of Contents
-----------------
2020-03-25 00:33:12 +08:00
* [Quick Start ](#quick-start )
2020-12-25 06:59:58 +08:00
* [On Demand ](#on-demand )
2020-03-26 01:53:24 +08:00
* [Documentation ](#documentation )
* [Performance results ](#performance-results )
* [Real-world usage ](#real-world-usage )
* [Bindings and Ports of simdjson ](#bindings-and-ports-of-simdjson )
2020-03-25 00:33:12 +08:00
* [About simdjson ](#about-simdjson )
2020-03-26 01:53:24 +08:00
* [Funding ](#funding )
* [Contributing to simdjson ](#contributing-to-simdjson )
* [License ](#license )
Quick Start
-----------
The simdjson library is easily consumable with a single .h and .cpp file.
2020-03-21 10:07:52 +08:00
2020-12-25 06:59:58 +08:00
0. Prerequisites: `g++` (version 7 or better) or `clang++` (version 6 or better), and a 64-bit
system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming
environments like Visual Studio and Xcode, but different steps are needed.
1. Pull [simdjson.h ](singleheader/simdjson.h ) and [simdjson.cpp ](singleheader/simdjson.cpp ) into a
directory, along with the sample file [twitter.json ](jsonexamples/twitter.json ).
2020-03-21 10:07:52 +08:00
```
wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json
```
2020-03-30 06:34:56 +08:00
2. Create `quickstart.cpp` :
2020-03-21 10:07:52 +08:00
2021-03-13 03:21:23 +08:00
```c++
2021-03-13 03:19:11 +08:00
#include "simdjson.h"
using namespace simdjson;
int main(void) {
ondemand::parser parser;
padded_string json = padded_string::load("twitter.json");
ondemand::document tweets = parser.iterate(json);
std::cout < < uint64_t ( tweets [ " search_metadata " ] [ " count " ] ) < < " results . " < < std::endl ;
}
2020-03-21 10:07:52 +08:00
```
2020-04-09 02:06:16 +08:00
3. `c++ -o quickstart quickstart.cpp simdjson.cpp`
2020-03-30 06:34:56 +08:00
4. `./quickstart`
2020-03-21 10:07:52 +08:00
```
100 results.
```
2019-03-13 03:13:25 +08:00
2020-03-26 01:53:24 +08:00
Documentation
-------------
2019-08-26 22:24:31 +08:00
2020-03-26 01:53:24 +08:00
Usage documentation is available:
2019-08-26 22:24:31 +08:00
2020-03-26 01:53:24 +08:00
* [Basics ](doc/basics.md ) is an overview of how to use simdjson and its APIs.
* [Performance ](doc/performance.md ) shows some more advanced scenarios and how to tune for them.
* [Implementation Selection ](doc/implementation-selection.md ) describes runtime CPU detection and
how you can work with it.
2021-03-17 23:08:44 +08:00
* [API ](https://simdjson.org/api/0.9.0/annotated.html ) contains the automatically generated API documentation.
2018-12-15 10:50:55 +08:00
2020-03-26 01:53:24 +08:00
Performance results
-------------------
2019-03-03 06:18:45 +08:00
2021-03-13 04:01:12 +08:00
The simdjson library uses three-quarters less instructions than state-of-the-art parser [RapidJSON ](https://rapidjson.org ). To our knowledge, simdjson is the first fully-validating JSON parser
2020-05-02 00:16:18 +08:00
to run at [gigabytes per second ](https://en.wikipedia.org/wiki/Gigabyte ) (GB/s) on commodity processors. It can parse millions of JSON documents per second on a single core.
2018-12-28 09:33:18 +08:00
2020-03-28 02:11:02 +08:00
The following figure represents parsing speed in GB/s for parsing various files
2021-03-13 04:01:12 +08:00
on an Intel Skylake processor (3.4 GHz) using the GNU GCC 10 compiler (with the -O3 flag).
We compare against the best and fastest C++ libraries on benchmarks that load and process the data.
2020-05-02 00:16:18 +08:00
The simdjson library offers full unicode ([UTF-8](https://en.wikipedia.org/wiki/UTF-8)) validation and exact
2021-03-17 05:32:38 +08:00
number parsing.
2020-03-28 02:11:02 +08:00
2021-03-13 04:01:52 +08:00
< img src = "doc/rome.png" width = "60%" >
2020-03-28 02:11:02 +08:00
2020-03-31 22:29:20 +08:00
The simdjson library offers high speed whether it processes tiny files (e.g., 300 bytes)
2020-03-28 02:11:02 +08:00
or larger files (e.g., 3MB). The following plot presents parsing
speed for [synthetic files over various sizes generated with a script ](https://github.com/simdjson/simdjson_experiments_vldb2019/blob/master/experiments/growing/gen.py ) on a 3.4 GHz Skylake processor (GNU GCC 9, -O3).
2021-03-13 04:02:47 +08:00
2021-03-13 04:01:52 +08:00
< img src = "doc/growing.png" width = "60%" >
2020-03-28 02:11:02 +08:00
[All our experiments are reproducible ](https://github.com/simdjson/simdjson_experiments_vldb2019 ).
2018-12-19 14:04:13 +08:00
2020-10-23 21:21:16 +08:00
For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing functions ](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md ).
2020-10-23 21:15:04 +08:00
2020-03-26 01:53:24 +08:00
Real-world usage
----------------
2018-12-28 09:41:38 +08:00
2020-03-25 00:33:12 +08:00
- [Microsoft FishStore ](https://github.com/microsoft/FishStore )
- [Yandex ClickHouse ](https://github.com/yandex/ClickHouse )
- [Clang Build Analyzer ](https://github.com/aras-p/ClangBuildAnalyzer )
2020-08-04 00:51:21 +08:00
- [Shopify HeapProfiler ](https://github.com/Shopify/heap-profiler )
2018-12-28 09:48:21 +08:00
2020-03-25 00:33:12 +08:00
If you are planning to use simdjson in a product, please work from one of our releases.
2018-12-28 09:48:21 +08:00
2020-03-26 01:53:24 +08:00
Bindings and Ports of simdjson
------------------------------
2019-11-27 03:13:17 +08:00
2020-03-25 00:33:12 +08:00
We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything).
2018-12-15 10:32:42 +08:00
2020-03-25 00:33:12 +08:00
- [ZippyJSON ](https://github.com/michaeleisel/zippyjson ): Swift bindings for the simdjson project.
2020-07-17 01:05:28 +08:00
- [libpy_simdjson ](https://github.com/gerrymanoim/libpy_simdjson/ ): high-speed Python bindings for simdjson using [libpy ](https://github.com/quantopian/libpy ).
2020-03-25 00:33:12 +08:00
- [pysimdjson ](https://github.com/TkTech/pysimdjson ): Python bindings for the simdjson project.
2021-03-10 05:29:36 +08:00
- [cysimdjson ](https://github.com/TeskaLabs/cysimdjson ): high-speed Python bindings for the simdjson project.
2020-04-01 04:44:28 +08:00
- [simdjson-rs ](https://github.com/simd-lite ): Rust port.
2020-03-25 00:33:12 +08:00
- [simdjson-rust ](https://github.com/SunDoge/simdjson-rust ): Rust wrapper (bindings).
- [SimdJsonSharp ](https://github.com/EgorBo/SimdJsonSharp ): C# version for .NET Core (bindings and full port).
- [simdjson_nodejs ](https://github.com/luizperes/simdjson_nodejs ): Node.js bindings for the simdjson project.
- [simdjson_php ](https://github.com/crazyxman/simdjson_php ): PHP bindings for the simdjson project.
- [simdjson_ruby ](https://github.com/saka1/simdjson_ruby ): Ruby bindings for the simdjson project.
2020-07-18 03:41:54 +08:00
- [fast_jsonparser ](https://github.com/anilmaurya/fast_jsonparser ): Ruby bindings for the simdjson project.
2020-03-25 00:33:12 +08:00
- [simdjson-go ](https://github.com/minio/simdjson-go ): Go port using Golang assembly.
- [rcppsimdjson ](https://github.com/eddelbuettel/rcppsimdjson ): R bindings.
2020-10-21 23:43:36 +08:00
- [simdjson_erlang ](https://github.com/ChomperT/simdjson_erlang ): erlang bindings.
2018-12-15 10:32:42 +08:00
2020-03-26 01:53:24 +08:00
About simdjson
--------------
The simdjson library takes advantage of modern microarchitectures, parallelizing with SIMD vector
instructions, reducing branch misprediction, and reducing data dependency to take advantage of each
CPU's multiple execution cores.
Some people [enjoy reading our paper ](https://arxiv.org/abs/1902.08318 ): A description of the design
2020-11-04 04:48:09 +08:00
and implementation of simdjson is in our research article:
2020-10-09 22:54:48 +08:00
- Geoff Langdale, Daniel Lemire, [Parsing Gigabytes of JSON per Second ](https://arxiv.org/abs/1902.08318 ), VLDB Journal 28 (6), 2019.
We have an in-depth paper focused on the UTF-8 validation:
- John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte ](https://arxiv.org/abs/2010.03090 ), Software: Practice & Experience (to appear)
2020-03-26 01:53:24 +08:00
We also have an informal [blog post providing some background and context ](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/ ).
2020-03-29 04:18:57 +08:00
For the video inclined, < br / >
[](http://www.youtube.com/watch?v=wlvKAT7SZIQ)< br />
2020-03-26 01:53:24 +08:00
(it was the best voted talk, we're kinda proud of it).
Funding
-------
2019-06-27 02:16:32 +08:00
2020-04-08 03:40:25 +08:00
The work is supported by the Natural Sciences and Engineering Research Council of Canada under grant
number RGPIN-2017-03910.
2019-06-27 02:16:32 +08:00
2019-03-03 06:18:45 +08:00
[license]: LICENSE
[license img]: https://img.shields.io/badge/License-Apache%202-blue.svg
2020-03-25 00:33:12 +08:00
2020-03-26 01:53:24 +08:00
Contributing to simdjson
------------------------
Head over to [CONTRIBUTING.md ](CONTRIBUTING.md ) for information on contributing to simdjson, and
[HACKING.md ](HACKING.md ) for information on source, building, and architecture/design.
License
-------
2020-03-25 00:33:12 +08:00
2020-05-02 00:16:18 +08:00
This code is made available under the [Apache License 2.0 ](https://www.apache.org/licenses/LICENSE-2.0.html ).
2020-03-25 00:33:12 +08:00
Under Windows, we build some tools using the windows/dirent_portable.h file (which is outside our library code): it under the liberal (business-friendly) MIT license.
2020-04-09 02:06:16 +08:00
2020-05-02 00:16:18 +08:00
For compilers that do not support [C++17 ](https://en.wikipedia.org/wiki/C%2B%2B17 ), we bundle the string-view library which is published under the Boost license (http://www.boost.org/LICENSE_1_0.txt). Like the Apache license, the Boost license is a permissive license allowing commercial redistribution.
2021-04-02 10:53:31 +08:00
For efficient number serialization, we bundle Florian Loitsch's implementation of the Grisu2 algorithm for binary to decimal floating-point numbers. The implementation was slightly modified by JSON for Modern C++ library. Both Florian Loitsch's implementation and JSON for Modern C++ are provided under the MIT license.
2021-04-13 22:25:02 +08:00
For runtime dispatching, we use some code from the PyTorch project licensed under 3-clause BSD.
