Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
SIMDJSON_DISABLE_DEPRECATED_API.
For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.
Closes#1264
This builds the CI fuzzers with the intended clang version. It also allows users to set the clang version locally,
in case they need to.
It also switches the CI fuzzers to use an optimized sanitizer build, to do something oss-fuzz doesn't and get more done in the short time the CI fuzzer runs.
Projects that link simdjson from MSVC with exceptions off will
include simdjson headers which transitively include STL headers.
The MSVC STL stipulates that _HAS_EXCEPTIONS=0 be defined or code
requiring exceptions will be enabled. This change adds a new job
to the appveyor build matrix to verify the build and tests with
exceptions disabled, and disables exceptions at the compiler level
when SIMDJSON_EXCEPTIONS is specified to OFF.
* add definitions for is_number and tie (by lemire)
* add fuzzer for element
* update fuzz documentation
* fix UB in creating an empty padded string
* don't bother null terminating padded_string, it is done by the std::memset already
* refactor fuzz data splitting into a separate class
This refactors the dynamic check of which implementations are supported at runtime.
It also reduces duplicated effort in the CI fuzzing job, the differential fuzzers don't need to run with different values of SIMDJSON_FORCE_IMPLEMENTATION.
There is also a convenience script to run the fuzzers locally, to quickly check that the fuzzers still build, run and no easy to find bugs are there. It should be handy not only when developing the fuzzers, but also when modifying simdjson.
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
* Make it possible to check that an implementation is supported at runtime.
* add CI fuzzing on arm 64 bit
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* Make it possible to check that an implementation is supported at runtime.
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* We need to accomodate cxxopts
Co-authored-by: Paul Dreik <github@pauldreik.se>
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
This adds a minifier fuzzer. There is also an utf-8 fuzzer, but it is disabled until #1187 is fixed.
Run all fuzzers bug the utf-8 one in the github CI fuzz.
This adds a fuzzer for at_pointer() which recently had a bug.
The #1142 bug had been found with this fuzzer
Also, it polishes the github action job:
cross pollinate the fuzzer corpora (lets fuzzers reuse results from other fuzzers)
use github action syntax instead of bash checks
only run on push if on master
This adds a fuzzer which parses the same input using all the available implementations (haswell, westmere, fallback on x64).
This should get the otherwise uncovered sourcefiles (mostly fallback) to show up in the fuzz coverage.
For instance, the fallback directory has only one line covered.
As of the 20200909 report, 1866 lines are covered out of 4478.
Also, it will detect if the implementations behave differently:
by making sure they all succeed, or all error
turning the parsed data into text again, should produce equal results
While at it, I corrected some minor things:
clean up building too many variants, run with forced implementation (closes#815 )
always store crashes as artefacts, good in case the fuzzer finds something
return value of the fuzzer function should always be 0
reduce log spam
introduce max size for the seed corpus and the CI fuzzer
* move from deprecated interface in fuzz dump raw tape
* update fuzz_dump to the non deprecated replacement
* replace use of deprecated api
* hopefully fix windows build
This enables the minify fuzzer, which has been disabled because it did not pass the oss-fuzz instrumentation test. Now it does, after changes in simdjson (https://github.com/lemire/simdjson/issues/186).
* get minify running (api change)
* disable benchmarks when compiling fuzzers
* catch exceptions from the minify fuzzer
* enable repeated corpus creation without recursive inclusion of zip
* remove leftover comment
This creates a "document" class with only user-facing document state (no parser internals).
- document: user-facing document state
- document::iterator: iterator (equivalent of ParsedJsonIterator)
- document::parser: parser state plus a "docked" document we parse into (equivalent of ParsedJson)
Usage:
```c++
auto doc = simdjson::document::parse(buf, len); // less efficient but simplest
```
```c++
simdjson::document::parser parser; // reusable parser
parser.allocate_capacity(len);
simdjson::document* doc = parser.parse(buf, len); // pointer to doc inside parser
doc = parser.parse(buf2, len); // reuses all buffers and overwrites doc; more efficient
```
* initial oss-fuzz friendly build
parts taken from libfmt, which I wrote and have the copyright to
* fix build error
* add script for building a corpus zip
see https://google.github.io/oss-fuzz/getting-started/new-project-guide/#seed-corpus
* fix zip command
* drop setting the C++ standard
* disable the minify fuzzer, does not pass oss-fuzz check-build test
* fix integer overflow in subnormal_power10
detected by oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714
* invoke the build like oss fuzz does
* document what the scripts are for and how to use them
* add a page about fuzzing
* add basic fuzzing using libFuzzer
* let cmake respect cflags, otherwise the fuzzer flags go unnoticed
also, integrates badly with oss-fuzz
* add new fuzzer for minification, simplify the old one
* add fuzzer for the dump example
* clang format
Paul Dreik
Jeremy Ong
Daniel Lemire
John Keiser
Furkan Usta