Including <iostream> has two problems:
* Compile times are worse because of over-inclusion
* Binary sizes are worse when statically linking libstdc++ because
iostreams cannot be dead-code-stripped
simdjson only needs std::ostream. Include the header declaring only what
we need (<ostream>), omitting stuff we don't need (std::cout and its
initialization, for example).
This commit should not change behavior, but it might break users who
assume that including <simdjson/simdjson.h> will make std::cout
available (such as many of simdjson's own files).
* upload corpus to https://www.pauldreik.se/ from the x64 github action job (keep the github action cache)
* drop the github action cache (which was not working anyway) for power fuzzer and download the fuzz corpus from https://www.pauldreik.se/ instead
* resurrect arm64 fuzzing on drone CI, downloading the fuzz corpus from https://www.pauldreik.se/
* update the fuzzing documentation
* Remove CMP0025 policy
This policy is already set to NEW by the minimum required version.
* Use HOMEPAGE_URL in the project call
* Use VERSION in the project call
* Detect if this is the top project
* Port simdjson-user-cmakecache to a CMake script
* Create a developer mode
The SIMDJSON_DEVELOPER_MODE option set to ON will enable targets that
are only useful for developers of simdjson.
* Consolidate root CML commands into logical sections
* Warn about intended use of developer mode
* Prettify the just_ascii test
* Remove redundant CMake variables
* Inline CML contents from include and src
* Raise minimum CMake requirement to 3.14
* Define proper install rules
* Restore thread support variable
* Add BUILD_SHARED_LIBS as a top level only option
* Force developer mode to be on in CI
* Include flags earlier in developer mode
* Set CMAKE_BUILD_TYPE conditionally
CMAKE_BUILD_TYPE is used only by single configuration generators and is
otherwise completely ignored.
* Remove useless static/shared options
simdjson now uses the CMake builtin BUILD_SHARED_LIBS to switch the
built artifact's type.
* Remove unused CMAKE_MODULE_PATH variable
* Refactor implementation switching into a module
* Factor exception option out into a module
* Reformat simdjson-flags.cmake
* Rename simdjson-flags to developer-options
* Accumulate properties into an include module
This is done this way to avoid using utility targets that must be
exported and installed, which could potentially be misused by users of
the library.
* Port impl definitions to props
* Port exception options to props
* Lift normal options to the top
* Port developer options to props
* Remove simdjson-flags from benchmark
* Document the developer mode in HACKING
* Fix include path in installed config file
* Fix formatting of prop commands
* Fix tests that include .cpp files
* Change GCC AVX fixes back to compile options
* Deprecate SIMDJSON_BUILD_STATIC
* Always link fuzz targets to simdjson
* Install CMake from simdjson's debian repo
* Add gnupg for apt-key
* Make sure ASan link flags come first
* Pass CI env variable to cmake invocation
* Install package for apt-add-repository
* Remove return() from flush macro
* Use directory level commands instead of props
* Restore the github repository variable
* Set developer mode unconditionally for checkperf
The CI env variable is only set in the CI and this target is always run
in developer mode.
* Attempt to fix ODR violation in parsing checks
These tests were compiling the simdjson.cpp file again and linking to
the simdjson library target causes ODR violations.
Instead of linking to the target, just inherit its props.
* Move variables before the source dir
* Mark props to be flushed after adding more
* Use props for every command for the library
* Use keyword form for linking libs
* Handle deprecation of SIMDJSON_JUST_LIBRARY
* Handle deprecations in a separate module
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* This gives the CMake install the necessarily information (and flags) to know
whether we have a Windows DLL and in such cases how to handle the linkage.
* Fixing issue 1243
* The tie must go.
* Having std::pair be a protected inheritance breaks on demand.
* Putting it back.
* You really want to use emplace.
* Fixing one botched test.
* Prettier test.
* Using safer code.
* Fixing unsafe code.
* Simplifying the fuzzer.
* Trying another way.
* Ok. It should work without exceptions.
* Removing trailing spaces.
* Entering a new UTF-8 test
* Maybe *I* had a bug in the tests.
* Replacing nulls with 1s.
* Let us try to be more verbose.
* Return 0.
* Fixing issue.
* Adding puzzler scenario.
* Fixing PPC64
Co-authored-by: Daniel Lemire <dlemire@rcs-power9-talos>
* first try
* use ubuntu 20.04, do the fuzzing
* new try at power fuzz
* hard code clang version
* setting env variables does not seem to work
* use fuzzer-no-link
* switch to Debian Buster for power fuzz
* use non-sanitizer build for power
* me not like yaml
* fix bad syntax
* add ndjson fuzzer
* reproduce #1310 in the newly added unit test
Had to replace the input, because:
1)
the fuzzer uses the first part of the input to determine
the batch_size to use, so that has to be cut off
2)
the master now protects against low values of batch_size
I also made the test not return early, so the error is triggered.
Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
SIMDJSON_DISABLE_DEPRECATED_API.
For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.
Closes#1264
This builds the CI fuzzers with the intended clang version. It also allows users to set the clang version locally,
in case they need to.
It also switches the CI fuzzers to use an optimized sanitizer build, to do something oss-fuzz doesn't and get more done in the short time the CI fuzzer runs.
Projects that link simdjson from MSVC with exceptions off will
include simdjson headers which transitively include STL headers.
The MSVC STL stipulates that _HAS_EXCEPTIONS=0 be defined or code
requiring exceptions will be enabled. This change adds a new job
to the appveyor build matrix to verify the build and tests with
exceptions disabled, and disables exceptions at the compiler level
when SIMDJSON_EXCEPTIONS is specified to OFF.
* add definitions for is_number and tie (by lemire)
* add fuzzer for element
* update fuzz documentation
* fix UB in creating an empty padded string
* don't bother null terminating padded_string, it is done by the std::memset already
* refactor fuzz data splitting into a separate class
This refactors the dynamic check of which implementations are supported at runtime.
It also reduces duplicated effort in the CI fuzzing job, the differential fuzzers don't need to run with different values of SIMDJSON_FORCE_IMPLEMENTATION.
There is also a convenience script to run the fuzzers locally, to quickly check that the fuzzers still build, run and no easy to find bugs are there. It should be handy not only when developing the fuzzers, but also when modifying simdjson.
- Allow user to specify SIMDJSON_BUILTIN_IMPLEMENTATION
- Make cmake -DSIMDJSON_IMPLEMENTATION=haswell *only* specify haswell
- Move negative implementation selection to
-DSIMDJSON_EXCLUDE_IMPLEMENTATION
- Automatically select SIMDJSON_BUILTIN_IMPLEMENTATION if
SIMDJSON_IMPLEMENTATION is set
- Move implementation enablement mostly to implementation files
- Make implementation enablement and selection simpler and more robust
- Fix bug where programs linked against simdjson were not passed
SIMDJSON_XXX_IMPLEMENTATION or SIMDJSON_EXCEPTIONS
* Make it possible to check that an implementation is supported at runtime.
* add CI fuzzing on arm 64 bit
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* Make it possible to check that an implementation is supported at runtime.
* Guarding the implementation accesses.
* Better doc.
* Updating cxxopts.
* We need to accomodate cxxopts
Co-authored-by: Paul Dreik <github@pauldreik.se>
This adds fuzzing on drone.io arm64
For some reason, leak detection had to be disabled. If it is enabled, the fuzzer falsely reports a crash at the end of fuzzing.
Closes: #1188
This adds a minifier fuzzer. There is also an utf-8 fuzzer, but it is disabled until #1187 is fixed.
Run all fuzzers bug the utf-8 one in the github CI fuzz.
This adds a fuzzer for at_pointer() which recently had a bug.
The #1142 bug had been found with this fuzzer
Also, it polishes the github action job:
cross pollinate the fuzzer corpora (lets fuzzers reuse results from other fuzzers)
use github action syntax instead of bash checks
only run on push if on master
This adds a fuzzer which parses the same input using all the available implementations (haswell, westmere, fallback on x64).
This should get the otherwise uncovered sourcefiles (mostly fallback) to show up in the fuzz coverage.
For instance, the fallback directory has only one line covered.
As of the 20200909 report, 1866 lines are covered out of 4478.
Also, it will detect if the implementations behave differently:
by making sure they all succeed, or all error
turning the parsed data into text again, should produce equal results
While at it, I corrected some minor things:
clean up building too many variants, run with forced implementation (closes#815 )
always store crashes as artefacts, good in case the fuzzer finds something
return value of the fuzzer function should always be 0
reduce log spam
introduce max size for the seed corpus and the CI fuzzer
* move from deprecated interface in fuzz dump raw tape
* update fuzz_dump to the non deprecated replacement
* replace use of deprecated api
* hopefully fix windows build
Daniel Lemire
Paul Dreik
strager
Dirk Stolle
friendlyanon
John Keiser
Jeremy Ong
Furkan Usta