Commit Graph

803 Commits

Author SHA1 Message Date
Daniel Lemire 7e331957c4 Adding ruby script. 2019-12-30 15:07:42 -05:00
Nexus Web Development 0dc80ccf21 Update README.md : Simple typo (#412)
Exemple to example.
2019-12-23 11:35:42 -05:00
Nexus Web Development f2b48ede4c Update JsonStream.md : simple typo (#413)
Exemple to example.
2019-12-23 11:35:09 -05:00
Stenal P Jolly 6cefdc2f5c Fix README typo (#407) 2019-12-23 11:08:55 -05:00
Paul Dreik 29e78413fe
output failed fuzz cases (#409)
this is to get more debug output from cases like this:
c9cd8e6211/checks?check_suite_id=351526761
https://pipelines.actions.githubusercontent.com/gFSIASDqcDhMdHkcuMJEdsUcdOsOrVBr8d56BjAkszcoMkibGp/_apis/pipelines/1/runs/274/signedlogcontent/3?urlExpires=2019-12-21T18%3A50%3A08.3387613Z&urlSigningMethod=HMACV1&urlSignature=DVF3u4jEs81xJP6Hmi8LLQlLOztJke8MsP62J0MXruQ%3D

which ends with:
2019-12-20T23:08:42.6830393Z ../src/generic/numberparsing.h:243:31: runtime error: 1e+311 is outside the range of representable values of type 'double'
2019-12-20T23:08:42.6836790Z SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../src/generic/numberparsing.h:243:31 in
2019-12-20T23:08:42.6837715Z MS: 2 InsertRepeatedBytes-CrossOver-; base unit: 47cf79b80bc84f2ec8b39c1c73daa6c1222ef624
2019-12-20T23:08:42.6866416Z artifact_prefix='./'; Test unit written to ./crash-3fa2f47d980d4bf5812af18fb0d0a0ce9b5cc65a
2019-12-20T23:08:42.6999448Z ##[error]Process completed with exit code 1.
2019-12-20T23:08:42.7030863Z Cleaning up orphan processes


The problem is that github actions does not store artifacts on failure.
2019-12-21 20:32:32 +01:00
Paul Dreik 8192e63a4b
update apt repos before installing (#408)
This should hopefully resolve the recent errors in the github action
for running the fuzzers.
2019-12-21 19:23:51 +01:00
Daniel Lemire b2ebdb0d07
I think we can align the numbers better (so it is prettier). (#399)
* I think we can align the numbers better (so it is prettier).

* Remove space before %, align third line better

Co-authored-by: John Keiser <john@johnkeiser.com>
2019-12-20 19:58:49 -05:00
dbj 9c3828fefe STRINGIFY implemented (#402)
* STRINGIFY implemented

* SIMDJSON_THREADS_ENABLED def/undef
2019-12-20 07:57:00 -05:00
John Keiser 60916318f7 Show miss rate, make it more accurate 2019-12-18 14:38:25 -08:00
John Keiser d7c83397e4 lookup+cont-check algorithm 2019-12-18 14:37:21 -08:00
Daniel Lemire 1d621bba37 Being more explicit about EMPTY errors. 2019-12-18 14:39:48 +00:00
John Keiser e2f349e7bd Measure impact of utf-8 blocks and structurals per block directly 2019-12-17 11:41:13 -08:00
Daniel Lemire 102262c7ab
Fixing issue386 (#396)
* Creating arch-specific bitmanipulation.h files.
* Improving system and compiler portability.
* We want to allow trailing_zeroes on zero inputs.
2019-12-16 19:09:18 -05:00
Daniel Lemire f02babe427 Adding analysis by @sebpop from https://github.com/lemire/simdjson/pull/391#issuecomment-565551462 2019-12-13 13:39:15 -05:00
Daniel Lemire fc6133b58f
Fixes issue 388 (#394) 2019-12-11 08:13:29 -05:00
Daniel Lemire 2bd65fa444
Adding @mswilson to the list of contributors 2019-12-10 13:14:12 -05:00
mswilson d33208c7db Correct detection of NEON support (#392)
... as the test as it is currently implemented will always evaluate to true.

Fixes #389
2019-12-10 13:12:17 -05:00
Daniel Lemire c9cd8e6211
PMULL is slow on ARM64, let us not rely on it? (#391) 2019-12-09 17:15:34 -05:00
Daniel Lemire 74a96878bc Correcting the spelling of macOS. 2019-12-05 15:09:53 -05:00
Daniel Lemire 7e28708e1d
Making it clearer that this is a 64-bit library 2019-12-05 09:18:49 -05:00
Daniel Lemire 1211c01ca1
Resolves issue 186 (#383)
* Resolves issue 186
https://github.com/lemire/simdjson/issues/186
2019-12-02 12:23:45 -05:00
Daniel Lemire f32b97733b Updating the json minifier benchmark to match that of the new API. 2019-12-02 10:46:03 -05:00
Jeremie Piotte 4e1c90f76f
Fix memory allocation of the max_depth in JsonStream. 2019-11-28 13:55:31 -05:00
Daniel Lemire e63f258470
missed one 2019-11-26 14:51:40 -05:00
Daniel Lemire ede9f9117f
Minor cleaning 2019-11-26 14:51:01 -05:00
Daniel Lemire 7c560fa137 Cleaning documentation. 2019-11-26 14:13:17 -05:00
piotte13 178a0842fe Refreshing the single header files. 2019-11-26 11:08:32 -05:00
piotte13 f345490cae Updating .gitignore for most popular IDEs 2019-11-26 10:59:18 -05:00
Jeremie Piotte db141e82c9
Specifying that RFC7464 is not supported 2019-11-26 10:33:33 -05:00
Jeremie Piotte f163155929 JsonStream documentation (#381)
* adding Multiline JSON competition chart to doc
* Completing the comments for JsonStream
* Adding a page for JsonStream's documentation.
2019-11-25 18:11:55 -05:00
John Keiser 9b6377fd80 Precalculate the ASCII path 2019-11-25 11:49:44 -08:00
John Keiser 7356b4532f Perform UTF-8 detection via flag lookup algorithm
- adds the alternative zwegner, range and lookup utf8 algorithms as well, for
ability to do "shootouts"
2019-11-25 11:49:44 -08:00
John Keiser 7d7bec856d Remove lookup_lower_4_bits
It's only a coincidence that it works in current uses: it doesn't do
what the name says. Particularly, if the high bit is 1 it will yield
0 even if the lower 4 bits would yield something else.
2019-11-25 11:49:44 -08:00
Paul Dreik c5504ef50b
run the oss fuzz initial seed corpus in CI (#378)
This makes sure the seed corpus keeps being healthy.
2019-11-23 22:49:41 +01:00
Daniel Lemire 3658ff650d
Delete Notes.md 2019-11-23 14:15:25 -05:00
Paul Dreik 6d14afd80e
Make threads optional in the cmake build (#376)
Only the simdjson library should optionally depend on threads,
the executables that link to simdjson will get the dependency
indirectly.

* add option for controlling threads (default is on)
* add CI testing with threading on/off for msvc, gcc and clang
* fix an unrelated copy paste comment error in the cirlce ci build conf
2019-11-22 21:51:46 +01:00
Jeremie Piotte 6e5178efc4
Update CONTRIBUTORS 2019-11-21 16:49:07 -05:00
Jeremie Piotte 29fc51522a
Introducing concurrency mode in JsonStream. (#373)
* JsonStream threaded prototype

* JsonStream Threaded version working. Still supporting non-threaded version.

* Fix where invalid files would enter infinite loop.

* SingleHeader update

* I will remove -pthread in cmake for now.

* Attempt at resolving the -pthread issue
2019-11-21 11:22:06 -05:00
Daniel Lemire 6cd8fb7982
Adding a getline benchmark (#344) 2019-11-20 20:33:16 -05:00
John Keiser ce824f8653 Decrease stage 1 step size to 64 bytes on Westmere/ARM
- Templatize scan_step() with STAGE1_STEP_SIZE
- Fix simd8::store()
- add NUM_CHUNKS to simd8
2019-11-18 21:58:07 -08:00
John Keiser 708f4a094d Move inline functions out of class definition for templating 2019-11-18 21:58:07 -08:00
Paul Dreik 2704b73399
Add fuzzer badge and improve fuzzer documentation (#367)
* Update Fuzzing.md

* add oss-fuzz badge
2019-11-13 16:57:20 +01:00
Paul Dreik 783ccd6c21
Add CI Fuzz job
This runs fuzzing for a short while, then executes the corpus through valgrind.
The extended corpus is uploaded to persistent storage on bintray.
2019-11-12 16:46:23 +01:00
Paul Dreik 3fd1c3b64a run short fuzzing and valgrind in github action 2019-11-11 22:17:32 +01:00
Daniel Lemire 58d249ca16
Introducing move assignments. (#363) 2019-11-09 10:34:32 -05:00
Jeremie Piotte bdc2b07339
Streams of JSON documents + Large files (>4GB) (#350) (#364)
* rough prototype working.  Needs more test and fine tuning.

* prototype working on large files.

* prototype working on large files.

* Adding benchmarks

* jsonstream API adjustment

* type

* minor fixes and cleaning.

* minor fixes and cleaning.

* removing warnings

* removing some copies

* runtime dispatch error fix

* makefile linking src/jsonstream.cpp

* fixing arm stage 1 headers

* fixing stage 2 headers

* fixing stage 1 arm header

* making jsonstream portable

* cleaning imports

* including <algorithms> for windows compiler

* cleaning benchmark imports

* adding jsonstream to amalgamation

* merged main into branch

* bug fix where JsonStream would bug on rare cases.

* Addind a JsonStream Demo to Amalgamation

* Fix for https://github.com/lemire/simdjson/issues/345

* Follow up test and fix for https://github.com/lemire/simdjson/issues/345 (#347)

* Final (?) fix for https://github.com/lemire/simdjson/issues/345

* Verbose basictest

* Being more forgiving of powers of ten.

* Let us zero the tail end.

* add basic fuzzers (#348)

* add basic fuzzing using libFuzzer

* let cmake respect cflags, otherwise the fuzzer flags go unnoticed

also, integrates badly with oss-fuzz

* add new fuzzer for minification, simplify the old one

* add fuzzer for the dump example

* clang format

* adding Paul Dreik

* rough prototype working.  Needs more test and fine tuning.

* prototype working on large files.

* prototype working on large files.

* Adding benchmarks

* jsonstream API adjustment

* type

* minor fixes and cleaning.

* Fixing issue 351 (#352)

* Fixing issues 351 and 353

* minor fixes and cleaning.

* removing warnings

* removing some copies

* Fix ARM compile errors on g++ 7.4 (#354)

* Fix ARM compilation errors

* Update singleheader

* runtime dispatch error fix

* makefile linking src/jsonstream.cpp

* fixing arm stage 1 headers

* fixing stage 2 headers

* fixing stage 1 arm header

* fix integer overflow in subnormal_power10 (#355)

detected by oss-fuzz

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714

* Adding new test file, following https://github.com/lemire/simdjson/pull/355

* making jsonstream portable

* cleaning imports

* including <algorithms> for windows compiler

* cleaning benchmark imports

* adding jsonstream to amalgamation

* merged main into branch

* bug fix where JsonStream would bug on rare cases.

* Addind a JsonStream Demo to Amalgamation

* merging main

* rough prototype working.  Needs more test and fine tuning.

* prototype working on large files.

* prototype working on large files.

* Adding benchmarks

* jsonstream API adjustment

* minor fixes and cleaning.

* minor fixes and cleaning.

* removing warnings

* removing some copies

* runtime dispatch error fix

* makefile linking src/jsonstream.cpp

* fixing arm stage 1 headers

* fixing stage 2 headers

* fixing stage 1 arm header

* making jsonstream portable

* cleaning imports

* including <algorithms> for windows compiler

* cleaning benchmark imports

* adding jsonstream to amalgamation

* bug fix where JsonStream would bug on rare cases.

* Addind a JsonStream Demo to Amalgamation

* rough prototype working.  Needs more test and fine tuning.

* minor fixes and cleaning.

* adding jsonstream to amalgamation

* merged main into branch

* Addind a JsonStream Demo to Amalgamation

* merging main

* merging main

* make file fix
2019-11-08 17:39:45 -05:00
Daniel Lemire 6888ca709d
Update README.md 2019-11-08 16:39:09 -05:00
Paul Dreik 8ae818e17c add ossfuzz support (#362)
* initial oss-fuzz friendly build

parts taken from libfmt, which I wrote and have the copyright to

* fix build error

* add script for building a corpus zip

see https://google.github.io/oss-fuzz/getting-started/new-project-guide/#seed-corpus

* fix zip command

* drop setting the C++ standard

* disable the minify fuzzer, does not pass oss-fuzz check-build test

* fix integer overflow in subnormal_power10

detected by oss-fuzz

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714

* invoke the build like oss fuzz does

* document what the scripts are for and how to use them

* add a page about fuzzing
2019-11-08 10:32:43 -05:00
Daniel Lemire c4f1baad31
Making get_corpus safer (#360) 2019-11-06 12:22:42 -05:00
Daniel Lemire 3439ce19c9
Adding a flag which allows us to disable AVX detection. This exposes a bug. (#356) 2019-11-06 10:39:26 -05:00