0d4e501239
Fixing the bug.
2020-06-17 10:06:16 -04:00
fd44c2a2ff
Merge pull request #927 from simdjson/dlemire/exposingthestringminifier
...
Exposing the string minifier.
2020-06-13 07:47:20 -07:00
a86a82b39c
Rename minify class to minifier so the minify() method is cleared up
2020-06-12 17:05:25 -07:00
4dfbf98e4e
Using a worker instead of a thread per batch ( #920 )
...
In the parse_many function, we have one thread doing the stage 1, while the main thread does stage 2. So if stage 1 and stage 2 take half the time, the parse_many could run at twice the speed. It is unlikely to do so. Still, we see benefits of about 40% due to threading.
To achieve this interleaving, we load the data in batches (blocks) of some size. In the current code (master), we create a new thread for each batch. Thread creation is expensive so our approach only works over sizeable batches. This PR improves things and makes parse_many faster when using small batches.
This fixes our parse_stream benchmark which is just busted.
This replaces the one-thread per batch routine by a worker object that reuses the same thread. In benchmarks, this allows us to get the same maximal speed, but with smaller processing blocks. It does not help much with larger blocks because the cost of the thread create gets amortized efficiently.
This PR makes parse_many beneficial over small datasets. It also makes us less dependent on the thread creation time.
Unfortunately, it is going to be difficult to say anything definitive in general. The cost of creating a thread varies widely depending on the OS. On some systems, it might be cheap, in others very expensive. It should be expected that the new code will depend less drastically on the performances of the underlying system, since we create juste one thread.
Co-authored-by: John Keiser <john@johnkeiser.com>
Co-authored-by: Daniel Lemire <lemire@gmai.com>
2020-06-12 16:51:18 -04:00
bbd61eb13f
Let tape writing be put in a register
2020-06-12 09:18:20 -07:00
a6e4933d93
Exposing the string minifier.
2020-06-11 13:07:18 -04:00
fe01da077e
Make threaded version work again
2020-06-07 16:21:00 -07:00
d43a4e9df9
Remove SUCCESS_AND_HAS_MORE (internal only value)
2020-06-07 16:20:55 -07:00
ef63a84a3e
Move document stream state to implementation
2020-06-07 16:20:44 -07:00
7a69da16e4
Fixing issue 906 ( #912 )
...
* Fixing issue 906
* Safe patching.
* Now with explanations.
* Bumping up memory allocation.
* Putting the patch back.
* fallback fixes.
Co-authored-by: Daniel Lemire <lemire@gmai.com>
2020-06-05 15:37:09 -04:00
b75fa26dc1
Move containing_scope and ret_address to .cpp
2020-06-01 12:15:55 -07:00
3d22a2d845
One weird trick: set a bogus error value in the parser impl
...
This makes us faster under both gcc and clang somehow.
2020-06-01 12:15:55 -07:00
1aab4752e2
Store all parser state in the implementation
2020-06-01 12:15:54 -07:00
6a71b24495
Reuse stored buf and len from parser
2020-06-01 12:14:09 -07:00
a3a9bde83e
Move DOM parsing into concrete interface implementation
2020-06-01 12:14:09 -07:00
40d57da83c
fixes issue 891 ( #893 )
2020-05-20 11:54:53 -04:00
e6c9dfbd91
Make include files more fine-grained
2020-05-19 14:42:04 -07:00
7ad4020829
Make main compilation chunks into .cpp files
2020-05-19 13:32:35 -07:00
a476531524
Share ref_address everywhere it's used
2020-05-19 13:30:34 -07:00
e03c5e9f23
We should guard the include ( #881 )
2020-05-13 20:02:46 -04:00
dbb3316511
Move current_string_buf_loc to stage 2
2020-05-11 06:11:32 -07:00
cd6f204c77
Move write_tape() to stage 2 code
2020-05-11 06:09:48 -07:00
269131ed21
Move on_number_* to stage 2 code
2020-05-11 06:04:54 -07:00
65d784e88e
Move on_start/end_string to stage 2 code
2020-05-11 05:49:40 -07:00
35afb6cae0
Move on_error, on_success to stage 2 code
2020-05-11 05:46:18 -07:00
4f25b6ac0c
Move on_end_* to stage 2 code
2020-05-11 05:34:49 -07:00
3d5ed1a7e3
Move on_start_* to stage 2 code
2020-05-11 05:30:35 -07:00
a03115a4a6
Move end_scope to stage 2 code
2020-05-11 05:24:12 -07:00
7219d28a31
Call end_scope directly from stage 2 code
2020-05-11 05:20:04 -07:00
0875bce68f
Don't pass depth to on_end_*
2020-05-11 05:15:39 -07:00
54fe302907
Don't pass depth to end_scope
2020-05-11 05:06:41 -07:00
edaa8f811f
Move on_start_* depth management to stage 2 code
2020-05-11 05:03:25 -07:00
2c8fd109de
Move increment_count to stage 2
2020-05-11 04:58:50 -07:00
16d88cc095
Don't pass depth to increment_count
2020-05-11 04:15:02 -07:00
2a6e6b3dbd
Cleaning string_view ( #872 )
...
* Cleaning string_view
* Corrected typo
* Alignment.
2020-05-10 16:05:52 -04:00
afb369950c
Disable Intellisense-only warnings in simdjson.h/cpp
2020-05-04 11:47:04 -07:00
1d06624d38
Unset /D_CRT_SECURE_NO_WARNINGS
...
- Also localize DISABLE_DEPRECATED_WARNING so that we catch other
deprecations
2020-05-04 11:35:05 -07:00
d40069a018
Disable deprecation warnings for VS builds
...
fopen/getenv are standard c++ that are not deprecated.
2020-05-04 11:34:00 -07:00
e04cbd71d0
Only install singleheader/simdjson.h as part of the public API
2020-05-02 01:44:11 +03:00
fc1ddcd2f8
Faster case-insensitive comparisons. ( #837 )
...
* Faster case-insensitive comparisons.
2020-04-30 15:52:28 -04:00
73d7d704c1
CMake: Remove export_private_library
...
Since we are exporting all the targets as part of the main simdjson target we do not need private
exports anymore
2020-04-30 02:06:19 +03:00
eee07e6cfd
Use the same export name for all targets
2020-04-29 23:47:27 +03:00
0f9dbf84b7
Fix incorrect check for case insensitive key lookup ( #824 )
2020-04-29 13:55:28 -04:00
2a1f8fa8f1
Provides support for clang under Windows. ( #817 )
2020-04-27 22:09:27 -04:00
49da7e74cd
usage.md -> basics.md ( #823 )
2020-04-27 16:03:19 -04:00
0514588175
Improves clang-cl build with Visual Studio ( #809 )
2020-04-27 08:59:32 -04:00
b99a7344c9
missing spaces.
2020-04-25 22:26:18 -04:00
f3ac0be0e6
Merge branch 'master' of github.com:simdjson/simdjson
2020-04-23 18:39:56 -04:00
18c9468af5
Fixed typo
2020-04-23 18:39:32 -04:00
d4239aaa8f
default initialisaiton ( #779 )
...
* padded_string.* default initialisation
parsedjson_iterator - copy constructor; depth_index not necessary
2020-04-23 18:32:11 -04:00
Daniel Lemire
John Keiser
Daniel Lemire
Pavel P
Furkan Usta
Nong Li
ostri