Commit Graph

2086 Commits

Author SHA1 Message Date
Convery e6258377d9
Additional define to silence the 32-bit portability warning. (#1279)
Under compilers like MSVC the `#pragma message` about portability will be issued for each translation module; regardless of whether or not `SIMDJSON_PORTABILITY_H` has already been defined. As such a new define `SIMDJSON_NO_PORTABILITY_WARNING`, can be defined prior to the inclusion of `<simdjson.h>` to silence it.
2020-11-02 08:37:13 -05:00
Jeremy Ong b3a26fd0cc
Do not enable -fPIC when compiling on Windows (#1280)
Regardless of the compiler, Windows targets do not support -fPIC,
as position independent code is already implicitly enabled. Compiling
simdjson with Clang on Windows will error because -fPIC is an
unsupported option for target 'x86_64-pc-windows-msvc'.
2020-11-02 08:18:04 -05:00
Paul Dreik 47669566da
fix unintended early return in ondemand unit test loops (#1263)
fix uninteded early return in ondemand unit test loops

go through and fix warnings appearing in qtcreator,

qualify with std::, add const, abort on error

get rid of ulp_distance, not needed anymore when parsing is exact
2020-11-01 19:08:40 +01:00
Paul Dreik 03a1bb0a5b
detect if amalgamation is needed (#1269)
output amalgamation date to be locale and timezone independent
2020-11-01 19:07:15 +01:00
Paul Dreik 54ffbbe7db
remove asserts from compute_float_64 (#1276) 2020-11-01 18:34:42 +01:00
Paul Dreik 0b82f07115
fix segfault in numberparsing #1273 (#1274)
This was a read overflow.
2020-11-01 18:27:21 +01:00
Paul Dreik 265db2e533
fix non ascii sources (#1275)
Master does not build because of non-ascii sources, merging without waiting for CI.
2020-11-01 11:14:01 +01:00
Paul Dreik 2b2abf4895 reamalgamate 2020-11-01 07:03:17 +01:00
Paul Dreik f93fb21c95
optionally disable deprecated apis (#1271)
Introduce cmake option SIMDJSON_DISABLE_DEPRECATED_API (default Off)
which turns off deprecated simdjson api functions by setting the macro
 SIMDJSON_DISABLE_DEPRECATED_API.

For non-cmake users, users will have to set SIMDJSON_DISABLE_DEPRECATED_API
by some other means to disable the api.

Closes #1264
2020-11-01 06:38:52 +01:00
Paul Dreik b7fe764e6c
fuzz more of generic/numberparsing.h (#1272)
make the ondemand fuzzer use more of the api
2020-11-01 06:29:16 +01:00
Paul Dreik 500e4c3572
fuzz with the intended clang version (#1267)
This builds the CI fuzzers with the intended clang version. It also allows users to set the clang version locally,
in case they need to.

It also switches the CI fuzzers to use an optimized sanitizer build, to do something oss-fuzz doesn't and get more done in the short time the CI fuzzer runs.
2020-10-31 08:22:49 +01:00
friendlyanon 55281c01fb
Fix unintended Drone CI break (#1266)
* Use 'set -e' as it is portable

* Install git on the i386 images

* Remove CMake PPA from arm64 images

CMake's PPA doesn't actually distribute arm64 binary packages. See
related issues:
https://gitlab.kitware.com/cmake/cmake/-/issues/17923
https://gitlab.kitware.com/cmake/cmake/-/issues/20122

* Use Debian Buster instead of Ubuntu Bionic Beaver

Buster has all the packages in apt that is needed, which is convenient

* Install Clang 6.0 explicitly

Make sure that what gets installed is clang 6.0 as the step name
suggests.
https://packages.debian.org/buster/clang-6.0

* Install CMake from buster-backports

Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
2020-10-30 22:47:40 -04:00
Daniel Lemire a8bf10ea5a Minor patch. 2020-10-30 14:51:50 -04:00
friendlyanon 9f60093b41
Update CMake to at least 3.16 in Drone CI (#1261)
* Add script for CMake PPA

* Call the CMake PPA script in Drone CI

"apt-get update -qq" can be omitted, as that command is already called
by the script to pull in necessary packages for the CMake GPG keys.

* Remove sudo calls in the CMake PPA script

This script is intended to be run in Docker images, where the default
user is already root.

* Use echo instead of printf

* Use /etc/os-release instead of lsb_release

lsd_release could be installed, but os-release is just more convenient
to grab the version code from at this point.

* On Debian images grab CMake from buster-backports

It's not wise to mix Ubuntu PPAs with Debian and buster-backports has
CMake 3.16, which is recent enough for our purposes.

Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
2020-10-30 13:55:35 -04:00
Daniel Lemire 562d4f4f58 Fixing a minor logical error. 2020-10-29 16:42:50 -04:00
Paul Dreik ac87437588
fuzz the on demand api (#1220) 2020-10-29 19:14:44 +01:00
Daniel Lemire b1444b4dfb
Mostly tiny changes, with one optimization to fallback for number parsing. (#1265)
* Mostly tiny changes, with one optimization to fallback for number parsing.

* Missed an update.
2020-10-29 11:18:11 -04:00
friendlyanon ed21875083
Remove old CMake version (#1259)
The images used by github already come with a sufficiently recent CMake
version instead of a very outdated one. See:
https://github.com/actions/virtual-environments

Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
2020-10-28 09:44:13 -04:00
Daniel Lemire 0249b5b311
Update CONTRIBUTORS 2020-10-27 18:44:28 -04:00
Danila Kutenin f46a0f64f2
PPC64 support (#1254)
* Initial PPC64 support

* Add travis CI

* Fix outdated cmake version for travis

* Fix indendtation

* Try another workaround for outdated cmake in travis

* Try beta cmake

* Add dash before beta

* Use builtin snaps

* Use cmake as rocksdb

* Test cmake on bionic

* Remove unnecessary things from travis

* Remove unnecessary things from travis

* Another try of compiler install

* Add all major compilers

* Add all major compilers

* Add all major compilers

* Tweak travis a bit

* Typo

* More robust travis

* Typos typos typos

* Add fewer compilers, add non specific build for clang and gcc, should be the final config

* CMAKE_FLAGS is in incorrect place

* Remove default implementation

* Limit build thread number

* Fall back prefix_xor to a usual implementation, no performance boost is noticed

* Test for power9 as it is the main architecture for OpenPOWER right now

* Add to documentation to build with power9 as the implementation is compatible but compiler optimizations is not

* Replace ARM with PPC in the comment
2020-10-27 18:43:39 -04:00
Jonathan Wakely 1fd0447dbb
Remove repeated words (#1252) 2020-10-26 20:41:01 -04:00
Jeremy Ong 9856201f5c
Disable exceptions when compiling with MSVC (#1256)
Projects that link simdjson from MSVC with exceptions off will
include simdjson headers which transitively include STL headers.
The MSVC STL stipulates that _HAS_EXCEPTIONS=0 be defined or code
requiring exceptions will be enabled. This change adds a new job
to the appveyor build matrix to verify the build and tests with
exceptions disabled, and disables exceptions at the compiler level
when SIMDJSON_EXCEPTIONS is specified to OFF.
2020-10-26 14:11:25 -04:00
Daniel Lemire a75c07065f
Fix for issue 1246. We document the relationship between parser instances and elements (#1250)
* Fix for issue 1246.

* Adopting John's wording.
2020-10-26 08:40:45 -04:00
Daniel Lemire 6a86ef5a7d Issue release 2020-10-23 09:32:25 -04:00
Daniel Lemire 500e5d6759
Update README.md 2020-10-23 09:21:16 -04:00
Daniel Lemire e3f2c9f29a
Update README.md 2020-10-23 09:18:47 -04:00
Daniel Lemire 3458e6248b
Update README.md 2020-10-23 09:15:04 -04:00
Daniel Lemire 14039d05a9
Adding a new benchmark for ondemand: distinct user id (#1239)
* Adding a distinct user id benchmark

* reenabling everything

* Removing an unnecessary "value()".

* Better tests of the examples and some fixes.

* Guarding exception code.
2020-10-23 08:47:01 -04:00
Daniel Lemire c592da4937
Adds yyjson to our internal benchmarks. (#1244) 2020-10-21 16:23:20 -04:00
Daniel Lemire 61fb4244f2
There are a couple of extra value() that should not be there (#1242) 2020-10-21 12:37:22 -04:00
Daniel Lemire a503e022d8
Update README.md 2020-10-21 11:43:36 -04:00
Daniel Lemire 0942dc0764
This fixes a typo and makes the types more explicit (#1241) 2020-10-20 17:41:37 -04:00
Daniel Lemire 0d6919dd99
Reenable the on-demand tests and allows us to convert a raw string into a C++ string. (#1232)
* Reenable the on-demand tests and allows us to convert a raw string into a C++ string.

* Fixing a 1-byte buffer overrun.

* More documentation.

* Adding more tests.

* Enabling the new tests

* Committing a nicer example.

* Not yet happy but this should fix our failures.

* Duh.

* Ok. Making it easier to get string_view instances from field instances.

* It is a struct.

* Trying to satisfy VS.

* Adopting John's name.
2020-10-19 20:22:24 -04:00
Daniel Lemire 3e8e797bc2 Typo. 2020-10-19 17:30:52 -04:00
Daniel Lemire 0a907ec694
Tweaking further the documentation. (#1237)
* Tweaking further the documentation.

* More details.

* Another sentence.

* Saving.

* Tweaking more
2020-10-19 16:51:04 -04:00
Paul Dreik f1b4a54991
add fuzz element (#1204)
* add definitions for is_number and tie (by lemire)
* add fuzzer for element
* update fuzz documentation
* fix UB in creating an empty padded string
* don't bother null terminating padded_string, it is done by the std::memset already
*  refactor fuzz data splitting into a separate class
2020-10-17 05:48:50 +02:00
Paul Dreik 58e7106df1
remove unused function parse_unsigned 2020-10-16 22:17:11 +02:00
Paul Dreik 7bf391c54a
fix potential use of uninitialized value warning, avoid casting away const
This fixes a "potentially use of uninitialized value" warning, as well as a cstyle cast to non-const.
2020-10-16 22:14:42 +02:00
Daniel Lemire 07a6e098c8
This would allow users to find out what builtin is. (#1227)
* This would allow users to find out what builtin is.

* Trying another approach.

* Added instructions.

* Cleaning up the printout.

* Let us be less invasive.

* Adding a comment.
2020-10-15 21:58:42 -04:00
Daniel Lemire e4897d6b54
We have hardcoded 32 (#1236) 2020-10-15 21:57:10 -04:00
Daniel Lemire 23026d966b Tweaking. 2020-10-15 21:55:23 -04:00
Daniel Lemire 3cd98df30d
This adds new tests regarding ordering. (#1233)
* This adds new tests regarding ordering.

* Updating the documentation with more examples.

* Adding compilation tests.

* Pruning code for exceptions.

* Guarding exceptionless.
2020-10-15 16:41:14 -04:00
Daniel Lemire 001be23258
Being more specific regarding the padding. (#1228)
* Being more specific regarding the padding.

* Even more precise.
2020-10-14 13:35:51 -04:00
Daniel Lemire c85b6682e0
This is a cleaner on-demand documentation (for discussion). (#1226)
* This is a cleaner on-demand documentation (for discussion).

* Added stable APIs.
2020-10-14 13:35:28 -04:00
Daniel Lemire bb2bc98a22
Fix issue https://github.com/simdjson/simdjson/issues/1127 (#1224) 2020-10-13 09:18:54 -04:00
Daniel Lemire 43da4f7ccc Corrected number 2020-10-12 17:59:13 -04:00
Daniel Lemire 37e6d1e9c7
new number parsing (#1222)
* Remove our dependency on strtod_l by bundling our own slow path.

* Ok. Let us drop strtod entirely.

* Trimming down the powers to -342.

* Removing useless line.

* Many more comments.

* Adding some DLL exports.

* Let the gods help those who rely on windows+gcc.

* Marking the subnormals as unlikely. This is pretty much "performance neutral", but it might help just a bit with twitter.json.
2020-10-10 12:47:49 -04:00
Paul Dreik 1d9926698e
update how boost.json is invoked, fix missing separators (#1203)
* initial try at adding boost json to the benchmark

* clean up

* qualify memcpy etc. with std::

* clang format

* extra space

* update benchmark with help from Vinnie Falco from Boost.json

* add missing separators
2020-10-09 18:22:37 -04:00
Daniel Lemire b04f64e02c
Simplifying slightly the logic in the cmake (#1219) 2020-10-09 18:10:58 -04:00
Daniel Lemire c6d710b14b Merge branch 'master' of github.com:simdjson/simdjson 2020-10-09 16:46:51 -04:00