When find_field_unordered is used on an empty object, it calls
json_iterator::reenter_child. reenter_child asserts that it doesn't
rewind too far back by consulting parser->start_positions.
When the On Demand parser sees an empty object, it fails to update
parser->start_positions. This means that the assertion in
json_iterator::reenter_child reads stale data, or potentially
uninitialized memory. Reading uninitialized memory can cause spurious
assertion failures and Valgrind memcheck reports:
Running missing_keys_for_empty_top_level_object ...
==170679== Conditional jump or move depends on uninitialised value(s)
==170679== at 0x4943D7: reenter_child (json_iterator-inl.h:208)
==170679== by 0x4943D7: find_field_unordered_raw (value_iterator-inl.h:197)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:13)
==170679== by 0x4943D7: find_field_unordered (object-inl.h:96)
==170679== by 0x4943D7: find_field_unordered (value-inl.h:110)
==170679== by 0x4943D7: find_field_unordered (document-inl.h:105)
==170679== by 0x4943D7: object_tests::missing_keys_for_empty_top_level_object() (ondemand_object_tests.cpp:117)
==170679== by 0x4CA761: object_tests::run() (ondemand_object_tests.cpp:1085)
==170679== by 0x8BA314: int test_main<bool ()>(int, char**, bool ( const&)()) (test_ondemand.h:81)
==170679== by 0x4CA9C8: main (ondemand_object_tests.cpp:1119)
==170679==
Fix the read of uninitialized or stale memory by updating
parser->start_positions regardless of whether we see an empty object or
an object with some keys.
This commit only affects builds where development checks
(SIMDJSON_DEVELOPMENT_CHECKS) are enabled. Builds where development
checks are disabled are unaffected by this bug.
* Verifies bug with missing keys.
* Allowing search from any key.
* Workaround for buggy msys
* Restricting how we can end key searches.
* Adding a few tests.
* This moves all DOM (benchmark + test) files to a subdir
* Missing file.
* CMake + DLL is not pretty.
* Capitalizing AND
* Fixing mismatch endif
* Flipping the order.
* onedemand => ondemand
* Remove CMP0025 policy
This policy is already set to NEW by the minimum required version.
* Use HOMEPAGE_URL in the project call
* Use VERSION in the project call
* Detect if this is the top project
* Port simdjson-user-cmakecache to a CMake script
* Create a developer mode
The SIMDJSON_DEVELOPER_MODE option set to ON will enable targets that
are only useful for developers of simdjson.
* Consolidate root CML commands into logical sections
* Warn about intended use of developer mode
* Prettify the just_ascii test
* Remove redundant CMake variables
* Inline CML contents from include and src
* Raise minimum CMake requirement to 3.14
* Define proper install rules
* Restore thread support variable
* Add BUILD_SHARED_LIBS as a top level only option
* Force developer mode to be on in CI
* Include flags earlier in developer mode
* Set CMAKE_BUILD_TYPE conditionally
CMAKE_BUILD_TYPE is used only by single configuration generators and is
otherwise completely ignored.
* Remove useless static/shared options
simdjson now uses the CMake builtin BUILD_SHARED_LIBS to switch the
built artifact's type.
* Remove unused CMAKE_MODULE_PATH variable
* Refactor implementation switching into a module
* Factor exception option out into a module
* Reformat simdjson-flags.cmake
* Rename simdjson-flags to developer-options
* Accumulate properties into an include module
This is done this way to avoid using utility targets that must be
exported and installed, which could potentially be misused by users of
the library.
* Port impl definitions to props
* Port exception options to props
* Lift normal options to the top
* Port developer options to props
* Remove simdjson-flags from benchmark
* Document the developer mode in HACKING
* Fix include path in installed config file
* Fix formatting of prop commands
* Fix tests that include .cpp files
* Change GCC AVX fixes back to compile options
* Deprecate SIMDJSON_BUILD_STATIC
* Always link fuzz targets to simdjson
* Install CMake from simdjson's debian repo
* Add gnupg for apt-key
* Make sure ASan link flags come first
* Pass CI env variable to cmake invocation
* Install package for apt-add-repository
* Remove return() from flush macro
* Use directory level commands instead of props
* Restore the github repository variable
* Set developer mode unconditionally for checkperf
The CI env variable is only set in the CI and this target is always run
in developer mode.
* Attempt to fix ODR violation in parsing checks
These tests were compiling the simdjson.cpp file again and linking to
the simdjson library target causes ODR violations.
Instead of linking to the target, just inherit its props.
* Move variables before the source dir
* Mark props to be flushed after adding more
* Use props for every command for the library
* Use keyword form for linking libs
* Handle deprecation of SIMDJSON_JUST_LIBRARY
* Handle deprecations in a separate module
Co-authored-by: friendlyanon <friendlyanon@users.noreply.github.com>
* Truncate final unclosed string.
* Adding more precise remarks.
* Better documentation and more robust code.
* ARM + PPC corrections.
* Patching ARM implementation with new stage1_mode parameter.
* Fixed most problems.
* Correcting white spaces and adding a remark.
* This adds the truncated_bytes() method to the stream instances.
* This implementations string serialization for On Demand instances.
* Adding more documentation.
* Another remark.
* Marking the new functions as inline.
* casts apparently do not work.
* Upgrading the API.
* Making the code really free from exceptions.
* At another fix for exceptionless.
* Modify to_chars so that it does not pad integers with '.0'.
* Negative 0 cannot be expressed as an integer.
* Again, accomodating exceptionless usage.
* Using x <= -0 does not allow you to determine the sign since 0 <= -0. I am not sure where
this bug comes from.
* This gives the CMake install the necessarily information (and flags) to know
whether we have a Windows DLL and in such cases how to handle the linkage.
This avoids a very unlikely buffer overrun that can occur in a particular kind of invalid JSON:
- the document is invalid with an unclosed top level array or object
- the last thing in the document is a number that ends at EOF
- the padding is filled entirely with numeric digits
Daniel Lemire
strager
Dirk Stolle
friendlyanon
John Keiser
John Keiser
David CARLIER