Commit Graph

1480 Commits

Author SHA1 Message Date
Mike Lischke f292d14abc The parser is parsing for the first time.
- Removed ultra simple test grammar + parser. No longer needed.
- Removed long list of keywords from (regular) test grammar.
- Fixed a number of toString() methods to get better debug output.
- Moved Ref typedefs from Declarations.h to the individual classes as defining them on the forward declarations totally confuses the XCode debugger.
- Removed reference to the owning ATN in an ATNState. We cannot guarantee to have the correct address there due to the way the states are created. The reference is not needed anyway.
- ATNDeserializationOptions now has verifyATN set by default (as in the Java target).
- Had to add a workaround for a weird situation: static initialization in ATNDeseralizer stopped working for no apparent reason. Need to investigate this.
- Added a few support methods to the CPPUtils, mostly to ease debug output creation.
- Added console listener by default to the listeners list (as done in the Java target).
- Fixed translation mistakes in the CommonTokenStream class.
- Fixed some memory leaks and exception handling bugs.
2016-04-17 13:13:15 +02:00
Mike Lischke eb0241f767 Another refactoring round.
- Removed a few unused classes.
- More raw pointers to smart pointers conversion: RuleContext, ParserRuleContext, ParseTree, Token, ParseTreeWalker, Tree...
- BitSet is now used directly instead of all those dynamic allocations and is a derived class instead of a composite.
- Replced ATNState equals with == operator overload.
- Correct wrong iterator over ATNConfigsets.
- Added utilitiy function that mimics Java's generic toString().
2016-04-13 19:05:56 +02:00
Mike Lischke 7f8ad7bd2d Applied consistent exception model and fixed is<> helper.
- Exceptions are now consistently thrown by value and captured by reference. C++11 exception_ptr and nested_exception are used when exception references are neeeded or when implementing the equivalent of Java's nesting.
- The is<> helper didn't handle properly (const) references, which is now explicitly handled. Added new unit tests for that.
- Fixed a number of places where a catch all was used to implement a "finally" (which hides exceptions).
- Changed exceptions to hold (temporary) raw pointers instead of shared pointers, as otherwise it is tried to free wrapped pointers which might just be references to static objects. Might later be updated again when we continue with removing raw pointers.
- Some smaller fixes.
- The generated simple parser now runs through without any error (yet, it doesn't do anything useful).

- ANTLR C++ target template:
  - Added getListener and genVisitor bool members to ANTLR's LexerFile + ParserFile classes, so can use them in the template.
  - Made addition of listener #include dependent on the new genListener member, which allows to run parser generation without listeners/visitors.
2016-04-09 16:36:52 +02:00
Mike Lischke d1b59ca5af Next overhaul (more TODOs fixed)
- Added an even simpler grammar to ease debugging while getting the lib into a working state.
- Added helper template is<> to ease frequent type checks (for value types, ref types and shared_ptr). Added some unit tests for that as well.
- Changed the MurmurHash::hashCode() function to take shared_ptr as this is the only variant we need. Had to change the MurmurHash unit tests for that.
- Removed conflicting IntStream::_EOF (and other variants). We use the C runtime EOF value instead.
- Changed all references to semantic contexts, prediction context and the prediction context cache to use shared_ptr<>. Created *Ref typedefs to simplify usage.
- Adjusted the C++ string templates for that.
- Fixed a number of memory leaks + some cleanup.
2016-04-06 15:07:25 +02:00
Mike Lischke 1ca5b38868 Fixed some TODOs + exceptions.
- Reworked the exception hierarchy to conform with the Java hierarchy (where we mimic that). Ultimative base class is std::exception, which uses std::string (char* actually) for messages, so all exceptions use std::string for that as well. Consider that as first step to rework the entire lib to use std::string instead of std::wstring (with utf-8 for full Unicode support).
- Removed ASSERTException + TODOException and fixed the places where they were used.
- Removed ANTLRException, which was only an intermediate layer without an equivalent on Java side.
- Replaced some equals() calls by == (with defined operator overloading).
- Enhanced Arrays::equals() to ensure it compiles only if the actual types being compared support the != operator (both value + reference types).
- Made the Recognizer class template free by using plain polymorphism. Some adjustments were need also in the Cpp template to support that. Could convert the .inl file to .cpp then.
2016-03-31 18:32:44 +02:00
Mike Lischke 29dedd17c4 New unit tests + template enhancements + more memory handling work
- Added IntervalSet unit tests. Fixed a few bugs found by that.
- Enhanced the demo grammar so that we use as many as possible template rules from Cpp.stg. Still not fully done it seems.
- Fixed bugs in size determination for arrays (vectors now).
- Simplified PredictionContext and SemanticContext (one template parameter less).
- Removed no longer used Utils.h/cpp. Fixed CPPUtils.h/cpp.
- Extended LexerXXCommand template rules to take a new grammar parameter (code gen has been updated) as we need this context in the Cpp target. This change requires to update all existing templates! Cannot do here as this is an old revision.
- Some cleanup.
2016-03-30 20:11:05 +02:00
Mike Lischke 3f78367457 Next overhaul
- Added first real unit test set and enable code coverage collection in XCode (for ANTLRInputStream).
- Reworked ANTLRFileStream::load, which is now more flexible (supports Unicode BOM + 3 possible encodings), can load from Unicode file names and has almost no platform code.
- Enabled strict data size and sign checks in XCode (clang) and fixed a million places...
- Started converting int to size_t where it makes more sense.
- Started working on const correctness.
- Fixed a ton of memory leaks.
- The ATN and ATNConfigSet classes now entirely work as value types. Same for Interval(Set). These seem to be the most critical data structures (ATNConfig + ATNState are pending).
- The abstract IntSet class is gone now.
- Murmur hash code now works with size_t instead of int (need to add unit tests for that).
- Fixed a number of TODOs and other smaller things.

- The Cpp template now properly handles grammar rule return values.
2016-03-27 22:01:03 +02:00
Mike Lischke e97820a27e Reworked handling of ATN instances throught the code + definition of IRecognizer.
ATNs are top level structures, which are created and kept by parser/lexer classes (or their simulator equivalents). Hence there are now value types in their controlling class and passed around as const &.

IRecognizer was a template class without real need, which has been changed to make it a simple interface easily usable without having to find C++ hacks for fancy Java wildcard generics.
2016-03-24 11:49:44 +01:00
Mike Lischke 6df5d025bf Complete formatting overhaul.
- Reformatted every single file to have a consistent indentation style using only space chars, with 2 chars per indentation. Reduced huge indentation due to deep namespace nesting by not indenting namespaces.
- Reduced #include usage to a minimum.
- Made copyright header the first entry in all files.
- Moved the previously mac-only prefix file (antlrcpp-Prefix.h) to the runtime. It can now be used by all platforms and includes all necessary standard headers.
- Removed a number of unused files.
2016-03-20 17:21:46 +01:00
Mike Lischke fec65477d8 ATN deserialization + some initialization and memory leak fixes
- ATN deserialization finally works.
- Changed a number of pointer to STL classes to just the STL classes and pass them around by const & where necessary.
2016-03-19 17:18:06 +01:00
Mike Lischke 11571fa092 A few more adjustments to make the merged changes work with this revision of antlr. 2016-03-18 15:27:17 +01:00
Mike Lischke 1672bc0739 Added all changes done so far. Since we are here on a very old revision I cannot simply merge, so all files have manually been copied and we have no history for these changes. 2016-03-18 14:22:42 +01:00
Terence Parr 6b2817f8bb get last not first when get() finds multiple matching nodes. 2013-12-20 12:47:19 -08:00
Terence Parr 8c5d088eb7 Merge pull request #378 from sharwell/polish
Tree patterns polish
2013-12-19 17:24:42 -08:00
Sam Harwell fb1880d82c Move ATNSerializer to runtime 2013-12-19 19:07:24 -06:00
Sam Harwell bc59f30857 Use ATNDeserializer methods instead of deprecated ATNSimulator methods 2013-12-19 19:07:23 -06:00
Sam Harwell 2a9a716c53 Remove unnecessary methods ParseTreeMatch.getText() and failed() (use getTree().getText() and !succeeded() instead) 2013-12-19 19:06:22 -06:00
Sam Harwell 72675075cf Remove unnecessary testing constructor 2013-12-19 19:06:21 -06:00
Sam Harwell 75b8174dc8 Clean up the result caching for getTokenTypeMap and getRuleIndexMap 2013-12-19 19:06:20 -06:00
Sam Harwell df61690758 Clean up the caching of ATN instances with bypass alternatives 2013-12-19 19:06:19 -06:00
Terence Parr 9ca6bf9bd3 fix null pointer bug with rule "a : a;" 2013-12-19 16:35:37 -08:00
Terence Parr bd91dc166d add getTokenTypeMap(), getRuleIndexMap() to recognizer. Gen new fields for that an ATN with bypass alts. Then methods for that: getATNWithBypassAlts(). Big changes to interface for ParseTreeMatch; create Parser.compileParseTreePattern() method. Convert rule names to rule indexes. 2013-11-24 14:04:46 -08:00
Terence Parr 4c52a103e1 cleanup 2013-11-22 11:31:59 -08:00
Terence Parr b2ec85d14d updated comments, cleaned up the API, made helper routines. 2013-11-22 11:08:16 -08:00
Terence Parr 168bce79d3 working on api 2013-11-22 09:55:40 -08:00
Terence Parr 4fdd520c2e got labels working 2013-11-21 16:01:02 -08:00
Terence Parr 1e83557819 reorg to pass round a match object in matches_() 2013-11-21 10:30:37 -08:00
Terence Parr e836544d30 add test for hidden tokens 2013-11-21 09:55:26 -08:00
Terence Parr e68757717f pass a lexer and a parser to the tree pattern match or instead of the classes. 2013-11-21 09:32:45 -08:00
Terence Parr 33d11709df Revert "fix comment"
This reverts commit e8f577e162.
2013-11-21 08:42:28 -08:00
Terence Parr e8f577e162 fix comment 2013-11-21 08:42:02 -08:00
Terence Parr 40789babf5 got tree pattern matching working on my test cases using a parser interpreter not been generated parser to compile patterns like "<ID> = <expr>;". It uses Sam's alteration to the ATN to have bypass alternatives for <expr> tags. 2013-11-20 17:54:57 -08:00
Terence Parr 2394b38995 added state to be consistent with enterRule(); most importantly, we need to set the state when we enter a recursive rule, which we are not doing at the moment. This was not a problem before because we never could directly call a recursive rule and the outer rule set the current state. Now that we are using recursive rules as start rules, we need to set the state. 2013-11-20 17:53:51 -08:00
Terence Parr 2791dd5619 Merge branch 'master' into tree-patterns 2013-11-20 16:57:16 -08:00
Terence Parr 8c57233db7 mv to runtime. damn git! 2013-11-20 16:53:49 -08:00
Terence Parr 1c71d05074 Merge branch 'master' into tree-patterns 2013-11-20 16:49:27 -08:00
Terence Parr cc07f8faec mv to runtime. was ref'ing v3 not v4 Token. 2013-11-20 16:47:17 -08:00
Terence Parr 578dcd0799 reformat 2013-11-20 16:36:34 -08:00
Terence Parr c7e9a410e9 rm unused field _contextStack 2013-11-20 16:35:55 -08:00
Terence Parr 48ce1f1809 Support getting the serialized ATN from a recognizer 2013-11-20 16:22:01 -08:00
Terence Parr d09510a936 Merge branch 'master' into tree-patterns 2013-11-20 16:09:36 -08:00
Terence Parr 41d9d5bfd4 Merge pull request #357 from sharwell/atn-deserializer
ATN deserializer
2013-11-20 14:33:07 -08:00
Terence Parr 42e5343287 Merge branch 'master' into tree-patterns 2013-11-20 14:31:08 -08:00
Terence Parr 5a6709aba6 update changes file, tweak parser interp test to use new api 2013-11-20 14:28:51 -08:00
Terence Parr 02f937b115 Merge branch 'master' into tree-patterns 2013-11-20 14:20:09 -08:00
Sam Harwell b447256f20 Extract ATN deserialization to a new class ATNDeserializer 2013-11-20 09:14:18 -06:00
Sam Harwell 6eb8111756 Generate method stub so external calls directly to LR rules do not require the precedence argument 2013-11-15 16:15:34 -06:00
Terence Parr 8cace4e182 pull master BaseTest 2013-11-14 14:47:49 -08:00
Terence Parr 2f902da3d2 pull master into branch 2013-11-14 14:43:50 -08:00
Sam Harwell 3309c9d85e Fix build warnings 2013-11-14 15:36:01 -06:00