a40073a8cc
Merge remote-tracking branch 'antlr/master' into non-greedy
2012-10-15 01:58:15 -05:00
bbe77782db
Fix state checking in new non-greedy handling
2012-10-15 00:56:25 -05:00
aa2223cd6a
Fix assertion in PredictionContext.mergeArrays
2012-10-14 22:40:34 -05:00
7c221f13ed
Unify EMPTY_FULL_CTX_INVOKING_STATE and EMPTY_INVOKING_STATE (fixes an array indexing error during closure)
2012-10-14 22:25:11 -05:00
025cc6187a
Fix handling of non-greedy blocks in the lexer (uses regex-style non-greedy with unordered alternatives)
2012-10-14 21:46:57 -05:00
28b243cda5
BlockEndState links to start state for each block in the ATN (constructed during deserialization)
2012-10-14 21:45:35 -05:00
0d30a7a60b
Add ATN representation for non-greedy decisions
2012-10-14 21:45:34 -05:00
c68c058280
Implement Array2DHashSet.iterator
2012-10-14 21:45:29 -05:00
a0ffc84952
Merge branch 'master' into new-conflicting-alts
2012-10-14 18:47:32 -07:00
327a3a0100
Revert "update comment on consume()"
...
This reverts commit 27a8e5e521
2012-10-14 18:46:54 -07:00
54871d52ff
Revert "let it consume EOF since we sometimes need to match that during prediction"
...
This reverts commit 643edf0263
2012-10-14 18:46:43 -07:00
75a01636d0
got ambiguity reporting set right I think. fixed unit tests to force exact ambig detection.
2012-10-14 18:42:43 -07:00
e9c83c375f
cleanup and refactor conflict detection code into PredictionMode.
2012-10-14 17:37:15 -07:00
01bbce6952
full LL is always correct now, if slower. Stops prediction when resolvesToJustOneViableAlt(). SLL always uses heuristic
2012-10-14 13:25:34 -07:00
e7ece0e90a
got fast prediction termination with exactAmbig option that forces ANTLR to scan ahead until it identifies the true ambiguity.
2012-10-13 18:12:33 -07:00
d6d3f6f506
missing ambig alt in msg.
2012-10-12 18:07:37 -07:00
e69734c501
rm loopsSimulateTailRecursion
2012-10-12 17:49:43 -07:00
2ecfe2671a
size!=cardinality with BitSet.
2012-10-11 20:20:02 -07:00
f20cd82920
I need EOF edges on rule stop states for start rules.
2012-10-11 18:55:13 -07:00
1723bbd6f5
no debug
2012-10-11 18:54:26 -07:00
6654281aab
Merge branch 'master' into new-conflicting-alts
2012-10-11 18:52:22 -07:00
27a8e5e521
update comment on consume()
2012-10-11 18:51:58 -07:00
643edf0263
let it consume EOF since we sometimes need to match that during prediction
2012-10-11 18:50:02 -07:00
badb48a987
almost there.
2012-10-11 18:47:47 -07:00
058ed51349
initial impl
2012-10-11 13:09:08 -07:00
ad438a70ac
updated the comments for SLL.
2012-10-11 12:09:38 -07:00
ebe633a2cc
replace ctor for single ctx and don't merge wildcards when payloads aren't the same. fixes a unit test.
2012-10-08 15:26:56 -07:00
7bc16f40ea
Merge branch 'prediction-context-fixes' of git://github.com/sharwell/antlr4
2012-10-08 11:37:29 -07:00
d274650765
add null ptr so start rule it doesn't crash if you label the alternatives.
2012-10-08 11:32:38 -07:00
9cea095d81
Fix PredictionContext.mergeRoot placing states in the wrong order
2012-10-08 10:26:48 -05:00
f43e3614a8
Fix invoking states for recursion contexts
2012-10-08 07:38:06 -05:00
1defbdcc5d
Fix parent pointers in parse trees for nested recursion contexts
2012-10-07 23:01:26 -05:00
05f1dc0d26
tweak
2012-10-01 16:09:18 -07:00
69c20f3cf6
Only perform array copy when necessary
2012-10-01 15:30:28 -05:00
3457cc73ea
Significant updates to UnbufferedCharStream to meet the IntStream and CharStream interface requirements
2012-10-01 15:29:51 -05:00
cd25890486
Many updates to UnbufferedTokenStream:
...
* Fix LT(-1) after a seek operation
* Prevent consume() after EOF is reached
* Use the EOF token provided by the TokenSource (no need to create a new one)
* Fix assigned token index when marks are in place and when add() is called by derived types
* Throw exception for some invalid use of mark/release/seek
* Prevent fill() from adding multiple EOF tokens
* Remove diagnostic println
* Updated field documentation
* Implement getTokenSource
2012-10-01 15:24:00 -05:00
5637a04eec
Extensively document the IntStream, CharStream, and TokenStream interfaces
2012-10-01 14:52:30 -05:00
fa3e6e5867
Move CharStream.EOF to IntStream.EOF
2012-10-01 14:09:11 -05:00
fa7015f798
Move test helper methods out of exposed UnbufferedCharStream API
2012-10-01 11:53:04 -05:00
1c65bcd02f
Move test helper methods out of exposed UnbufferedTokenStream API
2012-10-01 08:25:46 -05:00
68dd847c04
* Fixed the unbuffered streams, which actually buffered everything
...
up by mistake. tweaked a few comments.
* Added a getter to IntStream for the token factory
2012-09-30 16:45:30 -07:00
9845e4ff20
rm redundant field.
2012-09-30 12:46:01 -07:00
ac29e6cdac
got unbufferedchar working I think.
2012-09-30 12:37:35 -07:00
db1bfa9c03
ParserATNSimulator wasn't using Token type param.
2012-09-29 16:58:51 -07:00
b56cd0f587
simplified delete configs algorithm.
2012-09-29 12:48:20 -07:00
e78ecd418a
rm isGreedy from DecisionState, but allow ATN construction for lexer to be nongreedy. error if '.' in parser. rm unit tests for parser nongreedy
2012-09-29 12:33:00 -07:00
c8a51ccfad
got wildcard working properly with recursion now!!! new unit test.
2012-09-27 12:22:16 -07:00
19782e6d77
first wack at fixing nongreedy (ACTION | .)* for recursive rules.
2012-09-27 11:44:51 -07:00
bca63c64aa
the test rig did not allow lexer only grammars; a regression.
2012-09-23 18:05:31 -07:00
262a331a5b
recursive rule bug in lexer; the lexer ATN simulator was not checking for empty stack at rule stop states.
2012-09-23 18:04:46 -07:00
7ed6cdcc4b
TerminalNodeImpl.getSourceInterval didn't fulfill the contract of SyntaxTree.getSourceInterval
2012-09-22 20:10:01 -05:00
62e005c841
Add nb-configuration.xml files
2012-09-22 19:49:11 -05:00
4abedb08dd
tweak comments
2012-09-22 12:29:22 -07:00
ce38d61f3a
Merge branch 'flexiblehashmap' of github.com:sharwell/antlr4
2012-09-16 15:11:29 -07:00
1ce6b69651
Fix build warnings in FlexibleHashMap, reduce entry size
2012-09-16 13:44:34 -05:00
e9656e35f9
Fix codegen problem where notset and wildcard transitions could match invalid tokens and/or EOF
2012-09-16 12:56:10 -05:00
aa2d893a77
got first draft of new "go/stop" algorithm.
2012-09-09 19:20:33 -07:00
89656989e6
add new map that lets me specify hashCode/equals
2012-09-09 18:55:47 -07:00
3c5dec1172
add large comment describing the decision rule for continuing analysis.
2012-09-09 17:56:42 -07:00
4bbbff4e8e
comments and a rename
2012-09-08 15:23:33 -07:00
0a961645be
fix comments; fix unit test
2012-09-08 13:13:00 -07:00
726fe2c983
Make sure to call visitErrorNode instead of visitTerminal when you have an ErrorNode. Also fix listener calls for case where _buildParseTrees is false (adds terminal as children of the rule nodes but won't actually construct the parse tree).
2012-09-08 14:57:30 -05:00
7586afff27
don't pass new term node
2012-09-08 10:21:42 -07:00
64cd79a18a
add return values
2012-09-08 10:14:10 -07:00
2b5c3723ba
update comments from Sam's version
2012-09-06 14:17:13 -07:00
38e0ad238d
Merge branch 'bail-error-propagation' of github.com:sharwell/antlr4 into main
2012-09-04 19:00:54 -07:00
8dd215f4b4
Merge branch 'deserialize' of github.com:sharwell/antlr4 into main
2012-09-04 18:59:38 -07:00
201db8b6d0
merge sam's pulls
2012-09-04 18:59:20 -07:00
3565efbbc3
Merge branch 'parserrulecontext-tostring' of github.com:sharwell/antlr4 into main
2012-09-04 18:37:03 -07:00
20aa8cf4ec
update comments
2012-08-27 12:05:05 -07:00
d9ad3d0bde
rm as unneeded.
2012-08-27 12:02:16 -07:00
60d99e62dc
rm ParseListener; tested the tracer with left recursive rules; weird but deterministic for entry events.
2012-08-27 11:22:42 -07:00
050b27cf65
tweak comment
2012-08-26 18:09:52 -07:00
1a52e0212e
tweak comment
2012-08-26 17:21:35 -07:00
e33e355d66
tweak comment
2012-08-26 16:32:28 -07:00
ad737ebdf6
tweak
2012-08-26 16:03:59 -07:00
d8b4d6403a
add doxy stuff
2012-08-25 11:40:55 -07:00
a51f1e7055
Updated documentation in BailErrorStrategy
2012-08-21 13:24:39 -05:00
4508f96ce4
BailErrorStrategy propagates the RecognitionException to the current rule context and it's parents
2012-08-21 13:23:03 -05:00
dd12508f5d
Add overloads of RuleContext.toStringTree and RuleContext.toString which take a list of rule names instead of a Recognizer instance. Remove ParserRuleContext override because it does the same thing as the base method.
2012-08-19 11:43:21 -05:00
6f38942e31
Add overload of Trees.toStringTree that takes a list of rule names instead of a Recognizer instance
2012-08-19 11:38:13 -05:00
17f7efca0a
Never serialize edges for rule stop states since they can be trivially derived during ATN deserialization
2012-08-17 10:51:44 -05:00
1d92b311ce
Merge branch 'master' into main
2012-08-13 12:38:00 -07:00
3bfcb7b38f
Add ATNSimulator.verifyATN to verify several assumptions about the deserialized ATN
2012-08-12 11:59:42 -05:00
2975c90fac
Serialize/deserialize the value of BlockStartState.endState
2012-08-12 11:58:26 -05:00
0c073062f7
TokensStartState is not really a BlockStartState because it doesn't have an associated end state
2012-08-12 11:56:07 -05:00
9b42acbf4d
ATNSimulator.deserialize computes values for PlusBlockStartState.loopBackState and StarLoopEntryState.loopBackState
2012-08-12 11:56:06 -05:00
83d8903f9f
ATNSimulator.deserialize derives rule stop states and (in the lexer) the outgoing transitions for those states
2012-08-12 10:27:48 -05:00
2d62b73a14
Merge branch 'master' into main
2012-08-06 17:11:38 -07:00
199e9892dc
Should be checking for ErrorNode (interface) instead of ErrorNodeImpl (class)
2012-08-06 15:01:43 -05:00
cb09dd6d09
Move RuleNode, TerminalNode, ErrorNode, TerminalNodeImpl, and ErrorNodeImpl to top-level types
2012-08-06 15:01:00 -05:00
492980de71
Merge branch 'master' into main
2012-08-04 21:31:37 -07:00
2947fe6a2a
Fix ATN optimizer leaving loop back state numbers in an inconsistent state
2012-08-04 22:22:51 -05:00
0d92c25056
improve hashCode
2012-08-04 14:38:57 -07:00
1bec176eaa
Impl Sam's no viable alt avoidance idea that chooses min alt that dips into outer context. unit test
2012-08-04 14:18:57 -07:00
6d12cbfddb
fix regression; didn't work with stdin
2012-08-04 13:50:52 -07:00
c2b49bd94e
pull in Sam's ATN alt collapsing optimizations with optimizeStates off as it causes a class cast exception.
2012-08-04 13:32:07 -07:00
c7d1ea7e23
Merge remote-tracking branch 'sharwell/atn-optimization'
2012-08-04 12:08:43 -07:00
b7b2a45c8b
update comments
2012-08-04 12:00:13 -07:00
b160a3b14d
caching more operations in mergeCache, updated comments, added null chk in front of mergeCache ops.
2012-08-04 11:04:21 -07:00
88dedbd1e0
added mergeArray cache, which didnt help java much but made massive diff to Go grammar. a file, nat.go, went from 2min to 3s in speed. Fixed bug where SLL had preds but they ORd to NONE. made it resolve to min alt. DoubleKeyMap moved to runtime.
2012-08-03 20:27:23 -07:00
8a0af228d8
Revert "fix null ptr"
...
This reverts commit 95f5d858cb
2012-08-03 18:27:57 -07:00
b8c2f5b75b
added var for sll loop tail recursion default value; updated unit tests
2012-08-03 18:27:26 -07:00
e7b65057a6
added var for sll loop tail recursion default value; updated unit tests
2012-08-03 18:12:52 -07:00
95f5d858cb
fix null ptr
2012-08-03 16:56:41 -07:00
5019278204
Add ATN optimization to collapse multiple alternatives to a single SetTransition where possible. Currently disabled for parser grammars since the code generator doesn't support set transitions.
2012-08-03 08:33:11 -05:00
54a3759412
LexerATNSimulator and ParserATNSimulator switch on result of getSerializationType() instead of performing multiple type checks
2012-08-03 08:24:24 -05:00
86e47b9c02
update comments to explain SLL vs LL, predicate strategy, etc... (and did a small tweak in code)
2012-08-02 18:23:11 -07:00
06d7c150fd
slow again since i had to remove full LL context->alt cache (wasn't correct). But, threaded now makes bigger diff than before.
2012-08-02 17:01:53 -07:00
ff4eb0f744
Transition classes except for SetTransition are final
2012-08-02 12:15:36 -05:00
60df00be4f
Transition classes override getSerializationType()
2012-08-02 12:15:35 -05:00
754e6eb593
mirrored parser ATN sim changes in lexer ATN sim for thread safety. reduced contention for contextCache.
2012-08-01 18:25:25 -07:00
b0b30fec68
cleanup
2012-08-01 18:05:47 -07:00
ff24630494
add cmt
2012-08-01 15:03:32 -07:00
b0c9a9c7c2
tried optimization concerning recomputing DFA states in DFA, but didn't help much. left in as comment for now.
2012-08-01 14:51:37 -07:00
80560efcf8
wow. i think i got threaded working. rm readwrite lock, sync add add to shared context and add dfa state
2012-07-31 19:39:31 -07:00
3d2cbe087d
refactoring
2012-07-31 18:54:00 -07:00
abb268416e
Merge branch 'master' into main
2012-07-31 18:34:33 -07:00
b8f7acdeac
refactor to create DFAState target, fill it in, then add to DFA only after it's complete. Combine some helper methods. simpler.
2012-07-31 18:26:39 -07:00
fedb7c105b
tweaks
2012-07-30 22:24:12 -07:00
85ea296632
added D2 as dup temp of D state. almost ready to change position of addDFAState
2012-07-30 22:15:09 -07:00
8c820269e4
Fix IntegerList.binarySearch
2012-07-30 19:24:20 -05:00
c893f2af08
Pulled Sam's new IntegerList
2012-07-30 15:20:43 -07:00
dcdcad8feb
Add IntegerList.addAll overload for Collection<Integer>, add contains, sort, hashCode, equals, toString.
2012-07-30 15:17:02 -05:00
3bf99d6d88
Add IntervalSet.toIntegerList(), use IntegerStack to hold mode stack in Lexer
2012-07-30 15:13:26 -05:00
fa62570737
Add IntegerList and IntegerStack, behaves like ArrayList<Integer> and ArrayDeque<Integer> (mostly) except doesn't require boxing
2012-07-30 15:13:26 -05:00
c0ece0bd09
add read/write lock to lexer.
2012-07-29 17:21:10 -07:00
8836e13954
use ReentrantReadWriteLock to sync. works much better but still occasional illegal monitor state :(
2012-07-29 17:01:41 -07:00
f5f6715a07
back to core sync around creating DFA in decisionToDFA. not thread-safe for dfa update.
2012-07-29 14:33:18 -07:00
9bcfc83678
adding full LL prediction in DFAState from ctx -> predicted alt. Drops from 17s to 12s and costs only 4M more ram. woot!
2012-07-29 13:30:18 -07:00
79594a942e
synchronized lexer. just as fast now to build new parser/lexer each iter in TestJavaLR
2012-07-29 10:48:08 -07:00
ceab49e3ce
rm for now
2012-07-29 10:12:09 -07:00
e3e739dfc7
The lexer and parser ATN simulators' adaptivePredict now synchronize on the specific DFA of the decision to be simulated. This should prevent a lot of contention that would occur if we synchronize the entire adaptivePredict method. When the individual DFA are created, we also synchronize on the shared DFA[] table quickly to create a DFA and insert it into the array. Code generation modified to have _decisionToDFA generated at the top of both the parser and the lexer. Simulators created now with the recognizer, ATN, DFA[]. Not sure the LexerInterp/ParserInterp work but pushing ahead anyway for the moment.
2012-07-29 09:49:35 -07:00
8c8752a7b2
print file names if > 1 file
2012-07-28 18:33:21 -07:00
3f26d42d53
add Sam's getConflictingAlts potential bug fix as comment
2012-07-28 18:33:05 -07:00
68275eb998
rm singleton set; didn't help.
2012-07-28 13:23:52 -07:00
ee233f7dd3
add -SLL option, allow multiple input files, reuse same parser/lexer.
2012-07-28 13:07:51 -07:00
75fd3264ff
adding new equivalence class support code and new SingletonSet
2012-07-28 11:53:40 -07:00
680df17280
rm transient context cache for closure computations; slight speed improvement maybe. less complex.
2012-07-28 10:51:07 -07:00
eda95f7478
DFA cache on again; dropped from 25 to 16M!!!
2012-07-28 10:34:34 -07:00
8a301f59f8
try making smaller config list.
2012-07-28 10:29:04 -07:00
2ee5a2f3ae
release entire ConfigHashSet upon readonly. sames some mem and is faster.
2012-07-27 22:38:43 -07:00
7ff5544573
wipe out config lookup in ATNConfigSet when it goes readonly.
2012-07-27 22:20:23 -07:00
54fd28c29d
add() was wrong
2012-07-27 21:03:41 -07:00
8279c3da11
turn on local context cache.
2012-07-27 20:53:10 -07:00
14372f2515
only 14 unit tests fail.
2012-07-27 17:38:02 -07:00
b8f072e231
rebuilt new hashtable as T[][]
2012-07-27 17:18:33 -07:00
b843d6cf59
initial new hash table impl.
2012-07-27 14:51:07 -07:00
b83c90af2f
play with atnconfigset fields
2012-07-27 14:50:36 -07:00
9445d1fdd6
update comment
2012-07-27 10:38:43 -07:00
26165c5d54
atnconfigsets can be readonly now. no copying to put into new DFA state, set readonly instead. made atnconfigset.equals() more accurate. add -notree option to test java lr
2012-07-27 10:34:59 -07:00
093af0edce
update comment
2012-07-26 18:26:17 -07:00
6322ff3abd
add comment
2012-07-26 18:17:05 -07:00
0e24a66f07
reduce memory usage by creating LexerATNConfig a subclass with the lexer action index. saves about 2M on java.* with javalr.
2012-07-26 17:51:02 -07:00
3c7b4c2a33
big cleanup.
2012-07-26 17:28:10 -07:00
6b8e6cf0a7
reduces a few equals calls.
2012-07-26 10:58:11 -07:00
faafe8dfa2
no local/shared predctx caches. remove spurious array creation.
2012-07-26 10:35:52 -07:00
86a4f26596
still has dfa and local predict ctx cache, but doesn't help much, and takes time. Eval preds as we pass them in full ctx mode; no sempred ctx in configs then. removed LANDMINE!!!!!!!!! some lame code that uses fields like _startIndex; fix that. added bail option to testjavalr. 18s to parse java.* with javalr full ctx now.
2012-07-25 18:47:34 -07:00
05de714987
add chk for null contextCache.
2012-07-25 10:58:04 -07:00
3774c5752b
add more tracking var
2012-07-25 10:30:19 -07:00
3cec976f7e
Don't pursue the closure if there is just one state or it's predicting unique alt. Fixed up computeReachSet() to do this. Dropped from 6s to 4.5s for TestJavaLR -SLL on java.* with 45M max footprint.
2012-07-24 18:45:23 -07:00
2495ce6b20
add SLL option to the ATN simulator and use in TestJavaLR.
2012-07-24 17:42:47 -07:00
2ae4d4eebd
add ctx cache to merge op. now i get optimal graph it seems. doesn't seem longer for java.* with JavaLR. still landmine in Pattern.java, Foo.java. Made toDOTString sort predctx nodes by id for repeatable tests. updated unit tests.
2012-07-24 16:26:01 -07:00
ac4f00524e
add ctx cache.
2012-07-24 14:19:43 -07:00
9627652b67
tweak to compile
2012-07-24 12:20:56 -07:00
2673e08bfc
Use arrays T[] instead of Set<T> to save memory in SemanticContext instances
2012-07-24 12:30:24 -05:00
760c403151
Use arrays T[] instead of List<T> to save memory in the cached DFA
2012-07-24 12:30:03 -05:00
320868d661
added code chking size of ctxs. changed configsets -> configs. added recur getallnodes.
2012-07-24 10:04:51 -07:00
344a56fd1e
finished inter-dfa-state-context-cache but doesn't fix landmine. add debug_list_atn_decisions field.
2012-07-23 18:21:53 -07:00
1966379265
shelve to look back at sam's. added optimizeConfigs() stuff.
2012-07-23 17:50:22 -07:00
935fd2831e
add writeFile util. improve dbg msg
2012-07-23 10:29:12 -07:00
d8a9207041
improve combine common parents, return a or b in merge array of merged is a or b; new unit test.
2012-07-22 20:07:46 -07:00
e18b9132d9
cleanup and merge common parents for array merge; fixes unit test
2012-07-22 16:39:53 -07:00
f1e4d85d0a
fix cut/paste error.
2012-07-22 16:17:07 -07:00
f7eeca274f
reorg closure and fix bug where $ in arrayctx wouldn't perform global follow. fix case in array merge that didn't check both a[i], b[i] as $ (only matters in full ctx). unit tests for graph show fewer ctx's created.
2012-07-22 16:09:45 -07:00
44ae1dad0b
if merged array is 1 node, return singleton. more unit tests
2012-07-22 15:05:12 -07:00
3edb35d95e
cleanup; avoid array creation
2012-07-22 12:59:26 -07:00
829ad9191c
didn't join a$ + bx as [a,b]$ for local ctx. new unit test to chk
2012-07-22 12:57:30 -07:00
8695210903
cleanup, simplify array ctx equals
2012-07-22 12:17:34 -07:00
43424cd5df
do check for a==b in merge(); improve DOT output
2012-07-22 12:10:40 -07:00
43e986db8b
small hash code cleanup.
2012-07-22 11:27:44 -07:00
54d23066b2
merge of array and $ for rootIsWildcard was broken. Fixed JavaLR parsing. Still sticks on Pattern.java but not 5 hours anymore--faster up til that landmine.
2012-07-21 19:24:03 -07:00
36a725765e
cleanup; hashcode for single node pred ctx was wrong. failed to see [46] and [46] as same.
2012-07-21 17:13:16 -07:00
991014d3f2
clean up, add comments, fix graph unit tests, updated toDotString().
2012-07-21 15:10:15 -07:00
f78bf4d097
rm comment; looks inappropriate for branch
2012-07-21 10:45:59 -07:00
0141bc058a
merge from master,In preparation for continuing on this experiment branch.
2012-07-21 10:38:55 -07:00
885f6530ad
Use chained calls to append instead of string concatenation
2012-07-20 14:52:49 -05:00
44ef41ff29
Remove unnecessary variable initializations
2012-07-20 14:52:46 -05:00
3b9940b02a
Use isEmpty() instead of comparing size() with 0
2012-07-20 14:52:43 -05:00
2f0029a040
Improved handling of (potentially) null fields
2012-07-20 14:52:14 -05:00
4c4f767d17
Prevent NPE after reporting error
2012-07-20 14:52:13 -05:00
45e42d7243
Parameter to equals can be null
2012-07-20 14:51:54 -05:00
3a35f3cb08
Check object type in equals
2012-07-20 14:51:53 -05:00
29d71acef9
Update documentation
2012-07-18 16:12:37 -05:00
a37f8cf4f1
Add missing license notices
2012-07-18 15:57:48 -05:00
f220212a95
couldn't get Horstmann's routine to do EPS not PS so had to backtrack.
2012-07-14 16:32:04 -07:00
47362b2951
use Horstmann's PS save not mine; it'll handle Japanese char etc.. this way.
2012-07-14 15:38:33 -07:00
d626c4acd6
Simplify ParserRuleContext.toString
2012-07-13 14:26:56 -05:00
a3371ac9ea
rm unneeded override.
2012-07-09 17:32:20 -07:00
9a4227841d
-print -> -tree
2012-07-08 13:38:56 -07:00
707ff615b4
tweak javadoc
2012-07-08 12:18:43 -07:00
3b2c0a6177
fix bug in get hidden channel stuff. rewrote. update test.
2012-07-07 18:48:48 -07:00
ec47251bb2
unbuffered release() can get markers in weird order since we reset p to beginning of buffer. Might mark at 1 and then at release p = 0 etc... Don't look for errors. Just reset earliestMarker if needed.
2012-07-04 11:32:41 -07:00
6b6274b0c8
add type narrowed getters
2012-07-02 15:48:45 -07:00
71bd16020f
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:45:48 -07:00
111c44b3fd
add default ctor to Lexer
2012-07-02 12:45:30 -07:00
4329f00186
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:19:51 -07:00
ae7313f36a
factor out input.read() and make ctor for easy subclass.
2012-07-02 12:19:20 -07:00
3ad87ba12c
rm generic parameter from ANTLRErrorListener, ripple effect.
2012-07-01 22:34:35 -07:00
bb5790d6a9
Add convenience method.
2012-07-01 17:36:22 -07:00
37396d6040
clean up
2012-07-01 12:20:54 -07:00
db66257d7e
cleanup so Lexer lets us subclass to support multiple token emit()s per lex rule match.
2012-07-01 12:19:44 -07:00
0b4fa240c5
alter comment
2012-07-01 10:55:31 -07:00
f80166b39c
Rewrote the unbuffered token stream to use a type-adjusted version of the unbuffered character stream. Simpler and I get to remove fast queue and lookahead string classes. These unbuffered streams always prime the pump with the 1st single.
...
Added a unit test for the unbuffered token stream.
Made sure that the unbuffered streams move forward always on a consume
Removed the reset method from the unbuffered streams because it's meaningless to reset to the beginning of the buffer.
2012-07-01 09:39:11 -07:00
5c69d31e88
CommonTokenFactory now knows how to copy the text out of the character stream buffer before they disappear in unbuffered character strengths; added ctor.
...
Lexer now guarantees that the text of the current token is always available to the emit() method even if the character stream is unbuffered.
Added some hooks to see some of the internal data in the unbuffered character stream so that I can test it better.
Updated LexerInterpreter so that it uses the token factory.
Improved/added unit tests for the unbuffered character string.
Updated various comments
2012-06-30 16:40:16 -07:00
44355d6ff6
oops; don't need these afterall
2012-06-30 16:36:25 -07:00
3900606024
no need for range field. removed. overrirde tostring so nothing weird comes out.
2012-06-30 16:36:01 -07:00
0c22d12870
updated comments, remove duplicate code, add new functionality.
2012-06-29 13:02:44 -07:00
e861902a10
set hidden channel num to 1 not 99. default is chan 0
2012-06-29 12:03:54 -07:00
f396b4dc82
error->syntaxError in listener
2012-06-25 13:25:13 -07:00
b18475113b
Merge branch 'master' into interactive
2012-06-16 17:20:04 -07:00
13d9f6ed53
add comment
2012-06-08 17:04:46 -07:00
3ece2c8640
merge from master; this pulls in the cleanup and bug fix in the lexer related to semantic predicates.
2012-06-08 10:18:49 -07:00
abc0e2ef87
add getSpeculativeText(); text matched so far in a lexer rule. Use this in predicates not actions. add unit tests.
2012-06-07 18:31:36 -07:00
018e3c03e8
look for T not just TLexer if no parser just tokens.
2012-06-07 18:27:42 -07:00
b255509e96
fix a bug related to semantic predicates in the lexer and generally cleaned up variable and method names in the simulator. I moved all of the predicates to the right side of lexer rules in the unit tests. Later, we should ensure that predicates only occur on the right edge of lexer rules. We should state that the rule is not been accepted so we can't test things like getText(), we have to use more raw indexes into the character stream. In the lexer simulator, the addDFAState() method now does not try to compute whether there is a predicate in the configurations. That information has already been set into the ATNConfigSet by the getEpsilonTarget() method. [I should also point out that I have not tested the Java parsing in a while and now it hits a landmine on a number of common Java files in jdk :(. I'm not sure where that crept in]
2012-06-07 17:31:18 -07:00
5a7b3b0cab
fixed a bug where some decisions that could see predicates thought they were LL(1). Added HIT_PRED invalid token type to the lookahead set to indicate we had a predicate; later we wipe out that entire set before returning from "get token lookahead sets".
2012-06-01 18:09:25 -07:00
59af8ab76b
clean up constants
2012-06-01 17:38:54 -07:00
aab8806783
added setter for _token
2012-05-26 11:32:32 -07:00
e737653b4f
lexer getText() included lookahead char, which it shouldn't.
2012-05-23 14:09:32 -07:00
bd87ec95e2
add setters for line/charpos to lexer/lexersimulator
2012-05-22 16:21:41 -07:00
ed879ab4e3
working towards interactive lexer
2012-05-21 10:16:37 -07:00
6215390408
add ctor for consistency
2012-05-20 17:28:14 -07:00
5fc148381a
rm toString from token stream. it affects buffer.
2012-05-20 16:49:44 -07:00
990da7d1d8
add unit test
2012-05-19 18:08:56 -07:00
03d7b51d3c
throw exception upon bad token indexes, add getOffChannelTokensToLeft, right. pull up some off channel token methods from CommonTokenStream. renamed.
2012-05-19 14:22:52 -07:00
7be2ba9d42
remove unused type param
2012-05-19 13:06:27 -07:00
d46c7db52c
merge master
2012-05-15 12:19:48 -07:00
c590ba8fd8
don't look backwards for err msg if EOF is entire input. make sure we don't use -1 rule index for ruleNames[]
2012-04-29 12:12:42 -07:00
3ea6a5cb2f
Merge branch 'master' into stack-graphs-integration
...
Conflicts:
tool/playground/T.g
2012-04-26 12:01:28 -07:00
30f61b17e3
comment tweak
2012-04-15 18:30:28 -07:00
f5501979c6
Merge branch 'master' into stack-graphs-integration
2012-04-09 10:56:14 -07:00
35aeeb7c5c
turn on building parse trees by default since we do -listener by default.
2012-04-07 11:27:18 -07:00
d22eb34fe7
Merge branch 'master' into stack-graphs-integration
2012-04-05 16:35:56 -07:00
1ade15392b
Add license header
2012-04-04 15:33:39 -05:00
e11dc07275
Remove TokenRewriteStream
2012-04-04 15:33:03 -05:00
08e3ddefc0
Merge branch 'master' into stack-graphs-integration
2012-04-04 13:22:00 -07:00
585aa0a14b
Merge branch 'tokenstreamrewriter' of git://github.com/sharwell/antlr4
2012-04-04 13:20:58 -07:00
ff2655d379
Rename getSource() to getTokenStream()
2012-04-04 15:20:45 -05:00
bbd61dca35
Remove TokenStreamRewriter.getOriginalText() (use rewriter.getSource().getText() instead)
2012-04-04 10:15:29 -05:00
a85f1c17a1
Add TokenStreamRewriter.getSource()
2012-04-04 10:14:09 -05:00
e4612935f2
Only need TokenStream (was overly restricted to BufferedTokenStream)
2012-04-04 10:10:33 -05:00
16046a7ec1
Proper maven reference for org.abego.treelayout.core dependency
2012-04-04 09:52:06 -05:00
bf9c7c3a28
Remove old TokenRewriteStream in favor of new TokenStreamRewriter that is not a stream; it feeds off of the stream to provide transformations of that original stream. Updated the comment at the top of the file. Instead of toString() we use getText(). Altered the unit tests to create a buffered stream and then create the rewriter attached to that.
2012-04-02 18:10:32 -07:00
3849cd572d
rm SymbolStream<T>, change all to TokenStream. all but 5 tests pass.
2012-04-02 16:49:04 -07:00
9ef6127982
merging from master
2012-04-01 18:13:40 -07:00
53c5aecb85
Implement UnbufferedTokenStream.get(int) within marked intervals
2012-04-01 18:23:32 -05:00
8ec9ba14c9
Implement getText within marked intervals of unbuffered streams
2012-04-01 18:22:43 -05:00
d6e2e41c1f
Fix stop index
2012-04-01 14:29:06 -05:00
e1870d16e9
added convenience getText(ctx) method to TokenStream. unlikley to need in CharStream and can get another way
2012-04-01 12:23:45 -07:00
588e30bfe1
getText(...) moved to CharStream and TokenStream; cleaned up names. removed getInputString from parser.
2012-04-01 12:12:49 -07:00
f238d7579e
added getText() to ParseTree. getText(tokens) is still really what you want but convenient for ctx.type().getText() when it's just one token.
2012-03-31 17:27:11 -07:00
cc20a52cdd
allow special "tokens" start rule name so we can test pure lexer grammars.
2012-03-31 15:54:00 -07:00
c9aef6fdbe
Merge branch 'master' into stack-graphs-integration
2012-03-31 10:33:22 -07:00
e5d7c27b09
Updated comments to indicate preference of .g4 over .g
2012-03-30 11:37:22 -05:00
447a5620f0
add -diagnostics option to test rig.
2012-03-28 11:07:23 -07:00
a363e70582
pull from master
2012-03-27 16:22:01 -07:00
bf07f7907f
Merge branch 'override-ruleindex' of github.com:sharwell/antlr4
2012-03-27 15:09:50 -07:00
41e97791bb
Merge branch 'set-stop-token' of github.com:sharwell/antlr4
2012-03-27 15:08:57 -07:00
f307945d2a
Merge branch 'lexer-npe' of github.com:sharwell/antlr4
2012-03-27 15:06:17 -07:00
8465920343
Merge branch 'atnstate-reachability-test' of github.com:sharwell/antlr4
2012-03-27 15:05:39 -07:00
e8765ef241
Fix errors tracking current state on rule return
2012-03-26 23:29:53 -05:00
169f58a3ff
fix def of EMPTY->INVALID
2012-03-25 21:19:39 -07:00
aa34f33846
Move ParserRuleContext._s to a field in Recognizer
2012-03-25 21:51:48 -05:00
df2e24a228
Override getRuleIndex() in generated context objects so ParserRuleContext.ruleIndex field can be removed
2012-03-25 21:50:55 -05:00
bdda174af6
Set stop token even when RecognitionException occurs. Add ParserRuleContext.exception field to hold exception if one occurs. Resolves antlr/antlr4#49 .
2012-03-25 21:50:12 -05:00
2b5896e83b
Merge branch 'master' into stack-graphs-integration
2012-03-25 10:08:29 -07:00
542e700644
fix antlr/antlr4/#48 (getText from parse tree node). Don't NPE when start/stop null in getSourceInterval.
2012-03-25 10:01:27 -07:00
4ea3c73d1f
cleaned up interval stuff. moved getSourceInterval to ParserRuleContext; renamed create to of() in Interval.
2012-03-24 18:17:35 -07:00
c6365fb5e2
removed config list (huge) from default message.
2012-03-24 16:56:36 -07:00
e228e88ac5
tweak to remove insuff pred method
2012-03-24 16:48:26 -07:00
6791bf60cf
Merge commit '3f1f76d' into parrt-default-error-listener
...
Conflicts:
runtime/Java/src/org/antlr/v4/runtime/ANTLRErrorStrategy.java
runtime/Java/src/org/antlr/v4/runtime/DefaultErrorStrategy.java
runtime/Java/src/org/antlr/v4/runtime/DiagnosticErrorListener.java
runtime/Java/src/org/antlr/v4/runtime/atn/ParserATNSimulator.java
2012-03-24 16:44:47 -07:00
b1bcde76b7
Merge commit '768bfc0' into parrt-default-error-listener
2012-03-24 16:40:09 -07:00
b035ceec9e
EMPTY gives "" not "$" now. rename closure->closure_. had return in wrong spot in closure for-loop. fix fromRuleContext.
2012-03-20 19:06:52 -07:00
0f969af947
added in Sam's popAll to deal with loop end push/pop stuff. passes LR tests and all but 2 sempred tests. does some JavaLR.g4 parsing too.
2012-03-20 18:06:00 -07:00
9318391664
tweak to fromRuleContext. more sempred tests work.
2012-03-20 12:32:28 -07:00
55ed1a45bf
lexer wasn't using new prediction context. weird. put in functionality for $ merge in full ctx mode. removed configs array in ATNConfigSet. ATNConfigSet wasn't tracking lexerActionIndex from config. ATNConfigSet now maps key to ATNConfig.
2012-03-20 12:12:18 -07:00
5945b29a86
got more working
2012-03-19 19:58:04 -07:00
7e9a86a3e1
got TestATNInterpreter working
2012-03-19 18:47:41 -07:00
48d663667b
pull from master
2012-03-19 18:03:32 -07:00
2232ea5101
was not computing lookahead correctly in _LOOK. It assumed all epsilons were predicates.
2012-03-19 17:50:51 -07:00
85b40c7d2e
shelve
2012-03-19 17:47:57 -07:00
bd7796544d
Update comments
2012-03-19 08:33:55 -05:00
5e0f9a4490
Ensure target non-null in Transition constructor (additional runtime-side consistency check).
2012-03-19 08:33:36 -05:00
a70cb6f36a
Fix NPE in LexerATNSimulator.execDFA error recovery
2012-03-19 08:17:59 -05:00
974ae65cb9
shelve
2012-03-18 20:21:39 -07:00
90516272fe
snapshot
2012-03-18 15:39:08 -07:00
7233177441
got integrated graph stacks to compile. commit so i can look at sam's
2012-03-18 15:20:52 -07:00
3a79aa15ba
move new classes into proper package locations
2012-03-18 13:45:13 -07:00
dd0944b9c4
Simplify and improve performance of BufferedTokenStream.fill
2012-03-17 17:30:56 -05:00
Sam Harwell
Terence Parr