86a4f26596
still has dfa and local predict ctx cache, but doesn't help much, and takes time. Eval preds as we pass them in full ctx mode; no sempred ctx in configs then. removed LANDMINE!!!!!!!!! some lame code that uses fields like _startIndex; fix that. added bail option to testjavalr. 18s to parse java.* with javalr full ctx now.
2012-07-25 18:47:34 -07:00
05de714987
add chk for null contextCache.
2012-07-25 10:58:04 -07:00
3774c5752b
add more tracking var
2012-07-25 10:30:19 -07:00
3cec976f7e
Don't pursue the closure if there is just one state or it's predicting unique alt. Fixed up computeReachSet() to do this. Dropped from 6s to 4.5s for TestJavaLR -SLL on java.* with 45M max footprint.
2012-07-24 18:45:23 -07:00
2495ce6b20
add SLL option to the ATN simulator and use in TestJavaLR.
2012-07-24 17:42:47 -07:00
2ae4d4eebd
add ctx cache to merge op. now i get optimal graph it seems. doesn't seem longer for java.* with JavaLR. still landmine in Pattern.java, Foo.java. Made toDOTString sort predctx nodes by id for repeatable tests. updated unit tests.
2012-07-24 16:26:01 -07:00
ac4f00524e
add ctx cache.
2012-07-24 14:19:43 -07:00
9627652b67
tweak to compile
2012-07-24 12:20:56 -07:00
320868d661
added code chking size of ctxs. changed configsets -> configs. added recur getallnodes.
2012-07-24 10:04:51 -07:00
344a56fd1e
finished inter-dfa-state-context-cache but doesn't fix landmine. add debug_list_atn_decisions field.
2012-07-23 18:21:53 -07:00
1966379265
shelve to look back at sam's. added optimizeConfigs() stuff.
2012-07-23 17:50:22 -07:00
935fd2831e
add writeFile util. improve dbg msg
2012-07-23 10:29:12 -07:00
d8a9207041
improve combine common parents, return a or b in merge array of merged is a or b; new unit test.
2012-07-22 20:07:46 -07:00
e18b9132d9
cleanup and merge common parents for array merge; fixes unit test
2012-07-22 16:39:53 -07:00
f1e4d85d0a
fix cut/paste error.
2012-07-22 16:17:07 -07:00
f7eeca274f
reorg closure and fix bug where $ in arrayctx wouldn't perform global follow. fix case in array merge that didn't check both a[i], b[i] as $ (only matters in full ctx). unit tests for graph show fewer ctx's created.
2012-07-22 16:09:45 -07:00
44ae1dad0b
if merged array is 1 node, return singleton. more unit tests
2012-07-22 15:05:12 -07:00
3edb35d95e
cleanup; avoid array creation
2012-07-22 12:59:26 -07:00
829ad9191c
didn't join a$ + bx as [a,b]$ for local ctx. new unit test to chk
2012-07-22 12:57:30 -07:00
8695210903
cleanup, simplify array ctx equals
2012-07-22 12:17:34 -07:00
43424cd5df
do check for a==b in merge(); improve DOT output
2012-07-22 12:10:40 -07:00
43e986db8b
small hash code cleanup.
2012-07-22 11:27:44 -07:00
54d23066b2
merge of array and $ for rootIsWildcard was broken. Fixed JavaLR parsing. Still sticks on Pattern.java but not 5 hours anymore--faster up til that landmine.
2012-07-21 19:24:03 -07:00
36a725765e
cleanup; hashcode for single node pred ctx was wrong. failed to see [46] and [46] as same.
2012-07-21 17:13:16 -07:00
991014d3f2
clean up, add comments, fix graph unit tests, updated toDotString().
2012-07-21 15:10:15 -07:00
f78bf4d097
rm comment; looks inappropriate for branch
2012-07-21 10:45:59 -07:00
0141bc058a
merge from master,In preparation for continuing on this experiment branch.
2012-07-21 10:38:55 -07:00
885f6530ad
Use chained calls to append instead of string concatenation
2012-07-20 14:52:49 -05:00
44ef41ff29
Remove unnecessary variable initializations
2012-07-20 14:52:46 -05:00
3b9940b02a
Use isEmpty() instead of comparing size() with 0
2012-07-20 14:52:43 -05:00
2f0029a040
Improved handling of (potentially) null fields
2012-07-20 14:52:14 -05:00
4c4f767d17
Prevent NPE after reporting error
2012-07-20 14:52:13 -05:00
45e42d7243
Parameter to equals can be null
2012-07-20 14:51:54 -05:00
3a35f3cb08
Check object type in equals
2012-07-20 14:51:53 -05:00
29d71acef9
Update documentation
2012-07-18 16:12:37 -05:00
a37f8cf4f1
Add missing license notices
2012-07-18 15:57:48 -05:00
f220212a95
couldn't get Horstmann's routine to do EPS not PS so had to backtrack.
2012-07-14 16:32:04 -07:00
47362b2951
use Horstmann's PS save not mine; it'll handle Japanese char etc.. this way.
2012-07-14 15:38:33 -07:00
d626c4acd6
Simplify ParserRuleContext.toString
2012-07-13 14:26:56 -05:00
a3371ac9ea
rm unneeded override.
2012-07-09 17:32:20 -07:00
9a4227841d
-print -> -tree
2012-07-08 13:38:56 -07:00
707ff615b4
tweak javadoc
2012-07-08 12:18:43 -07:00
3b2c0a6177
fix bug in get hidden channel stuff. rewrote. update test.
2012-07-07 18:48:48 -07:00
ec47251bb2
unbuffered release() can get markers in weird order since we reset p to beginning of buffer. Might mark at 1 and then at release p = 0 etc... Don't look for errors. Just reset earliestMarker if needed.
2012-07-04 11:32:41 -07:00
6b6274b0c8
add type narrowed getters
2012-07-02 15:48:45 -07:00
71bd16020f
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:45:48 -07:00
111c44b3fd
add default ctor to Lexer
2012-07-02 12:45:30 -07:00
4329f00186
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:19:51 -07:00
ae7313f36a
factor out input.read() and make ctor for easy subclass.
2012-07-02 12:19:20 -07:00
3ad87ba12c
rm generic parameter from ANTLRErrorListener, ripple effect.
2012-07-01 22:34:35 -07:00
bb5790d6a9
Add convenience method.
2012-07-01 17:36:22 -07:00
37396d6040
clean up
2012-07-01 12:20:54 -07:00
db66257d7e
cleanup so Lexer lets us subclass to support multiple token emit()s per lex rule match.
2012-07-01 12:19:44 -07:00
0b4fa240c5
alter comment
2012-07-01 10:55:31 -07:00
f80166b39c
Rewrote the unbuffered token stream to use a type-adjusted version of the unbuffered character stream. Simpler and I get to remove fast queue and lookahead string classes. These unbuffered streams always prime the pump with the 1st single.
...
Added a unit test for the unbuffered token stream.
Made sure that the unbuffered streams move forward always on a consume
Removed the reset method from the unbuffered streams because it's meaningless to reset to the beginning of the buffer.
2012-07-01 09:39:11 -07:00
5c69d31e88
CommonTokenFactory now knows how to copy the text out of the character stream buffer before they disappear in unbuffered character strengths; added ctor.
...
Lexer now guarantees that the text of the current token is always available to the emit() method even if the character stream is unbuffered.
Added some hooks to see some of the internal data in the unbuffered character stream so that I can test it better.
Updated LexerInterpreter so that it uses the token factory.
Improved/added unit tests for the unbuffered character string.
Updated various comments
2012-06-30 16:40:16 -07:00
44355d6ff6
oops; don't need these afterall
2012-06-30 16:36:25 -07:00
3900606024
no need for range field. removed. overrirde tostring so nothing weird comes out.
2012-06-30 16:36:01 -07:00
0c22d12870
updated comments, remove duplicate code, add new functionality.
2012-06-29 13:02:44 -07:00
e861902a10
set hidden channel num to 1 not 99. default is chan 0
2012-06-29 12:03:54 -07:00
f396b4dc82
error->syntaxError in listener
2012-06-25 13:25:13 -07:00
b18475113b
Merge branch 'master' into interactive
2012-06-16 17:20:04 -07:00
13d9f6ed53
add comment
2012-06-08 17:04:46 -07:00
3ece2c8640
merge from master; this pulls in the cleanup and bug fix in the lexer related to semantic predicates.
2012-06-08 10:18:49 -07:00
abc0e2ef87
add getSpeculativeText(); text matched so far in a lexer rule. Use this in predicates not actions. add unit tests.
2012-06-07 18:31:36 -07:00
018e3c03e8
look for T not just TLexer if no parser just tokens.
2012-06-07 18:27:42 -07:00
b255509e96
fix a bug related to semantic predicates in the lexer and generally cleaned up variable and method names in the simulator. I moved all of the predicates to the right side of lexer rules in the unit tests. Later, we should ensure that predicates only occur on the right edge of lexer rules. We should state that the rule is not been accepted so we can't test things like getText(), we have to use more raw indexes into the character stream. In the lexer simulator, the addDFAState() method now does not try to compute whether there is a predicate in the configurations. That information has already been set into the ATNConfigSet by the getEpsilonTarget() method. [I should also point out that I have not tested the Java parsing in a while and now it hits a landmine on a number of common Java files in jdk :(. I'm not sure where that crept in]
2012-06-07 17:31:18 -07:00
5a7b3b0cab
fixed a bug where some decisions that could see predicates thought they were LL(1). Added HIT_PRED invalid token type to the lookahead set to indicate we had a predicate; later we wipe out that entire set before returning from "get token lookahead sets".
2012-06-01 18:09:25 -07:00
59af8ab76b
clean up constants
2012-06-01 17:38:54 -07:00
aab8806783
added setter for _token
2012-05-26 11:32:32 -07:00
e737653b4f
lexer getText() included lookahead char, which it shouldn't.
2012-05-23 14:09:32 -07:00
bd87ec95e2
add setters for line/charpos to lexer/lexersimulator
2012-05-22 16:21:41 -07:00
ed879ab4e3
working towards interactive lexer
2012-05-21 10:16:37 -07:00
6215390408
add ctor for consistency
2012-05-20 17:28:14 -07:00
5fc148381a
rm toString from token stream. it affects buffer.
2012-05-20 16:49:44 -07:00
990da7d1d8
add unit test
2012-05-19 18:08:56 -07:00
03d7b51d3c
throw exception upon bad token indexes, add getOffChannelTokensToLeft, right. pull up some off channel token methods from CommonTokenStream. renamed.
2012-05-19 14:22:52 -07:00
7be2ba9d42
remove unused type param
2012-05-19 13:06:27 -07:00
d46c7db52c
merge master
2012-05-15 12:19:48 -07:00
c590ba8fd8
don't look backwards for err msg if EOF is entire input. make sure we don't use -1 rule index for ruleNames[]
2012-04-29 12:12:42 -07:00
3ea6a5cb2f
Merge branch 'master' into stack-graphs-integration
...
Conflicts:
tool/playground/T.g
2012-04-26 12:01:28 -07:00
30f61b17e3
comment tweak
2012-04-15 18:30:28 -07:00
f5501979c6
Merge branch 'master' into stack-graphs-integration
2012-04-09 10:56:14 -07:00
35aeeb7c5c
turn on building parse trees by default since we do -listener by default.
2012-04-07 11:27:18 -07:00
d22eb34fe7
Merge branch 'master' into stack-graphs-integration
2012-04-05 16:35:56 -07:00
1ade15392b
Add license header
2012-04-04 15:33:39 -05:00
e11dc07275
Remove TokenRewriteStream
2012-04-04 15:33:03 -05:00
08e3ddefc0
Merge branch 'master' into stack-graphs-integration
2012-04-04 13:22:00 -07:00
585aa0a14b
Merge branch 'tokenstreamrewriter' of git://github.com/sharwell/antlr4
2012-04-04 13:20:58 -07:00
ff2655d379
Rename getSource() to getTokenStream()
2012-04-04 15:20:45 -05:00
bbd61dca35
Remove TokenStreamRewriter.getOriginalText() (use rewriter.getSource().getText() instead)
2012-04-04 10:15:29 -05:00
a85f1c17a1
Add TokenStreamRewriter.getSource()
2012-04-04 10:14:09 -05:00
e4612935f2
Only need TokenStream (was overly restricted to BufferedTokenStream)
2012-04-04 10:10:33 -05:00
16046a7ec1
Proper maven reference for org.abego.treelayout.core dependency
2012-04-04 09:52:06 -05:00
bf9c7c3a28
Remove old TokenRewriteStream in favor of new TokenStreamRewriter that is not a stream; it feeds off of the stream to provide transformations of that original stream. Updated the comment at the top of the file. Instead of toString() we use getText(). Altered the unit tests to create a buffered stream and then create the rewriter attached to that.
2012-04-02 18:10:32 -07:00
3849cd572d
rm SymbolStream<T>, change all to TokenStream. all but 5 tests pass.
2012-04-02 16:49:04 -07:00
9ef6127982
merging from master
2012-04-01 18:13:40 -07:00
53c5aecb85
Implement UnbufferedTokenStream.get(int) within marked intervals
2012-04-01 18:23:32 -05:00
8ec9ba14c9
Implement getText within marked intervals of unbuffered streams
2012-04-01 18:22:43 -05:00
d6e2e41c1f
Fix stop index
2012-04-01 14:29:06 -05:00
e1870d16e9
added convenience getText(ctx) method to TokenStream. unlikley to need in CharStream and can get another way
2012-04-01 12:23:45 -07:00
588e30bfe1
getText(...) moved to CharStream and TokenStream; cleaned up names. removed getInputString from parser.
2012-04-01 12:12:49 -07:00
f238d7579e
added getText() to ParseTree. getText(tokens) is still really what you want but convenient for ctx.type().getText() when it's just one token.
2012-03-31 17:27:11 -07:00
cc20a52cdd
allow special "tokens" start rule name so we can test pure lexer grammars.
2012-03-31 15:54:00 -07:00
c9aef6fdbe
Merge branch 'master' into stack-graphs-integration
2012-03-31 10:33:22 -07:00
e5d7c27b09
Updated comments to indicate preference of .g4 over .g
2012-03-30 11:37:22 -05:00
447a5620f0
add -diagnostics option to test rig.
2012-03-28 11:07:23 -07:00
a363e70582
pull from master
2012-03-27 16:22:01 -07:00
bf07f7907f
Merge branch 'override-ruleindex' of github.com:sharwell/antlr4
2012-03-27 15:09:50 -07:00
41e97791bb
Merge branch 'set-stop-token' of github.com:sharwell/antlr4
2012-03-27 15:08:57 -07:00
f307945d2a
Merge branch 'lexer-npe' of github.com:sharwell/antlr4
2012-03-27 15:06:17 -07:00
8465920343
Merge branch 'atnstate-reachability-test' of github.com:sharwell/antlr4
2012-03-27 15:05:39 -07:00
169f58a3ff
fix def of EMPTY->INVALID
2012-03-25 21:19:39 -07:00
df2e24a228
Override getRuleIndex() in generated context objects so ParserRuleContext.ruleIndex field can be removed
2012-03-25 21:50:55 -05:00
bdda174af6
Set stop token even when RecognitionException occurs. Add ParserRuleContext.exception field to hold exception if one occurs. Resolves antlr/antlr4#49 .
2012-03-25 21:50:12 -05:00
2b5896e83b
Merge branch 'master' into stack-graphs-integration
2012-03-25 10:08:29 -07:00
542e700644
fix antlr/antlr4/#48 (getText from parse tree node). Don't NPE when start/stop null in getSourceInterval.
2012-03-25 10:01:27 -07:00
4ea3c73d1f
cleaned up interval stuff. moved getSourceInterval to ParserRuleContext; renamed create to of() in Interval.
2012-03-24 18:17:35 -07:00
c6365fb5e2
removed config list (huge) from default message.
2012-03-24 16:56:36 -07:00
e228e88ac5
tweak to remove insuff pred method
2012-03-24 16:48:26 -07:00
6791bf60cf
Merge commit '3f1f76d' into parrt-default-error-listener
...
Conflicts:
runtime/Java/src/org/antlr/v4/runtime/ANTLRErrorStrategy.java
runtime/Java/src/org/antlr/v4/runtime/DefaultErrorStrategy.java
runtime/Java/src/org/antlr/v4/runtime/DiagnosticErrorListener.java
runtime/Java/src/org/antlr/v4/runtime/atn/ParserATNSimulator.java
2012-03-24 16:44:47 -07:00
b1bcde76b7
Merge commit '768bfc0' into parrt-default-error-listener
2012-03-24 16:40:09 -07:00
b035ceec9e
EMPTY gives "" not "$" now. rename closure->closure_. had return in wrong spot in closure for-loop. fix fromRuleContext.
2012-03-20 19:06:52 -07:00
0f969af947
added in Sam's popAll to deal with loop end push/pop stuff. passes LR tests and all but 2 sempred tests. does some JavaLR.g4 parsing too.
2012-03-20 18:06:00 -07:00
9318391664
tweak to fromRuleContext. more sempred tests work.
2012-03-20 12:32:28 -07:00
55ed1a45bf
lexer wasn't using new prediction context. weird. put in functionality for $ merge in full ctx mode. removed configs array in ATNConfigSet. ATNConfigSet wasn't tracking lexerActionIndex from config. ATNConfigSet now maps key to ATNConfig.
2012-03-20 12:12:18 -07:00
5945b29a86
got more working
2012-03-19 19:58:04 -07:00
7e9a86a3e1
got TestATNInterpreter working
2012-03-19 18:47:41 -07:00
48d663667b
pull from master
2012-03-19 18:03:32 -07:00
2232ea5101
was not computing lookahead correctly in _LOOK. It assumed all epsilons were predicates.
2012-03-19 17:50:51 -07:00
85b40c7d2e
shelve
2012-03-19 17:47:57 -07:00
bd7796544d
Update comments
2012-03-19 08:33:55 -05:00
5e0f9a4490
Ensure target non-null in Transition constructor (additional runtime-side consistency check).
2012-03-19 08:33:36 -05:00
a70cb6f36a
Fix NPE in LexerATNSimulator.execDFA error recovery
2012-03-19 08:17:59 -05:00
974ae65cb9
shelve
2012-03-18 20:21:39 -07:00
90516272fe
snapshot
2012-03-18 15:39:08 -07:00
7233177441
got integrated graph stacks to compile. commit so i can look at sam's
2012-03-18 15:20:52 -07:00
3a79aa15ba
move new classes into proper package locations
2012-03-18 13:45:13 -07:00
ea7037dd2d
missing semicolon
2012-03-16 14:29:07 -07:00
9a0aaacbee
rm k=1 chk to report early ambiguity.
2012-03-16 14:11:21 -07:00
ae871dcc07
Rename exec/exec to execDFA/execATN
2012-03-13 19:27:52 -05:00
aeb43069aa
Move reach variable inside loop. Remove unnecessary ATNConfigSet clone.
2012-03-13 19:25:39 -05:00
51a15ac4a7
ATNConfigSet prior to error is closure, not reach. Reach is empty.
2012-03-13 19:25:38 -05:00
83c680850e
Only create a new ATNConfigSet for reach when no DFA edge is available
2012-03-13 19:25:38 -05:00
9d774c6543
Separate handling of DFA and ATN transitions during lexer atn sim.
2012-03-13 19:25:37 -05:00
1f60da9cfd
Handle DFA error states in lexer atn exec
2012-03-13 19:25:37 -05:00
590558c4d8
Combine DFAExecState and ATNExecState so we no longer need to throw exceptions in LexerATNSimulator.failOrAccept for valid inputs
2012-03-13 19:25:31 -05:00
aca3e0ddb2
make trim parse trees singular
2012-03-12 15:40:22 -07:00
dc627158e6
loop invar. code motion
2012-03-12 14:19:19 -07:00
2fc9760ba7
Merge remote-tracking branch 'sharwell/ambig-sempred'
2012-03-12 13:50:45 -07:00
Terence Parr
Sam Harwell