db66257d7e
cleanup so Lexer lets us subclass to support multiple token emit()s per lex rule match.
2012-07-01 12:19:44 -07:00
0b4fa240c5
alter comment
2012-07-01 10:55:31 -07:00
6053ac5269
rm gunit; hmm...my commits are wacked on this/last one. sorry!
2012-07-01 09:45:35 -07:00
f80166b39c
Rewrote the unbuffered token stream to use a type-adjusted version of the unbuffered character stream. Simpler and I get to remove fast queue and lookahead string classes. These unbuffered streams always prime the pump with the 1st single.
...
Added a unit test for the unbuffered token stream.
Made sure that the unbuffered streams move forward always on a consume
Removed the reset method from the unbuffered streams because it's meaningless to reset to the beginning of the buffer.
2012-07-01 09:39:11 -07:00
5c69d31e88
CommonTokenFactory now knows how to copy the text out of the character stream buffer before they disappear in unbuffered character strengths; added ctor.
...
Lexer now guarantees that the text of the current token is always available to the emit() method even if the character stream is unbuffered.
Added some hooks to see some of the internal data in the unbuffered character stream so that I can test it better.
Updated LexerInterpreter so that it uses the token factory.
Improved/added unit tests for the unbuffered character string.
Updated various comments
2012-06-30 16:40:16 -07:00
44355d6ff6
oops; don't need these afterall
2012-06-30 16:36:25 -07:00
3900606024
no need for range field. removed. overrirde tostring so nothing weird comes out.
2012-06-30 16:36:01 -07:00
0c22d12870
updated comments, remove duplicate code, add new functionality.
2012-06-29 13:02:44 -07:00
e861902a10
set hidden channel num to 1 not 99. default is chan 0
2012-06-29 12:03:54 -07:00
f396b4dc82
error->syntaxError in listener
2012-06-25 13:25:13 -07:00
740208ee4d
test code.
2012-06-17 16:56:26 -07:00
b18475113b
Merge branch 'master' into interactive
2012-06-16 17:20:04 -07:00
bfc90d473c
fix help msg slightly.
2012-06-13 12:42:21 -07:00
13d9f6ed53
add comment
2012-06-08 17:04:46 -07:00
abc0e2ef87
add getSpeculativeText(); text matched so far in a lexer rule. Use this in predicates not actions. add unit tests.
2012-06-07 18:31:36 -07:00
018e3c03e8
look for T not just TLexer if no parser just tokens.
2012-06-07 18:27:42 -07:00
b255509e96
fix a bug related to semantic predicates in the lexer and generally cleaned up variable and method names in the simulator. I moved all of the predicates to the right side of lexer rules in the unit tests. Later, we should ensure that predicates only occur on the right edge of lexer rules. We should state that the rule is not been accepted so we can't test things like getText(), we have to use more raw indexes into the character stream. In the lexer simulator, the addDFAState() method now does not try to compute whether there is a predicate in the configurations. That information has already been set into the ATNConfigSet by the getEpsilonTarget() method. [I should also point out that I have not tested the Java parsing in a while and now it hits a landmine on a number of common Java files in jdk :(. I'm not sure where that crept in]
2012-06-07 17:31:18 -07:00
d3c238569f
fix unit test.
2012-06-06 15:44:10 -07:00
5a2197a74d
fix bug where X : 'x' {pred}? ; wasn't seen as alias from X to 'x'.
2012-06-02 11:44:27 -07:00
7192cc9a68
rm deadcode
2012-06-02 11:44:01 -07:00
e1524fe86c
add test
2012-06-02 11:28:25 -07:00
0e9bcfbeca
change in token type broke many tests; fixed. says 5 failures now.
2012-06-02 11:26:18 -07:00
5a7b3b0cab
fixed a bug where some decisions that could see predicates thought they were LL(1). Added HIT_PRED invalid token type to the lookahead set to indicate we had a predicate; later we wipe out that entire set before returning from "get token lookahead sets".
2012-06-01 18:09:25 -07:00
5a2fe297e3
ack. more cleanup work will be required for change of token types from 3..n to 1..n
2012-06-01 18:08:00 -07:00
8481979f69
rename method
2012-06-01 18:06:48 -07:00
59af8ab76b
clean up constants
2012-06-01 17:38:54 -07:00
94071d7d82
prevent negative indexes
2012-06-01 17:36:21 -07:00
de83aacb65
add unit test
2012-05-31 17:02:15 -07:00
aab8806783
added setter for _token
2012-05-26 11:32:32 -07:00
3b1d91e010
fix unit tests
2012-05-24 13:08:15 -07:00
6539c5f464
remove inappropriate attributes of lexer rules
2012-05-24 13:07:43 -07:00
e737653b4f
lexer getText() included lookahead char, which it shouldn't.
2012-05-23 14:09:32 -07:00
cd82465830
updated [...] parser so it handles < > and [ ] properly. no escapes now.
2012-05-23 12:48:07 -07:00
bc236a6e0b
no tree attribute
2012-05-22 16:54:45 -07:00
8078d39053
rm "st" attribute
2012-05-22 16:42:05 -07:00
bd87ec95e2
add setters for line/charpos to lexer/lexersimulator
2012-05-22 16:21:41 -07:00
868ca7ca6d
named actions in combined get copied to lexer now, but lexer::{...} are MOVED to lexer grammar.
2012-05-22 16:21:06 -07:00
ed879ab4e3
working towards interactive lexer
2012-05-21 10:16:37 -07:00
d378469c9c
rename
2012-05-20 18:06:43 -07:00
6215390408
add ctor for consistency
2012-05-20 17:28:14 -07:00
5fc148381a
rm toString from token stream. it affects buffer.
2012-05-20 16:49:44 -07:00
990da7d1d8
add unit test
2012-05-19 18:08:56 -07:00
03d7b51d3c
throw exception upon bad token indexes, add getOffChannelTokensToLeft, right. pull up some off channel token methods from CommonTokenStream. renamed.
2012-05-19 14:22:52 -07:00
7be2ba9d42
remove unused type param
2012-05-19 13:06:27 -07:00
c590ba8fd8
don't look backwards for err msg if EOF is entire input. make sure we don't use -1 rule index for ruleNames[]
2012-04-29 12:12:42 -07:00
a46368275d
Merge branch 'master' of github.com:parrt/antlr4
2012-04-26 12:26:27 -07:00
5fbc994342
update changes
2012-04-26 12:26:14 -07:00
2454182a38
Merge pull request #57 from parrt/master
...
Lots of new stuff,Pulled from my main development repo
2012-04-26 12:21:15 -07:00
6314b7d31b
-> becomes # for alt labels
2012-04-26 11:59:57 -07:00
30f61b17e3
comment tweak
2012-04-15 18:30:28 -07:00
Terence Parr
Terence Parr