f220212a95
couldn't get Horstmann's routine to do EPS not PS so had to backtrack.
2012-07-14 16:32:04 -07:00
47362b2951
use Horstmann's PS save not mine; it'll handle Japanese char etc.. this way.
2012-07-14 15:38:33 -07:00
a3371ac9ea
rm unneeded override.
2012-07-09 17:32:20 -07:00
9a4227841d
-print -> -tree
2012-07-08 13:38:56 -07:00
707ff615b4
tweak javadoc
2012-07-08 12:18:43 -07:00
3b2c0a6177
fix bug in get hidden channel stuff. rewrote. update test.
2012-07-07 18:48:48 -07:00
ec47251bb2
unbuffered release() can get markers in weird order since we reset p to beginning of buffer. Might mark at 1 and then at release p = 0 etc... Don't look for errors. Just reset earliestMarker if needed.
2012-07-04 11:32:41 -07:00
6b6274b0c8
add type narrowed getters
2012-07-02 15:48:45 -07:00
71bd16020f
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:45:48 -07:00
111c44b3fd
add default ctor to Lexer
2012-07-02 12:45:30 -07:00
4329f00186
Merge branch 'master' into nogenerics-in-error-listener-2nd-try
2012-07-02 12:19:51 -07:00
ae7313f36a
factor out input.read() and make ctor for easy subclass.
2012-07-02 12:19:20 -07:00
3ad87ba12c
rm generic parameter from ANTLRErrorListener, ripple effect.
2012-07-01 22:34:35 -07:00
bb5790d6a9
Add convenience method.
2012-07-01 17:36:22 -07:00
37396d6040
clean up
2012-07-01 12:20:54 -07:00
db66257d7e
cleanup so Lexer lets us subclass to support multiple token emit()s per lex rule match.
2012-07-01 12:19:44 -07:00
0b4fa240c5
alter comment
2012-07-01 10:55:31 -07:00
f80166b39c
Rewrote the unbuffered token stream to use a type-adjusted version of the unbuffered character stream. Simpler and I get to remove fast queue and lookahead string classes. These unbuffered streams always prime the pump with the 1st single.
...
Added a unit test for the unbuffered token stream.
Made sure that the unbuffered streams move forward always on a consume
Removed the reset method from the unbuffered streams because it's meaningless to reset to the beginning of the buffer.
2012-07-01 09:39:11 -07:00
5c69d31e88
CommonTokenFactory now knows how to copy the text out of the character stream buffer before they disappear in unbuffered character strengths; added ctor.
...
Lexer now guarantees that the text of the current token is always available to the emit() method even if the character stream is unbuffered.
Added some hooks to see some of the internal data in the unbuffered character stream so that I can test it better.
Updated LexerInterpreter so that it uses the token factory.
Improved/added unit tests for the unbuffered character string.
Updated various comments
2012-06-30 16:40:16 -07:00
44355d6ff6
oops; don't need these afterall
2012-06-30 16:36:25 -07:00
3900606024
no need for range field. removed. overrirde tostring so nothing weird comes out.
2012-06-30 16:36:01 -07:00
0c22d12870
updated comments, remove duplicate code, add new functionality.
2012-06-29 13:02:44 -07:00
e861902a10
set hidden channel num to 1 not 99. default is chan 0
2012-06-29 12:03:54 -07:00
f396b4dc82
error->syntaxError in listener
2012-06-25 13:25:13 -07:00
b18475113b
Merge branch 'master' into interactive
2012-06-16 17:20:04 -07:00
13d9f6ed53
add comment
2012-06-08 17:04:46 -07:00
abc0e2ef87
add getSpeculativeText(); text matched so far in a lexer rule. Use this in predicates not actions. add unit tests.
2012-06-07 18:31:36 -07:00
018e3c03e8
look for T not just TLexer if no parser just tokens.
2012-06-07 18:27:42 -07:00
b255509e96
fix a bug related to semantic predicates in the lexer and generally cleaned up variable and method names in the simulator. I moved all of the predicates to the right side of lexer rules in the unit tests. Later, we should ensure that predicates only occur on the right edge of lexer rules. We should state that the rule is not been accepted so we can't test things like getText(), we have to use more raw indexes into the character stream. In the lexer simulator, the addDFAState() method now does not try to compute whether there is a predicate in the configurations. That information has already been set into the ATNConfigSet by the getEpsilonTarget() method. [I should also point out that I have not tested the Java parsing in a while and now it hits a landmine on a number of common Java files in jdk :(. I'm not sure where that crept in]
2012-06-07 17:31:18 -07:00
5a7b3b0cab
fixed a bug where some decisions that could see predicates thought they were LL(1). Added HIT_PRED invalid token type to the lookahead set to indicate we had a predicate; later we wipe out that entire set before returning from "get token lookahead sets".
2012-06-01 18:09:25 -07:00
59af8ab76b
clean up constants
2012-06-01 17:38:54 -07:00
aab8806783
added setter for _token
2012-05-26 11:32:32 -07:00
e737653b4f
lexer getText() included lookahead char, which it shouldn't.
2012-05-23 14:09:32 -07:00
bd87ec95e2
add setters for line/charpos to lexer/lexersimulator
2012-05-22 16:21:41 -07:00
ed879ab4e3
working towards interactive lexer
2012-05-21 10:16:37 -07:00
6215390408
add ctor for consistency
2012-05-20 17:28:14 -07:00
5fc148381a
rm toString from token stream. it affects buffer.
2012-05-20 16:49:44 -07:00
990da7d1d8
add unit test
2012-05-19 18:08:56 -07:00
03d7b51d3c
throw exception upon bad token indexes, add getOffChannelTokensToLeft, right. pull up some off channel token methods from CommonTokenStream. renamed.
2012-05-19 14:22:52 -07:00
7be2ba9d42
remove unused type param
2012-05-19 13:06:27 -07:00
c590ba8fd8
don't look backwards for err msg if EOF is entire input. make sure we don't use -1 rule index for ruleNames[]
2012-04-29 12:12:42 -07:00
30f61b17e3
comment tweak
2012-04-15 18:30:28 -07:00
35aeeb7c5c
turn on building parse trees by default since we do -listener by default.
2012-04-07 11:27:18 -07:00
1ade15392b
Add license header
2012-04-04 15:33:39 -05:00
e11dc07275
Remove TokenRewriteStream
2012-04-04 15:33:03 -05:00
585aa0a14b
Merge branch 'tokenstreamrewriter' of git://github.com/sharwell/antlr4
2012-04-04 13:20:58 -07:00
ff2655d379
Rename getSource() to getTokenStream()
2012-04-04 15:20:45 -05:00
bbd61dca35
Remove TokenStreamRewriter.getOriginalText() (use rewriter.getSource().getText() instead)
2012-04-04 10:15:29 -05:00
a85f1c17a1
Add TokenStreamRewriter.getSource()
2012-04-04 10:14:09 -05:00
e4612935f2
Only need TokenStream (was overly restricted to BufferedTokenStream)
2012-04-04 10:10:33 -05:00
Terence Parr
Sam Harwell