forgetest1
/
antlr
forked from jasder/antlr
1
0
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

Signficantly simplified Token handling. 

In order to lower the overhead when passing around Token instances via smart pointers and because the ownership is clear (token streams own them), these instances can be passed as raw pointers.
This commit is contained in:
Mike Lischke 2016-06-13 13:12:02 +02:00
parent 23eeae4b44
commit b9c28d8c9c
61 changed files with 532 additions and 409 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
doc/building-antlr.md
View File

@ -112,7 +112,8 @@ antlr4-tool-testsuite/4.5.2-SNAPSHOT/antlr4-tool-testsuite-4.5.2-SNAPSHOT.jar
Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.5.2-SNAPSHOT purposes.
To build without running the tests (saves about 8 minutes), do this:
## Building without testing
To build without running the tests (saves more than 30 mins), do this:
```bash
mvn -DskipTests install
@ -122,4 +123,4 @@ mvn -DskipTests install
After download ANTLR source, just "import project from existing sources" and click on the "Maven Projects" tab in right gutter of IDE. It should build stuff in the background automatically and look like:
<img src=images/intellij-maven.png width=200>
<img src=images/intellij-maven.png width=200>

11
doc/cpp-target.md
View File

@ -77,16 +77,16 @@ This example assumes your grammar contains a parser rule named `key` for which t
There are a couple of things that only the C++ ANTLR target has to deal with. They are described here.
### Memory Management
Since C++ has no built-in memory management we need to take extra care for that. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable.
Since C++ has no built-in memory management we need to take extra care. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable. Generally, when you see a raw pointer in code consider this as being managed elsewehere. You should never try to manage such a pointer (delete, assign to smart pointer etc.).
### Unicode Support
Encoding is mostly an input issue, i.e. when the lexer converts text input into lexer tokens. The parser is completely encoding unaware. However, lexer input in in the grammar is defined by character ranges with either a single member (e.g. 'a' or [a]), an explicit range (e.g. 'a'..'z' or [a-z]), the full Unicode range (for a wildcard) and the full Unicode range minus a sub range (for negated ranges, e.g. ~[a]). The explicit ranges are encoded in the serialized ATN by 16bit numbers, hence cannot reach beyond 0xFFFF (the Unicode BMP), while the implicit ranges can include any value (and hence support the full Unicode set, up to 0x10FFFF).
Encoding is mostly an input issue, i.e. when the lexer converts text input into lexer tokens. The parser is completely encoding unaware. However, lexer input in the grammar is defined by character ranges with either a single member (e.g. 'a' or [a] or [abc]), an explicit range (e.g. 'a'..'z' or [a-z]), the full Unicode range (for a wildcard) and the full Unicode range minus a sub range (for negated ranges, e.g. ~[a]). The explicit ranges (including single member ranges) are encoded in the serialized ATN by 16bit numbers, hence cannot reach beyond 0xFFFF (the Unicode BMP), while the implicit ranges can include any value (and hence support the full Unicode set, up to 0x10FFFF).
> An interesting side note here is that the Java target fully supports Unicode as well, despite the inherent limitations from the serialized ATN. That's possible because the Java String class represents characters beyond the BMP as surrogate pairs (two 16bit values) and even reads them as 2 characters. To make this work a character range for an identifier in a grammar must include the surrogate pairs area (for a Java parser).
> An interesting side note here is that the Java target fully supports Unicode as well, despite the inherent limitations from the serialized ATN. That's possible because the Java String class represents characters beyond the BMP as surrogate pairs (two 16bit values) and even reads them as 2 separate input characters. To make this work a character range for an identifier in a grammar must include the surrogate pairs area (for a Java parser).
The C++ target however always expects UTF-8 input (either in a string or via a wide stream) which is then converted to UTF-32 (a char32_t array) and fed to the lexer. ANTLR, when parsing your grammar, limits character ranges explicitly to the BMP currently. So, in order to allow specifying the full Unicode set the C++ target uses a little trick: whenever an explicit character range includes the (unused) codepoint 0xFFFF in a grammar it is silently extended to the full Unicode range. It's clear that this is an all-or-nothing solution. You cannot define a subset of Unicode codepoints > 0xFFFF that way. This can only be solved if ANTLR supports larger character intervals.
The differences in handling characters beyond the BMP leads to a difference between Java and C++ parsers: the character offsets may not concur. This is because Java reads two 16bit values per Unicode char (if that falls into the surrogate area) while a C++ parser only reads one 32bit value. That usually doesn't have practical consequences, but might confuse people when comparing token positions.
The differences in handling characters beyond the BMP leads to a difference between Java and C++ lexers: the character offsets may not concur. This is because Java reads two 16bit values per Unicode char (if that falls into the surrogate area) while a C++ parser only reads one 32bit value. That usually doesn't have practical consequences, but might confuse people when comparing token positions.
### Named Actions
In order to help customizing the generated files there are a number of additional socalled **named actions**. These actions are tight to specific areas in the generated code and allow to add custom (target specific) code. All targets support these actions
@ -96,7 +96,6 @@ In order to help customizing the generated files there are a number of additiona
* @lexer::header
* @lexer::members
(and their scopeless alternatives `@header` and `@members`) where header doesn't mean a C/C++ header file, but the top of a code file. The content of the header action appears in all generated files at the first line. So it's good for things like license/copyright information.
The content of a *members* action is placed in the public section of lexer or parser class declarations. Hence it can be used for public variables or predicate functions used in a grammar predicate. Since all targets support *header* + *members* they are the best place for stuff that should be available also in generated files for other languages.
@ -109,7 +108,7 @@ In addition to that the C++ target supports many more such named actions. Unfort
* **@lexer::declarations** - Placed in the private section of the lexer declaration (generated sections in all classes strictly follow the pattern: public, protected, privat, from top to bottom). Use this for private vars etc.
* **@lexer::definitions** - Placed before other implementations in the cpp file (but after *@postinclude*). Use this to implement e.g. private types.
For the parser there are the same for actions as shown above for the lexer. In addition to that there are even more actions for visitor and listener classes, which are:
For the parser there are the same actions as shown above for the lexer. In addition to that there are even more actions for visitor and listener classes:
* **@parser::listenerpreinclude**
* **@parser::listenerpostinclude**

4
runtime-testsuite/resources/org/antlr/v4/test/runtime/cpp/Cpp.test.stg
View File

@ -240,7 +240,7 @@ protected:
};
public:
virtual Ref\<Token> nextToken() override {
virtual Token* nextToken() override {
if (dynamic_cast\<PositionAdjustingLexerATNSimulator *>(_interpreter) == nullptr) {
delete _interpreter;
_interpreter = new PositionAdjustingLexerATNSimulator(this, _atn, _decisionToDFA, _sharedContextCache);
@ -249,7 +249,7 @@ public:
return Lexer::nextToken();
}
virtual Ref\<Token> emit() override {
virtual Token* emit() override {
switch (type) {
case TOKENS:
handleAcceptPositionForKeyword("tokens");

4
runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java
View File

@ -892,8 +892,8 @@ public abstract class BaseCppTest {
+ "\n"
+ "class TreeShapeListener : public tree::ParseTreeListener {\n"
+ "public:\n"
+ " void visitTerminal(Ref\\<tree::TerminalNode> node) override {}\n"
+ " void visitErrorNode(Ref\\<tree::ErrorNode> node) override {}\n"
+ " void visitTerminal(Ref\\<tree::TerminalNode> const& node) override {}\n"
+ " void visitErrorNode(Ref\\<tree::ErrorNode> const& node) override {}\n"
+ " void exitEveryRule(ParserRuleContext *ctx) override {}\n"
+ " void enterEveryRule(ParserRuleContext *ctx) override {\n"
+ " for (auto child : ctx->children) {\n"

6
runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/TestLexerExec.java
View File

@ -4683,7 +4683,7 @@ public class TestLexerExec extends BaseCppTest {
public void testPositionAdjustingLexer() throws Exception {
mkdir(tmpdir);
StringBuilder grammarBuilder = new StringBuilder(2729);
StringBuilder grammarBuilder = new StringBuilder(2721);
grammarBuilder.append("lexer grammar PositionAdjustingLexer;\n");
grammarBuilder.append("\n");
grammarBuilder.append("@members {\n");
@ -4705,7 +4705,7 @@ public class TestLexerExec extends BaseCppTest {
grammarBuilder.append(" };\n");
grammarBuilder.append("\n");
grammarBuilder.append("public:\n");
grammarBuilder.append(" virtual Ref<Token> nextToken() override {\n");
grammarBuilder.append(" virtual Token* nextToken() override {\n");
grammarBuilder.append(" if (dynamic_cast<PositionAdjustingLexerATNSimulator *>(_interpreter) == nullptr) {\n");
grammarBuilder.append(" delete _interpreter;\n");
grammarBuilder.append(" _interpreter = new PositionAdjustingLexerATNSimulator(this, _atn, _decisionToDFA, _sharedContextCache);\n");
@ -4714,7 +4714,7 @@ public class TestLexerExec extends BaseCppTest {
grammarBuilder.append(" return Lexer::nextToken();\n");
grammarBuilder.append(" }\n");
grammarBuilder.append("\n");
grammarBuilder.append(" virtual Ref<Token> emit() override {\n");
grammarBuilder.append(" virtual Token* emit() override {\n");
grammarBuilder.append(" switch (type) {\n");
grammarBuilder.append(" case TOKENS:\n");
grammarBuilder.append(" handleAcceptPositionForKeyword(\"tokens\");\n");

4
runtime/Cpp/runtime/antlrcpp.xcodeproj/project.pbxproj
View File

@ -1073,7 +1073,7 @@
276E5CC51CDB57AA003FF4B4 /* LexerNoViableAltException.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LexerNoViableAltException.cpp; sourceTree = "<group>"; };
276E5CC61CDB57AA003FF4B4 /* LexerNoViableAltException.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LexerNoViableAltException.h; sourceTree = "<group>"; };
276E5CC71CDB57AA003FF4B4 /* ListTokenSource.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ListTokenSource.cpp; sourceTree = "<group>"; wrapsLines = 0; };
276E5CC81CDB57AA003FF4B4 /* ListTokenSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ListTokenSource.h; sourceTree = "<group>"; };
276E5CC81CDB57AA003FF4B4 /* ListTokenSource.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ListTokenSource.h; sourceTree = "<group>"; wrapsLines = 0; };
276E5CCA1CDB57AA003FF4B4 /* Interval.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Interval.cpp; sourceTree = "<group>"; };
276E5CCB1CDB57AA003FF4B4 /* Interval.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Interval.h; sourceTree = "<group>"; };
276E5CCC1CDB57AA003FF4B4 /* IntervalSet.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = IntervalSet.cpp; sourceTree = "<group>"; };
@ -1151,7 +1151,7 @@
276E5D1E1CDB57AA003FF4B4 /* Trees.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Trees.h; sourceTree = "<group>"; wrapsLines = 0; };
276E5D221CDB57AA003FF4B4 /* UnbufferedCharStream.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = UnbufferedCharStream.cpp; sourceTree = "<group>"; wrapsLines = 0; };
276E5D231CDB57AA003FF4B4 /* UnbufferedCharStream.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = UnbufferedCharStream.h; sourceTree = "<group>"; };
276E5D241CDB57AA003FF4B4 /* UnbufferedTokenStream.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = UnbufferedTokenStream.cpp; sourceTree = "<group>"; };
276E5D241CDB57AA003FF4B4 /* UnbufferedTokenStream.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = UnbufferedTokenStream.cpp; sourceTree = "<group>"; wrapsLines = 0; };
276E5D251CDB57AA003FF4B4 /* UnbufferedTokenStream.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = UnbufferedTokenStream.h; sourceTree = "<group>"; };
276E5D271CDB57AA003FF4B4 /* Vocabulary.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Vocabulary.cpp; sourceTree = "<group>"; wrapsLines = 0; };
276E5D281CDB57AA003FF4B4 /* Vocabulary.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Vocabulary.h; sourceTree = "<group>"; };

2
runtime/Cpp/runtime/src/ANTLRErrorListener.h
View File

@ -76,7 +76,7 @@ namespace antlr4 {
/// the reporting of an error. It is null in the case where
/// the parser was able to recover in line without exiting the
/// surrounding rule. </param>
virtual void syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line,
virtual void syntaxError(IRecognizer *recognizer, Token *offendingSymbol, size_t line,
int charPositionInLine, const std::string &msg, std::exception_ptr e) = 0;
/**

2
runtime/Cpp/runtime/src/ANTLRErrorStrategy.h
View File

@ -80,7 +80,7 @@ namespace antlr4 {
* @throws RecognitionException if the error strategy was not able to
* recover from the unexpected input symbol
*/
virtual Ref<Token> recoverInline(Parser *recognizer) = 0;
virtual Token* recoverInline(Parser *recognizer) = 0;
/// <summary>
/// This method is called to recover from exception {@code e}. This method is

2
runtime/Cpp/runtime/src/BailErrorStrategy.cpp
View File

@ -58,7 +58,7 @@ void BailErrorStrategy::recover(Parser *recognizer, std::exception_ptr e) {
}
}
Ref<Token> BailErrorStrategy::recoverInline(Parser *recognizer) {
Token* BailErrorStrategy::recoverInline(Parser *recognizer) {
InputMismatchException e(recognizer);
std::exception_ptr exception = std::make_exception_ptr(e);

4
runtime/Cpp/runtime/src/BailErrorStrategy.h
View File

@ -73,11 +73,9 @@ namespace antlr4 {
public:
virtual void recover(Parser *recognizer, std::exception_ptr e) override;
/// <summary>
/// Make sure we don't attempt to recover inline; if the parser
/// successfully recovers, it won't throw an exception.
/// </summary>
virtual Ref<Token> recoverInline(Parser *recognizer) override;
virtual Token* recoverInline(Parser *recognizer) override;
/// <summary>
/// Make sure we don't attempt to recover from problems in subrules. </summary>

2
runtime/Cpp/runtime/src/BaseErrorListener.cpp
View File

@ -34,7 +34,7 @@
using namespace antlr4;
void BaseErrorListener::syntaxError(IRecognizer * /*recognizer*/, Ref<Token> const& /*offendingSymbol*/, size_t /*line*/,
void BaseErrorListener::syntaxError(IRecognizer * /*recognizer*/, Token * /*offendingSymbol*/, size_t /*line*/,
int /*charPositionInLine*/, const std::string &/*msg*/, std::exception_ptr /*e*/) {
}

2
runtime/Cpp/runtime/src/BaseErrorListener.h
View File

@ -46,7 +46,7 @@ namespace antlr4 {
*/
class ANTLR4CPP_PUBLIC BaseErrorListener : public ANTLRErrorListener {
virtual void syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line, int charPositionInLine,
virtual void syntaxError(IRecognizer *recognizer, Token * offendingSymbol, size_t line, int charPositionInLine,
const std::string &msg, std::exception_ptr e) override;
virtual void reportAmbiguity(Parser *recognizer, const dfa::DFA &dfa, size_t startIndex, size_t stopIndex, bool exact,

73
runtime/Cpp/runtime/src/BufferedTokenStream.cpp
View File

@ -118,12 +118,13 @@ size_t BufferedTokenStream::fetch(size_t n) {
}
for (size_t i = 0; i < n; i++) {
Ref<Token> t = _tokenSource->nextToken();
if (is<WritableToken>(t)) {
(std::dynamic_pointer_cast<WritableToken>(t))->setTokenIndex((int)_tokens.size());
std::unique_ptr<Token> t(_tokenSource->nextToken());
if (is<WritableToken *>(t.get())) {
(static_cast<WritableToken *>(t.get()))->setTokenIndex((int)_tokens.size());
}
_tokens.push_back(t);
if (t->getType() == Token::EOF) {
_tokens.push_back(std::move(t));
if (_tokens.back()->getType() == Token::EOF) {
_fetchedEOF = true;
return i + 1;
}
@ -132,18 +133,18 @@ size_t BufferedTokenStream::fetch(size_t n) {
return n;
}
Ref<Token> BufferedTokenStream::get(size_t i) const {
Token* BufferedTokenStream::get(size_t i) const {
if (i >= _tokens.size()) {
throw IndexOutOfBoundsException(std::string("token index ") +
std::to_string(i) +
std::string(" out of range 0..") +
std::to_string(_tokens.size() - 1));
}
return _tokens[i];
return _tokens[i].get();
}
std::vector<Ref<Token>> BufferedTokenStream::get(size_t start, size_t stop) {
std::vector<Ref<Token>> subset;
std::vector<Token *> BufferedTokenStream::get(size_t start, size_t stop) {
std::vector<Token *> subset;
lazyInit();
@ -155,7 +156,7 @@ std::vector<Ref<Token>> BufferedTokenStream::get(size_t start, size_t stop) {
stop = _tokens.size() - 1;
}
for (size_t i = start; i <= stop; i++) {
Ref<Token> t = _tokens[i];
Token *t = _tokens[i].get();
if (t->getType() == Token::EOF) {
break;
}
@ -168,14 +169,14 @@ ssize_t BufferedTokenStream::LA(ssize_t i) {
return LT(i)->getType();
}
Ref<Token> BufferedTokenStream::LB(size_t k) {
Token* BufferedTokenStream::LB(size_t k) {
if (k > _p) {
return nullptr;
}
return _tokens[(size_t)(_p - k)];
return _tokens[_p - k].get();
}
Ref<Token> BufferedTokenStream::LT(ssize_t k) {
Token* BufferedTokenStream::LT(ssize_t k) {
lazyInit();
if (k == 0) {
return nullptr;
@ -188,10 +189,10 @@ Ref<Token> BufferedTokenStream::LT(ssize_t k) {
sync(i);
if (i >= _tokens.size()) { // return EOF token
// EOF must be last token
return _tokens.back();
return _tokens.back().get();
}
return _tokens[i];
return _tokens[i].get();
}
ssize_t BufferedTokenStream::adjustSeekIndex(size_t i) {
@ -217,15 +218,18 @@ void BufferedTokenStream::setTokenSource(TokenSource *tokenSource) {
_needSetup = true;
}
std::vector<Ref<Token>> BufferedTokenStream::getTokens() {
return _tokens;
std::vector<Token *> BufferedTokenStream::getTokens() {
std::vector<Token *> result;
for (auto &t : _tokens)
result.push_back(t.get());
return result;
}
std::vector<Ref<Token>> BufferedTokenStream::getTokens(int start, int stop) {
std::vector<Token *> BufferedTokenStream::getTokens(int start, int stop) {
return getTokens(start, stop, std::vector<int>());
}
std::vector<Ref<Token>> BufferedTokenStream::getTokens(int start, int stop, const std::vector<int> &types) {
std::vector<Token *> BufferedTokenStream::getTokens(int start, int stop, const std::vector<int> &types) {
lazyInit();
if (start < 0 || stop >= (int)_tokens.size() || stop < 0 || start >= (int)_tokens.size()) {
throw IndexOutOfBoundsException(std::string("start ") +
@ -236,15 +240,14 @@ std::vector<Ref<Token>> BufferedTokenStream::getTokens(int start, int stop, cons
std::to_string(_tokens.size() - 1));
}
std::vector<Ref<Token>> filteredTokens;
std::vector<Token *> filteredTokens;
if (start > stop) {
return filteredTokens;
}
// list = tokens[start:stop]:{T t, t.getType() in types}
for (size_t i = (size_t)start; i <= (size_t)stop; i++) {
Ref<Token> tok = _tokens[i];
Token *tok = _tokens[i].get();
if (types.empty() || std::find(types.begin(), types.end(), tok->getType()) != types.end()) {
filteredTokens.push_back(tok);
@ -253,7 +256,7 @@ std::vector<Ref<Token>> BufferedTokenStream::getTokens(int start, int stop, cons
return filteredTokens;
}
std::vector<Ref<Token>> BufferedTokenStream::getTokens(int start, int stop, int ttype) {
std::vector<Token *> BufferedTokenStream::getTokens(int start, int stop, int ttype) {
std::vector<int> s;
s.push_back(ttype);
return getTokens(start, stop, s);
@ -265,14 +268,14 @@ ssize_t BufferedTokenStream::nextTokenOnChannel(size_t i, size_t channel) {
return size() - 1;
}
Ref<Token> token = _tokens[i];
Token *token = _tokens[i].get();
while (token->getChannel() != channel) {
if (token->getType() == Token::EOF) {
return i;
}
i++;
sync(i);
token = _tokens[i];
token = _tokens[i].get();
}
return i;
}
@ -285,7 +288,7 @@ ssize_t BufferedTokenStream::previousTokenOnChannel(size_t i, size_t channel) {
}
while (true) {
Ref<Token> token = _tokens[i];
Token *token = _tokens[i].get();
if (token->getType() == Token::EOF || token->getChannel() == channel) {
return i;
}
@ -297,7 +300,7 @@ ssize_t BufferedTokenStream::previousTokenOnChannel(size_t i, size_t channel) {
return i;
}
std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToRight(size_t tokenIndex, size_t channel) {
std::vector<Token *> BufferedTokenStream::getHiddenTokensToRight(size_t tokenIndex, size_t channel) {
lazyInit();
if (tokenIndex >= _tokens.size()) {
throw IndexOutOfBoundsException(std::to_string(tokenIndex) + " not in 0.." + std::to_string(_tokens.size() - 1));
@ -316,11 +319,11 @@ std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToRight(size_t token
return filterForChannel(from, (size_t)to, channel);
}
std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToRight(size_t tokenIndex) {
std::vector<Token *> BufferedTokenStream::getHiddenTokensToRight(size_t tokenIndex) {
return getHiddenTokensToRight(tokenIndex, -1);
}
std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToLeft(size_t tokenIndex, size_t channel) {
std::vector<Token *> BufferedTokenStream::getHiddenTokensToLeft(size_t tokenIndex, size_t channel) {
lazyInit();
if (tokenIndex >= _tokens.size()) {
throw IndexOutOfBoundsException(std::to_string(tokenIndex) + " not in 0.." + std::to_string(_tokens.size() - 1));
@ -342,14 +345,14 @@ std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToLeft(size_t tokenI
return filterForChannel(from, to, channel);
}
std::vector<Ref<Token>> BufferedTokenStream::getHiddenTokensToLeft(size_t tokenIndex) {
std::vector<Token *> BufferedTokenStream::getHiddenTokensToLeft(size_t tokenIndex) {
return getHiddenTokensToLeft(tokenIndex, -1);
}
std::vector<Ref<Token>> BufferedTokenStream::filterForChannel(size_t from, size_t to, ssize_t channel) {
std::vector<Ref<Token>> hidden;
std::vector<Token *> BufferedTokenStream::filterForChannel(size_t from, size_t to, ssize_t channel) {
std::vector<Token *> hidden;
for (size_t i = from; i <= to; i++) {
Ref<Token> t = _tokens[i];
Token *t = _tokens[i].get();
if (channel == -1) {
if (t->getChannel() != Lexer::DEFAULT_TOKEN_CHANNEL) {
hidden.push_back(t);
@ -395,7 +398,7 @@ std::string BufferedTokenStream::getText(const misc::Interval &interval) {
std::stringstream ss;
for (size_t i = (size_t)start; i <= (size_t)stop; i++) {
Ref<Token> t = _tokens[i];
Token *t = _tokens[i].get();
if (t->getType() == Token::EOF) {
break;
}
@ -408,7 +411,7 @@ std::string BufferedTokenStream::getText(RuleContext *ctx) {
return getText(ctx->getSourceInterval());
}
std::string BufferedTokenStream::getText(Ref<Token> const& start, Ref<Token> const& stop) {
std::string BufferedTokenStream::getText(Token *start, Token *stop) {
if (start != nullptr && stop != nullptr) {
return getText(misc::Interval(start->getTokenIndex(), stop->getTokenIndex()));
}

30
runtime/Cpp/runtime/src/BufferedTokenStream.h
View File

@ -62,57 +62,57 @@ namespace antlr4 {
virtual size_t size() override;
virtual void consume() override;
virtual Ref<Token> get(size_t i) const override;
virtual Token* get(size_t i) const override;
/// Get all tokens from start..stop inclusively.
virtual std::vector<Ref<Token>> get(size_t start, size_t stop);
virtual std::vector<Token *> get(size_t start, size_t stop);
virtual ssize_t LA(ssize_t i) override;
virtual Ref<Token> LT(ssize_t k) override;
virtual Token* LT(ssize_t k) override;
/// Reset this token stream by setting its token source.
virtual void setTokenSource(TokenSource *tokenSource);
virtual std::vector<Ref<Token>> getTokens();
virtual std::vector<Ref<Token>> getTokens(int start, int stop);
virtual std::vector<Token *> getTokens();
virtual std::vector<Token *> getTokens(int start, int stop);
/// <summary>
/// Given a start and stop index, return a List of all tokens in
/// the token type BitSet. Return null if no tokens were found. This
/// method looks at both on and off channel tokens.
/// </summary>
virtual std::vector<Ref<Token>> getTokens(int start, int stop, const std::vector<int> &types);
virtual std::vector<Ref<Token>> getTokens(int start, int stop, int ttype);
virtual std::vector<Token *> getTokens(int start, int stop, const std::vector<int> &types);
virtual std::vector<Token *> getTokens(int start, int stop, int ttype);
/// Collect all tokens on specified channel to the right of
/// the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or
/// EOF. If channel is -1, find any non default channel token.
virtual std::vector<Ref<Token>> getHiddenTokensToRight(size_t tokenIndex, size_t channel);
virtual std::vector<Token *> getHiddenTokensToRight(size_t tokenIndex, size_t channel);
/// <summary>
/// Collect all hidden tokens (any off-default channel) to the right of
/// the current token up until we see a token on DEFAULT_TOKEN_CHANNEL
/// of EOF.
/// </summary>
virtual std::vector<Ref<Token>> getHiddenTokensToRight(size_t tokenIndex);
virtual std::vector<Token *> getHiddenTokensToRight(size_t tokenIndex);
/// <summary>
/// Collect all tokens on specified channel to the left of
/// the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
/// If channel is -1, find any non default channel token.
/// </summary>
virtual std::vector<Ref<Token>> getHiddenTokensToLeft(size_t tokenIndex, size_t channel);
virtual std::vector<Token *> getHiddenTokensToLeft(size_t tokenIndex, size_t channel);
/// <summary>
/// Collect all hidden tokens (any off-default channel) to the left of
/// the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
/// </summary>
virtual std::vector<Ref<Token>> getHiddenTokensToLeft(size_t tokenIndex);
virtual std::vector<Token *> getHiddenTokensToLeft(size_t tokenIndex);
virtual std::string getSourceName() const override;
virtual std::string getText() override;
virtual std::string getText(const misc::Interval &interval) override;
virtual std::string getText(RuleContext *ctx) override;
virtual std::string getText(Ref<Token> const& start, Ref<Token> const& stop) override;
virtual std::string getText(Token *start, Token *stop) override;
/// <summary>
/// Get all tokens from lexer until EOF </summary>
@ -129,7 +129,7 @@ namespace antlr4 {
* considered a complete view of the input once {@link #fetchedEOF} is set
* to {@code true}.
*/
std::vector<Ref<Token>> _tokens;
std::vector<std::unique_ptr<Token>> _tokens;
/**
* The index into {@link #tokens} of the current token (next token to
@ -175,7 +175,7 @@ namespace antlr4 {
/// <returns> The actual number of elements added to the buffer. </returns>
virtual size_t fetch(size_t n);
virtual Ref<Token> LB(size_t k);
virtual Token* LB(size_t k);
/// Allowed derived classes to modify the behavior of operations which change
/// the current stream position by adjusting the target token index of a seek
@ -212,7 +212,7 @@ namespace antlr4 {
*/
virtual ssize_t previousTokenOnChannel(size_t i, size_t channel);
virtual std::vector<Ref<Token>> filterForChannel(size_t from, size_t to, ssize_t channel);
virtual std::vector<Token *> filterForChannel(size_t from, size_t to, ssize_t channel);
bool isInitialized() const;

8
runtime/Cpp/runtime/src/CommonTokenFactory.cpp
View File

@ -45,10 +45,10 @@ CommonTokenFactory::CommonTokenFactory(bool copyText) : copyText(copyText) {
CommonTokenFactory::CommonTokenFactory() : CommonTokenFactory(false) {
}
Ref<CommonToken> CommonTokenFactory::create(std::pair<TokenSource*, CharStream*> source, int type,
std::unique_ptr<CommonToken> CommonTokenFactory::create(std::pair<TokenSource*, CharStream*> source, int type,
const std::string &text, int channel, int start, int stop, int line, int charPositionInLine) {
Ref<CommonToken> t = std::make_shared<CommonToken>(source, type, channel, start, stop);
std::unique_ptr<CommonToken> t(new CommonToken(source, type, channel, start, stop));
t->setLine(line);
t->setCharPositionInLine(charPositionInLine);
if (text != "") {
@ -60,6 +60,6 @@ Ref<CommonToken> CommonTokenFactory::create(std::pair<TokenSource*, CharStream*>
return t;
}
Ref<CommonToken> CommonTokenFactory::create(int type, const std::string &text) {
return std::make_shared<CommonToken>(type, text);
std::unique_ptr<CommonToken> CommonTokenFactory::create(int type, const std::string &text) {
return std::unique_ptr<CommonToken>(new CommonToken(type, text));
}

4
runtime/Cpp/runtime/src/CommonTokenFactory.h
View File

@ -91,10 +91,10 @@ namespace antlr4 {
*/
CommonTokenFactory();
virtual Ref<CommonToken> create(std::pair<TokenSource*, CharStream*> source, int type,
virtual std::unique_ptr<CommonToken> create(std::pair<TokenSource*, CharStream*> source, int type,
const std::string &text, int channel, int start, int stop, int line, int charPositionInLine) override;
virtual Ref<CommonToken> create(int type, const std::string &text) override;
virtual std::unique_ptr<CommonToken> create(int type, const std::string &text) override;
};
} // namespace antlr4

12
runtime/Cpp/runtime/src/CommonTokenStream.cpp
View File

@ -47,9 +47,9 @@ ssize_t CommonTokenStream::adjustSeekIndex(size_t i) {
return nextTokenOnChannel(i, channel);
}
Ref<Token> CommonTokenStream::LB(size_t k) {
Token* CommonTokenStream::LB(size_t k) {
if (k == 0 || k > _p) {
return Ref<Token>();
return nullptr;
}
ssize_t i = (ssize_t)_p;
@ -64,10 +64,10 @@ Ref<Token> CommonTokenStream::LB(size_t k) {
return nullptr;
}
return _tokens[i];
return _tokens[i].get();
}
Ref<Token> CommonTokenStream::LT(ssize_t k) {
Token* CommonTokenStream::LT(ssize_t k) {
lazyInit();
if (k == 0) {
return nullptr;
@ -86,14 +86,14 @@ Ref<Token> CommonTokenStream::LT(ssize_t k) {
n++;
}
return _tokens[i];
return _tokens[i].get();
}
int CommonTokenStream::getNumberOfOnChannelTokens() {
int n = 0;
fill();
for (size_t i = 0; i < _tokens.size(); i++) {
Ref<Token> t = _tokens[i];
Token *t = _tokens[i].get();
if (t->getChannel() == channel) {
n++;
}

4
runtime/Cpp/runtime/src/CommonTokenStream.h
View File

@ -94,10 +94,10 @@ namespace antlr4 {
protected:
virtual ssize_t adjustSeekIndex(size_t i) override;
virtual Ref<Token> LB(size_t k) override;
virtual Token* LB(size_t k) override;
public:
virtual Ref<Token> LT(ssize_t k) override;
virtual Token* LT(ssize_t k) override;
/// Count EOF just once.
virtual int getNumberOfOnChannelTokens();

2
runtime/Cpp/runtime/src/ConsoleErrorListener.cpp
View File

@ -35,7 +35,7 @@ using namespace antlr4;
ConsoleErrorListener ConsoleErrorListener::INSTANCE;
void ConsoleErrorListener::syntaxError(IRecognizer * /*recognizer*/, Ref<Token> const& /*offendingSymbol*/,
void ConsoleErrorListener::syntaxError(IRecognizer * /*recognizer*/, Token * /*offendingSymbol*/,
size_t line, int charPositionInLine, const std::string &msg, std::exception_ptr /*e*/) {
std::cerr << "line " << line << ":" << charPositionInLine << " " << msg << std::endl;
}

2
runtime/Cpp/runtime/src/ConsoleErrorListener.h
View File

@ -54,7 +54,7 @@ namespace antlr4 {
* line <em>line</em>:<em>charPositionInLine</em> <em>msg</em>
* </pre>
*/
virtual void syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line, int charPositionInLine,
virtual void syntaxError(IRecognizer *recognizer, Token * offendingSymbol, size_t line, int charPositionInLine,
const std::string &msg, std::exception_ptr e) override;
};

41
runtime/Cpp/runtime/src/DefaultErrorStrategy.cpp
View File

@ -48,6 +48,13 @@
using namespace antlr4;
using namespace antlrcpp;
DefaultErrorStrategy::DefaultErrorStrategy() {
InitializeInstanceFields();
}
DefaultErrorStrategy::~DefaultErrorStrategy() {
}
void DefaultErrorStrategy::reset(Parser *recognizer) {
endErrorCondition(recognizer);
}
@ -188,7 +195,7 @@ void DefaultErrorStrategy::reportUnwantedToken(Parser *recognizer) {
beginErrorCondition(recognizer);
Ref<Token> t = recognizer->getCurrentToken();
Token *t = recognizer->getCurrentToken();
std::string tokenName = getTokenErrorDisplay(t);
misc::IntervalSet expecting = getExpectedTokens(recognizer);
@ -204,16 +211,16 @@ void DefaultErrorStrategy::reportMissingToken(Parser *recognizer) {
beginErrorCondition(recognizer);
Ref<Token> t = recognizer->getCurrentToken();
Token *t = recognizer->getCurrentToken();
misc::IntervalSet expecting = getExpectedTokens(recognizer);
std::string msg = "missing " + expecting.toString(recognizer->getVocabulary()) + " at " + getTokenErrorDisplay(t);
recognizer->notifyErrorListeners(t, msg, nullptr);
}
Ref<Token> DefaultErrorStrategy::recoverInline(Parser *recognizer) {
Token* DefaultErrorStrategy::recoverInline(Parser *recognizer) {
// SINGLE TOKEN DELETION
Ref<Token> matchedSymbol = singleTokenDeletion(recognizer);
Token *matchedSymbol = singleTokenDeletion(recognizer);
if (matchedSymbol) {
// we have deleted the extra token.
// now, move past ttype token as if all were ok
@ -247,22 +254,22 @@ bool DefaultErrorStrategy::singleTokenInsertion(Parser *recognizer) {
return false;
}
Ref<Token> DefaultErrorStrategy::singleTokenDeletion(Parser *recognizer) {
Token* DefaultErrorStrategy::singleTokenDeletion(Parser *recognizer) {
ssize_t nextTokenType = recognizer->getInputStream()->LA(2);
misc::IntervalSet expecting = getExpectedTokens(recognizer);
if (expecting.contains((int)nextTokenType)) {
reportUnwantedToken(recognizer);
recognizer->consume(); // simply delete extra token
// we want to return the token we're actually matching
Ref<Token> matchedSymbol = recognizer->getCurrentToken();
Token *matchedSymbol = recognizer->getCurrentToken();
reportMatch(recognizer); // we know current token is correct
return matchedSymbol;
}
return nullptr;
}
Ref<Token> DefaultErrorStrategy::getMissingSymbol(Parser *recognizer) {
Ref<Token> currentSymbol = recognizer->getCurrentToken();
Token* DefaultErrorStrategy::getMissingSymbol(Parser *recognizer) {
Token *currentSymbol = recognizer->getCurrentToken();
misc::IntervalSet expecting = getExpectedTokens(recognizer);
ssize_t expectedTokenType = expecting.getMinElement(); // get any element
std::string tokenText;
@ -271,21 +278,24 @@ Ref<Token> DefaultErrorStrategy::getMissingSymbol(Parser *recognizer) {
} else {
tokenText = "<missing " + recognizer->getVocabulary().getDisplayName(expectedTokenType) + ">";
}
Ref<Token> current = currentSymbol;
Ref<Token> lookback = recognizer->getTokenStream()->LT(-1);
Token *current = currentSymbol;
Token *lookback = recognizer->getTokenStream()->LT(-1);
if (current->getType() == Token::EOF && lookback != nullptr) {
current = lookback;
}
return std::dynamic_pointer_cast<Token>(recognizer->getTokenFactory()->create({ current->getTokenSource(),
_missingSymbol = recognizer->getTokenFactory()->create({ current->getTokenSource(),
current->getTokenSource()->getInputStream() }, (int)expectedTokenType, tokenText, Token::DEFAULT_CHANNEL, -1, -1,
current->getLine(), current->getCharPositionInLine()));
current->getLine(), current->getCharPositionInLine());
return _missingSymbol.get();
}
misc::IntervalSet DefaultErrorStrategy::getExpectedTokens(Parser *recognizer) {
return recognizer->getExpectedTokens();
}
std::string DefaultErrorStrategy::getTokenErrorDisplay(Ref<Token> const& t) {
std::string DefaultErrorStrategy::getTokenErrorDisplay(Token *t) {
if (t == nullptr) {
return "<no Token>";
}
@ -300,11 +310,11 @@ std::string DefaultErrorStrategy::getTokenErrorDisplay(Ref<Token> const& t) {
return escapeWSAndQuote(s);
}
std::string DefaultErrorStrategy::getSymbolText(Ref<Token> const& symbol) {
std::string DefaultErrorStrategy::getSymbolText(Token *symbol) {
return symbol->getText();
}
int DefaultErrorStrategy::getSymbolType(Ref<Token> const& symbol) {
int DefaultErrorStrategy::getSymbolType(Token *symbol) {
return symbol->getType();
}
@ -347,4 +357,5 @@ void DefaultErrorStrategy::consumeUntil(Parser *recognizer, const misc::Interval
void DefaultErrorStrategy::InitializeInstanceFields() {
errorRecoveryMode = false;
lastErrorIndex = -1;
_missingSymbol = nullptr;
}

19
runtime/Cpp/runtime/src/DefaultErrorStrategy.h
View File

@ -42,9 +42,8 @@ namespace antlr4 {
*/
class ANTLR4CPP_PUBLIC DefaultErrorStrategy : public ANTLRErrorStrategy {
public:
DefaultErrorStrategy() {
InitializeInstanceFields();
}
DefaultErrorStrategy();
virtual ~DefaultErrorStrategy();
protected:
/**
@ -302,7 +301,7 @@ namespace antlr4 {
* is in the set of tokens that can follow the {@code ')'} token reference
* in rule {@code atom}. It can assume that you forgot the {@code ')'}.
*/
virtual Ref<Token> recoverInline(Parser *recognizer) override;
virtual Token* recoverInline(Parser *recognizer) override;
/// <summary>
/// This method implements the single-token insertion inline error recovery
@ -341,7 +340,7 @@ namespace antlr4 {
/// <returns> the successfully matched <seealso cref="Token"/> instance if single-token
/// deletion successfully recovers from the mismatched input, otherwise
/// {@code null} </returns>
virtual Ref<Token> singleTokenDeletion(Parser *recognizer);
virtual Token* singleTokenDeletion(Parser *recognizer);
/// <summary>
/// Conjure up a missing token during error recovery.
@ -363,7 +362,7 @@ namespace antlr4 {
/// If you change what tokens must be created by the lexer,
/// override this method to create the appropriate tokens.
/// </summary>
virtual Ref<Token> getMissingSymbol(Parser *recognizer);
virtual Token* getMissingSymbol(Parser *recognizer);
virtual misc::IntervalSet getExpectedTokens(Parser *recognizer);
@ -376,11 +375,11 @@ namespace antlr4 {
/// your token objects because you don't have to go modify your lexer
/// so that it creates a new class.
/// </summary>
virtual std::string getTokenErrorDisplay(Ref<Token> const& t);
virtual std::string getTokenErrorDisplay(Token *t);
virtual std::string getSymbolText(Ref<Token> const& symbol);
virtual std::string getSymbolText(Token *symbol);
virtual int getSymbolType(Ref<Token> const& symbol);
virtual int getSymbolType(Token *symbol);
virtual std::string escapeWSAndQuote(std::string &s);
@ -483,8 +482,8 @@ namespace antlr4 {
virtual void consumeUntil(Parser *recognizer, const misc::IntervalSet &set);
private:
std::unique_ptr<Token> _missingSymbol; // Temporarily created token.
void InitializeInstanceFields();
};
} // namespace antlr4

42
runtime/Cpp/runtime/src/Lexer.cpp
View File

@ -72,7 +72,7 @@ void Lexer::reset() {
getInterpreter<atn::LexerATNSimulator>()->reset();
}
Ref<Token> Lexer::nextToken() {
std::unique_ptr<Token> Lexer::nextToken() {
// Mark start location in char stream so unbuffered streams are
// guaranteed at least have text of current token
ssize_t tokenStartMarker = _input->mark();
@ -87,7 +87,7 @@ Ref<Token> Lexer::nextToken() {
outerContinue:
if (hitEOF) {
emitEOF();
return token;
return std::move(token);
}
token.reset();
@ -119,7 +119,7 @@ Ref<Token> Lexer::nextToken() {
if (token == nullptr) {
emit();
}
return token;
return std::move(token);
}
}
@ -173,24 +173,22 @@ CharStream* Lexer::getInputStream() {
return _input;
}
void Lexer::emit(Ref<Token> const& token) {
this->token = token;
void Lexer::emit(std::unique_ptr<Token> token) {
this->token = std::move(token);
}
Ref<Token> Lexer::emit() {
Ref<Token> t = std::dynamic_pointer_cast<Token>(_factory->create({ this, _input }, (int)type, _text, channel,
Token* Lexer::emit() {
emit(_factory->create({ this, _input }, (int)type, _text, channel,
tokenStartCharIndex, getCharIndex() - 1, (int)tokenStartLine, tokenStartCharPositionInLine));
emit(t);
return t;
return token.get();
}
Ref<Token> Lexer::emitEOF() {
Token* Lexer::emitEOF() {
int cpos = getCharPositionInLine();
size_t line = getLine();
Ref<Token> eof = std::dynamic_pointer_cast<Token>(_factory->create({ this, _input }, EOF, "", Token::DEFAULT_CHANNEL,
(int)_input->index(), (int)_input->index() - 1, (int)line, cpos));
emit(eof);
return eof;
emit(_factory->create({ this, _input }, EOF, "", Token::DEFAULT_CHANNEL, (int)_input->index(),
(int)_input->index() - 1, (int)line, cpos));
return token.get();
}
size_t Lexer::getLine() const {
@ -224,12 +222,12 @@ void Lexer::setText(const std::string &text) {
_text = text;
}
Ref<Token> Lexer::getToken() {
return token;
std::unique_ptr<Token> Lexer::getToken() {
return std::move(token);
}
void Lexer::setToken(Ref<Token> const& token) {
this->token = token;
void Lexer::setToken(std::unique_ptr<Token> token) {
this->token = std::move(token);
}
void Lexer::setType(ssize_t ttype) {
@ -248,11 +246,11 @@ int Lexer::getChannel() {
return channel;
}
std::vector<Ref<Token>> Lexer::getAllTokens() {
std::vector<Ref<Token>> tokens;
Ref<Token> t = nextToken();
std::vector<std::unique_ptr<Token>> Lexer::getAllTokens() {
std::vector<std::unique_ptr<Token>> tokens;
std::unique_ptr<Token> t = nextToken();
while (t->getType() != EOF) {
tokens.push_back(t);
tokens.push_back(std::move(t));
t = nextToken();
}
return tokens;

23
runtime/Cpp/runtime/src/Lexer.h
View File

@ -67,7 +67,11 @@ namespace antlr4 {
/// emissions, then set this to the last token to be matched or
/// something nonnull so that the auto token emit mechanism will not
/// emit another token.
Ref<Token> token;
// Life cycle of a token is this:
// Created by emit() (via the token factory) or by action code, holding ownership of it.
// Ownership is handed over to the token stream when calling nextToken().
std::unique_ptr<Token> token;
/// <summary>
/// What character index in the stream did the current token start at?
@ -108,7 +112,7 @@ namespace antlr4 {
virtual void reset();
/// Return a token from this source; i.e., match a token on the char stream.
virtual Ref<Token> nextToken() override;
virtual std::unique_ptr<Token> nextToken() override;
/// Instruct the lexer to skip creating a token for current lexer rule
/// and look for another token. nextToken() knows to keep looking when
@ -139,16 +143,16 @@ namespace antlr4 {
/// for efficiency reasons. Subclass and override this method, nextToken,
/// and getToken (to push tokens into a list and pull from that list
/// rather than a single variable as this implementation does).
virtual void emit(Ref<Token> const& token);
virtual void emit(std::unique_ptr<Token> token);
/// The standard method called to automatically emit a token at the
/// outermost lexical rule. The token object should point into the
/// char buffer start..stop. If there is a text override in 'text',
/// use that to set the token's text. Override this method to emit
/// custom Token objects or provide a new factory.
virtual Ref<Token> emit();
virtual Token* emit();
virtual Ref<Token> emitEOF();
virtual Token* emitEOF();
virtual size_t getLine() const override;
@ -169,11 +173,10 @@ namespace antlr4 {
/// changes to the text.
virtual void setText(const std::string &text);
/// <summary>
/// Override if emitting multiple tokens. </summary>
virtual Ref<Token> getToken();
/// Override if emitting multiple tokens.
virtual std::unique_ptr<Token> getToken();
virtual void setToken(Ref<Token> const& token);
virtual void setToken(std::unique_ptr<Token> token);
virtual void setType(ssize_t ttype);
@ -187,7 +190,7 @@ namespace antlr4 {
/// Return a list of all Token objects in input char stream.
/// Forces load of all tokens. Does not include EOF token.
virtual std::vector<Ref<Token>> getAllTokens();
virtual std::vector<std::unique_ptr<Token>> getAllTokens();
virtual void recover(const LexerNoViableAltException &e);

99
runtime/Cpp/runtime/src/ListTokenSource.cpp
View File

@ -37,95 +37,58 @@
using namespace antlr4;
ListTokenSource::ListTokenSource(std::vector<std::unique_ptr<Token>> tokens) : ListTokenSource(std::move(tokens), "") {
}
ListTokenSource::ListTokenSource(std::vector<std::unique_ptr<Token>> tokens_, const std::string &sourceName_)
: tokens(std::move(tokens_)), sourceName(sourceName_) {
InitializeInstanceFields();
if (tokens.empty()) {
throw "tokens cannot be null";
}
// Check if there is an eof token and create one if not.
if (tokens.back()->getType() != Token::EOF) {
Token *lastToken = tokens.back().get();
int start = -1;
int previousStop = lastToken->getStopIndex();
if (previousStop != -1) {
start = previousStop + 1;
}
int stop = std::max(-1, start - 1);
tokens.emplace_back((_factory->create({ this, getInputStream() }, Token::EOF, "EOF",
Token::DEFAULT_CHANNEL, start, stop, (int)lastToken->getLine(), lastToken->getCharPositionInLine())));
}
}
int ListTokenSource::getCharPositionInLine() {
if (i < tokens.size()) {
return tokens[i]->getCharPositionInLine();
} else if (eofToken != nullptr) {
return eofToken->getCharPositionInLine();
} else if (tokens.size() > 0) {
// have to calculate the result from the line/column of the previous
// token, along with the text of the token.
Ref<Token> lastToken = tokens.back();
std::string tokenText = lastToken->getText();
if (tokenText != "") {
int lastNewLine = (int)tokenText.rfind('\n');
if (lastNewLine >= 0) {
return (int)tokenText.length() - lastNewLine - 1;
}
}
return lastToken->getCharPositionInLine() + lastToken->getStopIndex() - lastToken->getStartIndex() + 1;
}
// only reach this if tokens is empty, meaning EOF occurs at the first
// position in the input
return 0;
}
Ref<Token> ListTokenSource::nextToken() {
if (i >= tokens.size()) {
if (eofToken == nullptr) {
int start = -1;
if (tokens.size() > 0) {
int previousStop = tokens.back()->getStopIndex();
if (previousStop != -1) {
start = previousStop + 1;
}
}
int stop = std::max(-1, start - 1);
eofToken = std::dynamic_pointer_cast<Token>(_factory->create({ this, getInputStream() }, Token::EOF, "EOF",
Token::DEFAULT_CHANNEL, start, stop, (int)getLine(), getCharPositionInLine()));
}
return eofToken;
std::unique_ptr<Token> ListTokenSource::nextToken() {
if (i < tokens.size()) {
return std::move(tokens[i++]);
}
Ref<Token> t = tokens[i];
if (i == tokens.size() - 1 && t->getType() == Token::EOF) {
eofToken = t;
}
i++;
return t;
return nullptr;
}
size_t ListTokenSource::getLine() const {
if (i < tokens.size()) {
return (size_t)tokens[i]->getLine();
} else if (eofToken != nullptr) {
return (size_t)eofToken->getLine();
} else if (tokens.size() > 0) {
// have to calculate the result from the line/column of the previous
// token, along with the text of the token.
Ref<Token> lastToken = tokens.back();
int line = lastToken->getLine();
std::string tokenText = lastToken->getText();
if (tokenText != "") {
for (size_t i = 0; i < tokenText.length(); i++) {
if (tokenText[i] == '\n') {
line++;
}
}
}
// if no text is available, assume the token did not contain any newline characters.
return (size_t)line;
}
// only reach this if tokens is empty, meaning EOF occurs at the first
// position in the input
return 1;
}
CharStream *ListTokenSource::getInputStream() {
if (i < tokens.size()) {
return tokens[i]->getInputStream();
} else if (eofToken != nullptr) {
return eofToken->getInputStream();
} else if (tokens.size() > 0) {
return tokens[tokens.size() - 1]->getInputStream();
} else if (!tokens.empty()) {
return tokens.back()->getInputStream();
}
// no input stream information is available

44
runtime/Cpp/runtime/src/ListTokenSource.h
View File

@ -36,59 +36,46 @@
namespace antlr4 {
/// <summary>
/// Provides an implementation of <seealso cref="TokenSource"/> as a wrapper around a list
/// of <seealso cref="Token"/> objects.
/// <p/>
///
/// If the final token in the list is an <seealso cref="Token#EOF"/> token, it will be used
/// as the EOF token for every call to <seealso cref="#nextToken"/> after the end of the
/// list is reached. Otherwise, an EOF token will be created.
/// </summary>
class ANTLR4CPP_PUBLIC ListTokenSource : public TokenSource {
protected:
const std::vector<Ref<Token>> tokens;
// This list will be emptied token by token as we call nextToken().
// Token streams can be used to buffer tokens for a while.
std::vector<std::unique_ptr<Token>> tokens;
private:
/// <summary>
/// The name of the input source. If this value is {@code null}, a call to
/// <seealso cref="#getSourceName"/> should return the source name used to create the
/// the next token in <seealso cref="#tokens"/> (or the previous token if the end of
/// the input has been reached).
/// </summary>
private:
const std::string sourceName;
/// <summary>
protected:
/// The index into <seealso cref="#tokens"/> of token to return by the next call to
/// <seealso cref="#nextToken"/>. The end of the input is indicated by this value
/// being greater than or equal to the number of items in <seealso cref="#tokens"/>.
/// </summary>
protected:
size_t i;
/// <summary>
/// This field caches the EOF token for the token source.
/// </summary>
Ref<Token> eofToken;
/// <summary>
private:
/// This is the backing field for <seealso cref="#getTokenFactory"/> and
/// <seealso cref="setTokenFactory"/>.
/// </summary>
private:
Ref<TokenFactory<CommonToken>> _factory = CommonTokenFactory::DEFAULT;
/// <summary>
public:
/// Constructs a new <seealso cref="ListTokenSource"/> instance from the specified
/// collection of <seealso cref="Token"/> objects.
/// </summary>
///
/// <param name="tokens"> The collection of <seealso cref="Token"/> objects to provide as a
/// <seealso cref="TokenSource"/>. </param>
/// <exception cref="NullPointerException"> if {@code tokens} is {@code null} </exception>
public:
template<typename T1>
ListTokenSource(std::vector<T1> tokens) : ListTokenSource(tokens, "") {
}
ListTokenSource(std::vector<std::unique_ptr<Token>> tokens);
/// <summary>
/// Constructs a new <seealso cref="ListTokenSource"/> instance from the specified
@ -102,17 +89,10 @@ namespace antlr4 {
/// been reached).
/// </param>
/// <exception cref="NullPointerException"> if {@code tokens} is {@code null} </exception>
template<typename T1>
ListTokenSource(std::vector<T1> tokens, const std::string &/*sourceName*/) {
InitializeInstanceFields();
if (tokens.empty()) {
throw "tokens cannot be nul";
}
}
ListTokenSource(std::vector<std::unique_ptr<Token>> tokens_, const std::string &sourceName_);
virtual int getCharPositionInLine() override;
virtual Ref<Token> nextToken() override;
virtual std::unique_ptr<Token> nextToken() override;
virtual size_t getLine() const override;
virtual CharStream* getInputStream() override;
virtual std::string getSourceName() override;

6
runtime/Cpp/runtime/src/NoViableAltException.cpp
View File

@ -40,12 +40,12 @@ NoViableAltException::NoViableAltException(Parser *recognizer)
recognizer->getCurrentToken(), nullptr, recognizer->getContext()) {
}
NoViableAltException::NoViableAltException(Parser *recognizer, TokenStream *input, Ref<Token> const& startToken,
Ref<Token> const& offendingToken, atn::ATNConfigSet *deadEndConfigs, Ref<ParserRuleContext> const& ctx)
NoViableAltException::NoViableAltException(Parser *recognizer, TokenStream *input,Token *startToken,
Token *offendingToken, atn::ATNConfigSet *deadEndConfigs, Ref<ParserRuleContext> const& ctx)
: RecognitionException("No viable alternative", recognizer, input, ctx, offendingToken), _deadEndConfigs(deadEndConfigs), _startToken(startToken) {
}
Ref<Token> NoViableAltException::getStartToken() const {
Token* NoViableAltException::getStartToken() const {
return _startToken;
}

8
runtime/Cpp/runtime/src/NoViableAltException.h
View File

@ -44,10 +44,10 @@ namespace antlr4 {
class ANTLR4CPP_PUBLIC NoViableAltException : public RecognitionException {
public:
NoViableAltException(Parser *recognizer); // LL(1) error
NoViableAltException(Parser *recognizer, TokenStream *input, Ref<Token> const& startToken,
Ref<Token> const& offendingToken, atn::ATNConfigSet *deadEndConfigs, Ref<ParserRuleContext> const& ctx);
NoViableAltException(Parser *recognizer, TokenStream *input,Token *startToken,
Token *offendingToken, atn::ATNConfigSet *deadEndConfigs, Ref<ParserRuleContext> const& ctx);
virtual Ref<Token> getStartToken() const;
virtual Token* getStartToken() const;
virtual atn::ATNConfigSet* getDeadEndConfigs() const;
private:
@ -58,7 +58,7 @@ namespace antlr4 {
/// not be buffering tokens so get a reference to it. (At the
/// time the error occurred, of course the stream needs to keep a
/// buffer all of the tokens but later we might not have access to those.)
Ref<Token> _startToken;
Token *_startToken;
};

16
runtime/Cpp/runtime/src/Parser.cpp
View File

@ -117,8 +117,8 @@ void Parser::reset() {
}
}
Ref<Token> Parser::match(int ttype) {
Ref<Token> t = getCurrentToken();
Token* Parser::match(int ttype) {
Token *t = getCurrentToken();
if (t->getType() == ttype) {
if (ttype == EOF) {
_matchedEOF = true;
@ -136,8 +136,8 @@ Ref<Token> Parser::match(int ttype) {
return t;
}
Ref<Token> Parser::matchWildcard() {
Ref<Token> t = getCurrentToken();
Token* Parser::matchWildcard() {
Token *t = getCurrentToken();
if (t->getType() > 0) {
_errHandler->reportMatch(this);
consume();
@ -290,7 +290,7 @@ void Parser::setTokenStream(TokenStream *input) {
_input = input;
}
Ref<Token> Parser::getCurrentToken() {
Token* Parser::getCurrentToken() {
return _input->LT(1);
}
@ -298,7 +298,7 @@ void Parser::notifyErrorListeners(const std::string &msg) {
notifyErrorListeners(getCurrentToken(), msg, nullptr);
}
void Parser::notifyErrorListeners(Ref<Token> const& offendingToken, const std::string &msg, std::exception_ptr e) {
void Parser::notifyErrorListeners(Token *offendingToken, const std::string &msg, std::exception_ptr e) {
_syntaxErrors++;
int line = -1;
int charPositionInLine = -1;
@ -309,8 +309,8 @@ void Parser::notifyErrorListeners(Ref<Token> const& offendingToken, const std::s
listener.syntaxError(this, offendingToken, (size_t)line, charPositionInLine, msg, e);
}
Ref<Token> Parser::consume() {
Ref<Token> o = getCurrentToken();
Token* Parser::consume() {
Token *o = getCurrentToken();
if (o->getType() != EOF) {
getInputStream()->consume();
}

10
runtime/Cpp/runtime/src/Parser.h
View File

@ -92,7 +92,7 @@ namespace antlr4 {
/// <exception cref="RecognitionException"> if the current input symbol did not match
/// {@code ttype} and the error strategy could not recover from the
/// mismatched symbol </exception>
virtual Ref<Token> match(int ttype);
virtual Token* match(int ttype);
/// <summary>
/// Match current input symbol as a wildcard. If the symbol type matches
@ -110,7 +110,7 @@ namespace antlr4 {
/// <exception cref="RecognitionException"> if the current input symbol did not match
/// a wildcard and the error strategy could not recover from the mismatched
/// symbol </exception>
virtual Ref<Token> matchWildcard();
virtual Token* matchWildcard();
/// <summary>
/// Track the <seealso cref="ParserRuleContext"/> objects during the parse and hook
@ -265,11 +265,11 @@ namespace antlr4 {
/// Match needs to return the current input symbol, which gets put
/// into the label for the associated token ref; e.g., x=ID.
/// </summary>
virtual Ref<Token> getCurrentToken();
virtual Token* getCurrentToken();
void notifyErrorListeners(const std::string &msg);
virtual void notifyErrorListeners(Ref<Token> const& offendingToken, const std::string &msg, std::exception_ptr e);
virtual void notifyErrorListeners(Token *offendingToken, const std::string &msg, std::exception_ptr e);
/// <summary>
/// Consume and return the <seealso cref="#getCurrentToken current symbol"/>.
@ -292,7 +292,7 @@ namespace antlr4 {
/// <seealso cref="ParseTreeListener#visitErrorNode"/> is called on any parse
/// listeners.
/// </summary>
virtual Ref<Token> consume();
virtual Token* consume();
/// <summary>
/// Always called by generated parsers upon entry to a rule. Access field

14
runtime/Cpp/runtime/src/ParserInterpreter.cpp
View File

@ -306,23 +306,23 @@ void ParserInterpreter::recover(RecognitionException &e) {
// no input consumed, better add an error node
if (is<InputMismatchException *>(&e)) {
InputMismatchException &ime = (InputMismatchException&)e;
Ref<Token> tok = e.getOffendingToken();
Token *tok = e.getOffendingToken();
int expectedTokenType = ime.getExpectedTokens().getMinElement(); // get any element
auto errToken = getTokenFactory()->create({ tok->getTokenSource(), tok->getTokenSource()->getInputStream() },
_errorToken = getTokenFactory()->create({ tok->getTokenSource(), tok->getTokenSource()->getInputStream() },
expectedTokenType, tok->getText(), Token::DEFAULT_CHANNEL, -1, -1, // invalid start/stop
tok->getLine(), tok->getCharPositionInLine());
_ctx->addErrorNode(std::dynamic_pointer_cast<Token>(errToken));
_ctx->addErrorNode(_errorToken.get());
}
else { // NoViableAlt
Ref<Token> tok = e.getOffendingToken();
auto errToken = getTokenFactory()->create({ tok->getTokenSource(), tok->getTokenSource()->getInputStream() },
Token *tok = e.getOffendingToken();
_errorToken = getTokenFactory()->create({ tok->getTokenSource(), tok->getTokenSource()->getInputStream() },
Token::INVALID_TYPE, tok->getText(), Token::DEFAULT_CHANNEL, -1, -1, // invalid start/stop
tok->getLine(), tok->getCharPositionInLine());
_ctx->addErrorNode(std::dynamic_pointer_cast<Token>(errToken));
_ctx->addErrorNode(_errorToken.get());
}
}
}
Ref<Token> ParserInterpreter::recoverInline() {
Token* ParserInterpreter::recoverInline() {
return _errHandler->recoverInline(this);
}

3
runtime/Cpp/runtime/src/ParserInterpreter.h
View File

@ -196,10 +196,11 @@ namespace antlr4 {
* tree.
*/
void recover(RecognitionException &e);
Ref<Token> recoverInline();
Token* recoverInline();
private:
const dfa::Vocabulary &_vocabulary;
std::unique_ptr<Token> _errorToken;
};
} // namespace antlr4

12
runtime/Cpp/runtime/src/ParserRuleContext.cpp
View File

@ -80,14 +80,14 @@ void ParserRuleContext::removeLastChild() {
}
}
Ref<tree::TerminalNode> ParserRuleContext::addChild(Ref<Token> const& matchedToken) {
Ref<tree::TerminalNode> ParserRuleContext::addChild(Token *matchedToken) {
Ref<tree::TerminalNodeImpl> t = std::make_shared<tree::TerminalNodeImpl>(matchedToken);
addChild(t);
t->parent = shared_from_this();
return t;
}
Ref<tree::ErrorNode> ParserRuleContext::addErrorNode(Ref<Token> const& badToken) {
Ref<tree::ErrorNode> ParserRuleContext::addErrorNode(Token *badToken) {
Ref<tree::ErrorNodeImpl> t = std::make_shared<tree::ErrorNodeImpl>(badToken);
addChild(t);
t->parent = shared_from_this();
@ -107,7 +107,7 @@ Ref<tree::TerminalNode> ParserRuleContext::getToken(int ttype, std::size_t i) {
for (auto o : children) {
if (is<tree::TerminalNode>(o)) {
Ref<tree::TerminalNode> tnode = std::dynamic_pointer_cast<tree::TerminalNode>(o);
Ref<Token> symbol = tnode->getSymbol();
Token *symbol = tnode->getSymbol();
if (symbol->getType() == ttype) {
if (j++ == i) {
return tnode;
@ -124,7 +124,7 @@ std::vector<Ref<tree::TerminalNode>> ParserRuleContext::getTokens(int ttype) {
for (auto &o : children) {
if (is<tree::TerminalNode>(o)) {
Ref<tree::TerminalNode> tnode = std::dynamic_pointer_cast<tree::TerminalNode>(o);
Ref<Token> symbol = tnode->getSymbol();
Token *symbol = tnode->getSymbol();
if (symbol->getType() == ttype) {
tokens.push_back(tnode);
}
@ -149,11 +149,11 @@ misc::Interval ParserRuleContext::getSourceInterval() {
return misc::Interval(start->getTokenIndex(), stop->getTokenIndex());
}
Ref<Token> ParserRuleContext::getStart() {
Token* ParserRuleContext::getStart() {
return start;
}
Ref<Token> ParserRuleContext::getStop() {
Token* ParserRuleContext::getStop() {
return stop;
}

16
runtime/Cpp/runtime/src/ParserRuleContext.h
View File

@ -91,7 +91,8 @@ namespace antlr4 {
/// </summary>
// public List<Integer> states;
Ref<Token> start, stop;
Token *start;
Token *stop;
/// The exception that forced this rule to return. If the rule successfully
/// completed, this is "null exception pointer".
@ -121,11 +122,12 @@ namespace antlr4 {
/// generic ruleContext object.
virtual void removeLastChild();
virtual Ref<tree::TerminalNode> addChild(Ref<Token> const& matchedToken);
virtual Ref<tree::TerminalNode> addChild(Token *matchedToken);
virtual Ref<tree::ErrorNode> addErrorNode(Token *badToken);
virtual Ref<tree::ErrorNode> addErrorNode(Ref<Token> const& badToken);
std::weak_ptr<ParserRuleContext> getParent() { return std::dynamic_pointer_cast<ParserRuleContext>(getParentReference().lock()); };
std::weak_ptr<ParserRuleContext> getParent() {
return std::dynamic_pointer_cast<ParserRuleContext>(getParentReference().lock());
};
virtual Ref<tree::TerminalNode> getToken(int ttype, std::size_t i);
@ -168,14 +170,14 @@ namespace antlr4 {
* Note that the range from start to stop is inclusive, so for rules that do not consume anything
* (for example, zero length or error productions) this token may exceed stop.
*/
virtual Ref<Token> getStart();
virtual Token*getStart();
/**
* Get the final token in this context.
* Note that the range from start to stop is inclusive, so for rules that do not consume anything
* (for example, zero length or error productions) this token may precede start.
*/
virtual Ref<Token> getStop();
virtual Token* getStop();
/// <summary>
/// Used for rule context info debugging during parse-time, not so much for ATN debugging </summary>

2
runtime/Cpp/runtime/src/ProxyErrorListener.cpp
View File

@ -49,7 +49,7 @@ void ProxyErrorListener::removeErrorListeners() {
_delegates.clear();
}
void ProxyErrorListener::syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line,
void ProxyErrorListener::syntaxError(IRecognizer *recognizer, Token *offendingSymbol, size_t line,
int charPositionInLine, const std::string &msg, std::exception_ptr e) {
for (auto listener : _delegates) {

2
runtime/Cpp/runtime/src/ProxyErrorListener.h
View File

@ -48,7 +48,7 @@ namespace antlr4 {
void removeErrorListener(ANTLRErrorListener *listener);
void removeErrorListeners();
void syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line, int charPositionInLine,
void syntaxError(IRecognizer *recognizer, Token *offendingSymbol, size_t line, int charPositionInLine,
const std::string &msg, std::exception_ptr e) override;
virtual void reportAmbiguity(Parser *recognizer, const dfa::DFA &dfa, size_t startIndex, size_t stopIndex, bool exact,

6
runtime/Cpp/runtime/src/RecognitionException.cpp
View File

@ -40,12 +40,12 @@
using namespace antlr4;
RecognitionException::RecognitionException(IRecognizer *recognizer, IntStream *input,
Ref<ParserRuleContext> const& ctx, Ref<Token> const& offendingToken)
Ref<ParserRuleContext> const& ctx, Token *offendingToken)
: RecognitionException("", recognizer, input, ctx, offendingToken) {
}
RecognitionException::RecognitionException(const std::string &message, IRecognizer *recognizer, IntStream *input,
Ref<ParserRuleContext> const& ctx, Ref<Token> const& offendingToken)
Ref<ParserRuleContext> const& ctx, Token *offendingToken)
: RuntimeException(message), _recognizer(recognizer), _input(input), _ctx(ctx), _offendingToken(offendingToken) {
InitializeInstanceFields();
if (recognizer != nullptr) {
@ -76,7 +76,7 @@ IntStream* RecognitionException::getInputStream() const {
return _input;
}
Ref<Token> RecognitionException::getOffendingToken() const {
Token* RecognitionException::getOffendingToken() const {
return _offendingToken;
}

8
runtime/Cpp/runtime/src/RecognitionException.h
View File

@ -50,15 +50,15 @@ namespace antlr4 {
/// The current Token when an error occurred. Since not all streams
/// support accessing symbols by index, we have to track the Token
/// instance itself.
Ref<Token> _offendingToken;
Token *_offendingToken;
int _offendingState;
public:
RecognitionException(IRecognizer *recognizer, IntStream *input, Ref<ParserRuleContext> const& ctx,
Ref<Token> const& offendingToken = Ref<Token>());
Token *offendingToken = nullptr);
RecognitionException(const std::string &message, IRecognizer *recognizer, IntStream *input,
Ref<ParserRuleContext> const& ctx, Ref<Token> const& offendingToken = Ref<Token>());
Ref<ParserRuleContext> const& ctx, Token *offendingToken = nullptr);
~RecognitionException() {}
/// Get the ATN state number the parser was in at the time the error
@ -104,7 +104,7 @@ namespace antlr4 {
/// available. </returns>
virtual IntStream* getInputStream() const;
virtual Ref<Token> getOffendingToken() const;
virtual Token* getOffendingToken() const;
/// <summary>
/// Gets the <seealso cref="Recognizer"/> where this exception occurred.

4
runtime/Cpp/runtime/src/TokenFactory.h
View File

@ -44,11 +44,11 @@ namespace antlr4 {
/// This is the method used to create tokens in the lexer and in the
/// error handling strategy. If text!=null, than the start and stop positions
/// are wiped to -1 in the text override is set in the CommonToken.
virtual Ref<Symbol> create(std::pair<TokenSource *, CharStream *> source, int type, const std::string &text,
virtual std::unique_ptr<Symbol> create(std::pair<TokenSource *, CharStream *> source, int type, const std::string &text,
int channel, int start, int stop, int line, int charPositionInLine) = 0;
/// Generically useful
virtual Ref<Symbol> create(int type, const std::string &text) = 0;
virtual std::unique_ptr<Symbol> create(int type, const std::string &text) = 0;
};
} // namespace antlr4

2
runtime/Cpp/runtime/src/TokenSource.h
View File

@ -58,7 +58,7 @@ namespace antlr4 {
/// <seealso cref="CharStream"/>). Do not fail/return upon lexing error; keep chewing
/// on the characters until you get a good one; errors are not passed through
/// to the parser.
virtual Ref<Token> nextToken() = 0;
virtual std::unique_ptr<Token> nextToken() = 0;
/// <summary>
/// Get the line number for the current position in the input stream. The

6
runtime/Cpp/runtime/src/TokenStream.h
View File

@ -50,7 +50,7 @@ namespace antlr4 {
public:
virtual ~TokenStream();
virtual Ref<Token> LT(ssize_t k) = 0;
virtual Token* LT(ssize_t k) = 0;
/// <summary>
/// Gets the <seealso cref="Token"/> at the specified {@code index} in the stream. When
@ -69,7 +69,7 @@ namespace antlr4 {
/// <exception cref="IllegalArgumentException"> if {code index} is less than 0 </exception>
/// <exception cref="UnsupportedOperationException"> if the stream does not support
/// retrieving the token at the specified index </exception>
virtual Ref<Token> get(size_t index) const = 0;
virtual Token* get(size_t index) const = 0;
/// Gets the underlying TokenSource which provides tokens for this stream.
virtual TokenSource* getTokenSource() const = 0;
@ -157,7 +157,7 @@ namespace antlr4 {
/// </returns>
/// <exception cref="UnsupportedOperationException"> if this stream does not support
/// this method for the specified tokens </exception>
virtual std::string getText(Ref<Token> const& start, Ref<Token> const& stop) = 0;
virtual std::string getText(Token *start, Token *stop) = 0;
};
} // namespace antlr4

2
runtime/Cpp/runtime/src/TokenStreamRewriter.cpp
View File

@ -302,7 +302,7 @@ std::string TokenStreamRewriter::getText(const std::string &programName, const I
while (i <= (size_t)stop && i < tokens->size()) {
RewriteOperation *op = indexToOp[i];
indexToOp.erase(i); // remove so any left have index size-1
Ref<Token> t = tokens->get(i);
Token *t = tokens->get(i);
if (op == nullptr) {
// no operation at that index, just dump token
if (t->getType() != Token::EOF) {

34
runtime/Cpp/runtime/src/UnbufferedTokenStream.cpp
View File

@ -45,7 +45,8 @@ using namespace antlr4;
UnbufferedTokenStream::UnbufferedTokenStream(TokenSource *tokenSource) : UnbufferedTokenStream(tokenSource, 256) {
}
UnbufferedTokenStream::UnbufferedTokenStream(TokenSource *tokenSource, int /*bufferSize*/) : _tokenSource(tokenSource)
UnbufferedTokenStream::UnbufferedTokenStream(TokenSource *tokenSource, int /*bufferSize*/)
: _tokenSource(tokenSource), _lastToken(nullptr), _lastTokenBufferStart(nullptr)
{
InitializeInstanceFields();
fill(1); // prime the pump
@ -54,17 +55,17 @@ UnbufferedTokenStream::UnbufferedTokenStream(TokenSource *tokenSource, int /*buf
UnbufferedTokenStream::~UnbufferedTokenStream() {
}
Ref<Token> UnbufferedTokenStream::get(size_t i) const
Token* UnbufferedTokenStream::get(size_t i) const
{ // get absolute index
size_t bufferStartIndex = getBufferStartIndex();
if (i < bufferStartIndex || i >= bufferStartIndex + _tokens.size()) {
throw IndexOutOfBoundsException(std::string("get(") + std::to_string(i) + std::string(") outside buffer: ")
+ std::to_string(bufferStartIndex) + std::string("..") + std::to_string(bufferStartIndex + _tokens.size()));
}
return _tokens[i - bufferStartIndex];
return _tokens[i - bufferStartIndex].get();
}
Ref<Token> UnbufferedTokenStream::LT(ssize_t i)
Token* UnbufferedTokenStream::LT(ssize_t i)
{
if (i == -1) {
return _lastToken;
@ -78,10 +79,10 @@ Ref<Token> UnbufferedTokenStream::LT(ssize_t i)
if (index >= (ssize_t)_tokens.size()) {
assert(_tokens.size() > 0 && _tokens.back()->getType() == EOF);
return _tokens.back();
return _tokens.back().get();
}
return _tokens[(size_t)index];
return _tokens[(size_t)index].get();
}
ssize_t UnbufferedTokenStream::LA(ssize_t i)
@ -104,7 +105,7 @@ std::string UnbufferedTokenStream::getText(RuleContext* ctx)
return getText(ctx->getSourceInterval());
}
std::string UnbufferedTokenStream::getText(Ref<Token> const& start, Ref<Token> const& stop)
std::string UnbufferedTokenStream::getText(Token *start, Token *stop)
{
return getText(misc::Interval(start->getTokenIndex(), stop->getTokenIndex()));
}
@ -116,7 +117,7 @@ void UnbufferedTokenStream::consume()
}
// buf always has at least tokens[p==0] in this method due to ctor
_lastToken = _tokens[_p]; // track last token for LT(-1)
_lastToken = _tokens[_p].get(); // track last token for LT(-1)
// if we're at last token and no markers, opportunity to flush buffer
if (_p == _tokens.size() - 1 && _numMarkers == 0) {
@ -156,21 +157,20 @@ size_t UnbufferedTokenStream::fill(size_t n)
return i;
}
Ref<Token> t = _tokenSource->nextToken();
add(t);
add(_tokenSource->nextToken());
}
return n;
}
void UnbufferedTokenStream::add(Ref<Token> t)
void UnbufferedTokenStream::add(std::unique_ptr<Token> t)
{
Ref<WritableToken> writable = std::dynamic_pointer_cast<WritableToken>(t);
if (writable) {
WritableToken *writable = dynamic_cast<WritableToken *>(t.get());
if (writable != nullptr) {
writable->setTokenIndex(int(getBufferStartIndex() + _tokens.size()));
}
_tokens.push_back(t);
_tokens.push_back(std::move(t));
}
/// <summary>
@ -203,7 +203,7 @@ void UnbufferedTokenStream::release(ssize_t marker)
if (_p > 0) {
// Copy tokens[p]..tokens[n-1] to tokens[0]..tokens[(n-1)-p], reset ptrs
// p is last valid token; move nothing if p==n as we have no valid char
std::vector<Ref<Token>>(_tokens.begin() + (ssize_t)_p, _tokens.end()).swap(_tokens);
_tokens.erase(_tokens.begin(), _tokens.begin() + (ssize_t)_p);
_p = 0;
}
@ -243,7 +243,7 @@ void UnbufferedTokenStream::seek(size_t index)
if (_p == 0) {
_lastToken = _lastTokenBufferStart;
} else {
_lastToken = _tokens[_p - 1];
_lastToken = _tokens[_p - 1].get();
}
}
@ -274,7 +274,7 @@ std::string UnbufferedTokenStream::getText(const misc::Interval &interval)
std::stringstream ss;
for (size_t i = a; i <= b; i++) {
Ref<Token> t = _tokens[i];
Token *t = _tokens[i].get();
if (i > 0)
ss << ", ";
ss << t->getText();

14
runtime/Cpp/runtime/src/UnbufferedTokenStream.h
View File

@ -41,8 +41,8 @@ namespace antlr4 {
UnbufferedTokenStream(TokenSource *tokenSource, int bufferSize);
virtual ~UnbufferedTokenStream();
virtual Ref<Token> get(size_t i) const override;
virtual Ref<Token> LT(ssize_t i) override;
virtual Token* get(size_t i) const override;
virtual Token* LT(ssize_t i) override;
virtual ssize_t LA(ssize_t i) override;
virtual TokenSource* getTokenSource() const override;
@ -50,7 +50,7 @@ namespace antlr4 {
virtual std::string getText(const misc::Interval &interval) override;
virtual std::string getText() override;
virtual std::string getText(RuleContext *ctx) override;
virtual std::string getText(Ref<Token> const& start, Ref<Token> const& stop) override;
virtual std::string getText(Token *start, Token *stop) override;
virtual void consume() override;
@ -80,7 +80,7 @@ namespace antlr4 {
/// we start filling at index 0 again.
/// </summary>
std::vector<Ref<Token>> _tokens;
std::vector<std::unique_ptr<Token>> _tokens;
/// <summary>
/// 0..n-1 index into <seealso cref="#tokens tokens"/> of next token.
@ -101,13 +101,13 @@ namespace antlr4 {
/// <summary>
/// This is the {@code LT(-1)} token for the current position.
/// </summary>
Ref<Token> _lastToken;
Token *_lastToken;
/// <summary>
/// When {@code numMarkers > 0}, this is the {@code LT(-1)} token for the
/// first token in <seealso cref="#tokens"/>. Otherwise, this is {@code null}.
/// </summary>
Ref<Token> _lastTokenBufferStart;
Token *_lastTokenBufferStart;
/// <summary>
/// Absolute token index. It's the index of the token about to be read via
@ -127,7 +127,7 @@ namespace antlr4 {
/// then EOF was reached before {@code n} tokens could be added.
/// </summary>
virtual size_t fill(size_t n);
virtual void add(Ref<Token> t);
virtual void add(std::unique_ptr<Token> t);
size_t getBufferStartIndex() const;

6
runtime/Cpp/runtime/src/support/CPPUtils.h
View File

@ -62,12 +62,6 @@ namespace antlrcpp {
ANTLR4CPP_PUBLIC FinalAction finally(std::function<void ()> f);
// Convenience functions to avoid lengthy dynamic_cast() != nullptr checks in many places.
/*
template <typename T1, typename T2>
inline bool is(T2 &obj) { // For value types.
return dynamic_cast<typename std::add_const<T1>::type *>(&obj) != nullptr;
}
*/
template <typename T1, typename T2>
inline bool is(T2 *obj) { // For pointer types.
return dynamic_cast<typename std::add_const<T1>::type>(obj) != nullptr;

2
runtime/Cpp/runtime/src/tree/ErrorNodeImpl.cpp
View File

@ -37,5 +37,5 @@ using namespace antlr4;
using namespace antlr4::misc;
using namespace antlr4::tree;
ErrorNodeImpl::ErrorNodeImpl(Ref<Token> const& token) : TerminalNodeImpl(token) {
ErrorNodeImpl::ErrorNodeImpl(Token *token) : TerminalNodeImpl(token) {
}

2
runtime/Cpp/runtime/src/tree/ErrorNodeImpl.h
View File

@ -47,7 +47,7 @@ namespace tree {
/// </summary>
class ANTLR4CPP_PUBLIC ErrorNodeImpl : public virtual TerminalNodeImpl, public virtual ErrorNode {
public:
ErrorNodeImpl(Ref<Token> const& token);
ErrorNodeImpl(Token *token);
template<typename T, typename T1>
T accept(ParseTreeVisitor<T1> *visitor) {

2
runtime/Cpp/runtime/src/tree/TerminalNode.h
View File

@ -38,7 +38,7 @@ namespace tree {
class ANTLR4CPP_PUBLIC TerminalNode : public ParseTree {
public:
virtual Ref<Token> getSymbol() = 0;
virtual Token* getSymbol() = 0;
};
} // namespace tree

5
runtime/Cpp/runtime/src/tree/TerminalNodeImpl.cpp
View File

@ -37,11 +37,10 @@
using namespace antlr4;
using namespace antlr4::tree;
TerminalNodeImpl::TerminalNodeImpl(Ref<Token> const& symbol) {
this->symbol = symbol;
TerminalNodeImpl::TerminalNodeImpl(Token *symbol_) : symbol(symbol_) {
}
Ref<Token> TerminalNodeImpl::getSymbol() {
Token* TerminalNodeImpl::getSymbol() {
return symbol;
}

6
runtime/Cpp/runtime/src/tree/TerminalNodeImpl.h
View File

@ -38,12 +38,12 @@ namespace tree {
class ANTLR4CPP_PUBLIC TerminalNodeImpl : public virtual TerminalNode {
public:
Ref<Token> symbol;
Token *symbol;
std::weak_ptr<ParseTree> parent;
TerminalNodeImpl(Ref<Token> const& symbol);
TerminalNodeImpl(Token *symbol);
virtual Ref<Token> getSymbol() override;
virtual Token* getSymbol() override;
virtual misc::Interval getSourceInterval() override;
virtual std::size_t getChildCount() override;

20
runtime/Cpp/runtime/src/tree/Trees.cpp
View File

@ -133,7 +133,7 @@ std::string Trees::getNodeText(Ref<Tree> const& t, const std::vector<std::string
} else if (is<ErrorNode>(t)) {
return t->toString();
} else if (is<TerminalNode>(t)) {
Ref<Token> symbol = (std::static_pointer_cast<TerminalNode>(t))->getSymbol();
Token *symbol = (std::static_pointer_cast<TerminalNode>(t))->getSymbol();
if (symbol != nullptr) {
std::string s = symbol->getText();
return s;
@ -258,24 +258,6 @@ Ref<ParserRuleContext> Trees::getRootOfSubtreeEnclosingRegion(Ref<ParseTree> con
return nullptr;
}
void Trees::stripChildrenOutOfRange(Ref<ParserRuleContext> const& t, Ref<ParserRuleContext> const& root,
size_t startIndex, size_t stopIndex) {
if (t == nullptr) {
return;
}
for (size_t i = 0; i < t->getChildCount(); ++i) {
Ref<ParseTree> child = t->getChild(i);
Interval range = child->getSourceInterval();
if (is<ParserRuleContext>(child) && (range.b < (int)startIndex || range.a > (int)stopIndex)) {
if (isAncestorOf(child, root)) { // replace only if subtree doesn't have displayed root
Ref<CommonToken> abbrev = std::make_shared<CommonToken>((int)Token::INVALID_TYPE, "...");
t->children[i] = std::make_shared<TerminalNodeImpl>(abbrev);
}
}
}
}
Ref<Tree> Trees::findNodeSuchThat(Ref<Tree> const& t, Ref<Predicate<Tree>> const& pred) {
if (pred->test(t)) {
return t;

11
runtime/Cpp/runtime/src/tree/Trees.h
View File

@ -93,17 +93,6 @@ namespace tree {
size_t startTokenIndex, // inclusive
size_t stopTokenIndex); // inclusive
/** Replace any subtree siblings of root that are completely to left
* or right of lookahead range with a CommonToken(Token.INVALID_TYPE,"...")
* node. The source interval for t is not altered to suit smaller range!
*
* WARNING: destructive to t.
*
* @since 4.5.1
*/
static void stripChildrenOutOfRange(Ref<ParserRuleContext> const& t, Ref<ParserRuleContext> const& root,
size_t startIndex, size_t stopIndex);
/** Return first node satisfying the pred
*
* @since 4.5.1

41
runtime/Cpp/runtime/src/tree/pattern/ParseTreePatternMatcher.cpp
View File

@ -101,17 +101,11 @@ ParseTreeMatch ParseTreePatternMatcher::match(Ref<ParseTree> const& tree, const
}
ParseTreePattern ParseTreePatternMatcher::compile(const std::string &pattern, int patternRuleIndex) {
std::vector<Ref<Token>> tokenList = tokenize(pattern);
ListTokenSource *tokenSrc = new ListTokenSource(tokenList); /* mem-check: deleted in finally block */
CommonTokenStream *tokens = new CommonTokenStream(tokenSrc); /* mem-check: deleted in finally block */
auto onExit = finally([tokenSrc, tokens]() {
delete tokenSrc;
delete tokens;
});
ListTokenSource tokenSrc(std::move(tokenize(pattern)));
CommonTokenStream tokens(&tokenSrc);
ParserInterpreter parserInterp(_parser->getGrammarFileName(), _parser->getVocabulary(),
_parser->getRuleNames(), _parser->getATNWithBypassAlts(), tokens);
_parser->getRuleNames(), _parser->getATNWithBypassAlts(), &tokens);
Ref<ParserRuleContext> tree;
try {
@ -136,7 +130,7 @@ ParseTreePattern ParseTreePatternMatcher::compile(const std::string &pattern, in
}
// Make sure tree pattern compilation checks for a complete parse
if (tokens->LA(1) != Token::EOF) {
if (tokens.LA(1) != Token::EOF) {
throw StartRuleDoesNotConsumeFullPattern();
}
@ -169,8 +163,8 @@ Ref<ParseTree> ParseTreePatternMatcher::matchImpl(Ref<ParseTree> const& tree,
Ref<ParseTree> mismatchedNode;
// both are tokens and they have same type
if (t1->getSymbol()->getType() == t2->getSymbol()->getType()) {
if (is<TokenTagToken>(t2->getSymbol())) { // x and <ID>
Ref<TokenTagToken> tokenTagToken = std::dynamic_pointer_cast<TokenTagToken>(t2->getSymbol());
if (is<TokenTagToken *>(t2->getSymbol())) { // x and <ID>
TokenTagToken *tokenTagToken = dynamic_cast<TokenTagToken *>(t2->getSymbol());
// track label->list-of-nodes for both token name and label (if any)
labels[tokenTagToken->getTokenName()].push_back(tree);
@ -200,7 +194,7 @@ Ref<ParseTree> ParseTreePatternMatcher::matchImpl(Ref<ParseTree> const& tree,
Ref<ParseTree> mismatchedNode;
// (expr ...) and <expr>
Ref<RuleTagToken> ruleTagToken = getRuleTagToken(r2);
RuleTagToken *ruleTagToken = getRuleTagToken(r2);
if (ruleTagToken != nullptr) {
//ParseTreeMatch *m = nullptr; // unused?
if (r1->RuleContext::getRuleContext()->getRuleIndex() == r2->RuleContext::getRuleContext()->getRuleIndex()) {
@ -242,25 +236,25 @@ Ref<ParseTree> ParseTreePatternMatcher::matchImpl(Ref<ParseTree> const& tree,
return tree;
}
Ref<RuleTagToken> ParseTreePatternMatcher::getRuleTagToken(Ref<ParseTree> const& t) {
RuleTagToken* ParseTreePatternMatcher::getRuleTagToken(Ref<ParseTree> const& t) {
if (is<RuleNode>(t)) {
Ref<RuleNode> r = std::dynamic_pointer_cast<RuleNode>(t);
if (r->getChildCount() == 1 && is<TerminalNode>(r->getChild(0))) {
Ref<TerminalNode> c = std::dynamic_pointer_cast<TerminalNode>(r->getChild(0));
if (is<RuleTagToken>(c->getSymbol())) {
return std::dynamic_pointer_cast<RuleTagToken>(c->getSymbol());
if (is<RuleTagToken *>(c->getSymbol())) {
return dynamic_cast<RuleTagToken *>(c->getSymbol());
}
}
}
return nullptr;
}
std::vector<Ref<Token>> ParseTreePatternMatcher::tokenize(const std::string &pattern) {
std::vector<std::unique_ptr<Token>> ParseTreePatternMatcher::tokenize(const std::string &pattern) {
// split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)
std::vector<Chunk> chunks = split(pattern);
// create token stream from text and tags
std::vector<Ref<Token>> tokens;
std::vector<std::unique_ptr<Token>> tokens;
for (auto chunk : chunks) {
if (is<TagChunk *>(&chunk)) {
TagChunk &tagChunk = (TagChunk&)chunk;
@ -270,15 +264,14 @@ std::vector<Ref<Token>> ParseTreePatternMatcher::tokenize(const std::string &pat
if (ttype == Token::INVALID_TYPE) {
throw IllegalArgumentException("Unknown token " + tagChunk.getTag() + " in pattern: " + pattern);
}
Ref<TokenTagToken> t = std::make_shared<TokenTagToken>(tagChunk.getTag(), (int)ttype, tagChunk.getLabel());
tokens.push_back(t);
tokens.emplace_back(new TokenTagToken(tagChunk.getTag(), (int)ttype, tagChunk.getLabel()));
} else if (islower(tagChunk.getTag()[0])) {
ssize_t ruleIndex = _parser->getRuleIndex(tagChunk.getTag());
if (ruleIndex == -1) {
throw IllegalArgumentException("Unknown rule " + tagChunk.getTag() + " in pattern: " + pattern);
}
int ruleImaginaryTokenType = _parser->getATNWithBypassAlts().ruleToTokenType[(size_t)ruleIndex];
tokens.push_back(std::make_shared<RuleTagToken>(tagChunk.getTag(), ruleImaginaryTokenType, tagChunk.getLabel()));
tokens.emplace_back(new RuleTagToken(tagChunk.getTag(), ruleImaginaryTokenType, tagChunk.getLabel()));
} else {
throw IllegalArgumentException("invalid tag: " + tagChunk.getTag() + " in pattern: " + pattern);
}
@ -286,9 +279,9 @@ std::vector<Ref<Token>> ParseTreePatternMatcher::tokenize(const std::string &pat
TextChunk &textChunk = (TextChunk&)chunk;
ANTLRInputStream input(textChunk.getText());
_lexer->setInputStream(&input);
Ref<Token> t = _lexer->nextToken();
std::unique_ptr<Token> t(_lexer->nextToken());
while (t->getType() != Token::EOF) {
tokens.push_back(t);
tokens.push_back(std::move(t));
t = _lexer->nextToken();
}
_lexer->setInputStream(nullptr);
@ -347,6 +340,7 @@ std::vector<Chunk> ParseTreePatternMatcher::split(const std::string &pattern) {
std::string text = pattern.substr(0, starts[0]);
chunks.push_back(TextChunk(text));
}
for (size_t i = 0; i < ntags; i++) {
// copy inside of <tag>
std::string tag = pattern.substr(starts[i] + _start.length(), stops[i] - (starts[i] + _start.length()));
@ -364,6 +358,7 @@ std::vector<Chunk> ParseTreePatternMatcher::split(const std::string &pattern) {
chunks.push_back(TextChunk(text));
}
}
if (ntags > 0) {
size_t afterLastTag = stops[ntags - 1] + _stop.length();
if (afterLastTag < n) { // copy text from end of last tag to end

4
runtime/Cpp/runtime/src/tree/pattern/ParseTreePatternMatcher.h
View File

@ -170,7 +170,7 @@ namespace pattern {
// ---- SUPPORT CODE ----
virtual std::vector<Ref<Token>> tokenize(const std::string &pattern);
virtual std::vector<std::unique_ptr<Token>> tokenize(const std::string &pattern);
/// Split "<ID> = <e:expr>;" into 4 chunks for tokenizing by tokenize().
virtual std::vector<Chunk> split(const std::string &pattern);
@ -191,7 +191,7 @@ namespace pattern {
Ref<ParseTree> const& patternTree, std::map<std::string, std::vector<Ref<ParseTree>>> &labels);
/// Is t <expr> subtree?
virtual Ref<RuleTagToken> getRuleTagToken(Ref<ParseTree> const& t);
virtual RuleTagToken* getRuleTagToken(Ref<ParseTree> const& t);
private:
Lexer *_lexer;

8
runtime/Cpp/runtime/src/tree/xpath/XPath.cpp
View File

@ -71,14 +71,14 @@ std::vector<XPathElement> XPath::split(const std::string &path) {
throw IllegalArgumentException(msg);
}
std::vector<Ref<Token>> tokens = tokenStream.getTokens();
std::vector<Token *> tokens = tokenStream.getTokens();
std::vector<XPathElement> elements;
size_t n = tokens.size();
size_t i = 0;
bool done = false;
while (!done && i < n) {
Ref<Token> el = tokens[i];
Ref<Token> next = nullptr;
Token *el = tokens[i];
Token *next = nullptr;
switch (el->getType()) {
case XPathLexer::ROOT:
case XPathLexer::ANYWHERE: {
@ -116,7 +116,7 @@ std::vector<XPathElement> XPath::split(const std::string &path) {
return elements;
}
XPathElement XPath::getXPathElement(const Ref<Token> &wordToken, bool anywhere) {
XPathElement XPath::getXPathElement(Token *wordToken, bool anywhere) {
if (wordToken->getType() == Token::EOF) {
throw IllegalArgumentException("Missing path element at end of path");
}

2
runtime/Cpp/runtime/src/tree/xpath/XPath.h
View File

@ -105,7 +105,7 @@ namespace xpath {
/// Convert word like {@code *} or {@code ID} or {@code expr} to a path
/// element. {@code anywhere} is {@code true} if {@code //} precedes the
/// word.
virtual XPathElement getXPathElement(const std::shared_ptr<Token> &wordToken, bool anywhere);
virtual XPathElement getXPathElement(Token *wordToken, bool anywhere);
};
} // namespace xpath

2
runtime/Cpp/runtime/src/tree/xpath/XPathLexerErrorListener.cpp
View File

@ -34,6 +34,6 @@
using namespace antlr4;
using namespace antlr4::tree::xpath;
void XPathLexerErrorListener::syntaxError(IRecognizer * /*recognizer*/, Ref<Token> const& /*offendingSymbol*/,
void XPathLexerErrorListener::syntaxError(IRecognizer * /*recognizer*/, Token * /*offendingSymbol*/,
size_t /*line*/, int /*charPositionInLine*/, const std::string &/*msg*/, std::exception_ptr /*e*/) {
}

2
runtime/Cpp/runtime/src/tree/xpath/XPathLexerErrorListener.h
View File

@ -39,7 +39,7 @@ namespace xpath {
class ANTLR4CPP_PUBLIC XPathLexerErrorListener : public BaseErrorListener {
public:
virtual void syntaxError(IRecognizer *recognizer, Ref<Token> const& offendingSymbol, size_t line,
virtual void syntaxError(IRecognizer *recognizer, Token *offendingSymbol, size_t line,
int charPositionInLine, const std::string &msg, std::exception_ptr e) override;
};

146
runtime/Cpp/test/M.cpp Normal file
View File

@ -0,0 +1,146 @@
// Generated from /var/folders/fv/13zbdzw17cdczn_rbnt5m_140000gn/T/TestCompositeLexers-1465807145492/M.g4 by ANTLR 4.5.3
#include "M.h"
using namespace antlr4;
M::M(CharStream *input) : Lexer(input) {
_interpreter = new atn::LexerATNSimulator(this, _atn, _decisionToDFA, _sharedContextCache);
}
M::~M() {
delete _interpreter;
}
std::string M::getGrammarFileName() const {
return "M.g4";
}
const std::vector<std::string>& M::getRuleNames() const {
return _ruleNames;
}
const std::vector<std::string>& M::getModeNames() const {
return _modeNames;
}
const std::vector<std::string>& M::getTokenNames() const {
return _tokenNames;
}
dfa::Vocabulary& M::getVocabulary() const {
return _vocabulary;
}
const std::vector<uint16_t> M::getSerializedATN() const {
return _serializedATN;
}
const atn::ATN& M::getATN() const {
return _atn;
}
void M::action(Ref<RuleContext> const& context, int ruleIndex, int actionIndex) {
switch (ruleIndex) {
case 0: AAction(std::dynamic_pointer_cast<RuleContext>(context), actionIndex); break;
case 2: BAction(std::dynamic_pointer_cast<RuleContext>(context), actionIndex); break;
default:
break;
}
}
void M::AAction(Ref<RuleContext> const& context, int actionIndex) {
switch (actionIndex) {
case 0: std::cout << "M.A" << std::endl; break;
default:
break;
}
}
void M::BAction(Ref<RuleContext> const& context, int actionIndex) {
switch (actionIndex) {
case 1: std::cout << "S.B" << std::endl; break;
default:
break;
}
}
// Static vars and initialization.
std::vector<dfa::DFA> M::_decisionToDFA;
atn::PredictionContextCache M::_sharedContextCache;
// We own the ATN which in turn owns the ATN states.
atn::ATN M::_atn;
std::vector<uint16_t> M::_serializedATN;
std::vector<std::string> M::_ruleNames = {
"A", "WS", "B"
};
std::vector<std::string> M::_modeNames = {
"DEFAULT_MODE"
};
std::vector<std::string> M::_literalNames = {
"", "", "", "'b'"
};
std::vector<std::string> M::_symbolicNames = {
"", "A", "WS", "B"
};
dfa::Vocabulary M::_vocabulary(_literalNames, _symbolicNames);
std::vector<std::string> M::_tokenNames;
M::Initializer::Initializer() {
// This code could be in a static initializer lambda, but VS doesn't allow access to private class members from there.
for (size_t i = 0; i < _symbolicNames.size(); ++i) {
std::string name = _vocabulary.getLiteralName(i);
if (name.empty()) {
name = _vocabulary.getSymbolicName(i);
}
if (name.empty()) {
_tokenNames.push_back("<INVALID>");
} else {
_tokenNames.push_back(name);
}
}
_serializedATN = {
0x3, 0x430, 0xd6d1, 0x8206, 0xad2d, 0x4417, 0xaef1, 0x8d80, 0xaadd,
0x2, 0x5, 0x14, 0x8, 0x1, 0x4, 0x2, 0x9, 0x2, 0x4, 0x3, 0x9, 0x3, 0x4,
0x4, 0x9, 0x4, 0x3, 0x2, 0x3, 0x2, 0x3, 0x2, 0x3, 0x2, 0x3, 0x3, 0x3,
0x3, 0x3, 0x3, 0x3, 0x3, 0x3, 0x4, 0x3, 0x4, 0x3, 0x4, 0x2, 0x2, 0x5,
0x3, 0x3, 0x5, 0x4, 0x7, 0x5, 0x3, 0x2, 0x3, 0x4, 0x2, 0xc, 0xc, 0x22,
0x22, 0x13, 0x2, 0x3, 0x3, 0x2, 0x2, 0x2, 0x2, 0x5, 0x3, 0x2, 0x2, 0x2,
0x2, 0x7, 0x3, 0x2, 0x2, 0x2, 0x3, 0x9, 0x3, 0x2, 0x2, 0x2, 0x5, 0xd,
0x3, 0x2, 0x2, 0x2, 0x7, 0x11, 0x3, 0x2, 0x2, 0x2, 0x9, 0xa, 0x7, 0x63,
0x2, 0x2, 0xa, 0xb, 0x5, 0x7, 0x4, 0x2, 0xb, 0xc, 0x8, 0x2, 0x2, 0x2,
0xc, 0x4, 0x3, 0x2, 0x2, 0x2, 0xd, 0xe, 0x9, 0x2, 0x2, 0x2, 0xe, 0xf,
0x3, 0x2, 0x2, 0x2, 0xf, 0x10, 0x8, 0x3, 0x3, 0x2, 0x10, 0x6, 0x3, 0x2,
0x2, 0x2, 0x11, 0x12, 0x7, 0x64, 0x2, 0x2, 0x12, 0x13, 0x8, 0x4, 0x4,
0x2, 0x13, 0x8, 0x3, 0x2, 0x2, 0x2, 0x3, 0x2, 0x5, 0x3, 0x2, 0x2, 0x8,
0x2, 0x2, 0x3, 0x4, 0x3,
};
atn::ATNDeserializer deserializer;
_atn = deserializer.deserialize(_serializedATN);
for (int i = 0; i < _atn.getNumberOfDecisions(); i++) {
_decisionToDFA.push_back(dfa::DFA(_atn.getDecisionState(i), i));
}
}
M::Initializer M::_init;

60
runtime/Cpp/test/M.h Normal file
View File

@ -0,0 +1,60 @@
// Generated from /var/folders/fv/13zbdzw17cdczn_rbnt5m_140000gn/T/TestCompositeLexers-1465807145492/M.g4 by ANTLR 4.5.3
#pragma once
#include "antlr4-runtime.h"
using namespace antlr4;
class M : public Lexer {
public:
enum {
A = 1, WS = 2, B = 3
};
M(CharStream *input);
~M();
virtual std::string getGrammarFileName() const override;
virtual const std::vector<std::string>& getRuleNames() const override;
virtual const std::vector<std::string>& getModeNames() const override;
virtual const std::vector<std::string>& getTokenNames() const override; // deprecated, use vocabulary instead
virtual dfa::Vocabulary& getVocabulary() const override;
virtual const std::vector<uint16_t> getSerializedATN() const;
virtual const atn::ATN& getATN() const override;
virtual void action(Ref<RuleContext> const& context, int ruleIndex, int actionIndex) override;
private:
static std::vector<dfa::DFA> _decisionToDFA;
static atn::PredictionContextCache _sharedContextCache;
static std::vector<std::string> _ruleNames;
static std::vector<std::string> _tokenNames;
static std::vector<std::string> _modeNames;
static std::vector<std::string> _literalNames;
static std::vector<std::string> _symbolicNames;
static dfa::Vocabulary _vocabulary;
static atn::ATN _atn;
static std::vector<uint16_t> _serializedATN;
// Individual action functions triggered by action() above.
void AAction(Ref<RuleContext> const& context, int actionIndex);
void BAction(Ref<RuleContext> const& context, int actionIndex);
// Individual semantic predicate functions triggered by sempred() above.
struct Initializer {
Initializer();
};
static Initializer _init;
};

6
tool/resources/org/antlr/v4/tool/templates/codegen/Cpp/Cpp.stg
View File

@ -941,7 +941,7 @@ AddToLabelList(a) ::= <<
<ctx(a.label)>-><a.listName>.push_back(<labelref(a.label)>);
>>
TokenLabelType() ::= "Ref\<<file.TokenLabelType; null = {Token}>>"
TokenLabelType() ::= "<file.TokenLabelType; null = {Token}>*"
TokenDeclHeader(t) ::= "<TokenLabelType()> <t.name>;"
TokenDecl(t) ::= "<! Variable Declaration !>"
@ -949,7 +949,7 @@ TokenDecl(t) ::= "<! Variable Declaration !>"
TokenTypeDeclHeader(t) ::= "<! Local Variable !>"
TokenTypeDecl(t) ::= "ssize_t <t.name>;"
TokenListDeclHeader(t) ::= "std::vector\<Ref\<Token>> <t.name>;"
TokenListDeclHeader(t) ::= "std::vector\<Token *> <t.name>;"
TokenListDecl(t) ::= "<! Variable Declaration !>"
RuleContextDeclHeader(r) ::= "Ref\<<parser.name>::<r.ctxName>> <r.name>;"
@ -1058,7 +1058,7 @@ labelref(x) ::= "<if (!x.isLocal)>std::dynamic_pointer_cast\<<x.ctx.name>>(_loca
ctx(actionChunk) ::= "std::dynamic_pointer_cast\<<actionChunk.ctx.name>>(_localctx)"
// used for left-recursive rules
recRuleAltPredicate(ruleName,opPrec) ::= "precpred(_ctx.get(), <opPrec>)"
recRuleAltPredicate(ruleName,opPrec) ::= "precpred(_ctx, <opPrec>)"
recRuleSetReturnAction(src,name) ::= "recRuleSetReturnAction(src,name) $<name>=$<src>.<name>;"
recRuleSetStopToken() ::= "_ctx->stop = _input->LT(-1);"