forked from jasder/antlr
Merge branch 'master' into LR-loop-fix
This commit is contained in:
Mike Lischke
commit
d324e9a46b
|
|
@ -1,5 +1,5 @@
|
|||
[cut]
|
||||
Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors most "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership.
|
||||
Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors must "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership.
|
||||
|
||||
Make sure you are already in the contributors.txt file or add a commit to this pull request with the appropriate change. Thanks!
|
||||
[/cut]
|
||||
|
|
|
|||
|
|
@ -6,6 +6,8 @@
|
|||
|
||||
*Given day-job constraints, my time working on this project is limited so I'll have to focus first on fixing bugs rather than changing/improving the feature set. Likely I'll do it in bursts every few months. Please do not be offended if your bug or pull request does not yield a response! --parrt*
|
||||
|
||||
[](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BF92STRXT8F8Q)
|
||||
|
||||
## Authors and major contributors
|
||||
|
||||
* [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-maven-plugin</artifactId>
|
||||
<packaging>maven-plugin</packaging>
|
||||
|
|
|
|||
|
|
@ -174,13 +174,20 @@ At this point, you should have an editor which displays an error icon next to th
|
|||
|
||||
What remains to be done is have our validate function actually validate the input. Finally ANTLR comes in the picture!
|
||||
|
||||
To start with, let's load ANTLR and your parser, listener etc.. Easy, since you could write:
|
||||
To start with, let's load ANTLR and your parser, listener etc..
|
||||
|
||||
The preferred approach for loading parser code is to bundle your parser, [as described here](javascript-target.md).
|
||||
You can then load it as part of the importScripts instruction at the start of your worker code.
|
||||
|
||||
Another approach is to load it using 'require'. Easy, since you could write:
|
||||
|
||||
```js
|
||||
var antlr4 = require('antlr4/index');
|
||||
```
|
||||
|
||||
This may work, but it's actually unreliable. The reason is that the require function used by ANTLR, which exactly mimics the NodeJS require function, uses a different syntax than the require function that comes with ACE. So we need to bring in a require function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find here. But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows:
|
||||
This may work, but it's actually unreliable. The reason is that the 'require' function that comes with ACE uses a different syntax than the 'require' function used by ANTLR, which follows the NodeJS 'require' convention.
|
||||
So we need to bring in a NodeJS compatible 'require' function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find in li/require.js.
|
||||
But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows:
|
||||
|
||||
```js
|
||||
// load nodejs compatible require
|
||||
|
|
@ -191,7 +198,8 @@ importScripts("../lib/require.js");
|
|||
var antlr4_require = require;
|
||||
require = ace_require;
|
||||
```
|
||||
Now it's safe to load antlr, and the parsers generated for your language. Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows:
|
||||
Now it's safe to load antlr and the parsers generated for your language.
|
||||
Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows:
|
||||
```js
|
||||
// load antlr4 and myLanguage
|
||||
var antlr4, mylanguage;
|
||||
|
|
@ -204,6 +212,7 @@ try {
|
|||
}
|
||||
```
|
||||
Please note the try-finally construct. ANTLR uses 'require' synchronously so it's perfectly safe to ignore the ACE 'require' while running ANTLR code. ACE itself does not guarantee synchronous execution, so you are much safer always switching 'require' back to 'ace_require'.
|
||||
|
||||
Now detecting deep syntax errors in your code is a task for your ANTLR listener or visitor or whatever piece of code you've delegated this to. We're not going to describe this here, since it would require some knowledge of your language. However, detecting grammar syntax errors is something ANTLR does beautifully (isn't that why you went for ANTLR in the first place?). So what we will illustrate here is how to report grammar syntax errors. I have no doubt that from there, you will be able to extend the validator to suit your specific needs.
|
||||
Whenever ANTLR encounters an unexpected token, it fires an error. By default, the error is routed to an error listener which simply writes to the console.
|
||||
What we need to do is replace this listener by our own listener, se we can route errors to the ACE editor. First, let's create such a listener:
|
||||
|
|
@ -243,5 +252,4 @@ var validate = function(input) {
|
|||
};
|
||||
```
|
||||
You know what? That's it! You now have an ACE editor that does syntax validation using ANTLR! I hope you find this useful, and simple enough to get started.
|
||||
What I did not address here is packaging, not something I'm an expert at. The good news is that it makes development simple, since I don't have to run any compilation process. I just edit my code, reload my editor page, and check how it goes.
|
||||
Now wait, hey! How do you debug this? Well, as usual, using Chrome, since neither Firefox or Safari are able to debug worker code. What a shame...
|
||||
WNow wait, hey! How do you debug this? Well, as usual, using Chrome, since no other browser is able to debug worker code. What a shame...
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Adding unit tests
|
||||
# ANTLR project unit tests
|
||||
|
||||
## Introduction
|
||||
|
||||
|
|
@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow
|
|||
The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
|
||||
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
|
||||
|
||||
## Requirements
|
||||
|
||||
In order to perform the tests on all target languages, you need to have the following languages installed:
|
||||
|
||||
* `mono` (e.g., `brew install mono`) on non-Windows boxes (on Windows it uses the Microsoft .net stack). Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
|
||||
* `nodejs`
|
||||
* Python 2.7
|
||||
* Python 3.5
|
||||
* Go
|
||||
* Swift 3 (via XCode 8.x) tested currently only osx
|
||||
* clang (for C++ target)
|
||||
*
|
||||
To **install into local repository** `~/.m2/repository/org/antlr`, do this:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
```
|
||||
|
||||
Now, make sure C# runtime is built and installed locally.
|
||||
|
||||
```bash
|
||||
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
|
||||
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
|
||||
rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll
|
||||
rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
|
||||
# build
|
||||
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
|
||||
```
|
||||
|
||||
C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.
|
||||
|
||||
|
||||
## Running the runtime tests
|
||||
|
||||
A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
|
||||
|
|
@ -49,26 +82,128 @@ And the result of testing the entire subdirectory:
|
|||
From `mvn`, on the commandline, you will see:
|
||||
|
||||
```bash
|
||||
$ cd antlr4
|
||||
$ mvn test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser
|
||||
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSets
|
||||
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestPerformance
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
|
||||
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
|
||||
...
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Reactor Summary:
|
||||
[INFO]
|
||||
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
|
||||
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
|
||||
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
|
||||
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
|
||||
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
|
||||
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
|
||||
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
|
||||
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Total time: 06:53 min
|
||||
[INFO] Finished at: 2016-11-16T15:36:56-08:00
|
||||
[INFO] Final Memory: 44M/458M
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
```
|
||||
|
||||
Note: That is actually result of running the much faster:
|
||||
|
||||
```bash
|
||||
mvn -Dparallel=methods -DthreadCount=4 install
|
||||
```
|
||||
|
||||
## Running test subsets
|
||||
|
||||
*From the `runtime-testsuite` dir*
|
||||
|
||||
### Run one test group across targets
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn -Dtest=TestParserExec test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestParserExec
|
||||
...
|
||||
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestParserExec
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests:
|
||||
|
||||
```
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=Test*Lexer* test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
|
||||
...
|
||||
```
|
||||
|
||||
### Run all tests for a single target
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.* test
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests in Java target only:
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.*Lexer* test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
|
||||
|
||||
Results :
|
||||
|
||||
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
|
||||
```
|
||||
|
||||
## Testing in parallel
|
||||
|
||||
Use this to run tests in parallel:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G"
|
||||
$ mvn -Dparallel=methods -DthreadCount=4 test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
|
||||
...
|
||||
```
|
||||
|
||||
This can be combined with other `-D` above.
|
||||
|
||||
## Adding a runtime test
|
||||
|
||||
To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
|
||||
|
|
@ -38,6 +38,7 @@ Receiving objects: 100% (59858/59858), 31.10 MiB | 819.00 KiB/s, done.
|
|||
Resolving deltas: 100% (31898/31898), done.
|
||||
Checking connectivity... done.
|
||||
$ cd antlr4
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn -DskipTests install
|
||||
...
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
|
|
@ -67,65 +68,10 @@ We do `install` not `compile` as tool tests and such refer to modules that must
|
|||
To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
```
|
||||
|
||||
# Testing tool and targets
|
||||
|
||||
In order to perform the tests on all target languages, you need to have the following languages installed:
|
||||
|
||||
* `mono` (e.g., `brew install mono`)
|
||||
* `nodejs`
|
||||
* Python 2.7
|
||||
* Python 3.5
|
||||
* Go
|
||||
* Swift 3 (via XCode 8.x) tested currently only osx
|
||||
* clang (for C++ target)
|
||||
|
||||
To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this:
|
||||
|
||||
```bash
|
||||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
$ mvn install # now "do it with feeling"
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
|
||||
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
|
||||
...
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Reactor Summary:
|
||||
[INFO]
|
||||
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
|
||||
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
|
||||
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
|
||||
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
|
||||
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
|
||||
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
|
||||
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
|
||||
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Total time: 06:53 min
|
||||
[INFO] Finished at: 2016-11-16T15:36:56-08:00
|
||||
[INFO] Final Memory: 44M/458M
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
```
|
||||
|
||||
Note: That is actually result of running the much faster:
|
||||
|
||||
`mvn -Dparallel=methods -DthreadCount=4 install`
|
||||
|
||||
|
||||
You should see these jars (when building 4.6-SNAPSHOT):
|
||||
|
||||
```bash
|
||||
|
|
@ -143,87 +89,17 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar
|
|||
|
||||
Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes.
|
||||
|
||||
## Running test subsets
|
||||
# Testing tool and targets
|
||||
|
||||
*From the `runtime-testsuite` dir*
|
||||
See [ANTLR project unit tests](antlr-project-testing.md).
|
||||
|
||||
### Run one test group across targets
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=TestParserExec test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestParserExec
|
||||
...
|
||||
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestParserExec
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests:
|
||||
|
||||
```
|
||||
$ mvn -Dtest=Test*Lexer* test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
|
||||
...
|
||||
```
|
||||
|
||||
### Run all tests for a single target
|
||||
|
||||
```bash
|
||||
$ mvn -Dtest=java.* test
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests in Java target only:
|
||||
|
||||
```bash
|
||||
$ mvn -Dtest=java.*Lexer* test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
|
||||
|
||||
Results :
|
||||
|
||||
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
|
||||
```
|
||||
|
||||
## Testing in parallel
|
||||
|
||||
Use this to run tests in parallel:
|
||||
|
||||
```bash
|
||||
$ mvn -Dparallel=methods -DthreadCount=4 test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
|
||||
...
|
||||
```
|
||||
|
||||
This can be combined with other `-D` above.
|
||||
|
||||
## Building without testing
|
||||
# Building without testing
|
||||
|
||||
To build without running the tests (saves a lot of time), do this:
|
||||
|
||||
```bash
|
||||
mvn -DskipTests install
|
||||
$ mvn -DskipTests install
|
||||
```
|
||||
|
||||
## Building ANTLR in Intellij IDE
|
||||
|
|
|
|||
|
|
@ -20,11 +20,13 @@ You will see that there are a whole bunch of files generated by this call. If vi
|
|||
|
||||
## Where can I get the runtime?
|
||||
|
||||
Once you've generated the lexer and/or parser code, you need to download or build the runtime. Prebuilt C++ runtime binaries for Windows (VS 2013 runtime), OSX and iOS are available on the ANTLR web site:
|
||||
Once you've generated the lexer and/or parser code, you need to download or build the runtime. Prebuilt C++ runtime binaries for Windows (Visual Studio 2013/2015), OSX/macOS and iOS are available on the ANTLR web site:
|
||||
|
||||
* http://www.antlr.org
|
||||
|
||||
Use CMake to build a Linux library (works also on OSX, however not for the iOS library). Building your own library on OSX or Windows is trivial, however. Just open the VS or XCode project, select target + arch and build it. Should work out of the box without any additional dependency.
|
||||
Use CMake to build a Linux library (works also on OSX, however not for the iOS library).
|
||||
|
||||
Instead of downloading a prebuilt binary you can also easily build your own library on OSX or Windows. Just use the provided projects for XCode or Visual Studio and build it. Should work out of the box without any additional dependency.
|
||||
|
||||
|
||||
## How do I run the generated lexer and/or parser?
|
||||
|
|
@ -76,6 +78,29 @@ This example assumes your grammar contains a parser rule named `key` for which t
|
|||
|
||||
There are a couple of things that only the C++ ANTLR target has to deal with. They are described here.
|
||||
|
||||
### Build Aspects
|
||||
The code generation (by running the ANTLR4 jar) allows to specify 2 values you might find useful for better integration of the generated files into your application (both are optional):
|
||||
|
||||
* A **namespace**: use the **`-package`** parameter to specify the namespace you want.
|
||||
* An **export macro**: especially in VC++ extra work is required to export your classes from a DLL. This is usually accomplished by a macro that has different values depending on whether you are creating the DLL or import it. The ANTLR4 runtime itself also uses one for its classes:
|
||||
|
||||
```c++
|
||||
#ifdef ANTLR4CPP_EXPORTS
|
||||
#define ANTLR4CPP_PUBLIC __declspec(dllexport)
|
||||
#else
|
||||
#ifdef ANTLR4CPP_STATIC
|
||||
#define ANTLR4CPP_PUBLIC
|
||||
#else
|
||||
#define ANTLR4CPP_PUBLIC __declspec(dllimport)
|
||||
#endif
|
||||
#endif
|
||||
```
|
||||
Just like the `ANTLR4CPP_PUBLIC` macro here you can specify your own one for the generated classes using the **`-export-macro`** parameter.
|
||||
|
||||
In order to create a static lib in Visual Studio define the `ANTLR4CPP_STATIC` macro in addition to the project settings that must be set for a static library (if you compile the runtime yourself).
|
||||
|
||||
For gcc and clang it is possible to use the `-fvisibility=hidden` setting to hide all symbols except those that are made default-visible (which has been defined for all public classes in the runtime).
|
||||
|
||||
### Memory Management
|
||||
Since C++ has no built-in memory management we need to take extra care. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable. Generally, when you see a raw pointer in code consider this as being managed elsewehere. You should never try to manage such a pointer (delete, assign to smart pointer etc.).
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,6 @@
|
|||
# C♯
|
||||
|
||||
See also [Sam Harwell's Alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs)
|
||||
|
||||
### Which frameworks are supported?
|
||||
## Which frameworks are supported?
|
||||
|
||||
The C# runtime is CLS compliant, and only requires a corresponding 3.5 .Net framework.
|
||||
|
||||
|
|
@ -13,19 +11,15 @@ In practice, the runtime has been extensively tested against:
|
|||
|
||||
No issue was found, so you should find that the runtime works pretty much against any recent .Net framework.
|
||||
|
||||
### How do I get started?
|
||||
## How do I get started?
|
||||
|
||||
You will find full instructions on the Git web page for ANTLR C# runtime.
|
||||
You will find full instructions on the [Git repo page for ANTLR C# runtime](https://github.com/antlr/antlr4/tree/master/runtime/CSharp).
|
||||
|
||||
### How do I use the runtime from my project?
|
||||
## How do I use the runtime from my project?
|
||||
|
||||
(i.e., How do I run the generated lexer and/or parser?)
|
||||
|
||||
Let's suppose that your grammar is named, as above, "MyGrammar".
|
||||
|
||||
Let's suppose this parser comprises a rule named "StartRule"
|
||||
|
||||
The tool will have generated for you the following files:
|
||||
Let's suppose that your grammar is named `MyGrammar`. The tool will generate for you the following files:
|
||||
|
||||
* MyGrammarLexer.cs
|
||||
* MyGrammarParser.cs
|
||||
|
|
@ -34,7 +28,7 @@ The tool will have generated for you the following files:
|
|||
* MyGrammarVisitor.js (if you have activated the -visitor option)
|
||||
* MyGrammarBaseVisitor.js (if you have activated the -visitor option)
|
||||
|
||||
Now a fully functioning code might look like the following:
|
||||
Now a fully functioning code might look like the following for start rule `StartRule`:
|
||||
|
||||
```
|
||||
using Antlr4.Runtime;
|
||||
|
|
@ -58,7 +52,7 @@ This program will work. But it won't be useful unless you do one of the followin
|
|||
|
||||
(please note that production code is target specific, so you can't have multi target grammars that include production code)
|
||||
|
||||
### How do I create and run a custom listener?
|
||||
## How do I create and run a custom listener?
|
||||
|
||||
Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value".
|
||||
|
||||
|
|
@ -96,4 +90,5 @@ ParseTreeWalker.DEFAULT.walk(printer, tree);
|
|||
|
||||
Further information can be found from The Definitive ANTLR Reference book.
|
||||
|
||||
The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#.
|
||||
The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#. See also [Sam Harwell's alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs)
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 439 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
|
|
@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema
|
|||
|
||||
* [Cutting an ANTLR Release](releasing-antlr.md)
|
||||
|
||||
* [Adding ANTLR unit tests](adding-tests.md)
|
||||
* [ANTLR project unit tests](antlr-project-testing.md)
|
||||
|
||||
* [Creating an ANTLR Language Target](creating-a-language-target.md)
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ The tests were conducted using Selenium. No issue was found, so you should find
|
|||
|
||||
## Is NodeJS supported?
|
||||
|
||||
The runtime has also been extensively tested against Node.js 0.10.33. No issue was found.
|
||||
The runtime has also been extensively tested against Node.js 0.12.7. No issue was found.
|
||||
|
||||
## How to create a JavaScript lexer or parser?
|
||||
|
||||
|
|
@ -31,7 +31,9 @@ For a full list of antlr4 tool options, please visit the [tool documentation pag
|
|||
|
||||
Once you've generated the lexer and/or parser code, you need to download the runtime.
|
||||
|
||||
The JavaScript runtime is available from the ANTLR web site [download section](http://www.antlr.org/download/index.html). The runtime is provided in the form of source code, so no additional installation is required.
|
||||
The JavaScript runtime is [available from npm](https://www.npmjs.com/package/antlr4).
|
||||
|
||||
If you can't use npm, the JavaScript runtime is also available from the ANTLR web site [download section](http://www.antlr.org/download/index.html). The runtime is provided in the form of source code, so no additional installation is required.
|
||||
|
||||
We will not document here how to refer to the runtime from your project, since this would differ a lot depending on your project type and IDE.
|
||||
|
||||
|
|
@ -47,11 +49,28 @@ However, it would be a bit of a problem when it comes to get it into a browser.
|
|||
<script src='lib/myscript.js'>
|
||||
```
|
||||
|
||||
In order to avoid having to do this, and also to have the exact same code for browsers and Node.js, we rely on a script which provides the equivalent of the Node.js 'require' function.
|
||||
To avoid having doing this, the preferred approach is to bundle antlr4 with your parser code, using webpack.
|
||||
|
||||
This script is provided by Torben Haase, and is NOT part of ANTLR JavaScript runtime, although the runtime heavily relies on it. Please note that syntax for 'require' in NodeJS is different from the one implemented by RequireJS and similar frameworks.
|
||||
You can get [information on webpack here](https://webpack.github.io).
|
||||
|
||||
So in short, assuming you have at the root of your web site, both the 'antlr4' directory and a 'lib' directory with 'require.js' inside it, all you need to put in your HTML header is the following:
|
||||
The steps to create your parsing code are the following:
|
||||
- generate your lexer, parser, listener and visitor using the antlr tool
|
||||
- write your parse tree handling code by providig your custom listener or visitor, and associated code, using 'require' to load antlr.
|
||||
- create an index.js file with the entry point to your parsing code (or several if required).
|
||||
- test your parsing logic thoroughly using node.js
|
||||
|
||||
You are now ready to bundle your parsing code as follows:
|
||||
- following webpack specs, create a webpack.config file
|
||||
- in the webpack.config file, exclude node.js only modules using: node: { module: "empty", net: "empty", fs: "empty" }
|
||||
- from the cmd line, nag-vigate to the directory containing webpack.config and type: webpack
|
||||
|
||||
This will produce a single js file containing all your parsing code. Easy to include in your web pages!
|
||||
|
||||
If you can't use webpack, you can use the lib/require.js script which implements the Node.js 'require' function in brwsers.
|
||||
|
||||
This script is provided by Torben Haase, and is NOT part of ANTLR JavaScript runtime.
|
||||
|
||||
Assuming you have, at the root of your web site, both the 'antlr4' directory and a 'lib' directory with 'require.js' inside it, all you need to put in your HTML header is the following:
|
||||
|
||||
```xml
|
||||
<script src='lib/require.js'>
|
||||
|
|
@ -62,16 +81,6 @@ So in short, assuming you have at the root of your web site, both the 'antlr4' d
|
|||
|
||||
This will load the runtime asynchronously.
|
||||
|
||||
## How do I get the runtime in Node.js?
|
||||
|
||||
Right now, there is no npm package available, so you need to register a link instead. This can be done by running the following command from the antlr4 directory:
|
||||
|
||||
```bash
|
||||
$ npm link antlr4
|
||||
```
|
||||
|
||||
This will install antlr4 using the package.json descriptor that comes with the script.
|
||||
|
||||
## How do I run the generated lexer and/or parser?
|
||||
|
||||
Let's suppose that your grammar is named, as above, "MyGrammar". Let's suppose this parser comprises a rule named "StartRule". The tool will have generated for you the following files:
|
||||
|
|
|
|||
|
|
@ -4,12 +4,22 @@
|
|||
|
||||
Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1).
|
||||
|
||||
### Delete existing release tag
|
||||
|
||||
Wack any existing tag as mvn will create one and it fails if already there.
|
||||
|
||||
```
|
||||
$ git tag -d 4.5.2
|
||||
$ git push origin :refs/tags/4.5.2
|
||||
$ git push upstream :refs/tags/4.5.2
|
||||
$ git tag -d 4.6
|
||||
$ git push origin :refs/tags/4.6
|
||||
$ git push upstream :refs/tags/4.6
|
||||
```
|
||||
|
||||
### Create release candidate tag
|
||||
|
||||
```bash
|
||||
$ git tag -a 4.6-rc1 -m 'heading towards 4.6'
|
||||
$ git push origin 4.6-rc1
|
||||
$ git push upstream 4.6-rc1
|
||||
```
|
||||
|
||||
## Bump version
|
||||
|
|
@ -22,15 +32,17 @@ Edit the repository looking for 4.5 or whatever and update it. Bump version in t
|
|||
* runtime/Python3/setup.py
|
||||
* runtime/Python3/src/antlr4/Recognizer.py
|
||||
* runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs
|
||||
* runtime/CSharp/build/version.ps1
|
||||
* runtime/JavaScript/src/antlr4/package.json
|
||||
* runtime/JavaScript/src/antlr4/Recognizer.js
|
||||
* runtime/Cpp/VERSION
|
||||
* runtime/Cpp/runtime/src/RuntimeMetaData.cpp
|
||||
* runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake
|
||||
* tool/src/org/antlr/v4/codegen/target/CppTarget.java
|
||||
* tool/src/org/antlr/v4/codegen/target/CSharpTarget.java
|
||||
* tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java
|
||||
* tool/src/org/antlr/v4/codegen/target/Python2Target.java
|
||||
* tool/src/org/antlr/v4/codegen/target/Python3Target.java
|
||||
* runtime/Cpp/VERSION
|
||||
* runtime/Cpp/RuntimeMetaData.cpp
|
||||
|
||||
Here is a simple script to display any line from the critical files with, say, `4.5` in it:
|
||||
|
||||
|
|
@ -144,24 +156,24 @@ It will start out by asking you the version number:
|
|||
|
||||
```
|
||||
...
|
||||
What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.2: : 4.5.2
|
||||
What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.5.2: :
|
||||
What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.5.2: :
|
||||
What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.5.2: :
|
||||
What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.5.2: :
|
||||
What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.5.2: :
|
||||
What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.5.2: : 4.5.2
|
||||
What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.3-SNAPSHOT:
|
||||
What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6: : 4.6
|
||||
What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.6: :
|
||||
What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.6: :
|
||||
What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.6: :
|
||||
What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.6: :
|
||||
What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.6: :
|
||||
What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.6: : 4.6
|
||||
What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6.1-SNAPSHOT:
|
||||
...
|
||||
```
|
||||
|
||||
Maven will go through your pom.xml files to update versions from 4.5.2-SNAPSHOT to 4.5.2 for release and then to 4.5.3-SNAPSHOT after release, which is done with:
|
||||
Maven will go through your pom.xml files to update versions from 4.6-SNAPSHOT to 4.6 for release and then to 4.6.1-SNAPSHOT after release, which is done with:
|
||||
|
||||
```bash
|
||||
mvn release:perform -Darguments="-DskipTests"
|
||||
```
|
||||
|
||||
Maven will use git to push pom.xml changes. (big smile)
|
||||
Maven will use git to push pom.xml changes.
|
||||
|
||||
Now, go here:
|
||||
|
||||
|
|
@ -174,11 +186,11 @@ and on the left click "Staging Repositories". You click the staging repo and clo
|
|||
Copy the jars to antlr.org site and update download/index.html
|
||||
|
||||
```bash
|
||||
cp ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.5.2.jar
|
||||
cp ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-4.5.2-complete.jar
|
||||
cp ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.6.jar
|
||||
cp ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.6-complete.jar
|
||||
cd ~/antlr/sites/website-antlr4/download
|
||||
git add antlr-4.5.2-complete.jar
|
||||
git add antlr-runtime-4.5.2.jar
|
||||
git add antlr-4.6-complete.jar
|
||||
git add antlr-runtime-4.6.jar
|
||||
```
|
||||
|
||||
Update on site:
|
||||
|
|
@ -190,7 +202,7 @@ Update on site:
|
|||
* scripts/topnav.js
|
||||
|
||||
```
|
||||
git commit -a -m 'add 4.5.2 jars'
|
||||
git commit -a -m 'add 4.6 jars'
|
||||
git push origin gh-pages
|
||||
```
|
||||
|
||||
|
|
@ -200,8 +212,8 @@ git push origin gh-pages
|
|||
|
||||
```bash
|
||||
cd runtime/JavaScript/src
|
||||
zip -r /tmp/antlr-javascript-runtime-4.5.2.zip antlr4
|
||||
cp /tmp/antlr-javascript-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download
|
||||
zip -r /tmp/antlr-javascript-runtime-4.6.zip antlr4
|
||||
cp /tmp/antlr-javascript-runtime-4.6.zip ~/antlr/sites/website-antlr4/download
|
||||
# git add, commit, push
|
||||
```
|
||||
|
||||
|
|
@ -209,7 +221,7 @@ Move target to website
|
|||
|
||||
```bash
|
||||
pushd ~/antlr/sites/website-antlr4/download
|
||||
git add antlr-javascript-runtime-4.5.2.zip
|
||||
git add antlr-javascript-runtime-4.6.zip
|
||||
git commit -a -m 'update JS runtime'
|
||||
git push origin gh-pages
|
||||
popd
|
||||
|
|
@ -217,6 +229,70 @@ popd
|
|||
|
||||
### CSharp
|
||||
|
||||
*Publishing to Nuget from Linux/MacOSX*
|
||||
|
||||
**Getting ready to run Nuget**
|
||||
|
||||
Of course you need Mono and `nuget` to be installed. On mac:
|
||||
|
||||
```bash
|
||||
brew install mono
|
||||
brew install nuget
|
||||
```
|
||||
|
||||
Or, you can [download nuget.exe](https://dist.nuget.org/win-x86-commandline/latest/nuget.exe).
|
||||
|
||||
From the shell on mac, you can check all is ok by typing
|
||||
|
||||
```bash
|
||||
nuget
|
||||
```
|
||||
|
||||
This should display the nuget help.
|
||||
|
||||
**Creating the assembly**
|
||||
|
||||
```bash
|
||||
$ cd runtime/CSharp/runtime/CSharp/Antlr4.Runtime
|
||||
$ xbuild /p:Configuration=Release Antlr4.Runtime.mono.csproj
|
||||
...
|
||||
Copying file from '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.Standard.dll' to '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/lib/Release/Antlr4.Runtime.Standard.dll'
|
||||
Done building project "/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Antlr4.Runtime.mono.csproj".
|
||||
```
|
||||
|
||||
Alternately, you may want to build ANTLR using Xamarin Studio Community (free).
|
||||
|
||||
**Packaging for NuGet**
|
||||
|
||||
```bash
|
||||
cd runtime/CSharp/runtime/CSharp
|
||||
```
|
||||
|
||||
which is where the `Package.nuspec` file resides.
|
||||
|
||||
Type the following command:
|
||||
|
||||
```bash
|
||||
$ nuget pack Package.nuspec
|
||||
Attempting to build package from 'Package.nuspec'.
|
||||
Successfully created package '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime.Standard.4.6.0.nupkg'.
|
||||
```
|
||||
|
||||
This should display: Successfully created package *<package-path>*
|
||||
|
||||
**Publishing to NuGet**
|
||||
|
||||
You need to be a NuGet owner for "ANTLR 4 Standard Runtime"
|
||||
As a registered NuGet user, you can then manually upload the package spec here (`runtime/CSharp/runtime/CSharp/Package.nuspec`): [https://www.nuget.org/packages/manage/upload](https://www.nuget.org/packages/manage/upload)
|
||||
|
||||
Alternately, you can publish from the cmd line. You need to get your NuGet key from [https://www.nuget.org/account#](https://www.nuget.org/account#) and then from the cmd line, you can then type:
|
||||
|
||||
```bash
|
||||
nuget push Antlr4.Runtime.Standard.<version>.nupkg <your-key> -Source https://www.nuget.org/api/v2/package
|
||||
```
|
||||
|
||||
**Creating DLLs**
|
||||
|
||||
```bash
|
||||
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
|
||||
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
|
||||
|
|
@ -225,15 +301,15 @@ rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
|
|||
# build
|
||||
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
|
||||
# zip it up to get a version number on zip filename
|
||||
zip --junk-paths /tmp/antlr-csharp-runtime-4.5.2.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll
|
||||
cp /tmp/antlr-csharp-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download
|
||||
zip --junk-paths /tmp/antlr-csharp-runtime-4.6.zip Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.Standard.dll
|
||||
cp /tmp/antlr-csharp-runtime-4.6.zip ~/antlr/sites/website-antlr4/download
|
||||
```
|
||||
|
||||
Move target to website
|
||||
|
||||
```bash
|
||||
pushd ~/antlr/sites/website-antlr4/download
|
||||
git add antlr-csharp-runtime-4.5.2.zip
|
||||
git add antlr-csharp-runtime-4.6.zip
|
||||
git commit -a -m 'update C# runtime'
|
||||
git push origin gh-pages
|
||||
popd
|
||||
|
|
@ -282,7 +358,7 @@ python setup.py register -r pypi
|
|||
python setup.py sdist bdist_wininst upload -r pypi
|
||||
```
|
||||
|
||||
Add links to the artifacts from download.html
|
||||
There are links to the artifacts in [download.html](http://www.antlr.org/download.html) already.
|
||||
|
||||
### C++
|
||||
|
||||
|
|
@ -302,6 +378,7 @@ On a Mac (with XCode 7+ installed):
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
./deploy-macos.sh
|
||||
cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-macos.zip
|
||||
```
|
||||
|
||||
On any Mac or Linux machine:
|
||||
|
|
@ -309,6 +386,7 @@ On any Mac or Linux machine:
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
./deploy-source.sh
|
||||
cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-source.zip
|
||||
```
|
||||
|
||||
On a Windows machine the build scripts checks if VS 2013 and/or VS 2015 are installed and builds binaries for each, if found. This script requires 7z to be installed (http://7-zip.org).
|
||||
|
|
@ -316,15 +394,17 @@ On a Windows machine the build scripts checks if VS 2013 and/or VS 2015 are inst
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
deploy-windows.cmd
|
||||
cp antlr4-cpp-runtime-vs2015.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-vs2015.zip
|
||||
```
|
||||
|
||||
Move target to website (**_rename to a specific ANTLR version first if needed_**):
|
||||
|
||||
```bash
|
||||
pushd ~/antlr/sites/website-antlr4/download
|
||||
git add antlr4cpp-runtime-macos.zip
|
||||
git add antlr4cpp-runtime-windows.zip
|
||||
git add antlr4cpp-runtime-source.zip
|
||||
# vi index.html
|
||||
git add antlr4cpp-runtime-4.6-macos.zip
|
||||
git add antlr4cpp-runtime-4.6-windows.zip
|
||||
git add antlr4cpp-runtime-4.6-source.zip
|
||||
git commit -a -m 'update C++ runtime'
|
||||
git push origin gh-pages
|
||||
popd
|
||||
|
|
@ -332,7 +412,7 @@ popd
|
|||
|
||||
## Update javadoc for runtime and tool
|
||||
|
||||
First gen javadoc:
|
||||
First, gen javadoc:
|
||||
|
||||
```bash
|
||||
$ cd antlr4
|
||||
|
|
@ -346,9 +426,9 @@ cd ~/antlr/sites/website-antlr4/api
|
|||
git checkout gh-pages
|
||||
git pull origin gh-pages
|
||||
cd Java
|
||||
jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2-javadoc.jar
|
||||
jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6-javadoc.jar
|
||||
cd ../JavaTool
|
||||
jar xvf ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2-javadoc.jar
|
||||
jar xvf ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-javadoc.jar
|
||||
git commit -a -m 'freshen api doc'
|
||||
git push origin gh-pages
|
||||
```
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate
|
|||
|
||||
## Generating JUnit Tests
|
||||
|
||||
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run
|
||||
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run
|
||||
|
||||
```bash
|
||||
$ mvn -Pgen generate-test-sources
|
||||
|
|
|
|||
|
|
@ -1,47 +1,99 @@
|
|||
# ANTLR4 Language Target, Runtime for Swift
|
||||
|
||||
### Usage
|
||||
## Install ANTLR4
|
||||
|
||||
#### 1. Install ANTLR4
|
||||
Make sure you have the ANTLR
|
||||
installed.[The getting started guide](getting-started.md) should get
|
||||
you started.
|
||||
|
||||
[The getting started guide](getting-started.md) should get you started.
|
||||
## Create a Swift lexer or parser
|
||||
This is pretty much the same as creating a Java lexer or parser,
|
||||
except you need to specify the language target, for example:
|
||||
|
||||
#### 2. create a Swift lexer or parser
|
||||
This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
|
||||
```
|
||||
$ antlr4 -Dlanguage=Swift MyGrammar.g4
|
||||
```
|
||||
For a full list of antlr4 tool options, please visit the
|
||||
[tool documentation page](tool-options.md).
|
||||
|
||||
```
|
||||
$ antlr4 -Dlanguage=Swift MyGrammar.g4
|
||||
```
|
||||
For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md).
|
||||
## Build your Swift project with ANTLR runtime
|
||||
|
||||
#### 3. Get the Swift ANTLR runtime
|
||||
You will find Swift runtime (framework project) in
|
||||
The following instructions are assuming Xcode as the IDE:
|
||||
|
||||
```
|
||||
antlr4/runtime/Swift
|
||||
```
|
||||
* __Add parser/lexer to project__. Make sure the parsers/lexers
|
||||
generated in __step 2__ are added to the project. To do this, you can
|
||||
drag the generated files from Finder to the Xcode IDE. Remember to
|
||||
check __Copy items if needed__ to make sure the files are actually
|
||||
moved into the project folder instead of symbolic links (see the
|
||||
screenshot below). After moving you will be able to see your files in
|
||||
the project navigator. But when you open one of the files, you will
|
||||
see Xcode complaining the module "Antlr4" could not be found at the
|
||||
import statement. This is expected, since we still need the ANTLR
|
||||
Swift runtime for those missing symbols.
|
||||
|
||||
#### 4. Example playground
|
||||
<img src=images/dragfile.png width="500">
|
||||
|
||||
* __Download ANTLR runtime__. Due to unstable ABI of Swift language,
|
||||
there will not be a single "library" for the Swift ANTLR runtime for
|
||||
now. To get Swift ANTLR runtime, clone the ANTLR repository. Open it
|
||||
in finder. From the root directory of the repo, go to runtime/Swift
|
||||
folder. You will see the Xcode project manifest file:
|
||||
__Antlr4.xcodeproj__.
|
||||
|
||||
* __Import ANTLR Swift runtime into project__. Drag Antlr4.xcodeproj
|
||||
into your project, after this is done, your Xcode project navigator
|
||||
will be something like the screenshot below. In this case, your own
|
||||
project is "Smalltalk", and you will be able to see the
|
||||
Antlr4.xcodeproj shown as a contained project. The error message will
|
||||
still be there, that's because we still need to tell Xcode how to find
|
||||
the runtime.
|
||||
|
||||
<img src=images/xcodenav.png width="300">
|
||||
|
||||
* __Build ANTLR runtime__. By expanding the "Products" folder in the
|
||||
inner project (Antlr4.xcodeproj), you will see two Antlr4.framework
|
||||
files. ".framework" file is the swift version of ".jar", ".a" as in
|
||||
JAVA, C/C++ Initially those two files should be red, that's because
|
||||
they are not built. To build, click the "target selection" button
|
||||
right next to your Xcode run button. And in the drop down select the
|
||||
target you want to build. And you will see the two Antlr4.framework
|
||||
files are for iOS and OSX, as shown below. After target selection,
|
||||
press "CMD+B", and Xcode will build the framework for you. Then you
|
||||
will see one of the frameworks become black.
|
||||
|
||||
<img src=images/targetselection.png width="500">
|
||||
|
||||
* __Add dependencies__. Simply adding ANTLR Swift runtime and build
|
||||
the artifact is not enough. You still need to specify
|
||||
dependencies. Click your own project (Smalltalk), and you will see
|
||||
project setting page. Go to "Build Phase", and inside it make sure
|
||||
your ANTLR Swift runtime framework is added to both "__Target
|
||||
Dependencies__" and "__Link Binary With Libraries__" sections, as
|
||||
shown below. After correctly added dependencies, the error message for
|
||||
importing library will be gone.
|
||||
|
||||
<img src=images/xcodedep.png width="800">
|
||||
|
||||
## Example playground
|
||||
|
||||
The Swift runtime includes an Xcode playground to get started with.
|
||||
|
||||
In Xcode, open `antlr4/runtime/Swift/Antlr4.xcworkspace`. Select
|
||||
"Antlr4 OSX > My Mac" as the build target, and build the project as normal.
|
||||
The playground should then be active.
|
||||
First go to the ANTLR4 repository, and open
|
||||
`runtime/Swift/Antlr4.xcworkspace` in Xcode. Select "Antlr4 OSX > My
|
||||
Mac" as the build target, and build the project as normal. The
|
||||
playground should then be active.
|
||||
|
||||
The playground includes a simple grammar called "Hello", and an example for
|
||||
walking the parse tree. You should see in the playground output that it is
|
||||
printing messages for each node in the parse tree as it walks.
|
||||
The playground includes a simple grammar called "Hello", and an
|
||||
example for walking the parse tree. You should see in the playground
|
||||
output that it is printing messages for each node in the parse tree as
|
||||
it walks.
|
||||
|
||||
The playground shows how to create a lexer, token stream, and parser, and
|
||||
how to execute the parse.
|
||||
The grammar is defined in the playground's `Resources/Hello.g4`. The
|
||||
parser was generated from the grammar using ANTLR like this:
|
||||
|
||||
The grammar is defined in the playground's `Resources/Hello.g4`. The parser
|
||||
was generated from the grammar using Antlr4 like this:
|
||||
|
||||
```
|
||||
cd 'antlr4/runtime/Swift/Antlr4 playground.playground/Resources'
|
||||
```
|
||||
antlr4 -Dlanguage=Swift -visitor -o ../Sources/Autogen Hello.g4
|
||||
```
|
||||
|
||||
The example tree walker is in `Sources/HelloWalker.swift`.
|
||||
The example tree walker is in Sources/HelloWalker.swift.
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ This page lists the available and upcoming ANTLR runtimes. Please note that you
|
|||
|
||||
New features generally appear in the Java target and then migrate to the other targets, but these other targets don't always get updated in the same overall tool release. This section tries to identify features added to Java that have not been added to the other targets.
|
||||
|
||||
|Feature|Java|C♯|JavaScript|Python2|Python3|Swift|C++|
|
||||
|---|---|---|---|---|---|---|---|
|
||||
|Ambiguous tree construction|4.5.1|-|-|-|-|-|-|
|
||||
|Feature|Java|C♯|Python2|Python3|JavaScript|Go|C++|Swift|
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
|Ambiguous tree construction|4.5.1|-|-|-|-|-|-|-|
|
||||
|
||||
|
|
|
|||
2
pom.xml
2
pom.xml
|
|
@ -13,7 +13,7 @@
|
|||
</parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<packaging>pom</packaging>
|
||||
|
||||
<name>ANTLR 4</name>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-test-annotations</artifactId>
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-testsuite</artifactId>
|
||||
<name>ANTLR 4 Runtime Tests (2nd generation)</name>
|
||||
|
|
@ -79,107 +79,59 @@
|
|||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<testSourceDirectory>test</testSourceDirectory>
|
||||
<resources>
|
||||
<resource>
|
||||
<directory>resources</directory>
|
||||
</resource>
|
||||
<resource>
|
||||
<directory>../runtime</directory>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<version>2.19.1</version>
|
||||
<configuration>
|
||||
<includes>
|
||||
<include>**/csharp/Test*.java</include>
|
||||
<include>**/java/Test*.java</include>
|
||||
<include>**/go/Test*.java</include>
|
||||
<include>**/javascript/node/Test*.java</include>
|
||||
<include>**/python2/Test*.java</include>
|
||||
<include>**/python3/Test*.java</include>
|
||||
<include>${antlr.tests.swift}</include>
|
||||
</includes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-jar-plugin</artifactId>
|
||||
<version>2.4</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<goals>
|
||||
<goal>test-jar</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
<build>
|
||||
<testSourceDirectory>test</testSourceDirectory>
|
||||
<resources>
|
||||
<resource>
|
||||
<directory>resources</directory>
|
||||
</resource>
|
||||
<resource>
|
||||
<directory>../runtime</directory>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<version>2.19.1</version>
|
||||
<configuration>
|
||||
<includes>
|
||||
<include>**/csharp/Test*.java</include>
|
||||
<include>**/java/Test*.java</include>
|
||||
<include>**/go/Test*.java</include>
|
||||
<include>**/javascript/node/Test*.java</include>
|
||||
<include>**/python2/Test*.java</include>
|
||||
<include>**/python3/Test*.java</include>
|
||||
<include>${antlr.tests.swift}</include>
|
||||
</includes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-jar-plugin</artifactId>
|
||||
<version>2.4</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<goals>
|
||||
<goal>test-jar</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<profiles>
|
||||
<profile>
|
||||
<id>includeSwiftTests</id>
|
||||
<activation>
|
||||
<os>
|
||||
<family>mac</family>
|
||||
</os>
|
||||
</activation>
|
||||
<properties>
|
||||
<antlr.tests.swift>**/swift/Test*.java</antlr.tests.swift>
|
||||
</properties>
|
||||
</profile>
|
||||
|
||||
<profile>
|
||||
<id>gen</id>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.codehaus.mojo</groupId>
|
||||
<artifactId>exec-maven-plugin</artifactId>
|
||||
<version>1.4.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>generate-test-sources</phase>
|
||||
<goals>
|
||||
<goal>java</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<mainClass>org.antlr.v4.testgen.TestGenerator</mainClass>
|
||||
<arguments>
|
||||
<argument>-root</argument>
|
||||
<argument>${basedir}</argument>
|
||||
<argument>-outdir</argument>
|
||||
<argument>${basedir}/test</argument>
|
||||
<argument>-templates</argument>
|
||||
<argument>${basedir}/resources/org/antlr/v4/test/runtime/templates</argument>
|
||||
</arguments>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
<profile>
|
||||
<id>tests</id>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<configuration>
|
||||
<includes>
|
||||
<include>**/Test*.java</include>
|
||||
</includes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
</profiles>
|
||||
<profiles>
|
||||
<profile>
|
||||
<id>includeSwiftTests</id>
|
||||
<activation>
|
||||
<os>
|
||||
<family>mac</family>
|
||||
</os>
|
||||
</activation>
|
||||
<properties>
|
||||
<antlr.tests.swift>**/swift/Test*.java</antlr.tests.swift>
|
||||
</properties>
|
||||
</profile>
|
||||
</profiles>
|
||||
</project>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-test-annotation-processors</artifactId>
|
||||
|
|
|
|||
|
|
@ -55,7 +55,9 @@ import java.io.InputStream;
|
|||
import java.io.InputStreamReader;
|
||||
import java.lang.reflect.InvocationTargetException;
|
||||
import java.lang.reflect.Method;
|
||||
import java.net.URISyntaxException;
|
||||
import java.net.URL;
|
||||
import java.nio.file.Paths;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
|
|
@ -508,7 +510,7 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
return files;
|
||||
}
|
||||
|
||||
private String runProcess(ProcessBuilder builder, String description) throws Exception {
|
||||
private String runProcess(ProcessBuilder builder, String description, boolean showStderr) throws Exception {
|
||||
// System.out.println("BUILDER: "+builder.command());
|
||||
Process process = builder.start();
|
||||
StreamVacuum stdoutVacuum = new StreamVacuum(process.getInputStream());
|
||||
|
|
@ -521,10 +523,10 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
String output = stdoutVacuum.toString();
|
||||
if ( stderrVacuum.toString().length()>0 ) {
|
||||
this.stderrDuringParse = stderrVacuum.toString();
|
||||
// System.err.println(this.stderrDuringParse);
|
||||
if ( showStderr ) System.err.println(this.stderrDuringParse);
|
||||
}
|
||||
if (errcode != 0) {
|
||||
String err = "execution failed with error code: "+errcode;
|
||||
String err = "execution of '"+description+"' failed with error code: "+errcode;
|
||||
if ( this.stderrDuringParse!=null ) {
|
||||
this.stderrDuringParse += err;
|
||||
}
|
||||
|
|
@ -536,11 +538,11 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
return output;
|
||||
}
|
||||
|
||||
private String runCommand(String command[], String workPath, String description) throws Exception {
|
||||
private String runCommand(String command[], String workPath, String description, boolean showStderr) throws Exception {
|
||||
ProcessBuilder builder = new ProcessBuilder(command);
|
||||
builder.directory(new File(workPath));
|
||||
|
||||
return runProcess(builder, description);
|
||||
return runProcess(builder, description, showStderr);
|
||||
}
|
||||
|
||||
// TODO: add a buildRuntimeOnWindows variant.
|
||||
|
|
@ -550,8 +552,9 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
|
||||
try {
|
||||
String command[] = { "cmake", ".", /*"-DCMAKE_CXX_COMPILER=clang++",*/ "-DCMAKE_BUILD_TYPE=release" };
|
||||
if (runCommand(command, runtimePath, "antlr runtime cmake") == null)
|
||||
if (runCommand(command, runtimePath, "antlr runtime cmake", false) == null) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
catch (Exception e) {
|
||||
System.err.println("can't configure antlr cpp runtime cmake file");
|
||||
|
|
@ -559,11 +562,21 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
|
||||
try {
|
||||
String command[] = { "make", "-j", "8" }; // Assuming a reasonable amount of available CPU cores.
|
||||
if (runCommand(command, runtimePath, "building antlr runtime") == null)
|
||||
if (runCommand(command, runtimePath, "building antlr runtime", true) == null)
|
||||
return false;
|
||||
}
|
||||
catch (Exception e) {
|
||||
System.err.println("can't compile antlr cpp runtime");
|
||||
e.printStackTrace(System.err);
|
||||
try {
|
||||
String command[] = { "ls", "-la" };
|
||||
String output = runCommand(command, runtimePath + "/dist/", "printing library folder content", true);
|
||||
System.out.println(output);
|
||||
}
|
||||
catch (Exception e2) {
|
||||
System.err.println("can't even list folder content");
|
||||
e2.printStackTrace(System.err);
|
||||
}
|
||||
}
|
||||
|
||||
/* for debugging
|
||||
|
|
@ -593,7 +606,7 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
if ( !runtimeBuiltOnce ) {
|
||||
try {
|
||||
String command[] = {"clang++", "--version"};
|
||||
String output = runCommand(command, tmpdir, "printing compiler version");
|
||||
String output = runCommand(command, tmpdir, "printing compiler version", false);
|
||||
System.out.println("Compiler version is: "+output);
|
||||
}
|
||||
catch (Exception e) {
|
||||
|
|
@ -613,7 +626,7 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
String libExtension = (getOS().equals("mac")) ? "dylib" : "so";
|
||||
try {
|
||||
String command[] = { "ln", "-s", runtimePath + "/dist/libantlr4-runtime." + libExtension };
|
||||
if (runCommand(command, tmpdir, "sym linking C++ runtime") == null)
|
||||
if (runCommand(command, tmpdir, "sym linking C++ runtime", true) == null)
|
||||
return null;
|
||||
}
|
||||
catch (Exception e) {
|
||||
|
|
@ -625,7 +638,7 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
try {
|
||||
List<String> command2 = new ArrayList<String>(Arrays.asList("clang++", "-std=c++11", "-I", includePath, "-L.", "-lantlr4-runtime", "-o", "a.out"));
|
||||
command2.addAll(allCppFiles(tmpdir));
|
||||
if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary") == null) {
|
||||
if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary", true) == null) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
|
@ -642,7 +655,7 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
builder.directory(new File(tmpdir));
|
||||
Map<String, String> env = builder.environment();
|
||||
env.put("LD_PRELOAD", runtimePath + "/dist/libantlr4-runtime." + libExtension);
|
||||
String output = runProcess(builder, "running test binary");
|
||||
String output = runProcess(builder, "running test binary", false);
|
||||
if ( output.length()==0 ) {
|
||||
output = null;
|
||||
}
|
||||
|
|
@ -664,11 +677,21 @@ public class BaseCppTest implements RuntimeTestSupport {
|
|||
|
||||
protected String locateRuntime() {
|
||||
final ClassLoader loader = Thread.currentThread().getContextClassLoader();
|
||||
final URL runtimeSrc = loader.getResource("Cpp");
|
||||
if (runtimeSrc == null) {
|
||||
final URL runtimeURL = loader.getResource("Cpp");
|
||||
if (runtimeURL == null) {
|
||||
throw new RuntimeException("Cannot find runtime");
|
||||
}
|
||||
return runtimeSrc.getPath();
|
||||
// Windows not getting runtime right. See:
|
||||
// http://stackoverflow.com/questions/6164448/convert-url-to-normal-windows-filename-java
|
||||
// it was coming back "/C:/projects/antlr4-l7imv/runtime-testsuite/target/classes/Cpp"
|
||||
String p;
|
||||
try {
|
||||
p = Paths.get(runtimeURL.toURI()).toFile().toString();
|
||||
}
|
||||
catch (URISyntaxException use) {
|
||||
p = "Can't find runtime";
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
List<ANTLRMessage> getMessagesOfType(List<ANTLRMessage> msgs, Class<? extends ANTLRMessage> c) {
|
||||
|
|
|
|||
|
|
@ -302,30 +302,6 @@ public class LexerExecDescriptors {
|
|||
|
||||
}
|
||||
|
||||
public static class CharSetWithReversedRange extends BaseLexerTestDescriptor {
|
||||
public String input = "9";
|
||||
/**
|
||||
A
|
||||
[@0,0:0='9',<1>,1:0]
|
||||
[@1,1:0='<EOF>',<-1>,1:1]
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String output;
|
||||
|
||||
public String errors = null;
|
||||
public String startRule = "";
|
||||
public String grammarName = "L";
|
||||
|
||||
/**
|
||||
lexer grammar L;
|
||||
A : [z-a9]+ {<writeln("\"A\"")>} ;
|
||||
WS : [ \n]+ -> skip ;
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String grammar;
|
||||
|
||||
}
|
||||
|
||||
public static class EOFByItself extends BaseLexerTestDescriptor {
|
||||
public String input = "";
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -112,7 +112,7 @@ public class ParseTreesDescriptors {
|
|||
|
||||
@Override
|
||||
public boolean ignore(String targetName) {
|
||||
return !targetName.matches("Java|Python2|Python3|Node");
|
||||
return !targetName.matches("Java|Python2|Python3|Node|Swift");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,30 @@ import org.antlr.v4.test.runtime.BaseLexerTestDescriptor;
|
|||
import org.antlr.v4.test.runtime.CommentHasStringValue;
|
||||
|
||||
public class SemPredEvalLexerDescriptors {
|
||||
// Test for https://github.com/antlr/antlr4/issues/958
|
||||
public static class RuleSempredFunction extends BaseLexerTestDescriptor {
|
||||
public String input = "aaa";
|
||||
/**
|
||||
[@0,0:0='a',<1>,1:0]
|
||||
[@1,1:1='a',<1>,1:1]
|
||||
[@2,2:2='a',<1>,1:2]
|
||||
[@3,3:2='<EOF>',<-1>,1:3]
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String output;
|
||||
|
||||
public String errors = null;
|
||||
public String startRule = "";
|
||||
public String grammarName = "L";
|
||||
|
||||
/**
|
||||
lexer grammar L;
|
||||
T : 'a' {<True()>}? ;
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String grammar;
|
||||
}
|
||||
|
||||
public static class DisableRule extends BaseLexerTestDescriptor {
|
||||
public String input = "enum abc";
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -1,11 +1,18 @@
|
|||
# C# target for ANTLR 4
|
||||
|
||||
### Note to existing users
|
||||
### Note to historical users
|
||||
|
||||
Versions of ANTLR 4.4.x and before managed the C# target as a [separate tool provided by Sam Harwell](https://github.com/tunnelvisionlabs/antlr4cs/releases/tag/v4.3.0). As of 4.5, we our releasing a (mono-compatible) C# target derived from Sam's with the main tool. ANTLR 4.5 is now able to generate C#, Java, Python 2, Python 3 and JavaScript. Sam continues to work on his version of the ANTLR tool and so a C# target is also available through that.
|
||||
Versions of ANTLR 4.4.x and before managed the C#
|
||||
target as part of a [separate tool provided by Sam Harwell](https://github.com/tunnelvisionlabs/antlr4cs/releases/tag/v4.3.0).
|
||||
As of 4.5, we our releasing a (mono-compatible) C# target together
|
||||
with the main tool.
|
||||
|
||||
This makes it possible to use ANTLR 4 in any C# development environment, including of course Microsoft Visual Studio, but also Xamarin Studio, which runs on MacOS X.
|
||||
Given Microsoft's recent commitment to *cross-platform developer experiences*, we believe this is a great opportunity for C# developers.
|
||||
Mono compatibility makes it possible to use ANTLR 4 in any C# development
|
||||
environment, including of course Microsoft Visual Studio, but also Xamarin Studio, which runs on MacOS X.
|
||||
Given Microsoft's recent commitment to *cross-platform developer experiences*,
|
||||
we believe this is a great opportunity for C# developers.
|
||||
|
||||
Releasing the runtime with the tool ensures that you can get the exact same behavior across many languages: Java, C#, Python, JavaScript, Go, Swift and C++.
|
||||
|
||||
## Getting Started
|
||||
|
||||
|
|
@ -16,39 +23,37 @@ You can install *any* of the following versions of Java to use this target.
|
|||
|
||||
If you already have one of the following installed, you should check to make sure the installation is up-to-date.
|
||||
|
||||
* Java 8 runtime environment (x86 or x64)
|
||||
* Java 8 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
* Java 7 runtime environment (x86 or x64)
|
||||
* Java 7 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
* Java 6 runtime environment (x86 or x64)
|
||||
* Java 6 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
|
||||
### Step 2: Download the tool and runtime, and install the runtime
|
||||
### Step 2: Download the tool
|
||||
|
||||
You need to download the following from the ANTLR web site:
|
||||
You need to download the ANTLR tool from the ANTLR web site.
|
||||
This is a Java archive (*.jar) used to generate the C# code from an ANTLR grammar.
|
||||
|
||||
1. The ANTLR tool. This is a Java archive (*.jar) used to generate the C# code from an ANTLR grammar.
|
||||
2. The ANTLR runtime assembly. This is a regular .Net assembly (*.dll).
|
||||
|
||||
You then need to install the runtime assembly in the GAC.
|
||||
This is required to easily reference the runtime from your project.
|
||||
Go the the directory where you downloaded the runtime, and type the following command:
|
||||
`gacutil -i Antlr4.Runtime.dll`
|
||||
|
||||
### Step 3: Add a reference to the ANTLR runtime in your project
|
||||
|
||||
We trust that you know how to do this :-).
|
||||
|
||||
### Step 4: Add or create a grammar file (*.g4) in your project
|
||||
### Step 3: Add or create a grammar file (*.g4) in your project
|
||||
|
||||
To avoid confusing your IDE, we suggest setting the build action to None for this file.
|
||||
See the docs and the book to learn about writing lexer and parser grammars.
|
||||
|
||||
### Step 5: Generate the C# code
|
||||
|
||||
This can be done either from the cmd line, or by adding a custom pre-build command.
|
||||
At minimal, the cmd line should look as follows: ``java -jar antlr4-4.5.jar -Dlanguage=CSharp grammar.g4``
|
||||
### Step 4: Generate the C# code
|
||||
|
||||
This can be done either from the cmd line, or by adding a custom pre-build command in your project.
|
||||
At minimal, the cmd line should look as follows: ``java -jar antlr4-4.6.jar -Dlanguage=CSharp grammar.g4``
|
||||
This will generate the files, which you can then integrate in your project.
|
||||
This is just a quick start. The tool has many useful options to control generation, please refer to its documentation.
|
||||
|
||||
This is just a quick start. The tool has many useful options to control generation, please refer to its documentation.
|
||||
|
||||
### Step 5: Add a reference to the ANTLR runtime in your project
|
||||
|
||||
The Antlr 4 standard runtime for C# is now available from NuGet.
|
||||
We trust that you know how to do add NuGet references to your project :-).
|
||||
The package id is Antlr.4.Runtime. We do not support other packages.
|
||||
|
||||
|
||||
### Step 6: You're done!
|
||||
|
||||
Of course, the generated code is not going to meet your requirement by magic.
|
||||
|
|
@ -60,10 +65,10 @@ While the latter works, it is no longer the recommended approach, because it is
|
|||
|
||||
See the web site for examples of using the generated code.
|
||||
|
||||
To learn more about ANTLR 4, read the book.
|
||||
To learn more about ANTLR 4, read [the book](http://a.co/2n4rJlb).
|
||||
|
||||
### Visual Studio integration
|
||||
|
||||
If you require tighter Visual Studio integration, you can use the tools from Tunnel Vision Labs.
|
||||
If you require tighter Visual Studio integration, you can use the tools from [Tunnel Vision Labs](http://tunnelvisionlabs.com/).
|
||||
(please note however that they use a different tool and runtime)
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
<OutputType>Library</OutputType>
|
||||
<AppDesignerFolder>Properties</AppDesignerFolder>
|
||||
<RootNamespace>Antlr4.Runtime</RootNamespace>
|
||||
<AssemblyName>Antlr4.Runtime</AssemblyName>
|
||||
<AssemblyName>Antlr4.Runtime.Standard</AssemblyName>
|
||||
<TargetFrameworkVersion>v3.5</TargetFrameworkVersion>
|
||||
<FileAlignment>512</FileAlignment>
|
||||
<BaseIntermediateOutputPath>obj\net20\</BaseIntermediateOutputPath>
|
||||
|
|
@ -21,7 +21,7 @@
|
|||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>full</DebugType>
|
||||
<Optimize>false</Optimize>
|
||||
<OutputPath>bin\net35\Debug\</OutputPath>
|
||||
<OutputPath>lib\Debug</OutputPath>
|
||||
<DefineConstants>DEBUG;TRACE;NET35PLUS</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
|
|
@ -29,7 +29,7 @@
|
|||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
|
||||
<Optimize>true</Optimize>
|
||||
<OutputPath>bin\net35\Release\</OutputPath>
|
||||
<OutputPath>lib\Release</OutputPath>
|
||||
<DefineConstants>TRACE;NET35PLUS</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
<OutputType>Library</OutputType>
|
||||
<AppDesignerFolder>Properties</AppDesignerFolder>
|
||||
<RootNamespace>Antlr4.Runtime</RootNamespace>
|
||||
<AssemblyName>Antlr4.Runtime</AssemblyName>
|
||||
<AssemblyName>Antlr4.Runtime.Standard</AssemblyName>
|
||||
<TargetFrameworkVersion>v3.5</TargetFrameworkVersion>
|
||||
<FileAlignment>512</FileAlignment>
|
||||
<BaseIntermediateOutputPath>obj\net35\</BaseIntermediateOutputPath>
|
||||
|
|
|
|||
|
|
@ -42,8 +42,8 @@ using System.Runtime.InteropServices;
|
|||
// You can specify all the values or you can default the Build and Revision Numbers
|
||||
// by using the '*' as shown below:
|
||||
// [assembly: AssemblyVersion("1.0.*")]
|
||||
[assembly: AssemblyVersion("4.5.3.0")]
|
||||
[assembly: AssemblyVersion("4.6.0")]
|
||||
#if !COMPACT
|
||||
[assembly: AssemblyFileVersion("4.5.3.0")]
|
||||
[assembly: AssemblyInformationalVersion("4.5.3.0")]
|
||||
[assembly: AssemblyFileVersion("4.6.0")]
|
||||
[assembly: AssemblyInformationalVersion("4.6.0")]
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -5,8 +5,6 @@ VisualStudioVersion = 12.0.30110.0
|
|||
MinimumVisualStudioVersion = 10.0.40219.1
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{47C0086D-577C-43DA-ADC7-544F27656E45}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
..\..\tool\resources\org\antlr\v4\tool\templates\codegen\CSharp\CSharp.stg = ..\..\tool\resources\org\antlr\v4\tool\templates\codegen\CSharp\CSharp.stg
|
||||
..\..\tool\src\org\antlr\v4\codegen\CSharpTarget.java = ..\..\tool\src\org\antlr\v4\codegen\CSharpTarget.java
|
||||
..\..\Readme.md = ..\..\Readme.md
|
||||
EndProjectSection
|
||||
EndProject
|
||||
|
|
|
|||
|
|
@ -0,0 +1,24 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
|
||||
<metadata>
|
||||
<id>Antlr4.Runtime.Standard</id>
|
||||
<version>4.6.0</version>
|
||||
<language>en-us</language>
|
||||
<title>ANTLR 4 Standard Runtime</title>
|
||||
<description>The standard C# ANTLR 4 runtime from the ANTLR Organization</description>
|
||||
<summary>The runtime library for parsers generated by the C# target of the standard ANTLR 4 tool.</summary>
|
||||
<authors>Eric Vergnaud, Terence Parr, Sam Harwell</authors>
|
||||
<owners>The ANTLR Organization</owners>
|
||||
<releaseNotes>https://github.com/antlr/antlr4/releases</releaseNotes>
|
||||
<requireLicenseAcceptance>true</requireLicenseAcceptance>
|
||||
<licenseUrl>https://github.com/antlr/antlr4/blob/master/LICENSE.txt</licenseUrl>
|
||||
<projectUrl>https://github.com/antlr/antlr4</projectUrl>
|
||||
<iconUrl>https://raw.github.com/antlr/website-antlr4/master/images/icons/antlr.png</iconUrl>
|
||||
<copyright>Copyright (c) 2012-2016 The ANTLR Project. All rights reserved.</copyright>
|
||||
<tags>antlr parsing grammar</tags>
|
||||
<dependencies />
|
||||
</metadata>
|
||||
<files>
|
||||
<file src="Antlr4.Runtime/lib/Release/Antlr4.Runtime.Standard.dll" target="lib/net35/"/>
|
||||
</files>
|
||||
</package>
|
||||
|
|
@ -1 +1 @@
|
|||
4.5.4
|
||||
4.6
|
||||
|
|
|
|||
|
|
@ -19,11 +19,11 @@
|
|||
#
|
||||
# next make a build folder e.g. ~/buildfolder/
|
||||
# from the buildfolder, run cmake ~/srcfolder; make
|
||||
#
|
||||
#
|
||||
###############################################################
|
||||
# # minimum required CMAKE version
|
||||
# CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2 FATAL_ERROR)
|
||||
#
|
||||
#
|
||||
# LIST( APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake )
|
||||
#
|
||||
# # compiler must be 11 or 14
|
||||
|
|
@ -49,7 +49,7 @@
|
|||
# add_executable(demo main.cpp ${antlr4cpp_src_files_antlrcpptest})
|
||||
# add_dependencies(demo antlr4cpp antlr4cpp_generation_antlrcpptest)
|
||||
# target_link_libraries(demo antlr4-runtime)
|
||||
#
|
||||
#
|
||||
###############################################################
|
||||
|
||||
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2)
|
||||
|
|
@ -65,8 +65,8 @@ set(ANTLR4CPP_EXTERNAL_ROOT ${CMAKE_BINARY_DIR}/externals/antlr4cpp)
|
|||
|
||||
# external repository
|
||||
# GIT_REPOSITORY https://github.com/antlr/antlr4.git
|
||||
set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/DanMcLaughlin/antlr4.git")
|
||||
set(ANTLR4CPP_EXTERNAL_TAG "v4.5.4")
|
||||
set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/antlr/antlr4.git")
|
||||
set(ANTLR4CPP_EXTERNAL_TAG "4.6-rc1")
|
||||
|
||||
if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}")
|
||||
message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}")
|
||||
|
|
@ -81,13 +81,13 @@ endif()
|
|||
# set(ANTLR4CPP_JAR_LOCATION ${ANTLR4CPP_EXTERNAL_ROOT}/${ANTLR4CPP_JAR_NAME})
|
||||
#
|
||||
# !TODO! Ensure Antlr tool available - copy from internet
|
||||
#
|
||||
#
|
||||
# # !TODO! this shold be calculated based on the tags
|
||||
# if (NOT ANTLR4CPP_JAR_NAME)
|
||||
# # default location to find antlr Java binary
|
||||
# set(ANTLR4CPP_JAR_NAME antlr4-4.5.4-SNAPSHOT.jar)
|
||||
# endif()
|
||||
#
|
||||
#
|
||||
# if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}")
|
||||
# # download Java tool if not installed
|
||||
# ExternalProject_ADD(
|
||||
|
|
@ -162,20 +162,20 @@ endforeach(src_path)
|
|||
|
||||
set(ANTLR4CPP_LIBS "${INSTALL_DIR}/lib")
|
||||
|
||||
# antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so
|
||||
# antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a
|
||||
# antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so
|
||||
# antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a
|
||||
|
||||
############ Generate runtime #################
|
||||
# macro to add dependencies to target
|
||||
#
|
||||
# Param 1 project name
|
||||
# Param 1 project name
|
||||
# Param 1 namespace (postfix for dependencies)
|
||||
# Param 2 Lexer file (full path)
|
||||
# Param 3 Parser File (full path)
|
||||
#
|
||||
# output
|
||||
#
|
||||
# antlr4cpp_src_files_{namespace} - src files for add_executable
|
||||
# antlr4cpp_src_files_{namespace} - src files for add_executable
|
||||
# antlr4cpp_include_dirs_{namespace} - include dir for generated headers
|
||||
# antlr4cpp_generation_{namespace} - for add_dependencies tracking
|
||||
|
||||
|
|
@ -184,18 +184,18 @@ macro(antlr4cpp_process_grammar
|
|||
antlr4cpp_project_namespace
|
||||
antlr4cpp_grammar_lexer
|
||||
antlr4cpp_grammar_parser)
|
||||
|
||||
|
||||
if(EXISTS "${ANTLR4CPP_JAR_LOCATION}")
|
||||
message(STATUS "Found antlr tool: ${ANTLR4CPP_JAR_LOCATION}")
|
||||
else()
|
||||
message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}")
|
||||
endif()
|
||||
|
||||
|
||||
add_custom_target("antlr4cpp_generation_${antlr4cpp_project_namespace}"
|
||||
COMMAND
|
||||
COMMAND
|
||||
${CMAKE_COMMAND} -E make_directory ${ANTLR4CPP_GENERATED_SRC_DIR}
|
||||
COMMAND
|
||||
"${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}"
|
||||
"${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}"
|
||||
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
|
||||
DEPENDS "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}"
|
||||
)
|
||||
|
|
@ -213,9 +213,9 @@ macro(antlr4cpp_process_grammar
|
|||
)
|
||||
endforeach(generated_file)
|
||||
message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} Generated: ${generated_files}")
|
||||
|
||||
|
||||
# export generated include directory
|
||||
set(antlr4cpp_include_dirs_${antlr4cpp_project_namespace} ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace})
|
||||
message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} include: ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}")
|
||||
|
||||
|
||||
endmacro()
|
||||
|
|
|
|||
|
|
@ -3,10 +3,14 @@
|
|||
* can be found in the LICENSE.txt file in the project root.
|
||||
*/
|
||||
|
||||
#include "misc/Interval.h"
|
||||
#include "TokenSource.h"
|
||||
#include "support/StringUtils.h"
|
||||
#include "CharStream.h"
|
||||
#include "Recognizer.h"
|
||||
#include "Vocabulary.h"
|
||||
|
||||
#include "misc/Interval.h"
|
||||
|
||||
#include "support/StringUtils.h"
|
||||
#include "support/CPPUtils.h"
|
||||
|
||||
#include "CommonToken.h"
|
||||
|
|
@ -149,6 +153,10 @@ antlr4::CharStream *CommonToken::getInputStream() const {
|
|||
}
|
||||
|
||||
std::string CommonToken::toString() const {
|
||||
return toString(nullptr);
|
||||
}
|
||||
|
||||
std::string CommonToken::toString(Recognizer *r) const {
|
||||
std::stringstream ss;
|
||||
|
||||
std::string channelStr;
|
||||
|
|
@ -164,8 +172,12 @@ std::string CommonToken::toString() const {
|
|||
txt = "<no text>";
|
||||
}
|
||||
|
||||
std::string typeString = std::to_string(symbolToNumeric(_type));
|
||||
if (r != nullptr)
|
||||
typeString = r->getVocabulary().getDisplayName(_type);
|
||||
|
||||
ss << "[@" << symbolToNumeric(getTokenIndex()) << "," << symbolToNumeric(_start) << ":" << symbolToNumeric(_stop)
|
||||
<< "='" << txt << "',<" << symbolToNumeric(_type) << ">" << channelStr << "," << _line << ":"
|
||||
<< "='" << txt << "',<" << typeString << ">" << channelStr << "," << _line << ":"
|
||||
<< getCharPositionInLine() << "]";
|
||||
|
||||
return ss.str();
|
||||
|
|
|
|||
|
|
@ -150,6 +150,7 @@ namespace antlr4 {
|
|||
|
||||
virtual std::string toString() const override;
|
||||
|
||||
virtual std::string toString(Recognizer *r) const;
|
||||
private:
|
||||
void InitializeInstanceFields();
|
||||
};
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ std::map<std::string, size_t> Recognizer::getTokenTypeMap() {
|
|||
if (iterator != _tokenTypeMapCache.end()) {
|
||||
result = iterator->second;
|
||||
} else {
|
||||
for (size_t i = 0; i < getATN().maxTokenType; ++i) {
|
||||
for (size_t i = 0; i <= getATN().maxTokenType; ++i) {
|
||||
std::string literalName = vocabulary.getLiteralName(i);
|
||||
if (!literalName.empty()) {
|
||||
result[literalName] = i;
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
|
||||
using namespace antlr4;
|
||||
|
||||
const std::string RuntimeMetaData::VERSION = "4.5.4";
|
||||
const std::string RuntimeMetaData::VERSION = "4.6";
|
||||
|
||||
std::string RuntimeMetaData::getRuntimeVersion() {
|
||||
return VERSION;
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
|
||||
#pragma once
|
||||
|
||||
#include "antlr4-common.h"
|
||||
|
||||
namespace antlr4 {
|
||||
|
||||
/// The default mechanism for creating tokens. It's used by default in Lexer and
|
||||
|
|
|
|||
|
|
@ -78,7 +78,6 @@
|
|||
#else
|
||||
#define ANTLR4CPP_PUBLIC
|
||||
#endif
|
||||
#define EXPIMP_TEMPLATE
|
||||
#else
|
||||
#define GUID_LIBUUID
|
||||
#if __GNUC__ >= 6
|
||||
|
|
@ -86,7 +85,6 @@
|
|||
#else
|
||||
#define ANTLR4CPP_PUBLIC
|
||||
#endif
|
||||
#define EXPIMP_TEMPLATE
|
||||
#endif
|
||||
|
||||
#include "support/guid.h"
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ namespace antlr4 {
|
|||
namespace misc {
|
||||
|
||||
// Helpers to convert certain unsigned symbols (e.g. Token::EOF) to their original numeric value (e.g. -1)
|
||||
// and vice version. This is needed mostly for intervals to keep their original order and for toString()
|
||||
// and vice versa. This is needed mostly for intervals to keep their original order and for toString()
|
||||
// methods to print the original numeric value (e.g. for tests).
|
||||
size_t numericToSymbol(ssize_t v);
|
||||
ssize_t symbolToNumeric(size_t v);
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime</artifactId>
|
||||
|
|
@ -41,6 +41,7 @@
|
|||
<version>1.0</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>deploy</phase>
|
||||
<goals>
|
||||
<goal>dot</goal>
|
||||
</goals>
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "antlr4",
|
||||
"version": "4.6",
|
||||
"version": "4.6.0",
|
||||
"description": "JavaScript runtime for ANTLR4",
|
||||
"main": "src/antlr4/index.js",
|
||||
"repository": "antlr/antlr4.git",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
publishing to npm
|
||||
=================
|
||||
The JavaScript runtime is published on npm.
|
||||
There is nothing to build since JavaScript is based on source code only.
|
||||
The JavaScript itself is tested using npm, so assumption is npm is already installed.
|
||||
The current npm version used is 3.10.9.
|
||||
The publishing itself relies on the information in src/antlr4/package.json.
|
||||
To publish, from Terminal, you need to navigate to the src/ directory.
|
||||
Once there, login: npm login
|
||||
Then npm publish antlr4
|
||||
That's it!
|
||||
|
||||
|
|
@ -484,14 +484,14 @@ class DefaultErrorStrategy(ErrorStrategy):
|
|||
expecting = self.getExpectedTokens(recognizer)
|
||||
expectedTokenType = expecting[0] # get any element
|
||||
if expectedTokenType==Token.EOF:
|
||||
tokenText = "<missing EOF>"
|
||||
tokenText = u"<missing EOF>"
|
||||
else:
|
||||
name = None
|
||||
if expectedTokenType < len(recognizer.literalNames):
|
||||
name = recognizer.literalNames[expectedTokenType]
|
||||
if name is None and expectedTokenType < len(recognizer.symbolicNames):
|
||||
name = recognizer.symbolicNames[expectedTokenType]
|
||||
tokenText = "<missing " + str(name) + ">"
|
||||
tokenText = u"<missing " + unicode(name) + u">"
|
||||
current = currentSymbol
|
||||
lookback = recognizer.getTokenStream().LT(-1)
|
||||
if current.type==Token.EOF and lookback is not None:
|
||||
|
|
|
|||
|
|
@ -1,59 +0,0 @@
|
|||
# file GENERATED by distutils, do NOT edit
|
||||
LICENSE.txt
|
||||
README.txt
|
||||
RELEASE-4.5.txt
|
||||
setup.py
|
||||
src/antlr4/BufferedTokenStream.py
|
||||
src/antlr4/CommonTokenFactory.py
|
||||
src/antlr4/CommonTokenStream.py
|
||||
src/antlr4/FileStream.py
|
||||
src/antlr4/InputStream.py
|
||||
src/antlr4/IntervalSet.py
|
||||
src/antlr4/LL1Analyzer.py
|
||||
src/antlr4/Lexer.py
|
||||
src/antlr4/ListTokenSource.py
|
||||
src/antlr4/Parser.py
|
||||
src/antlr4/ParserInterpreter.py
|
||||
src/antlr4/ParserRuleContext.py
|
||||
src/antlr4/PredictionContext.py
|
||||
src/antlr4/Recognizer.py
|
||||
src/antlr4/RuleContext.py
|
||||
src/antlr4/Token.py
|
||||
src/antlr4/Utils.py
|
||||
src/antlr4/__init__.py
|
||||
src/antlr4/atn/ATN.py
|
||||
src/antlr4/atn/ATNConfig.py
|
||||
src/antlr4/atn/ATNConfigSet.py
|
||||
src/antlr4/atn/ATNDeserializationOptions.py
|
||||
src/antlr4/atn/ATNDeserializer.py
|
||||
src/antlr4/atn/ATNSimulator.py
|
||||
src/antlr4/atn/ATNState.py
|
||||
src/antlr4/atn/ATNType.py
|
||||
src/antlr4/atn/LexerATNSimulator.py
|
||||
src/antlr4/atn/LexerAction.py
|
||||
src/antlr4/atn/LexerActionExecutor.py
|
||||
src/antlr4/atn/ParserATNSimulator.py
|
||||
src/antlr4/atn/PredictionMode.py
|
||||
src/antlr4/atn/SemanticContext.py
|
||||
src/antlr4/atn/Transition.py
|
||||
src/antlr4/atn/__init__.py
|
||||
src/antlr4/dfa/DFA.py
|
||||
src/antlr4/dfa/DFASerializer.py
|
||||
src/antlr4/dfa/DFAState.py
|
||||
src/antlr4/dfa/__init__.py
|
||||
src/antlr4/error/DiagnosticErrorListener.py
|
||||
src/antlr4/error/ErrorListener.py
|
||||
src/antlr4/error/ErrorStrategy.py
|
||||
src/antlr4/error/Errors.py
|
||||
src/antlr4/error/__init__.py
|
||||
src/antlr4/tree/Chunk.py
|
||||
src/antlr4/tree/ParseTreeMatch.py
|
||||
src/antlr4/tree/ParseTreePattern.py
|
||||
src/antlr4/tree/ParseTreePatternMatcher.py
|
||||
src/antlr4/tree/RuleTagToken.py
|
||||
src/antlr4/tree/TokenTagToken.py
|
||||
src/antlr4/tree/Tree.py
|
||||
src/antlr4/tree/Trees.py
|
||||
src/antlr4/tree/__init__.py
|
||||
src/antlr4/xpath/XPath.py
|
||||
src/antlr4/xpath/__init__.py
|
||||
|
|
@ -473,8 +473,6 @@ public class BufferedTokenStream: TokenStream {
|
|||
|
||||
|
||||
public func getText() throws -> String {
|
||||
try lazyInit()
|
||||
try fill()
|
||||
return try getText(Interval.of(0, size() - 1))
|
||||
}
|
||||
|
||||
|
|
@ -485,7 +483,7 @@ public class BufferedTokenStream: TokenStream {
|
|||
if start < 0 || stop < 0 {
|
||||
return ""
|
||||
}
|
||||
try lazyInit()
|
||||
try fill()
|
||||
if stop >= tokens.count {
|
||||
stop = tokens.count - 1
|
||||
}
|
||||
|
|
|
|||
|
|
@ -255,9 +255,13 @@ public class CommonToken: WritableToken {
|
|||
}
|
||||
|
||||
public var description: String {
|
||||
return toString(nil)
|
||||
}
|
||||
|
||||
public func toString(_ r: Recognizer<ATNSimulator>?) -> String {
|
||||
var channelStr: String = ""
|
||||
if channel > 0 {
|
||||
channelStr = "channel=\(channel)"
|
||||
channelStr = ",channel=\(channel)"
|
||||
}
|
||||
var txt: String
|
||||
if let tokenText = getText() {
|
||||
|
|
@ -267,15 +271,19 @@ public class CommonToken: WritableToken {
|
|||
} else {
|
||||
txt = "<no text>"
|
||||
}
|
||||
let desc: StringBuilder = StringBuilder()
|
||||
desc.append("[@\(getTokenIndex()),")
|
||||
desc.append("\(start):\(stop)='\(txt)',")
|
||||
desc.append("<\(type)>\(channelStr),")
|
||||
desc.append("\(line):\(getCharPositionInLine())]")
|
||||
|
||||
return desc.toString()
|
||||
var typeString = "\(type)"
|
||||
if let r = r {
|
||||
typeString = r.getVocabulary().getDisplayName(type);
|
||||
}
|
||||
return "[@"+getTokenIndex()+","+start+":"+stop+"='"+txt+"',<"+typeString+">"+channelStr+","+line+":"+getCharPositionInLine()+"]"
|
||||
// let desc: StringBuilder = StringBuilder()
|
||||
// desc.append("[@\(getTokenIndex()),")
|
||||
// desc.append("\(start):\(stop)='\(txt)',")
|
||||
// desc.append("<\(typeString)>\(channelStr),")
|
||||
// desc.append("\(line):\(getCharPositionInLine())]")
|
||||
//
|
||||
// return desc.toString()
|
||||
}
|
||||
|
||||
public var visited: Bool {
|
||||
get {
|
||||
return _visited
|
||||
|
|
|
|||
|
|
@ -79,6 +79,18 @@ open class ParserRuleContext: RuleContext {
|
|||
|
||||
self.start = ctx.start
|
||||
self.stop = ctx.stop
|
||||
|
||||
// copy any error nodes to alt label node
|
||||
if ctx.children != nil{
|
||||
self.children = Array<ParseTree>()
|
||||
// reset parent pointer for any error nodes
|
||||
for child: ParseTree in ctx.children! {
|
||||
if child is ErrorNode{
|
||||
self.children?.append(child)
|
||||
( (child as! ErrorNode)).parent = self
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public init(_ parent: ParserRuleContext?, _ invokingStateNumber: Int) {
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ open class Recognizer<ATNInterpreter:ATNSimulator> {
|
|||
if result == nil {
|
||||
result = Dictionary<String, Int>()
|
||||
let length = self.getATN().maxTokenType
|
||||
for i in 0..<length {
|
||||
for i in 0...length {
|
||||
let literalName: String? = vocabulary.getLiteralName(i)
|
||||
if literalName != nil {
|
||||
result![literalName!] = i
|
||||
|
|
|
|||
|
|
@ -232,13 +232,20 @@
|
|||
* both SLL and LL parsing. Erroneous input will therefore require 2 passes over
|
||||
* the input.</p>
|
||||
*/
|
||||
import Foundation
|
||||
|
||||
open class ParserATNSimulator: ATNSimulator {
|
||||
public let debug: Bool = false
|
||||
public let debug_list_atn_decisions: Bool = false
|
||||
public let dfa_debug: Bool = false
|
||||
public let retry_debug: Bool = false
|
||||
|
||||
/** Just in case this optimization is bad, add an ENV variable to turn it off */
|
||||
public static let TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT: Bool = {
|
||||
if let value = ProcessInfo.processInfo.environment["TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT"] {
|
||||
return NSString(string: value).boolValue
|
||||
}
|
||||
return false
|
||||
}()
|
||||
internal final var parser: Parser
|
||||
|
||||
public final var decisionToDFA: [DFA]
|
||||
|
|
@ -1441,6 +1448,10 @@ open class ParserATNSimulator: ATNSimulator {
|
|||
}
|
||||
let length = p.getNumberOfTransitions()
|
||||
for i in 0..<length {
|
||||
if i == 0 &&
|
||||
canDropLoopEntryEdgeInLeftRecursiveRule(config) {
|
||||
continue
|
||||
}
|
||||
let t: Transition = p.transition(i)
|
||||
let continueCollecting: Bool =
|
||||
!(t is ActionTransition) && collectPredicates
|
||||
|
|
@ -1506,7 +1517,168 @@ open class ParserATNSimulator: ATNSimulator {
|
|||
//print("That took: "+(finishTime-startTime)+ " ms");
|
||||
}
|
||||
|
||||
|
||||
/** Implements first-edge (loop entry) elimination as an optimization
|
||||
* during closure operations. See antlr/antlr4#1398.
|
||||
*
|
||||
* The optimization is to avoid adding the loop entry config when
|
||||
* the exit path can only lead back to the same
|
||||
* StarLoopEntryState after popping context at the rule end state
|
||||
* (traversing only epsilon edges, so we're still in closure, in
|
||||
* this same rule).
|
||||
*
|
||||
* We need to detect any state that can reach loop entry on
|
||||
* epsilon w/o exiting rule. We don't have to look at FOLLOW
|
||||
* links, just ensure that all stack tops for config refer to key
|
||||
* states in LR rule.
|
||||
*
|
||||
* To verify we are in the right situation we must first check
|
||||
* closure is at a StarLoopEntryState generated during LR removal.
|
||||
* Then we check that each stack top of context is a return state
|
||||
* from one of these cases:
|
||||
*
|
||||
* 1. 'not' expr, '(' type ')' expr. The return state points at loop entry state
|
||||
* 2. expr op expr. The return state is the block end of internal block of (...)*
|
||||
* 3. 'between' expr 'and' expr. The return state of 2nd expr reference.
|
||||
* That state points at block end of internal block of (...)*.
|
||||
* 4. expr '?' expr ':' expr. The return state points at block end,
|
||||
* which points at loop entry state.
|
||||
*
|
||||
* If any is true for each stack top, then closure does not add a
|
||||
* config to the current config set for edge[0], the loop entry branch.
|
||||
*
|
||||
* Conditions fail if any context for the current config is:
|
||||
*
|
||||
* a. empty (we'd fall out of expr to do a global FOLLOW which could
|
||||
* even be to some weird spot in expr) or,
|
||||
* b. lies outside of expr or,
|
||||
* c. lies within expr but at a state not the BlockEndState
|
||||
* generated during LR removal
|
||||
*
|
||||
* Do we need to evaluate predicates ever in closure for this case?
|
||||
*
|
||||
* No. Predicates, including precedence predicates, are only
|
||||
* evaluated when computing a DFA start state. I.e., only before
|
||||
* the lookahead (but not parser) consumes a token.
|
||||
*
|
||||
* There are no epsilon edges allowed in LR rule alt blocks or in
|
||||
* the "primary" part (ID here). If closure is in
|
||||
* StarLoopEntryState any lookahead operation will have consumed a
|
||||
* token as there are no epsilon-paths that lead to
|
||||
* StarLoopEntryState. We do not have to evaluate predicates
|
||||
* therefore if we are in the generated StarLoopEntryState of a LR
|
||||
* rule. Note that when making a prediction starting at that
|
||||
* decision point, decision d=2, compute-start-state performs
|
||||
* closure starting at edges[0], edges[1] emanating from
|
||||
* StarLoopEntryState. That means it is not performing closure on
|
||||
* StarLoopEntryState during compute-start-state.
|
||||
*
|
||||
* How do we know this always gives same prediction answer?
|
||||
*
|
||||
* Without predicates, loop entry and exit paths are ambiguous
|
||||
* upon remaining input +b (in, say, a+b). Either paths lead to
|
||||
* valid parses. Closure can lead to consuming + immediately or by
|
||||
* falling out of this call to expr back into expr and loop back
|
||||
* again to StarLoopEntryState to match +b. In this special case,
|
||||
* we choose the more efficient path, which is to take the bypass
|
||||
* path.
|
||||
*
|
||||
* The lookahead language has not changed because closure chooses
|
||||
* one path over the other. Both paths lead to consuming the same
|
||||
* remaining input during a lookahead operation. If the next token
|
||||
* is an operator, lookahead will enter the choice block with
|
||||
* operators. If it is not, lookahead will exit expr. Same as if
|
||||
* closure had chosen to enter the choice block immediately.
|
||||
*
|
||||
* Closure is examining one config (some loopentrystate, some alt,
|
||||
* context) which means it is considering exactly one alt. Closure
|
||||
* always copies the same alt to any derived configs.
|
||||
*
|
||||
* How do we know this optimization doesn't mess up precedence in
|
||||
* our parse trees?
|
||||
*
|
||||
* Looking through expr from left edge of stat only has to confirm
|
||||
* that an input, say, a+b+c; begins with any valid interpretation
|
||||
* of an expression. The precedence actually doesn't matter when
|
||||
* making a decision in stat seeing through expr. It is only when
|
||||
* parsing rule expr that we must use the precedence to get the
|
||||
* right interpretation and, hence, parse tree.
|
||||
*
|
||||
* @since 4.6
|
||||
*/
|
||||
internal func canDropLoopEntryEdgeInLeftRecursiveRule(_ config: ATNConfig) -> Bool {
|
||||
if ParserATNSimulator.TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT {
|
||||
return false
|
||||
}
|
||||
let p: ATNState = config.state
|
||||
guard let configContext = config.context else {
|
||||
return false
|
||||
}
|
||||
// First check to see if we are in StarLoopEntryState generated during
|
||||
// left-recursion elimination. For efficiency, also check if
|
||||
// the context has an empty stack case. If so, it would mean
|
||||
// global FOLLOW so we can't perform optimization
|
||||
if p.getStateType() != ATNState.STAR_LOOP_ENTRY ||
|
||||
!( (p as! StarLoopEntryState)).precedenceRuleDecision || // Are we the special loop entry/exit state?
|
||||
configContext.isEmpty() || // If SLL wildcard
|
||||
configContext.hasEmptyPath(){
|
||||
return false
|
||||
}
|
||||
|
||||
// Require all return states to return back to the same rule
|
||||
// that p is in.
|
||||
let numCtxs: Int = configContext.size()
|
||||
for i in 0 ..< numCtxs { // for each stack context
|
||||
let returnState: ATNState = atn.states[configContext.getReturnState(i)]!
|
||||
if returnState.ruleIndex != p.ruleIndex
|
||||
{return false}
|
||||
}
|
||||
|
||||
let decisionStartState: BlockStartState = (p.transition(0).target as! BlockStartState)
|
||||
let blockEndStateNum: Int = decisionStartState.endState!.stateNumber
|
||||
let blockEndState: BlockEndState = (atn.states[blockEndStateNum] as! BlockEndState)
|
||||
|
||||
// Verify that the top of each stack context leads to loop entry/exit
|
||||
// state through epsilon edges and w/o leaving rule.
|
||||
for i in 0 ..< numCtxs { // for each stack context
|
||||
let returnStateNumber: Int = configContext.getReturnState(i)
|
||||
let returnState: ATNState = atn.states[returnStateNumber]!
|
||||
// all states must have single outgoing epsilon edge
|
||||
if returnState.getNumberOfTransitions() != 1 || !returnState.transition(0).isEpsilon(){
|
||||
return false
|
||||
}
|
||||
// Look for prefix op case like 'not expr', (' type ')' expr
|
||||
let returnStateTarget: ATNState = returnState.transition(0).target
|
||||
if returnState.getStateType() == ATNState.BLOCK_END &&
|
||||
returnStateTarget == p {
|
||||
continue
|
||||
}
|
||||
// Look for 'expr op expr' or case where expr's return state is block end
|
||||
// of (...)* internal block; the block end points to loop back
|
||||
// which points to p but we don't need to check that
|
||||
if returnState == blockEndState{
|
||||
continue
|
||||
}
|
||||
// Look for ternary expr ? expr : expr. The return state points at block end,
|
||||
// which points at loop entry state
|
||||
if returnStateTarget == blockEndState{
|
||||
continue
|
||||
}
|
||||
// Look for complex prefix 'between expr and expr' case where 2nd expr's
|
||||
// return state points at block end state of (...)* internal block
|
||||
if returnStateTarget.getStateType() == ATNState.BLOCK_END &&
|
||||
returnStateTarget.getNumberOfTransitions() == 1 &&
|
||||
returnStateTarget.transition(0).isEpsilon() &&
|
||||
returnStateTarget.transition(0).target == p{
|
||||
continue
|
||||
}
|
||||
|
||||
// anything else ain't conforming
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
open func getRuleName(_ index: Int) -> String {
|
||||
if index >= 0 {
|
||||
return parser.getRuleNames()[index]
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-tool-testsuite</artifactId>
|
||||
<name>ANTLR 4 Tool Tests</name>
|
||||
|
|
|
|||
|
|
@ -320,4 +320,21 @@ public class TestSymbolIssues extends BaseJavaToolTest {
|
|||
|
||||
testErrors(test, false);
|
||||
}
|
||||
|
||||
@Test public void testCharsCollision() throws Exception {
|
||||
String[] test = {
|
||||
"lexer grammar L;\n" +
|
||||
"TOKEN_RANGE: [aa-f];\n" +
|
||||
"TOKEN_RANGE_2: [A-FD-J];\n" +
|
||||
"TOKEN_RANGE_3: 'Z' | 'K'..'R' | 'O'..'V';\n" +
|
||||
"TOKEN_RANGE_4: 'g'..'l' | [g-l];\n", // Handling in ATNOptimizer.
|
||||
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:2:18: chars \"a-f\" used multiple times in set [aa-f]\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:3:18: chars \"D-J\" used multiple times in set [A-FD-J]\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:4:13: chars \"O-V\" used multiple times in set 'Z' | 'K'..'R' | 'O'..'V'\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4::: chars \"g-l\" used multiple times in set [g-l]\n"
|
||||
};
|
||||
|
||||
testErrors(test, false);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -487,11 +487,39 @@ public class TestToolSyntaxErrors extends BaseJavaToolTest {
|
|||
"Error3: '';\n" +
|
||||
"NotError: ' ';";
|
||||
String expected =
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:2:8: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:2:16: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:3:8: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:4:15: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:5:8: string literals cannot be empty\n";
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:2:8: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:2:16: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:3:8: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:4:15: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:5:8: string literals and sets cannot be empty: ''\n";
|
||||
|
||||
String[] pair = new String[] {
|
||||
grammar,
|
||||
expected
|
||||
};
|
||||
|
||||
super.testErrors(pair, true);
|
||||
}
|
||||
|
||||
@Test public void testInvalidCharSetAndRange() {
|
||||
String grammar =
|
||||
"lexer grammar Test;\n" +
|
||||
"INVALID_RANGE: 'GH'..'LM';\n" +
|
||||
"INVALID_RANGE_2: 'F'..'A' | 'Z';\n" +
|
||||
"VALID_STRING_LITERALS: '\\u1234' | '\\t' | [\\-\\]];\n" +
|
||||
"INVALID_CHAR_SET: [f-az][];\n" +
|
||||
"INVALID_CHAR_SET_2: [\\u24\\uA2][\\u24];\n" + //https://github.com/antlr/antlr4/issues/1077
|
||||
"INVALID_CHAR_SET_3: [\\t\\{];";
|
||||
|
||||
String expected =
|
||||
"error(" + ErrorType.INVALID_LITERAL_IN_LEXER_SET.code + "): Test.g4:2:23: multi-character literals are not allowed in lexer sets: 'GH'\n" +
|
||||
"error(" + ErrorType.INVALID_LITERAL_IN_LEXER_SET.code + "): Test.g4:2:29: multi-character literals are not allowed in lexer sets: 'LM'\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:3:26: string literals and sets cannot be empty: 'F'..'A'\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:5:23: string literals and sets cannot be empty: [f-a]\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:5:29: string literals and sets cannot be empty: []\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:6:23: invalid escape sequence\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:6:33: invalid escape sequence\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:7:23: invalid escape sequence\n";
|
||||
|
||||
String[] pair = new String[] {
|
||||
grammar,
|
||||
|
|
|
|||
57
tool/pom.xml
57
tool/pom.xml
|
|
@ -9,40 +9,40 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4</artifactId>
|
||||
<name>ANTLR 4 Tool</name>
|
||||
<url>http://www.antlr.org</url>
|
||||
<description>The ANTLR 4 grammar compiler.</description>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-runtime</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr-runtime</artifactId>
|
||||
<version>3.5.2</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>ST4</artifactId>
|
||||
<version>4.0.8</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.abego.treelayout</groupId>
|
||||
<artifactId>org.abego.treelayout.core</artifactId>
|
||||
<version>1.0.3</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>javax.json</groupId>
|
||||
<artifactId>javax.json-api</artifactId>
|
||||
<version>1.0</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-runtime</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr-runtime</artifactId>
|
||||
<version>3.5.2</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>ST4</artifactId>
|
||||
<version>4.0.8</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.abego.treelayout</groupId>
|
||||
<artifactId>org.abego.treelayout.core</artifactId>
|
||||
<version>1.0.3</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.glassfish</groupId>
|
||||
<artifactId>javax.json</artifactId>
|
||||
<version>1.0.4</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<sourceDirectory>src</sourceDirectory>
|
||||
|
|
@ -73,6 +73,7 @@
|
|||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>deploy</phase>
|
||||
<goals>
|
||||
<goal>javadoc</goal>
|
||||
</goals>
|
||||
|
|
|
|||
|
|
@ -720,6 +720,7 @@ if ( <if(invert)><m.varName> \<= 0 || <else>!<endif>(<expr>) ) {
|
|||
<if(m.labels)><m.labels:{l | <labelref(l)> = }><endif>ErrorHandler.RecoverInline(this);
|
||||
}
|
||||
else {
|
||||
ErrorHandler.ReportMatch(this);
|
||||
Consume();
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -43,7 +43,7 @@ cppTypeInitMap ::= [
|
|||
LexerHeader(lexer, atn, actionFuncs, sempredFuncs, superClass = {antlr4::Lexer}) ::= <<
|
||||
<namedActions.context>
|
||||
|
||||
class <lexer.name> : public <superClass> {
|
||||
class <file.exportMacro> <lexer.name> : public <superClass> {
|
||||
public:
|
||||
<if (lexer.tokens)>
|
||||
enum {
|
||||
|
|
@ -266,7 +266,7 @@ bool <if (r.factory.g.lexer)><lexer.name><else><parser.name><endif>::<r.name>Sem
|
|||
ParserHeader(parser, funcs, atn, sempredFuncs, superClass = {antlr4::Parser}) ::= <<
|
||||
<namedActions.context>
|
||||
|
||||
class <parser.name> : public <superClass> {
|
||||
class <file.exportMacro> <parser.name> : public <superClass> {
|
||||
public:
|
||||
<if (parser.tokens)>
|
||||
enum {
|
||||
|
|
@ -526,7 +526,7 @@ LeftRecursiveRuleFunction(currentRule, args, code, locals, ruleCtx, altLabelCtxs
|
|||
>>
|
||||
|
||||
StructDeclHeader(struct, ctorAttrs, attrs, getters, dispatchMethods, interfaces, extensionMembers) ::= <<
|
||||
class <struct.name> : public <if (contextSuperClass)><contextSuperClass><else>antlr4::ParserRuleContext<endif><if(interfaces)>, <interfaces; separator=", "><endif> {
|
||||
class <file.exportMacro> <struct.name> : public <if (contextSuperClass)><contextSuperClass><else>antlr4::ParserRuleContext<endif><if(interfaces)>, <interfaces; separator=", "><endif> {
|
||||
public:
|
||||
<attrs: {a | <a>;}; separator="\n">
|
||||
<if (ctorAttrs)><struct.name>(antlr4::ParserRuleContext *parent, size_t invokingState);<endif>
|
||||
|
|
@ -578,7 +578,7 @@ void <parser.name>::<struct.name>::copyFrom(<struct.name> *ctx) {
|
|||
>>
|
||||
|
||||
AltLabelStructDeclHeader(struct, attrs, getters, dispatchMethods) ::= <<
|
||||
class <struct.name> : public <currentRule.name; format = "cap">Context {
|
||||
class <file.exportMacro> <struct.name> : public <currentRule.name; format = "cap">Context {
|
||||
public:
|
||||
<struct.name>(<currentRule.name; format = "cap">Context *ctx);
|
||||
|
||||
|
|
@ -816,7 +816,9 @@ setState(<m.stateNumber>);
|
|||
<capture>
|
||||
if (<if (invert)><m.varName> == 0 || <m.varName> == Token::EOF || <else>!<endif>(<expr>)) {
|
||||
<if (m.labels)><m.labels: {l | <labelref(l)> = }><endif>_errHandler->recoverInline(this);
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
_errHandler->reportMatch(this);
|
||||
consume();
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ namespace <file.genPackage> {
|
|||
* which can be extended to create a listener which only needs to handle a subset
|
||||
* of the available methods.
|
||||
*/
|
||||
class <file.grammarName>BaseListener : public <file.grammarName>Listener {
|
||||
class <file.exportMacro> <file.grammarName>BaseListener : public <file.grammarName>Listener {
|
||||
public:
|
||||
<namedActions.baselistenerdeclarations>
|
||||
|
||||
|
|
@ -192,7 +192,7 @@ namespace <file.genPackage> {
|
|||
/**
|
||||
* This interface defines an abstract listener for a parse tree produced by <file.parserName>.
|
||||
*/
|
||||
class <file.grammarName>Listener : public antlr4::tree::ParseTreeListener {
|
||||
class <file.exportMacro> <file.grammarName>Listener : public antlr4::tree::ParseTreeListener {
|
||||
public:
|
||||
<namedActions.listenerdeclarations>
|
||||
|
||||
|
|
@ -248,7 +248,7 @@ namespace <file.genPackage> {
|
|||
* This class provides an empty implementation of <file.grammarName>Visitor, which can be
|
||||
* extended to create a visitor which only needs to handle a subset of the available methods.
|
||||
*/
|
||||
class <file.grammarName>BaseVisitor : public <file.grammarName>Visitor {
|
||||
class <file.exportMacro> <file.grammarName>BaseVisitor : public <file.grammarName>Visitor {
|
||||
public:
|
||||
<namedActions.basevisitordeclarations>
|
||||
|
||||
|
|
@ -304,7 +304,7 @@ VisitorFileHeader(file, header, namedActions) ::= <<
|
|||
* This class defines an abstract visitor for a parse tree
|
||||
* produced by <file.parserName>.
|
||||
*/
|
||||
class <file.grammarName>Visitor : public antlr4::tree::AbstractParseTreeVisitor {
|
||||
class <file.exportMacro> <file.grammarName>Visitor : public antlr4::tree::AbstractParseTreeVisitor {
|
||||
public:
|
||||
<namedActions.visitordeclarations>
|
||||
|
||||
|
|
|
|||
|
|
@ -405,7 +405,7 @@ func (p *<parser.name>) <currentRule.name; format="cap">(<currentRule.args:{a |
|
|||
<endif>
|
||||
}
|
||||
}()
|
||||
|
||||
|
||||
<if(currentRule.hasLookaheadBlock)>
|
||||
var _alt int
|
||||
<endif>
|
||||
|
|
@ -808,6 +808,7 @@ var _lt = p.GetTokenStream().LT(1)<! TODO: Should LT be called always like Invok
|
|||
p.GetErrorHandler().RecoverInline(p)
|
||||
<endif>
|
||||
} else {
|
||||
p.GetErrorHandler().ReportMatch(p)
|
||||
p.Consume()
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -643,7 +643,10 @@ setState(<m.stateNumber>);
|
|||
<capture>
|
||||
if ( <if(invert)><m.varName> \<= 0 || <else>!<endif>(<expr>) ) {
|
||||
<if(m.labels)><m.labels:{l | <labelref(l)> = (Token)}><endif>_errHandler.recoverInline(this);
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
if ( _input.LA(1)==Token.EOF ) matchedEOF = true;
|
||||
_errHandler.reportMatch(this);
|
||||
consume();
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -236,7 +236,7 @@ case <index>:
|
|||
* overriding implementation impossible to maintain.
|
||||
*/
|
||||
RuleSempredFunction(r, actions) ::= <<
|
||||
<if(parser)><parser.name><else><lexer.name><endif>.prototype.<r.name>_sempred = function(localctx, predIndex) {
|
||||
<if (r.factory.g.lexer)><lexer.name><else><parser.name><endif>.prototype.<r.name>_sempred = function(localctx, predIndex) {
|
||||
switch(predIndex) {
|
||||
<actions:{index| case <index>:
|
||||
return <actions.(index)>;}; separator="\n">
|
||||
|
|
@ -523,6 +523,7 @@ this.state = <m.stateNumber>;
|
|||
<if(m.labels)><m.labels:{l | <labelref(l)> = }><endif>this._errHandler.recoverInline(this);
|
||||
}
|
||||
else {
|
||||
this._errHandler.reportMatch(this);
|
||||
this.consume();
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -493,6 +493,7 @@ self.state = <m.stateNumber>
|
|||
<if(invert)>if <m.varName> \<= 0 or <expr><else>if not(<expr>)<endif>:
|
||||
<if(m.labels)><m.labels:{l | <labelref(l)> = }><else> <endif>self._errHandler.recoverInline(self)
|
||||
else:
|
||||
self._errHandler.reportMatch(self)
|
||||
self.consume()
|
||||
>>
|
||||
|
||||
|
|
|
|||
|
|
@ -501,6 +501,7 @@ self.state = <m.stateNumber>
|
|||
<if(invert)>if <m.varName> \<= 0 or <expr><else>if not(<expr>)<endif>:
|
||||
<if(m.labels)><m.labels:{l | <labelref(l)> = }><else> <endif>self._errHandler.recoverInline(self)
|
||||
else:
|
||||
self._errHandler.reportMatch(self)
|
||||
self.consume()
|
||||
>>
|
||||
|
||||
|
|
|
|||
|
|
@ -696,7 +696,9 @@ setState(<m.stateNumber>)
|
|||
<capture>
|
||||
if (<if(invert)><m.varName> \<= 0 || <else>!<endif>(<expr>)) {
|
||||
<if(m.labels)><m.labels:{l | <labelref(l)> = }><endif>try _errHandler.recoverInline(self)<if(m.labels)> as Token<endif>
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
_errHandler.reportMatch(self)
|
||||
try consume()
|
||||
}
|
||||
>>
|
||||
|
|
|
|||
|
|
@ -112,6 +112,7 @@ public class Tool {
|
|||
public boolean gen_visitor = false;
|
||||
public boolean gen_dependencies = false;
|
||||
public String genPackage = null;
|
||||
public String exportMacro = null; // C++ specific, need to allow setting a macro to set declspec for generated classes for VC++.
|
||||
public Map<String, String> grammarOptions = null;
|
||||
public boolean warnings_are_errors = false;
|
||||
public boolean longMessages = false;
|
||||
|
|
@ -128,6 +129,7 @@ public class Tool {
|
|||
new Option("gen_visitor", "-visitor", "generate parse tree visitor"),
|
||||
new Option("gen_visitor", "-no-visitor", "don't generate parse tree visitor (default)"),
|
||||
new Option("genPackage", "-package", OptionArgType.STRING, "specify a package/namespace for the generated code"),
|
||||
new Option("exportMacro", "-export-macro", OptionArgType.STRING, "C++ only, specify a macro for import/export of generated classes"),
|
||||
new Option("gen_dependencies", "-depend", "generate file dependencies"),
|
||||
new Option("", "-D<option>=value", "set/override a grammar-level option"),
|
||||
new Option("warnings_are_errors", "-Werror", "treat warnings as errors"),
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import org.antlr.v4.runtime.atn.SetTransition;
|
|||
import org.antlr.v4.runtime.atn.Transition;
|
||||
import org.antlr.v4.runtime.misc.Interval;
|
||||
import org.antlr.v4.runtime.misc.IntervalSet;
|
||||
import org.antlr.v4.tool.ErrorType;
|
||||
import org.antlr.v4.tool.Grammar;
|
||||
import org.antlr.v4.tool.Rule;
|
||||
|
||||
|
|
@ -94,20 +95,34 @@ public class ATNOptimizer {
|
|||
Transition matchTransition = decision.transition(j).target.transition(0);
|
||||
if (matchTransition instanceof NotSetTransition) {
|
||||
throw new UnsupportedOperationException("Not yet implemented.");
|
||||
} else {
|
||||
matchSet.addAll(matchTransition.label());
|
||||
}
|
||||
IntervalSet set = matchTransition.label();
|
||||
int minElem = set.getMinElement();
|
||||
int maxElem = set.getMaxElement();
|
||||
for (int k = minElem; k <= maxElem; k++) {
|
||||
if (matchSet.contains(k)) {
|
||||
char setMin = (char) set.getMinElement();
|
||||
char setMax = (char) set.getMaxElement();
|
||||
// TODO: Token is missing (i.e. position in source will not be displayed).
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName,
|
||||
null, (char) minElem + "-" + (char) maxElem, "[" + setMin + "-" + setMax + "]");
|
||||
break;
|
||||
}
|
||||
}
|
||||
matchSet.addAll(set);
|
||||
}
|
||||
|
||||
Transition newTransition;
|
||||
if (matchSet.getIntervals().size() == 1) {
|
||||
if (matchSet.size() == 1) {
|
||||
newTransition = new AtomTransition(blockEndState, matchSet.getMinElement());
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
Interval matchInterval = matchSet.getIntervals().get(0);
|
||||
newTransition = new RangeTransition(blockEndState, matchInterval.a, matchInterval.b);
|
||||
}
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
newTransition = new SetTransition(blockEndState, matchSet);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ import org.antlr.v4.tool.LexerGrammar;
|
|||
import org.antlr.v4.tool.Rule;
|
||||
import org.antlr.v4.tool.ast.ActionAST;
|
||||
import org.antlr.v4.tool.ast.GrammarAST;
|
||||
import org.antlr.v4.tool.ast.RangeAST;
|
||||
import org.antlr.v4.tool.ast.TerminalAST;
|
||||
import org.stringtemplate.v4.ST;
|
||||
import org.stringtemplate.v4.STGroup;
|
||||
|
|
@ -253,6 +254,7 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
ATNState right = newState(b);
|
||||
int t1 = CharSupport.getCharValueFromGrammarCharLiteral(a.getText());
|
||||
int t2 = CharSupport.getCharValueFromGrammarCharLiteral(b.getText());
|
||||
checkRange(a, b, t1, t2);
|
||||
left.addTransition(new RangeTransition(right, t1, t2));
|
||||
a.atnState = left;
|
||||
b.atnState = left;
|
||||
|
|
@ -268,7 +270,10 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
if ( t.getType()==ANTLRParser.RANGE ) {
|
||||
int a = CharSupport.getCharValueFromGrammarCharLiteral(t.getChild(0).getText());
|
||||
int b = CharSupport.getCharValueFromGrammarCharLiteral(t.getChild(1).getText());
|
||||
set.add(a, b);
|
||||
if (checkRange((GrammarAST) t.getChild(0), (GrammarAST) t.getChild(1), a, b)) {
|
||||
checkSetCollision(associatedAST, set, a, b);
|
||||
set.add(a,b);
|
||||
}
|
||||
}
|
||||
else if ( t.getType()==ANTLRParser.LEXER_CHAR_SET ) {
|
||||
set.addAll(getSetFromCharSetLiteral(t));
|
||||
|
|
@ -276,12 +281,12 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
else if ( t.getType()==ANTLRParser.STRING_LITERAL ) {
|
||||
int c = CharSupport.getCharValueFromGrammarCharLiteral(t.getText());
|
||||
if ( c != -1 ) {
|
||||
checkSetCollision(associatedAST, set, c);
|
||||
set.add(c);
|
||||
}
|
||||
else {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, t.getToken(), t.getText());
|
||||
|
||||
}
|
||||
}
|
||||
else if ( t.getType()==ANTLRParser.TOKEN_REF ) {
|
||||
|
|
@ -307,6 +312,27 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
return new Handle(left, right);
|
||||
}
|
||||
|
||||
protected boolean checkRange(GrammarAST leftNode, GrammarAST rightNode, int leftValue, int rightValue) {
|
||||
boolean result = true;
|
||||
if (leftValue == -1) {
|
||||
result = false;
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, leftNode.getToken(), leftNode.getText());
|
||||
}
|
||||
if (rightValue == -1) {
|
||||
result = false;
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, rightNode.getToken(), rightNode.getText());
|
||||
}
|
||||
if (!result) return result;
|
||||
|
||||
if (rightValue < leftValue) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, leftNode.parent.getToken(), leftNode.getText() + ".." + rightNode.getText());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** For a lexer, a string is a sequence of char to match. That is,
|
||||
* "fog" is treated as 'f' 'o' 'g' not as a single transition in
|
||||
* the DFA. Machine== o-'f'->o-'o'->o-'g'->o and has n+1 states
|
||||
|
|
@ -315,12 +341,19 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
@Override
|
||||
public Handle stringLiteral(TerminalAST stringLiteralAST) {
|
||||
String chars = stringLiteralAST.getText();
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(chars);
|
||||
int n = chars.length();
|
||||
ATNState left = newState(stringLiteralAST);
|
||||
ATNState right;
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(chars);
|
||||
if (chars == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE,
|
||||
g.fileName, stringLiteralAST.getToken());
|
||||
return new Handle(left, left);
|
||||
}
|
||||
|
||||
int n = chars.length();
|
||||
ATNState prev = left;
|
||||
ATNState right = null;
|
||||
for (int i=0; i<n; i++) {
|
||||
right = null;
|
||||
for (int i = 0; i < n; i++) {
|
||||
right = newState(stringLiteralAST);
|
||||
prev.addTransition(new AtomTransition(right, chars.charAt(i)));
|
||||
prev = right;
|
||||
|
|
@ -342,34 +375,90 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
|
||||
public IntervalSet getSetFromCharSetLiteral(GrammarAST charSetAST) {
|
||||
String chars = charSetAST.getText();
|
||||
chars = chars.substring(1, chars.length()-1);
|
||||
String cset = '"'+ chars +'"';
|
||||
chars = chars.substring(1, chars.length() - 1);
|
||||
String cset = '"' + chars + '"';
|
||||
IntervalSet set = new IntervalSet();
|
||||
|
||||
if (chars.length() == 0) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, charSetAST.getToken(), "[]");
|
||||
return set;
|
||||
}
|
||||
// unescape all valid escape char like \n, leaving escaped dashes as '\-'
|
||||
// so we can avoid seeing them as '-' range ops.
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(cset);
|
||||
// now make x-y become set of char
|
||||
if (chars == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE,
|
||||
g.fileName, charSetAST.getToken());
|
||||
return set;
|
||||
}
|
||||
int n = chars.length();
|
||||
for (int i=0; i< n; i++) {
|
||||
// now make x-y become set of char
|
||||
for (int i = 0; i < n; i++) {
|
||||
int c = chars.charAt(i);
|
||||
if ( c=='\\' && (i+1)<n && chars.charAt(i+1)=='-' ) { // \-
|
||||
if (c == '\\' && i+1 < n && chars.charAt(i+1) == '-') { // \-
|
||||
checkSetCollision(charSetAST, set, '-');
|
||||
set.add('-');
|
||||
i++;
|
||||
}
|
||||
else if ( (i+2)<n && chars.charAt(i+1)=='-' ) { // range x-y
|
||||
else if (i+2 < n && chars.charAt(i+1) == '-') { // range x-y
|
||||
int x = c;
|
||||
int y = chars.charAt(i+2);
|
||||
if ( x<=y ) set.add(x,y);
|
||||
i+=2;
|
||||
if (x <= y) {
|
||||
checkSetCollision(charSetAST, set, x, y);
|
||||
set.add(x,y);
|
||||
}
|
||||
else {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, charSetAST.getToken(), "[" + (char) x + "-" + (char) y + "]");
|
||||
}
|
||||
i += 2;
|
||||
}
|
||||
else {
|
||||
checkSetCollision(charSetAST, set, c);
|
||||
set.add(c);
|
||||
}
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
protected void checkSetCollision(GrammarAST ast, IntervalSet set, int el) {
|
||||
if (set.contains(el)) {
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName, ast.getToken(),
|
||||
(char)el, ast.getText());
|
||||
}
|
||||
}
|
||||
|
||||
protected void checkSetCollision(GrammarAST ast, IntervalSet set, int a, int b) {
|
||||
for (int i = a; i <= b; i++) {
|
||||
if (set.contains(i)) {
|
||||
String setText;
|
||||
if (ast.getChildren() == null) {
|
||||
setText = ast.getText();
|
||||
}
|
||||
else {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (Object child : ast.getChildren()) {
|
||||
if (child instanceof RangeAST) {
|
||||
sb.append(((RangeAST) child).getChild(0).getText());
|
||||
sb.append("..");
|
||||
sb.append(((RangeAST) child).getChild(1).getText());
|
||||
}
|
||||
else {
|
||||
sb.append(((GrammarAST)child).getText());
|
||||
}
|
||||
sb.append(" | ");
|
||||
}
|
||||
sb.replace(sb.length() - 3, sb.length(), "");
|
||||
setText = sb.toString();
|
||||
}
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName, ast.getToken(),
|
||||
(char)a + "-" + (char)b, setText);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Handle tokenRef(TerminalAST node) {
|
||||
// Ref to EOF in lexer yields char transition on -1
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ import java.util.Map;
|
|||
|
||||
public class LexerFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public boolean genListener; // from -listener cmd-line
|
||||
public boolean genVisitor; // from -visitor cmd-line
|
||||
@ModelElement public Lexer lexer;
|
||||
|
|
@ -21,6 +22,7 @@ public class LexerFile extends OutputFile {
|
|||
super(factory, fileName);
|
||||
namedActions = buildNamedActions(factory.getGrammar());
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
genListener = factory.getGrammar().tool.gen_listener;
|
||||
genVisitor = factory.getGrammar().tool.gen_visitor;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ import java.util.Set;
|
|||
*/
|
||||
public class ListenerFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public String grammarName;
|
||||
public String parserName;
|
||||
/**
|
||||
|
|
@ -61,5 +62,6 @@ public class ListenerFile extends OutputFile {
|
|||
ActionAST ast = g.namedActions.get("header");
|
||||
if ( ast!=null ) header = new Action(factory, ast);
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ import java.util.Map;
|
|||
/** */
|
||||
public class ParserFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public boolean genListener; // from -listener cmd-line
|
||||
public boolean genVisitor; // from -visitor cmd-line
|
||||
@ModelElement public Parser parser;
|
||||
|
|
@ -28,6 +29,7 @@ public class ParserFile extends OutputFile {
|
|||
Grammar g = factory.getGrammar();
|
||||
namedActions = buildNamedActions(factory.getGrammar());
|
||||
genPackage = g.tool.genPackage;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
// need the below members in the ST for Python, C++
|
||||
genListener = g.tool.gen_listener;
|
||||
genVisitor = g.tool.gen_visitor;
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import java.util.Set;
|
|||
|
||||
public class VisitorFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public String grammarName;
|
||||
public String parserName;
|
||||
/**
|
||||
|
|
@ -58,5 +59,6 @@ public class VisitorFile extends OutputFile {
|
|||
ActionAST ast = g.namedActions.get("header");
|
||||
if ( ast!=null ) header = new Action(factory, ast);
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,6 +28,8 @@ public class CharSupport {
|
|||
ANTLRLiteralEscapedCharValue['\\'] = '\\';
|
||||
ANTLRLiteralEscapedCharValue['\''] = '\'';
|
||||
ANTLRLiteralEscapedCharValue['"'] = '"';
|
||||
ANTLRLiteralEscapedCharValue['-'] = '-';
|
||||
ANTLRLiteralEscapedCharValue[']'] = ']';
|
||||
ANTLRLiteralCharValueEscape['\n'] = "\\n";
|
||||
ANTLRLiteralCharValueEscape['\r'] = "\\r";
|
||||
ANTLRLiteralCharValueEscape['\t'] = "\\t";
|
||||
|
|
@ -76,32 +78,6 @@ public class CharSupport {
|
|||
return getCharValueFromCharInGrammarLiteral(literal.substring(1,literal.length()-1));
|
||||
}
|
||||
|
||||
/** Given char x or \t or \u1234 return the char value;
|
||||
* Unnecessary escapes like '\{' yield -1.
|
||||
*/
|
||||
public static int getCharValueFromCharInGrammarLiteral(String cstr) {
|
||||
switch ( cstr.length() ) {
|
||||
case 1 :
|
||||
// 'x'
|
||||
return cstr.charAt(0); // no escape char
|
||||
case 2 :
|
||||
if ( cstr.charAt(0)!='\\' ) return -1;
|
||||
// '\x' (antlr lexer will catch invalid char)
|
||||
if ( Character.isDigit(cstr.charAt(1)) ) return -1;
|
||||
int escChar = cstr.charAt(1);
|
||||
int charVal = ANTLRLiteralEscapedCharValue[escChar];
|
||||
if ( charVal==0 ) return -1;
|
||||
return charVal;
|
||||
case 6 :
|
||||
// '\u1234'
|
||||
if ( !cstr.startsWith("\\u") ) return -1;
|
||||
String unicodeChars = cstr.substring(2, cstr.length());
|
||||
return Integer.parseInt(unicodeChars, 16);
|
||||
default :
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
public static String getStringFromGrammarStringLiteral(String literal) {
|
||||
StringBuilder buf = new StringBuilder();
|
||||
int i = 1; // skip first quote
|
||||
|
|
@ -110,19 +86,60 @@ public class CharSupport {
|
|||
int end = i+1;
|
||||
if ( literal.charAt(i) == '\\' ) {
|
||||
end = i+2;
|
||||
if ( (i+1)>=n ) break; // ignore spurious \ on end
|
||||
if ( literal.charAt(i+1) == 'u' ) end = i+6;
|
||||
if ( i+1 < n && literal.charAt(i+1) == 'u' ) {
|
||||
for (end = i + 2; end < i + 6; end++) {
|
||||
if ( end>n ) return null; // invalid escape sequence.
|
||||
char charAt = literal.charAt(end);
|
||||
if (!Character.isDigit(charAt) && !(charAt >= 'a' && charAt <= 'f') && !(charAt >= 'A' && charAt <= 'F')) {
|
||||
return null; // invalid escape sequence.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if ( end>n ) break;
|
||||
if ( end>n ) return null; // invalid escape sequence.
|
||||
String esc = literal.substring(i, end);
|
||||
int c = getCharValueFromCharInGrammarLiteral(esc);
|
||||
if ( c==-1 ) { buf.append(esc); }
|
||||
if ( c==-1 ) {
|
||||
return null; // invalid escape sequence.
|
||||
}
|
||||
else buf.append((char)c);
|
||||
i = end;
|
||||
}
|
||||
return buf.toString();
|
||||
}
|
||||
|
||||
/** Given char x or \t or \u1234 return the char value;
|
||||
* Unnecessary escapes like '\{' yield -1.
|
||||
*/
|
||||
public static int getCharValueFromCharInGrammarLiteral(String cstr) {
|
||||
switch ( cstr.length() ) {
|
||||
case 1:
|
||||
// 'x'
|
||||
return cstr.charAt(0); // no escape char
|
||||
case 2:
|
||||
if ( cstr.charAt(0)!='\\' ) return -1;
|
||||
// '\x' (antlr lexer will catch invalid char)
|
||||
if ( Character.isDigit(cstr.charAt(1)) ) return -1;
|
||||
int escChar = cstr.charAt(1);
|
||||
int charVal = ANTLRLiteralEscapedCharValue[escChar];
|
||||
if ( charVal==0 ) return -1;
|
||||
return charVal;
|
||||
case 6:
|
||||
// '\u1234'
|
||||
if ( !cstr.startsWith("\\u") ) return -1;
|
||||
String unicodeChars = cstr.substring(2, cstr.length());
|
||||
int result = -1;
|
||||
try {
|
||||
result = Integer.parseInt(unicodeChars, 16);
|
||||
}
|
||||
catch (NumberFormatException e) {
|
||||
}
|
||||
return result;
|
||||
default:
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
public static String capitalize(String s) {
|
||||
return Character.toUpperCase(s.charAt(0)) + s.substring(1);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -471,7 +471,7 @@ public class BasicSemanticChecks extends GrammarTreeVisitor {
|
|||
protected void enterTerminal(GrammarAST tree) {
|
||||
String text = tree.getText();
|
||||
if (text.equals("''")) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_NOT_ALLOWED, g.fileName, tree.token);
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED, g.fileName, tree.token, "''");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -982,11 +982,15 @@ public enum ErrorType {
|
|||
*
|
||||
* <p>empty strings not allowed</p>
|
||||
*
|
||||
* <pre>A: '''test''';</pre>
|
||||
* <pre>B: '';</pre>
|
||||
* <pre>C: 'test' '';</pre>
|
||||
* <pre>
|
||||
* A: '''test''';
|
||||
* B: '';
|
||||
* C: 'test' '';
|
||||
* D: [];
|
||||
* E: [f-a];
|
||||
* </pre>
|
||||
*/
|
||||
EMPTY_STRINGS_NOT_ALLOWED(174, "string literals cannot be empty", ErrorSeverity.ERROR),
|
||||
EMPTY_STRINGS_AND_SETS_NOT_ALLOWED(174, "string literals and sets cannot be empty: <arg>", ErrorSeverity.ERROR),
|
||||
/**
|
||||
* Compiler Error 175.
|
||||
*
|
||||
|
|
@ -1027,6 +1031,19 @@ public enum ErrorType {
|
|||
* <p>T00: 'a00' -> skip, more;</p>
|
||||
*/
|
||||
INCOMPATIBLE_COMMANDS(179, "incompatible commands <arg> and <arg2>", ErrorSeverity.WARNING),
|
||||
/**
|
||||
* Compiler Warning 180.
|
||||
*
|
||||
* <p>chars "a-f" used multiple times in set [a-fc-m]</p>
|
||||
*
|
||||
* <pre>
|
||||
* A: [aa-z]; // warning
|
||||
* B: [a-fc-m]; // warning
|
||||
* </pre>
|
||||
*
|
||||
* TODO: Does not work with fragment rules.
|
||||
*/
|
||||
CHARACTERS_COLLISION_IN_SET(180, "chars \"<arg>\" used multiple times in set <arg2>", ErrorSeverity.WARNING),
|
||||
|
||||
/*
|
||||
* Backward incompatibility errors
|
||||
|
|
|
|||
|
|
@ -81,6 +81,7 @@ public class Grammar implements AttributeResolver {
|
|||
parserOptions.add("TokenLabelType");
|
||||
parserOptions.add("tokenVocab");
|
||||
parserOptions.add("language");
|
||||
parserOptions.add("exportMacro");
|
||||
}
|
||||
|
||||
public static final Set<String> lexerOptions = parserOptions;
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ package org.antlr.v4.tool.ast;
|
|||
|
||||
import org.antlr.runtime.Token;
|
||||
import org.antlr.v4.misc.CharSupport;
|
||||
import org.antlr.v4.tool.ErrorType;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
|
|
@ -41,6 +42,10 @@ public abstract class GrammarASTWithOptions extends GrammarAST {
|
|||
String v = value.getText();
|
||||
if ( v.startsWith("'") || v.startsWith("\"") ) {
|
||||
v = CharSupport.getStringFromGrammarStringLiteral(v);
|
||||
if (v == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE, g.fileName, value.getToken());
|
||||
v = "";
|
||||
}
|
||||
}
|
||||
return v;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue