jasder
/
antlr
1
0
Fork 1
Code Issues Pull Requests Releases 1 Wiki Activity

Merge branch 'master_upstream'

This commit is contained in:
Mike Lischke 2016-12-15 09:16:26 +01:00
parent 76fcb6d37e 81cb4367bb
commit 50afb22068
24 changed files with 577 additions and 363 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@ -1,5 +1,5 @@
[cut] [cut]
Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors most "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership. Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors must "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership.
Make sure you are already in the contributors.txt file or add a commit to this pull request with the appropriate change. Thanks! Make sure you are already in the contributors.txt file or add a commit to this pull request with the appropriate change. Thanks!
[/cut] [/cut]

2
README.md
View File

@ -6,6 +6,8 @@
*Given day-job constraints, my time working on this project is limited so I'll have to focus first on fixing bugs rather than changing/improving the feature set. Likely I'll do it in bursts every few months. Please do not be offended if your bug or pull request does not yield a response! --parrt* *Given day-job constraints, my time working on this project is limited so I'll have to focus first on fixing bugs rather than changing/improving the feature set. Likely I'll do it in bursts every few months. Please do not be offended if your bug or pull request does not yield a response! --parrt*
[![Donate](https://www.paypal.com/en_US/i/btn/x-click-butcc-donate.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BF92STRXT8F8Q)
## Authors and major contributors ## Authors and major contributors
* [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu * [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu

165
doc/adding-tests.md → doc/antlr-project-testing.md
View File

@ -1,4 +1,4 @@
# Adding unit tests # ANTLR project unit tests
## Introduction ## Introduction
@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow
The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). [Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
## Requirements
In order to perform the tests on all target languages, you need to have the following languages installed:
* `mono` (e.g., `brew install mono`) on non-Windows boxes (on Windows it uses the Microsoft .net stack). Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
* `nodejs`
* Python 2.7
* Python 3.5
* Go
* Swift 3 (via XCode 8.x) tested currently only osx
* clang (for C++ target)
*
To **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
```
Now, make sure C# runtime is built and installed locally.
```bash
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll
rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
# build
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
```
C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.
## Running the runtime tests ## Running the runtime tests
A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
@ -49,26 +82,128 @@ And the result of testing the entire subdirectory:
From `mvn`, on the commandline, you will see: From `mvn`, on the commandline, you will see:
```bash ```bash
$ cd antlr4
$ mvn test
...
------------------------------------------------------- -------------------------------------------------------
T E S T S T E S T S
------------------------------------------------------- -------------------------------------------------------
Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
Running org.antlr.v4.test.runtime.javascript.node.TestSets Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
Running org.antlr.v4.test.runtime.javascript.node.TestPerformance antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec ...
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer [INFO] ------------------------------------------------------------------------
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec [INFO] Reactor Summary:
Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec [INFO]
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec [INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:53 min
[INFO] Finished at: 2016-11-16T15:36:56-08:00
[INFO] Final Memory: 44M/458M
[INFO] ------------------------------------------------------------------------
```
Note: That is actually result of running the much faster:
```bash
mvn -Dparallel=methods -DthreadCount=4 install
```
## Running test subsets
*From the `runtime-testsuite` dir*
### Run one test group across targets
```bash
$ cd runtime-testsuite
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn -Dtest=TestParserExec test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestParserExec
...
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
Running org.antlr.v4.test.runtime.csharp.TestParserExec
... ...
``` ```
Or run all lexer related tests:
```
$ cd runtime-testsuite
$ mvn -Dtest=Test*Lexer* test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
...
```
### Run all tests for a single target
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.* test
...
```
Or run all lexer related tests in Java target only:
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.*Lexer* test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
Running org.antlr.v4.test.runtime.java.TestLexerErrors
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
Running org.antlr.v4.test.runtime.java.TestLexerExec
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
Results :
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
```
## Testing in parallel
Use this to run tests in parallel:
```bash
$ export MAVEN_OPTS="-Xmx1G"
$ mvn -Dparallel=methods -DthreadCount=4 test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
...
```
This can be combined with other `-D` above.
## Adding a runtime test ## Adding a runtime test
To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of: To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:

136
doc/building-antlr.md
View File

@ -38,6 +38,7 @@ Receiving objects: 100% (59858/59858), 31.10 MiB | 819.00 KiB/s, done.
Resolving deltas: 100% (31898/31898), done. Resolving deltas: 100% (31898/31898), done.
Checking connectivity... done. Checking connectivity... done.
$ cd antlr4 $ cd antlr4
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn -DskipTests install $ mvn -DskipTests install
... ...
[INFO] ------------------------------------------------------------------------ [INFO] ------------------------------------------------------------------------
@ -67,65 +68,10 @@ We do `install` not `compile` as tool tests and such refer to modules that must
To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this: To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash ```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine $ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
``` ```
# Testing tool and targets
In order to perform the tests on all target languages, you need to have the following languages installed:
* `mono` (e.g., `brew install mono`)
* `nodejs`
* Python 2.7
* Python 3.5
* Go
* Swift 3 (via XCode 8.x) tested currently only osx
* clang (for C++ target)
To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
$ mvn install # now "do it with feeling"
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:53 min
[INFO] Finished at: 2016-11-16T15:36:56-08:00
[INFO] Final Memory: 44M/458M
[INFO] ------------------------------------------------------------------------
```
Note: That is actually result of running the much faster:
`mvn -Dparallel=methods -DthreadCount=4 install`
You should see these jars (when building 4.6-SNAPSHOT): You should see these jars (when building 4.6-SNAPSHOT):
```bash ```bash
@ -143,87 +89,17 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar
Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes.
## Running test subsets # Testing tool and targets
*From the `runtime-testsuite` dir* See [ANTLR project unit tests](antlr-project-testing.md).
### Run one test group across targets
```bash # Building without testing
$ cd runtime-testsuite
$ mvn -Dtest=TestParserExec test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestParserExec
...
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
Running org.antlr.v4.test.runtime.csharp.TestParserExec
...
```
Or run all lexer related tests:
```
$ mvn -Dtest=Test*Lexer* test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
...
```
### Run all tests for a single target
```bash
$ mvn -Dtest=java.* test
...
```
Or run all lexer related tests in Java target only:
```bash
$ mvn -Dtest=java.*Lexer* test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
Running org.antlr.v4.test.runtime.java.TestLexerErrors
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
Running org.antlr.v4.test.runtime.java.TestLexerExec
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
Results :
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
```
## Testing in parallel
Use this to run tests in parallel:
```bash
$ mvn -Dparallel=methods -DthreadCount=4 test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
...
```
This can be combined with other `-D` above.
## Building without testing
To build without running the tests (saves a lot of time), do this: To build without running the tests (saves a lot of time), do this:
```bash ```bash
mvn -DskipTests install $ mvn -DskipTests install
``` ```
## Building ANTLR in Intellij IDE ## Building ANTLR in Intellij IDE

2
doc/index.md
View File

@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema
* [Cutting an ANTLR Release](releasing-antlr.md) * [Cutting an ANTLR Release](releasing-antlr.md)
* [Adding ANTLR unit tests](adding-tests.md) * [ANTLR project unit tests](antlr-project-testing.md)
* [Creating an ANTLR Language Target](creating-a-language-target.md) * [Creating an ANTLR Language Target](creating-a-language-target.md)

68
doc/releasing-antlr.md
View File

@ -4,12 +4,22 @@
Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1). Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1).
### Delete existing release tag
Wack any existing tag as mvn will create one and it fails if already there. Wack any existing tag as mvn will create one and it fails if already there.
``` ```
$ git tag -d 4.5.2 $ git tag -d 4.6
$ git push origin :refs/tags/4.5.2 $ git push origin :refs/tags/4.6
$ git push upstream :refs/tags/4.5.2 $ git push upstream :refs/tags/4.6
```
### Create release candidate tag
```bash
$ git tag -a 4.6-rc1 -m 'heading towards 4.6'
$ git push origin 4.6-rc1
$ git push upstream 4.6-rc1
``` ```
## Bump version ## Bump version
@ -22,15 +32,17 @@ Edit the repository looking for 4.5 or whatever and update it. Bump version in t
* runtime/Python3/setup.py * runtime/Python3/setup.py
* runtime/Python3/src/antlr4/Recognizer.py * runtime/Python3/src/antlr4/Recognizer.py
* runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs * runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs
* runtime/CSharp/build/version.ps1
* runtime/JavaScript/src/antlr4/package.json * runtime/JavaScript/src/antlr4/package.json
* runtime/JavaScript/src/antlr4/Recognizer.js * runtime/JavaScript/src/antlr4/Recognizer.js
* runtime/Cpp/VERSION
* runtime/Cpp/runtime/src/RuntimeMetaData.cpp
* runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake
* tool/src/org/antlr/v4/codegen/target/CppTarget.java * tool/src/org/antlr/v4/codegen/target/CppTarget.java
* tool/src/org/antlr/v4/codegen/target/CSharpTarget.java * tool/src/org/antlr/v4/codegen/target/CSharpTarget.java
* tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java * tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java
* tool/src/org/antlr/v4/codegen/target/Python2Target.java * tool/src/org/antlr/v4/codegen/target/Python2Target.java
* tool/src/org/antlr/v4/codegen/target/Python3Target.java * tool/src/org/antlr/v4/codegen/target/Python3Target.java
* runtime/Cpp/VERSION
* runtime/Cpp/RuntimeMetaData.cpp
Here is a simple script to display any line from the critical files with, say, `4.5` in it: Here is a simple script to display any line from the critical files with, say, `4.5` in it:
@ -144,24 +156,24 @@ It will start out by asking you the version number:
``` ```
... ...
What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.2: : 4.5.2 What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6: : 4.6
What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.5.2: : What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.6: :
What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.5.2: : What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.6: :
What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.5.2: : What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.6: :
What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.5.2: : What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.6: :
What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.5.2: : What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.6: :
What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.5.2: : 4.5.2 What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.6: : 4.6
What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.3-SNAPSHOT: What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6.1-SNAPSHOT:
... ...
``` ```
Maven will go through your pom.xml files to update versions from 4.5.2-SNAPSHOT to 4.5.2 for release and then to 4.5.3-SNAPSHOT after release, which is done with: Maven will go through your pom.xml files to update versions from 4.6-SNAPSHOT to 4.6 for release and then to 4.6.1-SNAPSHOT after release, which is done with:
```bash ```bash
mvn release:perform -Darguments="-DskipTests" mvn release:perform -Darguments="-DskipTests"
``` ```
Maven will use git to push pom.xml changes. (big smile) Maven will use git to push pom.xml changes.
Now, go here: Now, go here:
@ -174,11 +186,11 @@ and on the left click "Staging Repositories". You click the staging repo and clo
Copy the jars to antlr.org site and update download/index.html Copy the jars to antlr.org site and update download/index.html
```bash ```bash
cp ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.5.2.jar cp ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.6.jar
cp ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-4.5.2-complete.jar cp ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.6-complete.jar
cd ~/antlr/sites/website-antlr4/download cd ~/antlr/sites/website-antlr4/download
git add antlr-4.5.2-complete.jar git add antlr-4.6-complete.jar
git add antlr-runtime-4.5.2.jar git add antlr-runtime-4.6.jar
``` ```
Update on site: Update on site:
@ -190,7 +202,7 @@ Update on site:
* scripts/topnav.js * scripts/topnav.js
``` ```
git commit -a -m 'add 4.5.2 jars' git commit -a -m 'add 4.6 jars'
git push origin gh-pages git push origin gh-pages
``` ```
@ -200,8 +212,8 @@ git push origin gh-pages
```bash ```bash
cd runtime/JavaScript/src cd runtime/JavaScript/src
zip -r /tmp/antlr-javascript-runtime-4.5.2.zip antlr4 zip -r /tmp/antlr-javascript-runtime-4.6.zip antlr4
cp /tmp/antlr-javascript-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download cp /tmp/antlr-javascript-runtime-4.6.zip ~/antlr/sites/website-antlr4/download
# git add, commit, push # git add, commit, push
``` ```
@ -209,7 +221,7 @@ Move target to website
```bash ```bash
pushd ~/antlr/sites/website-antlr4/download pushd ~/antlr/sites/website-antlr4/download
git add antlr-javascript-runtime-4.5.2.zip git add antlr-javascript-runtime-4.6.zip
git commit -a -m 'update JS runtime' git commit -a -m 'update JS runtime'
git push origin gh-pages git push origin gh-pages
popd popd
@ -225,15 +237,15 @@ rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
# build # build
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
# zip it up to get a version number on zip filename # zip it up to get a version number on zip filename
zip --junk-paths /tmp/antlr-csharp-runtime-4.5.2.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll zip --junk-paths /tmp/antlr-csharp-runtime-4.6.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll
cp /tmp/antlr-csharp-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download cp /tmp/antlr-csharp-runtime-4.6.zip ~/antlr/sites/website-antlr4/download
``` ```
Move target to website Move target to website
```bash ```bash
pushd ~/antlr/sites/website-antlr4/download pushd ~/antlr/sites/website-antlr4/download
git add antlr-csharp-runtime-4.5.2.zip git add antlr-csharp-runtime-4.6.zip
git commit -a -m 'update C# runtime' git commit -a -m 'update C# runtime'
git push origin gh-pages git push origin gh-pages
popd popd
@ -346,9 +358,9 @@ cd ~/antlr/sites/website-antlr4/api
git checkout gh-pages git checkout gh-pages
git pull origin gh-pages git pull origin gh-pages
cd Java cd Java
jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2-javadoc.jar jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6-javadoc.jar
cd ../JavaTool cd ../JavaTool
jar xvf ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2-javadoc.jar jar xvf ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-javadoc.jar
git commit -a -m 'freshen api doc' git commit -a -m 'freshen api doc'
git push origin gh-pages git push origin gh-pages
``` ```

2
doc/runtimetests-overview.md
View File

@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate
## Generating JUnit Tests ## Generating JUnit Tests
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run
```bash ```bash
$ mvn -Pgen generate-test-sources $ mvn -Pgen generate-test-sources

6
doc/targets.md
View File

@ -14,7 +14,7 @@ This page lists the available and upcoming ANTLR runtimes. Please note that you
New features generally appear in the Java target and then migrate to the other targets, but these other targets don't always get updated in the same overall tool release. This section tries to identify features added to Java that have not been added to the other targets. New features generally appear in the Java target and then migrate to the other targets, but these other targets don't always get updated in the same overall tool release. This section tries to identify features added to Java that have not been added to the other targets.
|Feature|Java|C&sharp;|JavaScript|Python2|Python3|Swift|C++| |Feature|Java|C&sharp;|Python2|Python3|JavaScript|Go|C++|Swift|
|---|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|---|---|
|Ambiguous tree construction|4.5.1|-|-|-|-|-|-| |Ambiguous tree construction|4.5.1|-|-|-|-|-|-|-|

156
runtime-testsuite/pom.xml
View File

@ -79,107 +79,59 @@
</dependency> </dependency>
</dependencies> </dependencies>
<build> <build>
<testSourceDirectory>test</testSourceDirectory> <testSourceDirectory>test</testSourceDirectory>
<resources> <resources>
<resource> <resource>
<directory>resources</directory> <directory>resources</directory>
</resource> </resource>
<resource> <resource>
<directory>../runtime</directory> <directory>../runtime</directory>
</resource> </resource>
</resources> </resources>
<plugins> <plugins>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId> <artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version> <version>2.19.1</version>
<configuration> <configuration>
<includes> <includes>
<include>**/csharp/Test*.java</include> <include>**/csharp/Test*.java</include>
<include>**/java/Test*.java</include> <include>**/java/Test*.java</include>
<include>**/go/Test*.java</include> <include>**/go/Test*.java</include>
<include>**/javascript/node/Test*.java</include> <include>**/javascript/node/Test*.java</include>
<include>**/python2/Test*.java</include> <include>**/python2/Test*.java</include>
<include>**/python3/Test*.java</include> <include>**/python3/Test*.java</include>
<include>${antlr.tests.swift}</include> <include>${antlr.tests.swift}</include>
</includes> </includes>
</configuration> </configuration>
</plugin> </plugin>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId> <artifactId>maven-jar-plugin</artifactId>
<version>2.4</version> <version>2.4</version>
<executions> <executions>
<execution> <execution>
<goals> <goals>
<goal>test-jar</goal> <goal>test-jar</goal>
</goals> </goals>
</execution> </execution>
</executions> </executions>
</plugin> </plugin>
</plugins> </plugins>
</build> </build>
<profiles> <profiles>
<profile> <profile>
<id>includeSwiftTests</id> <id>includeSwiftTests</id>
<activation> <activation>
<os> <os>
<family>mac</family> <family>mac</family>
</os> </os>
</activation> </activation>
<properties> <properties>
<antlr.tests.swift>**/swift/Test*.java</antlr.tests.swift> <antlr.tests.swift>**/swift/Test*.java</antlr.tests.swift>
</properties> </properties>
</profile> </profile>
</profiles>
<profile>
<id>gen</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.4.0</version>
<executions>
<execution>
<phase>generate-test-sources</phase>
<goals>
<goal>java</goal>
</goals>
<configuration>
<mainClass>org.antlr.v4.testgen.TestGenerator</mainClass>
<arguments>
<argument>-root</argument>
<argument>${basedir}</argument>
<argument>-outdir</argument>
<argument>${basedir}/test</argument>
<argument>-templates</argument>
<argument>${basedir}/resources/org/antlr/v4/test/runtime/templates</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>tests</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/Test*.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project> </project>

51
runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java
View File

@ -55,7 +55,9 @@ import java.io.InputStream;
import java.io.InputStreamReader; import java.io.InputStreamReader;
import java.lang.reflect.InvocationTargetException; import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method; import java.lang.reflect.Method;
import java.net.URISyntaxException;
import java.net.URL; import java.net.URL;
import java.nio.file.Paths;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Arrays; import java.util.Arrays;
import java.util.Collections; import java.util.Collections;
@ -508,7 +510,7 @@ public class BaseCppTest implements RuntimeTestSupport {
return files; return files;
} }
private String runProcess(ProcessBuilder builder, String description) throws Exception { private String runProcess(ProcessBuilder builder, String description, boolean showStderr) throws Exception {
// System.out.println("BUILDER: "+builder.command()); // System.out.println("BUILDER: "+builder.command());
Process process = builder.start(); Process process = builder.start();
StreamVacuum stdoutVacuum = new StreamVacuum(process.getInputStream()); StreamVacuum stdoutVacuum = new StreamVacuum(process.getInputStream());
@ -521,10 +523,10 @@ public class BaseCppTest implements RuntimeTestSupport {
String output = stdoutVacuum.toString(); String output = stdoutVacuum.toString();
if ( stderrVacuum.toString().length()>0 ) { if ( stderrVacuum.toString().length()>0 ) {
this.stderrDuringParse = stderrVacuum.toString(); this.stderrDuringParse = stderrVacuum.toString();
// System.err.println(this.stderrDuringParse); if ( showStderr ) System.err.println(this.stderrDuringParse);
} }
if (errcode != 0) { if (errcode != 0) {
String err = "execution failed with error code: "+errcode; String err = "execution of '"+description+"' failed with error code: "+errcode;
if ( this.stderrDuringParse!=null ) { if ( this.stderrDuringParse!=null ) {
this.stderrDuringParse += err; this.stderrDuringParse += err;
} }
@ -536,11 +538,11 @@ public class BaseCppTest implements RuntimeTestSupport {
return output; return output;
} }
private String runCommand(String command[], String workPath, String description) throws Exception { private String runCommand(String command[], String workPath, String description, boolean showStderr) throws Exception {
ProcessBuilder builder = new ProcessBuilder(command); ProcessBuilder builder = new ProcessBuilder(command);
builder.directory(new File(workPath)); builder.directory(new File(workPath));
return runProcess(builder, description); return runProcess(builder, description, showStderr);
} }
// TODO: add a buildRuntimeOnWindows variant. // TODO: add a buildRuntimeOnWindows variant.
@ -550,8 +552,9 @@ public class BaseCppTest implements RuntimeTestSupport {
try { try {
String command[] = { "cmake", ".", /*"-DCMAKE_CXX_COMPILER=clang++",*/ "-DCMAKE_BUILD_TYPE=release" }; String command[] = { "cmake", ".", /*"-DCMAKE_CXX_COMPILER=clang++",*/ "-DCMAKE_BUILD_TYPE=release" };
if (runCommand(command, runtimePath, "antlr runtime cmake") == null) if (runCommand(command, runtimePath, "antlr runtime cmake", false) == null) {
return false; return false;
}
} }
catch (Exception e) { catch (Exception e) {
System.err.println("can't configure antlr cpp runtime cmake file"); System.err.println("can't configure antlr cpp runtime cmake file");
@ -559,11 +562,21 @@ public class BaseCppTest implements RuntimeTestSupport {
try { try {
String command[] = { "make", "-j", "8" }; // Assuming a reasonable amount of available CPU cores. String command[] = { "make", "-j", "8" }; // Assuming a reasonable amount of available CPU cores.
if (runCommand(command, runtimePath, "building antlr runtime") == null) if (runCommand(command, runtimePath, "building antlr runtime", true) == null)
return false; return false;
} }
catch (Exception e) { catch (Exception e) {
System.err.println("can't compile antlr cpp runtime"); System.err.println("can't compile antlr cpp runtime");
e.printStackTrace(System.err);
try {
String command[] = { "ls", "-la" };
String output = runCommand(command, runtimePath + "/dist/", "printing library folder content", true);
System.out.println(output);
}
catch (Exception e2) {
System.err.println("can't even list folder content");
e2.printStackTrace(System.err);
}
} }
/* for debugging /* for debugging
@ -593,7 +606,7 @@ public class BaseCppTest implements RuntimeTestSupport {
if ( !runtimeBuiltOnce ) { if ( !runtimeBuiltOnce ) {
try { try {
String command[] = {"clang++", "--version"}; String command[] = {"clang++", "--version"};
String output = runCommand(command, tmpdir, "printing compiler version"); String output = runCommand(command, tmpdir, "printing compiler version", false);
System.out.println("Compiler version is: "+output); System.out.println("Compiler version is: "+output);
} }
catch (Exception e) { catch (Exception e) {
@ -613,7 +626,7 @@ public class BaseCppTest implements RuntimeTestSupport {
String libExtension = (getOS().equals("mac")) ? "dylib" : "so"; String libExtension = (getOS().equals("mac")) ? "dylib" : "so";
try { try {
String command[] = { "ln", "-s", runtimePath + "/dist/libantlr4-runtime." + libExtension }; String command[] = { "ln", "-s", runtimePath + "/dist/libantlr4-runtime." + libExtension };
if (runCommand(command, tmpdir, "sym linking C++ runtime") == null) if (runCommand(command, tmpdir, "sym linking C++ runtime", true) == null)
return null; return null;
} }
catch (Exception e) { catch (Exception e) {
@ -625,7 +638,7 @@ public class BaseCppTest implements RuntimeTestSupport {
try { try {
List<String> command2 = new ArrayList<String>(Arrays.asList("clang++", "-std=c++11", "-I", includePath, "-L.", "-lantlr4-runtime", "-o", "a.out")); List<String> command2 = new ArrayList<String>(Arrays.asList("clang++", "-std=c++11", "-I", includePath, "-L.", "-lantlr4-runtime", "-o", "a.out"));
command2.addAll(allCppFiles(tmpdir)); command2.addAll(allCppFiles(tmpdir));
if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary") == null) { if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary", true) == null) {
return null; return null;
} }
} }
@ -642,7 +655,7 @@ public class BaseCppTest implements RuntimeTestSupport {
builder.directory(new File(tmpdir)); builder.directory(new File(tmpdir));
Map<String, String> env = builder.environment(); Map<String, String> env = builder.environment();
env.put("LD_PRELOAD", runtimePath + "/dist/libantlr4-runtime." + libExtension); env.put("LD_PRELOAD", runtimePath + "/dist/libantlr4-runtime." + libExtension);
String output = runProcess(builder, "running test binary"); String output = runProcess(builder, "running test binary", false);
if ( output.length()==0 ) { if ( output.length()==0 ) {
output = null; output = null;
} }
@ -664,11 +677,21 @@ public class BaseCppTest implements RuntimeTestSupport {
protected String locateRuntime() { protected String locateRuntime() {
final ClassLoader loader = Thread.currentThread().getContextClassLoader(); final ClassLoader loader = Thread.currentThread().getContextClassLoader();
final URL runtimeSrc = loader.getResource("Cpp"); final URL runtimeURL = loader.getResource("Cpp");
if (runtimeSrc == null) { if (runtimeURL == null) {
throw new RuntimeException("Cannot find runtime"); throw new RuntimeException("Cannot find runtime");
} }
return runtimeSrc.getPath(); // Windows not getting runtime right. See:
// http://stackoverflow.com/questions/6164448/convert-url-to-normal-windows-filename-java
// it was coming back "/C:/projects/antlr4-l7imv/runtime-testsuite/target/classes/Cpp"
String p;
try {
p = Paths.get(runtimeURL.toURI()).toFile().toString();
}
catch (URISyntaxException use) {
p = "Can't find runtime";
}
return p;
} }
List<ANTLRMessage> getMessagesOfType(List<ANTLRMessage> msgs, Class<? extends ANTLRMessage> c) { List<ANTLRMessage> getMessagesOfType(List<ANTLRMessage> msgs, Class<? extends ANTLRMessage> c) {

2
runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParseTreesDescriptors.java
View File

@ -112,7 +112,7 @@ public class ParseTreesDescriptors {
@Override @Override
public boolean ignore(String targetName) { public boolean ignore(String targetName) {
return !targetName.matches("Java|Python2|Python3|Node"); return !targetName.matches("Java|Python2|Python3|Node|Swift");
} }
} }

24
runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/SemPredEvalLexerDescriptors.java
View File

@ -10,6 +10,30 @@ import org.antlr.v4.test.runtime.BaseLexerTestDescriptor;
import org.antlr.v4.test.runtime.CommentHasStringValue; import org.antlr.v4.test.runtime.CommentHasStringValue;
public class SemPredEvalLexerDescriptors { public class SemPredEvalLexerDescriptors {
// Test for https://github.com/antlr/antlr4/issues/958
public static class RuleSempredFunction extends BaseLexerTestDescriptor {
public String input = "aaa";
/**
[@0,0:0='a',<1>,1:0]
[@1,1:1='a',<1>,1:1]
[@2,2:2='a',<1>,1:2]
[@3,3:2='<EOF>',<-1>,1:3]
*/
@CommentHasStringValue
public String output;
public String errors = null;
public String startRule = "";
public String grammarName = "L";
/**
lexer grammar L;
T : 'a' {<True()>}? ;
*/
@CommentHasStringValue
public String grammar;
}
public static class DisableRule extends BaseLexerTestDescriptor { public static class DisableRule extends BaseLexerTestDescriptor {
public String input = "enum abc"; public String input = "enum abc";
/** /**

6
runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs
View File

@ -42,8 +42,8 @@ using System.Runtime.InteropServices;
// You can specify all the values or you can default the Build and Revision Numbers // You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below: // by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")] // [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("4.5.3.0")] [assembly: AssemblyVersion("4.6.0")]
#if !COMPACT #if !COMPACT
[assembly: AssemblyFileVersion("4.5.3.0")] [assembly: AssemblyFileVersion("4.6.0")]
[assembly: AssemblyInformationalVersion("4.5.3.0")] [assembly: AssemblyInformationalVersion("4.6.0")]
#endif #endif

2
runtime/Cpp/VERSION
View File

@ -1 +1 @@
4.5.4 4.6

34
runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake
View File

@ -19,11 +19,11 @@
# #
# next make a build folder e.g. ~/buildfolder/ # next make a build folder e.g. ~/buildfolder/
# from the buildfolder, run cmake ~/srcfolder; make # from the buildfolder, run cmake ~/srcfolder; make
# #
############################################################### ###############################################################
# # minimum required CMAKE version # # minimum required CMAKE version
# CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2 FATAL_ERROR) # CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2 FATAL_ERROR)
# #
# LIST( APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake ) # LIST( APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake )
# #
# # compiler must be 11 or 14 # # compiler must be 11 or 14
@ -49,7 +49,7 @@
# add_executable(demo main.cpp ${antlr4cpp_src_files_antlrcpptest}) # add_executable(demo main.cpp ${antlr4cpp_src_files_antlrcpptest})
# add_dependencies(demo antlr4cpp antlr4cpp_generation_antlrcpptest) # add_dependencies(demo antlr4cpp antlr4cpp_generation_antlrcpptest)
# target_link_libraries(demo antlr4-runtime) # target_link_libraries(demo antlr4-runtime)
# #
############################################################### ###############################################################
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2) CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2)
@ -65,8 +65,8 @@ set(ANTLR4CPP_EXTERNAL_ROOT ${CMAKE_BINARY_DIR}/externals/antlr4cpp)
# external repository # external repository
# GIT_REPOSITORY https://github.com/antlr/antlr4.git # GIT_REPOSITORY https://github.com/antlr/antlr4.git
set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/DanMcLaughlin/antlr4.git") set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/antlr/antlr4.git")
set(ANTLR4CPP_EXTERNAL_TAG "v4.5.4") set(ANTLR4CPP_EXTERNAL_TAG "4.6-rc1")
if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}") if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}")
message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}") message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}")
@ -81,13 +81,13 @@ endif()
# set(ANTLR4CPP_JAR_LOCATION ${ANTLR4CPP_EXTERNAL_ROOT}/${ANTLR4CPP_JAR_NAME}) # set(ANTLR4CPP_JAR_LOCATION ${ANTLR4CPP_EXTERNAL_ROOT}/${ANTLR4CPP_JAR_NAME})
# #
# !TODO! Ensure Antlr tool available - copy from internet # !TODO! Ensure Antlr tool available - copy from internet
# #
# # !TODO! this shold be calculated based on the tags # # !TODO! this shold be calculated based on the tags
# if (NOT ANTLR4CPP_JAR_NAME) # if (NOT ANTLR4CPP_JAR_NAME)
# # default location to find antlr Java binary # # default location to find antlr Java binary
# set(ANTLR4CPP_JAR_NAME antlr4-4.5.4-SNAPSHOT.jar) # set(ANTLR4CPP_JAR_NAME antlr4-4.5.4-SNAPSHOT.jar)
# endif() # endif()
# #
# if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}") # if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}")
# # download Java tool if not installed # # download Java tool if not installed
# ExternalProject_ADD( # ExternalProject_ADD(
@ -162,20 +162,20 @@ endforeach(src_path)
set(ANTLR4CPP_LIBS "${INSTALL_DIR}/lib") set(ANTLR4CPP_LIBS "${INSTALL_DIR}/lib")
# antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so # antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so
# antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a # antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a
############ Generate runtime ################# ############ Generate runtime #################
# macro to add dependencies to target # macro to add dependencies to target
# #
# Param 1 project name # Param 1 project name
# Param 1 namespace (postfix for dependencies) # Param 1 namespace (postfix for dependencies)
# Param 2 Lexer file (full path) # Param 2 Lexer file (full path)
# Param 3 Parser File (full path) # Param 3 Parser File (full path)
# #
# output # output
# #
# antlr4cpp_src_files_{namespace} - src files for add_executable # antlr4cpp_src_files_{namespace} - src files for add_executable
# antlr4cpp_include_dirs_{namespace} - include dir for generated headers # antlr4cpp_include_dirs_{namespace} - include dir for generated headers
# antlr4cpp_generation_{namespace} - for add_dependencies tracking # antlr4cpp_generation_{namespace} - for add_dependencies tracking
@ -184,18 +184,18 @@ macro(antlr4cpp_process_grammar
antlr4cpp_project_namespace antlr4cpp_project_namespace
antlr4cpp_grammar_lexer antlr4cpp_grammar_lexer
antlr4cpp_grammar_parser) antlr4cpp_grammar_parser)
if(EXISTS "${ANTLR4CPP_JAR_LOCATION}") if(EXISTS "${ANTLR4CPP_JAR_LOCATION}")
message(STATUS "Found antlr tool: ${ANTLR4CPP_JAR_LOCATION}") message(STATUS "Found antlr tool: ${ANTLR4CPP_JAR_LOCATION}")
else() else()
message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}") message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}")
endif() endif()
add_custom_target("antlr4cpp_generation_${antlr4cpp_project_namespace}" add_custom_target("antlr4cpp_generation_${antlr4cpp_project_namespace}"
COMMAND COMMAND
${CMAKE_COMMAND} -E make_directory ${ANTLR4CPP_GENERATED_SRC_DIR} ${CMAKE_COMMAND} -E make_directory ${ANTLR4CPP_GENERATED_SRC_DIR}
COMMAND COMMAND
"${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}" "${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}"
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}" WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
DEPENDS "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}" DEPENDS "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}"
) )
@ -213,9 +213,9 @@ macro(antlr4cpp_process_grammar
) )
endforeach(generated_file) endforeach(generated_file)
message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} Generated: ${generated_files}") message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} Generated: ${generated_files}")
# export generated include directory # export generated include directory
set(antlr4cpp_include_dirs_${antlr4cpp_project_namespace} ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}) set(antlr4cpp_include_dirs_${antlr4cpp_project_namespace} ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace})
message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} include: ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}") message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} include: ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}")
endmacro() endmacro()

2
runtime/Cpp/runtime/src/RuntimeMetaData.cpp
View File

@ -7,7 +7,7 @@
using namespace antlr4; using namespace antlr4;
const std::string RuntimeMetaData::VERSION = "4.5.4"; const std::string RuntimeMetaData::VERSION = "4.6";
std::string RuntimeMetaData::getRuntimeVersion() { std::string RuntimeMetaData::getRuntimeVersion() {
return VERSION; return VERSION;

4
runtime/Python2/src/antlr4/error/ErrorStrategy.py
View File

@ -484,14 +484,14 @@ class DefaultErrorStrategy(ErrorStrategy):
expecting = self.getExpectedTokens(recognizer) expecting = self.getExpectedTokens(recognizer)
expectedTokenType = expecting[0] # get any element expectedTokenType = expecting[0] # get any element
if expectedTokenType==Token.EOF: if expectedTokenType==Token.EOF:
tokenText = "<missing EOF>" tokenText = u"<missing EOF>"
else: else:
name = None name = None
if expectedTokenType < len(recognizer.literalNames): if expectedTokenType < len(recognizer.literalNames):
name = recognizer.literalNames[expectedTokenType] name = recognizer.literalNames[expectedTokenType]
if name is None and expectedTokenType < len(recognizer.symbolicNames): if name is None and expectedTokenType < len(recognizer.symbolicNames):
name = recognizer.symbolicNames[expectedTokenType] name = recognizer.symbolicNames[expectedTokenType]
tokenText = "<missing " + str(name) + ">" tokenText = u"<missing " + unicode(name) + u">"
current = currentSymbol current = currentSymbol
lookback = recognizer.getTokenStream().LT(-1) lookback = recognizer.getTokenStream().LT(-1)
if current.type==Token.EOF and lookback is not None: if current.type==Token.EOF and lookback is not None:

4
runtime/Swift/Antlr4/org/antlr/v4/runtime/BufferedTokenStream.swift
View File

@ -473,8 +473,6 @@ public class BufferedTokenStream: TokenStream {
public func getText() throws -> String { public func getText() throws -> String {
try lazyInit()
try fill()
return try getText(Interval.of(0, size() - 1)) return try getText(Interval.of(0, size() - 1))
} }
@ -485,7 +483,7 @@ public class BufferedTokenStream: TokenStream {
if start < 0 || stop < 0 { if start < 0 || stop < 0 {
return "" return ""
} }
try lazyInit() try fill()
if stop >= tokens.count { if stop >= tokens.count {
stop = tokens.count - 1 stop = tokens.count - 1
} }

26
runtime/Swift/Antlr4/org/antlr/v4/runtime/CommonToken.swift
View File

@ -255,9 +255,13 @@ public class CommonToken: WritableToken {
} }
public var description: String { public var description: String {
return toString(nil)
}
public func toString(_ r: Recognizer<ATNSimulator>?) -> String {
var channelStr: String = "" var channelStr: String = ""
if channel > 0 { if channel > 0 {
channelStr = "channel=\(channel)" channelStr = ",channel=\(channel)"
} }
var txt: String var txt: String
if let tokenText = getText() { if let tokenText = getText() {
@ -267,15 +271,19 @@ public class CommonToken: WritableToken {
} else { } else {
txt = "<no text>" txt = "<no text>"
} }
let desc: StringBuilder = StringBuilder() var typeString = "\(type)"
desc.append("[@\(getTokenIndex()),") if let r = r {
desc.append("\(start):\(stop)='\(txt)',") typeString = r.getVocabulary().getDisplayName(type);
desc.append("<\(type)>\(channelStr),") }
desc.append("\(line):\(getCharPositionInLine())]") return "[@"+getTokenIndex()+","+start+":"+stop+"='"+txt+"',<"+typeString+">"+channelStr+","+line+":"+getCharPositionInLine()+"]"
// let desc: StringBuilder = StringBuilder()
return desc.toString() // desc.append("[@\(getTokenIndex()),")
// desc.append("\(start):\(stop)='\(txt)',")
// desc.append("<\(typeString)>\(channelStr),")
// desc.append("\(line):\(getCharPositionInLine())]")
//
// return desc.toString()
} }
public var visited: Bool { public var visited: Bool {
get { get {
return _visited return _visited

12
runtime/Swift/Antlr4/org/antlr/v4/runtime/ParserRuleContext.swift
View File

@ -79,6 +79,18 @@ open class ParserRuleContext: RuleContext {
self.start = ctx.start self.start = ctx.start
self.stop = ctx.stop self.stop = ctx.stop
// copy any error nodes to alt label node
if ctx.children != nil{
self.children = Array<ParseTree>()
// reset parent pointer for any error nodes
for child: ParseTree in ctx.children! {
if child is ErrorNode{
self.children?.append(child)
( (child as! ErrorNode)).parent = self
}
}
}
} }
public init(_ parent: ParserRuleContext?, _ invokingStateNumber: Int) { public init(_ parent: ParserRuleContext?, _ invokingStateNumber: Int) {

2
runtime/Swift/Antlr4/org/antlr/v4/runtime/Recognizer.swift
View File

@ -62,7 +62,7 @@ open class Recognizer<ATNInterpreter:ATNSimulator> {
if result == nil { if result == nil {
result = Dictionary<String, Int>() result = Dictionary<String, Int>()
let length = self.getATN().maxTokenType let length = self.getATN().maxTokenType
for i in 0..<length { for i in 0...length {
let literalName: String? = vocabulary.getLiteralName(i) let literalName: String? = vocabulary.getLiteralName(i)
if literalName != nil { if literalName != nil {
result![literalName!] = i result![literalName!] = i

176
runtime/Swift/Antlr4/org/antlr/v4/runtime/atn/ParserATNSimulator.swift
View File

@ -232,13 +232,20 @@
* both SLL and LL parsing. Erroneous input will therefore require 2 passes over * both SLL and LL parsing. Erroneous input will therefore require 2 passes over
* the input.</p> * the input.</p>
*/ */
import Foundation
open class ParserATNSimulator: ATNSimulator { open class ParserATNSimulator: ATNSimulator {
public let debug: Bool = false public let debug: Bool = false
public let debug_list_atn_decisions: Bool = false public let debug_list_atn_decisions: Bool = false
public let dfa_debug: Bool = false public let dfa_debug: Bool = false
public let retry_debug: Bool = false public let retry_debug: Bool = false
/** Just in case this optimization is bad, add an ENV variable to turn it off */
public static let TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT: Bool = {
if let value = ProcessInfo.processInfo.environment["TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT"] {
return NSString(string: value).boolValue
}
return false
}()
internal final var parser: Parser internal final var parser: Parser
public final var decisionToDFA: [DFA] public final var decisionToDFA: [DFA]
@ -1441,6 +1448,10 @@ open class ParserATNSimulator: ATNSimulator {
} }
let length = p.getNumberOfTransitions() let length = p.getNumberOfTransitions()
for i in 0..<length { for i in 0..<length {
if i == 0 &&
canDropLoopEntryEdgeInLeftRecursiveRule(config) {
continue
}
let t: Transition = p.transition(i) let t: Transition = p.transition(i)
let continueCollecting: Bool = let continueCollecting: Bool =
!(t is ActionTransition) && collectPredicates !(t is ActionTransition) && collectPredicates
@ -1506,7 +1517,168 @@ open class ParserATNSimulator: ATNSimulator {
//print("That took: "+(finishTime-startTime)+ " ms"); //print("That took: "+(finishTime-startTime)+ " ms");
} }
/** Implements first-edge (loop entry) elimination as an optimization
* during closure operations. See antlr/antlr4#1398.
*
* The optimization is to avoid adding the loop entry config when
* the exit path can only lead back to the same
* StarLoopEntryState after popping context at the rule end state
* (traversing only epsilon edges, so we're still in closure, in
* this same rule).
*
* We need to detect any state that can reach loop entry on
* epsilon w/o exiting rule. We don't have to look at FOLLOW
* links, just ensure that all stack tops for config refer to key
* states in LR rule.
*
* To verify we are in the right situation we must first check
* closure is at a StarLoopEntryState generated during LR removal.
* Then we check that each stack top of context is a return state
* from one of these cases:
*
* 1. 'not' expr, '(' type ')' expr. The return state points at loop entry state
* 2. expr op expr. The return state is the block end of internal block of (...)*
* 3. 'between' expr 'and' expr. The return state of 2nd expr reference.
* That state points at block end of internal block of (...)*.
* 4. expr '?' expr ':' expr. The return state points at block end,
* which points at loop entry state.
*
* If any is true for each stack top, then closure does not add a
* config to the current config set for edge[0], the loop entry branch.
*
* Conditions fail if any context for the current config is:
*
* a. empty (we'd fall out of expr to do a global FOLLOW which could
* even be to some weird spot in expr) or,
* b. lies outside of expr or,
* c. lies within expr but at a state not the BlockEndState
* generated during LR removal
*
* Do we need to evaluate predicates ever in closure for this case?
*
* No. Predicates, including precedence predicates, are only
* evaluated when computing a DFA start state. I.e., only before
* the lookahead (but not parser) consumes a token.
*
* There are no epsilon edges allowed in LR rule alt blocks or in
* the "primary" part (ID here). If closure is in
* StarLoopEntryState any lookahead operation will have consumed a
* token as there are no epsilon-paths that lead to
* StarLoopEntryState. We do not have to evaluate predicates
* therefore if we are in the generated StarLoopEntryState of a LR
* rule. Note that when making a prediction starting at that
* decision point, decision d=2, compute-start-state performs
* closure starting at edges[0], edges[1] emanating from
* StarLoopEntryState. That means it is not performing closure on
* StarLoopEntryState during compute-start-state.
*
* How do we know this always gives same prediction answer?
*
* Without predicates, loop entry and exit paths are ambiguous
* upon remaining input +b (in, say, a+b). Either paths lead to
* valid parses. Closure can lead to consuming + immediately or by
* falling out of this call to expr back into expr and loop back
* again to StarLoopEntryState to match +b. In this special case,
* we choose the more efficient path, which is to take the bypass
* path.
*
* The lookahead language has not changed because closure chooses
* one path over the other. Both paths lead to consuming the same
* remaining input during a lookahead operation. If the next token
* is an operator, lookahead will enter the choice block with
* operators. If it is not, lookahead will exit expr. Same as if
* closure had chosen to enter the choice block immediately.
*
* Closure is examining one config (some loopentrystate, some alt,
* context) which means it is considering exactly one alt. Closure
* always copies the same alt to any derived configs.
*
* How do we know this optimization doesn't mess up precedence in
* our parse trees?
*
* Looking through expr from left edge of stat only has to confirm
* that an input, say, a+b+c; begins with any valid interpretation
* of an expression. The precedence actually doesn't matter when
* making a decision in stat seeing through expr. It is only when
* parsing rule expr that we must use the precedence to get the
* right interpretation and, hence, parse tree.
*
* @since 4.6
*/
internal func canDropLoopEntryEdgeInLeftRecursiveRule(_ config: ATNConfig) -> Bool {
if ParserATNSimulator.TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT {
return false
}
let p: ATNState = config.state
guard let configContext = config.context else {
return false
}
// First check to see if we are in StarLoopEntryState generated during
// left-recursion elimination. For efficiency, also check if
// the context has an empty stack case. If so, it would mean
// global FOLLOW so we can't perform optimization
if p.getStateType() != ATNState.STAR_LOOP_ENTRY ||
!( (p as! StarLoopEntryState)).precedenceRuleDecision || // Are we the special loop entry/exit state?
configContext.isEmpty() || // If SLL wildcard
configContext.hasEmptyPath(){
return false
}
// Require all return states to return back to the same rule
// that p is in.
let numCtxs: Int = configContext.size()
for i in 0 ..< numCtxs { // for each stack context
let returnState: ATNState = atn.states[configContext.getReturnState(i)]!
if returnState.ruleIndex != p.ruleIndex
{return false}
}
let decisionStartState: BlockStartState = (p.transition(0).target as! BlockStartState)
let blockEndStateNum: Int = decisionStartState.endState!.stateNumber
let blockEndState: BlockEndState = (atn.states[blockEndStateNum] as! BlockEndState)
// Verify that the top of each stack context leads to loop entry/exit
// state through epsilon edges and w/o leaving rule.
for i in 0 ..< numCtxs { // for each stack context
let returnStateNumber: Int = configContext.getReturnState(i)
let returnState: ATNState = atn.states[returnStateNumber]!
// all states must have single outgoing epsilon edge
if returnState.getNumberOfTransitions() != 1 || !returnState.transition(0).isEpsilon(){
return false
}
// Look for prefix op case like 'not expr', (' type ')' expr
let returnStateTarget: ATNState = returnState.transition(0).target
if returnState.getStateType() == ATNState.BLOCK_END &&
returnStateTarget == p {
continue
}
// Look for 'expr op expr' or case where expr's return state is block end
// of (...)* internal block; the block end points to loop back
// which points to p but we don't need to check that
if returnState == blockEndState{
continue
}
// Look for ternary expr ? expr : expr. The return state points at block end,
// which points at loop entry state
if returnStateTarget == blockEndState{
continue
}
// Look for complex prefix 'between expr and expr' case where 2nd expr's
// return state points at block end state of (...)* internal block
if returnStateTarget.getStateType() == ATNState.BLOCK_END &&
returnStateTarget.getNumberOfTransitions() == 1 &&
returnStateTarget.transition(0).isEpsilon() &&
returnStateTarget.transition(0).target == p{
continue
}
// anything else ain't conforming
return false
}
return true
}
open func getRuleName(_ index: Int) -> String { open func getRuleName(_ index: Int) -> String {
if index >= 0 { if index >= 0 {
return parser.getRuleNames()[index] return parser.getRuleNames()[index]

54
tool/pom.xml
View File

@ -16,33 +16,33 @@
<url>http://www.antlr.org</url> <url>http://www.antlr.org</url>
<description>The ANTLR 4 grammar compiler.</description> <description>The ANTLR 4 grammar compiler.</description>
<dependencies> <dependencies>
<dependency> <dependency>
<groupId>org.antlr</groupId> <groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId> <artifactId>antlr4-runtime</artifactId>
<version>${project.version}</version> <version>${project.version}</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.antlr</groupId> <groupId>org.antlr</groupId>
<artifactId>antlr-runtime</artifactId> <artifactId>antlr-runtime</artifactId>
<version>3.5.2</version> <version>3.5.2</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.antlr</groupId> <groupId>org.antlr</groupId>
<artifactId>ST4</artifactId> <artifactId>ST4</artifactId>
<version>4.0.8</version> <version>4.0.8</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.abego.treelayout</groupId> <groupId>org.abego.treelayout</groupId>
<artifactId>org.abego.treelayout.core</artifactId> <artifactId>org.abego.treelayout.core</artifactId>
<version>1.0.3</version> <version>1.0.3</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>javax.json</groupId> <groupId>org.glassfish</groupId>
<artifactId>javax.json-api</artifactId> <artifactId>javax.json</artifactId>
<version>1.0</version> <version>1.0.4</version>
</dependency> </dependency>
</dependencies> </dependencies>
<build> <build>
<sourceDirectory>src</sourceDirectory> <sourceDirectory>src</sourceDirectory>

2
tool/resources/org/antlr/v4/tool/templates/codegen/JavaScript/JavaScript.stg
View File

@ -236,7 +236,7 @@ case <index>:
* overriding implementation impossible to maintain. * overriding implementation impossible to maintain.
*/ */
RuleSempredFunction(r, actions) ::= << RuleSempredFunction(r, actions) ::= <<
<if(parser)><parser.name><else><lexer.name><endif>.prototype.<r.name>_sempred = function(localctx, predIndex) { <if (r.factory.g.lexer)><lexer.name><else><parser.name><endif>.prototype.<r.name>_sempred = function(localctx, predIndex) {
switch(predIndex) { switch(predIndex) {
<actions:{index| case <index>: <actions:{index| case <index>:
return <actions.(index)>;}; separator="\n"> return <actions.(index)>;}; separator="\n">