diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1d7960a69..e2af2261b 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,5 +1,5 @@ [cut] -Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors most "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership. +Thank you for proposing a contribution to the ANTLR project. In order to accept changes from the outside world, all contributors must "sign" the [contributors.txt](https://github.com/antlr/antlr4/blob/master/contributors.txt) contributors certificate of origin. It's an unfortunate reality of today's fuzzy and bizarre world of open-source ownership. Make sure you are already in the contributors.txt file or add a commit to this pull request with the appropriate change. Thanks! [/cut] diff --git a/README.md b/README.md index 86a69d4c3..d26d18a66 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ *Given day-job constraints, my time working on this project is limited so I'll have to focus first on fixing bugs rather than changing/improving the feature set. Likely I'll do it in bursts every few months. Please do not be offended if your bug or pull request does not yield a response! --parrt* +[![Donate](https://www.paypal.com/en_US/i/btn/x-click-butcc-donate.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BF92STRXT8F8Q) + ## Authors and major contributors * [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu diff --git a/doc/adding-tests.md b/doc/antlr-project-testing.md similarity index 61% rename from doc/adding-tests.md rename to doc/antlr-project-testing.md index de4787041..e29b00cec 100644 --- a/doc/adding-tests.md +++ b/doc/antlr-project-testing.md @@ -1,4 +1,4 @@ -# Adding unit tests +# ANTLR project unit tests ## Introduction @@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow The grammars are strings representing StringTemplates (`ST` objects) so `` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the [Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). +## Requirements + +In order to perform the tests on all target languages, you need to have the following languages installed: + +* `mono` (e.g., `brew install mono`) on non-Windows boxes (on Windows it uses the Microsoft .net stack). Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below +* `nodejs` +* Python 2.7 +* Python 3.5 +* Go +* Swift 3 (via XCode 8.x) tested currently only osx +* clang (for C++ target) +* +To **install into local repository** `~/.m2/repository/org/antlr`, do this: + +```bash +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux +$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine +``` + +Now, make sure C# runtime is built and installed locally. + +```bash +cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp +# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it +rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll +rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll +# build +xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj +``` + +C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib. + + ## Running the runtime tests A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of @@ -49,26 +82,128 @@ And the result of testing the entire subdirectory: From `mvn`, on the commandline, you will see: ```bash +$ cd antlr4 +$ mvn test +... ------------------------------------------------------- T E S T S ------------------------------------------------------- -Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec -Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors -Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser -Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSets -Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec -Running org.antlr.v4.test.runtime.javascript.node.TestPerformance -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer -Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec -Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec -Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec +Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451 +Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081 +Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj +Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec +Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864 +antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4] +... +[INFO] ------------------------------------------------------------------------ +[INFO] Reactor Summary: +[INFO] +[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s] +[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s] +[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s] +[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s] +[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s] +[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s] +[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min] +[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s] +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 06:53 min +[INFO] Finished at: 2016-11-16T15:36:56-08:00 +[INFO] Final Memory: 44M/458M +[INFO] ------------------------------------------------------------------------ +``` + +Note: That is actually result of running the much faster: + +```bash +mvn -Dparallel=methods -DthreadCount=4 install +``` + +## Running test subsets + +*From the `runtime-testsuite` dir* + +### Run one test group across targets + +```bash +$ cd runtime-testsuite +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux +$ mvn -Dtest=TestParserExec test +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.cpp.TestParserExec +... +Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec +Running org.antlr.v4.test.runtime.csharp.TestParserExec ... ``` +Or run all lexer related tests: + +``` +$ cd runtime-testsuite +$ mvn -Dtest=Test*Lexer* test +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers +... +``` + +### Run all tests for a single target + +```bash +$ cd runtime-testsuite +$ mvn -Dtest=java.* test +... +``` + +Or run all lexer related tests in Java target only: + +```bash +$ cd runtime-testsuite +$ mvn -Dtest=java.*Lexer* test +... +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.java.TestCompositeLexers +Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec +Running org.antlr.v4.test.runtime.java.TestLexerErrors +Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec +Running org.antlr.v4.test.runtime.java.TestLexerExec +Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec +Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer +Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec + +Results : + +Tests run: 59, Failures: 0, Errors: 0, Skipped: 0 +``` + +## Testing in parallel + +Use this to run tests in parallel: + +```bash +$ export MAVEN_OPTS="-Xmx1G" +$ mvn -Dparallel=methods -DthreadCount=4 test +... +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false +... +``` + +This can be combined with other `-D` above. + ## Adding a runtime test To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of: diff --git a/doc/building-antlr.md b/doc/building-antlr.md index b4a52e5b9..ac220bc30 100644 --- a/doc/building-antlr.md +++ b/doc/building-antlr.md @@ -38,6 +38,7 @@ Receiving objects: 100% (59858/59858), 31.10 MiB | 819.00 KiB/s, done. Resolving deltas: 100% (31898/31898), done. Checking connectivity... done. $ cd antlr4 +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn -DskipTests install ... [INFO] ------------------------------------------------------------------------ @@ -67,65 +68,10 @@ We do `install` not `compile` as tool tests and such refer to modules that must To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this: ```bash +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests=true # make sure all artifacts are visible on this machine ``` -# Testing tool and targets - -In order to perform the tests on all target languages, you need to have the following languages installed: - -* `mono` (e.g., `brew install mono`) -* `nodejs` -* Python 2.7 -* Python 3.5 -* Go -* Swift 3 (via XCode 8.x) tested currently only osx -* clang (for C++ target) - -To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this: - -```bash -$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine -$ mvn install # now "do it with feeling" -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451 -Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081 -Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec -Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864 -antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4] -... -[INFO] ------------------------------------------------------------------------ -[INFO] Reactor Summary: -[INFO] -[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s] -[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s] -[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s] -[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s] -[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s] -[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s] -[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min] -[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s] -[INFO] ------------------------------------------------------------------------ -[INFO] BUILD SUCCESS -[INFO] ------------------------------------------------------------------------ -[INFO] Total time: 06:53 min -[INFO] Finished at: 2016-11-16T15:36:56-08:00 -[INFO] Final Memory: 44M/458M -[INFO] ------------------------------------------------------------------------ -``` - -Note: That is actually result of running the much faster: - -`mvn -Dparallel=methods -DthreadCount=4 install` - - You should see these jars (when building 4.6-SNAPSHOT): ```bash @@ -143,87 +89,17 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. -## Running test subsets +# Testing tool and targets -*From the `runtime-testsuite` dir* +See [ANTLR project unit tests](antlr-project-testing.md). -### Run one test group across targets -```bash -$ cd runtime-testsuite -$ mvn -Dtest=TestParserExec test -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.cpp.TestParserExec -... -Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec -Running org.antlr.v4.test.runtime.csharp.TestParserExec -... -``` - -Or run all lexer related tests: - -``` -$ mvn -Dtest=Test*Lexer* test -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers -... -``` - -### Run all tests for a single target - -```bash -$ mvn -Dtest=java.* test -... -``` - -Or run all lexer related tests in Java target only: - -```bash -$ mvn -Dtest=java.*Lexer* test -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.java.TestCompositeLexers -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec -Running org.antlr.v4.test.runtime.java.TestLexerErrors -Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec -Running org.antlr.v4.test.runtime.java.TestLexerExec -Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec -Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer -Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec - -Results : - -Tests run: 59, Failures: 0, Errors: 0, Skipped: 0 -``` - -## Testing in parallel - -Use this to run tests in parallel: - -```bash -$ mvn -Dparallel=methods -DthreadCount=4 test -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false -... -``` - -This can be combined with other `-D` above. - -## Building without testing +# Building without testing To build without running the tests (saves a lot of time), do this: ```bash -mvn -DskipTests install +$ mvn -DskipTests install ``` ## Building ANTLR in Intellij IDE diff --git a/doc/index.md b/doc/index.md index fce24d1b2..d5c55518e 100644 --- a/doc/index.md +++ b/doc/index.md @@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema * [Cutting an ANTLR Release](releasing-antlr.md) -* [Adding ANTLR unit tests](adding-tests.md) +* [ANTLR project unit tests](antlr-project-testing.md) * [Creating an ANTLR Language Target](creating-a-language-target.md) diff --git a/doc/releasing-antlr.md b/doc/releasing-antlr.md index 715b282d2..01ce06a01 100644 --- a/doc/releasing-antlr.md +++ b/doc/releasing-antlr.md @@ -4,12 +4,22 @@ Create a pre-release or full release at github; [Example 4.5-rc-1](https://github.com/antlr/antlr4/releases/tag/4.5-rc-1). +### Delete existing release tag + Wack any existing tag as mvn will create one and it fails if already there. ``` -$ git tag -d 4.5.2 -$ git push origin :refs/tags/4.5.2 -$ git push upstream :refs/tags/4.5.2 +$ git tag -d 4.6 +$ git push origin :refs/tags/4.6 +$ git push upstream :refs/tags/4.6 +``` + +### Create release candidate tag + +```bash +$ git tag -a 4.6-rc1 -m 'heading towards 4.6' +$ git push origin 4.6-rc1 +$ git push upstream 4.6-rc1 ``` ## Bump version @@ -22,15 +32,17 @@ Edit the repository looking for 4.5 or whatever and update it. Bump version in t * runtime/Python3/setup.py * runtime/Python3/src/antlr4/Recognizer.py * runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs + * runtime/CSharp/build/version.ps1 * runtime/JavaScript/src/antlr4/package.json * runtime/JavaScript/src/antlr4/Recognizer.js + * runtime/Cpp/VERSION + * runtime/Cpp/runtime/src/RuntimeMetaData.cpp + * runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake * tool/src/org/antlr/v4/codegen/target/CppTarget.java * tool/src/org/antlr/v4/codegen/target/CSharpTarget.java * tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java * tool/src/org/antlr/v4/codegen/target/Python2Target.java * tool/src/org/antlr/v4/codegen/target/Python3Target.java - * runtime/Cpp/VERSION - * runtime/Cpp/RuntimeMetaData.cpp Here is a simple script to display any line from the critical files with, say, `4.5` in it: @@ -144,24 +156,24 @@ It will start out by asking you the version number: ``` ... -What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.2: : 4.5.2 -What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.5.2: : -What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.5.2: : -What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.5.2: : -What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.5.2: : -What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.5.2: : -What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.5.2: : 4.5.2 -What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.5.3-SNAPSHOT: +What is the release version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6: : 4.6 +What is the release version for "ANTLR 4 Runtime"? (org.antlr:antlr4-runtime) 4.6: : +What is the release version for "ANTLR 4 Tool"? (org.antlr:antlr4) 4.6: : +What is the release version for "ANTLR 4 Maven plugin"? (org.antlr:antlr4-maven-plugin) 4.6: : +What is the release version for "ANTLR 4 Runtime Test Generator"? (org.antlr:antlr4-runtime-testsuite) 4.6: : +What is the release version for "ANTLR 4 Tool Tests"? (org.antlr:antlr4-tool-testsuite) 4.6: : +What is SCM release tag or label for "ANTLR 4"? (org.antlr:antlr4-master) antlr4-master-4.6: : 4.6 +What is the new development version for "ANTLR 4"? (org.antlr:antlr4-master) 4.6.1-SNAPSHOT: ... ``` -Maven will go through your pom.xml files to update versions from 4.5.2-SNAPSHOT to 4.5.2 for release and then to 4.5.3-SNAPSHOT after release, which is done with: +Maven will go through your pom.xml files to update versions from 4.6-SNAPSHOT to 4.6 for release and then to 4.6.1-SNAPSHOT after release, which is done with: ```bash mvn release:perform -Darguments="-DskipTests" ``` -Maven will use git to push pom.xml changes. (big smile) +Maven will use git to push pom.xml changes. Now, go here: @@ -174,11 +186,11 @@ and on the left click "Staging Repositories". You click the staging repo and clo Copy the jars to antlr.org site and update download/index.html ```bash -cp ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.5.2.jar -cp ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2.jar ~/antlr/sites/website-antlr4/download/antlr-4.5.2-complete.jar +cp ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6.jar ~/antlr/sites/website-antlr4/download/antlr-runtime-4.6.jar +cp ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-complete.jar ~/antlr/sites/website-antlr4/download/antlr-4.6-complete.jar cd ~/antlr/sites/website-antlr4/download -git add antlr-4.5.2-complete.jar -git add antlr-runtime-4.5.2.jar +git add antlr-4.6-complete.jar +git add antlr-runtime-4.6.jar ``` Update on site: @@ -190,7 +202,7 @@ Update on site: * scripts/topnav.js ``` -git commit -a -m 'add 4.5.2 jars' +git commit -a -m 'add 4.6 jars' git push origin gh-pages ``` @@ -200,8 +212,8 @@ git push origin gh-pages ```bash cd runtime/JavaScript/src -zip -r /tmp/antlr-javascript-runtime-4.5.2.zip antlr4 -cp /tmp/antlr-javascript-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download +zip -r /tmp/antlr-javascript-runtime-4.6.zip antlr4 +cp /tmp/antlr-javascript-runtime-4.6.zip ~/antlr/sites/website-antlr4/download # git add, commit, push ``` @@ -209,7 +221,7 @@ Move target to website ```bash pushd ~/antlr/sites/website-antlr4/download -git add antlr-javascript-runtime-4.5.2.zip +git add antlr-javascript-runtime-4.6.zip git commit -a -m 'update JS runtime' git push origin gh-pages popd @@ -225,15 +237,15 @@ rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll # build xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj # zip it up to get a version number on zip filename -zip --junk-paths /tmp/antlr-csharp-runtime-4.5.2.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll -cp /tmp/antlr-csharp-runtime-4.5.2.zip ~/antlr/sites/website-antlr4/download +zip --junk-paths /tmp/antlr-csharp-runtime-4.6.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll +cp /tmp/antlr-csharp-runtime-4.6.zip ~/antlr/sites/website-antlr4/download ``` Move target to website ```bash pushd ~/antlr/sites/website-antlr4/download -git add antlr-csharp-runtime-4.5.2.zip +git add antlr-csharp-runtime-4.6.zip git commit -a -m 'update C# runtime' git push origin gh-pages popd @@ -346,9 +358,9 @@ cd ~/antlr/sites/website-antlr4/api git checkout gh-pages git pull origin gh-pages cd Java -jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.5.2/antlr4-runtime-4.5.2-javadoc.jar +jar xvf ~/.m2/repository/org/antlr/antlr4-runtime/4.6/antlr4-runtime-4.6-javadoc.jar cd ../JavaTool -jar xvf ~/.m2/repository/org/antlr/antlr4/4.5.2/antlr4-4.5.2-javadoc.jar +jar xvf ~/.m2/repository/org/antlr/antlr4/4.6/antlr4-4.6-javadoc.jar git commit -a -m 'freshen api doc' git push origin gh-pages ``` diff --git a/doc/runtimetests-overview.md b/doc/runtimetests-overview.md index d7247a20c..13085869e 100644 --- a/doc/runtimetests-overview.md +++ b/doc/runtimetests-overview.md @@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate ## Generating JUnit Tests -The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run +The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run ```bash $ mvn -Pgen generate-test-sources diff --git a/doc/targets.md b/doc/targets.md index 6c36bf5b4..418630bb1 100644 --- a/doc/targets.md +++ b/doc/targets.md @@ -14,7 +14,7 @@ This page lists the available and upcoming ANTLR runtimes. Please note that you New features generally appear in the Java target and then migrate to the other targets, but these other targets don't always get updated in the same overall tool release. This section tries to identify features added to Java that have not been added to the other targets. -|Feature|Java|C♯|JavaScript|Python2|Python3|Swift|C++| -|---|---|---|---|---|---|---|---| -|Ambiguous tree construction|4.5.1|-|-|-|-|-|-| +|Feature|Java|C♯|Python2|Python3|JavaScript|Go|C++|Swift| +|---|---|---|---|---|---|---|---|---| +|Ambiguous tree construction|4.5.1|-|-|-|-|-|-|-| diff --git a/runtime-testsuite/pom.xml b/runtime-testsuite/pom.xml index 50555409f..afbc3aa17 100644 --- a/runtime-testsuite/pom.xml +++ b/runtime-testsuite/pom.xml @@ -79,107 +79,59 @@ - - test - - - resources - - - ../runtime - - - - - org.apache.maven.plugins - maven-surefire-plugin - 2.19.1 - - - **/csharp/Test*.java - **/java/Test*.java - **/go/Test*.java - **/javascript/node/Test*.java - **/python2/Test*.java - **/python3/Test*.java - ${antlr.tests.swift} - - - - - org.apache.maven.plugins - maven-jar-plugin - 2.4 - - - - test-jar - - - - - - + + test + + + resources + + + ../runtime + + + + + org.apache.maven.plugins + maven-surefire-plugin + 2.19.1 + + + **/csharp/Test*.java + **/java/Test*.java + **/go/Test*.java + **/javascript/node/Test*.java + **/python2/Test*.java + **/python3/Test*.java + ${antlr.tests.swift} + + + + + org.apache.maven.plugins + maven-jar-plugin + 2.4 + + + + test-jar + + + + + + - - - includeSwiftTests - - - mac - - - - **/swift/Test*.java - - - - - gen - - - - org.codehaus.mojo - exec-maven-plugin - 1.4.0 - - - generate-test-sources - - java - - - org.antlr.v4.testgen.TestGenerator - - -root - ${basedir} - -outdir - ${basedir}/test - -templates - ${basedir}/resources/org/antlr/v4/test/runtime/templates - - - - - - - - - - tests - - - - org.apache.maven.plugins - maven-surefire-plugin - - - **/Test*.java - - - - - - - + + + includeSwiftTests + + + mac + + + + **/swift/Test*.java + + + diff --git a/runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java b/runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java index 146812f8f..3beffe94d 100644 --- a/runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java +++ b/runtime-testsuite/test/org/antlr/v4/test/runtime/cpp/BaseCppTest.java @@ -55,7 +55,9 @@ import java.io.InputStream; import java.io.InputStreamReader; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; +import java.net.URISyntaxException; import java.net.URL; +import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -508,7 +510,7 @@ public class BaseCppTest implements RuntimeTestSupport { return files; } - private String runProcess(ProcessBuilder builder, String description) throws Exception { + private String runProcess(ProcessBuilder builder, String description, boolean showStderr) throws Exception { // System.out.println("BUILDER: "+builder.command()); Process process = builder.start(); StreamVacuum stdoutVacuum = new StreamVacuum(process.getInputStream()); @@ -521,10 +523,10 @@ public class BaseCppTest implements RuntimeTestSupport { String output = stdoutVacuum.toString(); if ( stderrVacuum.toString().length()>0 ) { this.stderrDuringParse = stderrVacuum.toString(); -// System.err.println(this.stderrDuringParse); + if ( showStderr ) System.err.println(this.stderrDuringParse); } if (errcode != 0) { - String err = "execution failed with error code: "+errcode; + String err = "execution of '"+description+"' failed with error code: "+errcode; if ( this.stderrDuringParse!=null ) { this.stderrDuringParse += err; } @@ -536,11 +538,11 @@ public class BaseCppTest implements RuntimeTestSupport { return output; } - private String runCommand(String command[], String workPath, String description) throws Exception { + private String runCommand(String command[], String workPath, String description, boolean showStderr) throws Exception { ProcessBuilder builder = new ProcessBuilder(command); builder.directory(new File(workPath)); - return runProcess(builder, description); + return runProcess(builder, description, showStderr); } // TODO: add a buildRuntimeOnWindows variant. @@ -550,8 +552,9 @@ public class BaseCppTest implements RuntimeTestSupport { try { String command[] = { "cmake", ".", /*"-DCMAKE_CXX_COMPILER=clang++",*/ "-DCMAKE_BUILD_TYPE=release" }; - if (runCommand(command, runtimePath, "antlr runtime cmake") == null) + if (runCommand(command, runtimePath, "antlr runtime cmake", false) == null) { return false; + } } catch (Exception e) { System.err.println("can't configure antlr cpp runtime cmake file"); @@ -559,11 +562,21 @@ public class BaseCppTest implements RuntimeTestSupport { try { String command[] = { "make", "-j", "8" }; // Assuming a reasonable amount of available CPU cores. - if (runCommand(command, runtimePath, "building antlr runtime") == null) + if (runCommand(command, runtimePath, "building antlr runtime", true) == null) return false; } catch (Exception e) { System.err.println("can't compile antlr cpp runtime"); + e.printStackTrace(System.err); + try { + String command[] = { "ls", "-la" }; + String output = runCommand(command, runtimePath + "/dist/", "printing library folder content", true); + System.out.println(output); + } + catch (Exception e2) { + System.err.println("can't even list folder content"); + e2.printStackTrace(System.err); + } } /* for debugging @@ -593,7 +606,7 @@ public class BaseCppTest implements RuntimeTestSupport { if ( !runtimeBuiltOnce ) { try { String command[] = {"clang++", "--version"}; - String output = runCommand(command, tmpdir, "printing compiler version"); + String output = runCommand(command, tmpdir, "printing compiler version", false); System.out.println("Compiler version is: "+output); } catch (Exception e) { @@ -613,7 +626,7 @@ public class BaseCppTest implements RuntimeTestSupport { String libExtension = (getOS().equals("mac")) ? "dylib" : "so"; try { String command[] = { "ln", "-s", runtimePath + "/dist/libantlr4-runtime." + libExtension }; - if (runCommand(command, tmpdir, "sym linking C++ runtime") == null) + if (runCommand(command, tmpdir, "sym linking C++ runtime", true) == null) return null; } catch (Exception e) { @@ -625,7 +638,7 @@ public class BaseCppTest implements RuntimeTestSupport { try { List command2 = new ArrayList(Arrays.asList("clang++", "-std=c++11", "-I", includePath, "-L.", "-lantlr4-runtime", "-o", "a.out")); command2.addAll(allCppFiles(tmpdir)); - if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary") == null) { + if (runCommand(command2.toArray(new String[0]), tmpdir, "building test binary", true) == null) { return null; } } @@ -642,7 +655,7 @@ public class BaseCppTest implements RuntimeTestSupport { builder.directory(new File(tmpdir)); Map env = builder.environment(); env.put("LD_PRELOAD", runtimePath + "/dist/libantlr4-runtime." + libExtension); - String output = runProcess(builder, "running test binary"); + String output = runProcess(builder, "running test binary", false); if ( output.length()==0 ) { output = null; } @@ -664,11 +677,21 @@ public class BaseCppTest implements RuntimeTestSupport { protected String locateRuntime() { final ClassLoader loader = Thread.currentThread().getContextClassLoader(); - final URL runtimeSrc = loader.getResource("Cpp"); - if (runtimeSrc == null) { + final URL runtimeURL = loader.getResource("Cpp"); + if (runtimeURL == null) { throw new RuntimeException("Cannot find runtime"); } - return runtimeSrc.getPath(); + // Windows not getting runtime right. See: + // http://stackoverflow.com/questions/6164448/convert-url-to-normal-windows-filename-java + // it was coming back "/C:/projects/antlr4-l7imv/runtime-testsuite/target/classes/Cpp" + String p; + try { + p = Paths.get(runtimeURL.toURI()).toFile().toString(); + } + catch (URISyntaxException use) { + p = "Can't find runtime"; + } + return p; } List getMessagesOfType(List msgs, Class c) { diff --git a/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParseTreesDescriptors.java b/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParseTreesDescriptors.java index feb32081f..6925fc8eb 100644 --- a/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParseTreesDescriptors.java +++ b/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParseTreesDescriptors.java @@ -112,7 +112,7 @@ public class ParseTreesDescriptors { @Override public boolean ignore(String targetName) { - return !targetName.matches("Java|Python2|Python3|Node"); + return !targetName.matches("Java|Python2|Python3|Node|Swift"); } } diff --git a/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/SemPredEvalLexerDescriptors.java b/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/SemPredEvalLexerDescriptors.java index fc43a6f83..1a9e6e621 100644 --- a/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/SemPredEvalLexerDescriptors.java +++ b/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/SemPredEvalLexerDescriptors.java @@ -10,6 +10,30 @@ import org.antlr.v4.test.runtime.BaseLexerTestDescriptor; import org.antlr.v4.test.runtime.CommentHasStringValue; public class SemPredEvalLexerDescriptors { + // Test for https://github.com/antlr/antlr4/issues/958 + public static class RuleSempredFunction extends BaseLexerTestDescriptor { + public String input = "aaa"; + /** + [@0,0:0='a',<1>,1:0] + [@1,1:1='a',<1>,1:1] + [@2,2:2='a',<1>,1:2] + [@3,3:2='',<-1>,1:3] + */ + @CommentHasStringValue + public String output; + + public String errors = null; + public String startRule = ""; + public String grammarName = "L"; + + /** + lexer grammar L; + T : 'a' {}? ; + */ + @CommentHasStringValue + public String grammar; + } + public static class DisableRule extends BaseLexerTestDescriptor { public String input = "enum abc"; /** diff --git a/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs b/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs index 3bb88e173..580d104e3 100644 --- a/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs +++ b/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs @@ -42,8 +42,8 @@ using System.Runtime.InteropServices; // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] -[assembly: AssemblyVersion("4.5.3.0")] +[assembly: AssemblyVersion("4.6.0")] #if !COMPACT -[assembly: AssemblyFileVersion("4.5.3.0")] -[assembly: AssemblyInformationalVersion("4.5.3.0")] +[assembly: AssemblyFileVersion("4.6.0")] +[assembly: AssemblyInformationalVersion("4.6.0")] #endif diff --git a/runtime/Cpp/VERSION b/runtime/Cpp/VERSION index 0ed2bfe96..b3d791d75 100644 --- a/runtime/Cpp/VERSION +++ b/runtime/Cpp/VERSION @@ -1 +1 @@ -4.5.4 +4.6 diff --git a/runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake b/runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake index 1cb579a4d..eda023845 100644 --- a/runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake +++ b/runtime/Cpp/cmake/ExternalAntlr4Cpp.cmake @@ -19,11 +19,11 @@ # # next make a build folder e.g. ~/buildfolder/ # from the buildfolder, run cmake ~/srcfolder; make -# +# ############################################################### # # minimum required CMAKE version # CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2 FATAL_ERROR) -# +# # LIST( APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake ) # # # compiler must be 11 or 14 @@ -49,7 +49,7 @@ # add_executable(demo main.cpp ${antlr4cpp_src_files_antlrcpptest}) # add_dependencies(demo antlr4cpp antlr4cpp_generation_antlrcpptest) # target_link_libraries(demo antlr4-runtime) -# +# ############################################################### CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12.2) @@ -65,8 +65,8 @@ set(ANTLR4CPP_EXTERNAL_ROOT ${CMAKE_BINARY_DIR}/externals/antlr4cpp) # external repository # GIT_REPOSITORY https://github.com/antlr/antlr4.git -set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/DanMcLaughlin/antlr4.git") -set(ANTLR4CPP_EXTERNAL_TAG "v4.5.4") +set(ANTLR4CPP_EXTERNAL_REPO "https://github.com/antlr/antlr4.git") +set(ANTLR4CPP_EXTERNAL_TAG "4.6-rc1") if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}") message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}") @@ -81,13 +81,13 @@ endif() # set(ANTLR4CPP_JAR_LOCATION ${ANTLR4CPP_EXTERNAL_ROOT}/${ANTLR4CPP_JAR_NAME}) # # !TODO! Ensure Antlr tool available - copy from internet -# +# # # !TODO! this shold be calculated based on the tags # if (NOT ANTLR4CPP_JAR_NAME) # # default location to find antlr Java binary # set(ANTLR4CPP_JAR_NAME antlr4-4.5.4-SNAPSHOT.jar) # endif() -# +# # if(NOT EXISTS "${ANTLR4CPP_JAR_LOCATION}") # # download Java tool if not installed # ExternalProject_ADD( @@ -162,20 +162,20 @@ endforeach(src_path) set(ANTLR4CPP_LIBS "${INSTALL_DIR}/lib") -# antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so -# antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a +# antlr4_shared ${INSTALL_DIR}/lib/libantlr4-runtime.so +# antlr4_static ${INSTALL_DIR}/lib/libantlr4-runtime.a ############ Generate runtime ################# # macro to add dependencies to target # -# Param 1 project name +# Param 1 project name # Param 1 namespace (postfix for dependencies) # Param 2 Lexer file (full path) # Param 3 Parser File (full path) # # output # -# antlr4cpp_src_files_{namespace} - src files for add_executable +# antlr4cpp_src_files_{namespace} - src files for add_executable # antlr4cpp_include_dirs_{namespace} - include dir for generated headers # antlr4cpp_generation_{namespace} - for add_dependencies tracking @@ -184,18 +184,18 @@ macro(antlr4cpp_process_grammar antlr4cpp_project_namespace antlr4cpp_grammar_lexer antlr4cpp_grammar_parser) - + if(EXISTS "${ANTLR4CPP_JAR_LOCATION}") message(STATUS "Found antlr tool: ${ANTLR4CPP_JAR_LOCATION}") else() message(FATAL_ERROR "Unable to find antlr tool. ANTLR4CPP_JAR_LOCATION:${ANTLR4CPP_JAR_LOCATION}") endif() - + add_custom_target("antlr4cpp_generation_${antlr4cpp_project_namespace}" - COMMAND + COMMAND ${CMAKE_COMMAND} -E make_directory ${ANTLR4CPP_GENERATED_SRC_DIR} COMMAND - "${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}" + "${Java_JAVA_EXECUTABLE}" -jar "${ANTLR4CPP_JAR_LOCATION}" -Werror -Dlanguage=Cpp -listener -visitor -o "${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}" -package ${antlr4cpp_project_namespace} "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}" WORKING_DIRECTORY "${CMAKE_BINARY_DIR}" DEPENDS "${antlr4cpp_grammar_lexer}" "${antlr4cpp_grammar_parser}" ) @@ -213,9 +213,9 @@ macro(antlr4cpp_process_grammar ) endforeach(generated_file) message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} Generated: ${generated_files}") - + # export generated include directory set(antlr4cpp_include_dirs_${antlr4cpp_project_namespace} ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}) message(STATUS "Antlr4Cpp ${antlr4cpp_project_namespace} include: ${ANTLR4CPP_GENERATED_SRC_DIR}/${antlr4cpp_project_namespace}") - + endmacro() diff --git a/runtime/Cpp/runtime/src/RuntimeMetaData.cpp b/runtime/Cpp/runtime/src/RuntimeMetaData.cpp index 15f940e90..f5bdacac5 100755 --- a/runtime/Cpp/runtime/src/RuntimeMetaData.cpp +++ b/runtime/Cpp/runtime/src/RuntimeMetaData.cpp @@ -7,7 +7,7 @@ using namespace antlr4; -const std::string RuntimeMetaData::VERSION = "4.5.4"; +const std::string RuntimeMetaData::VERSION = "4.6"; std::string RuntimeMetaData::getRuntimeVersion() { return VERSION; diff --git a/runtime/Python2/src/antlr4/error/ErrorStrategy.py b/runtime/Python2/src/antlr4/error/ErrorStrategy.py index ec6a5d4c7..586a7933d 100644 --- a/runtime/Python2/src/antlr4/error/ErrorStrategy.py +++ b/runtime/Python2/src/antlr4/error/ErrorStrategy.py @@ -484,14 +484,14 @@ class DefaultErrorStrategy(ErrorStrategy): expecting = self.getExpectedTokens(recognizer) expectedTokenType = expecting[0] # get any element if expectedTokenType==Token.EOF: - tokenText = "" + tokenText = u"" else: name = None if expectedTokenType < len(recognizer.literalNames): name = recognizer.literalNames[expectedTokenType] if name is None and expectedTokenType < len(recognizer.symbolicNames): name = recognizer.symbolicNames[expectedTokenType] - tokenText = "" + tokenText = u"" current = currentSymbol lookback = recognizer.getTokenStream().LT(-1) if current.type==Token.EOF and lookback is not None: diff --git a/runtime/Swift/Antlr4/org/antlr/v4/runtime/BufferedTokenStream.swift b/runtime/Swift/Antlr4/org/antlr/v4/runtime/BufferedTokenStream.swift index 1c1122812..1f0ed2c44 100644 --- a/runtime/Swift/Antlr4/org/antlr/v4/runtime/BufferedTokenStream.swift +++ b/runtime/Swift/Antlr4/org/antlr/v4/runtime/BufferedTokenStream.swift @@ -473,8 +473,6 @@ public class BufferedTokenStream: TokenStream { public func getText() throws -> String { - try lazyInit() - try fill() return try getText(Interval.of(0, size() - 1)) } @@ -485,7 +483,7 @@ public class BufferedTokenStream: TokenStream { if start < 0 || stop < 0 { return "" } - try lazyInit() + try fill() if stop >= tokens.count { stop = tokens.count - 1 } diff --git a/runtime/Swift/Antlr4/org/antlr/v4/runtime/CommonToken.swift b/runtime/Swift/Antlr4/org/antlr/v4/runtime/CommonToken.swift index f9d8541ff..4ca4094da 100644 --- a/runtime/Swift/Antlr4/org/antlr/v4/runtime/CommonToken.swift +++ b/runtime/Swift/Antlr4/org/antlr/v4/runtime/CommonToken.swift @@ -255,9 +255,13 @@ public class CommonToken: WritableToken { } public var description: String { + return toString(nil) + } + + public func toString(_ r: Recognizer?) -> String { var channelStr: String = "" if channel > 0 { - channelStr = "channel=\(channel)" + channelStr = ",channel=\(channel)" } var txt: String if let tokenText = getText() { @@ -267,15 +271,19 @@ public class CommonToken: WritableToken { } else { txt = "" } - let desc: StringBuilder = StringBuilder() - desc.append("[@\(getTokenIndex()),") - desc.append("\(start):\(stop)='\(txt)',") - desc.append("<\(type)>\(channelStr),") - desc.append("\(line):\(getCharPositionInLine())]") - - return desc.toString() + var typeString = "\(type)" + if let r = r { + typeString = r.getVocabulary().getDisplayName(type); + } + return "[@"+getTokenIndex()+","+start+":"+stop+"='"+txt+"',<"+typeString+">"+channelStr+","+line+":"+getCharPositionInLine()+"]" +// let desc: StringBuilder = StringBuilder() +// desc.append("[@\(getTokenIndex()),") +// desc.append("\(start):\(stop)='\(txt)',") +// desc.append("<\(typeString)>\(channelStr),") +// desc.append("\(line):\(getCharPositionInLine())]") +// +// return desc.toString() } - public var visited: Bool { get { return _visited diff --git a/runtime/Swift/Antlr4/org/antlr/v4/runtime/ParserRuleContext.swift b/runtime/Swift/Antlr4/org/antlr/v4/runtime/ParserRuleContext.swift index 1e36ac7a1..13c91d6ef 100644 --- a/runtime/Swift/Antlr4/org/antlr/v4/runtime/ParserRuleContext.swift +++ b/runtime/Swift/Antlr4/org/antlr/v4/runtime/ParserRuleContext.swift @@ -79,6 +79,18 @@ open class ParserRuleContext: RuleContext { self.start = ctx.start self.stop = ctx.stop + + // copy any error nodes to alt label node + if ctx.children != nil{ + self.children = Array() + // reset parent pointer for any error nodes + for child: ParseTree in ctx.children! { + if child is ErrorNode{ + self.children?.append(child) + ( (child as! ErrorNode)).parent = self + } + } + } } public init(_ parent: ParserRuleContext?, _ invokingStateNumber: Int) { diff --git a/runtime/Swift/Antlr4/org/antlr/v4/runtime/Recognizer.swift b/runtime/Swift/Antlr4/org/antlr/v4/runtime/Recognizer.swift index 735a5dd37..811c5da10 100644 --- a/runtime/Swift/Antlr4/org/antlr/v4/runtime/Recognizer.swift +++ b/runtime/Swift/Antlr4/org/antlr/v4/runtime/Recognizer.swift @@ -62,7 +62,7 @@ open class Recognizer { if result == nil { result = Dictionary() let length = self.getATN().maxTokenType - for i in 0.. */ +import Foundation open class ParserATNSimulator: ATNSimulator { public let debug: Bool = false public let debug_list_atn_decisions: Bool = false public let dfa_debug: Bool = false public let retry_debug: Bool = false - + /** Just in case this optimization is bad, add an ENV variable to turn it off */ + public static let TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT: Bool = { + if let value = ProcessInfo.processInfo.environment["TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT"] { + return NSString(string: value).boolValue + } + return false + }() internal final var parser: Parser public final var decisionToDFA: [DFA] @@ -1441,6 +1448,10 @@ open class ParserATNSimulator: ATNSimulator { } let length = p.getNumberOfTransitions() for i in 0.. Bool { + if ParserATNSimulator.TURN_OFF_LR_LOOP_ENTRY_BRANCH_OPT { + return false + } + let p: ATNState = config.state + guard let configContext = config.context else { + return false + } + // First check to see if we are in StarLoopEntryState generated during + // left-recursion elimination. For efficiency, also check if + // the context has an empty stack case. If so, it would mean + // global FOLLOW so we can't perform optimization + if p.getStateType() != ATNState.STAR_LOOP_ENTRY || + !( (p as! StarLoopEntryState)).precedenceRuleDecision || // Are we the special loop entry/exit state? + configContext.isEmpty() || // If SLL wildcard + configContext.hasEmptyPath(){ + return false + } + + // Require all return states to return back to the same rule + // that p is in. + let numCtxs: Int = configContext.size() + for i in 0 ..< numCtxs { // for each stack context + let returnState: ATNState = atn.states[configContext.getReturnState(i)]! + if returnState.ruleIndex != p.ruleIndex + {return false} + } + + let decisionStartState: BlockStartState = (p.transition(0).target as! BlockStartState) + let blockEndStateNum: Int = decisionStartState.endState!.stateNumber + let blockEndState: BlockEndState = (atn.states[blockEndStateNum] as! BlockEndState) + + // Verify that the top of each stack context leads to loop entry/exit + // state through epsilon edges and w/o leaving rule. + for i in 0 ..< numCtxs { // for each stack context + let returnStateNumber: Int = configContext.getReturnState(i) + let returnState: ATNState = atn.states[returnStateNumber]! + // all states must have single outgoing epsilon edge + if returnState.getNumberOfTransitions() != 1 || !returnState.transition(0).isEpsilon(){ + return false + } + // Look for prefix op case like 'not expr', (' type ')' expr + let returnStateTarget: ATNState = returnState.transition(0).target + if returnState.getStateType() == ATNState.BLOCK_END && + returnStateTarget == p { + continue + } + // Look for 'expr op expr' or case where expr's return state is block end + // of (...)* internal block; the block end points to loop back + // which points to p but we don't need to check that + if returnState == blockEndState{ + continue + } + // Look for ternary expr ? expr : expr. The return state points at block end, + // which points at loop entry state + if returnStateTarget == blockEndState{ + continue + } + // Look for complex prefix 'between expr and expr' case where 2nd expr's + // return state points at block end state of (...)* internal block + if returnStateTarget.getStateType() == ATNState.BLOCK_END && + returnStateTarget.getNumberOfTransitions() == 1 && + returnStateTarget.transition(0).isEpsilon() && + returnStateTarget.transition(0).target == p{ + continue + } + + // anything else ain't conforming + return false + } + + return true + } + open func getRuleName(_ index: Int) -> String { if index >= 0 { return parser.getRuleNames()[index] diff --git a/tool/pom.xml b/tool/pom.xml index 328cab73b..40e71bb7f 100644 --- a/tool/pom.xml +++ b/tool/pom.xml @@ -16,33 +16,33 @@ http://www.antlr.org The ANTLR 4 grammar compiler. - - - org.antlr - antlr4-runtime - ${project.version} - - - org.antlr - antlr-runtime - 3.5.2 - - - org.antlr - ST4 - 4.0.8 - - - org.abego.treelayout - org.abego.treelayout.core - 1.0.3 - - - javax.json - javax.json-api - 1.0 - - + + + org.antlr + antlr4-runtime + ${project.version} + + + org.antlr + antlr-runtime + 3.5.2 + + + org.antlr + ST4 + 4.0.8 + + + org.abego.treelayout + org.abego.treelayout.core + 1.0.3 + + + org.glassfish + javax.json + 1.0.4 + + src diff --git a/tool/resources/org/antlr/v4/tool/templates/codegen/JavaScript/JavaScript.stg b/tool/resources/org/antlr/v4/tool/templates/codegen/JavaScript/JavaScript.stg index 501e117a6..2e10728f4 100644 --- a/tool/resources/org/antlr/v4/tool/templates/codegen/JavaScript/JavaScript.stg +++ b/tool/resources/org/antlr/v4/tool/templates/codegen/JavaScript/JavaScript.stg @@ -236,7 +236,7 @@ case : * overriding implementation impossible to maintain. */ RuleSempredFunction(r, actions) ::= << -.prototype._sempred = function(localctx, predIndex) { +.prototype._sempred = function(localctx, predIndex) { switch(predIndex) { : return ;}; separator="\n">