diff --git a/doc/adding-tests.md b/doc/antlr-project-testing.md similarity index 61% rename from doc/adding-tests.md rename to doc/antlr-project-testing.md index de4787041..7303a8bd6 100644 --- a/doc/adding-tests.md +++ b/doc/antlr-project-testing.md @@ -1,4 +1,4 @@ -# Adding unit tests +# ANTLR project unit tests ## Introduction @@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow The grammars are strings representing StringTemplates (`ST` objects) so `` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the [Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). +## Requirements + +In order to perform the tests on all target languages, you need to have the following languages installed: + +* `mono` (e.g., `brew install mono`) Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below +* `nodejs` +* Python 2.7 +* Python 3.5 +* Go +* Swift 3 (via XCode 8.x) tested currently only osx +* clang (for C++ target) +* +To **install into local repository** `~/.m2/repository/org/antlr`, do this: + +```bash +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux +$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine +``` + +Now, make sure C# runtime is built and installed locally. + +```bash +cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp +# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it +rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll +rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll +# build +xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj +``` + +C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib. + + ## Running the runtime tests A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of @@ -49,26 +82,128 @@ And the result of testing the entire subdirectory: From `mvn`, on the commandline, you will see: ```bash +$ cd antlr4 +$ mvn test +... ------------------------------------------------------- T E S T S ------------------------------------------------------- -Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec -Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors -Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser -Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSets -Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec -Running org.antlr.v4.test.runtime.javascript.node.TestPerformance -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec -Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer -Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec -Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec -Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec +Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451 +Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081 +Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj +Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec +Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers +dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864 +antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4] +... +[INFO] ------------------------------------------------------------------------ +[INFO] Reactor Summary: +[INFO] +[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s] +[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s] +[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s] +[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s] +[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s] +[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s] +[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min] +[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s] +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 06:53 min +[INFO] Finished at: 2016-11-16T15:36:56-08:00 +[INFO] Final Memory: 44M/458M +[INFO] ------------------------------------------------------------------------ +``` + +Note: That is actually result of running the much faster: + +```bash +mvn -Dparallel=methods -DthreadCount=4 install +``` + +## Running test subsets + +*From the `runtime-testsuite` dir* + +### Run one test group across targets + +```bash +$ cd runtime-testsuite +$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux +$ mvn -Dtest=TestParserExec test +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.cpp.TestParserExec +... +Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec +Running org.antlr.v4.test.runtime.csharp.TestParserExec ... ``` +Or run all lexer related tests: + +``` +$ cd runtime-testsuite +$ mvn -Dtest=Test*Lexer* test +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers +... +``` + +### Run all tests for a single target + +```bash +$ cd runtime-testsuite +$ mvn -Dtest=java.* test +... +``` + +Or run all lexer related tests in Java target only: + +```bash +$ cd runtime-testsuite +$ mvn -Dtest=java.*Lexer* test +... +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Running org.antlr.v4.test.runtime.java.TestCompositeLexers +Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec +Running org.antlr.v4.test.runtime.java.TestLexerErrors +Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec +Running org.antlr.v4.test.runtime.java.TestLexerExec +Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec +Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer +Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec + +Results : + +Tests run: 59, Failures: 0, Errors: 0, Skipped: 0 +``` + +## Testing in parallel + +Use this to run tests in parallel: + +```bash +$ export MAVEN_OPTS="-Xmx1G" +$ mvn -Dparallel=methods -DthreadCount=4 test +... +------------------------------------------------------- + T E S T S +------------------------------------------------------- +Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false +... +``` + +This can be combined with other `-D` above. + ## Adding a runtime test To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of: diff --git a/doc/building-antlr.md b/doc/building-antlr.md index 3d21356ff..ac220bc30 100644 --- a/doc/building-antlr.md +++ b/doc/building-antlr.md @@ -72,64 +72,6 @@ $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests=true # make sure all artifacts are visible on this machine ``` -# Testing tool and targets - -In order to perform the tests on all target languages, you need to have the following languages installed: - -* `mono` (e.g., `brew install mono`) -* `nodejs` -* Python 2.7 -* Python 3.5 -* Go -* Swift 3 (via XCode 8.x) tested currently only osx -* clang (for C++ target) - -To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this: - -```bash -$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux -$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine -$ mvn install # now "do it with feeling" -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451 -Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081 -Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec -Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers -dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864 -antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4] -... -[INFO] ------------------------------------------------------------------------ -[INFO] Reactor Summary: -[INFO] -[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s] -[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s] -[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s] -[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s] -[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s] -[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s] -[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min] -[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s] -[INFO] ------------------------------------------------------------------------ -[INFO] BUILD SUCCESS -[INFO] ------------------------------------------------------------------------ -[INFO] Total time: 06:53 min -[INFO] Finished at: 2016-11-16T15:36:56-08:00 -[INFO] Final Memory: 44M/458M -[INFO] ------------------------------------------------------------------------ -``` - -Note: That is actually result of running the much faster: - -```bash -mvn -Dparallel=methods -DthreadCount=4 install -``` - You should see these jars (when building 4.6-SNAPSHOT): ```bash @@ -147,87 +89,12 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. -## Running test subsets +# Testing tool and targets -*From the `runtime-testsuite` dir* +See [ANTLR project unit tests](antlr-project-testing.md). -### Run one test group across targets -```bash -$ cd runtime-testsuite -$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux -$ mvn -Dtest=TestParserExec test -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.cpp.TestParserExec -... -Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec -Running org.antlr.v4.test.runtime.csharp.TestParserExec -... -``` - -Or run all lexer related tests: - -``` -$ cd runtime-testsuite -$ mvn -Dtest=Test*Lexer* test -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers -... -``` - -### Run all tests for a single target - -```bash -$ cd runtime-testsuite -$ mvn -Dtest=java.* test -... -``` - -Or run all lexer related tests in Java target only: - -```bash -$ cd runtime-testsuite -$ mvn -Dtest=java.*Lexer* test -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Running org.antlr.v4.test.runtime.java.TestCompositeLexers -Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec -Running org.antlr.v4.test.runtime.java.TestLexerErrors -Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec -Running org.antlr.v4.test.runtime.java.TestLexerExec -Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec -Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer -Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec - -Results : - -Tests run: 59, Failures: 0, Errors: 0, Skipped: 0 -``` - -## Testing in parallel - -Use this to run tests in parallel: - -```bash -$ export MAVEN_OPTS="-Xmx1G" -$ mvn -Dparallel=methods -DthreadCount=4 test -... -------------------------------------------------------- - T E S T S -------------------------------------------------------- -Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false -... -``` - -This can be combined with other `-D` above. - -## Building without testing +# Building without testing To build without running the tests (saves a lot of time), do this: diff --git a/doc/index.md b/doc/index.md index fce24d1b2..d5c55518e 100644 --- a/doc/index.md +++ b/doc/index.md @@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema * [Cutting an ANTLR Release](releasing-antlr.md) -* [Adding ANTLR unit tests](adding-tests.md) +* [ANTLR project unit tests](antlr-project-testing.md) * [Creating an ANTLR Language Target](creating-a-language-target.md) diff --git a/doc/runtimetests-overview.md b/doc/runtimetests-overview.md index d7247a20c..13085869e 100644 --- a/doc/runtimetests-overview.md +++ b/doc/runtimetests-overview.md @@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate ## Generating JUnit Tests -The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run +The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run ```bash $ mvn -Pgen generate-test-sources