forked from jasder/antlr
update doc; move testing info to single file. rename doc file.
This commit is contained in:
parrt
parent
3f00594313
commit
7430b74d98
|
|
@ -1,4 +1,4 @@
|
|||
# Adding unit tests
|
||||
# ANTLR project unit tests
|
||||
|
||||
## Introduction
|
||||
|
||||
|
|
@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow
|
|||
The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
|
||||
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
|
||||
|
||||
## Requirements
|
||||
|
||||
In order to perform the tests on all target languages, you need to have the following languages installed:
|
||||
|
||||
* `mono` (e.g., `brew install mono`) Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
|
||||
* `nodejs`
|
||||
* Python 2.7
|
||||
* Python 3.5
|
||||
* Go
|
||||
* Swift 3 (via XCode 8.x) tested currently only osx
|
||||
* clang (for C++ target)
|
||||
*
|
||||
To **install into local repository** `~/.m2/repository/org/antlr`, do this:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
```
|
||||
|
||||
Now, make sure C# runtime is built and installed locally.
|
||||
|
||||
```bash
|
||||
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
|
||||
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
|
||||
rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll
|
||||
rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
|
||||
# build
|
||||
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
|
||||
```
|
||||
|
||||
C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.
|
||||
|
||||
|
||||
## Running the runtime tests
|
||||
|
||||
A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
|
||||
|
|
@ -49,26 +82,128 @@ And the result of testing the entire subdirectory:
|
|||
From `mvn`, on the commandline, you will see:
|
||||
|
||||
```bash
|
||||
$ cd antlr4
|
||||
$ mvn test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser
|
||||
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSets
|
||||
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestPerformance
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec
|
||||
Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
|
||||
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
|
||||
...
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Reactor Summary:
|
||||
[INFO]
|
||||
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
|
||||
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
|
||||
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
|
||||
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
|
||||
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
|
||||
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
|
||||
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
|
||||
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Total time: 06:53 min
|
||||
[INFO] Finished at: 2016-11-16T15:36:56-08:00
|
||||
[INFO] Final Memory: 44M/458M
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
```
|
||||
|
||||
Note: That is actually result of running the much faster:
|
||||
|
||||
```bash
|
||||
mvn -Dparallel=methods -DthreadCount=4 install
|
||||
```
|
||||
|
||||
## Running test subsets
|
||||
|
||||
*From the `runtime-testsuite` dir*
|
||||
|
||||
### Run one test group across targets
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn -Dtest=TestParserExec test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestParserExec
|
||||
...
|
||||
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestParserExec
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests:
|
||||
|
||||
```
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=Test*Lexer* test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
|
||||
...
|
||||
```
|
||||
|
||||
### Run all tests for a single target
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.* test
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests in Java target only:
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.*Lexer* test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
|
||||
|
||||
Results :
|
||||
|
||||
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
|
||||
```
|
||||
|
||||
## Testing in parallel
|
||||
|
||||
Use this to run tests in parallel:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G"
|
||||
$ mvn -Dparallel=methods -DthreadCount=4 test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
|
||||
...
|
||||
```
|
||||
|
||||
This can be combined with other `-D` above.
|
||||
|
||||
## Adding a runtime test
|
||||
|
||||
To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
|
||||
|
|
@ -72,64 +72,6 @@ $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
|||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
```
|
||||
|
||||
# Testing tool and targets
|
||||
|
||||
In order to perform the tests on all target languages, you need to have the following languages installed:
|
||||
|
||||
* `mono` (e.g., `brew install mono`)
|
||||
* `nodejs`
|
||||
* Python 2.7
|
||||
* Python 3.5
|
||||
* Go
|
||||
* Swift 3 (via XCode 8.x) tested currently only osx
|
||||
* clang (for C++ target)
|
||||
|
||||
To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
|
||||
$ mvn install # now "do it with feeling"
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
|
||||
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
|
||||
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
|
||||
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
|
||||
...
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Reactor Summary:
|
||||
[INFO]
|
||||
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
|
||||
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
|
||||
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
|
||||
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
|
||||
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
|
||||
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
|
||||
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
|
||||
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] BUILD SUCCESS
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
[INFO] Total time: 06:53 min
|
||||
[INFO] Finished at: 2016-11-16T15:36:56-08:00
|
||||
[INFO] Final Memory: 44M/458M
|
||||
[INFO] ------------------------------------------------------------------------
|
||||
```
|
||||
|
||||
Note: That is actually result of running the much faster:
|
||||
|
||||
```bash
|
||||
mvn -Dparallel=methods -DthreadCount=4 install
|
||||
```
|
||||
|
||||
You should see these jars (when building 4.6-SNAPSHOT):
|
||||
|
||||
```bash
|
||||
|
|
@ -147,87 +89,12 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar
|
|||
|
||||
Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes.
|
||||
|
||||
## Running test subsets
|
||||
# Testing tool and targets
|
||||
|
||||
*From the `runtime-testsuite` dir*
|
||||
See [ANTLR project unit tests](antlr-project-testing.md).
|
||||
|
||||
### Run one test group across targets
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
|
||||
$ mvn -Dtest=TestParserExec test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestParserExec
|
||||
...
|
||||
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
|
||||
Running org.antlr.v4.test.runtime.csharp.TestParserExec
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests:
|
||||
|
||||
```
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=Test*Lexer* test
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
|
||||
...
|
||||
```
|
||||
|
||||
### Run all tests for a single target
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.* test
|
||||
...
|
||||
```
|
||||
|
||||
Or run all lexer related tests in Java target only:
|
||||
|
||||
```bash
|
||||
$ cd runtime-testsuite
|
||||
$ mvn -Dtest=java.*Lexer* test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
|
||||
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerErrors
|
||||
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestLexerExec
|
||||
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
|
||||
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
|
||||
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
|
||||
|
||||
Results :
|
||||
|
||||
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
|
||||
```
|
||||
|
||||
## Testing in parallel
|
||||
|
||||
Use this to run tests in parallel:
|
||||
|
||||
```bash
|
||||
$ export MAVEN_OPTS="-Xmx1G"
|
||||
$ mvn -Dparallel=methods -DthreadCount=4 test
|
||||
...
|
||||
-------------------------------------------------------
|
||||
T E S T S
|
||||
-------------------------------------------------------
|
||||
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
|
||||
...
|
||||
```
|
||||
|
||||
This can be combined with other `-D` above.
|
||||
|
||||
## Building without testing
|
||||
# Building without testing
|
||||
|
||||
To build without running the tests (saves a lot of time), do this:
|
||||
|
||||
|
|
|
|||
|
|
@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema
|
|||
|
||||
* [Cutting an ANTLR Release](releasing-antlr.md)
|
||||
|
||||
* [Adding ANTLR unit tests](adding-tests.md)
|
||||
* [ANTLR project unit tests](antlr-project-testing.md)
|
||||
|
||||
* [Creating an ANTLR Language Target](creating-a-language-target.md)
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate
|
|||
|
||||
## Generating JUnit Tests
|
||||
|
||||
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run
|
||||
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run
|
||||
|
||||
```bash
|
||||
$ mvn -Pgen generate-test-sources
|
||||
|
|
|
|||
Loading…
Reference in New Issue