forgetest1
/
antlr
forked from jasder/antlr
1
0
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

update doc; move testing info to single file. rename doc file.

This commit is contained in:
parrt 2016-12-14 11:51:28 -08:00
parent 3f00594313
commit 7430b74d98
4 changed files with 155 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

165
doc/adding-tests.md → doc/antlr-project-testing.md
View File

@ -1,4 +1,4 @@
# Adding unit tests # ANTLR project unit tests
## Introduction ## Introduction
@ -35,6 +35,39 @@ The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allow
The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
[Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). [Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
## Requirements
In order to perform the tests on all target languages, you need to have the following languages installed:
* `mono` (e.g., `brew install mono`) Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
* `nodejs`
* Python 2.7
* Python 3.5
* Go
* Swift 3 (via XCode 8.x) tested currently only osx
* clang (for C++ target)
*
To **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
```
Now, make sure C# runtime is built and installed locally.
```bash
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
rm Antlr4.Runtime/bin/net20/Release/Antlr4.Runtime.dll
rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
# build
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
```
C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.
## Running the runtime tests ## Running the runtime tests
A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
@ -49,26 +82,128 @@ And the result of testing the entire subdirectory:
From `mvn`, on the commandline, you will see: From `mvn`, on the commandline, you will see:
```bash ```bash
$ cd antlr4
$ mvn test
...
------------------------------------------------------- -------------------------------------------------------
T E S T S T E S T S
------------------------------------------------------- -------------------------------------------------------
Running org.antlr.v4.test.runtime.javascript.node.TestCompositeLexers Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.581 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
Running org.antlr.v4.test.runtime.javascript.node.TestLexerErrors Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.721 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalParser Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.084 sec Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
Running org.antlr.v4.test.runtime.javascript.node.TestSets Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.798 sec dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
Running org.antlr.v4.test.runtime.javascript.node.TestPerformance antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 sec ...
Running org.antlr.v4.test.runtime.javascript.node.TestSemPredEvalLexer [INFO] ------------------------------------------------------------------------
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.994 sec [INFO] Reactor Summary:
Running org.antlr.v4.test.runtime.javascript.node.TestLexerExec [INFO]
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.433 sec [INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:53 min
[INFO] Finished at: 2016-11-16T15:36:56-08:00
[INFO] Final Memory: 44M/458M
[INFO] ------------------------------------------------------------------------
```
Note: That is actually result of running the much faster:
```bash
mvn -Dparallel=methods -DthreadCount=4 install
```
## Running test subsets
*From the `runtime-testsuite` dir*
### Run one test group across targets
```bash
$ cd runtime-testsuite
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn -Dtest=TestParserExec test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestParserExec
...
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
Running org.antlr.v4.test.runtime.csharp.TestParserExec
... ...
``` ```
Or run all lexer related tests:
```
$ cd runtime-testsuite
$ mvn -Dtest=Test*Lexer* test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
...
```
### Run all tests for a single target
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.* test
...
```
Or run all lexer related tests in Java target only:
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.*Lexer* test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
Running org.antlr.v4.test.runtime.java.TestLexerErrors
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
Running org.antlr.v4.test.runtime.java.TestLexerExec
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
Results :
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
```
## Testing in parallel
Use this to run tests in parallel:
```bash
$ export MAVEN_OPTS="-Xmx1G"
$ mvn -Dparallel=methods -DthreadCount=4 test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
...
```
This can be combined with other `-D` above.
## Adding a runtime test ## Adding a runtime test
To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of: To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:

139
doc/building-antlr.md
View File

@ -72,64 +72,6 @@ $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine $ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
``` ```
# Testing tool and targets
In order to perform the tests on all target languages, you need to have the following languages installed:
* `mono` (e.g., `brew install mono`)
* `nodejs`
* Python 2.7
* Python 3.5
* Go
* Swift 3 (via XCode 8.x) tested currently only osx
* clang (for C++ target)
To run the tests and **install into local repository** `~/.m2/repository/org/antlr`, do this:
```bash
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
$ mvn install # now "do it with feeling"
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
[INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
[INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
[INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
[INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
[INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
[INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
[INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:53 min
[INFO] Finished at: 2016-11-16T15:36:56-08:00
[INFO] Final Memory: 44M/458M
[INFO] ------------------------------------------------------------------------
```
Note: That is actually result of running the much faster:
```bash
mvn -Dparallel=methods -DthreadCount=4 install
```
You should see these jars (when building 4.6-SNAPSHOT): You should see these jars (when building 4.6-SNAPSHOT):
```bash ```bash
@ -147,87 +89,12 @@ antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar
Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes.
## Running test subsets # Testing tool and targets
*From the `runtime-testsuite` dir* See [ANTLR project unit tests](antlr-project-testing.md).
### Run one test group across targets
```bash # Building without testing
$ cd runtime-testsuite
$ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
$ mvn -Dtest=TestParserExec test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestParserExec
...
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
Running org.antlr.v4.test.runtime.csharp.TestParserExec
...
```
Or run all lexer related tests:
```
$ cd runtime-testsuite
$ mvn -Dtest=Test*Lexer* test
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
...
```
### Run all tests for a single target
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.* test
...
```
Or run all lexer related tests in Java target only:
```bash
$ cd runtime-testsuite
$ mvn -Dtest=java.*Lexer* test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.java.TestCompositeLexers
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
Running org.antlr.v4.test.runtime.java.TestLexerErrors
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
Running org.antlr.v4.test.runtime.java.TestLexerExec
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
Results :
Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
```
## Testing in parallel
Use this to run tests in parallel:
```bash
$ export MAVEN_OPTS="-Xmx1G"
$ mvn -Dparallel=methods -DthreadCount=4 test
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
...
```
This can be combined with other `-D` above.
## Building without testing
To build without running the tests (saves a lot of time), do this: To build without running the tests (saves a lot of time), do this:

2
doc/index.md
View File

@ -63,6 +63,6 @@ This documentation is a reference and summarizes grammar syntax and the key sema
* [Cutting an ANTLR Release](releasing-antlr.md) * [Cutting an ANTLR Release](releasing-antlr.md)
* [Adding ANTLR unit tests](adding-tests.md) * [ANTLR project unit tests](antlr-project-testing.md)
* [Creating an ANTLR Language Target](creating-a-language-target.md) * [Creating an ANTLR Language Target](creating-a-language-target.md)

2
doc/runtimetests-overview.md
View File

@ -29,7 +29,7 @@ The underlying process of running the tests is quite a complicated setup to cate
## Generating JUnit Tests ## Generating JUnit Tests
The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [adding-tests.md](adding-tests.md) file. As lined out there you have to run The test specification part makes heavy use of the StringTemplate engine to allow defining target language agnostic tests. For that all tests are described in template (`stg`) files. You can find them in the [templates](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates) subfolder of the runtime tests folder. Read more about the folder structure in the [antlr-project-testing.md](antlr-project-testing.md) file. As lined out there you have to run
```bash ```bash
$ mvn -Pgen generate-test-sources $ mvn -Pgen generate-test-sources