out of date. use maven
See also Deploying ANTLR mvn artifacts.
Most programmers do not need the information on this page because they will simply download the appropriate jar(s) or use ANTLR through maven (via ANTLR's antlr4-maven-plugin). If you would like to fork the project and fix bugs or tweak the runtime code generation, then you will almost certainly need to build ANTLR itself. There are two components:
- the tool that compiles grammars down into parsers and lexers in one of the target languages
- the runtime used by those generated parsers and lexers.
As of 4.4, we use a Python script in the main directory called bild.py, which will bootstrap by pulling in the bilder.py library. You don't need to know much about what's inside, but the code is reasonably easy to read.
I will assume that the root directory is /tmp for the purposes of explaining how to build ANTLR in this document.
Get the source
The first step is to get the Java source code from the ANTLR 4 repository at github. You can download the repository from github, but the easiest thing to do is simply clone the repository on your local disk:
$ cd /tmp
/tmp $ git clone git@github.com:antlr/antlr4.git
Cloning into 'antlr4'...
remote: Counting objects: 43273, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 43273 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (43273/43273), 18.76 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (22419/22419), done.
Checking connectivity... done.
Really though you'll need all the targets to build jars etc...
cd /tmp
git clone git@github.com:antlr/antlr4.git
git clone git@github.com:antlr/antlr4-csharp.git
git clone git@github.com:antlr/antlr4-javascript.git
git clone git@github.com:antlr/antlr4-python2.git
git clone git@github.com:antlr/antlr4-python3.git
Freshen repos
cd /tmp
cd antlr4; git pull origin master; cd ..
cd antlr4-csharp; git pull origin master; cd ..
cd antlr4-javascript; git pull origin master; cd ..
cd antlr4-python2; git pull origin master; cd ..
cd antlr4-python3; git pull origin master; cd ..
also useful:
cd /tmp
cd antlr4; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-csharp; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-javascript; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-python2; git commit -a -m 'msg'; git push origin master; cd ..
cd antlr4-python3; git commit -a -m 'msg'; git push origin master; cd ..
Compiling
First, make sure you have java installed. Type java on command line and it should launch. If not, install Java.
Now, let's make sure everything is clean just out of habit:
/tmp $ cd antlr4
/tmp/antlr4 $ ./bild.py clean
bootstrapping; downloading bilder.py
target clean
bild succeeded
Compiling the source code is easy (but note that compile will fail until you download the target repos—see below):
/tmp/antlr4 $ ./bild.py compile
target compile
require parsers
build compile
bild succeeded
Take a look in the bild.log file that is always produced if you'd like to see precisely what commands were executed and their output:
/tmp/antlr4 $ head bild.log
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:107] platform=darwin
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:108] jdk={'1.6': '/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home', '1.7': '/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home', '1.8': '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home'}
[01/13/15 18:56:16] require parsers
[01/13/15 18:56:16 require ./bild.py:74 bilder.py:370] require parsers
[01/13/15 18:56:16 antlr3 ./bild.py:67 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/parse/GrammarTreeVisitor.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/BlockSetTransformer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/LeftRecursiveRuleWalker.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRLexer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRParser.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ATNBuilder.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ActionSplitter.g
[01/13/15 18:56:19 antlr3 ./bild.py:69 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/codegen -lib /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/codegen/SourceGenTriggers.g
[01/13/15 18:56:20 antlr4 ./bild.py:71 bilder.py:423] java -cp /Users/parrt/.bild/jars/antlr-4.4-complete.jar org.antlr.v4.Tool -o /private/tmp/antlr4/gen4/org/antlr/v4/runtime/tree/xpath -package org.antlr.v4.runtime.tree.xpath /private/tmp/antlr4/runtime/Java/src/org/antlr/v4/runtime/tree/xpath/XPathLexer.g4
[01/13/15 18:56:20] build compile
[01/13/15 18:56:20 require ./bild.py:74 bilder.py:378] build compile
[01/13/15 18:56:20 javac ./bild.py:81 bilder.py:452] javac -version
...
BTW, if you get an error about Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar see stackoverflow solution. Just:
$ unset JAVA_TOOL_OPTIONS
You will also notice that the build script has pulled in the necessary jars, sort of like maven would:
$ ls ~/.bild/jars/
ST-4.0.8.jar antlr-4.5-SNAPSHOT-complete.jar bild-junit.jar
antlr-3.5.1-complete.jar antlr-4.5-complete.jar hamcrest-core-1.3.jar
antlr-3.5.1-runtime.jar antlr-4.5-rc-1-complete.jar junit-4.11.jar
antlr-4.4-complete.jar antlr4-4.5-complete.jar
Making tool and runtime jars
Because the build script expects all targets (the usual case), you must pull all of those target repositories in as well. We have kept them as separate repositories for organizational purposes and not because they are separate projects.
Here are the github repositories for the various targets (Java is contained within the main antlr4 tool):
You can run the following commands:
cd /tmp
git clone https://github.com/antlr/antlr4.git
git clone git@github.com:antlr/antlr4-csharp.git
git clone git@github.com:antlr/antlr4-javascript.git
git clone git@github.com:antlr/antlr4-python2.git
git clone git@github.com:antlr/antlr4-python3.git
Then go back into the antlr4 where the build script is and try to make a jar to see if everything is working.
/tmp $ cd antlr4
/tmp/antlr4 $ grep VERSION bild.py | head -2
BOOTSTRAP_VERSION = "4.4"
VERSION = "4.5-SNAPSHOT"
/tmp/antlr4 $ ./bild.py mkjar
target mkjar
require compile
require parsers
build compile
build mkjar_complete
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
require compile
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
Generated dist/antlr4-4.5-SNAPSHOT.jar
bild succeeded
/tmp/antlr4 $ ls dist
antlr4-4.5-SNAPSHOT-complete.jar antlr4-4.5-SNAPSHOT.jar
You'll see that it builds the complete jar twice. ANTLR 4.5 has an ANTLR 4 grammar in it which must be built with the previous version because we cannot assume that the current version is available. Once we have built a valid ANTLR 4.5 using, say, 4.4, then we rebuild it using itself.
Testing tool and targets
In order to perform the tests, make sure that you have all of the targets per the previous section. Also you will need mono and `nodejs; on OS X:
$ brew install mono
$ brew install node
Then, launch the tests of all targets with:
./bild.py tests
This should peg all of your CPUs as it tries to launch the tests in parallel. Note that we have seen some tests fail when the C# target tests are run in parallel. They work fine in serial always (e.g., from your development environment).
You should see output like
...
Testing Python3 ...
org.antlr.v4.test.rt.py3.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.py3.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.py3.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLeftRecursion: 95 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerExec: 38 tests, 0 failures
Python3 tests complete
Testing JavaScript ...
org.antlr.v4.test.rt.js.node.TestBitSetWordSize: 1 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerExec: 38 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLeftRecursion: 95 tests, 0 failures
JavaScript tests complete
bild succeeded
Building ANTLR in Intellij
I build ANTLR during development in Intellij directly. The necessary project files are included in the repository in directory .idea and files:
$ cd /tmp/antlr4
$ find . -name '*.iml'
./antlr4.iml
./runtime/Java/runtime.iml
./tool/tool.iml
Making ANTLR runtime jar OSGi ready
Without being OSGi compatible, the ANTLR runtime cannot be used within eclipse. That's annoying. To fix this, we have a pull request that explains what the issue is with OSGi and describes how all we need to do is change the manifest file in the jar.
First, download bnd tool and make an alias for convenience:
alias bnd='java -jar /usr/local/lib/biz.aQute.bnd-3.0.0.jar'
Our regular mkjar bild target generates the following manifest:
Manifest-Version: 1.0
Implementation-Vendor: ANTLR
Implementation-Title: ANTLR 4 Runtime
Implementation-Version: 4.5
Implementation-Vendor-Id: org.antlr
Build-Jdk: 1.6
Built-By: parrt
Created-By: http://www.bildtool.org
Interesting. This prints exactly the existing jar manifest looks like:
$ bnd antlr4-4.5.jar
[MANIFEST antlr4-4.5]
Build-Jdk 1.6
Built-By parrt
Created-By http://www.bildtool.org
Implementation-Title ANTLR 4 Runtime
Implementation-Vendor ANTLR
Implementation-Vendor-Id org.antlr
Implementation-Version 4.5
Manifest-Version 1.0
Well that was easy, I ran the following to get the proper jar.
cd dist
bnd wrap --output antlr4-4.5-osgi.jar antlr4-4.5.jar
The manifest file it generates is:
Manifest-Version: 1.0
Bnd-LastModified: 1421371533578
Build-Jdk: 1.6
Built-By: parrt
Bundle-ManifestVersion: 2
Bundle-Name: dist
Bundle-SymbolicName: dist
Bundle-Version: 0
Created-By: 1.7.0_25 (Oracle Corporation)
Export-Package: org.abego.treelayout,org.abego.treelayout.internal.util,
org.abego.treelayout.internal.util.java.lang,org.abego.treelayout.inter
nal.util.java.lang.string,org.abego.treelayout.internal.util.java.util,
org.abego.treelayout.util;uses:="org.abego.treelayout",org.antlr.v4.run
time;uses:="javax.print,javax.swing,org.antlr.v4.runtime.atn,org.antlr.
v4.runtime.dfa,org.antlr.v4.runtime.misc,org.antlr.v4.runtime.tree,org.
antlr.v4.runtime.tree.pattern",org.antlr.v4.runtime.atn;uses:="org.antl
r.v4.runtime,org.antlr.v4.runtime.dfa,org.antlr.v4.runtime.misc",org.an
tlr.v4.runtime.dfa;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.atn
",org.antlr.v4.runtime.misc;uses:="javax.annotation.processing,javax.la
ng.model,javax.lang.model.element,javax.print,javax.swing,org.antlr.v4.
runtime",org.antlr.v4.runtime.tree;uses:="org.antlr.v4.runtime,org.antl
r.v4.runtime.misc",org.antlr.v4.runtime.tree.gui;uses:="javax.print,jav
ax.swing,org.abego.treelayout,org.antlr.v4.runtime.tree",org.antlr.v4.r
untime.tree.pattern;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.mi
sc,org.antlr.v4.runtime.tree",org.antlr.v4.runtime.tree.xpath;uses:="or
g.antlr.v4.runtime,org.antlr.v4.runtime.atn,org.antlr.v4.runtime.dfa,or
g.antlr.v4.runtime.tree"
Implementation-Title: ANTLR 4 Runtime
Implementation-Vendor: ANTLR
Implementation-Vendor-Id: org.antlr
Implementation-Version: 4.5
Import-Package: javax.annotation.processing;resolution:=optional,javax.i
mageio;resolution:=optional,javax.lang.model;resolution:=optional,javax
.lang.model.element;resolution:=optional,javax.lang.model.type;resoluti
on:=optional,javax.lang.model.util;resolution:=optional,javax.print;res
olution:=optional,javax.print.attribute;resolution:=optional,javax.swin
g;resolution:=optional,javax.swing.border;resolution:=optional,javax.sw
ing.event;resolution:=optional,javax.swing.filechooser;resolution:=opti
onal,javax.swing.tree;resolution:=optional,javax.tools;resolution:=opti
onal
Originally-Created-By: http://www.bildtool.org
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.6))"
Tool: Bnd-3.0.0.201501151451
Use the following to test the updated jar:
bnd print --verify antlr4-4.5-osgi.jar
Ok, added to mkjar target so happens automatically now. It downloads 2.4 bnd (not 3.0) and wraps.
Editing the ANTLR github wiki
If you are a kind soul and would like to fix some of the documentation associated with building and managing handler, such as this page, you can fork antlr/antlr4 at github and then clone from YOUR_ID:
$ cd /tmp
$ git clone git@github.com:YOUR_ID/antlr4.wiki.git