Merge branch 'master_upstream'
This commit is contained in:
Mike Lischke
commit
a9dc3ebd86
|
|
@ -8,7 +8,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-maven-plugin</artifactId>
|
||||
<packaging>maven-plugin</packaging>
|
||||
|
|
|
|||
|
|
@ -174,13 +174,20 @@ At this point, you should have an editor which displays an error icon next to th
|
|||
|
||||
What remains to be done is have our validate function actually validate the input. Finally ANTLR comes in the picture!
|
||||
|
||||
To start with, let's load ANTLR and your parser, listener etc.. Easy, since you could write:
|
||||
To start with, let's load ANTLR and your parser, listener etc..
|
||||
|
||||
The preferred approach for loading parser code is to bundle your parser, [as described here](javascript-target.md).
|
||||
You can then load it as part of the importScripts instruction at the start of your worker code.
|
||||
|
||||
Another approach is to load it using 'require'. Easy, since you could write:
|
||||
|
||||
```js
|
||||
var antlr4 = require('antlr4/index');
|
||||
```
|
||||
|
||||
This may work, but it's actually unreliable. The reason is that the require function used by ANTLR, which exactly mimics the NodeJS require function, uses a different syntax than the require function that comes with ACE. So we need to bring in a require function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find here. But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows:
|
||||
This may work, but it's actually unreliable. The reason is that the 'require' function that comes with ACE uses a different syntax than the 'require' function used by ANTLR, which follows the NodeJS 'require' convention.
|
||||
So we need to bring in a NodeJS compatible 'require' function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find in li/require.js.
|
||||
But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows:
|
||||
|
||||
```js
|
||||
// load nodejs compatible require
|
||||
|
|
@ -191,7 +198,8 @@ importScripts("../lib/require.js");
|
|||
var antlr4_require = require;
|
||||
require = ace_require;
|
||||
```
|
||||
Now it's safe to load antlr, and the parsers generated for your language. Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows:
|
||||
Now it's safe to load antlr and the parsers generated for your language.
|
||||
Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows:
|
||||
```js
|
||||
// load antlr4 and myLanguage
|
||||
var antlr4, mylanguage;
|
||||
|
|
@ -204,6 +212,7 @@ try {
|
|||
}
|
||||
```
|
||||
Please note the try-finally construct. ANTLR uses 'require' synchronously so it's perfectly safe to ignore the ACE 'require' while running ANTLR code. ACE itself does not guarantee synchronous execution, so you are much safer always switching 'require' back to 'ace_require'.
|
||||
|
||||
Now detecting deep syntax errors in your code is a task for your ANTLR listener or visitor or whatever piece of code you've delegated this to. We're not going to describe this here, since it would require some knowledge of your language. However, detecting grammar syntax errors is something ANTLR does beautifully (isn't that why you went for ANTLR in the first place?). So what we will illustrate here is how to report grammar syntax errors. I have no doubt that from there, you will be able to extend the validator to suit your specific needs.
|
||||
Whenever ANTLR encounters an unexpected token, it fires an error. By default, the error is routed to an error listener which simply writes to the console.
|
||||
What we need to do is replace this listener by our own listener, se we can route errors to the ACE editor. First, let's create such a listener:
|
||||
|
|
@ -243,5 +252,4 @@ var validate = function(input) {
|
|||
};
|
||||
```
|
||||
You know what? That's it! You now have an ACE editor that does syntax validation using ANTLR! I hope you find this useful, and simple enough to get started.
|
||||
What I did not address here is packaging, not something I'm an expert at. The good news is that it makes development simple, since I don't have to run any compilation process. I just edit my code, reload my editor page, and check how it goes.
|
||||
Now wait, hey! How do you debug this? Well, as usual, using Chrome, since neither Firefox or Safari are able to debug worker code. What a shame...
|
||||
WNow wait, hey! How do you debug this? Well, as usual, using Chrome, since no other browser is able to debug worker code. What a shame...
|
||||
|
|
|
|||
|
|
@ -1,8 +1,6 @@
|
|||
# C♯
|
||||
|
||||
See also [Sam Harwell's Alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs)
|
||||
|
||||
### Which frameworks are supported?
|
||||
## Which frameworks are supported?
|
||||
|
||||
The C# runtime is CLS compliant, and only requires a corresponding 3.5 .Net framework.
|
||||
|
||||
|
|
@ -13,19 +11,15 @@ In practice, the runtime has been extensively tested against:
|
|||
|
||||
No issue was found, so you should find that the runtime works pretty much against any recent .Net framework.
|
||||
|
||||
### How do I get started?
|
||||
## How do I get started?
|
||||
|
||||
You will find full instructions on the Git web page for ANTLR C# runtime.
|
||||
You will find full instructions on the [Git repo page for ANTLR C# runtime](https://github.com/antlr/antlr4/tree/master/runtime/CSharp).
|
||||
|
||||
### How do I use the runtime from my project?
|
||||
## How do I use the runtime from my project?
|
||||
|
||||
(i.e., How do I run the generated lexer and/or parser?)
|
||||
|
||||
Let's suppose that your grammar is named, as above, "MyGrammar".
|
||||
|
||||
Let's suppose this parser comprises a rule named "StartRule"
|
||||
|
||||
The tool will have generated for you the following files:
|
||||
Let's suppose that your grammar is named `MyGrammar`. The tool will generate for you the following files:
|
||||
|
||||
* MyGrammarLexer.cs
|
||||
* MyGrammarParser.cs
|
||||
|
|
@ -34,7 +28,7 @@ The tool will have generated for you the following files:
|
|||
* MyGrammarVisitor.js (if you have activated the -visitor option)
|
||||
* MyGrammarBaseVisitor.js (if you have activated the -visitor option)
|
||||
|
||||
Now a fully functioning code might look like the following:
|
||||
Now a fully functioning code might look like the following for start rule `StartRule`:
|
||||
|
||||
```
|
||||
using Antlr4.Runtime;
|
||||
|
|
@ -58,7 +52,7 @@ This program will work. But it won't be useful unless you do one of the followin
|
|||
|
||||
(please note that production code is target specific, so you can't have multi target grammars that include production code)
|
||||
|
||||
### How do I create and run a custom listener?
|
||||
## How do I create and run a custom listener?
|
||||
|
||||
Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value".
|
||||
|
||||
|
|
@ -96,4 +90,5 @@ ParseTreeWalker.DEFAULT.walk(printer, tree);
|
|||
|
||||
Further information can be found from The Definitive ANTLR Reference book.
|
||||
|
||||
The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#.
|
||||
The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#. See also [Sam Harwell's alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs)
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 439 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
|
|
@ -15,7 +15,7 @@ The tests were conducted using Selenium. No issue was found, so you should find
|
|||
|
||||
## Is NodeJS supported?
|
||||
|
||||
The runtime has also been extensively tested against Node.js 0.10.33. No issue was found.
|
||||
The runtime has also been extensively tested against Node.js 0.12.7. No issue was found.
|
||||
|
||||
## How to create a JavaScript lexer or parser?
|
||||
|
||||
|
|
@ -31,7 +31,9 @@ For a full list of antlr4 tool options, please visit the [tool documentation pag
|
|||
|
||||
Once you've generated the lexer and/or parser code, you need to download the runtime.
|
||||
|
||||
The JavaScript runtime is available from the ANTLR web site [download section](http://www.antlr.org/download/index.html). The runtime is provided in the form of source code, so no additional installation is required.
|
||||
The JavaScript runtime is [available from npm](https://www.npmjs.com/package/antlr4).
|
||||
|
||||
If you can't use npm, the JavaScript runtime is also available from the ANTLR web site [download section](http://www.antlr.org/download/index.html). The runtime is provided in the form of source code, so no additional installation is required.
|
||||
|
||||
We will not document here how to refer to the runtime from your project, since this would differ a lot depending on your project type and IDE.
|
||||
|
||||
|
|
@ -47,11 +49,28 @@ However, it would be a bit of a problem when it comes to get it into a browser.
|
|||
<script src='lib/myscript.js'>
|
||||
```
|
||||
|
||||
In order to avoid having to do this, and also to have the exact same code for browsers and Node.js, we rely on a script which provides the equivalent of the Node.js 'require' function.
|
||||
To avoid having doing this, the preferred approach is to bundle antlr4 with your parser code, using webpack.
|
||||
|
||||
This script is provided by Torben Haase, and is NOT part of ANTLR JavaScript runtime, although the runtime heavily relies on it. Please note that syntax for 'require' in NodeJS is different from the one implemented by RequireJS and similar frameworks.
|
||||
You can get [information on webpack here](https://webpack.github.io).
|
||||
|
||||
So in short, assuming you have at the root of your web site, both the 'antlr4' directory and a 'lib' directory with 'require.js' inside it, all you need to put in your HTML header is the following:
|
||||
The steps to create your parsing code are the following:
|
||||
- generate your lexer, parser, listener and visitor using the antlr tool
|
||||
- write your parse tree handling code by providig your custom listener or visitor, and associated code, using 'require' to load antlr.
|
||||
- create an index.js file with the entry point to your parsing code (or several if required).
|
||||
- test your parsing logic thoroughly using node.js
|
||||
|
||||
You are now ready to bundle your parsing code as follows:
|
||||
- following webpack specs, create a webpack.config file
|
||||
- in the webpack.config file, exclude node.js only modules using: node: { module: "empty", net: "empty", fs: "empty" }
|
||||
- from the cmd line, nag-vigate to the directory containing webpack.config and type: webpack
|
||||
|
||||
This will produce a single js file containing all your parsing code. Easy to include in your web pages!
|
||||
|
||||
If you can't use webpack, you can use the lib/require.js script which implements the Node.js 'require' function in brwsers.
|
||||
|
||||
This script is provided by Torben Haase, and is NOT part of ANTLR JavaScript runtime.
|
||||
|
||||
Assuming you have, at the root of your web site, both the 'antlr4' directory and a 'lib' directory with 'require.js' inside it, all you need to put in your HTML header is the following:
|
||||
|
||||
```xml
|
||||
<script src='lib/require.js'>
|
||||
|
|
@ -62,16 +81,6 @@ So in short, assuming you have at the root of your web site, both the 'antlr4' d
|
|||
|
||||
This will load the runtime asynchronously.
|
||||
|
||||
## How do I get the runtime in Node.js?
|
||||
|
||||
Right now, there is no npm package available, so you need to register a link instead. This can be done by running the following command from the antlr4 directory:
|
||||
|
||||
```bash
|
||||
$ npm link antlr4
|
||||
```
|
||||
|
||||
This will install antlr4 using the package.json descriptor that comes with the script.
|
||||
|
||||
## How do I run the generated lexer and/or parser?
|
||||
|
||||
Let's suppose that your grammar is named, as above, "MyGrammar". Let's suppose this parser comprises a rule named "StartRule". The tool will have generated for you the following files:
|
||||
|
|
|
|||
|
|
@ -229,6 +229,70 @@ popd
|
|||
|
||||
### CSharp
|
||||
|
||||
*Publishing to Nuget from Linux/MacOSX*
|
||||
|
||||
**Getting ready to run Nuget**
|
||||
|
||||
Of course you need Mono and `nuget` to be installed. On mac:
|
||||
|
||||
```bash
|
||||
brew install mono
|
||||
brew install nuget
|
||||
```
|
||||
|
||||
Or, you can [download nuget.exe](https://dist.nuget.org/win-x86-commandline/latest/nuget.exe).
|
||||
|
||||
From the shell on mac, you can check all is ok by typing
|
||||
|
||||
```bash
|
||||
nuget
|
||||
```
|
||||
|
||||
This should display the nuget help.
|
||||
|
||||
**Creating the assembly**
|
||||
|
||||
```bash
|
||||
$ cd runtime/CSharp/runtime/CSharp/Antlr4.Runtime
|
||||
$ xbuild /p:Configuration=Release Antlr4.Runtime.mono.csproj
|
||||
...
|
||||
Copying file from '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.Standard.dll' to '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/lib/Release/Antlr4.Runtime.Standard.dll'
|
||||
Done building project "/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Antlr4.Runtime.mono.csproj".
|
||||
```
|
||||
|
||||
Alternately, you may want to build ANTLR using Xamarin Studio Community (free).
|
||||
|
||||
**Packaging for NuGet**
|
||||
|
||||
```bash
|
||||
cd runtime/CSharp/runtime/CSharp
|
||||
```
|
||||
|
||||
which is where the `Package.nuspec` file resides.
|
||||
|
||||
Type the following command:
|
||||
|
||||
```bash
|
||||
$ nuget pack Package.nuspec
|
||||
Attempting to build package from 'Package.nuspec'.
|
||||
Successfully created package '/Users/parrt/antlr/code/antlr4/runtime/CSharp/runtime/CSharp/Antlr4.Runtime.Standard.4.6.0.nupkg'.
|
||||
```
|
||||
|
||||
This should display: Successfully created package *<package-path>*
|
||||
|
||||
**Publishing to NuGet**
|
||||
|
||||
You need to be a NuGet owner for "ANTLR 4 Standard Runtime"
|
||||
As a registered NuGet user, you can then manually upload the package spec here (`runtime/CSharp/runtime/CSharp/Package.nuspec`): [https://www.nuget.org/packages/manage/upload](https://www.nuget.org/packages/manage/upload)
|
||||
|
||||
Alternately, you can publish from the cmd line. You need to get your NuGet key from [https://www.nuget.org/account#](https://www.nuget.org/account#) and then from the cmd line, you can then type:
|
||||
|
||||
```bash
|
||||
nuget push Antlr4.Runtime.Standard.<version>.nupkg <your-key> -Source https://www.nuget.org/api/v2/package
|
||||
```
|
||||
|
||||
**Creating DLLs**
|
||||
|
||||
```bash
|
||||
cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
|
||||
# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
|
||||
|
|
@ -237,7 +301,7 @@ rm Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.dll
|
|||
# build
|
||||
xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
|
||||
# zip it up to get a version number on zip filename
|
||||
zip --junk-paths /tmp/antlr-csharp-runtime-4.6.zip Antlr4.Runtime/bin/net35/Release/Antlr4.Runtime.dll
|
||||
zip --junk-paths /tmp/antlr-csharp-runtime-4.6.zip Antlr4.Runtime/obj/net20/Release/Antlr4.Runtime.Standard.dll
|
||||
cp /tmp/antlr-csharp-runtime-4.6.zip ~/antlr/sites/website-antlr4/download
|
||||
```
|
||||
|
||||
|
|
@ -294,7 +358,7 @@ python setup.py register -r pypi
|
|||
python setup.py sdist bdist_wininst upload -r pypi
|
||||
```
|
||||
|
||||
Add links to the artifacts from download.html
|
||||
There are links to the artifacts in [download.html](http://www.antlr.org/download.html) already.
|
||||
|
||||
### C++
|
||||
|
||||
|
|
@ -314,6 +378,7 @@ On a Mac (with XCode 7+ installed):
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
./deploy-macos.sh
|
||||
cp antlr4-cpp-runtime-macos.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-macos.zip
|
||||
```
|
||||
|
||||
On any Mac or Linux machine:
|
||||
|
|
@ -321,6 +386,7 @@ On any Mac or Linux machine:
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
./deploy-source.sh
|
||||
cp antlr4-cpp-runtime-source.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-source.zip
|
||||
```
|
||||
|
||||
On a Windows machine the build scripts checks if VS 2013 and/or VS 2015 are installed and builds binaries for each, if found. This script requires 7z to be installed (http://7-zip.org).
|
||||
|
|
@ -328,15 +394,17 @@ On a Windows machine the build scripts checks if VS 2013 and/or VS 2015 are inst
|
|||
```bash
|
||||
cd runtime/Cpp
|
||||
deploy-windows.cmd
|
||||
cp antlr4-cpp-runtime-vs2015.zip ~/antlr/sites/website-antlr4/download/antlr4-cpp-runtime-4.6-vs2015.zip
|
||||
```
|
||||
|
||||
Move target to website (**_rename to a specific ANTLR version first if needed_**):
|
||||
|
||||
```bash
|
||||
pushd ~/antlr/sites/website-antlr4/download
|
||||
git add antlr4cpp-runtime-macos.zip
|
||||
git add antlr4cpp-runtime-windows.zip
|
||||
git add antlr4cpp-runtime-source.zip
|
||||
# vi index.html
|
||||
git add antlr4cpp-runtime-4.6-macos.zip
|
||||
git add antlr4cpp-runtime-4.6-windows.zip
|
||||
git add antlr4cpp-runtime-4.6-source.zip
|
||||
git commit -a -m 'update C++ runtime'
|
||||
git push origin gh-pages
|
||||
popd
|
||||
|
|
@ -344,7 +412,7 @@ popd
|
|||
|
||||
## Update javadoc for runtime and tool
|
||||
|
||||
First gen javadoc:
|
||||
First, gen javadoc:
|
||||
|
||||
```bash
|
||||
$ cd antlr4
|
||||
|
|
|
|||
|
|
@ -1,47 +1,99 @@
|
|||
# ANTLR4 Language Target, Runtime for Swift
|
||||
|
||||
### Usage
|
||||
## Install ANTLR4
|
||||
|
||||
#### 1. Install ANTLR4
|
||||
Make sure you have the ANTLR
|
||||
installed.[The getting started guide](getting-started.md) should get
|
||||
you started.
|
||||
|
||||
[The getting started guide](getting-started.md) should get you started.
|
||||
## Create a Swift lexer or parser
|
||||
This is pretty much the same as creating a Java lexer or parser,
|
||||
except you need to specify the language target, for example:
|
||||
|
||||
#### 2. create a Swift lexer or parser
|
||||
This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
|
||||
```
|
||||
$ antlr4 -Dlanguage=Swift MyGrammar.g4
|
||||
```
|
||||
For a full list of antlr4 tool options, please visit the
|
||||
[tool documentation page](tool-options.md).
|
||||
|
||||
```
|
||||
$ antlr4 -Dlanguage=Swift MyGrammar.g4
|
||||
```
|
||||
For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md).
|
||||
## Build your Swift project with ANTLR runtime
|
||||
|
||||
#### 3. Get the Swift ANTLR runtime
|
||||
You will find Swift runtime (framework project) in
|
||||
The following instructions are assuming Xcode as the IDE:
|
||||
|
||||
```
|
||||
antlr4/runtime/Swift
|
||||
```
|
||||
* __Add parser/lexer to project__. Make sure the parsers/lexers
|
||||
generated in __step 2__ are added to the project. To do this, you can
|
||||
drag the generated files from Finder to the Xcode IDE. Remember to
|
||||
check __Copy items if needed__ to make sure the files are actually
|
||||
moved into the project folder instead of symbolic links (see the
|
||||
screenshot below). After moving you will be able to see your files in
|
||||
the project navigator. But when you open one of the files, you will
|
||||
see Xcode complaining the module "Antlr4" could not be found at the
|
||||
import statement. This is expected, since we still need the ANTLR
|
||||
Swift runtime for those missing symbols.
|
||||
|
||||
#### 4. Example playground
|
||||
<img src=images/dragfile.png width="500">
|
||||
|
||||
* __Download ANTLR runtime__. Due to unstable ABI of Swift language,
|
||||
there will not be a single "library" for the Swift ANTLR runtime for
|
||||
now. To get Swift ANTLR runtime, clone the ANTLR repository. Open it
|
||||
in finder. From the root directory of the repo, go to runtime/Swift
|
||||
folder. You will see the Xcode project manifest file:
|
||||
__Antlr4.xcodeproj__.
|
||||
|
||||
* __Import ANTLR Swift runtime into project__. Drag Antlr4.xcodeproj
|
||||
into your project, after this is done, your Xcode project navigator
|
||||
will be something like the screenshot below. In this case, your own
|
||||
project is "Smalltalk", and you will be able to see the
|
||||
Antlr4.xcodeproj shown as a contained project. The error message will
|
||||
still be there, that's because we still need to tell Xcode how to find
|
||||
the runtime.
|
||||
|
||||
<img src=images/xcodenav.png width="300">
|
||||
|
||||
* __Build ANTLR runtime__. By expanding the "Products" folder in the
|
||||
inner project (Antlr4.xcodeproj), you will see two Antlr4.framework
|
||||
files. ".framework" file is the swift version of ".jar", ".a" as in
|
||||
JAVA, C/C++ Initially those two files should be red, that's because
|
||||
they are not built. To build, click the "target selection" button
|
||||
right next to your Xcode run button. And in the drop down select the
|
||||
target you want to build. And you will see the two Antlr4.framework
|
||||
files are for iOS and OSX, as shown below. After target selection,
|
||||
press "CMD+B", and Xcode will build the framework for you. Then you
|
||||
will see one of the frameworks become black.
|
||||
|
||||
<img src=images/targetselection.png width="500">
|
||||
|
||||
* __Add dependencies__. Simply adding ANTLR Swift runtime and build
|
||||
the artifact is not enough. You still need to specify
|
||||
dependencies. Click your own project (Smalltalk), and you will see
|
||||
project setting page. Go to "Build Phase", and inside it make sure
|
||||
your ANTLR Swift runtime framework is added to both "__Target
|
||||
Dependencies__" and "__Link Binary With Libraries__" sections, as
|
||||
shown below. After correctly added dependencies, the error message for
|
||||
importing library will be gone.
|
||||
|
||||
<img src=images/xcodedep.png width="800">
|
||||
|
||||
## Example playground
|
||||
|
||||
The Swift runtime includes an Xcode playground to get started with.
|
||||
|
||||
In Xcode, open `antlr4/runtime/Swift/Antlr4.xcworkspace`. Select
|
||||
"Antlr4 OSX > My Mac" as the build target, and build the project as normal.
|
||||
The playground should then be active.
|
||||
First go to the ANTLR4 repository, and open
|
||||
`runtime/Swift/Antlr4.xcworkspace` in Xcode. Select "Antlr4 OSX > My
|
||||
Mac" as the build target, and build the project as normal. The
|
||||
playground should then be active.
|
||||
|
||||
The playground includes a simple grammar called "Hello", and an example for
|
||||
walking the parse tree. You should see in the playground output that it is
|
||||
printing messages for each node in the parse tree as it walks.
|
||||
The playground includes a simple grammar called "Hello", and an
|
||||
example for walking the parse tree. You should see in the playground
|
||||
output that it is printing messages for each node in the parse tree as
|
||||
it walks.
|
||||
|
||||
The playground shows how to create a lexer, token stream, and parser, and
|
||||
how to execute the parse.
|
||||
The grammar is defined in the playground's `Resources/Hello.g4`. The
|
||||
parser was generated from the grammar using ANTLR like this:
|
||||
|
||||
The grammar is defined in the playground's `Resources/Hello.g4`. The parser
|
||||
was generated from the grammar using Antlr4 like this:
|
||||
|
||||
```
|
||||
cd 'antlr4/runtime/Swift/Antlr4 playground.playground/Resources'
|
||||
```
|
||||
antlr4 -Dlanguage=Swift -visitor -o ../Sources/Autogen Hello.g4
|
||||
```
|
||||
|
||||
The example tree walker is in `Sources/HelloWalker.swift`.
|
||||
The example tree walker is in Sources/HelloWalker.swift.
|
||||
|
||||
|
|
|
|||
2
pom.xml
2
pom.xml
|
|
@ -13,7 +13,7 @@
|
|||
</parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<packaging>pom</packaging>
|
||||
|
||||
<name>ANTLR 4</name>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-test-annotations</artifactId>
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-testsuite</artifactId>
|
||||
<name>ANTLR 4 Runtime Tests (2nd generation)</name>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime-test-annotation-processors</artifactId>
|
||||
|
|
|
|||
|
|
@ -302,30 +302,6 @@ public class LexerExecDescriptors {
|
|||
|
||||
}
|
||||
|
||||
public static class CharSetWithReversedRange extends BaseLexerTestDescriptor {
|
||||
public String input = "9";
|
||||
/**
|
||||
A
|
||||
[@0,0:0='9',<1>,1:0]
|
||||
[@1,1:0='<EOF>',<-1>,1:1]
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String output;
|
||||
|
||||
public String errors = null;
|
||||
public String startRule = "";
|
||||
public String grammarName = "L";
|
||||
|
||||
/**
|
||||
lexer grammar L;
|
||||
A : [z-a9]+ {<writeln("\"A\"")>} ;
|
||||
WS : [ \n]+ -> skip ;
|
||||
*/
|
||||
@CommentHasStringValue
|
||||
public String grammar;
|
||||
|
||||
}
|
||||
|
||||
public static class EOFByItself extends BaseLexerTestDescriptor {
|
||||
public String input = "";
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -1,11 +1,18 @@
|
|||
# C# target for ANTLR 4
|
||||
|
||||
### Note to existing users
|
||||
### Note to historical users
|
||||
|
||||
Versions of ANTLR 4.4.x and before managed the C# target as a [separate tool provided by Sam Harwell](https://github.com/tunnelvisionlabs/antlr4cs/releases/tag/v4.3.0). As of 4.5, we our releasing a (mono-compatible) C# target derived from Sam's with the main tool. ANTLR 4.5 is now able to generate C#, Java, Python 2, Python 3 and JavaScript. Sam continues to work on his version of the ANTLR tool and so a C# target is also available through that.
|
||||
Versions of ANTLR 4.4.x and before managed the C#
|
||||
target as part of a [separate tool provided by Sam Harwell](https://github.com/tunnelvisionlabs/antlr4cs/releases/tag/v4.3.0).
|
||||
As of 4.5, we our releasing a (mono-compatible) C# target together
|
||||
with the main tool.
|
||||
|
||||
This makes it possible to use ANTLR 4 in any C# development environment, including of course Microsoft Visual Studio, but also Xamarin Studio, which runs on MacOS X.
|
||||
Given Microsoft's recent commitment to *cross-platform developer experiences*, we believe this is a great opportunity for C# developers.
|
||||
Mono compatibility makes it possible to use ANTLR 4 in any C# development
|
||||
environment, including of course Microsoft Visual Studio, but also Xamarin Studio, which runs on MacOS X.
|
||||
Given Microsoft's recent commitment to *cross-platform developer experiences*,
|
||||
we believe this is a great opportunity for C# developers.
|
||||
|
||||
Releasing the runtime with the tool ensures that you can get the exact same behavior across many languages: Java, C#, Python, JavaScript, Go, Swift and C++.
|
||||
|
||||
## Getting Started
|
||||
|
||||
|
|
@ -16,39 +23,37 @@ You can install *any* of the following versions of Java to use this target.
|
|||
|
||||
If you already have one of the following installed, you should check to make sure the installation is up-to-date.
|
||||
|
||||
* Java 8 runtime environment (x86 or x64)
|
||||
* Java 8 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
* Java 7 runtime environment (x86 or x64)
|
||||
* Java 7 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
* Java 6 runtime environment (x86 or x64)
|
||||
* Java 6 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
|
||||
|
||||
### Step 2: Download the tool and runtime, and install the runtime
|
||||
### Step 2: Download the tool
|
||||
|
||||
You need to download the following from the ANTLR web site:
|
||||
You need to download the ANTLR tool from the ANTLR web site.
|
||||
This is a Java archive (*.jar) used to generate the C# code from an ANTLR grammar.
|
||||
|
||||
1. The ANTLR tool. This is a Java archive (*.jar) used to generate the C# code from an ANTLR grammar.
|
||||
2. The ANTLR runtime assembly. This is a regular .Net assembly (*.dll).
|
||||
|
||||
You then need to install the runtime assembly in the GAC.
|
||||
This is required to easily reference the runtime from your project.
|
||||
Go the the directory where you downloaded the runtime, and type the following command:
|
||||
`gacutil -i Antlr4.Runtime.dll`
|
||||
|
||||
### Step 3: Add a reference to the ANTLR runtime in your project
|
||||
|
||||
We trust that you know how to do this :-).
|
||||
|
||||
### Step 4: Add or create a grammar file (*.g4) in your project
|
||||
### Step 3: Add or create a grammar file (*.g4) in your project
|
||||
|
||||
To avoid confusing your IDE, we suggest setting the build action to None for this file.
|
||||
See the docs and the book to learn about writing lexer and parser grammars.
|
||||
|
||||
### Step 5: Generate the C# code
|
||||
|
||||
This can be done either from the cmd line, or by adding a custom pre-build command.
|
||||
At minimal, the cmd line should look as follows: ``java -jar antlr4-4.5.jar -Dlanguage=CSharp grammar.g4``
|
||||
### Step 4: Generate the C# code
|
||||
|
||||
This can be done either from the cmd line, or by adding a custom pre-build command in your project.
|
||||
At minimal, the cmd line should look as follows: ``java -jar antlr4-4.6.jar -Dlanguage=CSharp grammar.g4``
|
||||
This will generate the files, which you can then integrate in your project.
|
||||
This is just a quick start. The tool has many useful options to control generation, please refer to its documentation.
|
||||
|
||||
This is just a quick start. The tool has many useful options to control generation, please refer to its documentation.
|
||||
|
||||
### Step 5: Add a reference to the ANTLR runtime in your project
|
||||
|
||||
The Antlr 4 standard runtime for C# is now available from NuGet.
|
||||
We trust that you know how to do add NuGet references to your project :-).
|
||||
The package id is Antlr.4.Runtime. We do not support other packages.
|
||||
|
||||
|
||||
### Step 6: You're done!
|
||||
|
||||
Of course, the generated code is not going to meet your requirement by magic.
|
||||
|
|
@ -60,10 +65,10 @@ While the latter works, it is no longer the recommended approach, because it is
|
|||
|
||||
See the web site for examples of using the generated code.
|
||||
|
||||
To learn more about ANTLR 4, read the book.
|
||||
To learn more about ANTLR 4, read [the book](http://a.co/2n4rJlb).
|
||||
|
||||
### Visual Studio integration
|
||||
|
||||
If you require tighter Visual Studio integration, you can use the tools from Tunnel Vision Labs.
|
||||
If you require tighter Visual Studio integration, you can use the tools from [Tunnel Vision Labs](http://tunnelvisionlabs.com/).
|
||||
(please note however that they use a different tool and runtime)
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
<OutputType>Library</OutputType>
|
||||
<AppDesignerFolder>Properties</AppDesignerFolder>
|
||||
<RootNamespace>Antlr4.Runtime</RootNamespace>
|
||||
<AssemblyName>Antlr4.Runtime</AssemblyName>
|
||||
<AssemblyName>Antlr4.Runtime.Standard</AssemblyName>
|
||||
<TargetFrameworkVersion>v3.5</TargetFrameworkVersion>
|
||||
<FileAlignment>512</FileAlignment>
|
||||
<BaseIntermediateOutputPath>obj\net20\</BaseIntermediateOutputPath>
|
||||
|
|
@ -21,7 +21,7 @@
|
|||
<DebugSymbols>true</DebugSymbols>
|
||||
<DebugType>full</DebugType>
|
||||
<Optimize>false</Optimize>
|
||||
<OutputPath>bin\net35\Debug\</OutputPath>
|
||||
<OutputPath>lib\Debug</OutputPath>
|
||||
<DefineConstants>DEBUG;TRACE;NET35PLUS</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
|
|
@ -29,7 +29,7 @@
|
|||
</PropertyGroup>
|
||||
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
|
||||
<Optimize>true</Optimize>
|
||||
<OutputPath>bin\net35\Release\</OutputPath>
|
||||
<OutputPath>lib\Release</OutputPath>
|
||||
<DefineConstants>TRACE;NET35PLUS</DefineConstants>
|
||||
<ErrorReport>prompt</ErrorReport>
|
||||
<WarningLevel>4</WarningLevel>
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
<OutputType>Library</OutputType>
|
||||
<AppDesignerFolder>Properties</AppDesignerFolder>
|
||||
<RootNamespace>Antlr4.Runtime</RootNamespace>
|
||||
<AssemblyName>Antlr4.Runtime</AssemblyName>
|
||||
<AssemblyName>Antlr4.Runtime.Standard</AssemblyName>
|
||||
<TargetFrameworkVersion>v3.5</TargetFrameworkVersion>
|
||||
<FileAlignment>512</FileAlignment>
|
||||
<BaseIntermediateOutputPath>obj\net35\</BaseIntermediateOutputPath>
|
||||
|
|
|
|||
|
|
@ -5,8 +5,6 @@ VisualStudioVersion = 12.0.30110.0
|
|||
MinimumVisualStudioVersion = 10.0.40219.1
|
||||
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{47C0086D-577C-43DA-ADC7-544F27656E45}"
|
||||
ProjectSection(SolutionItems) = preProject
|
||||
..\..\tool\resources\org\antlr\v4\tool\templates\codegen\CSharp\CSharp.stg = ..\..\tool\resources\org\antlr\v4\tool\templates\codegen\CSharp\CSharp.stg
|
||||
..\..\tool\src\org\antlr\v4\codegen\CSharpTarget.java = ..\..\tool\src\org\antlr\v4\codegen\CSharpTarget.java
|
||||
..\..\Readme.md = ..\..\Readme.md
|
||||
EndProjectSection
|
||||
EndProject
|
||||
|
|
|
|||
|
|
@ -0,0 +1,24 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
|
||||
<metadata>
|
||||
<id>Antlr4.Runtime.Standard</id>
|
||||
<version>4.6.0</version>
|
||||
<language>en-us</language>
|
||||
<title>ANTLR 4 Standard Runtime</title>
|
||||
<description>The standard C# ANTLR 4 runtime from the ANTLR Organization</description>
|
||||
<summary>The runtime library for parsers generated by the C# target of the standard ANTLR 4 tool.</summary>
|
||||
<authors>Eric Vergnaud, Terence Parr, Sam Harwell</authors>
|
||||
<owners>The ANTLR Organization</owners>
|
||||
<releaseNotes>https://github.com/antlr/antlr4/releases</releaseNotes>
|
||||
<requireLicenseAcceptance>true</requireLicenseAcceptance>
|
||||
<licenseUrl>https://github.com/antlr/antlr4/blob/master/LICENSE.txt</licenseUrl>
|
||||
<projectUrl>https://github.com/antlr/antlr4</projectUrl>
|
||||
<iconUrl>https://raw.github.com/antlr/website-antlr4/master/images/icons/antlr.png</iconUrl>
|
||||
<copyright>Copyright (c) 2012-2016 The ANTLR Project. All rights reserved.</copyright>
|
||||
<tags>antlr parsing grammar</tags>
|
||||
<dependencies />
|
||||
</metadata>
|
||||
<files>
|
||||
<file src="Antlr4.Runtime/lib/Release/Antlr4.Runtime.Standard.dll" target="lib/net35/"/>
|
||||
</files>
|
||||
</package>
|
||||
|
|
@ -3,10 +3,14 @@
|
|||
* can be found in the LICENSE.txt file in the project root.
|
||||
*/
|
||||
|
||||
#include "misc/Interval.h"
|
||||
#include "TokenSource.h"
|
||||
#include "support/StringUtils.h"
|
||||
#include "CharStream.h"
|
||||
#include "Recognizer.h"
|
||||
#include "Vocabulary.h"
|
||||
|
||||
#include "misc/Interval.h"
|
||||
|
||||
#include "support/StringUtils.h"
|
||||
#include "support/CPPUtils.h"
|
||||
|
||||
#include "CommonToken.h"
|
||||
|
|
@ -149,6 +153,10 @@ antlr4::CharStream *CommonToken::getInputStream() const {
|
|||
}
|
||||
|
||||
std::string CommonToken::toString() const {
|
||||
return toString(nullptr);
|
||||
}
|
||||
|
||||
std::string CommonToken::toString(Recognizer *r) const {
|
||||
std::stringstream ss;
|
||||
|
||||
std::string channelStr;
|
||||
|
|
@ -164,8 +172,12 @@ std::string CommonToken::toString() const {
|
|||
txt = "<no text>";
|
||||
}
|
||||
|
||||
std::string typeString = std::to_string(symbolToNumeric(_type));
|
||||
if (r != nullptr)
|
||||
typeString = r->getVocabulary().getDisplayName(_type);
|
||||
|
||||
ss << "[@" << symbolToNumeric(getTokenIndex()) << "," << symbolToNumeric(_start) << ":" << symbolToNumeric(_stop)
|
||||
<< "='" << txt << "',<" << symbolToNumeric(_type) << ">" << channelStr << "," << _line << ":"
|
||||
<< "='" << txt << "',<" << typeString << ">" << channelStr << "," << _line << ":"
|
||||
<< getCharPositionInLine() << "]";
|
||||
|
||||
return ss.str();
|
||||
|
|
|
|||
|
|
@ -150,6 +150,7 @@ namespace antlr4 {
|
|||
|
||||
virtual std::string toString() const override;
|
||||
|
||||
virtual std::string toString(Recognizer *r) const;
|
||||
private:
|
||||
void InitializeInstanceFields();
|
||||
};
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
|
||||
#pragma once
|
||||
|
||||
#include "antlr4-common.h"
|
||||
|
||||
namespace antlr4 {
|
||||
|
||||
/// The default mechanism for creating tokens. It's used by default in Lexer and
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ namespace antlr4 {
|
|||
namespace misc {
|
||||
|
||||
// Helpers to convert certain unsigned symbols (e.g. Token::EOF) to their original numeric value (e.g. -1)
|
||||
// and vice version. This is needed mostly for intervals to keep their original order and for toString()
|
||||
// and vice versa. This is needed mostly for intervals to keep their original order and for toString()
|
||||
// methods to print the original numeric value (e.g. for tests).
|
||||
size_t numericToSymbol(ssize_t v);
|
||||
ssize_t symbolToNumeric(size_t v);
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
<relativePath>../../pom.xml</relativePath>
|
||||
</parent>
|
||||
<artifactId>antlr4-runtime</artifactId>
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "antlr4",
|
||||
"version": "4.6",
|
||||
"version": "4.6.0",
|
||||
"description": "JavaScript runtime for ANTLR4",
|
||||
"main": "src/antlr4/index.js",
|
||||
"repository": "antlr/antlr4.git",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
publishing to npm
|
||||
=================
|
||||
The JavaScript runtime is published on npm.
|
||||
There is nothing to build since JavaScript is based on source code only.
|
||||
The JavaScript itself is tested using npm, so assumption is npm is already installed.
|
||||
The current npm version used is 3.10.9.
|
||||
The publishing itself relies on the information in src/antlr4/package.json.
|
||||
To publish, from Terminal, you need to navigate to the src/ directory.
|
||||
Once there, login: npm login
|
||||
Then npm publish antlr4
|
||||
That's it!
|
||||
|
||||
|
|
@ -1,59 +0,0 @@
|
|||
# file GENERATED by distutils, do NOT edit
|
||||
LICENSE.txt
|
||||
README.txt
|
||||
RELEASE-4.5.txt
|
||||
setup.py
|
||||
src/antlr4/BufferedTokenStream.py
|
||||
src/antlr4/CommonTokenFactory.py
|
||||
src/antlr4/CommonTokenStream.py
|
||||
src/antlr4/FileStream.py
|
||||
src/antlr4/InputStream.py
|
||||
src/antlr4/IntervalSet.py
|
||||
src/antlr4/LL1Analyzer.py
|
||||
src/antlr4/Lexer.py
|
||||
src/antlr4/ListTokenSource.py
|
||||
src/antlr4/Parser.py
|
||||
src/antlr4/ParserInterpreter.py
|
||||
src/antlr4/ParserRuleContext.py
|
||||
src/antlr4/PredictionContext.py
|
||||
src/antlr4/Recognizer.py
|
||||
src/antlr4/RuleContext.py
|
||||
src/antlr4/Token.py
|
||||
src/antlr4/Utils.py
|
||||
src/antlr4/__init__.py
|
||||
src/antlr4/atn/ATN.py
|
||||
src/antlr4/atn/ATNConfig.py
|
||||
src/antlr4/atn/ATNConfigSet.py
|
||||
src/antlr4/atn/ATNDeserializationOptions.py
|
||||
src/antlr4/atn/ATNDeserializer.py
|
||||
src/antlr4/atn/ATNSimulator.py
|
||||
src/antlr4/atn/ATNState.py
|
||||
src/antlr4/atn/ATNType.py
|
||||
src/antlr4/atn/LexerATNSimulator.py
|
||||
src/antlr4/atn/LexerAction.py
|
||||
src/antlr4/atn/LexerActionExecutor.py
|
||||
src/antlr4/atn/ParserATNSimulator.py
|
||||
src/antlr4/atn/PredictionMode.py
|
||||
src/antlr4/atn/SemanticContext.py
|
||||
src/antlr4/atn/Transition.py
|
||||
src/antlr4/atn/__init__.py
|
||||
src/antlr4/dfa/DFA.py
|
||||
src/antlr4/dfa/DFASerializer.py
|
||||
src/antlr4/dfa/DFAState.py
|
||||
src/antlr4/dfa/__init__.py
|
||||
src/antlr4/error/DiagnosticErrorListener.py
|
||||
src/antlr4/error/ErrorListener.py
|
||||
src/antlr4/error/ErrorStrategy.py
|
||||
src/antlr4/error/Errors.py
|
||||
src/antlr4/error/__init__.py
|
||||
src/antlr4/tree/Chunk.py
|
||||
src/antlr4/tree/ParseTreeMatch.py
|
||||
src/antlr4/tree/ParseTreePattern.py
|
||||
src/antlr4/tree/ParseTreePatternMatcher.py
|
||||
src/antlr4/tree/RuleTagToken.py
|
||||
src/antlr4/tree/TokenTagToken.py
|
||||
src/antlr4/tree/Tree.py
|
||||
src/antlr4/tree/Trees.py
|
||||
src/antlr4/tree/__init__.py
|
||||
src/antlr4/xpath/XPath.py
|
||||
src/antlr4/xpath/__init__.py
|
||||
|
|
@ -10,7 +10,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4-tool-testsuite</artifactId>
|
||||
<name>ANTLR 4 Tool Tests</name>
|
||||
|
|
|
|||
|
|
@ -320,4 +320,21 @@ public class TestSymbolIssues extends BaseJavaToolTest {
|
|||
|
||||
testErrors(test, false);
|
||||
}
|
||||
|
||||
@Test public void testCharsCollision() throws Exception {
|
||||
String[] test = {
|
||||
"lexer grammar L;\n" +
|
||||
"TOKEN_RANGE: [aa-f];\n" +
|
||||
"TOKEN_RANGE_2: [A-FD-J];\n" +
|
||||
"TOKEN_RANGE_3: 'Z' | 'K'..'R' | 'O'..'V';\n" +
|
||||
"TOKEN_RANGE_4: 'g'..'l' | [g-l];\n", // Handling in ATNOptimizer.
|
||||
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:2:18: chars \"a-f\" used multiple times in set [aa-f]\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:3:18: chars \"D-J\" used multiple times in set [A-FD-J]\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4:4:13: chars \"O-V\" used multiple times in set 'Z' | 'K'..'R' | 'O'..'V'\n" +
|
||||
"warning(" + ErrorType.CHARACTERS_COLLISION_IN_SET.code + "): L.g4::: chars \"g-l\" used multiple times in set [g-l]\n"
|
||||
};
|
||||
|
||||
testErrors(test, false);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -487,11 +487,39 @@ public class TestToolSyntaxErrors extends BaseJavaToolTest {
|
|||
"Error3: '';\n" +
|
||||
"NotError: ' ';";
|
||||
String expected =
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:2:8: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:2:16: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:3:8: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:4:15: string literals cannot be empty\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_NOT_ALLOWED.code + "): T.g4:5:8: string literals cannot be empty\n";
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:2:8: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:2:16: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:3:8: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:4:15: string literals and sets cannot be empty: ''\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): T.g4:5:8: string literals and sets cannot be empty: ''\n";
|
||||
|
||||
String[] pair = new String[] {
|
||||
grammar,
|
||||
expected
|
||||
};
|
||||
|
||||
super.testErrors(pair, true);
|
||||
}
|
||||
|
||||
@Test public void testInvalidCharSetAndRange() {
|
||||
String grammar =
|
||||
"lexer grammar Test;\n" +
|
||||
"INVALID_RANGE: 'GH'..'LM';\n" +
|
||||
"INVALID_RANGE_2: 'F'..'A' | 'Z';\n" +
|
||||
"VALID_STRING_LITERALS: '\\u1234' | '\\t' | [\\-\\]];\n" +
|
||||
"INVALID_CHAR_SET: [f-az][];\n" +
|
||||
"INVALID_CHAR_SET_2: [\\u24\\uA2][\\u24];\n" + //https://github.com/antlr/antlr4/issues/1077
|
||||
"INVALID_CHAR_SET_3: [\\t\\{];";
|
||||
|
||||
String expected =
|
||||
"error(" + ErrorType.INVALID_LITERAL_IN_LEXER_SET.code + "): Test.g4:2:23: multi-character literals are not allowed in lexer sets: 'GH'\n" +
|
||||
"error(" + ErrorType.INVALID_LITERAL_IN_LEXER_SET.code + "): Test.g4:2:29: multi-character literals are not allowed in lexer sets: 'LM'\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:3:26: string literals and sets cannot be empty: 'F'..'A'\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:5:23: string literals and sets cannot be empty: [f-a]\n" +
|
||||
"error(" + ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED.code + "): Test.g4:5:29: string literals and sets cannot be empty: []\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:6:23: invalid escape sequence\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:6:33: invalid escape sequence\n" +
|
||||
"error(" + ErrorType.INVALID_ESCAPE_SEQUENCE.code + "): Test.g4:7:23: invalid escape sequence\n";
|
||||
|
||||
String[] pair = new String[] {
|
||||
grammar,
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<parent>
|
||||
<groupId>org.antlr</groupId>
|
||||
<artifactId>antlr4-master</artifactId>
|
||||
<version>4.6-SNAPSHOT</version>
|
||||
<version>4.6.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<artifactId>antlr4</artifactId>
|
||||
<name>ANTLR 4 Tool</name>
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import org.antlr.v4.runtime.atn.SetTransition;
|
|||
import org.antlr.v4.runtime.atn.Transition;
|
||||
import org.antlr.v4.runtime.misc.Interval;
|
||||
import org.antlr.v4.runtime.misc.IntervalSet;
|
||||
import org.antlr.v4.tool.ErrorType;
|
||||
import org.antlr.v4.tool.Grammar;
|
||||
import org.antlr.v4.tool.Rule;
|
||||
|
||||
|
|
@ -94,20 +95,34 @@ public class ATNOptimizer {
|
|||
Transition matchTransition = decision.transition(j).target.transition(0);
|
||||
if (matchTransition instanceof NotSetTransition) {
|
||||
throw new UnsupportedOperationException("Not yet implemented.");
|
||||
} else {
|
||||
matchSet.addAll(matchTransition.label());
|
||||
}
|
||||
IntervalSet set = matchTransition.label();
|
||||
int minElem = set.getMinElement();
|
||||
int maxElem = set.getMaxElement();
|
||||
for (int k = minElem; k <= maxElem; k++) {
|
||||
if (matchSet.contains(k)) {
|
||||
char setMin = (char) set.getMinElement();
|
||||
char setMax = (char) set.getMaxElement();
|
||||
// TODO: Token is missing (i.e. position in source will not be displayed).
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName,
|
||||
null, (char) minElem + "-" + (char) maxElem, "[" + setMin + "-" + setMax + "]");
|
||||
break;
|
||||
}
|
||||
}
|
||||
matchSet.addAll(set);
|
||||
}
|
||||
|
||||
Transition newTransition;
|
||||
if (matchSet.getIntervals().size() == 1) {
|
||||
if (matchSet.size() == 1) {
|
||||
newTransition = new AtomTransition(blockEndState, matchSet.getMinElement());
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
Interval matchInterval = matchSet.getIntervals().get(0);
|
||||
newTransition = new RangeTransition(blockEndState, matchInterval.a, matchInterval.b);
|
||||
}
|
||||
} else {
|
||||
}
|
||||
else {
|
||||
newTransition = new SetTransition(blockEndState, matchSet);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ import org.antlr.v4.tool.LexerGrammar;
|
|||
import org.antlr.v4.tool.Rule;
|
||||
import org.antlr.v4.tool.ast.ActionAST;
|
||||
import org.antlr.v4.tool.ast.GrammarAST;
|
||||
import org.antlr.v4.tool.ast.RangeAST;
|
||||
import org.antlr.v4.tool.ast.TerminalAST;
|
||||
import org.stringtemplate.v4.ST;
|
||||
import org.stringtemplate.v4.STGroup;
|
||||
|
|
@ -253,6 +254,7 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
ATNState right = newState(b);
|
||||
int t1 = CharSupport.getCharValueFromGrammarCharLiteral(a.getText());
|
||||
int t2 = CharSupport.getCharValueFromGrammarCharLiteral(b.getText());
|
||||
checkRange(a, b, t1, t2);
|
||||
left.addTransition(new RangeTransition(right, t1, t2));
|
||||
a.atnState = left;
|
||||
b.atnState = left;
|
||||
|
|
@ -268,7 +270,10 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
if ( t.getType()==ANTLRParser.RANGE ) {
|
||||
int a = CharSupport.getCharValueFromGrammarCharLiteral(t.getChild(0).getText());
|
||||
int b = CharSupport.getCharValueFromGrammarCharLiteral(t.getChild(1).getText());
|
||||
set.add(a, b);
|
||||
if (checkRange((GrammarAST) t.getChild(0), (GrammarAST) t.getChild(1), a, b)) {
|
||||
checkSetCollision(associatedAST, set, a, b);
|
||||
set.add(a,b);
|
||||
}
|
||||
}
|
||||
else if ( t.getType()==ANTLRParser.LEXER_CHAR_SET ) {
|
||||
set.addAll(getSetFromCharSetLiteral(t));
|
||||
|
|
@ -276,12 +281,12 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
else if ( t.getType()==ANTLRParser.STRING_LITERAL ) {
|
||||
int c = CharSupport.getCharValueFromGrammarCharLiteral(t.getText());
|
||||
if ( c != -1 ) {
|
||||
checkSetCollision(associatedAST, set, c);
|
||||
set.add(c);
|
||||
}
|
||||
else {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, t.getToken(), t.getText());
|
||||
|
||||
}
|
||||
}
|
||||
else if ( t.getType()==ANTLRParser.TOKEN_REF ) {
|
||||
|
|
@ -307,6 +312,27 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
return new Handle(left, right);
|
||||
}
|
||||
|
||||
protected boolean checkRange(GrammarAST leftNode, GrammarAST rightNode, int leftValue, int rightValue) {
|
||||
boolean result = true;
|
||||
if (leftValue == -1) {
|
||||
result = false;
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, leftNode.getToken(), leftNode.getText());
|
||||
}
|
||||
if (rightValue == -1) {
|
||||
result = false;
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_LITERAL_IN_LEXER_SET,
|
||||
g.fileName, rightNode.getToken(), rightNode.getText());
|
||||
}
|
||||
if (!result) return result;
|
||||
|
||||
if (rightValue < leftValue) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, leftNode.parent.getToken(), leftNode.getText() + ".." + rightNode.getText());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** For a lexer, a string is a sequence of char to match. That is,
|
||||
* "fog" is treated as 'f' 'o' 'g' not as a single transition in
|
||||
* the DFA. Machine== o-'f'->o-'o'->o-'g'->o and has n+1 states
|
||||
|
|
@ -315,12 +341,19 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
@Override
|
||||
public Handle stringLiteral(TerminalAST stringLiteralAST) {
|
||||
String chars = stringLiteralAST.getText();
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(chars);
|
||||
int n = chars.length();
|
||||
ATNState left = newState(stringLiteralAST);
|
||||
ATNState right;
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(chars);
|
||||
if (chars == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE,
|
||||
g.fileName, stringLiteralAST.getToken());
|
||||
return new Handle(left, left);
|
||||
}
|
||||
|
||||
int n = chars.length();
|
||||
ATNState prev = left;
|
||||
ATNState right = null;
|
||||
for (int i=0; i<n; i++) {
|
||||
right = null;
|
||||
for (int i = 0; i < n; i++) {
|
||||
right = newState(stringLiteralAST);
|
||||
prev.addTransition(new AtomTransition(right, chars.charAt(i)));
|
||||
prev = right;
|
||||
|
|
@ -342,34 +375,90 @@ public class LexerATNFactory extends ParserATNFactory {
|
|||
|
||||
public IntervalSet getSetFromCharSetLiteral(GrammarAST charSetAST) {
|
||||
String chars = charSetAST.getText();
|
||||
chars = chars.substring(1, chars.length()-1);
|
||||
String cset = '"'+ chars +'"';
|
||||
chars = chars.substring(1, chars.length() - 1);
|
||||
String cset = '"' + chars + '"';
|
||||
IntervalSet set = new IntervalSet();
|
||||
|
||||
if (chars.length() == 0) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, charSetAST.getToken(), "[]");
|
||||
return set;
|
||||
}
|
||||
// unescape all valid escape char like \n, leaving escaped dashes as '\-'
|
||||
// so we can avoid seeing them as '-' range ops.
|
||||
chars = CharSupport.getStringFromGrammarStringLiteral(cset);
|
||||
// now make x-y become set of char
|
||||
if (chars == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE,
|
||||
g.fileName, charSetAST.getToken());
|
||||
return set;
|
||||
}
|
||||
int n = chars.length();
|
||||
for (int i=0; i< n; i++) {
|
||||
// now make x-y become set of char
|
||||
for (int i = 0; i < n; i++) {
|
||||
int c = chars.charAt(i);
|
||||
if ( c=='\\' && (i+1)<n && chars.charAt(i+1)=='-' ) { // \-
|
||||
if (c == '\\' && i+1 < n && chars.charAt(i+1) == '-') { // \-
|
||||
checkSetCollision(charSetAST, set, '-');
|
||||
set.add('-');
|
||||
i++;
|
||||
}
|
||||
else if ( (i+2)<n && chars.charAt(i+1)=='-' ) { // range x-y
|
||||
else if (i+2 < n && chars.charAt(i+1) == '-') { // range x-y
|
||||
int x = c;
|
||||
int y = chars.charAt(i+2);
|
||||
if ( x<=y ) set.add(x,y);
|
||||
i+=2;
|
||||
if (x <= y) {
|
||||
checkSetCollision(charSetAST, set, x, y);
|
||||
set.add(x,y);
|
||||
}
|
||||
else {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED,
|
||||
g.fileName, charSetAST.getToken(), "[" + (char) x + "-" + (char) y + "]");
|
||||
}
|
||||
i += 2;
|
||||
}
|
||||
else {
|
||||
checkSetCollision(charSetAST, set, c);
|
||||
set.add(c);
|
||||
}
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
protected void checkSetCollision(GrammarAST ast, IntervalSet set, int el) {
|
||||
if (set.contains(el)) {
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName, ast.getToken(),
|
||||
(char)el, ast.getText());
|
||||
}
|
||||
}
|
||||
|
||||
protected void checkSetCollision(GrammarAST ast, IntervalSet set, int a, int b) {
|
||||
for (int i = a; i <= b; i++) {
|
||||
if (set.contains(i)) {
|
||||
String setText;
|
||||
if (ast.getChildren() == null) {
|
||||
setText = ast.getText();
|
||||
}
|
||||
else {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (Object child : ast.getChildren()) {
|
||||
if (child instanceof RangeAST) {
|
||||
sb.append(((RangeAST) child).getChild(0).getText());
|
||||
sb.append("..");
|
||||
sb.append(((RangeAST) child).getChild(1).getText());
|
||||
}
|
||||
else {
|
||||
sb.append(((GrammarAST)child).getText());
|
||||
}
|
||||
sb.append(" | ");
|
||||
}
|
||||
sb.replace(sb.length() - 3, sb.length(), "");
|
||||
setText = sb.toString();
|
||||
}
|
||||
g.tool.errMgr.grammarError(ErrorType.CHARACTERS_COLLISION_IN_SET, g.fileName, ast.getToken(),
|
||||
(char)a + "-" + (char)b, setText);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Handle tokenRef(TerminalAST node) {
|
||||
// Ref to EOF in lexer yields char transition on -1
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ import java.util.Map;
|
|||
|
||||
public class LexerFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -export-macro cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public boolean genListener; // from -listener cmd-line
|
||||
public boolean genVisitor; // from -visitor cmd-line
|
||||
@ModelElement public Lexer lexer;
|
||||
|
|
@ -22,7 +22,7 @@ public class LexerFile extends OutputFile {
|
|||
super(factory, fileName);
|
||||
namedActions = buildNamedActions(factory.getGrammar());
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().tool.exportMacro;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
genListener = factory.getGrammar().tool.gen_listener;
|
||||
genVisitor = factory.getGrammar().tool.gen_visitor;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ import java.util.Set;
|
|||
*/
|
||||
public class ListenerFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -export-macro cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public String grammarName;
|
||||
public String parserName;
|
||||
/**
|
||||
|
|
@ -62,6 +62,6 @@ public class ListenerFile extends OutputFile {
|
|||
ActionAST ast = g.namedActions.get("header");
|
||||
if ( ast!=null ) header = new Action(factory, ast);
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().tool.exportMacro;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ import java.util.Map;
|
|||
/** */
|
||||
public class ParserFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro ; // from -export-macro cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public boolean genListener; // from -listener cmd-line
|
||||
public boolean genVisitor; // from -visitor cmd-line
|
||||
@ModelElement public Parser parser;
|
||||
|
|
@ -29,7 +29,7 @@ public class ParserFile extends OutputFile {
|
|||
Grammar g = factory.getGrammar();
|
||||
namedActions = buildNamedActions(factory.getGrammar());
|
||||
genPackage = g.tool.genPackage;
|
||||
exportMacro = g.tool.exportMacro;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
// need the below members in the ST for Python, C++
|
||||
genListener = g.tool.gen_listener;
|
||||
genVisitor = g.tool.gen_visitor;
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ import java.util.Set;
|
|||
|
||||
public class VisitorFile extends OutputFile {
|
||||
public String genPackage; // from -package cmd-line
|
||||
public String exportMacro; // from -export-macro cmd-line
|
||||
public String exportMacro; // from -DexportMacro cmd-line
|
||||
public String grammarName;
|
||||
public String parserName;
|
||||
/**
|
||||
|
|
@ -59,6 +59,6 @@ public class VisitorFile extends OutputFile {
|
|||
ActionAST ast = g.namedActions.get("header");
|
||||
if ( ast!=null ) header = new Action(factory, ast);
|
||||
genPackage = factory.getGrammar().tool.genPackage;
|
||||
exportMacro = factory.getGrammar().tool.exportMacro;
|
||||
exportMacro = factory.getGrammar().getOptionString("exportMacro");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,6 +28,8 @@ public class CharSupport {
|
|||
ANTLRLiteralEscapedCharValue['\\'] = '\\';
|
||||
ANTLRLiteralEscapedCharValue['\''] = '\'';
|
||||
ANTLRLiteralEscapedCharValue['"'] = '"';
|
||||
ANTLRLiteralEscapedCharValue['-'] = '-';
|
||||
ANTLRLiteralEscapedCharValue[']'] = ']';
|
||||
ANTLRLiteralCharValueEscape['\n'] = "\\n";
|
||||
ANTLRLiteralCharValueEscape['\r'] = "\\r";
|
||||
ANTLRLiteralCharValueEscape['\t'] = "\\t";
|
||||
|
|
@ -76,32 +78,6 @@ public class CharSupport {
|
|||
return getCharValueFromCharInGrammarLiteral(literal.substring(1,literal.length()-1));
|
||||
}
|
||||
|
||||
/** Given char x or \t or \u1234 return the char value;
|
||||
* Unnecessary escapes like '\{' yield -1.
|
||||
*/
|
||||
public static int getCharValueFromCharInGrammarLiteral(String cstr) {
|
||||
switch ( cstr.length() ) {
|
||||
case 1 :
|
||||
// 'x'
|
||||
return cstr.charAt(0); // no escape char
|
||||
case 2 :
|
||||
if ( cstr.charAt(0)!='\\' ) return -1;
|
||||
// '\x' (antlr lexer will catch invalid char)
|
||||
if ( Character.isDigit(cstr.charAt(1)) ) return -1;
|
||||
int escChar = cstr.charAt(1);
|
||||
int charVal = ANTLRLiteralEscapedCharValue[escChar];
|
||||
if ( charVal==0 ) return -1;
|
||||
return charVal;
|
||||
case 6 :
|
||||
// '\u1234'
|
||||
if ( !cstr.startsWith("\\u") ) return -1;
|
||||
String unicodeChars = cstr.substring(2, cstr.length());
|
||||
return Integer.parseInt(unicodeChars, 16);
|
||||
default :
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
public static String getStringFromGrammarStringLiteral(String literal) {
|
||||
StringBuilder buf = new StringBuilder();
|
||||
int i = 1; // skip first quote
|
||||
|
|
@ -110,19 +86,60 @@ public class CharSupport {
|
|||
int end = i+1;
|
||||
if ( literal.charAt(i) == '\\' ) {
|
||||
end = i+2;
|
||||
if ( (i+1)>=n ) break; // ignore spurious \ on end
|
||||
if ( literal.charAt(i+1) == 'u' ) end = i+6;
|
||||
if ( i+1 < n && literal.charAt(i+1) == 'u' ) {
|
||||
for (end = i + 2; end < i + 6; end++) {
|
||||
if ( end>n ) return null; // invalid escape sequence.
|
||||
char charAt = literal.charAt(end);
|
||||
if (!Character.isDigit(charAt) && !(charAt >= 'a' && charAt <= 'f') && !(charAt >= 'A' && charAt <= 'F')) {
|
||||
return null; // invalid escape sequence.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if ( end>n ) break;
|
||||
if ( end>n ) return null; // invalid escape sequence.
|
||||
String esc = literal.substring(i, end);
|
||||
int c = getCharValueFromCharInGrammarLiteral(esc);
|
||||
if ( c==-1 ) { buf.append(esc); }
|
||||
if ( c==-1 ) {
|
||||
return null; // invalid escape sequence.
|
||||
}
|
||||
else buf.append((char)c);
|
||||
i = end;
|
||||
}
|
||||
return buf.toString();
|
||||
}
|
||||
|
||||
/** Given char x or \t or \u1234 return the char value;
|
||||
* Unnecessary escapes like '\{' yield -1.
|
||||
*/
|
||||
public static int getCharValueFromCharInGrammarLiteral(String cstr) {
|
||||
switch ( cstr.length() ) {
|
||||
case 1:
|
||||
// 'x'
|
||||
return cstr.charAt(0); // no escape char
|
||||
case 2:
|
||||
if ( cstr.charAt(0)!='\\' ) return -1;
|
||||
// '\x' (antlr lexer will catch invalid char)
|
||||
if ( Character.isDigit(cstr.charAt(1)) ) return -1;
|
||||
int escChar = cstr.charAt(1);
|
||||
int charVal = ANTLRLiteralEscapedCharValue[escChar];
|
||||
if ( charVal==0 ) return -1;
|
||||
return charVal;
|
||||
case 6:
|
||||
// '\u1234'
|
||||
if ( !cstr.startsWith("\\u") ) return -1;
|
||||
String unicodeChars = cstr.substring(2, cstr.length());
|
||||
int result = -1;
|
||||
try {
|
||||
result = Integer.parseInt(unicodeChars, 16);
|
||||
}
|
||||
catch (NumberFormatException e) {
|
||||
}
|
||||
return result;
|
||||
default:
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
public static String capitalize(String s) {
|
||||
return Character.toUpperCase(s.charAt(0)) + s.substring(1);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -471,7 +471,7 @@ public class BasicSemanticChecks extends GrammarTreeVisitor {
|
|||
protected void enterTerminal(GrammarAST tree) {
|
||||
String text = tree.getText();
|
||||
if (text.equals("''")) {
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_NOT_ALLOWED, g.fileName, tree.token);
|
||||
g.tool.errMgr.grammarError(ErrorType.EMPTY_STRINGS_AND_SETS_NOT_ALLOWED, g.fileName, tree.token, "''");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -982,11 +982,15 @@ public enum ErrorType {
|
|||
*
|
||||
* <p>empty strings not allowed</p>
|
||||
*
|
||||
* <pre>A: '''test''';</pre>
|
||||
* <pre>B: '';</pre>
|
||||
* <pre>C: 'test' '';</pre>
|
||||
* <pre>
|
||||
* A: '''test''';
|
||||
* B: '';
|
||||
* C: 'test' '';
|
||||
* D: [];
|
||||
* E: [f-a];
|
||||
* </pre>
|
||||
*/
|
||||
EMPTY_STRINGS_NOT_ALLOWED(174, "string literals cannot be empty", ErrorSeverity.ERROR),
|
||||
EMPTY_STRINGS_AND_SETS_NOT_ALLOWED(174, "string literals and sets cannot be empty: <arg>", ErrorSeverity.ERROR),
|
||||
/**
|
||||
* Compiler Error 175.
|
||||
*
|
||||
|
|
@ -1027,6 +1031,19 @@ public enum ErrorType {
|
|||
* <p>T00: 'a00' -> skip, more;</p>
|
||||
*/
|
||||
INCOMPATIBLE_COMMANDS(179, "incompatible commands <arg> and <arg2>", ErrorSeverity.WARNING),
|
||||
/**
|
||||
* Compiler Warning 180.
|
||||
*
|
||||
* <p>chars "a-f" used multiple times in set [a-fc-m]</p>
|
||||
*
|
||||
* <pre>
|
||||
* A: [aa-z]; // warning
|
||||
* B: [a-fc-m]; // warning
|
||||
* </pre>
|
||||
*
|
||||
* TODO: Does not work with fragment rules.
|
||||
*/
|
||||
CHARACTERS_COLLISION_IN_SET(180, "chars \"<arg>\" used multiple times in set <arg2>", ErrorSeverity.WARNING),
|
||||
|
||||
/*
|
||||
* Backward incompatibility errors
|
||||
|
|
|
|||
|
|
@ -81,6 +81,7 @@ public class Grammar implements AttributeResolver {
|
|||
parserOptions.add("TokenLabelType");
|
||||
parserOptions.add("tokenVocab");
|
||||
parserOptions.add("language");
|
||||
parserOptions.add("exportMacro");
|
||||
}
|
||||
|
||||
public static final Set<String> lexerOptions = parserOptions;
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ package org.antlr.v4.tool.ast;
|
|||
|
||||
import org.antlr.runtime.Token;
|
||||
import org.antlr.v4.misc.CharSupport;
|
||||
import org.antlr.v4.tool.ErrorType;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
|
|
@ -41,6 +42,10 @@ public abstract class GrammarASTWithOptions extends GrammarAST {
|
|||
String v = value.getText();
|
||||
if ( v.startsWith("'") || v.startsWith("\"") ) {
|
||||
v = CharSupport.getStringFromGrammarStringLiteral(v);
|
||||
if (v == null) {
|
||||
g.tool.errMgr.grammarError(ErrorType.INVALID_ESCAPE_SEQUENCE, g.fileName, value.getToken());
|
||||
v = "";
|
||||
}
|
||||
}
|
||||
return v;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue