Restructure C# runtime and fix Issue #2693 (#3057)

* Fixes for https://github.com/antlr/antlr4/issues/2693

* Adding script for comparing Java and C# profile output.

* Update lexer grammar.

Tightening up the grammar rules. Still Java9.

* Tighten up grammar.

* Adding in new base classes for C# runtime tests for profiling.

* Complete test for Profile = true.

Update ProfileDescriptor to now parse and output profile. The grammar is asm8080 from grammars-v4, tightened up. The input is the example provided there, truncated to included fewer lines as that causes a null-ptr crash with the older runtime. I verified by modifying the .csproj in /tmp.

* Restructuring the Antlr C# runtime so that it is consistent with all other runtimes, a source directory (now antlr4/runtime/CSharp/src), and a test directory (antlr4/runtime/CSharp). In the test area, I added a test for profiling in issue-2593. This test requires the Antlr tool and Antlr C# tool to be build. The path is assumed in a relative path to the test, ../../../../tool/target/antlr4-*-SNAPSHOT-complete.jar, with globbing performed. The test simply checks the return result, output not important. There are no changes to the runtime C# source files other than placing them under src/. Several other build files were changed to reflect the new location of the Antlr C# runtime. I updated the instructions for users on how to build the runtime, including information on checking the environment--now explicitly specified here so people know what to install!

.circleci/scripts/install-linux-dotnet.sh

@@ -15,5 +15,5 @@ echo "done installing .Net SDK"
 # we need to build the runtime before test run, since we used "--no-dependencies"
 # when we call dotnet cli for restore and build, in order to speed up
 echo "building runtime..."
-dotnet build -c Release -f netstandard2.0 runtime/CSharp/Antlr4.csproj
+dotnet build -c Release -f netstandard2.0 runtime/CSharp/src/Antlr4.csproj
 echo "done building runtime"

.travis/run-tests-dotnet.sh

@@ -7,7 +7,7 @@ export PATH=$PATH:/Users/travis/.dotnet
 # we need to build the runtime before test run, since we used "--no-dependencies"
 # when we call dotnet cli for restore and build, in order to speed up
 
-dotnet build -c Release -f netstandard2.0 ../runtime/CSharp/Antlr4.csproj
+dotnet build -c Release -f netstandard2.0 ../runtime/CSharp/src/Antlr4.csproj
 
 # call test
 

appveyor.yml

@@ -12,9 +12,9 @@ install:
   - cinst -y dart-sdk --version=2.8.4
 build_script:
   - mvn -DskipTests install --batch-mode
-  - dotnet build runtime/CSharp/Antlr4.csproj -c Release
+  - dotnet build runtime/CSharp/src/Antlr4.csproj -c Release
 after_build:
-  - dotnet pack runtime/CSharp/Antlr4.csproj -c Release
+  - dotnet pack runtime/CSharp/src/Antlr4.csproj -c Release
 test_script:
   - mvn install -Dantlr-php-php="C:\tools\php\php.exe" -Dantlr-dart-dart="C:\tools\dart-sdk\bin\dart.exe" -Dantlr-dart-pub="C:\tools\dart-sdk\bin\pub.bat" -Dantlr-dart-dart2native="C:\tools\dart-sdk\bin\dart2native.bat" -Dantlr-python2-python="C:\Python27\python.exe" -Dantlr-python3-python="C:\Python35\python.exe" --batch-mode
 artifacts:

doc/antlr-project-testing.md

@@ -57,11 +57,9 @@ $ mvn install -DskipTests=true   # make sure all artifacts are visible on this m
 Now, make sure C# runtime is built and installed locally.
 
 ```bash
-cd ~/antlr/code/antlr4/runtime/CSharp/runtime/CSharp
+cd ~/antlr/code/antlr4/runtime/CSharp/src
-# kill previous ones manually as "xbuild /t:Clean" didn't seem to do it
+rm -rf `find . -name '{obj,bin}'`
-find . -name '*.dll' -exec rm {} \;
+dotnet build -c Release runtime/CSharp/src/Antlr4.csproj
-# build
-xbuild /p:Configuration=Release Antlr4.Runtime/Antlr4.Runtime.mono.csproj
 ```
 
 C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.

doc/building-antlr.md

@@ -25,6 +25,22 @@ Checking connectivity... done.
 Checking out files: 100% (1427/1427), done.
 ```
 
+# Check your environment
+
+If you are starting from a clean, minimum Ubuntu OS, check your environment.
+
+
+```bash
+$ sudo apt-get update
+$ # Get Java
+$ java > /dev/null 2>&1
+$ if [[ "$?" != "0" ]]; then sudo apt install -y openjdk-11-jre-headless; fi
+$ # Get Mvn
+$ mvn > /dev/null 2>&1
+$ if [[ "$?" != "0" ]]; then sudo apt install -y maven; fi
+
+```
+
 # Compile
 
 ```bash

doc/releasing-antlr.md

@@ -54,7 +54,7 @@ Edit the repository looking for 4.5 or whatever and update it. Bump version in t
  * runtime/Python2/src/antlr4/Recognizer.py
  * runtime/Python3/setup.py
  * runtime/Python3/src/antlr4/Recognizer.py
- * runtime/CSharp/Antlr4.csproj
+ * runtime/CSharp/src/Antlr4.csproj
  * runtime/PHP/src/RuntimeMetaData.php
  * runtime/JavaScript/package.json
  * runtime/JavaScript/src/antlr4/Recognizer.js
@@ -80,8 +80,8 @@ Here is a simple script to display any line from the critical files with, say, `
 
 ```bash
 mvn clean
-rm -rf runtime/CSharp/bin
+rm -rf runtime/CSharp/src/bin
-rm -rf runtime/CSharp/obj
+rm -rf runtime/CSharp/src/obj
 rm -rf runtime/gen
 find tool runtime -type f -exec grep -l '4\.9' {} \;
 find runtime runtime -type f -exec grep -l '4\.9' {} \;
@@ -322,10 +322,10 @@ Of course you need Mono and `nuget` to be installed. On mac:
 
 From @kvanTTT: Install `dotnet` on any platform (see https://dotnet.microsoft.com/download) and run the following command on any OS (Win, Linux, macOS):
 
-* building: `dotnet build runtime/CSharp/Antlr4.csproj -c Release`
+* building: `dotnet build runtime/CSharp/src/Antlr4.csproj -c Release`
-  Output `.dll` will be in `runtime/CSharp/bin/Release/netstandard2.0` or in `runtime/CSharp/bin/Release/netstandard2.1`
+  Output `.dll` will be in `runtime/CSharp/src/bin/Release/netstandard2.0` or in `runtime/CSharp/src/bin/Release/netstandard2.1`
-* packing: `dotnet pack runtime/CSharp/Antlr4.csproj -c Release`
+* packing: `dotnet pack runtime/CSharp/src/Antlr4.csproj -c Release`
-  Output `.nupkg` will be in `runtime/CSharp/bin/Release/Antlr4.Runtime.Standard.4.9.0.nupkg`
+  Output `.nupkg` will be in `runtime/CSharp/src/bin/Release/Antlr4.Runtime.Standard.4.9.1.nupkg`
 
 Alternatively, you can install Visual Studio 2017 and make sure to check boxes with .NET Core SDK.
 

runtime-testsuite/test/org/antlr/v4/test/runtime/csharp/BaseCSharpTest.java

@@ -195,7 +195,7 @@ public class BaseCSharpTest extends BaseRuntimeTestSupport implements RuntimeTes
 
 			// find runtime package
 			final ClassLoader loader = Thread.currentThread().getContextClassLoader();
-			final URL runtimeProj = loader.getResource("CSharp/Antlr4.csproj");
+			final URL runtimeProj = loader.getResource("CSharp/src/Antlr4.csproj");
 			if (runtimeProj == null) {
 				throw new RuntimeException("C# runtime project file not found!");
 			}

runtime/CSharp/src/Antlr4.csproj

@@ -3,7 +3,7 @@
     <Company>The ANTLR Organization</Company>
     <Version>4.9.1</Version>
     <NeutralLanguage>en-US</NeutralLanguage>
-    <TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>
+    <TargetFramework>netstandard2.0</TargetFramework>
     <NoWarn>$(NoWarn);CS1591;CS1574;CS1580</NoWarn>
     <GenerateDocumentationFile>true</GenerateDocumentationFile>
     <AssemblyName>Antlr4.Runtime.Standard</AssemblyName>

runtime/CSharp/src/Atn/AmbiguityInfo.cs

@@ -41,6 +41,9 @@ namespace Antlr4.Runtime.Atn
     /// <since>4.3</since>
     public class AmbiguityInfo : DecisionEventInfo
     {
+        /** The set of alternative numbers for this decision event that lead to a valid parse. */
+        public BitSet ambigAlts;
+
         /// <summary>
         /// Constructs a new instance of the
         /// <see cref="AmbiguityInfo"/>
@@ -48,19 +51,30 @@ namespace Antlr4.Runtime.Atn
         /// specified detailed ambiguity information.
         /// </summary>
         /// <param name="decision">The decision number</param>
-        /// <param name="state">
+        /// <param name="configs">The final configuration set identifying the ambiguous
-        /// The final simulator state identifying the ambiguous
         /// alternatives for the current input
         /// </param>
+        /// <param name="ambigAlts">The set of alternatives in the decision that lead to a valid parse.
+        /// The predicted alt is the min(ambigAlts)
+        /// </param>
         /// <param name="input">The input token stream</param>
         /// <param name="startIndex">The start index for the current prediction</param>
         /// <param name="stopIndex">
         /// The index at which the ambiguity was identified during
         /// prediction
         /// </param>
-        public AmbiguityInfo(int decision, SimulatorState state, ITokenStream input, int startIndex, int stopIndex)
+        /// <param name="fullCtx">@code true} if the ambiguity was identified during LL
-            : base(decision, state, input, startIndex, stopIndex, state.useContext)
+        /// prediction; otherwise, {@code false} if the ambiguity was identified
+        /// during SLL prediction
+        /// </param>
+        public AmbiguityInfo(int decision,
+                 ATNConfigSet configs,
+                 BitSet ambigAlts,
+                 ITokenStream input, int startIndex, int stopIndex,
+                 bool fullCtx)
+        : base(decision, configs, input, startIndex, stopIndex, fullCtx)
         {
+            this.ambigAlts = ambigAlts;
         }
     }
 }

runtime/CSharp/src/Atn/ContextSensitivityInfo.cs

@@ -35,9 +35,8 @@ namespace Antlr4.Runtime.Atn
         /// with the specified detailed context sensitivity information.
         /// </summary>
         /// <param name="decision">The decision number</param>
-        /// <param name="state">
+        /// <param name="configs">The final configuration set identifying the ambiguous
-        /// The final simulator state containing the unique
+        /// alternatives for the current input
-        /// alternative identified by full-context prediction
         /// </param>
         /// <param name="input">The input token stream</param>
         /// <param name="startIndex">The start index for the current prediction</param>
@@ -45,8 +44,8 @@ namespace Antlr4.Runtime.Atn
         /// The index at which the context sensitivity was
         /// identified during full-context prediction
         /// </param>
-        public ContextSensitivityInfo(int decision, SimulatorState state, ITokenStream input, int startIndex, int stopIndex)
+        public ContextSensitivityInfo(int decision, ATNConfigSet configs, ITokenStream input, int startIndex, int stopIndex)
-            : base(decision, state, input, startIndex, stopIndex, true)
+            : base(decision, configs, input, startIndex, stopIndex, true)
         {
         }
     }

runtime/CSharp/src/Atn/DecisionEventInfo.cs

@@ -25,15 +25,13 @@ namespace Antlr4.Runtime.Atn
         /// <seealso cref="ATN.decisionToState"/>
         public readonly int decision;
 
-        /// <summary>
+        /// <summary>The configuration set containing additional information relevant to the
-        /// The simulator state containing additional information relevant to the
+        /// prediction state when the current event occurred, or {@code null} if no
-        /// prediction state when the current event occurred, or
+        /// additional information is relevant or available.</summary>
-        /// <see langword="null"/>
+        /// <remarks>The configuration set containing additional information relevant to the
-        /// if no
+        /// prediction state when the current event occurred, or {@code null} if no
-        /// additional information is relevant or available.
+        /// additional information is relevant or available.</remarks>
-        /// </summary>
+        public readonly ATNConfigSet configs;
-        [Nullable]
-        public readonly SimulatorState state;
 
         /// <summary>The input token stream which is being parsed.</summary>
         /// <remarks>The input token stream which is being parsed.</remarks>
@@ -63,14 +61,17 @@ namespace Antlr4.Runtime.Atn
         /// </summary>
         public readonly bool fullCtx;
 
-        public DecisionEventInfo(int decision, SimulatorState state, ITokenStream input, int startIndex, int stopIndex, bool fullCtx)
+        public DecisionEventInfo(int decision,
+            ATNConfigSet configs,
+            ITokenStream input, int startIndex, int stopIndex,
+            bool fullCtx)
         {
             this.decision = decision;
             this.fullCtx = fullCtx;
             this.stopIndex = stopIndex;
             this.input = input;
             this.startIndex = startIndex;
-            this.state = state;
+            this.configs = configs;
         }
     }
 }

runtime/CSharp/src/Atn/ErrorInfo.cs

@@ -30,17 +30,18 @@ namespace Antlr4.Runtime.Atn
         /// specified detailed syntax error information.
         /// </summary>
         /// <param name="decision">The decision number</param>
-        /// <param name="state">
+        /// <param name="configs">The final configuration set reached during prediction
-        /// The final simulator state reached during prediction
+        /// prior to reaching the {@link ATNSimulator#ERROR} state
-        /// prior to reaching the
-        /// <see cref="ATNSimulator.ERROR"/>
-        /// state
         /// </param>
         /// <param name="input">The input token stream</param>
         /// <param name="startIndex">The start index for the current prediction</param>
         /// <param name="stopIndex">The index at which the syntax error was identified</param>
-        public ErrorInfo(int decision, SimulatorState state, ITokenStream input, int startIndex, int stopIndex)
+        /// <param name="fullCtx">{@code true} if the syntax error was identified during LL
-            : base(decision, state, input, startIndex, stopIndex, state.useContext)
+        /// prediction; otherwise, {@code false} if the syntax error was identified
+        /// during SLL prediction
+        /// </param>
+        public ErrorInfo(int decision, ATNConfigSet configs, ITokenStream input, int startIndex, int stopIndex, bool fullCtx)
+            : base(decision, configs, input, startIndex, stopIndex, fullCtx)
         {
         }
     }

runtime/CSharp/src/Atn/LookaheadEventInfo.cs

@@ -19,6 +19,13 @@ namespace Antlr4.Runtime.Atn
     /// <since>4.3</since>
     public class LookaheadEventInfo : DecisionEventInfo
     {
+        /// <summary>The alternative chosen by adaptivePredict(), not necessarily
+        ///  the outermost alt shown for a rule; left-recursive rules have
+        ///  user-level alts that differ from the rewritten rule with a (...) block
+        ///  and a (..)* loop.
+        /// </summary>
+        public int predictedAlt;
+
         /// <summary>
         /// Constructs a new instance of the
         /// <see cref="LookaheadEventInfo"/>
@@ -26,18 +33,15 @@ namespace Antlr4.Runtime.Atn
         /// the specified detailed lookahead information.
         /// </summary>
         /// <param name="decision">The decision number</param>
-        /// <param name="state">
+        /// <param name="configs">The final configuration set containing the necessary
-        /// The final simulator state containing the necessary
+        /// information to determine the result of a prediction, or {@code null} if
-        /// information to determine the result of a prediction, or
+        /// the final configuration set is not available
-        /// <see langword="null"/>
-        /// if
-        /// the final state is not available
         /// </param>
+        /// <param name="predictedAlt"></param>
         /// <param name="input">The input token stream</param>
         /// <param name="startIndex">The start index for the current prediction</param>
         /// <param name="stopIndex">The index at which the prediction was finally made</param>
         /// <param name="fullCtx">
-        ///
         /// <see langword="true"/>
         /// if the current lookahead is part of an LL
         /// prediction; otherwise,
@@ -45,9 +49,10 @@ namespace Antlr4.Runtime.Atn
         /// if the current lookahead is part of
         /// an SLL prediction
         /// </param>
-        public LookaheadEventInfo(int decision, SimulatorState state, ITokenStream input, int startIndex, int stopIndex, bool fullCtx)
+        public LookaheadEventInfo(int decision, ATNConfigSet configs, int predictedAlt, ITokenStream input, int startIndex, int stopIndex, bool fullCtx)
-            : base(decision, state, input, startIndex, stopIndex, fullCtx)
+            : base(decision, configs, input, startIndex, stopIndex, fullCtx)
         {
+            this.predictedAlt = predictedAlt;
         }
     }
 }

runtime/CSharp/src/Atn/PredicateEvalInfo.cs

@@ -49,7 +49,6 @@ namespace Antlr4.Runtime.Atn
         /// class with the
         /// specified detailed predicate evaluation information.
         /// </summary>
-        /// <param name="state">The simulator state</param>
         /// <param name="decision">The decision number</param>
         /// <param name="input">The input token stream</param>
         /// <param name="startIndex">The start index for the current prediction</param>
@@ -68,10 +67,15 @@ namespace Antlr4.Runtime.Atn
         /// <see cref="predictedAlt"/>
         /// for more information.
         /// </param>
+        /// <param name="fullCtx">{@code true} if the semantic context was
+        /// evaluated during LL prediction; otherwise, {@code false} if the semantic
+        /// context was evaluated during SLL prediction
+        /// </param>
+        ///
         /// <seealso cref="ParserATNSimulator.EvalSemanticContext(SemanticContext, ParserRuleContext, int, bool)"/>
         /// <seealso cref="SemanticContext.Eval"/>
-        public PredicateEvalInfo(SimulatorState state, int decision, ITokenStream input, int startIndex, int stopIndex, SemanticContext semctx, bool evalResult, int predictedAlt)
+        public PredicateEvalInfo(int decision, ITokenStream input, int startIndex, int stopIndex, SemanticContext semctx, bool evalResult, int predictedAlt, bool fullCtx)
-            : base(decision, state, input, startIndex, stopIndex, state.useContext)
+            : base(decision, new ATNConfigSet(), input, startIndex, stopIndex, fullCtx)
         {
             this.semctx = semctx;
             this.evalResult = evalResult;

runtime/CSharp/src/Atn/ProfilingATNSimulator.cs

@@ -71,7 +71,7 @@ namespace Antlr4.Runtime.Atn
 			{
 				decisions[decision].SLL_MaxLook = SLL_k;
 				decisions[decision].SLL_MaxLookEvent =
-						new LookaheadEventInfo(decision, null/*, alt*/, input, startIndex, sllStopIndex, false);
+						new LookaheadEventInfo(decision, null, alt, input, startIndex, sllStopIndex, false);
 			}
 
 			if (llStopIndex >= 0)
@@ -83,7 +83,7 @@ namespace Antlr4.Runtime.Atn
 				{
 					decisions[decision].LL_MaxLook = LL_k;
 					decisions[decision].LL_MaxLookEvent =
-							new LookaheadEventInfo(decision, null/*, alt*/, input, startIndex, llStopIndex, true);
+							new LookaheadEventInfo(decision, null, alt, input, startIndex, llStopIndex, true);
 				}
 			}
 
@@ -108,7 +108,7 @@ namespace Antlr4.Runtime.Atn
 			if (existingTargetState == ERROR)
 			{
 				decisions[currentDecision].errors.Add(
-						new ErrorInfo(currentDecision, null /*previousD.configs*/, input, startIndex, sllStopIndex)
+						new ErrorInfo(currentDecision, previousD.configSet, input, startIndex, sllStopIndex, false)
 				);
 			}
 		}
@@ -143,7 +143,7 @@ namespace Antlr4.Runtime.Atn
 			else { // no reach on current lookahead symbol. ERROR.
 				   // TODO: does not handle delayed errors per getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule()
 				decisions[currentDecision].errors.Add(
-					new ErrorInfo(currentDecision, null /*closure*/, input, startIndex, llStopIndex)
+					new ErrorInfo(currentDecision, closure, input, startIndex, llStopIndex, true)
 				);
 			}
 		}
@@ -154,7 +154,7 @@ namespace Antlr4.Runtime.Atn
 			}
 			else { // no reach on current lookahead symbol. ERROR.
 				decisions[currentDecision].errors.Add(
-					new ErrorInfo(currentDecision, null /*closure*/, input, startIndex, sllStopIndex)
+					new ErrorInfo(currentDecision, closure, input, startIndex, sllStopIndex, false)
 				);
 			}
 		}
@@ -168,7 +168,7 @@ namespace Antlr4.Runtime.Atn
 			bool fullContext = llStopIndex >= 0;
 			int stopIndex = fullContext ? llStopIndex : sllStopIndex;
 			decisions[currentDecision].predicateEvals.Add(
-				new PredicateEvalInfo(null , currentDecision, input, startIndex, stopIndex, pred, result, alt/*, fullCtx*/)
+				new PredicateEvalInfo(currentDecision, input, startIndex, stopIndex, pred, result, alt, fullCtx)
 			);
 		}
 
@@ -193,14 +193,14 @@ namespace Antlr4.Runtime.Atn
 		if (prediction != conflictingAltResolvedBySLL)
 		{
 			decisions[currentDecision].contextSensitivities.Add(
-					new ContextSensitivityInfo(currentDecision, null /*configs*/, input, startIndex, stopIndex)
+					new ContextSensitivityInfo(currentDecision, configs, input, startIndex, stopIndex)
 			);
 		}
 			base.ReportContextSensitivity(dfa, prediction, configs, startIndex, stopIndex);
 	}
 
 	protected override void ReportAmbiguity(DFA dfa, DFAState D, int startIndex, int stopIndex, bool exact,
-		                                    BitSet ambigAlts, ATNConfigSet configSet)
+		                                    BitSet ambigAlts, ATNConfigSet configs)
 	{
 		int prediction;
 		if (ambigAlts != null)
@@ -208,22 +208,22 @@ namespace Antlr4.Runtime.Atn
 			prediction = ambigAlts.NextSetBit(0);
 		}
 		else {
-				prediction = configSet.GetAlts().NextSetBit(0);
+				prediction = configs.GetAlts().NextSetBit(0);
 		}
-			if (configSet.fullCtx && prediction != conflictingAltResolvedBySLL)
+			if (configs.fullCtx && prediction != conflictingAltResolvedBySLL)
 		{
 			// Even though this is an ambiguity we are reporting, we can
 			// still detect some context sensitivities.  Both SLL and LL
 			// are showing a conflict, hence an ambiguity, but if they resolve
 			// to different minimum alternatives we have also identified a
 			// context sensitivity.
-			decisions[currentDecision].contextSensitivities.Add( new ContextSensitivityInfo(currentDecision, null /*configs*/, input, startIndex, stopIndex) );
+			decisions[currentDecision].contextSensitivities.Add( new ContextSensitivityInfo(currentDecision, configs, input, startIndex, stopIndex) );
 		}
 		decisions[currentDecision].ambiguities.Add(
-			new AmbiguityInfo(currentDecision, null /*configs, ambigAlts*/,
+			new AmbiguityInfo(currentDecision, configs, ambigAlts,
-				              input, startIndex, stopIndex/*, configs.IsFullContext*/)
+				              input, startIndex, stopIndex, configs.fullCtx)
 		);
-		base.ReportAmbiguity(dfa, D, startIndex, stopIndex, exact, ambigAlts, configSet);
+		base.ReportAmbiguity(dfa, D, startIndex, stopIndex, exact, ambigAlts, configs);
 	}
 
 	// ---------------------------------------------------------------------