meta-parser-generator 1.0.4 → 1.1.0

package/README.md

@@ -4,26 +4,43 @@
 npm install meta-parser-generator
 ```
 
-Meta Parser Generator will help you generate an efficient parser using a grammar and a token definition.
-Meta programming is used to generate a single self contained parser file.
-
-This code has been extracted from https://github.com/batiste/blop-language
+Meta Parser Generator will help you generate an efficient parser using grammar and a token definition.
+Meta programming is used to generate a single self-contained parser file.
 
 ## Characteristics
 
-  * LL parser (Left to Right parser), arbitrary look ahead
-  * Direct Left recursion support (no indirect)
+  * PEG parser (Parsing Expression Grammar) with ordered choice
+  * Packrat parsing with memoization for linear time complexity
+  * Direct left recursion support using Guido van Rossum's algorithm
   * Parser code is generated from a grammar
-  * Good parsing performance (provided your grammar is efficient)
-  * Decent error reporting on parsing error
-  * Memoization
-  * Small source code (~500 lines of code), no dependencies
+  * Good parsing performance (O(n) with memoization)
+  * Excellent error reporting with context
+  * Small source code (~600 lines of code), no dependencies
+  
+### Important: Grammar Order Matters
+
+Unlike LL or LR parsers, PEG parsers use **ordered choice**. The first matching alternative is selected, and no backtracking occurs across alternatives. This means:
+
+```javascript
+// This grammar will NEVER match 'number' because 'name' matches first!
+'VALUE': [
+  ['name'],    // matches ANY identifier including '123abc'
+  ['number'],  // NEVER reached if name is defined as /^[\w]+/
+]
+
+// Correct order: more specific rules first
+'VALUE': [
+  ['number'],  // try number first
+  ['name'],    // then try name
+]
+```
   
 ## How to generate and use a parser
 
 This will generate a mathematical operation parser
   
 ```javascript
+// generator.js
 const { generateParser } = require('meta-parser-generator');
 const path = require('path');
 
@@ -53,7 +70,13 @@ const grammar = {
     ['number'],
   ],
 };
+```
+
+Then execute this script `node generate.js`
 
+```javascript
+// generate.js
+const { tokensDefinition, grammar } = require('./generator');
 // this generate the executable parser file
 generateParser(grammar, tokensDefinition, path.resolve(__dirname, './parser.js'));
 console.log('parser generated');
@@ -79,41 +102,49 @@ let ast = parse('9+10-190.3');
 console.log(ast)
 ```
 
-### How does generated parser works?
+### How does the generated parser work?
+
+Each grammar rule you write is transformed into a function, and those grammar functions call each other until the input parsing is successful. The parser uses:
 
-Each grammar rule you write is transformed into a function, and those grammar functions call each other until the input parsing is sucessful. Therefor the JavaScript call stack is used by the generated parser. So if you design a very recursive grammar, you might trigger a "Maximum call stack size exceeded" error for a large input. 
+1. **PEG Ordered Choice**: For each rule with multiple alternatives, tries them in order and returns the first match
+2. **Packrat Parsing**: Memoization prevents re-parsing the same position, guaranteeing O(n) time complexity
+3. **Left Recursion Handling**: Uses a special memoization strategy based on Guido van Rossum's algorithm
 
-In our example `MATH` grammar rule above you have a left recursion. It means you can parse expressions such as 1+2+3+4+5+...X, where X is the maximum stack size of V8.
+The JavaScript call stack is used by the generated parser. So, if you design a very recursive grammar, you might trigger a "Maximum call stack size exceeded" error for a large input.
 
-To know the default maximum stack size of V8 you can run `node --v8-options | grep stack-size`. If the default size is not enough for your grammar, use this option to extend the size. You can also try to rewrite your grammar in order to be less recursive.
+In our example, the `MATH` grammar rule has left recursion, allowing you to parse expressions like 1+2+3+4+5+...X, where X is limited by V8's stack size.
 
-Anything that can be handled by a modifier rather than recursion will not use the call stack and should be preffered.
+To find out the default maximum stack size of V8, run `node --v8-options | grep stack-size`. If the default size is not enough, you can extend it or rewrite your grammar.
+
+**Best practice**: Use modifiers (`*`, `+`, `?`) instead of recursion when possible - they don't use the call stack and handle large inputs better.
+
+**Note**: For very large files, the memoization cache can grow significantly. The parser clears the cache between parse calls, but memory usage during parsing is proportional to input size × grammar complexity.
 
 ### AST interface
 
 ```typescript
 type ASTNode = RuleNode | Token
 
-interface RuleNode {
-    stream_index: number                // position of the first token for this rule in the token stream
-    type: str                           // name of the rule
-    subRule: number                     // index of this grammar rule in the subrule array
-    children: [ASTNode]                 // list of children
-    named: { [key: string]: ASTNode; }  // named elements withing this rule, see named aliases 
+export interface RuleNode {
+  stream_index: number                // position of the first token for this rule in the token stream
+  type: string                        // name of the rule
+  sub_rule_index: number              // index of this grammar rule in the sub_rule_index array
+  children: [ASTNode]                 // list of children
+  named: { [key: string]: ASTNode; }  // named elements withing this rule, see named aliases 
 }
 ```
 
 At the leaf of the AST you will find the final tokens. They have a slightly different interface
 
 ```typescript
-interface Token {
-    stream_index: number // position of the token in the token stream
-    type: str            // name of token
-    value: str           // the value of the token
-    len: number          // shortcut for value.length
-    lineStart: number    // line start position in the input
-    columnStart: number  // column start position in the input
-    start: number        // character start position in the input 
+export interface Token {
+  stream_index: number // position of the token in the token stream
+  type: string         // name of token
+  value: string        // the value of the token
+  len: number          // shortcut for value.length
+  line_start: number   // line start position in the input
+  column_start: number // column start position in the input
+  start: number        // character start position in the input 
 }
 ```
 
@@ -139,8 +170,9 @@ To facilitate your work with the AST, you can name a rule or a token using a col
 
 ```typescript
 'MATH': [
-  ['MATH', 'math_operator:operator', 'number:num'], // math_operator and number token are named
-  ['number:num'],                                   // here only number is named
+  ['MATH', 'math_operator:operator', 'number:num'], // tokens math_operator and number
+                                                    // are named with operator and num
+  ['number:num'],                                   // here only number is named with num
 ]
 ```
 
@@ -168,3 +200,7 @@ The util function `displayError` will display detailed informations about a toke
 is based on the first grammar rule found that consume the most token from the stream.
 
 <img src="/error.png" width="800">
+
+## Projects using this parser
+
+* The Blop language https://github.com/batiste/blop-language

package/dist/metaParserGenerator.d.ts

@@ -0,0 +1,22 @@
+import { TokensDefinition, Grammar } from './types';
+/**
+ * Generates the tokenizer function code as an array of strings
+ * @param tokenDef - Token definitions mapping token names to patterns
+ * @returns Array of code lines for the tokenizer
+ */
+export declare function generateTokenizer(tokenDef: TokensDefinition): string[];
+/**
+ * Generates the complete parser code from grammar and token definitions
+ * @param grammar - Grammar rules defining the language structure
+ * @param tokensDef - Token definitions mapping token names to patterns
+ * @param debug - Whether to include debug logging in generated code
+ * @returns Array of code lines for the complete parser
+ */
+export declare function generate(grammar: Grammar, tokensDef: TokensDefinition, debug: boolean): string[];
+/**
+ * Generates a parser file from grammar and token definitions
+ * @param grammar - Grammar rules defining the language structure
+ * @param tokensDefinition - Token definitions mapping token names to patterns
+ * @param filename - Output path for the generated parser file
+ */
+export declare function generateParser(grammar: Grammar, tokensDefinition: TokensDefinition, filename: string): void;

package/dist/metaParserGenerator.js

@@ -0,0 +1,377 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || function (mod) {
+    if (mod && mod.__esModule) return mod;
+    var result = {};
+    if (mod != null) for (var k in mod) if (k !== "default" && Object.prototype.hasOwnProperty.call(mod, k)) __createBinding(result, mod, k);
+    __setModuleDefault(result, mod);
+    return result;
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.generateParser = exports.generate = exports.generateTokenizer = void 0;
+const fs = __importStar(require("fs"));
+const utils_1 = require("./utils");
+const recordFailure = `
+let best_failure;
+let best_failure_array = [];
+let best_failure_index = 0;
+
+// Records parsing failures at the deepest position reached
+// Collects all failures at the same position to potentially show "expected: X, Y, or Z"
+function record_failure(failure, i) {
+  // New deepest position reached - reset tracking
+  if (i > best_failure_index) {
+    best_failure_array = [];
+    best_failure = null;
+    best_failure_index = i;
+  }
+  // Record this failure
+  best_failure_array.push(failure);
+  // Keep first failure as primary for error messages
+  if (!best_failure) {
+    best_failure = failure;
+  }
+}
+
+
+// Memoization cache for regular rules
+// Note: For very large inputs, this cache can grow to O(n * m) where:
+// n = input size, m = number of grammar rules
+// Cache is cleared between parse() calls to prevent memory leaks
+let cache = {};
+
+function memoize(name, func) {
+  return function memoize_inner(stream, index) {
+    const key = \`\${name}-\${index}\`;
+    let value = cache[key];
+    if (value !== undefined) {
+      return value;
+    }
+    value = func(stream, index);
+    cache[key] = value;
+    return value;
+  };
+}
+
+// Separate cache for left-recursive rules
+let cacheR = {};
+
+// based on https://medium.com/@gvanrossum_83706/left-recursive-peg-grammars-65dab3c580e1
+function memoize_left_recur(name, func) {
+  return function memoize_inner(stream, index) {
+    const key = \`\${name}-\${index}\`;
+    let value = cacheR[key];
+    if (value !== undefined) {
+      return value;
+    }
+    // prime this rule with a failure
+    cacheR[key] = false;
+    let lastpos;
+    let lastvalue = value;
+    while (true) {
+      value = func(stream, index);
+      if (!value) break;
+      if (value.last_index <= lastpos) break;
+      lastpos = value.last_index;
+      lastvalue = value;
+      cacheR[key] = value;
+    }
+    return lastvalue;
+  };
+}
+
+`;
+/**
+ * Generates the tokenizer function code as an array of strings
+ * @param tokenDef - Token definitions mapping token names to patterns
+ * @returns Array of code lines for the tokenizer
+ */
+function generateTokenizer(tokenDef) {
+    const output = [];
+    const keys = Object.keys(tokenDef);
+    for (let i = 0; i < keys.length; i++) {
+        const key = keys[i];
+        if ((/:|\?/g).test(key)) {
+            throw new Error('Reserved word in token name');
+        }
+    }
+    output.push('function _tokenize(tokenDef, input, char, stream) {');
+    output.push('  let match;');
+    let key;
+    for (let i = 0; i < keys.length; i++) {
+        key = keys[i];
+        const token = tokenDef[key];
+        if (token.str) {
+            const strLen = token.str.length;
+            if (token.str.indexOf("'") > -1 || token.str.indexOf('\n') > -1) {
+                output.push(`  if (input.substr(char, ${strLen}) === \`${token.str}\`) {`);
+                output.push(`    return [\`${token.str}\`, '${key}'];`);
+            }
+            else {
+                output.push(`  if (input.substr(char, ${strLen}) === '${token.str}') {`);
+                output.push(`    return ['${token.str}', '${key}'];`);
+            }
+            output.push('  }');
+        }
+        else if (token.reg) {
+            output.push(`  match = input.substring(char).match(tokenDef.${key}.reg);`);
+            output.push('  if (match !== null) {');
+            output.push(`    return [match[0], '${key}'];`);
+            output.push('  }');
+        }
+        else if (token.func) {
+            output.push(`  match = tokenDef.${key}.func(input.substring(char), stream);`);
+            output.push('  if (match !== undefined) {');
+            output.push(`    return [match, '${key}'];`);
+            output.push('  }');
+        }
+        else {
+            throw new Error(`Tokenizer error: Invalid token ${key} without a reg, str or func property`);
+        }
+    }
+    output.push(`  return [null, '${key}'];`);
+    output.push('}');
+    output.push('function tokenize(tokenDef, input) {');
+    output.push(`  const stream = [];
+  const originalInput = input;
+  let lastToken;
+  let key;
+  let candidate = null;
+  const len = input.length;
+  let char = 0;
+  let index = 0;
+  let line = 0;
+  let column = 0;
+  while (char < len) {
+    [candidate, key] = _tokenize(tokenDef, originalInput, char, stream);
+    if (candidate !== null) {
+      const candidateLen = candidate.length;
+      lastToken = {
+        type: key,
+        value: candidate,
+        start: char,
+        stream_index: index,
+        len: candidateLen,
+        line_start: line,
+        column_start: column,
+      };
+      // Only split if there might be newlines (optimization)
+      if (candidate.indexOf('\\n') !== -1) {
+        const lines = candidate.split('\\n');
+        line += lines.length - 1;
+        column = lines[lines.length - 1].length;
+      } else {
+        column += candidateLen;
+      }
+      lastToken.lineEnd = line;
+      lastToken.columnEnd = column;
+      stream.push(lastToken);
+      index++;
+      char += candidateLen;
+    } else {
+      if (stream.length === 0) {
+        throw new Error('Tokenizer error: total match failure');
+      }
+      if (lastToken) {
+        lastToken.pointer += lastToken.len;
+      }
+      let msg = \`Tokenizer error, no matching token found for \${originalInput.slice(char, char + 26)}\`;
+      if (lastToken) {
+        msg += \` After token of type \${lastToken.type}: \${lastToken.value}\`;
+      }
+      const error = new Error(msg);
+      error.token = lastToken;
+      throw error;
+    }
+  }
+  stream.push({
+    type: 'EOS', value: '<End Of Stream>', char, index,
+  });
+  return stream;
+}
+`);
+    return output;
+}
+exports.generateTokenizer = generateTokenizer;
+/**
+ * Generates code for a specific grammar rule alternative
+ * @param name - Name of the grammar rule
+ * @param index - Index of this alternative in the rule
+ * @param ruleItems - Array of rule items (tokens and sub-rules) in this alternative
+ * @param tokensDef - Token definitions
+ * @param debug - Whether to include debug logging
+ * @returns Array of code lines for this rule function
+ */
+function generatesub_rule_index(name, index, ruleItems, tokensDef, debug) {
+    const output = [];
+    output.push(`let ${name}_${index} = (stream, index) => {`);
+    let i = 0;
+    output.push('  let i = index;');
+    output.push('  const children = [];');
+    output.push('  const named = {};');
+    output.push(`  const node = {
+    children, stream_index: index, name: '${name}',
+    sub_rule_index: ${index}, type: '${name}', named,
+  };`);
+    ruleItems.forEach((rule) => {
+        // terminal rule
+        if (tokensDef[rule.value] || rule.value === 'EOS') {
+            debug ? output.push('  console.log(i, stream[i])') : null;
+            if (rule.repeatable) {
+                output.push(`  while(stream[i].type === '${rule.value}') {`);
+                if (rule.alias) {
+                    output.push(`    named['${rule.alias}'] ? null : named['${rule.alias}'] = []`);
+                    output.push(`    named['${rule.alias}'].push(stream[i])`);
+                }
+                output.push('    children.push(stream[i]); i++;');
+                output.push('  }');
+            }
+            else if (rule.optional) {
+                output.push(`  if (stream[i].type === '${rule.value}') {`);
+                rule.alias ? output.push(`    named['${rule.alias}'] = stream[i];`) : null;
+                output.push('    children.push(stream[i]); i++;');
+                output.push('  }');
+            }
+            else {
+                output.push(`
+  if (stream[i].type !== '${rule.value}') {
+    if (i >= best_failure_index) {
+      const failure = {
+        type: '${name}', sub_rule_index: ${index},
+        sub_rule_stream_index: i - index, sub_rule_token_index: ${i},
+        stream_index: i, token: stream[i], first_token: stream[index], success: false,
+      };
+      record_failure(failure, i);
+    }
+    return false;
+  }
+`);
+                rule.alias ? output.push(`  named['${rule.alias}'] = stream[i];`) : null;
+                output.push('  children.push(stream[i]); i++;');
+            }
+            i++;
+            // calling another rule in the grammar
+        }
+        else {
+            if (rule.function) {
+                output.push(`  if (!(${rule.value})(node)) { return false; }`);
+            }
+            else if (rule.repeatable) {
+                output.push(`  let _rule_${i} = ${rule.value}(stream, i);`); // doing the call
+                output.push(`  while (_rule_${i}) {`);
+                if (rule.alias) {
+                    output.push(`    named['${rule.alias}'] ? null : named['${rule.alias}'] = [];`);
+                    output.push(`    named['${rule.alias}'].push(_rule_${i});`);
+                }
+                output.push(`    children.push(_rule_${i});`);
+                output.push(`    i = _rule_${i}.last_index;`);
+                output.push(`    _rule_${i} = ${rule.value}(stream, i);`);
+                output.push('  }');
+            }
+            else if (!rule.optional) {
+                output.push(`  const _rule_${i} = ${rule.value}(stream, i);`); // doing the call
+                output.push(`  if (!_rule_${i}) return false;`);
+                rule.alias ? output.push(`  named['${rule.alias}'] = _rule_${i};`) : null;
+                output.push(`  children.push(_rule_${i});`);
+                output.push(`  i = _rule_${i}.last_index;`);
+            }
+            else {
+                output.push(`  const _rule_${i} = ${rule.value}(stream, i);`); // doing the call
+                output.push(`  if (_rule_${i}) {`);
+                output.push(`    children.push(_rule_${i});`);
+                rule.alias ? output.push(`    named['${rule.alias}'] = _rule_${i};`) : null;
+                output.push(`    i = _rule_${i}.last_index;`);
+                output.push('  }');
+            }
+            i++;
+        }
+    });
+    output.push('  node.success = i === stream.length; node.last_index = i;');
+    output.push('  return node;');
+    output.push('};');
+    if (ruleItems[0].leftRecursion) {
+        output.push(`${name}_${index} = memoize_left_recur('${name}_${index}', ${name}_${index});`);
+    }
+    else {
+        output.push(`${name}_${index} = memoize('${name}_${index}', ${name}_${index});`);
+    }
+    output.push('\n');
+    return output;
+}
+/**
+ * Generates the complete parser code from grammar and token definitions
+ * @param grammar - Grammar rules defining the language structure
+ * @param tokensDef - Token definitions mapping token names to patterns
+ * @param debug - Whether to include debug logging in generated code
+ * @returns Array of code lines for the complete parser
+ */
+function generate(grammar, tokensDef, debug) {
+    let output = [];
+    (0, utils_1.checkGrammarAndTokens)(grammar, tokensDef);
+    const newGrammar = (0, utils_1.preprocessGrammar)(grammar);
+    const entries = Object.keys(newGrammar);
+    output.push('// This code is automatically generated by the meta parser, do not modify');
+    output.push('// produced with metaParserGenerator.js');
+    output.push(recordFailure);
+    entries.forEach((key) => {
+        let i = 0;
+        const metaSub = [];
+        newGrammar[key].forEach((ruleItems) => {
+            output = output.concat(generatesub_rule_index(key, i, ruleItems, tokensDef, debug));
+            metaSub.push(`${key}_${i}`);
+            i++;
+        });
+        output.push(`function ${key}(stream, index) {`);
+        const st = metaSub.map(sub => `${sub}(stream, index)`).join('\n    || ');
+        output.push(`  return ${st};`);
+        output.push('}');
+    });
+    output = output.concat(generateTokenizer(tokensDef));
+    output.push(`module.exports = {
+  parse: (stream) => {
+    best_failure = null;
+    best_failure_index = 0;
+    best_failure_array = [];
+    cache = {};
+    cacheR = {};
+    const result = START(stream, 0);
+    if (!result) {
+      return {
+        ...best_failure,
+        best_failure_array, 
+      }
+    }
+    return result;
+  },
+  tokenize,
+};
+`);
+    return output;
+}
+exports.generate = generate;
+/**
+ * Generates a parser file from grammar and token definitions
+ * @param grammar - Grammar rules defining the language structure
+ * @param tokensDefinition - Token definitions mapping token names to patterns
+ * @param filename - Output path for the generated parser file
+ */
+function generateParser(grammar, tokensDefinition, filename) {
+    fs.writeFileSync(filename, generate(grammar, tokensDefinition, false).join('\n'));
+}
+exports.generateParser = generateParser;
+//# sourceMappingURL=metaParserGenerator.js.map

package/dist/metaParserGenerator.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"metaParserGenerator.js","sourceRoot":"","sources":["../metaParserGenerator.ts"],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;AACA,uCAAyB;AACzB,mCAAmE;AAInE,MAAM,aAAa,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAqErB,CAAC;AAEF;;;;GAIG;AACH,SAAgB,iBAAiB,CAAC,QAA0B;IAC1D,MAAM,MAAM,GAAa,EAAE,CAAC;IAC5B,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;IACnC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE;QACpC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,IAAI,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE;YACvB,MAAM,IAAI,KAAK,CAAC,6BAA6B,CAAC,CAAC;SAChD;KACF;IAED,MAAM,CAAC,IAAI,CAAC,qDAAqD,CAAC,CAAC;IACnE,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;IAC5B,IAAI,GAAG,CAAC;IACR,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE;QACpC,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACd,MAAM,KAAK,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC;QAC5B,IAAI,KAAK,CAAC,GAAG,EAAE;YACb,MAAM,MAAM,GAAG,KAAK,CAAC,GAAG,CAAC,MAAM,CAAC;YAChC,IAAI,KAAK,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,IAAI,KAAK,CAAC,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE;gBAC/D,MAAM,CAAC,IAAI,CAAC,4BAA4B,MAAM,WAAW,KAAK,CAAC,GAAG,OAAO,CAAC,CAAC;gBAC3E,MAAM,CAAC,IAAI,CAAC,iBAAiB,KAAK,CAAC,GAAG,QAAQ,GAAG,KAAK,CAAC,CAAC;aACzD;iBAAM;gBACL,MAAM,CAAC,IAAI,CAAC,4BAA4B,MAAM,UAAU,KAAK,CAAC,GAAG,MAAM,CAAC,CAAC;gBACzE,MAAM,CAAC,IAAI,CAAC,gBAAgB,KAAK,CAAC,GAAG,OAAO,GAAG,KAAK,CAAC,CAAC;aACvD;YACD,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;SACpB;aAAM,IAAI,KAAK,CAAC,GAAG,EAAE;YACpB,MAAM,CAAC,IAAI,CAAC,kDAAkD,GAAG,QAAQ,CAAC,CAAC;YAC3E,MAAM,CAAC,IAAI,CAAC,yBAAyB,CAAC,CAAC;YACvC,MAAM,CAAC,IAAI,CAAC,0BAA0B,GAAG,KAAK,CAAC,CAAC;YAChD,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;SACpB;aAAM,IAAI,KAAK,CAAC,IAAI,EAAE;YACrB,MAAM,CAAC,IAAI,CAAC,sBAAsB,GAAG,uCAAuC,CAAC,CAAC;YAC9E,MAAM,CAAC,IAAI,CAAC,8BAA8B,CAAC,CAAC;YAC5C,MAAM,CAAC,IAAI,CAAC,uBAAuB,GAAG,KAAK,CAAC,CAAC;YAC7C,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;SACpB;aAAM;YACL,MAAM,IAAI,KAAK,CAAC,kCAAkC,GAAG,sCAAsC,CAAC,CAAC;SAC9F;KACF;IACD,MAAM,CAAC,IAAI,CAAC,oBAAoB,GAAG,KAAK,CAAC,CAAC;IAC1C,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAEjB,MAAM,CAAC,IAAI,CAAC,sCAAsC,CAAC,CAAC;IACpD,MAAM,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAyDb,CAAC,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAvGD,8CAuGC;AAED;;;;;;;;GAQG;AACH,SAAS,sBAAsB,CAC7B,IAAY,EACZ,KAAa,EACb,SAA0B,EAC1B,SAA2B,EAC3B,KAAc;IAEd,MAAM,MAAM,GAAa,EAAE,CAAC;IAC5B,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,IAAI,KAAK,yBAAyB,CAAC,CAAC;IAC3D,IAAI,CAAC,GAAG,CAAC,CAAC;IACV,MAAM,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC;IAChC,MAAM,CAAC,IAAI,CAAC,wBAAwB,CAAC,CAAC;IACtC,MAAM,CAAC,IAAI,CAAC,qBAAqB,CAAC,CAAC;IACnC,MAAM,CAAC,IAAI,CAAC;4CAC8B,IAAI;sBAC1B,KAAK,YAAY,IAAI;KACtC,CAAC,CAAC;IACL,SAAS,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,EAAE;QACzB,gBAAgB;QAChB,IAAI,SAAS,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,IAAI,CAAC,KAAK,KAAK,KAAK,EAAE;YACjD,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,6BAA6B,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;YAC1D,IAAI,IAAI,CAAC,UAAU,EAAE;gBACnB,MAAM,CAAC,IAAI,CAAC,+BAA+B,IAAI,CAAC,KAAK,MAAM,CAAC,CAAC;gBAC7D,IAAI,IAAI,CAAC,KAAK,EAAE;oBACd,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,sBAAsB,IAAI,CAAC,KAAK,SAAS,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,oBAAoB,CAAC,CAAC;iBAC3D;gBACD,MAAM,CAAC,IAAI,CAAC,oCAAoC,CAAC,CAAC;gBAClD,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;aACpB;iBAAM,IAAI,IAAI,CAAC,QAAQ,EAAE;gBACxB,MAAM,CAAC,IAAI,CAAC,6BAA6B,IAAI,CAAC,KAAK,MAAM,CAAC,CAAC;gBAC3D,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,iBAAiB,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;gBAC3E,MAAM,CAAC,IAAI,CAAC,oCAAoC,CAAC,CAAC;gBAClD,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;aACpB;iBAAM;gBACL,MAAM,CAAC,IAAI,CAAC;4BACQ,IAAI,CAAC,KAAK;;;iBAGrB,IAAI,sBAAsB,KAAK;kEACkB,CAAC;;;;;;;CAOlE,CAAC,CAAC;gBACK,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,YAAY,IAAI,CAAC,KAAK,iBAAiB,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;gBACzE,MAAM,CAAC,IAAI,CAAC,kCAAkC,CAAC,CAAC;aACjD;YACD,CAAC,EAAE,CAAC;YACN,sCAAsC;SACrC;aAAM;YACL,IAAI,IAAI,CAAC,QAAQ,EAAE;gBACjB,MAAM,CAAC,IAAI,CAAC,WAAW,IAAI,CAAC,KAAK,4BAA4B,CAAC,CAAC;aAChE;iBAAM,IAAI,IAAI,CAAC,UAAU,EAAE;gBAC1B,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,MAAM,IAAI,CAAC,KAAK,cAAc,CAAC,CAAC,CAAC,iBAAiB;gBAC9E,MAAM,CAAC,IAAI,CAAC,kBAAkB,CAAC,KAAK,CAAC,CAAC;gBACtC,IAAI,IAAI,CAAC,KAAK,EAAE;oBACd,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,sBAAsB,IAAI,CAAC,KAAK,UAAU,CAAC,CAAC;oBAChF,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,iBAAiB,CAAC,IAAI,CAAC,CAAC;iBAC7D;gBACD,MAAM,CAAC,IAAI,CAAC,2BAA2B,CAAC,IAAI,CAAC,CAAC;gBAC9C,MAAM,CAAC,IAAI,CAAC,iBAAiB,CAAC,cAAc,CAAC,CAAC;gBAC9C,MAAM,CAAC,IAAI,CAAC,aAAa,CAAC,MAAM,IAAI,CAAC,KAAK,cAAc,CAAC,CAAC;gBAC1D,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;aACpB;iBAAM,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE;gBACzB,MAAM,CAAC,IAAI,CAAC,iBAAiB,CAAC,MAAM,IAAI,CAAC,KAAK,cAAc,CAAC,CAAC,CAAC,iBAAiB;gBAChF,MAAM,CAAC,IAAI,CAAC,gBAAgB,CAAC,iBAAiB,CAAC,CAAC;gBAChD,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,YAAY,IAAI,CAAC,KAAK,cAAc,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;gBAC1E,MAAM,CAAC,IAAI,CAAC,yBAAyB,CAAC,IAAI,CAAC,CAAC;gBAC5C,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,cAAc,CAAC,CAAC;aAC7C;iBAAM;gBACL,MAAM,CAAC,IAAI,CAAC,iBAAiB,CAAC,MAAM,IAAI,CAAC,KAAK,cAAc,CAAC,CAAC,CAAC,iBAAiB;gBAChF,MAAM,CAAC,IAAI,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;gBACnC,MAAM,CAAC,IAAI,CAAC,2BAA2B,CAAC,IAAI,CAAC,CAAC;gBAC9C,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,cAAc,IAAI,CAAC,KAAK,cAAc,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;gBAC5E,MAAM,CAAC,IAAI,CAAC,iBAAiB,CAAC,cAAc,CAAC,CAAC;gBAC9C,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;aACpB;YACD,CAAC,EAAE,CAAC;SACL;IACH,CAAC,CAAC,CAAC;IACH,MAAM,CAAC,IAAI,CAAC,4DAA4D,CAAC,CAAC;IAC1E,MAAM,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC;IAC9B,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAClB,IAAI,SAAS,CAAC,CAAC,CAAC,CAAC,aAAa,EAAE;QAC9B,MAAM,CAAC,IAAI,CAAC,GAAG,IAAI,IAAI,KAAK,0BAA0B,IAAI,IAAI,KAAK,MAAM,IAAI,IAAI,KAAK,IAAI,CAAC,CAAC;KAC7F;SAAM;QACL,MAAM,CAAC,IAAI,CAAC,GAAG,IAAI,IAAI,KAAK,eAAe,IAAI,IAAI,KAAK,MAAM,IAAI,IAAI,KAAK,IAAI,CAAC,CAAC;KAClF;IACD,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAClB,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;;;;GAMG;AACH,SAAgB,QAAQ,CAAC,OAAgB,EAAE,SAA2B,EAAE,KAAc;IACpF,IAAI,MAAM,GAAa,EAAE,CAAC;IAC1B,IAAA,6BAAqB,EAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IAC1C,MAAM,UAAU,GAAG,IAAA,yBAAiB,EAAC,OAAO,CAAC,CAAC;IAC9C,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;IACxC,MAAM,CAAC,IAAI,CAAC,2EAA2E,CAAC,CAAC;IACzF,MAAM,CAAC,IAAI,CAAC,yCAAyC,CAAC,CAAC;IACvD,MAAM,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IAC3B,OAAO,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,EAAE;QACtB,IAAI,CAAC,GAAG,CAAC,CAAC;QACV,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,UAAU,CAAC,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,SAAS,EAAE,EAAE;YACpC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,sBAAsB,CAAC,GAAG,EAAE,CAAC,EAAE,SAAS,EAAE,SAAS,EAAE,KAAK,CAAC,CAAC,CAAC;YACpF,OAAO,CAAC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC;YAC5B,CAAC,EAAE,CAAC;QACN,CAAC,CAAC,CAAC;QACH,MAAM,CAAC,IAAI,CAAC,YAAY,GAAG,mBAAmB,CAAC,CAAC;QAChD,MAAM,EAAE,GAAG,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,GAAG,iBAAiB,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QACzE,MAAM,CAAC,IAAI,CAAC,YAAY,EAAE,GAAG,CAAC,CAAC;QAC/B,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IACnB,CAAC,CAAC,CAAC;IACH,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,iBAAiB,CAAC,SAAS,CAAC,CAAC,CAAC;IACrD,MAAM,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;CAkBb,CAAC,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AA1CD,4BA0CC;AAED;;;;;GAKG;AACH,SAAgB,cAAc,CAAC,OAAgB,EAAE,gBAAkC,EAAE,QAAgB;IACnG,EAAE,CAAC,aAAa,CAAC,QAAQ,EACvB,QAAQ,CAAC,OAAO,EAAE,gBAAgB,EAAE,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;AAC3D,CAAC;AAHD,wCAGC"}

package/dist/types.d.ts

@@ -0,0 +1,47 @@
+export type ASTNode = RuleNode | Token;
+export interface RuleNode {
+    stream_index: number;
+    type: string;
+    sub_rule_index: number;
+    children: [ASTNode];
+    named: {
+        [key: string]: ASTNode;
+    };
+}
+export interface Token {
+    stream_index: number;
+    type: string;
+    value: string;
+    len: number;
+    line_start: number;
+    column_start: number;
+    start: number;
+}
+export interface TokenDefinition {
+    str?: string;
+    reg?: RegExp;
+    func?: (input: string, stream: Token[]) => string | undefined;
+    verbose?: string;
+}
+export type TokensDefinition = Record<string, TokenDefinition>;
+export type Grammar = Record<string, string[][]>;
+export interface ProcessedRule {
+    value: string;
+    alias?: string;
+    optional: boolean;
+    repeatable: boolean;
+    leftRecursion: boolean;
+    function?: boolean;
+}
+export type ProcessedGrammar = Record<string, ProcessedRule[][]>;
+export interface ParseFailure {
+    type: string;
+    sub_rule_index: number;
+    sub_rule_stream_index: number;
+    sub_rule_token_index: number;
+    stream_index: number;
+    token: Token;
+    first_token: Token;
+    success: false;
+    best_failure_array?: ParseFailure[];
+}

package/dist/types.js

@@ -0,0 +1,3 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+//# sourceMappingURL=types.js.map

package/dist/types.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.js","sourceRoot":"","sources":["../types.ts"],"names":[],"mappings":""}

package/dist/utils.d.ts

@@ -0,0 +1,29 @@
+import { ASTNode, Token, TokensDefinition, Grammar, ProcessedGrammar, ParseFailure } from "./types";
+declare function streamContext(token: Token, firstToken: Token, stream: Token[]): string;
+/**
+ * Displays a formatted error message for parsing failures
+ * @param stream - Token stream
+ * @param tokensDefinition - Token definitions
+ * @param grammar - Grammar rules
+ * @param bestFailure - The parse failure with details
+ */
+declare function displayError(stream: Token[], tokensDefinition: TokensDefinition, grammar: Grammar, bestFailure: ParseFailure): never;
+/**
+ * Prints the AST tree structure to console
+ * @param node - Root AST node to print
+ * @param sp - Spacing/indentation string
+ */
+declare function printTree(node: ASTNode, sp: string): void;
+/**
+ * Validates that grammar and token definitions don't have overlapping keys
+ * @param grammar - Grammar rules
+ * @param tokensDefinition - Token definitions
+ */
+declare function checkGrammarAndTokens(grammar: Grammar, tokensDefinition: TokensDefinition): void;
+/**
+ * Preprocesses grammar rules to extract modifiers and aliases
+ * @param rules - Raw grammar rules
+ * @returns Processed grammar with parsed modifiers and metadata
+ */
+declare function preprocessGrammar(rules: Grammar): ProcessedGrammar;
+export { streamContext, preprocessGrammar, checkGrammarAndTokens, displayError, printTree, };