dbrownell-parserlib 0.8.1__tar.gz → 0.8.3__tar.gz

{dbrownell_parserlib-0.8.1 → dbrownell_parserlib-0.8.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dbrownell-parserlib
-Version: 0.8.1
+Version: 0.8.3
 Summary: Functionality useful when creating parsers.
 Author: Dave Brownell
 Author-email: Dave Brownell <github@DavidBrownell.com>
@@ -47,10 +47,153 @@ Description-Content-Type: text/markdown
 - [License](#license)
 
 ## Overview
-TODO: Complete this section
+`dbrownell_ParserLib` is a Python library that provides utility functionality for creating parsers, with a primary focus on [ANTLR](https://antlr.org) (Another Tool for Language Recognition) integration. The library simplifies the process of building parsers that produce strongly-typed abstract syntax trees (ASTs) with comprehensive source location tracking.
+
+### Key Features
+- **ANTLR Integration**: Streamlined wrapper around ANTLR for Python, handling grammar compilation and parser generation
+- **AST Representation**: Expression-based AST nodes with automatic parent-child relationship management
+- **Source Location Tracking**: Every AST node tracks its source file location (filename, line, column ranges) for detailed error reporting
+- **Visitor Pattern**: Flexible visitor pattern implementation with fine-grained traversal control
+- **Error Handling**: Rich error objects that associate messages with source locations and extract Python exception tracebacks
+- **Multi-threaded Parsing**: Built-in support for parsing multiple files concurrently
+- **Workspace Support**: Parse entire directory trees with customizable file filtering
+- **Whitespace Handling**: Mixins for both insignificant and significant whitespace grammars (e.g., Python-like indentation)
 
 ### How to use `dbrownell_ParserLib`
-TODO: Complete this section
+
+The typical workflow for creating a parser with this library involves:
+
+#### 1. Define an ANTLR Grammar
+Create a `.g4` grammar file defining your language syntax:
+```antlr
+grammar Calculator;
+
+expr : expr op=('*'|'/') expr   # BinaryOp
+     | expr op=('+'|'-') expr   # BinaryOp
+     | INT                       # Number
+     ;
+
+INT : [0-9]+ ;
+WS : [ \t\n\r]+ -> skip ;
+```
+
+#### 2. Build the Grammar
+Use `BuildAntlrGrammar` to generate the lexer and parser:
+```python
+from dbrownell_ParserLib import BuildAntlrGrammar
+from pathlib import Path
+
+BuildAntlrGrammar(
+    dm,  # DoneManager instance
+    Path("Calculator.g4"),
+    Path("output_dir"),
+)
+```
+
+#### 3. Create Custom Visitor
+Implement a visitor that converts ANTLR parse trees to Expression objects:
+```python
+from dbrownell_ParserLib import (
+    Expression,
+    InsignificantWhitespaceAntlrVisitorMixin,
+    TerminalExpression,
+)
+from dataclasses import dataclass
+
+from CalculatorVisitor import CalculatorVisitor as GeneratedVisitor
+
+
+
+@dataclass(eq=False)
+class BinaryExpression(Expression):
+    left: Expression
+    operator: TerminalExpression[str]
+    right: Expression
+
+    def _GenerateAcceptDetails(self):
+        yield "left", self.left
+        yield "operator", self.operator
+        yield "right", self.right
+
+class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisitor):
+
+    def visitBinaryOp(self, ctx):
+        children = self.GetChildren(ctx)
+        self._stack.append(
+            BinaryExpression(
+                self.CreateRegion(ctx),
+                children[0],
+                TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText()),
+                children[1],
+            )
+        )
+
+    def visitNumber(self, ctx):
+        self._stack.append(
+            TerminalExpression(self.CreateRegion(ctx), int(ctx.INT().getText()))
+        )
+```
+
+#### 4. Create Parser Instance
+Use `CreateAntlrParser` to build a callable parser:
+```python
+from dbrownell_ParserLib import CreateAntlrParser
+
+parser = CreateAntlrParser(
+    CalculatorLexer,
+    CalculatorParser,
+    CalculatorVisitor,
+    lambda p: p.expr(),  # Entry point rule
+)
+```
+from pathlib import Path
+
+from dbrownell_ParserLib import Error
+
+
+
+#### 5. Parse Files
+Parse single files, multiple files, or entire workspaces:
+```python
+# Parse a single file
+results = parser(dm, Path("input.txt"), None)
+
+# Parse multiple files
+results = parser(dm, [Path("file1.txt"), Path("file2.txt")], None)
+
+# Check results
+for filepath, result in results.items():
+    if isinstance(result, Error):
+        print(f"Parse error in {filepath}: {result}")
+    else:
+        # result is the visitor containing the parsed AST
+        ast = result._stack[0]
+        # Process the AST
+```
+
+from contextlib import contextmanager
+
+#### 6. Traverse the AST
+Use the visitor pattern to process your AST:
+```python
+    @contextmanager
+
+from dbrownell_ParserLib import ExpressionVisitorHelper, VisitResult
+
+class EvaluatorVisitor(ExpressionVisitorHelper):
+    def OnBinaryExpression(self, expression):
+    @contextmanager
+
+        # Process binary expressions
+        yield VisitResult.Continue
+
+        # Process terminal values
+        yield VisitResult.Continue
+
+ast.Accept(EvaluatorVisitor())
+```
+
+For detailed examples, see the test files in the repository, particularly the end-to-end tests that demonstrate complete parsing workflows.
 
 <!-- Content below this delimiter will be copied to the generated README.md file. DO NOT REMOVE THIS COMMENT, as it will cause regeneration to fail. -->
 

{dbrownell_parserlib-0.8.1 → dbrownell_parserlib-0.8.3}/README.md

@@ -25,10 +25,153 @@
 - [License](#license)
 
 ## Overview
-TODO: Complete this section
+`dbrownell_ParserLib` is a Python library that provides utility functionality for creating parsers, with a primary focus on [ANTLR](https://antlr.org) (Another Tool for Language Recognition) integration. The library simplifies the process of building parsers that produce strongly-typed abstract syntax trees (ASTs) with comprehensive source location tracking.
+
+### Key Features
+- **ANTLR Integration**: Streamlined wrapper around ANTLR for Python, handling grammar compilation and parser generation
+- **AST Representation**: Expression-based AST nodes with automatic parent-child relationship management
+- **Source Location Tracking**: Every AST node tracks its source file location (filename, line, column ranges) for detailed error reporting
+- **Visitor Pattern**: Flexible visitor pattern implementation with fine-grained traversal control
+- **Error Handling**: Rich error objects that associate messages with source locations and extract Python exception tracebacks
+- **Multi-threaded Parsing**: Built-in support for parsing multiple files concurrently
+- **Workspace Support**: Parse entire directory trees with customizable file filtering
+- **Whitespace Handling**: Mixins for both insignificant and significant whitespace grammars (e.g., Python-like indentation)
 
 ### How to use `dbrownell_ParserLib`
-TODO: Complete this section
+
+The typical workflow for creating a parser with this library involves:
+
+#### 1. Define an ANTLR Grammar
+Create a `.g4` grammar file defining your language syntax:
+```antlr
+grammar Calculator;
+
+expr : expr op=('*'|'/') expr   # BinaryOp
+     | expr op=('+'|'-') expr   # BinaryOp
+     | INT                       # Number
+     ;
+
+INT : [0-9]+ ;
+WS : [ \t\n\r]+ -> skip ;
+```
+
+#### 2. Build the Grammar
+Use `BuildAntlrGrammar` to generate the lexer and parser:
+```python
+from dbrownell_ParserLib import BuildAntlrGrammar
+from pathlib import Path
+
+BuildAntlrGrammar(
+    dm,  # DoneManager instance
+    Path("Calculator.g4"),
+    Path("output_dir"),
+)
+```
+
+#### 3. Create Custom Visitor
+Implement a visitor that converts ANTLR parse trees to Expression objects:
+```python
+from dbrownell_ParserLib import (
+    Expression,
+    InsignificantWhitespaceAntlrVisitorMixin,
+    TerminalExpression,
+)
+from dataclasses import dataclass
+
+from CalculatorVisitor import CalculatorVisitor as GeneratedVisitor
+
+
+
+@dataclass(eq=False)
+class BinaryExpression(Expression):
+    left: Expression
+    operator: TerminalExpression[str]
+    right: Expression
+
+    def _GenerateAcceptDetails(self):
+        yield "left", self.left
+        yield "operator", self.operator
+        yield "right", self.right
+
+class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisitor):
+
+    def visitBinaryOp(self, ctx):
+        children = self.GetChildren(ctx)
+        self._stack.append(
+            BinaryExpression(
+                self.CreateRegion(ctx),
+                children[0],
+                TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText()),
+                children[1],
+            )
+        )
+
+    def visitNumber(self, ctx):
+        self._stack.append(
+            TerminalExpression(self.CreateRegion(ctx), int(ctx.INT().getText()))
+        )
+```
+
+#### 4. Create Parser Instance
+Use `CreateAntlrParser` to build a callable parser:
+```python
+from dbrownell_ParserLib import CreateAntlrParser
+
+parser = CreateAntlrParser(
+    CalculatorLexer,
+    CalculatorParser,
+    CalculatorVisitor,
+    lambda p: p.expr(),  # Entry point rule
+)
+```
+from pathlib import Path
+
+from dbrownell_ParserLib import Error
+
+
+
+#### 5. Parse Files
+Parse single files, multiple files, or entire workspaces:
+```python
+# Parse a single file
+results = parser(dm, Path("input.txt"), None)
+
+# Parse multiple files
+results = parser(dm, [Path("file1.txt"), Path("file2.txt")], None)
+
+# Check results
+for filepath, result in results.items():
+    if isinstance(result, Error):
+        print(f"Parse error in {filepath}: {result}")
+    else:
+        # result is the visitor containing the parsed AST
+        ast = result._stack[0]
+        # Process the AST
+```
+
+from contextlib import contextmanager
+
+#### 6. Traverse the AST
+Use the visitor pattern to process your AST:
+```python
+    @contextmanager
+
+from dbrownell_ParserLib import ExpressionVisitorHelper, VisitResult
+
+class EvaluatorVisitor(ExpressionVisitorHelper):
+    def OnBinaryExpression(self, expression):
+    @contextmanager
+
+        # Process binary expressions
+        yield VisitResult.Continue
+
+        # Process terminal values
+        yield VisitResult.Continue
+
+ast.Accept(EvaluatorVisitor())
+```
+
+For detailed examples, see the test files in the repository, particularly the end-to-end tests that demonstrate complete parsing workflows.
 
 <!-- Content below this delimiter will be copied to the generated README.md file. DO NOT REMOVE THIS COMMENT, as it will cause regeneration to fail. -->
 

{dbrownell_parserlib-0.8.1 → dbrownell_parserlib-0.8.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dbrownell-parserlib"
-version = "0.8.1"
+version = "0.8.3"
 #          ^^^^^
 # Wheel names will be generated according to this value. Do not manually modify this value; instead
 # update it according to committed changes by running this command from the root of the repository:

{dbrownell_parserlib-0.8.1 → dbrownell_parserlib-0.8.3}/src/dbrownell_ParserLib/antlr/antlr_parser.py

@@ -485,7 +485,7 @@ class _ErrorListener(antlr4.DiagnosticErrorListener):
             return
 
         location = Location(line, column + 1)
-        error = Error(msg, Region(self._source, location, location))
+        error = Error(f"Syntax error: {msg}", Region(self._source, location, location))
         ex = AntlrParserException(error)
 
         if e is None:

{dbrownell_parserlib-0.8.1 → dbrownell_parserlib-0.8.3}/src/dbrownell_ParserLib/antlr/antlr_visitor_mixins.py

@@ -1,5 +1,7 @@
 """Various ANTLR visitor mixins that can be used to add functionality to the generated visitor classes."""
 
+import re
+
 from abc import ABC, abstractmethod
 from typing import cast, override, TYPE_CHECKING
 
@@ -35,14 +37,36 @@ class AntlrVisitorMixinBase(ABC):
         self._current_line: int = 0
 
     # ----------------------------------------------------------------------
-    @abstractmethod
     def CreateRegion(
         self,
-        ctx: antlr4.ParserRuleContext,
+        ctx: antlr4.TerminalNode | antlr4.ParserRuleContext,
     ) -> Region:
         """Create a `Region` from the given context."""
 
-        raise NotImplementedError()  # pragma: no cover
+        if isinstance(ctx, antlr4.TerminalNode):
+            token = cast(antlr4.Token, ctx.symbol)  # ty: ignore[unresolved-attribute]
+
+            assert token.column is not None
+
+            start_line = token.line
+            start_col = token.column + 1
+            end_line = token.line
+            end_col = token.column + 1 + len(token.text)
+
+            assert start_line is not None
+            assert end_line is not None
+
+            region = Region(
+                self.filename,
+                Location(start_line, start_col),
+                Location(end_line, end_col),
+            )
+        else:
+            region = self._CreateRegionFromRule(ctx)
+
+        self._OnProgress(region.end.line)
+
+        return region
 
     # ----------------------------------------------------------------------
     def GetChildren(self, ctx: antlr4.ParserRuleContext) -> list[object]:
@@ -64,49 +88,28 @@ class AntlrVisitorMixinBase(ABC):
             self._current_line = end_line
             self._on_progress_func(end_line)
 
+    # ----------------------------------------------------------------------
+    @abstractmethod
+    def _CreateRegionFromRule(self, ctx: antlr4.ParserRuleContext) -> Region: ...
+
 
 # ----------------------------------------------------------------------
 class InsignificantWhitespaceAntlrVisitorMixin(AntlrVisitorMixinBase):
     """Mixin that adds functionality used for grammars that ignore insignificant whitespace."""
 
     # ----------------------------------------------------------------------
-    @override
-    def CreateRegion(
-        self,
-        ctx: antlr4.ParserRuleContext | antlr4.TerminalNode,
-    ) -> Region:
-        """Create a `Region` from the given context."""
-
-        if isinstance(ctx, antlr4.ParserRuleContext):
-            start_token = ctx.start
-            stop_token = ctx.stop
-
-            if start_token is None or stop_token is None:
-                msg = "Context does not have start or stop token"
-                raise ValueError(msg)
+    def _CreateRegionFromRule(self, ctx: antlr4.ParserRuleContext) -> Region:
+        start_token = ctx.start
+        stop_token = ctx.stop
 
-            start_line = start_token.line
-            start_col = start_token.column + 1
-            end_line = stop_token.line
-            end_col = stop_token.column + 1
-
-            if start_line == end_line:
-                end_col += len(stop_token.text)
-            else:
-                end_col += 1
-
-        elif isinstance(ctx, antlr4.TerminalNode):
-            token = cast(antlr4.Token, ctx.symbol)  # ty: ignore[unresolved-attribute]
-
-            assert token.column is not None
-
-            start_line = token.line
-            start_col = token.column + 1
-            end_line = token.line
-            end_col = token.column + 1 + len(token.text)
+        if start_token is None or stop_token is None:
+            msg = "Context does not have start or stop token"
+            raise ValueError(msg)
 
-        else:
-            assert False, ctx  # noqa: B011, PT015  # pragma: no cover
+        start_line = start_token.line
+        start_col = start_token.column + 1
+        end_line = stop_token.line
+        end_col = stop_token.column + 1 + len(stop_token.text)
 
         assert start_line is not None
         assert end_line is not None
@@ -133,52 +136,34 @@ class SignificantWhitespaceAntlrVisitorMixin(AntlrVisitorMixinBase):
         self._newline_token = newline_token
         self._newline_token_string = newline_token_string
 
+        self._newline_indent_regex = re.compile(r"^\r?\n[ \t]*$")
+
     # ----------------------------------------------------------------------
     @override
-    def CreateRegion(
+    def _CreateRegionFromRule(
         self,
         ctx: antlr4.ParserRuleContext,
     ) -> Region:
         assert isinstance(ctx.start, antlr4.Token), ctx.start
         assert isinstance(ctx.stop, antlr4.Token), ctx.stop
 
-        if ctx.stop.type == self._dedent_token:
-            stop_line = ctx.stop.line
-            stop_col = ctx.stop.column
-
-        elif ctx.stop.type == self._newline_token and ctx.stop.text == self._newline_token_string:
-            if ctx.stop.line == ctx.start.line:
-                # This is the scenario where the statement is followed by a dedent followed by another
-                # statement. We don't want the range of this item to overlap with the range of the
-                # next item, so use the values as they are, even though it means that a statement
-                # that terminates with a newline will not have the newline here.
-                stop_line = ctx.stop.line
-                stop_col = ctx.stop.column
-            else:
-                stop_line = ctx.stop.line
-                stop_col = ctx.stop.column if ctx.stop.column == 0 else ctx.start.column
-
-        else:
-            stop_line = ctx.stop.line
-
-            content = ctx.stop.text
-            lines = content.split("\n")
-
-            if ctx.stop.type == self._newline_token:
-                assert content.startswith("\n"), content
-                assert len(lines) == 2, lines  # noqa: PLR2004
-
-            stop_col = len(lines[-1])
-
-            if stop_line == ctx.stop.line:
-                stop_col += ctx.stop.column
+        if ctx.stop.text == self._newline_token_string or self._newline_indent_regex.match(ctx.stop.text):
+            assert ctx.stop.line == ctx.start.line or ctx.stop.line - 1 == ctx.start.line, (
+                ctx.start.line,
+                ctx.stop.line,
+            )
 
-            stop_line += len(lines) - 1
+            stop_line = ctx.start.line
+            stop_column = ctx.start.column + 1 + len(ctx.start.text)
 
-        self._OnProgress(stop_line)
+            return Region(
+                self.filename,
+                Location(ctx.start.line, ctx.start.column + 1),
+                Location(stop_line, stop_column),
+            )
 
         return Region(
             self.filename,
             Location(ctx.start.line, ctx.start.column + 1),
-            Location(stop_line, stop_col),
+            Location(ctx.stop.line, ctx.stop.column + 1),
         )