dbrownell-parserlib 0.10.2__tar.gz → 0.11.0__tar.gz

{dbrownell_parserlib-0.10.2 → dbrownell_parserlib-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: dbrownell-parserlib
-Version: 0.10.2
+Version: 0.11.0
 Summary: Functionality useful when creating parsers.
 Author: Dave Brownell
 Author-email: Dave Brownell <github@DavidBrownell.com>
@@ -56,8 +56,9 @@ Description-Content-Type: text/markdown
 - **Visitor Pattern**: Flexible visitor pattern implementation with fine-grained traversal control
 - **Error Handling**: Rich error objects that associate messages with source locations and extract Python exception tracebacks
 - **Multi-threaded Parsing**: Built-in support for parsing multiple files concurrently
-- **Workspace Support**: Parse entire directory trees with customizable file filtering
-- **Whitespace Handling**: Mixins for both insignificant and significant whitespace grammars (e.g., Python-like indentation)
+- **Workspace Support**: Parse translation units organized across multiple workspace roots
+- **Whitespace Handling**: A single visitor mixin supports both insignificant and significant whitespace grammars (e.g., Python-like indentation); grammars that define `INDENT`/`DEDENT` tokens are detected automatically
+- **Multiline String Extraction**: Extract multiline string content that must be vertically aligned with its opening token, with detailed errors on misalignment
 
 ### How to use `dbrownell_ParserLib`
 
@@ -101,7 +102,7 @@ Implement a visitor that converts ANTLR parse trees to Expression objects:
 ```python
 from dataclasses import dataclass
 
-from dbrownell_ParserLib.antlr.antlr_visitor_mixins import InsignificantWhitespaceAntlrVisitorMixin
+from dbrownell_ParserLib.antlr.antlr_visitor_mixin import AntlrVisitorMixin
 from dbrownell_ParserLib.expression import Expression
 from dbrownell_ParserLib.terminal_expression import TerminalExpression
 
@@ -120,8 +121,10 @@ class BinaryExpression(Expression):
         yield "right", self.right
 
 
-class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisitor):
-    def visitExpr(self, ctx):
+class CalculatorVisitor(AntlrVisitorMixin, GeneratedVisitor):
+    def visitBinaryOp(self, ctx):
+        operator = TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText())
+
         children = self.GetChildren(ctx)
         assert len(children) == 2, children
 
@@ -129,20 +132,19 @@ class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisit
             BinaryExpression(
                 self.CreateRegion(ctx),
                 children[0],
-                TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText()),
-
+                operator,
                 children[1],
             )
         )
 
     def visitNumber(self, ctx):
         self._stack.append(
-            TerminalExpression(self.CreateRegion(ctx), int(ctx.getText()))
+            TerminalExpression[int](self.CreateRegion(ctx), int(ctx.getText()))
         )
 ```
 
 #### 4. Create Parser Instance
-Use `CreateAntlrParser` to build a callable parser:
+Use `CreateAntlrParser` to build a callable parser. Grammars that define `INDENT` and `DEDENT` tokens (significant whitespace) are detected automatically; no additional configuration is required.
 
 ```python
 from dbrownell_ParserLib.antlr.antlr_parser import CreateAntlrParser

{dbrownell_parserlib-0.10.2 → dbrownell_parserlib-0.11.0}/README.md

@@ -34,8 +34,9 @@
 - **Visitor Pattern**: Flexible visitor pattern implementation with fine-grained traversal control
 - **Error Handling**: Rich error objects that associate messages with source locations and extract Python exception tracebacks
 - **Multi-threaded Parsing**: Built-in support for parsing multiple files concurrently
-- **Workspace Support**: Parse entire directory trees with customizable file filtering
-- **Whitespace Handling**: Mixins for both insignificant and significant whitespace grammars (e.g., Python-like indentation)
+- **Workspace Support**: Parse translation units organized across multiple workspace roots
+- **Whitespace Handling**: A single visitor mixin supports both insignificant and significant whitespace grammars (e.g., Python-like indentation); grammars that define `INDENT`/`DEDENT` tokens are detected automatically
+- **Multiline String Extraction**: Extract multiline string content that must be vertically aligned with its opening token, with detailed errors on misalignment
 
 ### How to use `dbrownell_ParserLib`
 
@@ -79,7 +80,7 @@ Implement a visitor that converts ANTLR parse trees to Expression objects:
 ```python
 from dataclasses import dataclass
 
-from dbrownell_ParserLib.antlr.antlr_visitor_mixins import InsignificantWhitespaceAntlrVisitorMixin
+from dbrownell_ParserLib.antlr.antlr_visitor_mixin import AntlrVisitorMixin
 from dbrownell_ParserLib.expression import Expression
 from dbrownell_ParserLib.terminal_expression import TerminalExpression
 
@@ -98,8 +99,10 @@ class BinaryExpression(Expression):
         yield "right", self.right
 
 
-class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisitor):
-    def visitExpr(self, ctx):
+class CalculatorVisitor(AntlrVisitorMixin, GeneratedVisitor):
+    def visitBinaryOp(self, ctx):
+        operator = TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText())
+
         children = self.GetChildren(ctx)
         assert len(children) == 2, children
 
@@ -107,20 +110,19 @@ class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisit
             BinaryExpression(
                 self.CreateRegion(ctx),
                 children[0],
-                TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText()),
-
+                operator,
                 children[1],
             )
         )
 
     def visitNumber(self, ctx):
         self._stack.append(
-            TerminalExpression(self.CreateRegion(ctx), int(ctx.getText()))
+            TerminalExpression[int](self.CreateRegion(ctx), int(ctx.getText()))
         )
 ```
 
 #### 4. Create Parser Instance
-Use `CreateAntlrParser` to build a callable parser:
+Use `CreateAntlrParser` to build a callable parser. Grammars that define `INDENT` and `DEDENT` tokens (significant whitespace) are detected automatically; no additional configuration is required.
 
 ```python
 from dbrownell_ParserLib.antlr.antlr_parser import CreateAntlrParser

{dbrownell_parserlib-0.10.2 → dbrownell_parserlib-0.11.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dbrownell-parserlib"
-version = "0.10.2"
+version = "0.11.0"
 #          ^^^^^
 # Wheel names will be generated according to this value. Do not manually modify this value; instead
 # update it according to committed changes by running this command from the root of the repository:

{dbrownell_parserlib-0.10.2 → dbrownell_parserlib-0.11.0}/src/dbrownell_ParserLib/antlr/antlr_parser.py

@@ -102,6 +102,9 @@ class CreateVisitorFuncType(Protocol):
         on_progress_func: Callable[[int], None],
         *,
         is_included_file: bool,
+        parser_newline_string: str | None,
+        indent_token_type: int | None,
+        dedent_token_type: int | None,
     ) -> antlr4.ParseTreeVisitor: ...
 
 
@@ -117,9 +120,18 @@ def CreateAntlrParser(
     get_ast_func: Callable[[antlr4.Parser], antlr4.ParserRuleContext],
     custom_lexer_init_func: Callable[[antlr4.Lexer], None] | None = None,
     custom_parser_init_func: Callable[[antlr4.Parser], None] | None = None,
+    parser_newline_string: str | bool | None = None,  # noqa: FBT001
 ) -> AntlrParser:
     """Create an `AntlrParser` that uses the given code generated by ANTLR."""
 
+    indent_token_type = getattr(antlr_parser, "INDENT", None)
+    dedent_token_type = getattr(antlr_parser, "DEDENT", None)
+
+    if parser_newline_string is None and indent_token_type is not None and dedent_token_type is not None:
+        parser_newline_string = "newLine"
+    elif isinstance(parser_newline_string, bool):
+        parser_newline_string = None
+
     # ----------------------------------------------------------------------
     def CreateLexer(input_stream: antlr4.InputStream) -> antlr4.Lexer:
         lexer = antlr_lexer(input_stream)
@@ -145,6 +157,9 @@ def CreateAntlrParser(
         CreateParser,
         create_visitor_func,
         get_ast_func,
+        indent_token_type,
+        dedent_token_type,
+        parser_newline_string,
     )
 
 
@@ -162,12 +177,19 @@ class _AntlrParser(AntlrParser):
         create_parser_func: Callable[[antlr4.CommonTokenStream], antlr4.Parser],
         create_visitor_func: CreateVisitorFuncType,
         get_ast_func: Callable[[antlr4.Parser], antlr4.ParserRuleContext],
+        indent_token_type: int | None,
+        dedent_token_type: int | None,
+        parser_newline_string: str | None,
     ) -> None:
         self._create_lexer_func = create_lexer_func
         self._create_parser_func = create_parser_func
         self._create_visitor_func = create_visitor_func
         self._get_ast_func = get_ast_func
 
+        self._indent_token_type = indent_token_type
+        self._dedent_token_type = dedent_token_type
+        self._parser_newline_string = parser_newline_string
+
         self._execute_state: _AntlrParser._ExecuteState | None = None
 
     # ----------------------------------------------------------------------
@@ -433,6 +455,9 @@ class _AntlrParser(AntlrParser):
                         fullpath,
                         lambda line: cast(None, status.OnProgress(line, None)),
                         is_included_file=is_included_file,
+                        indent_token_type=self._indent_token_type,
+                        dedent_token_type=self._dedent_token_type,
+                        parser_newline_string=self._parser_newline_string,
                     )
 
                     ast = self._get_ast_func(parser)

dbrownell_parserlib-0.11.0/src/dbrownell_ParserLib/antlr/antlr_visitor_mixin.py

@@ -0,0 +1,150 @@
+# noqa: D100
+import re
+
+from typing import cast, TYPE_CHECKING
+
+import antlr4
+
+from dbrownell_ParserLib.region import Location, Region
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+    from pathlib import Path
+
+
+# ----------------------------------------------------------------------
+class AntlrVisitorMixin:
+    """Mixin that provides common visitation functionality; it is used along side ANTLR-generated visitors."""
+
+    # ----------------------------------------------------------------------
+    def __init__(
+        self,
+        filename: Path,
+        on_progress_func: Callable[[int], None],
+        *,
+        is_included_file: bool,
+        parser_newline_string: str | None,
+        indent_token_type: int | None,
+        dedent_token_type: int | None,
+    ) -> None:
+        self.filename = filename
+        self.is_included_file = is_included_file
+
+        # Protected data
+        self._on_progress_func = on_progress_func
+        self._parser_newline_string = parser_newline_string
+        self._indent_token_type = indent_token_type
+        self._dedent_token_type = dedent_token_type
+        self._stack: list[object] = []
+
+        # Private data
+        self._current_line: int = 0
+        self._stop_token_regex = re.compile(r"^\r?\n[ \t]*$")
+
+    # ----------------------------------------------------------------------
+    def CreateRegion(
+        self,
+        ctx: antlr4.TerminalNode | antlr4.ParserRuleContext,
+    ) -> Region:
+        """Create a `Region` from the given context."""
+
+        if isinstance(ctx, antlr4.TerminalNode):
+            start_token = cast(antlr4.Token, ctx.symbol)  # ty: ignore[unresolved-attribute]
+            end_location = self._CreateEndLocation(start_token)
+        else:
+            start_token = ctx.start
+            stop_token = ctx.stop
+
+            assert ctx.children is not None
+            assert start_token is not None
+            assert stop_token is not None
+
+            end_location: Location | None = None
+
+            if self._parser_newline_string is not None:
+                if self._IsStopToken(stop_token):
+                    assert len(ctx.children) == 2, ctx.children  # noqa: PLR2004
+                    end_location = self._CreateEndLocation(start_token)
+                elif stop_token.type == self._dedent_token_type:
+                    end_location = Location(stop_token.line, stop_token.column + 1)
+
+            if end_location is None:
+                end_location = self._CreateEndLocation(stop_token)
+
+        assert start_token.line is not None
+        assert start_token.column is not None
+
+        region = Region(
+            self.filename,
+            Location(start_token.line, start_token.column + 1),
+            end_location,
+        )
+
+        self._OnProgress(region.end.line)
+
+        return region
+
+    # ----------------------------------------------------------------------
+    def GetChildren(self, ctx: antlr4.ParserRuleContext) -> list[object]:
+        """Return the results of visiting the children of the given context."""
+
+        prev_num_stack_items = len(self._stack)
+
+        cast(antlr4.ParseTreeVisitor, self).visitChildren(ctx)
+
+        results = self._stack[prev_num_stack_items:]
+        del self._stack[prev_num_stack_items:]
+
+        return results
+
+    # ----------------------------------------------------------------------
+    # |  Protected Methods
+    def _OnProgress(self, end_line: int) -> None:
+        if end_line > self._current_line:
+            self._current_line = end_line
+            self._on_progress_func(end_line)
+
+    # ----------------------------------------------------------------------
+    # ----------------------------------------------------------------------
+    # ----------------------------------------------------------------------
+    @staticmethod
+    def _CreateEndLocation(token: antlr4.Token) -> Location:
+        assert token.text is not None
+        assert token.line is not None
+        assert token.column is not None
+
+        return CreateEndLocation(token.text, token.line, token.column + 1)
+
+    # ----------------------------------------------------------------------
+    def _IsStopToken(self, token: antlr4.Token) -> bool:
+        assert token.text is not None
+
+        return (
+            token.text == self._parser_newline_string or self._stop_token_regex.match(token.text) is not None
+        )
+
+
+# ----------------------------------------------------------------------
+def CreateEndLocation(text: str, line: int, column: int) -> Location:
+    """Create a `Location` based on the text and current line and column."""
+
+    # Note that this function will not be invoked outside of this file; it is implemented as a
+    # function so that it can be exercised by automated tests.
+
+    assert line > 0, line
+    assert column > 0, column
+
+    lines = text.splitlines()
+
+    last_nonblank_line = None
+
+    for line_index, line_text in enumerate(lines):
+        if line_text.strip():
+            last_nonblank_line = line_index
+
+    assert last_nonblank_line is not None
+
+    if last_nonblank_line == 0:
+        return Location(line, column + len(lines[0]))
+
+    return Location(line + last_nonblank_line, len(lines[last_nonblank_line]) + 1)

dbrownell_parserlib-0.10.2/src/dbrownell_ParserLib/antlr/antlr_visitor_mixins.py

@@ -1,169 +0,0 @@
-"""Various ANTLR visitor mixins that can be used to add functionality to the generated visitor classes."""
-
-import re
-
-from abc import ABC, abstractmethod
-from typing import cast, override, TYPE_CHECKING
-
-import antlr4
-
-from dbrownell_ParserLib.region import Location, Region
-
-if TYPE_CHECKING:
-    from collections.abc import Callable
-    from pathlib import Path
-
-
-# ----------------------------------------------------------------------
-class AntlrVisitorMixinBase(ABC):
-    """Base class for visitor mixins. This class is not meant to be used directly, but rather to be inherited by other mixin classes."""
-
-    # ----------------------------------------------------------------------
-    def __init__(
-        self,
-        filename: Path,
-        on_progress_func: Callable[[int], None],
-        *,
-        is_included_file: bool,
-    ) -> None:
-        self.filename = filename
-        self.is_included_file = is_included_file
-
-        # Protected data
-        self._on_progress_func = on_progress_func
-        self._stack: list[object] = []
-
-        # Private data
-        self._current_line: int = 0
-
-    # ----------------------------------------------------------------------
-    def CreateRegion(
-        self,
-        ctx: antlr4.TerminalNode | antlr4.ParserRuleContext,
-    ) -> Region:
-        """Create a `Region` from the given context."""
-
-        if isinstance(ctx, antlr4.TerminalNode):
-            token = cast(antlr4.Token, ctx.symbol)  # ty: ignore[unresolved-attribute]
-
-            assert token.column is not None
-
-            start_line = token.line
-            start_col = token.column + 1
-            end_line = token.line
-            end_col = token.column + 1 + len(token.text)
-
-            assert start_line is not None
-            assert end_line is not None
-
-            region = Region(
-                self.filename,
-                Location(start_line, start_col),
-                Location(end_line, end_col),
-            )
-        else:
-            region = self._CreateRegionFromRule(ctx)
-
-        self._OnProgress(region.end.line)
-
-        return region
-
-    # ----------------------------------------------------------------------
-    def GetChildren(self, ctx: antlr4.ParserRuleContext) -> list[object]:
-        """Return the results of visiting the children of the given context."""
-
-        prev_num_stack_items = len(self._stack)
-
-        cast(antlr4.ParseTreeVisitor, self).visitChildren(ctx)
-
-        results = self._stack[prev_num_stack_items:]
-        del self._stack[prev_num_stack_items:]
-
-        return results
-
-    # ----------------------------------------------------------------------
-    # |  Protected Methods
-    def _OnProgress(self, end_line: int) -> None:
-        if end_line > self._current_line:
-            self._current_line = end_line
-            self._on_progress_func(end_line)
-
-    # ----------------------------------------------------------------------
-    @abstractmethod
-    def _CreateRegionFromRule(self, ctx: antlr4.ParserRuleContext) -> Region: ...
-
-
-# ----------------------------------------------------------------------
-class InsignificantWhitespaceAntlrVisitorMixin(AntlrVisitorMixinBase):
-    """Mixin that adds functionality used for grammars that ignore insignificant whitespace."""
-
-    # ----------------------------------------------------------------------
-    def _CreateRegionFromRule(self, ctx: antlr4.ParserRuleContext) -> Region:
-        start_token = ctx.start
-        stop_token = ctx.stop
-
-        if start_token is None or stop_token is None:
-            msg = "Context does not have start or stop token"
-            raise ValueError(msg)
-
-        start_line = start_token.line
-        start_col = start_token.column + 1
-        end_line = stop_token.line
-        end_col = stop_token.column + 1 + len(stop_token.text)
-
-        assert start_line is not None
-        assert end_line is not None
-
-        return Region(
-            self.filename,
-            Location(start_line, start_col),
-            Location(end_line, end_col),
-        )
-
-
-# ----------------------------------------------------------------------
-class SignificantWhitespaceAntlrVisitorMixin(AntlrVisitorMixinBase):
-    """Mixin that adds functionality used for grammars that treat whitespace as significant."""
-
-    # ----------------------------------------------------------------------
-    def __init__(
-        self,
-        dedent_token: int,
-        newline_token: int,
-        newline_token_string: str,
-    ) -> None:
-        self._dedent_token = dedent_token
-        self._newline_token = newline_token
-        self._newline_token_string = newline_token_string
-
-        self._newline_indent_regex = re.compile(r"^\r?\n[ \t]*$")
-
-    # ----------------------------------------------------------------------
-    @override
-    def _CreateRegionFromRule(
-        self,
-        ctx: antlr4.ParserRuleContext,
-    ) -> Region:
-        assert isinstance(ctx.start, antlr4.Token), ctx.start
-        assert isinstance(ctx.stop, antlr4.Token), ctx.stop
-
-        if ctx.stop.text == self._newline_token_string or self._newline_indent_regex.match(ctx.stop.text):
-            assert ctx.stop.line == ctx.start.line or ctx.stop.line - 1 == ctx.start.line, (
-                ctx.start.line,
-                ctx.stop.line,
-            )
-
-            stop_line = ctx.start.line
-            stop_column = ctx.start.column + 1 + len(ctx.start.text)
-
-            return Region(
-                self.filename,
-                Location(ctx.start.line, ctx.start.column + 1),
-                Location(stop_line, stop_column),
-            )
-
-        return Region(
-            self.filename,
-            Location(ctx.start.line, ctx.start.column + 1),
-            Location(ctx.stop.line, ctx.stop.column + 1),
-        )