textmate-grammar-python 0.1.0__tar.gz → 0.1.2__tar.gz

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: textmate-grammar-python
-Version: 0.1.0
+Version: 0.1.2
 Summary: An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format.
 Home-page: https://github.com/watermarkhu/textmate-grammar-python
 License: MIT
@@ -11,6 +11,7 @@ Requires-Python: >=3.11,<4.0
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Requires-Dist: PyYAML (>=6.0.1,<7.0.0)
 Requires-Dist: onigurumacffi (>=1.3.0,<2.0.0)
 Project-URL: Repository, https://github.com/watermarkhu/textmate-grammar-python
@@ -20,14 +21,26 @@ Description-Content-Type: text/markdown
 
 An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format. 
 
+
+## Usage
+Install the module using `pip install textmate-grammar-python`.
+
+Before tokenization is possible, a `LanguageParser` needs to be initialized using a loaded grammar. 
+
 ```python
->>> from textmate_grammar.language import LanguageParser
->>> from textmate_grammar.grammars import matlab
+from textmate_grammar.language import LanguageParser
+from textmate_grammar.grammars import matlab
+parser = LanguageParser(matlab.GRAMMAR)
+```
 
->>> parser = LanguageParser(matlab.GRAMMAR)
->>> element = parser.parse_string("value = num2str(10);")
+After this, one can either choose to call `parser.parsing_string` to parse a input string directly, or call `parser.parse_file` with the path to the appropiate source file as the first argument, such as the the example `example.py`. 
+
+The parsed `element` object can be displayed directly by calling the `print` method. By default the element is printed as an element tree in a dictionary format. 
 
+```python
+>>> element = parser.parse_string("value = num2str(10);")
 >>> element.print()
+
 {'token': 'source.matlab',
  'children': [{'token': 'meta.assignment.variable.single.matlab', 
                'children': [{'token': 'variable.other.readwrite.matlab', 'content': 'value'}]},
@@ -39,7 +52,12 @@ An interpreter for grammar files as defined by TextMate and used in VSCode, impl
                'children': [{'token': 'constant.numeric.decimal.matlab', 'content': '10'}]},
               {'token': 'punctuation.terminator.semicolon.matlab', 'content': ';'}]}
 
+```
+Alternatively, with the keyword argument `flatten` the element is displayed as a list per unique token. Here the first item in the list is the starting position (line, column) of the unique tokenized element. 
+
+```python
 >>> element.print(flatten=True)
+
 [[(0, 0), 'value', ['source.matlab', 'meta.assignment.variable.single.matlab', 'variable.other.readwrite.matlab']],
  [(0, 5), ' ', ['source.matlab']],
  [(0, 6), '=', ['source.matlab', 'keyword.operator.assignment.matlab']],
@@ -51,25 +69,37 @@ An interpreter for grammar files as defined by TextMate and used in VSCode, impl
  [(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]
 ```
 
+## Supported Languages
+- [MATLAB](https://github.com/mathworks/MATLAB-Language-grammar)
+
+## TODO
+- Implement Begin/While pattern, required for other grammars.
+
 ## Sources
 - [Textmate guide](https://www.apeth.com/nonblog/stories/textmatebundle.html)
 - [VSCode Syntax Highlighting guide](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide)
 - [vscode-textmate](https://github.com/microsoft/vscode-textmate)
 - [Macromates texmate](https://macromates.com/textmate/manual/)
 
-## TODO
-- [x] Use yaml in stead of json
-- [x] Grammar initialization with *repository* at any level
-- [x] Proper line-by-line parsing
-- [x] `\G` anchoring
-- [x] Option to *disable* a grammar
-- [x] Option to *applyEndPatternLast*
-- [x] Implement *injections*
-- [ ] Implement Begin/While pattern
 
-## Supported Languages
-- [MATLAB](https://github.com/mathworks/MATLAB-Language-grammar)
+## Development
+
+Install the repository after cloning with [poetry](https://python-poetry.org/)
+
+```bash
+> pip install poetry
+> cd textmate-grammar-python
+> poetry install
+```
+
+Run unit tests
+```bash
+> tox run
+```
+
+Run regression testing against vscode-textmate (will install npm and required packages)
+```bash
+> tox run -e regression
+```
 
-## TextMate 2 features that are not used
-- [Format strings](https://macromates.com/blog/2011/format-strings/)
 

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/README.md

@@ -2,14 +2,26 @@
 
 An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format. 
 
+
+## Usage
+Install the module using `pip install textmate-grammar-python`.
+
+Before tokenization is possible, a `LanguageParser` needs to be initialized using a loaded grammar. 
+
 ```python
->>> from textmate_grammar.language import LanguageParser
->>> from textmate_grammar.grammars import matlab
+from textmate_grammar.language import LanguageParser
+from textmate_grammar.grammars import matlab
+parser = LanguageParser(matlab.GRAMMAR)
+```
 
->>> parser = LanguageParser(matlab.GRAMMAR)
->>> element = parser.parse_string("value = num2str(10);")
+After this, one can either choose to call `parser.parsing_string` to parse a input string directly, or call `parser.parse_file` with the path to the appropiate source file as the first argument, such as the the example `example.py`. 
+
+The parsed `element` object can be displayed directly by calling the `print` method. By default the element is printed as an element tree in a dictionary format. 
 
+```python
+>>> element = parser.parse_string("value = num2str(10);")
 >>> element.print()
+
 {'token': 'source.matlab',
  'children': [{'token': 'meta.assignment.variable.single.matlab', 
                'children': [{'token': 'variable.other.readwrite.matlab', 'content': 'value'}]},
@@ -21,7 +33,12 @@ An interpreter for grammar files as defined by TextMate and used in VSCode, impl
                'children': [{'token': 'constant.numeric.decimal.matlab', 'content': '10'}]},
               {'token': 'punctuation.terminator.semicolon.matlab', 'content': ';'}]}
 
+```
+Alternatively, with the keyword argument `flatten` the element is displayed as a list per unique token. Here the first item in the list is the starting position (line, column) of the unique tokenized element. 
+
+```python
 >>> element.print(flatten=True)
+
 [[(0, 0), 'value', ['source.matlab', 'meta.assignment.variable.single.matlab', 'variable.other.readwrite.matlab']],
  [(0, 5), ' ', ['source.matlab']],
  [(0, 6), '=', ['source.matlab', 'keyword.operator.assignment.matlab']],
@@ -33,24 +50,36 @@ An interpreter for grammar files as defined by TextMate and used in VSCode, impl
  [(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]
 ```
 
+## Supported Languages
+- [MATLAB](https://github.com/mathworks/MATLAB-Language-grammar)
+
+## TODO
+- Implement Begin/While pattern, required for other grammars.
+
 ## Sources
 - [Textmate guide](https://www.apeth.com/nonblog/stories/textmatebundle.html)
 - [VSCode Syntax Highlighting guide](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide)
 - [vscode-textmate](https://github.com/microsoft/vscode-textmate)
 - [Macromates texmate](https://macromates.com/textmate/manual/)
 
-## TODO
-- [x] Use yaml in stead of json
-- [x] Grammar initialization with *repository* at any level
-- [x] Proper line-by-line parsing
-- [x] `\G` anchoring
-- [x] Option to *disable* a grammar
-- [x] Option to *applyEndPatternLast*
-- [x] Implement *injections*
-- [ ] Implement Begin/While pattern
 
-## Supported Languages
-- [MATLAB](https://github.com/mathworks/MATLAB-Language-grammar)
+## Development
+
+Install the repository after cloning with [poetry](https://python-poetry.org/)
+
+```bash
+> pip install poetry
+> cd textmate-grammar-python
+> poetry install
+```
+
+Run unit tests
+```bash
+> tox run
+```
+
+Run regression testing against vscode-textmate (will install npm and required packages)
+```bash
+> tox run -e regression
+```
 
-## TextMate 2 features that are not used
-- [Format strings](https://macromates.com/blog/2011/format-strings/)

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "textmate-grammar-python"
-version = "0.1.0"
+version = "0.1.2"
 description = "An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format."
 authors = ["Mark Shui Hu <watermarkhu@gmail.com>"]
 license = "MIT"
@@ -17,6 +17,7 @@ PyYAML = "^6.0.1"
 [tool.poetry.group.test.dependencies]
 pytest = "^7.3.1"
 pytest-icdiff = "^0.8"
+tox = "^4.11.4"
 
 [build-system]
 requires = ["poetry-core"]

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/textmate_grammar/captures.py

@@ -104,7 +104,7 @@ class Capture(object):
                 continue
 
             # Dispatch the parse
-            self.kwargs.pop("leading_chars", None)
+            self.kwargs.pop("greedy", None)
             parsed, captured_elements, _ = parser._parse(
                 self.handler,
                 starting=group_starting,

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/textmate_grammar/grammars/matlab/grammar.yaml

@@ -18,7 +18,7 @@ repository:
     beginCaptures:
       '1':
         name: punctuation.definition.function.anonymous.matlab
-    end: (?=;|(?<!(?:\.\.\..*))\n|%)
+    end: (?=;|(?<!(?:\.{3}.*))\n|%)
     name: meta.function.anonymous.matlab
     patterns:
     - begin: \G(\()
@@ -40,7 +40,7 @@ repository:
       beginCaptures:
         '1':
           name: punctuation.section.group.begin.matlab
-      end: (\))?[^\S\n]*(?=;|(?<!(?:\.\.\..*))\n|%)
+      end: (\))?[^\S\n]*(?=;|(?<!(?:\.{3}.*))\n|%)
       endCaptures:
         '1':
           name: punctuation.section.group.end.matlab
@@ -62,7 +62,7 @@ repository:
       name: meta.for.matlab
       patterns:
       - begin: \G(?!$)
-        end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         name: meta.for.declaration.matlab
         patterns:
         - include: $self
@@ -81,7 +81,7 @@ repository:
       name: meta.if.matlab
       patterns:
       - begin: \G(?!$)
-        end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         name: meta.if.declaration.matlab
         patterns:
         - include: $self
@@ -92,7 +92,7 @@ repository:
         name: meta.elseif.matlab
         patterns:
         - begin: \G(?!$)
-          end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+          end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
           name: meta.elseif.declaration.matlab
           patterns:
           - include: $self
@@ -114,7 +114,7 @@ repository:
       name: meta.for.parallel.matlab
       patterns:
       - begin: \G(?!$)
-        end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         name: meta.for.parallel.declaration.matlab
         patterns:
         - include: $self
@@ -130,7 +130,7 @@ repository:
       name: meta.repeat.parallel.matlab
       patterns:
       - begin: \G(?!$)
-        end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         name: meta.repeat.parallel.declaration.matlab
         patterns:
         - include: $self
@@ -152,11 +152,11 @@ repository:
             name: keyword.control.switch.case.matlab
           '3':
             begin: \G(?!$)
-            end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+            end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
             name: meta.case.declaration.matlab
             patterns:
             - include: $self
-        match: (\s*)(?<=^|[\s,;])(case)\b(.*?)(?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        match: (\s*)(?<=^|[\s,;])(case)\b(.*?)(?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         name: meta.case.matlab
       - captures:
           '2':
@@ -196,7 +196,7 @@ repository:
       name: meta.while.matlab
       patterns:
       - begin: \G
-        end: (?<!\.\.\.)(?:(?=([,;])(?![^(]*\)))|$)
+        end: (?<!\.{3})(?:(?=([,;])(?![^(]*\)))|$)
         endCaptures:
           '1':
             include: $self
@@ -235,7 +235,7 @@ repository:
       name: meta.class.matlab
       patterns:
       - begin: \G
-        end: (?<!\.\.\.)(?=\n)
+        end: (?<!\.{3})(?=\n)
         name: meta.class.declaration.matlab
         patterns:
         - begin: \G(\([^)]*\))?\s*
@@ -262,14 +262,14 @@ repository:
                 - include: '#string'
               - include: '#comments'
               - include: '#line_continuation'
-          end: (?<!\.\.\.)(?=\s*%|\n)
+          end: (?<!\.{3})(?=\s*%|\n)
           patterns:
           - begin: \G\s*([a-zA-Z][a-zA-Z0-9_]*)
             beginCaptures:
               '1':
                 comment: Class name
                 name: entity.name.type.class.matlab
-            end: (?<!\.\.\.)(?=\n)
+            end: (?<!\.{3})(?=\n)
             patterns:
             - comment: Optional inheritance operator
               match: <
@@ -414,7 +414,7 @@ repository:
         - include: '#string_quoted_single'
     comment: "\t\t\t\t\t  1 \t\t\t\t   2\t\t3 \t\t\t\t   4\t\t\t\t\t\t\t\t\t\t\t\t\
       \t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t   "
-    match: (?<=^|[^.]\n|;|,|=)([^\S\n]*)(?# A> )(\b\w+\b)([^\S\n]+)(?# B> )((?!(\+|-|\*|\.\*|\/|\.\/|\\|\.\\|\^|\.\^|==|~=|&|&&|\||\|\||=|:|>|>=|<|<=|\.\.\.)[^\S\n]?)[^\s({=;%][^\n;%]*)
+    match: (?<=^|[^.]\n|;|,|=)([^\S\n]*)(?# A> )(\b\w+\b)([^\S\n]+)(?# B> )((?!(\+|-|\*|\.\*|\/|\.\/|\\|\.\\|\^|\.\^|==|~=|&|&&|\||\|\||=|:|>|>=|<|<=|\.{3})[^\S\n]?)[^\s({=;%][^\n;%]*)
     name: meta.function-call.command.matlab
   comment_block:
     begin: (^[\s]*)(%\{)[^\S\n]*+\n
@@ -525,10 +525,10 @@ repository:
       name: meta.function.matlab
       patterns:
       - begin: \G
-        end: (?<=[\)\n])
+        end: (?<=\))|(?>(?<!\.{3}.*)\n)
         name: meta.function.declaration.matlab
         patterns:
-        - begin: \G(?=.*?=)
+        - begin: \G(?=[^\(]*?(?:=|\[|\.{3}))
           comment: Function output
           contentName: meta.assignment.variable.output.matlab
           end: \s*(=)\s*
@@ -541,21 +541,21 @@ repository:
           - captures:
               '1':
                 name: punctuation.section.assignment.group.end.matlab
-            match: (\])\s*\z
+            match: (\])\s*
           - match: '[a-zA-Z][a-zA-Z0-9_]*'
             name: variable.parameter.output.matlab
           - match: ','
             name: punctuation.separator.parameter.comma.matlab
-          - include: '#comments'
           - include: '#line_continuation'
+          - include: '#comments'
         - comment: Function name
-          match: '[a-zA-Z][a-zA-Z0-9_.]*(?=[^a-zA-Z0-9_.])'
+          match: '[a-zA-Z][a-zA-Z0-9_]*(?>\.[a-zA-Z0-9_]+)*'
           name: entity.name.function.matlab
           patterns:
           - match: \.
             name: punctuation.accessor.dot.matlab
           - include: '#line_continuation'
-        - begin: (?<=[a-zA-Z0-9_])\s*\(
+        - begin: \s*\(
           beginCaptures:
             '0':
               name: punctuation.definition.parameters.begin.matlab
@@ -574,6 +574,7 @@ repository:
             name: punctuation.separator.parameter.comma.matlab
           - include: '#comments'
           - include: '#line_continuation'
+        - include: '#line_continuation'
         - include: '#comments'
       - begin: "(?x)\n\t\t\t\t\t\t\t\t\t(^\\s*)\t\t\t\t\t\t\t\t# Leading whitespace\n\
           \t\t\t\t\t\t\t\t\t(arguments)\\b([^%]*)\n\t\t\t\t\t\t\t\t\t\\s*\n\t\t\t\t\
@@ -617,7 +618,7 @@ repository:
     comment: "\n\t\t\t\tFunction call; matrix or array index (subscript)\n\t\t\t\t\
       We don't include $self here to avoid matching command syntax inside (), [],\
       \ {}\n\t\t\t"
-    end: (\)|(?<!\.\.\..*)\n)
+    end: (\)|(?<!\.{3}.*)\n)
     endCaptures:
       '0':
         name: punctuation.section.parens.end.matlab
@@ -658,7 +659,7 @@ repository:
         name: punctuation.section.parens.begin.matlab
     comment: Index array with expression evaluated to variable name or number
     contentName: meta.parens.matlab
-    end: (\)|(?<!\.\.\..*)\n)
+    end: (\)|(?<!\.{3}.*)\n)
     endCaptures:
       '0':
         name: punctuation.section.parens.end.matlab
@@ -673,7 +674,7 @@ repository:
         - include: '#variables'
     comment: We don't include $self here to avoid matching command syntax inside (),
       [], {}
-    end: (\}|(?<!\.\.\..*)\n)
+    end: (\}|(?<!\.{3}.*)\n)
     patterns:
     - include: '#end_in_parentheses'
     - include: '#rules_before_command_dual'
@@ -685,7 +686,7 @@ repository:
       '2':
         name: comment.continuation.line.matlab
     comment: Line continuations
-    match: (\.\.\.)(.*)$
+    match: (\.{3})(.*)$
     name: meta.continuation.line.matlab
   multiple_assignment:
     begin: \[(?=[^\]]+\]\s*=[a-zA-Z0-9_\s(])
@@ -740,19 +741,19 @@ repository:
       match: (?<!\w)\?(?=\w)
       name: keyword.operator.other.question.matlab
     - comment: Arithmetic operator symbols
-      match: (?<=[a-zA-Z0-9\s])(\+|-|\*|\.\*|/|\./|\\|\.\\|\^|\.\^)(?=[a-zA-Z0-9\s]|(?:\.\.\.))
+      match: (?<=[a-zA-Z0-9\s])(\+|-|\*|\.\*|/|\./|\\|\.\\|\^|\.\^)(?=[a-zA-Z0-9\s]|(?:\.{3}))
       name: keyword.operator.arithmetic.matlab
     - comment: Operator symbols
-      match: (?<=[a-zA-Z0-9\s])(==|~=|&|&&|\||\|\|)(?=[a-zA-Z0-9\s]|(?:\.\.\.))
+      match: (?<=[a-zA-Z0-9\s])(==|~=|&|&&|\||\|\|)(?=[a-zA-Z0-9\s]|(?:\.{3}))
       name: keyword.operator.logical.matlab
     - comment: Assignment operator
       match: (?<=[a-zA-Z0-9\s])(=)(?!=)
       name: keyword.operator.assignment.matlab
     - comment: Colon operator for vector creation/indexing
-      match: (?<=[a-zA-Z0-9_\s(){,]|^):(?=[a-zA-Z0-9_\s()},]|$||(?:\.\.\.))
+      match: (?<=[a-zA-Z0-9_\s(){,]|^):(?=[a-zA-Z0-9_\s()},]|$||(?:\.{3}))
       name: keyword.operator.vector.colon.matlab
     - comment: Comparative operators
-      match: (?<=[a-zA-Z0-9\s])(>|>=|<|<=)(?=[a-zA-Z0-9\s]|(?:\.\.\.))
+      match: (?<=[a-zA-Z0-9\s])(>|>=|<|<=)(?=[a-zA-Z0-9\s]|(?:\.{3}))
       name: keyword.operator.relational.matlab
   parentheses:
     begin: \(
@@ -762,7 +763,7 @@ repository:
     comment: "\n\t\t\t\tOperator precedence in expressions\n\t\t\t\tWe don't include\
       \ $self here to avoid matching command syntax inside (), [], {}\n\t\t\t"
     contentName: meta.parens.matlab
-    end: (\)|(?<!\.\.\..*)\n)
+    end: (\)|(?<!\.{3}.*)\n)
     endCaptures:
       '0':
         name: punctuation.section.parens.end.matlab
@@ -935,7 +936,7 @@ repository:
     comment: "\n\t\t\t\tSuperclass method call; matrix or array index (subscript)\n\
       \t\t\t\tWe don't include $self here to avoid matching command syntax inside\
       \ (), [], {}\n\t\t\t"
-    end: (\)|(?<!\.\.\..*)\n)
+    end: (\)|(?<!\.{3}.*)\n)
     endCaptures:
       '0':
         name: punctuation.section.parens.end.matlab

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/textmate_grammar/handler.py

@@ -1,4 +1,4 @@
-from onigurumacffi import _Pattern as Pattern, _Match as Match
+from onigurumacffi import _Pattern as Pattern, _Match as Match, compile
 
 from .logging import LOGGER
 from .exceptions import FileNotFound, ImpossibleSpan
@@ -16,6 +16,7 @@ class ContentHandler(object):
     are index by a tuple (line_number, line_position). Additionally, the handler contains the
     search method to match a search span against a input oniguruma regex pattern.
     """
+    notLookForwardEOL = compile("(?<!\(\?=[^\(]*)\$")
 
     def __init__(self, source: str) -> None:
         self.source = source
@@ -150,7 +151,7 @@ class ContentHandler(object):
         pattern: Pattern,
         starting: POS,
         boundary: POS | None = None,
-        leading_chars: int = 0,
+        greedy: bool = False,
         **kwargs,
     ) -> (Match | None, tuple[POS, POS] | None):
         """Matches the stream against a capture group.
@@ -167,7 +168,7 @@ class ContentHandler(object):
         """
 
         if pattern._pattern in ["\\z", "\\Z"]:
-            leading_chars = 2
+            greedy = True
 
         # Get line from starting (and boundary) positions
         if boundary and starting[0] == boundary[0]:
@@ -184,7 +185,7 @@ class ContentHandler(object):
         # Check that no charaters are skipped in case ws-only is enabled
         if matching:
             leading_string = line[init_pos : matching.start()]
-            if leading_string and not (leading_chars == 2 or (leading_chars == 1 and leading_string.isspace())):
+            if leading_string and not (greedy or (not greedy and leading_string.isspace())):
                 return None, None
         else:
             return None, None
@@ -197,11 +198,11 @@ class ContentHandler(object):
         if boundary and close_pos > boundary:
             return None, None
 
-        if leading_string and not leading_string.isspace() and leading_chars == 2:
+        if leading_string and not leading_string.isspace() and greedy:
             LOGGER.warning(f"skipping < {leading_string} >", position=start_pos, depth=kwargs.get("depth", 0))
 
         # Include \n in match span if pattern matches on end of line $
-        if "$" in pattern._pattern and matching.end() + 1 == self.line_lengths[starting[0]]:
+        if self.notLookForwardEOL.search(pattern._pattern) and matching.end() + 1 == self.line_lengths[starting[0]]:
             newline_matching = pattern.search(line[:-1])
             if newline_matching and newline_matching.span() == matching.span():
                 close_pos = (starting[0], matching.end() + 1)

{textmate_grammar_python-0.1.0 → textmate_grammar_python-0.1.2}/textmate_grammar/parser.py

@@ -285,7 +285,7 @@ class PatternsParser(ParserHasPatterns):
         handler: ContentHandler,
         starting: POS,
         boundary: POS | None = None,
-        leading_chars: int = 1,
+        greedy: bool = False,
         find_one: bool = True,
         **kwargs,
     ) -> tuple[bool, list[ContentElement], tuple[int, int]]:
@@ -306,7 +306,7 @@ class PatternsParser(ParserHasPatterns):
                     handler,
                     current,
                     boundary=boundary,
-                    leading_chars=leading_chars,
+                    greedy=greedy,
                     **kwargs,
                 )
                 if parsed:
@@ -322,7 +322,7 @@ class PatternsParser(ParserHasPatterns):
                 if find_one:
                     break
 
-            if not parsed and leading_chars != 2:
+            if not parsed and not greedy:
                 # Try again if previously allowed no leading white space charaters, only when multple patterns are to be found
                 options_span, options_elements = {}, {}
                 for parser in patterns:
@@ -330,7 +330,7 @@ class PatternsParser(ParserHasPatterns):
                         handler,
                         current,
                         boundary=boundary,
-                        leading_chars=2,
+                        greedy=True,
                         **kwargs,
                     )
                     if parsed:
@@ -349,8 +349,21 @@ class PatternsParser(ParserHasPatterns):
                     LOGGER.info(
                         f"{self.__class__.__name__} chosen pattern of {parser}", self, current, kwargs.get("depth", 0)
                     )
-                else:
+                elif self != self.language:
                     break
+                else:
+                    remainder = handler.read_line(current)
+                    if not remainder.isspace():
+                        LOGGER.warning(
+                            f"{self.__class__.__name__} remainder of line not parsed: {remainder}", self, current, kwargs.get("depth", 0)
+                        )
+                    if current[0] + 1 <= len(handler.lines):
+                        current = (current[0] + 1, 0)
+                    else:
+                        LOGGER.debug(
+                            f"{self.__class__.__name__} EOF encountered", self, current, kwargs.get("depth", 0)
+                        )
+                        break
 
             if current == starting:
                 LOGGER.warning(
@@ -370,13 +383,12 @@ class PatternsParser(ParserHasPatterns):
                     break
 
         if self.token:
-            closing = boundary if starting == current else current
             elements = [
                 ContentElement(
                     token=self.token,
                     grammar=self.grammar,
-                    content=handler.read_pos(starting, closing),
-                    characters=handler.chars(starting, closing),
+                    content=handler.read_pos(starting, boundary),
+                    characters=handler.chars(starting, boundary),
                     children=elements,
                 )
             ]
@@ -433,7 +445,7 @@ class BeginEndParser(ParserHasPatterns):
         handler: ContentHandler,
         starting: POS,
         boundary: POS | None = None,
-        leading_chars: int = 1,
+        greedy: bool = False,
         **kwargs,
     ) -> (bool, list[ContentElement], tuple[POS, POS] | None):
         """The parse method for grammars for which a begin/end pattern is provided."""
@@ -444,7 +456,7 @@ class BeginEndParser(ParserHasPatterns):
             starting,
             boundary=boundary,
             parsers=self.parsers_begin,
-            leading_chars=leading_chars,
+            greedy=greedy,
             **kwargs
         )
 
@@ -473,7 +485,7 @@ class BeginEndParser(ParserHasPatterns):
             # Try to find patterns first with no leading whitespace charaters allowed
             for parser in patterns:
                 parsed, capture_elements, capture_span = parser._parse(
-                    handler, current, boundary=boundary, leading_chars=1, **kwargs
+                    handler, current, boundary=boundary, greedy=False, **kwargs
                 )
                 if parsed:
                     if parser == self:
@@ -490,7 +502,7 @@ class BeginEndParser(ParserHasPatterns):
                 current,
                 boundary=boundary,
                 parsers=self.parsers_end,
-                leading_chars=1,
+                greedy=False,
                 **kwargs
             )
 
@@ -507,7 +519,7 @@ class BeginEndParser(ParserHasPatterns):
                         handler,
                         current,
                         boundary=boundary,
-                        leading_chars=2,
+                        greedy=True,
                         **kwargs,
                     )
                     if parsed:
@@ -538,7 +550,7 @@ class BeginEndParser(ParserHasPatterns):
                     current,
                     boundary=boundary,
                     parsers=self.parsers_end,
-                    leading_chars=2,
+                    greedy=True,
                     **kwargs
                 )