links-notation 0.27.1__tar.gz → 0.28.0__tar.gz

{links_notation-0.27.1/links_notation.egg-info → links_notation-0.28.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: links-notation
-Version: 0.27.1
+Version: 0.28.0
 Summary: Python implementation of the Links Notation parser
 Author-email: "Link.Foundation" <drakonard@gmail.com>
 License-Expression: Unlicense
@@ -111,6 +111,50 @@ links = parser.parse(text)
 # Produces: (3: papa loves mama)
 ```
 
+### How a Document Was Written (CST)
+
+Parsing keeps how every ID was written. Quoting never changes a reference:
+`1` and `'1'` are the same reference, so equality, `str()`, `repr()` and
+`format()` stay as they were. Each parsed link also carries `raw_id`, the ID
+exactly as written, and `quote`, the delimiter it was written between:
+
+```python
+from links_notation import Parser
+
+[link] = Parser().parse("(t 1 '1')")
+_, bare, quoted = link.values
+bare == quoted                # True: the same reference
+(bare.raw_id, bare.quote)     # ('1', None)
+(quoted.raw_id, quoted.quote) # ("'1'", "'")
+link.format()                 # '(t 1 1)'
+```
+
+`raw_id` is `None` for links without an ID, links built in code, and links
+made by indentation path combination. Use `Parser(syntax=False)` (AST mode) to
+leave it out. `reference_quote(raw)` reads the delimiter of one written
+reference.
+
+`parse_cst(input)` returns the lossless concrete syntax tree. Every character,
+including comments, blank lines, indentation, quotes and line endings, belongs
+to exactly one token, so the original text is always recovered:
+
+```python
+cst = Parser().parse_cst("(t 1 '1') # note\n")
+str(cst)  # "(t 1 '1') # note\n"
+[(token.kind, token.text) for token in cst.tokens()]
+# [('open_paren', '('), ('reference', 't'), ('whitespace', ' '), ...]
+cst.to_s_expression()
+# (document (line (group open_paren:"(" (line reference:"t" ...)
+#   close_paren:")") whitespace:" " comment:"# note" newline:"\n"))
+```
+
+A `document` node holds `line` nodes. A line holds its tokens up to and
+including its line ending. A `group` node holds `(`, the lines inside it, and
+`)`. Reference tokens carry `value` (what the reference means) and `quote`.
+`parse_cst` parses the document first, so it refuses what `parse` refuses,
+and it also refuses unbalanced parentheses. See the
+[issue 364 case study](../docs/case-studies/issue-364/README.md).
+
 ### Streaming Parsing
 
 `StreamParser` accepts arbitrary chunks and emits only complete top-level
@@ -136,12 +180,15 @@ stream also exposes `position`, `drain()`, `reset()`, and a buffer-size limit.
 
 The main parser class for Links Notation.
 
-- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True)`:
+- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True, syntax: bool = True)`:
   Create a parser; with `comments=False` a `#` is an ordinary character instead
-  of the start of a comment
+  of the start of a comment; with `syntax=False` (AST mode) parsed links leave
+  `raw_id` and `quote` as `None`
 - `parse(input_text: str) -> List[Link]`: Parse Links Notation text into Link
   objects; raises `ParseError` when the text does not parse or nests links
   deeper than `max_depth`
+- `parse_cst(input_text: str) -> CstNode`: Read the lossless concrete syntax
+  tree; `str()` of it is the input
 - `DEFAULT_MAX_DEPTH`: The default `max_depth`, 64, the same in every
   implementation
 
@@ -181,6 +228,9 @@ Represents a link in Links Notation.
 - `format(less_parentheses: bool = False) -> str`: Format as string
 - `simplify() -> Link`: Simplify link structure
 - `combine(other: Link) -> Link`: Combine with another link
+- `raw_id`: The ID exactly as the parsed document wrote it, or `None`; ignored
+  by `==`, `str()`, `repr()` and `format()`
+- `quote`: `'`, `"` or `` ` `` when `raw_id` is quoted, otherwise `None`
 
 ### format_links
 

{links_notation-0.27.1 → links_notation-0.28.0}/README.md

@@ -77,6 +77,50 @@ links = parser.parse(text)
 # Produces: (3: papa loves mama)
 ```
 
+### How a Document Was Written (CST)
+
+Parsing keeps how every ID was written. Quoting never changes a reference:
+`1` and `'1'` are the same reference, so equality, `str()`, `repr()` and
+`format()` stay as they were. Each parsed link also carries `raw_id`, the ID
+exactly as written, and `quote`, the delimiter it was written between:
+
+```python
+from links_notation import Parser
+
+[link] = Parser().parse("(t 1 '1')")
+_, bare, quoted = link.values
+bare == quoted                # True: the same reference
+(bare.raw_id, bare.quote)     # ('1', None)
+(quoted.raw_id, quoted.quote) # ("'1'", "'")
+link.format()                 # '(t 1 1)'
+```
+
+`raw_id` is `None` for links without an ID, links built in code, and links
+made by indentation path combination. Use `Parser(syntax=False)` (AST mode) to
+leave it out. `reference_quote(raw)` reads the delimiter of one written
+reference.
+
+`parse_cst(input)` returns the lossless concrete syntax tree. Every character,
+including comments, blank lines, indentation, quotes and line endings, belongs
+to exactly one token, so the original text is always recovered:
+
+```python
+cst = Parser().parse_cst("(t 1 '1') # note\n")
+str(cst)  # "(t 1 '1') # note\n"
+[(token.kind, token.text) for token in cst.tokens()]
+# [('open_paren', '('), ('reference', 't'), ('whitespace', ' '), ...]
+cst.to_s_expression()
+# (document (line (group open_paren:"(" (line reference:"t" ...)
+#   close_paren:")") whitespace:" " comment:"# note" newline:"\n"))
+```
+
+A `document` node holds `line` nodes. A line holds its tokens up to and
+including its line ending. A `group` node holds `(`, the lines inside it, and
+`)`. Reference tokens carry `value` (what the reference means) and `quote`.
+`parse_cst` parses the document first, so it refuses what `parse` refuses,
+and it also refuses unbalanced parentheses. See the
+[issue 364 case study](../docs/case-studies/issue-364/README.md).
+
 ### Streaming Parsing
 
 `StreamParser` accepts arbitrary chunks and emits only complete top-level
@@ -102,12 +146,15 @@ stream also exposes `position`, `drain()`, `reset()`, and a buffer-size limit.
 
 The main parser class for Links Notation.
 
-- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True)`:
+- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True, syntax: bool = True)`:
   Create a parser; with `comments=False` a `#` is an ordinary character instead
-  of the start of a comment
+  of the start of a comment; with `syntax=False` (AST mode) parsed links leave
+  `raw_id` and `quote` as `None`
 - `parse(input_text: str) -> List[Link]`: Parse Links Notation text into Link
   objects; raises `ParseError` when the text does not parse or nests links
   deeper than `max_depth`
+- `parse_cst(input_text: str) -> CstNode`: Read the lossless concrete syntax
+  tree; `str()` of it is the input
 - `DEFAULT_MAX_DEPTH`: The default `max_depth`, 64, the same in every
   implementation
 
@@ -147,6 +194,9 @@ Represents a link in Links Notation.
 - `format(less_parentheses: bool = False) -> str`: Format as string
 - `simplify() -> Link`: Simplify link structure
 - `combine(other: Link) -> Link`: Combine with another link
+- `raw_id`: The ID exactly as the parsed document wrote it, or `None`; ignored
+  by `==`, `str()`, `repr()` and `format()`
+- `quote`: `'`, `"` or `` ` `` when `raw_id` is quoted, otherwise `None`
 
 ### format_links
 

{links_notation-0.27.1 → links_notation-0.28.0}/README.ru.md

@@ -77,6 +77,51 @@ links = parser.parse(text)
 # Результат: (3: папа любит маму)
 ```
 
+### Как документ был записан (CST)
+
+Парсер сохраняет, как была записана каждая ссылка. Кавычки не меняют ссылку:
+`1` и `'1'` - одна и та же ссылка, поэтому сравнение, `str()`, `repr()` и
+`format()` остаются прежними. Каждая разобранная связь хранит `raw_id` - ID в
+точности так, как он записан, и `quote` - кавычку, в которую он заключён:
+
+```python
+from links_notation import Parser
+
+[link] = Parser().parse("(t 1 '1')")
+_, bare, quoted = link.values
+bare == quoted                # True: это одна ссылка
+(bare.raw_id, bare.quote)     # ('1', None)
+(quoted.raw_id, quoted.quote) # ("'1'", "'")
+link.format()                 # '(t 1 1)'
+```
+
+`raw_id` равен `None` у связей без ID, у связей, созданных в коде, и у связей,
+построенных объединением путей по отступам. `Parser(syntax=False)` (режим AST)
+не сохраняет его. `reference_quote(raw)` возвращает кавычку одной записанной
+ссылки.
+
+`parse_cst(input)` возвращает конкретное синтаксическое дерево без потерь:
+каждый символ, включая комментарии, пустые строки, отступы, кавычки и
+переводы строк, принадлежит ровно одному токену, поэтому исходный текст
+всегда восстанавливается:
+
+```python
+cst = Parser().parse_cst("(t 1 '1') # note\n")
+str(cst)  # "(t 1 '1') # note\n"
+[(token.kind, token.text) for token in cst.tokens()]
+# [('open_paren', '('), ('reference', 't'), ('whitespace', ' '), ...]
+cst.to_s_expression()
+# (document (line (group open_paren:"(" (line reference:"t" ...)
+#   close_paren:")") whitespace:" " comment:"# note" newline:"\n"))
+```
+
+Узел `document` содержит узлы `line`; строка содержит свои токены вместе с
+переводом строки; узел `group` содержит `(`, строки внутри и `)`. Токены
+ссылок хранят `value` (что означает ссылка) и `quote`. `parse_cst` сначала
+разбирает документ, поэтому отклоняет то, что отклоняет `parse`, а также
+несбалансированные скобки. Подробности - в
+[разборе issue 364](../docs/case-studies/issue-364/README.md).
+
 ### Потоковый разбор
 
 `StreamParser` принимает произвольные порции данных и выдаёт только завершённые
@@ -103,12 +148,15 @@ for link in parse_chunks(["одна свя", "зь\nвторая связь"]):
 
 Основной класс парсера для Links Notation.
 
-- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True)`:
+- `__init__(max_input_size: int = 10 * 1024 * 1024, max_depth: int = DEFAULT_MAX_DEPTH, comments: bool = True, syntax: bool = True)`:
   создать парсер; при `comments=False` `#` — обычный символ, а не начало
-  комментария
+  комментария; при `syntax=False` (режим AST) у разобранных связей `raw_id` и
+  `quote` равны `None`
 - `parse(input_text: str) -> List[Link]`: Парсинг текста Links Notation в
   объекты Link; выбрасывает `ParseError`, если текст не удалось разобрать или
   связи в нем вложены глубже `max_depth`
+- `parse_cst(input_text: str) -> CstNode`: конкретное синтаксическое дерево без
+  потерь; `str()` от него равен входному тексту
 - `DEFAULT_MAX_DEPTH`: значение `max_depth` по умолчанию, 64, одинаковое во всех
   реализациях
 
@@ -149,6 +197,9 @@ Nesting too deep at line 1, column 4: nesting depth exceeds the maximum of 3
 - `format(less_parentheses: bool = False) -> str`: Форматирование в строку
 - `simplify() -> Link`: Упрощение структуры связи
 - `combine(other: Link) -> Link`: Объединение с другой связью
+- `raw_id`: ID в точности так, как он записан в разобранном документе, или
+  `None`; не учитывается в `==`, `str()`, `repr()` и `format()`
+- `quote`: `'`, `"` или `` ` ``, если `raw_id` записан в кавычках, иначе `None`
 
 ### format_links
 

{links_notation-0.27.1 → links_notation-0.28.0}/links_notation/__init__.py

@@ -22,10 +22,12 @@ from .binary import (
 from .binary import format_document as format_binary_document
 from .binary import format_reference as format_binary_reference
 from .comments import strip_comments
+from .cst import CstKind, CstNode, build_cst
 from .format_config import FormatConfig
 from .formatter import format_links
 from .link import Link
 from .parser import DEFAULT_MAX_DEPTH, ParseError, Parser
+from .quotes import reference_quote
 from .reference_literal import decode_reference_literal, encode_reference_literal
 from .stream_parser import StreamParseError, StreamParser, StreamPosition, parse_async_chunks, parse_chunks
 
@@ -78,5 +80,9 @@ __all__ = [
     "parse_chunks",
     "FormatConfig",
     "strip_comments",
+    "CstKind",
+    "CstNode",
+    "build_cst",
+    "reference_quote",
     "__version__",
 ]

links_notation-0.28.0/links_notation/cst.py

@@ -0,0 +1,251 @@
+"""
+The concrete syntax tree of a Links Notation document.
+
+Every character of a document belongs to exactly one token, so the text of the
+tree is the document itself, comments, blank lines, indentation, quotes and
+line endings included. A document is a list of ``line`` nodes; a line holds the
+tokens up to and including its line ending, and a parenthesized ``group`` node
+holds its own parentheses and the lines written between them. A group is part
+of the line it opens on, however many lines it spans.
+
+The tree records how a document was written; :meth:`Parser.parse` reads what it
+means. They agree because a document is parsed before its tree is built (see
+:meth:`Parser.parse_cst`), and the tokens are read with the same rules the
+parser uses: comments are found by :func:`strip_comments`, delimited references
+by the same reader, and versioned literals by the same pattern.
+"""
+
+import re
+from typing import List, Optional, Tuple
+
+from .comments import strip_comments
+from .parser import ParseError
+from .quotes import QUOTE_CHARS, _parse_quoted_string_at
+from .reference_literal import decode_reference_literal
+
+
+class CstKind:
+    """The kinds of nodes and tokens of a concrete syntax tree."""
+
+    #: The whole document: its lines.
+    DOCUMENT = "document"
+    #: One line: its tokens and groups, up to and including its line ending.
+    LINE = "line"
+    #: A parenthesized group: ``(``, the lines inside it, and ``)``.
+    GROUP = "group"
+    #: A run of spaces and tabs.
+    WHITESPACE = "whitespace"
+    #: One line ending: LF, CRLF or CR.
+    NEWLINE = "newline"
+    #: A comment, from ``#`` up to the line ending.
+    COMMENT = "comment"
+    #: ``(``
+    OPEN_PAREN = "open_paren"
+    #: ``)``
+    CLOSE_PAREN = "close_paren"
+    #: ``:``
+    COLON = "colon"
+    #: A bare, quoted or versioned literal reference.
+    REFERENCE = "reference"
+
+
+_NODE_KINDS = frozenset((CstKind.DOCUMENT, CstKind.LINE, CstKind.GROUP))
+
+#: How a token's text is escaped in an S-expression.
+_ESCAPES = {"\\": "\\\\", '"': '\\"', "\n": "\\n", "\r": "\\r", "\t": "\\t"}
+_ESCAPED = re.compile(r'[\\"\n\r\t]')
+
+#: A versioned literal.
+_LITERAL = re.compile(r"~[0-9]+\{[^} \t\r\n()]*\}")
+
+#: What ends a bare reference.
+_BARE_REFERENCE_END = frozenset(" \t\r\n(:)")
+
+
+class CstNode:
+    """One node or token of a concrete syntax tree."""
+
+    def __init__(
+        self,
+        kind: str,
+        text: str = "",
+        children: Optional[List["CstNode"]] = None,
+        value: Optional[str] = None,
+        quote: Optional[str] = None,
+    ):
+        """
+        Initialize a node or token.
+
+        Args:
+            kind: One of the :class:`CstKind` values
+            text: A token's text exactly as written
+            children: A node's children, in document order
+            value: What a reference token means
+            quote: The delimiter of a quoted reference token
+        """
+        #: One of the :class:`CstKind` values.
+        self.kind = kind
+        #: The children of a node, in document order; empty for a token.
+        self.children: List["CstNode"] = children if children is not None else []
+        #: What a reference token means, its delimiters and escapes resolved
+        #: (the ID a parsed link has); None for every other node and token.
+        self.value = value
+        #: The delimiter a quoted reference token was written between (``'``,
+        #: ``"`` or a backtick); None for a bare or literal reference and every
+        #: other kind.
+        self.quote = quote
+        self._text = text
+
+    @property
+    def is_token(self) -> bool:
+        """Whether this is a token rather than a node with children."""
+        return self.kind not in _NODE_KINDS
+
+    @property
+    def text(self) -> str:
+        """The exact text this node or token was read from."""
+        if self.is_token:
+            return self._text
+        return "".join(child.text for child in self.children)
+
+    def __str__(self) -> str:
+        """The exact text this node or token was read from."""
+        return self.text
+
+    def __repr__(self) -> str:
+        """Developer-friendly representation."""
+        return f"CstNode({self.to_s_expression()})"
+
+    def tokens(self) -> List["CstNode"]:
+        """Every token of this subtree, in document order."""
+        if self.is_token:
+            return [self]
+        return [token for child in self.children for token in child.tokens()]
+
+    def to_s_expression(self) -> str:
+        """
+        The tree as an S-expression.
+
+        A node is written ``(kind child ...)`` and a token ``kind:"text"``, the
+        text escaped as a JSON string would escape ``\\``, ``"``, line feeds,
+        carriage returns and tabs. Two trees are the same exactly when their
+        S-expressions are.
+
+        >>> from links_notation import Parser
+        >>> Parser().parse_cst("a '1'").to_s_expression()
+        '(document (line reference:"a" whitespace:" " reference:"\\'1\\'"))'
+        """
+        if self.is_token:
+            return f"{self.kind}:{_quote_text(self._text)}"
+        return "(" + " ".join([self.kind] + [child.to_s_expression() for child in self.children]) + ")"
+
+
+def _quote_text(text: str) -> str:
+    """Write ``text`` between double quotes, escaped."""
+    return '"' + _ESCAPED.sub(lambda match: _ESCAPES[match.group()], text) + '"'
+
+
+class _Container:
+    """An open document or group, with the line being filled in it."""
+
+    def __init__(self, node: CstNode):
+        self.node = node
+        self.line: Optional[CstNode] = None
+
+
+def build_cst(input_text: str, comments: bool = True) -> CstNode:
+    """
+    The concrete syntax tree of a document.
+
+    The document is not validated here; :meth:`Parser.parse_cst` parses it
+    first, so use that unless the document is already known to parse.
+
+    Args:
+        input_text: The document as written
+        comments: Whether ``#`` starts a comment (default: True)
+
+    Returns:
+        The ``document`` node
+
+    Raises:
+        ParseError: If the parentheses are unbalanced
+    """
+    # References are read in the document the parser reads: the one with its
+    # comments blanked, which keeps every position of the one written.
+    prepared = strip_comments(input_text) if comments else input_text
+    document = CstNode(CstKind.DOCUMENT)
+    # The open containers, innermost last.
+    containers = [_Container(document)]
+
+    def add(node: CstNode) -> None:
+        container = containers[-1]
+        if container.line is None:
+            container.line = CstNode(CstKind.LINE)
+            container.node.children.append(container.line)
+        container.line.children.append(node)
+
+    def token(kind: str, start: int, end: int) -> CstNode:
+        return CstNode(kind, text=input_text[start:end])
+
+    length = len(input_text)
+    position = 0
+    while position < length:
+        character = input_text[position]
+        end = position + 1
+
+        if comments and character == "#" and prepared[position] != "#":
+            # A blanked "#" opens a comment, which runs to the line ending.
+            while end < length and input_text[end] not in ("\n", "\r"):
+                end += 1
+            add(token(CstKind.COMMENT, position, end))
+        elif character in ("\r", "\n"):
+            if character == "\r" and end < length and input_text[end] == "\n":
+                end += 1
+            add(token(CstKind.NEWLINE, position, end))
+            containers[-1].line = None
+        elif character in (" ", "\t"):
+            while end < length and input_text[end] in (" ", "\t"):
+                end += 1
+            add(token(CstKind.WHITESPACE, position, end))
+        elif character == "(":
+            group = CstNode(CstKind.GROUP, children=[token(CstKind.OPEN_PAREN, position, end)])
+            add(group)
+            containers.append(_Container(group))
+        elif character == ")":
+            if len(containers) == 1:
+                raise ParseError(f'Unbalanced ")" at offset {position}', offset=position)
+            containers.pop().node.children.append(token(CstKind.CLOSE_PAREN, position, end))
+        elif character == ":":
+            add(token(CstKind.COLON, position, end))
+        else:
+            reference, end = _reference(input_text, prepared, position)
+            add(reference)
+        position = end
+
+    if len(containers) != 1:
+        raise ParseError('Unbalanced "(" at the end of the document')
+    return document
+
+
+def _reference(input_text: str, prepared: str, start: int) -> Tuple[CstNode, int]:
+    """
+    The reference token that starts at ``start``, read as the parser reads it,
+    and where it ends.
+    """
+    if prepared[start] == "~":
+        literal = _LITERAL.match(prepared, start)
+        if literal is not None:
+            end = literal.end()
+            text = input_text[start:end]
+            return CstNode(CstKind.REFERENCE, text=text, value=decode_reference_literal(text)), end
+    if prepared[start] in QUOTE_CHARS:
+        quoted = _parse_quoted_string_at(prepared, start)
+        if quoted is not None:
+            value, end = quoted
+            reference = CstNode(CstKind.REFERENCE, text=input_text[start:end], value=value, quote=input_text[start])
+            return reference, end
+    end = start + 1
+    while end < len(prepared) and prepared[end] not in _BARE_REFERENCE_END:
+        end += 1
+    text = input_text[start:end]
+    return CstNode(CstKind.REFERENCE, text=text, value=text), end

{links_notation-0.27.1 → links_notation-0.28.0}/links_notation/link.py

@@ -4,6 +4,7 @@ Link class representing a Lino link with optional ID and values.
 
 from typing import TYPE_CHECKING, List, Optional, Union
 
+from .quotes import reference_quote
 from .reference_literal import format_reference
 
 if TYPE_CHECKING:
@@ -31,6 +32,36 @@ class Link:
         self.id = link_id
         self.values = values if values is not None else []
         self._is_from_path_combination = False
+        self._raw_id: Optional[str] = None
+
+    @property
+    def raw_id(self) -> Optional[str]:
+        """
+        The ID exactly as the parsed document wrote it: ``'1'`` for a quoted
+        ``1``, ``1`` for a bare one, ``~1{31}`` for a versioned literal.
+
+        None for a link without an ID, a link built in code, a link made by
+        combining the lines of an indented document, and every link read by a
+        parser created with ``syntax=False``. It takes no part in equality,
+        ``str()``, ``repr()`` or ``format()``.
+        """
+        return self._raw_id
+
+    @raw_id.setter
+    def raw_id(self, raw: Optional[str]) -> None:
+        if raw is not None and not isinstance(raw, str):
+            raise TypeError("raw_id must be a string or None")
+        self._raw_id = raw
+
+    @property
+    def quote(self) -> Optional[str]:
+        """
+        The delimiter the ID was written between: ``'``, ``"`` or a backtick
+        when it was quoted, and None when it was bare, a versioned literal, or
+        not read from a document (see :attr:`raw_id`). ``1`` and ``'1'`` stay
+        equal links; this only says how each was written.
+        """
+        return reference_quote(self._raw_id)
 
     def __str__(self) -> str:
         """String representation using standard formatting."""