flow-toon-format 0.9.0b2__py3-none-any.whl

flow_toon_format-0.9.0b2.dist-info/METADATA

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: flow_toon_format
+Version: 0.9.0b2
+Summary: A compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage
+Project-URL: Homepage, https://github.com/toon-format/toon-python
+Project-URL: Repository, https://github.com/toon-format/toon-python
+Project-URL: Documentation, https://github.com/toon-format/spec
+Project-URL: Bug Tracker, https://github.com/toon-format/toon-python/issues
+Author-email: Johann Schopplich <hello@johannschopplich.com>
+License: MIT
+License-File: LICENSE
+Keywords: data-format,llm,serialization,token-efficient,toon
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Requires-Dist: typing-extensions>=4.0.0; python_version < '3.10'
+Description-Content-Type: text/markdown
+
+# TOON Format for Python
+
+[![Tests](https://github.com/toon-format/toon-python/actions/workflows/test.yml/badge.svg)](https://github.com/toon-format/toon-python/actions)
+[![Python Versions](https://img.shields.io/pypi/pyversions/toon_format.svg)](https://pypi.org/project/toon_format/)
+
+> **⚠️ Beta Status (v0.9.x):** This library is in active development and working towards spec compliance. Beta published to PyPI. API may change before 1.0.0 release.
+
+Compact, human-readable serialization format for LLM contexts with **30-60% token reduction** vs JSON. Combines YAML-like indentation with CSV-like tabular arrays. Working towards full compatibility with the [official TOON specification](https://github.com/toon-format/spec).
+
+**Key Features:** Minimal syntax • Tabular arrays for uniform data • Array length validation • Python 3.8+ • Comprehensive test coverage.
+
+```bash
+# Beta published to PyPI - install from source:
+git clone https://github.com/toon-format/toon-python.git
+cd toon-python
+uv sync
+
+# Or install directly from GitHub:
+pip install git+https://github.com/toon-format/toon-python.git
+```
+
+## Quick Start
+
+```python
+from toon_format import encode, decode
+
+# Simple object
+encode({"name": "Alice", "age": 30})
+# name: Alice
+# age: 30
+
+# Tabular array (uniform objects)
+encode([{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}])
+# [2,]{id,name}:
+#   1,Alice
+#   2,Bob
+
+# Decode back to Python
+decode("items[2]: apple,banana")
+# {'items': ['apple', 'banana']}
+```
+
+## CLI Usage
+
+```bash
+# Auto-detect format by extension
+toon input.json -o output.toon      # Encode
+toon data.toon -o output.json       # Decode
+echo '{"x": 1}' | toon -            # Stdin/stdout
+
+# Options
+toon data.json --encode --delimiter "\t" --length-marker
+toon data.toon --decode --no-strict --indent 4
+```
+
+**Options:** `-e/--encode` `-d/--decode` `-o/--output` `--delimiter` `--indent` `--length-marker` `--no-strict`
+
+## API Reference
+
+### `encode(value, options=None)` → `str`
+
+```python
+encode({"id": 123}, {"delimiter": "\t", "indent": 4, "lengthMarker": "#"})
+```
+
+**Options:**
+- `delimiter`: `","` (default), `"\t"`, `"|"`
+- `indent`: Spaces per level (default: `2`)
+- `lengthMarker`: `""` (default) or `"#"` to prefix array lengths
+
+### `decode(input_str, options=None)` → `Any`
+
+```python
+decode("id: 123", {"indent": 2, "strict": True})
+```
+
+**Options:**
+- `indent`: Expected indent size (default: `2`)
+- `strict`: Validate syntax, lengths, delimiters (default: `True`)
+
+### Token Counting & Comparison
+
+Measure token efficiency and compare formats:
+
+```python
+from toon_format import estimate_savings, compare_formats, count_tokens
+
+# Measure savings
+data = {"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}
+result = estimate_savings(data)
+print(f"Saves {result['savings_percent']:.1f}% tokens")  # Saves 42.3% tokens
+
+# Visual comparison
+print(compare_formats(data))
+# Format Comparison
+# ────────────────────────────────────────────────
+# Format      Tokens    Size (chars)
+# JSON            45             123
+# TOON            28              85
+# ────────────────────────────────────────────────
+# Savings: 17 tokens (37.8%)
+
+# Count tokens directly
+toon_str = encode(data)
+tokens = count_tokens(toon_str)  # Uses tiktoken (gpt5/gpt5-mini)
+```
+
+**Requires tiktoken:** `uv add tiktoken` (benchmark features are optional)
+
+## Format Specification
+
+| Type | Example Input | TOON Output |
+|------|---------------|-------------|
+| **Object** | `{"name": "Alice", "age": 30}` | `name: Alice`<br>`age: 30` |
+| **Primitive Array** | `[1, 2, 3]` | `[3]: 1,2,3` |
+| **Tabular Array** | `[{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]` | `[2,]{id,name}:`<br>&nbsp;&nbsp;`1,A`<br>&nbsp;&nbsp;`2,B` |
+| **Mixed Array** | `[{"x": 1}, 42, "hi"]` | `[3]:`<br>&nbsp;&nbsp;`- x: 1`<br>&nbsp;&nbsp;`- 42`<br>&nbsp;&nbsp;`- hi` |
+
+**Quoting:** Only when necessary (empty, keywords, numeric strings, whitespace, structural chars, delimiters)
+
+**Type Normalization:** `Infinity/NaN/Functions` → `null` • `Decimal` → `float` • `datetime` → ISO 8601 • `-0` → `0`
+
+## Development
+
+```bash
+# Setup (requires uv: https://docs.astral.sh/uv/)
+git clone https://github.com/toon-format/toon-python.git
+cd toon-python
+uv sync
+
+# Run tests (792 tests, 91% coverage, 85% enforced)
+uv run pytest --cov=toon_format --cov-report=term
+
+# Code quality
+uv run ruff check src/ tests/        # Lint
+uv run ruff format src/ tests/       # Format
+uv run mypy src/                     # Type check
+```
+
+**CI/CD:** GitHub Actions • Python 3.8-3.14 • Coverage enforcement • PR coverage comments
+
+## Project Status & Roadmap
+
+Following semantic versioning towards 1.0.0:
+
+- **v0.8.x** - Initial code set, tests, documentation ✅
+- **v0.9.x** - Serializer improvements, spec compliance testing, publishing setup (current)
+- **v1.0.0-rc.x** - Release candidates for production readiness
+- **v1.0.0** - First stable release with full spec compliance
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
+
+## Documentation
+
+- [📘 Full Documentation](docs/) - Complete guides and references
+- [🔧 API Reference](docs/api.md) - Detailed function documentation
+- [📋 Format Specification](docs/format.md) - TOON syntax and rules
+- [🤖 LLM Integration](docs/llm-integration.md) - Best practices for LLM usage
+- [📜 TOON Spec](https://github.com/toon-format/spec) - Official specification
+- [🐛 Issues](https://github.com/toon-format/toon-python/issues) - Bug reports and features
+- [🤝 Contributing](CONTRIBUTING.md) - Contribution guidelines
+
+## Contributors
+
+- [Xavi Vinaixa](https://github.com/xaviviro)
+- [David Pirogov](https://github.com/davidpirogov)
+- [Justar](https://github.com/Justar96)
+- [Johann Schopplich](https://github.com/johannschopplich)
+
+## License
+
+MIT License – see [LICENSE](LICENSE) for details

flow_toon_format-0.9.0b2.dist-info/RECORD

@@ -0,0 +1,24 @@
+toon_format/__init__.py,sha256=NBvplQppXveRV0Sy2oG4qogjqoy_2yCYGNOQge0Wp0c,1124
+toon_format/__main__.py,sha256=e4hF2NKL3x7ZC1DG--XXkX90l71KMxCvE5jk_LJlVDM,264
+toon_format/_literal_utils.py,sha256=jQxZAGcY14rWrZ3o5Ra0LwJ9RLiP3uW6aPgbqEHr3jE,2043
+toon_format/_parsing_utils.py,sha256=Pq73LVs8UrDnSvBGOqRYoThUGfyIfI8PgbYi_wvOpzM,5153
+toon_format/_scanner.py,sha256=qIjnfIi58zqKtfLeJEd5__cSjj7Plpoe83aJx_BIZ_g,8387
+toon_format/_string_utils.py,sha256=eFr5fV1QOQiN332AaD486_7ZYNrYdruRFOEYZDzFVyg,4463
+toon_format/_validation.py,sha256=UQzvAAq4ZvuzZdgKNugoXe8STUaA3GPy0sX8o5_XaqE,4129
+toon_format/cli.py,sha256=-kOfsoIkR0AiiqXsTOHvWr1BiWEffCh7UFppJTKBEDw,5540
+toon_format/constants.py,sha256=OGhLhHm-th9hLDrc07Aqv5TgT18SACqYscfpp7rnwtw,1765
+toon_format/decoder.py,sha256=hz_aPaxsStr7DeJ66iZui--HhxFuogtyuDav8tZBtws,24474
+toon_format/encoder.py,sha256=BB6bdgCV9dSQPjmiKpiFn9OMVdexKnvl0Y_G_TbKCJo,1770
+toon_format/encoders.py,sha256=dSplBNgaZGwKH_DiaDiACZybCpwp5FiJNJZocIiBIQo,15995
+toon_format/logging_config.py,sha256=ExOUSyY4wF-MaR6Vle5YQYFFN9I4nU6RY8b5wymfm0o,2886
+toon_format/normalize.py,sha256=j82iVfAd45GYBktwx-89SNAmHyB20MNBNBjCtxMrpC8,7183
+toon_format/primitives.py,sha256=2E_dCaKTygPBY8lQSOrlFVZlj0euSlRrXjIdwKY581c,5181
+toon_format/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+toon_format/types.py,sha256=zB43j8-n5KALM5nKGlfvFB0wvAmIHbTPALn2KVFYCQY,1785
+toon_format/utils.py,sha256=X1jgHerPe4KRpJa2zSd_OuiFNIuybEixFpmE6K3f0ps,6221
+toon_format/writer.py,sha256=rQh3CVwttFkXLHvE0Jd4EntebTAc4YAntNnbuP0K_HQ,1752
+flow_toon_format-0.9.0b2.dist-info/METADATA,sha256=dPnRfnlgHRCzp_cKxmFLdGCUiD3Y5xsltX1oQew2VV0,7443
+flow_toon_format-0.9.0b2.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+flow_toon_format-0.9.0b2.dist-info/entry_points.txt,sha256=zZEiQ-mNPtmkya4I6IEGmxcnPH4hjEyf-mmXXHJfS6Y,46
+flow_toon_format-0.9.0b2.dist-info/licenses/LICENSE,sha256=f_Jm96-xCJNqEx5Xu5zaijPWDcQ5S1Mu3vclEIOkmTg,1197
+flow_toon_format-0.9.0b2.dist-info/RECORD,,

flow_toon_format-0.9.0b2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.28.0
+Root-Is-Purelib: true
+Tag: py3-none-any

flow_toon_format-0.9.0b2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+toon = toon_format.cli:main

flow_toon_format-0.9.0b2.dist-info/licenses/LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Copyright (c) 2025-PRESENT Xavi Vinaixa
+Copyright (c) 2025-PRESENT David Pirogov
+Copyright (c) 2025-PRESENT Justar
+Copyright (c) 2025-PRESENT Johann Schopplich
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

toon_format/__init__.py

@@ -0,0 +1,40 @@
+# Copyright (c) 2025 TOON Format Organization
+# SPDX-License-Identifier: MIT
+"""TOON Format for Python.
+
+Token-Oriented Object Notation (TOON) is a compact, human-readable serialization
+format optimized for LLM contexts. Achieves 30-60% token reduction vs JSON while
+maintaining readability and structure.
+
+This package provides encoding and decoding functionality with 100% compatibility
+with the official TOON specification (v1.3).
+
+Example:
+    >>> from toon_format import encode, decode
+    >>> data = {"name": "Alice", "age": 30}
+    >>> toon = encode(data)
+    >>> print(toon)
+    name: Alice
+    age: 30
+    >>> decode(toon)
+    {'name': 'Alice', 'age': 30}
+"""
+
+from .decoder import ToonDecodeError, decode
+from .encoder import encode
+from .types import DecodeOptions, Delimiter, DelimiterKey, EncodeOptions
+from .utils import compare_formats, count_tokens, estimate_savings
+
+__version__ = "0.9.0-beta.1"
+__all__ = [
+    "encode",
+    "decode",
+    "ToonDecodeError",
+    "Delimiter",
+    "DelimiterKey",
+    "EncodeOptions",
+    "DecodeOptions",
+    "count_tokens",
+    "estimate_savings",
+    "compare_formats",
+]

toon_format/__main__.py

@@ -0,0 +1,13 @@
+# Copyright (c) 2025 TOON Format Organization
+# SPDX-License-Identifier: MIT
+"""CLI entry point for TOON format.
+
+Allows running the package as a module: python -m toon_format
+"""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

toon_format/_literal_utils.py

@@ -0,0 +1,70 @@
+# Copyright (c) 2025 TOON Format Organization
+# SPDX-License-Identifier: MIT
+"""Utilities for detecting literal token types.
+
+This module provides functions to identify different types of literal
+values in TOON syntax, such as booleans, null, and numeric literals.
+Used during decoding to distinguish between literal values and strings.
+"""
+
+from .constants import FALSE_LITERAL, NULL_LITERAL, TRUE_LITERAL
+
+
+def is_boolean_or_null_literal(token: str) -> bool:
+    """Check if a token is a boolean or null literal (`true`, `false`, `null`).
+
+    Args:
+        token: The token to check
+
+    Returns:
+        True if the token is a boolean or null literal
+
+    Examples:
+        >>> is_boolean_or_null_literal("true")
+        True
+        >>> is_boolean_or_null_literal("null")
+        True
+        >>> is_boolean_or_null_literal("hello")
+        False
+    """
+    return token == TRUE_LITERAL or token == FALSE_LITERAL or token == NULL_LITERAL
+
+
+def is_numeric_literal(token: str) -> bool:
+    """Check if a token represents a valid numeric literal.
+
+    Rejects numbers with leading zeros (except `"0"` itself or decimals like `"0.5"`).
+    Per Section 7.3 of the TOON specification.
+
+    Args:
+        token: The token to check
+
+    Returns:
+        True if the token is a valid numeric literal
+
+    Examples:
+        >>> is_numeric_literal("42")
+        True
+        >>> is_numeric_literal("3.14")
+        True
+        >>> is_numeric_literal("0.5")
+        True
+        >>> is_numeric_literal("0123")  # Leading zero - not valid
+        False
+        >>> is_numeric_literal("hello")
+        False
+    """
+    if not token:
+        return False
+
+    # Must not have leading zeros (except for `"0"` itself or decimals like `"0.5"`)
+    if len(token) > 1 and token[0] == "0" and token[1] != ".":
+        return False
+
+    # Check if it's a valid number
+    try:
+        num = float(token)
+        # Reject NaN and infinity
+        return not (num != num or not (-float("inf") < num < float("inf")))
+    except ValueError:
+        return False

toon_format/_parsing_utils.py

@@ -0,0 +1,167 @@
+# Copyright (c) 2025 TOON Format Organization
+# SPDX-License-Identifier: MIT
+"""Parsing utilities for quote-aware string processing.
+
+This module provides utilities for parsing TOON strings while respecting
+quoted sections and escape sequences. Used extensively in decoder for
+finding delimiters and structural characters outside of quoted strings.
+"""
+
+from typing import Iterator, List, Tuple
+
+from .constants import BACKSLASH, DOUBLE_QUOTE
+
+
+def iter_unquoted(line: str, start: int = 0) -> Iterator[Tuple[int, str, bool]]:
+    """Iterate over characters in a line, tracking quote state.
+
+    This is the core utility for quote-aware string processing. It handles:
+    - Tracking quote boundaries
+    - Skipping escaped characters within quotes
+    - Yielding (index, character, is_quoted) tuples
+
+    Args:
+        line: The line to iterate over
+        start: Starting position (default: 0)
+
+    Yields:
+        Tuple of (index, char, is_quoted) for each character
+
+    Examples:
+        >>> list(iter_unquoted('a"b:c"d'))
+        [(0, 'a', False), (1, '"', False), (2, 'b', True), (3, ':', True),
+         (4, 'c', True), (5, '"', True), (6, 'd', False)]
+    """
+    in_quotes = False
+    i = start
+
+    while i < len(line):
+        char = line[i]
+
+        if char == DOUBLE_QUOTE:
+            # Yield quote with current state, THEN toggle for next char
+            yield (i, char, in_quotes)
+            in_quotes = not in_quotes
+        elif char == BACKSLASH and i + 1 < len(line) and in_quotes:
+            # Escaped character - yield backslash, then skip and yield next char
+            yield (i, char, True)
+            i += 1
+            if i < len(line):
+                yield (i, line[i], True)
+        else:
+            yield (i, char, in_quotes)
+
+        i += 1
+
+
+def find_unquoted_char(line: str, target_char: str, start: int = 0) -> int:
+    """Find first occurrence of target character outside of quoted strings.
+
+    Args:
+        line: Line to search
+        target_char: Character to find
+        start: Starting position (default: 0)
+
+    Returns:
+        Index of character, or -1 if not found
+
+    Examples:
+        >>> find_unquoted_char('a:b"c:d"e', ':')
+        1
+        >>> find_unquoted_char('a"b:c"d:e', ':', 0)
+        7
+        >>> find_unquoted_char('"a:b":c', ':', 0)
+        5
+    """
+    for i, char, is_quoted in iter_unquoted(line, start):
+        if char == target_char and not is_quoted:
+            return i
+    return -1
+
+
+def parse_delimited_values(line: str, delimiter: str) -> List[str]:
+    """Parse delimiter-separated values, respecting quotes and escapes.
+
+    This function splits a line on the delimiter, but only at unquoted positions.
+    Quotes and escape sequences within quoted sections are preserved.
+
+    Args:
+        line: Line content
+        delimiter: Active delimiter (e.g., ',', '\\t', '|')
+
+    Returns:
+        List of token strings (with quotes and escapes preserved)
+
+    Examples:
+        >>> parse_delimited_values('a,b,c', ',')
+        ['a', 'b', 'c']
+        >>> parse_delimited_values('a,"b,c",d', ',')
+        ['a', '"b,c"', 'd']
+        >>> parse_delimited_values('"a,b",c', ',')
+        ['"a,b"', 'c']
+    """
+    tokens: List[str] = []
+    current: List[str] = []
+
+    for i, char, is_quoted in iter_unquoted(line):
+        if char == delimiter and not is_quoted:
+            # Split on unquoted delimiter
+            tokens.append("".join(current))
+            current = []
+        else:
+            current.append(char)
+
+    # Add final token (always add, even if empty, to handle trailing delimiters)
+    if current or tokens:
+        tokens.append("".join(current))
+
+    return tokens
+
+
+def split_at_unquoted_char(line: str, target_char: str) -> Tuple[str, str]:
+    """Split a line at the first unquoted occurrence of target character.
+
+    Args:
+        line: Line content
+        target_char: Character to split on
+
+    Returns:
+        Tuple of (before, after) strings
+
+    Raises:
+        ValueError: If target character not found outside quotes
+
+    Examples:
+        >>> split_at_unquoted_char('key: value', ':')
+        ('key', ' value')
+        >>> split_at_unquoted_char('"key:1": value', ':')
+        ('"key:1"', ' value')
+    """
+    idx = find_unquoted_char(line, target_char)
+    if idx == -1:
+        raise ValueError(f"Character '{target_char}' not found outside quotes")
+    return (line[:idx], line[idx + 1 :])
+
+
+def find_first_unquoted(line: str, chars: List[str], start: int = 0) -> Tuple[int, str]:
+    """Find the first occurrence of any character in chars, outside quotes.
+
+    Args:
+        line: Line to search
+        chars: List of characters to search for
+        start: Starting position (default: 0)
+
+    Returns:
+        Tuple of (index, character) for first match, or (-1, '') if none found
+
+    Examples:
+        >>> find_first_unquoted('a:b,c', [':', ','])
+        (1, ':')
+        >>> find_first_unquoted('a"b:c",d', [':', ','])
+        (7, ',')
+    """
+    char_set = set(chars)
+    for i, char, is_quoted in iter_unquoted(line, start):
+        if char in char_set and not is_quoted:
+            return (i, char)
+    return (-1, "")