datoon 2.0.0__tar.gz → 3.0.0__tar.gz

{datoon-2.0.0 → datoon-3.0.0}/.gitignore

@@ -40,6 +40,7 @@ benchmarks/results/**/*.json
 # Local Codex/Claude artifacts
 .agents/skills/
 .claude/
+.omc/
 skills-lock.json
 
 # Generated by setuptools-scm during local builds/tests

{datoon-2.0.0 → datoon-3.0.0}/CHANGELOG.md

@@ -1,3 +1,22 @@
+# [3.0.0](https://github.com/andrii-su/datoon/compare/v2.0.0...v3.0.0) (2026-11-11)
+
+
+* feat!: selectable token counters (auto, chars, tiktoken, claude, hf, callable) ([cab2bfb](https://github.com/andrii-su/datoon/commit/cab2bfb9f1106f2a4cf40122588249acb0371f94))
+
+
+### Bug Fixes
+
+* **mcp:** accept only local token counters as a tool argument ([1f2e54d](https://github.com/andrii-su/datoon/commit/1f2e54d4f83ea266590933f0762019d2d37f7270))
+
+
+### BREAKING CHANGES
+
+* `ConversionConfig.token_encoding` is replaced by
+`tokenizer`; the CLI flag `--encoding` and the MCP `encoding` argument are
+replaced by `--tokenizer` / `tokenizer` (default from `DATOON_TOKENIZER`,
+else `auto`). `ConversionReport` gains a `tokenizer` field. Every counter
+returns at least 1, including for empty text.
+
 # [2.0.0](https://github.com/andrii-su/datoon/compare/v1.9.1...v2.0.0) (2026-11-11)
 
 

{datoon-2.0.0 → datoon-3.0.0}/CLAUDE.md

@@ -32,6 +32,7 @@ datoon/
 │   ├── analyzer.py                   # Payload structure suitability checks
 │   ├── converter.py                  # JSON normalization, TOON encoder call, policy gating
 │   ├── models.py                     # Config/report/outcome dataclasses
+│   ├── tokenizers.py                 # Token counter specs and estimates
 │   ├── cli.py                        # CLI entrypoint
 │   └── mcp_server.py                 # MCP server tools
 │
@@ -68,6 +69,7 @@ Edit these files directly:
 | Conversion policy | `src/datoon/converter.py` |
 | Payload analysis | `src/datoon/analyzer.py` |
 | Public data contracts | `src/datoon/models.py` |
+| Token counters | `src/datoon/tokenizers.py` |
 | CLI behavior | `src/datoon/cli.py` |
 | MCP behavior | `src/datoon/mcp_server.py` |
 | Skill behavior | `skills/datoon/SKILL.md` |
@@ -109,7 +111,8 @@ It runs on pull requests and pushes to `main`. The old auto-sync-on-main flow wa
 - `--force` is for controlled experiments and should bypass gating, not hide conversion errors.
 - Auto mode skips before conversion when analysis rejects the payload, and skips after conversion when estimated savings are below `min_savings_ratio`.
 - Keep skip reasons user-facing, specific, and stable enough for CLI/MCP troubleshooting.
-- Token counts are estimates. Be precise about whether a number is payload-token estimate, visible output estimate, or real model usage.
+- Token counts are estimates, whichever tokenizer produced them (`claude:` counts included). Be precise about whether a number is payload-token estimate, visible output estimate, or real model usage, and name the tokenizer it came from.
+- Only the `auto` tokenizer falls back silently; an explicit spec with a missing package or credentials must raise `DatoonError`, in auto and force mode alike.
 - Benchmarks in README must come from committed scripts/artifacts.
 - Keep generated caches out of commits: `.venv/`, `.pytest_cache/`, `.ruff_cache/`, `__pycache__/`, local benchmark result JSON under `benchmarks/results/`.
 

{datoon-2.0.0 → datoon-3.0.0}/INSTALL.md

@@ -42,9 +42,29 @@ Optional extras:
 
 ```bash
 pip install "datoon[tokens]"  # tiktoken token estimates
+pip install "datoon[claude]"  # Claude token counts via the Anthropic API
+pip install "datoon[hf]"      # Hugging Face tokenizers
 pip install "datoon[mcp]"     # MCP server dependencies
 ```
 
+### Choosing a tokenizer
+
+Select the token counter with `--tokenizer <spec>` (CLI), the `tokenizer` argument (MCP tools), or `DATOON_TOKENIZER`; the default is `auto`. The MCP tools accept only local counters (`auto`, `chars`, `tiktoken`) per call; `claude:` and `hf:` reach the network, so only the server operator can enable them through `DATOON_TOKENIZER`.
+
+| Spec | Counts with | Needs |
+|---|---|---|
+| `auto` | `tiktoken:o200k_base` when tiktoken is installed, else `chars:3` | — |
+| `chars` / `chars:<N>` | `ceil(len(text) / N)`, default `N = 3` | — |
+| `tiktoken` / `tiktoken:<encoding>` | tiktoken, default `o200k_base` | `datoon[tokens]` |
+| `claude:<model-id>` | Anthropic token counting API | `datoon[claude]`, Anthropic credentials (e.g. `ANTHROPIC_API_KEY`) |
+| `hf:<repo-id>` | Hugging Face `tokenizers` tokenizer | `datoon[hf]` |
+
+Local counters suit runtime gating; `claude:` and `hf:` suit benchmarks and evals where counts for the target model matter. `claude:` makes network calls, and Anthropic documents its counts as estimates. `hf:` downloads the tokenizer from the Hugging Face Hub on first use.
+
+```bash
+datoon input.json --tokenizer claude:claude-opus-5-5 --report-stdout
+```
+
 ## CLI
 
 Read from stdin:

{datoon-2.0.0 → datoon-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datoon
-Version: 2.0.0
+Version: 3.0.0
 Summary: Smart structured-data-to-TOON gateway with pragmatic auto-gating for LLM prompts.
 Author-email: Andrii Suruhov <andrii.suruhov@gmail.com>
 License: MIT License
@@ -44,6 +44,10 @@ Requires-Dist: pre-commit>=4.6.0; extra == "dev"
 Requires-Dist: pyyaml>=6.0; extra == "dev"
 Provides-Extra: tokens
 Requires-Dist: tiktoken>=0.13.0; extra == "tokens"
+Provides-Extra: claude
+Requires-Dist: anthropic>=2.0.0; extra == "claude"
+Provides-Extra: hf
+Requires-Dist: tokenizers>=0.23.0; extra == "hf"
 Provides-Extra: mcp
 Requires-Dist: mcp>=1.28.0; extra == "mcp"
 Provides-Extra: yaml
@@ -56,7 +60,7 @@ Requires-Dist: fastavro>=1.9; extra == "columnar"
 Provides-Extra: numbers
 Requires-Dist: numbers-parser>=4.4; extra == "numbers"
 Provides-Extra: all
-Requires-Dist: datoon[columnar,excel,mcp,numbers,tokens,yaml]; extra == "all"
+Requires-Dist: datoon[claude,columnar,excel,hf,mcp,numbers,tokens,yaml]; extra == "all"
 Dynamic: license-file
 
 <p align="center">
@@ -212,6 +216,10 @@ pip install "datoon[numbers]"
 # with tiktoken-based token counting
 pip install "datoon[tokens]"
 
+# with Claude token counting (Anthropic API) / Hugging Face tokenizers
+pip install "datoon[claude]"
+pip install "datoon[hf]"
+
 # with MCP server
 pip install "datoon[mcp]"
 
@@ -257,7 +265,22 @@ For Claude Code plugin, Codex, and MCP config → [**INSTALL.md**](./INSTALL.md)
 1. **Convert + estimate** — TOON encoder runs in-process, token savings calculated
 1. **Gate savings** — below threshold → return JSON; above → return TOON with report
 
-Every path returns a `ConversionReport` with `decision`, `reason`, and token estimates. Pipelines never get silent surprises.
+Every path returns a `ConversionReport` with `decision`, `reason`, token estimates, and the `tokenizer` that produced them. Pipelines never get silent surprises.
+
+## Choosing a tokenizer
+
+Token estimates drive the savings gate, so pick the counter that fits the job. Pass one spec with `--tokenizer`, the `tokenizer` MCP argument, `ConversionConfig(tokenizer=...)`, or the `DATOON_TOKENIZER` environment variable; in Python any callable `str -> int` works too. The MCP tools accept only local counters (`auto`, `chars`, `tiktoken`) per call; `claude:` and `hf:` reach the network, so only the server operator can enable them through `DATOON_TOKENIZER`.
+
+| Spec | Counts with | Needs |
+|---|---|---|
+| `auto` (default) | `tiktoken:o200k_base` when tiktoken is installed, else `chars:3` | — |
+| `chars` / `chars:<N>` | `ceil(len(text) / N)`, default `N = 3` | — |
+| `tiktoken` / `tiktoken:<encoding>` | tiktoken, default `o200k_base` | `datoon[tokens]` |
+| `claude:<model-id>` | Anthropic token counting API for that model | `datoon[claude]`, Anthropic credentials (e.g. `ANTHROPIC_API_KEY`) |
+| `hf:<repo-id>` | Hugging Face `tokenizers` tokenizer from the Hub | `datoon[hf]` |
+| a callable (Python) | your own function, reported as `custom` | — |
+
+Use the local counters (`auto`, `chars`, `tiktoken`) to gate payloads at runtime: they are fast and run locally (tiktoken fetches its encoding file once on first use). Use `claude:` or `hf:` in benchmarks and evals where counts for the target model matter. `claude:` makes network calls, and Anthropic documents its counts as estimates. Only `auto` falls back silently: an explicit spec with a missing package fails with an error that names the extra to install, and `claude:` without Anthropic credentials fails too. Every report records the resolved name in `tokenizer`.
 
 ______________________________________________________________________
 
@@ -420,6 +443,7 @@ ______________________________________________________________________
 | `--min-savings` | `0.15` | Minimum relative token savings required |
 | `--max-depth` | `6` | Maximum nesting depth for auto-conversion |
 | `--min-uniform-rows` | `3` | Minimum rows in uniform object arrays |
+| `--tokenizer <spec>` | `$DATOON_TOKENIZER`, else `auto` | Token counter for estimates — see [Choosing a tokenizer](#choosing-a-tokenizer) |
 | `--report <path>` | — | Write JSON conversion report to file |
 | `--report-stdout` | — | Print JSON conversion report to stderr |
 | `-o <path>` | stdout | Output file path |

{datoon-2.0.0 → datoon-3.0.0}/README.md

@@ -151,6 +151,10 @@ pip install "datoon[numbers]"
 # with tiktoken-based token counting
 pip install "datoon[tokens]"
 
+# with Claude token counting (Anthropic API) / Hugging Face tokenizers
+pip install "datoon[claude]"
+pip install "datoon[hf]"
+
 # with MCP server
 pip install "datoon[mcp]"
 
@@ -196,7 +200,22 @@ For Claude Code plugin, Codex, and MCP config → [**INSTALL.md**](./INSTALL.md)
 1. **Convert + estimate** — TOON encoder runs in-process, token savings calculated
 1. **Gate savings** — below threshold → return JSON; above → return TOON with report
 
-Every path returns a `ConversionReport` with `decision`, `reason`, and token estimates. Pipelines never get silent surprises.
+Every path returns a `ConversionReport` with `decision`, `reason`, token estimates, and the `tokenizer` that produced them. Pipelines never get silent surprises.
+
+## Choosing a tokenizer
+
+Token estimates drive the savings gate, so pick the counter that fits the job. Pass one spec with `--tokenizer`, the `tokenizer` MCP argument, `ConversionConfig(tokenizer=...)`, or the `DATOON_TOKENIZER` environment variable; in Python any callable `str -> int` works too. The MCP tools accept only local counters (`auto`, `chars`, `tiktoken`) per call; `claude:` and `hf:` reach the network, so only the server operator can enable them through `DATOON_TOKENIZER`.
+
+| Spec | Counts with | Needs |
+|---|---|---|
+| `auto` (default) | `tiktoken:o200k_base` when tiktoken is installed, else `chars:3` | — |
+| `chars` / `chars:<N>` | `ceil(len(text) / N)`, default `N = 3` | — |
+| `tiktoken` / `tiktoken:<encoding>` | tiktoken, default `o200k_base` | `datoon[tokens]` |
+| `claude:<model-id>` | Anthropic token counting API for that model | `datoon[claude]`, Anthropic credentials (e.g. `ANTHROPIC_API_KEY`) |
+| `hf:<repo-id>` | Hugging Face `tokenizers` tokenizer from the Hub | `datoon[hf]` |
+| a callable (Python) | your own function, reported as `custom` | — |
+
+Use the local counters (`auto`, `chars`, `tiktoken`) to gate payloads at runtime: they are fast and run locally (tiktoken fetches its encoding file once on first use). Use `claude:` or `hf:` in benchmarks and evals where counts for the target model matter. `claude:` makes network calls, and Anthropic documents its counts as estimates. Only `auto` falls back silently: an explicit spec with a missing package fails with an error that names the extra to install, and `claude:` without Anthropic credentials fails too. Every report records the resolved name in `tokenizer`.
 
 ______________________________________________________________________
 
@@ -359,6 +378,7 @@ ______________________________________________________________________
 | `--min-savings` | `0.15` | Minimum relative token savings required |
 | `--max-depth` | `6` | Maximum nesting depth for auto-conversion |
 | `--min-uniform-rows` | `3` | Minimum rows in uniform object arrays |
+| `--tokenizer <spec>` | `$DATOON_TOKENIZER`, else `auto` | Token counter for estimates — see [Choosing a tokenizer](#choosing-a-tokenizer) |
 | `--report <path>` | — | Write JSON conversion report to file |
 | `--report-stdout` | — | Print JSON conversion report to stderr |
 | `-o <path>` | stdout | Output file path |

{datoon-2.0.0 → datoon-3.0.0}/benchmarks/agent_skill_eval/generate_payloads.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Generate deterministic JSON payloads for agent skill evaluation."""
 
 from __future__ import annotations
@@ -67,12 +66,10 @@ def expected_for(payload: dict[str, object]) -> dict[str, object]:
         region_totals[region] = region_totals.get(region, 0) + revenue
         category_totals[category] = category_totals.get(category, 0) + revenue
 
-    top_region = sorted(region_totals.items(), key=lambda item: (-item[1], item[0]))[0][
+    top_region = min(region_totals.items(), key=lambda item: (-item[1], item[0]))[0][0]
+    top_category = min(category_totals.items(), key=lambda item: (-item[1], item[0]))[
         0
-    ]
-    top_category = sorted(
-        category_totals.items(), key=lambda item: (-item[1], item[0])
-    )[0][0]
+    ][0]
 
     return {
         "scenario": payload["scenario"],

{datoon-2.0.0 → datoon-3.0.0}/benchmarks/run.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Benchmark JSON vs TOON efficiency and datoon auto-conversion decisions."""
 
 from __future__ import annotations
@@ -9,7 +8,7 @@ import json
 import statistics
 import sys
 from dataclasses import asdict, dataclass
-from datetime import datetime, timezone
+from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
 
@@ -126,7 +125,7 @@ def benchmark_payload(
                 raw_error=str(exc),
             )
 
-    json_tokens = estimate_tokens(raw_json)
+    json_tokens = estimate_tokens(raw_json, auto_config.tokenizer)
 
     toon_tokens: int | None = None
     raw_savings_pct: float | None = None
@@ -248,16 +247,17 @@ def save_results(
 ) -> Path:
     """Save benchmark output to versioned JSON artifact."""
     RESULTS_DIR.mkdir(parents=True, exist_ok=True)
-    timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
+    timestamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")
     output = {
         "metadata": {
             "script_version": SCRIPT_VERSION,
-            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "generated_at": datetime.now(UTC).isoformat(),
             "payloads_sha256": payloads_sha,
             "config": {
                 "min_savings_ratio": config.min_savings_ratio,
                 "max_depth": config.max_depth,
                 "min_uniform_rows": config.min_uniform_rows,
+                "tokenizer": config.tokenizer,
             },
         },
         "summary": asdict(summary),
@@ -348,6 +348,7 @@ def dry_run(
     print(f"- min_savings_ratio: {config.min_savings_ratio}")
     print(f"- max_depth: {config.max_depth}")
     print(f"- min_uniform_rows: {config.min_uniform_rows}")
+    print(f"- tokenizer: {config.tokenizer}")
     print(f"- json payloads: {len(payloads)}")
     for payload in payloads:
         print(
@@ -384,6 +385,15 @@ def parse_args() -> argparse.Namespace:
         default=3,
         help="Auto mode minimum rows for uniform object arrays.",
     )
+    parser.add_argument(
+        "--tokenizer",
+        default="auto",
+        metavar="SPEC",
+        help=(
+            "Token counter for estimates: auto, chars[:N], tiktoken[:ENCODING], "
+            "claude:MODEL_ID (network) or hf:REPO_ID (default: auto)."
+        ),
+    )
     parser.add_argument(
         "--dry-run",
         action="store_true",
@@ -413,12 +423,14 @@ def main() -> int:
         max_depth=args.max_depth,
         min_uniform_rows=args.min_uniform_rows,
         force=False,
+        tokenizer=args.tokenizer,
     )
     force_config = ConversionConfig(
         min_savings_ratio=0.0,
         max_depth=max(64, args.max_depth),
         min_uniform_rows=2,
         force=True,
+        tokenizer=args.tokenizer,
     )
 
     if args.dry_run:

{datoon-2.0.0 → datoon-3.0.0}/pyproject.toml

@@ -27,12 +27,14 @@ dependencies = [
 [project.optional-dependencies]
 dev = ["pytest>=9.0.3", "pre-commit>=4.6.0", "pyyaml>=6.0"]
 tokens = ["tiktoken>=0.13.0"]
+claude = ["anthropic>=2.0.0"]
+hf = ["tokenizers>=0.23.0"]
 mcp = ["mcp>=1.28.0"]
 yaml = ["pyyaml>=6.0"]
 excel = ["openpyxl>=3.1"]
 columnar = ["pyarrow>=15.0", "fastavro>=1.9"]
 numbers = ["numbers-parser>=4.4"]
-all = ["datoon[tokens,mcp,yaml,excel,columnar,numbers]"]
+all = ["datoon[tokens,claude,hf,mcp,yaml,excel,columnar,numbers]"]
 
 [project.urls]
 Homepage = "https://github.com/andrii-su/datoon"

{datoon-2.0.0 → datoon-3.0.0}/scripts/ab_skill_benchmark.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """A/B benchmark for datoon skill usage in Claude CLI.
 
 This script compares the same prompts in two modes:
@@ -18,7 +17,7 @@ import math
 import statistics
 import subprocess
 from dataclasses import asdict, dataclass
-from datetime import datetime, timezone
+from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any
 
@@ -427,7 +426,7 @@ def main() -> int:
     if args.payload_limit < 1:
         raise ValueError("--payload-limit must be >= 1.")
 
-    timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
+    timestamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")
     log_path = args.output_dir / "logs" / f"ab_skill_{timestamp}.log"
     logger = configure_logging(log_path, verbose=args.verbose)
 
@@ -606,7 +605,7 @@ def main() -> int:
 
     output = {
         "metadata": {
-            "generated_at": datetime.now(timezone.utc).isoformat(),
+            "generated_at": datetime.now(UTC).isoformat(),
             "payload_ids": payload_ids,
             "iterations": args.iterations,
             "model": args.model,

{datoon-2.0.0 → datoon-3.0.0}/scripts/install.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Install, dry-run, or uninstall datoon integrations."""
 
 from __future__ import annotations
@@ -10,7 +9,7 @@ import shutil
 import subprocess
 import sys
 from dataclasses import dataclass
-from datetime import datetime, timezone
+from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any, Literal
 
@@ -74,7 +73,7 @@ def read_json_object(path: Path) -> dict[str, Any]:
     except json.JSONDecodeError as exc:
         raise ValueError(f"Malformed JSON in {path}: {exc.msg}") from exc
     if not isinstance(parsed, dict):
-        raise ValueError(f"Expected JSON object in {path}")
+        raise TypeError(f"Expected JSON object in {path}")
     return parsed
 
 
@@ -83,7 +82,7 @@ def write_json_with_backup(path: Path, data: dict[str, Any]) -> Path | None:
     path.parent.mkdir(parents=True, exist_ok=True)
     backup_path: Path | None = None
     if path.exists():
-        timestamp = datetime.now(timezone.utc).strftime("%Y%m%d%H%M%S%f")
+        timestamp = datetime.now(UTC).strftime("%Y%m%d%H%M%S%f")
         backup_path = path.with_name(f"{path.name}.bak.{timestamp}")
         index = 1
         while backup_path.exists():

{datoon-2.0.0 → datoon-3.0.0}/scripts/summarize_agent_skill_eval.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Summarize artifact-based agent skill evaluation results."""
 
 from __future__ import annotations

{datoon-2.0.0 → datoon-3.0.0}/scripts/validate_plugin_metadata.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Validate plugin and marketplace metadata consistency."""
 
 from __future__ import annotations
@@ -35,7 +34,7 @@ def read_json(path: Path) -> dict[str, Any]:
     except FileNotFoundError as exc:
         raise FileNotFoundError(f"Missing required metadata file: {path}") from exc
     if not isinstance(parsed, dict):
-        raise ValueError(f"Expected JSON object in {path}")
+        raise TypeError(f"Expected JSON object in {path}")
     return parsed
 
 

{datoon-2.0.0 → datoon-3.0.0}/scripts/validate_skill_sync.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 """Validate datoon skill copies and packaged .skill artifact consistency."""
 
 from __future__ import annotations
@@ -56,8 +55,7 @@ def validate() -> list[str]:
         mirror_hash = _sha256_bytes(mirror_bytes)
         if mirror_hash != source_hash:
             errors.append(
-                f"Out-of-sync SKILL.md copy: {mirror_path} "
-                f"(expected hash {source_hash}, got {mirror_hash})"
+                f"Out-of-sync SKILL.md copy: {mirror_path} (expected hash {source_hash}, got {mirror_hash})"
             )
 
     archived_bytes = _read_archive_member(SKILL_ARCHIVE_PATH, SKILL_ARCHIVE_MEMBER)

{datoon-2.0.0 → datoon-3.0.0}/src/datoon/__init__.py

@@ -12,17 +12,21 @@ from datoon.models import (
     PayloadAnalysis,
 )
 from datoon.readers import detect_format, read_tabular
+from datoon.tokenizers import TokenCounter, estimate_tokens, resolve_tokenizer
 
 __all__ = [
-    "__version__",
-    "convert_json_for_llm",
-    "read_tabular",
-    "detect_format",
     "ConversionConfig",
     "ConversionOutcome",
     "ConversionReport",
-    "PayloadAnalysis",
     "DatoonError",
+    "PayloadAnalysis",
+    "TokenCounter",
+    "__version__",
+    "convert_json_for_llm",
+    "detect_format",
+    "estimate_tokens",
+    "read_tabular",
+    "resolve_tokenizer",
 ]
 
 try:

{datoon-2.0.0 → datoon-3.0.0}/src/datoon/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '2.0.0'
-__version_tuple__ = version_tuple = (2, 0, 0)
+__version__ = version = '3.0.0'
+__version_tuple__ = version_tuple = (3, 0, 0)
 
-__commit_id__ = commit_id = 'g02282b3f8'
+__commit_id__ = commit_id = 'gb00f5623a'

{datoon-2.0.0 → datoon-3.0.0}/src/datoon/cli.py

@@ -4,14 +4,16 @@ from __future__ import annotations
 
 import argparse
 import json
+import os
 import sys
 from collections.abc import Sequence
 from pathlib import Path
 
 from datoon import __version__
 from datoon.converter import DatoonError, convert_json_for_llm
-from datoon.models import DEFAULT_TOKEN_ENCODING, ConversionConfig
+from datoon.models import ConversionConfig
 from datoon.readers import ALL_FORMATS, BINARY_FORMATS, detect_format, read_tabular
+from datoon.tokenizers import DEFAULT_TOKENIZER, TOKENIZER_ENV_VAR
 
 
 def build_parser() -> argparse.ArgumentParser:
@@ -82,11 +84,14 @@ def build_parser() -> argparse.ArgumentParser:
         help="Table index within a Numbers sheet (default: 0).",
     )
     parser.add_argument(
-        "--encoding",
-        default=DEFAULT_TOKEN_ENCODING,
+        "--tokenizer",
+        default=None,
+        metavar="SPEC",
         help=(
-            "tiktoken encoding for token estimates "
-            f"(default: {DEFAULT_TOKEN_ENCODING})."
+            "Token counter for estimates: auto, chars[:N], tiktoken[:ENCODING], "
+            "claude:MODEL_ID (network, needs datoon[claude]) or hf:REPO_ID "
+            f"(needs datoon[hf]). Default: ${TOKENIZER_ENV_VAR}, "
+            f"else {DEFAULT_TOKENIZER}."
         ),
     )
     parser.add_argument(
@@ -172,7 +177,9 @@ def main(argv: Sequence[str] | None = None) -> int:
             max_depth=args.max_depth,
             min_uniform_rows=args.min_uniform_rows,
             force=args.force,
-            token_encoding=args.encoding,
+            tokenizer=(
+                args.tokenizer or os.environ.get(TOKENIZER_ENV_VAR) or DEFAULT_TOKENIZER
+            ),
         )
 
         fmt = _resolve_format(args.input, args.format)

{datoon-2.0.0 → datoon-3.0.0}/src/datoon/converter.py

@@ -3,8 +3,6 @@
 from __future__ import annotations
 
 import json
-import math
-from functools import lru_cache
 from typing import Any
 
 from toon_format import encode
@@ -12,12 +10,12 @@ from toon_format import encode
 from datoon.analyzer import analyze_payload
 from datoon.errors import DatoonError
 from datoon.models import (
-    DEFAULT_TOKEN_ENCODING,
     ConversionConfig,
     ConversionOutcome,
     ConversionReport,
     PayloadAnalysis,
 )
+from datoon.tokenizers import estimate_tokens, resolve_tokenizer
 
 __all__ = ["DatoonError", "convert_json_for_llm", "estimate_tokens"]
 
@@ -42,26 +40,6 @@ def _normalize_json(raw_text: str) -> tuple[Any, str]:
     return parsed, normalized
 
 
-@lru_cache(maxsize=8)
-def _load_token_encoder(encoding: str) -> Any | None:
-    """Load a token encoder once per name, None when the optional dep is absent."""
-    try:
-        import tiktoken  # type: ignore
-    except ImportError:
-        return None
-
-    return tiktoken.get_encoding(encoding)
-
-
-def estimate_tokens(text: str, encoding: str = DEFAULT_TOKEN_ENCODING) -> int:
-    """Estimate token count with tiktoken when installed, else use a char heuristic."""
-    encoder = _load_token_encoder(encoding)
-    if encoder is not None:
-        return len(encoder.encode(text))
-
-    return max(1, math.ceil(len(text) / 3))
-
-
 def _encode_toon(parsed: Any) -> str:
     """Encode an already-parsed JSON value as TOON with the official encoder."""
     try:
@@ -81,6 +59,7 @@ def _build_skip_outcome(
     reason: str,
     config: ConversionConfig,
     input_tokens: int,
+    tokenizer_name: str,
     analysis: PayloadAnalysis,
 ) -> ConversionOutcome:
     """Build a standard skip outcome to keep reporting consistent."""
@@ -91,16 +70,23 @@ def _build_skip_outcome(
         input_token_estimate=input_tokens,
         output_token_estimate=input_tokens,
         savings_ratio=0.0,
+        tokenizer=tokenizer_name,
         analysis=analysis,
     )
     return ConversionOutcome(payload_text=normalized_json, report=report)
 
 
 def convert_json_for_llm(raw_text: str, config: ConversionConfig) -> ConversionOutcome:
-    """Convert JSON to TOON when structure and savings meet configured policy."""
+    """Convert JSON to TOON when structure and savings meet configured policy.
+
+    The tokenizer is resolved once per call and counts both the JSON and the
+    TOON text. A tokenizer that cannot be resolved is a configuration error and
+    raises ``DatoonError`` in auto and force mode alike.
+    """
     parsed, normalized_json = _normalize_json(raw_text)
+    counter, tokenizer_name = resolve_tokenizer(config.tokenizer)
     analysis = analyze_payload(parsed, config)
-    input_tokens = estimate_tokens(normalized_json, config.token_encoding)
+    input_tokens = estimate_tokens(normalized_json, counter)
 
     if not analysis.is_candidate and not config.force:
         return _build_skip_outcome(
@@ -108,6 +94,7 @@ def convert_json_for_llm(raw_text: str, config: ConversionConfig) -> ConversionO
             reason=analysis.reason,
             config=config,
             input_tokens=input_tokens,
+            tokenizer_name=tokenizer_name,
             analysis=analysis,
         )
 
@@ -122,10 +109,11 @@ def convert_json_for_llm(raw_text: str, config: ConversionConfig) -> ConversionO
             reason="TOON encoding failed; falling back to JSON.",
             config=config,
             input_tokens=input_tokens,
+            tokenizer_name=tokenizer_name,
             analysis=analysis,
         )
 
-    output_tokens = estimate_tokens(toon_text, config.token_encoding)
+    output_tokens = estimate_tokens(toon_text, counter)
     savings_ratio = (input_tokens - output_tokens) / input_tokens
 
     if savings_ratio < config.min_savings_ratio and not config.force:
@@ -137,6 +125,7 @@ def convert_json_for_llm(raw_text: str, config: ConversionConfig) -> ConversionO
             ),
             config=config,
             input_tokens=input_tokens,
+            tokenizer_name=tokenizer_name,
             analysis=analysis,
         )
 
@@ -150,6 +139,7 @@ def convert_json_for_llm(raw_text: str, config: ConversionConfig) -> ConversionO
         input_token_estimate=input_tokens,
         output_token_estimate=output_tokens,
         savings_ratio=savings_ratio,
+        tokenizer=tokenizer_name,
         analysis=analysis,
     )
     return ConversionOutcome(payload_text=toon_text, report=report)