cli-consumption 0.3.1__tar.gz → 0.3.3__tar.gz

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/CHANGELOG.md

@@ -6,6 +6,35 @@ use [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- Added deterministic, compressed offline snapshot files authenticated with Ed25519,
+  with bounded verification-before-parsing and idempotent SQLite/PostgreSQL ingestion.
+
+### Fixed
+
+- Continue now applies provider-specific cache semantics, preventing Anthropic cache
+  reads and writes from being omitted from input totals, and reports persisted
+  conversation-compaction markers.
+- Codex now preserves valid token accounting when provider counters are inconsistent,
+  keeps partial work and compaction records from referencing nonexistent turns, and
+  safely handles malformed metadata and timestamps.
+- OpenCode now rejects timezone-naive text timestamps consistently while retaining its
+  OpenCode 1.18.23 message/part extraction for models, tokens, and tools.
+
+## [0.3.2] - 2026-08-31
+
+### Changed
+
+- Version releases now attach the validated wheel and source distribution to an
+  automatically generated GitHub Release after PyPI publication succeeds.
+
+### Fixed
+
+- OpenCode 1.18.23 SQLite stores now extract model calls and token usage from
+  `message.data` and tool calls from `part.data`; incomplete current schemas are
+  reported as incompatible instead of silently producing zero usage.
+
 ## [0.3.1] - 2026-08-31
 
 ### Changed
@@ -102,7 +131,8 @@ use [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 - Refreshed the provider guide for the first minor release ([#26]).
 
-[Unreleased]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.1...HEAD
+[Unreleased]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.2...HEAD
+[0.3.2]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.1...v0.3.2
 [0.3.1]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.3.0...v0.3.1
 [0.3.0]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/Guillaume-Lombardo/cli-consumption/compare/v0.2.0...v0.2.1

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cli-consumption
-Version: 0.3.1
+Version: 0.3.3
 Summary: Analyze and consolidate AI coding CLI consumption across machines.
 Project-URL: Homepage, https://github.com/Guillaume-Lombardo/cli-consumption
 Project-URL: Documentation, https://github.com/Guillaume-Lombardo/cli-consumption#readme
@@ -31,6 +31,8 @@ Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
 Provides-Extra: server
 Requires-Dist: fastapi>=0.115; extra == 'server'
 Requires-Dist: uvicorn>=0.34; extra == 'server'
+Provides-Extra: snapshots
+Requires-Dist: cryptography>=45; extra == 'snapshots'
 Provides-Extra: sync
 Requires-Dist: httpx>=0.27; extra == 'sync'
 Description-Content-Type: text/markdown
@@ -75,7 +77,8 @@ the optional runtime capabilities you use:
 
 - `cli-consumption[sync]` for the sync client;
 - `cli-consumption[server]` for the collector service;
-- `cli-consumption[postgres]` for PostgreSQL.
+- `cli-consumption[postgres]` for PostgreSQL;
+- `cli-consumption[snapshots]` for signed, compressed offline snapshot files.
 
 Extras can be combined, for example `cli-consumption[server,postgres]` on a central
 collector.
@@ -103,7 +106,7 @@ uv run cli-consumption collect --provider codex --database usage.sqlite
 Use `--source [LABEL=]PATH` for trusted offline copies and repeated
 `--project NAME=PATH_PREFIX` mappings for stable project labels. See the
 [usage and operations guide](https://github.com/Guillaume-Lombardo/cli-consumption/blob/main/docs/usage.md)
-for multi-machine collection, reporting,
+for signed offline transfers, multi-machine collection, reporting,
 PostgreSQL, retention, synchronization, readiness, and automation.
 
 ## Supported CLIs
@@ -152,6 +155,7 @@ for ingestion, idempotency, migrations, report limits, and collector behavior.
 | Command | Purpose |
 | --- | --- |
 | `collect` | Collect local or copied provider data into SQL. |
+| `snapshot` | Create or ingest signed, compressed offline snapshot files. |
 | `sync` | Collect and send metadata-only snapshots to a central API. |
 | `serve` | Run the central collection API. |
 | `export` | Write the HTML dashboard and optional CSV tables. |

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/README.md

@@ -38,7 +38,8 @@ the optional runtime capabilities you use:
 
 - `cli-consumption[sync]` for the sync client;
 - `cli-consumption[server]` for the collector service;
-- `cli-consumption[postgres]` for PostgreSQL.
+- `cli-consumption[postgres]` for PostgreSQL;
+- `cli-consumption[snapshots]` for signed, compressed offline snapshot files.
 
 Extras can be combined, for example `cli-consumption[server,postgres]` on a central
 collector.
@@ -66,7 +67,7 @@ uv run cli-consumption collect --provider codex --database usage.sqlite
 Use `--source [LABEL=]PATH` for trusted offline copies and repeated
 `--project NAME=PATH_PREFIX` mappings for stable project labels. See the
 [usage and operations guide](https://github.com/Guillaume-Lombardo/cli-consumption/blob/main/docs/usage.md)
-for multi-machine collection, reporting,
+for signed offline transfers, multi-machine collection, reporting,
 PostgreSQL, retention, synchronization, readiness, and automation.
 
 ## Supported CLIs
@@ -115,6 +116,7 @@ for ingestion, idempotency, migrations, report limits, and collector behavior.
 | Command | Purpose |
 | --- | --- |
 | `collect` | Collect local or copied provider data into SQL. |
+| `snapshot` | Create or ingest signed, compressed offline snapshot files. |
 | `sync` | Collect and send metadata-only snapshots to a central API. |
 | `serve` | Run the central collection API. |
 | `export` | Write the HTML dashboard and optional CSV tables. |

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "cli-consumption"
-version = "0.3.1"
+version = "0.3.3"
 description = "Analyze and consolidate AI coding CLI consumption across machines."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -34,6 +34,9 @@ dependencies = [
 postgres = ["psycopg[binary]>=3.2"]
 server = ["fastapi>=0.115", "uvicorn>=0.34"]
 sync = ["httpx>=0.27"]
+snapshots = [
+    "cryptography>=45",
+]
 
 [project.urls]
 Homepage = "https://github.com/Guillaume-Lombardo/cli-consumption"

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/src/cli_consumption/adapters/_shared.py

@@ -166,15 +166,22 @@ def tokens(
     reasoning: object = 0,
     total: object = 0,
 ) -> dict[str, int]:
-    uncached_n = counter(uncached)
-    cached_n = counter(cached)
-    write_n = counter(cache_write)
-    visible_n = counter(visible)
-    reasoning_n = counter(reasoning)
-    input_n = bounded_sum(uncached_n, cached_n, write_n)
-    output_n = bounded_sum(visible_n, reasoning_n)
-    attributed = bounded_sum(input_n, output_n)
-    total_n = max(attributed, counter(total))
+    remaining = MAX_BIGINT
+    uncached_n = min(counter(uncached), remaining)
+    remaining -= uncached_n
+    cached_n = min(counter(cached), remaining)
+    remaining -= cached_n
+    write_n = min(counter(cache_write), remaining)
+    remaining -= write_n
+    visible_n = min(counter(visible), remaining)
+    remaining -= visible_n
+    reasoning_n = min(counter(reasoning), remaining)
+    remaining -= reasoning_n
+    input_n = uncached_n + cached_n + write_n
+    output_n = visible_n + reasoning_n
+    attributed = input_n + output_n
+    unattributed_n = min(max(0, counter(total) - attributed), remaining)
+    total_n = attributed + unattributed_n
     return {
         "input_tokens": input_n,
         "cached_input_tokens": cached_n,
@@ -184,13 +191,34 @@ def tokens(
         "total_tokens": total_n,
         "uncached_input_tokens": uncached_n,
         "visible_output_tokens": visible_n,
-        "unattributed_tokens": max(0, total_n - attributed),
+        "unattributed_tokens": unattributed_n,
     }
 
 
 def add_tokens(target: dict[str, Any], value: dict[str, int]) -> None:
-    for field, amount in value.items():
-        target[field] = bounded_sum(int(target[field]), amount)
+    remaining = MAX_BIGINT
+    for field in (
+        "uncached_input_tokens",
+        "cached_input_tokens",
+        "cache_write_input_tokens",
+        "visible_output_tokens",
+        "reasoning_output_tokens",
+        "unattributed_tokens",
+    ):
+        amount = min(remaining, int(target[field]) + value[field])
+        target[field] = amount
+        remaining -= amount
+    target["input_tokens"] = (
+        target["uncached_input_tokens"]
+        + target["cached_input_tokens"]
+        + target["cache_write_input_tokens"]
+    )
+    target["output_tokens"] = (
+        target["visible_output_tokens"] + target["reasoning_output_tokens"]
+    )
+    target["total_tokens"] = (
+        target["input_tokens"] + target["output_tokens"] + target["unattributed_tokens"]
+    )
 
 
 def new_turn(

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/src/cli_consumption/adapters/codex.py

@@ -87,11 +87,14 @@ AGENT_ROLE_ALIASES = {
 SAFE_DIMENSION = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.:/+-]*")
 
 
-def parse_timestamp(value: str | None) -> datetime | None:
-    if not value:
+def parse_timestamp(value: object) -> datetime | None:
+    if not isinstance(value, str) or not value:
         return None
     try:
-        return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC)
+        parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+        if parsed.tzinfo is None or parsed.utcoffset() is None:
+            return None
+        return parsed.astimezone(UTC)
     except ValueError:
         return None
 
@@ -99,7 +102,8 @@ def parse_timestamp(value: str | None) -> datetime | None:
 def infer_project(
     metadata: dict[str, Any], mappings: list[tuple[str, str]]
 ) -> tuple[str, str]:
-    cwd = str(metadata.get("cwd") or "").rstrip("/\\")
+    raw_cwd = metadata.get("cwd")
+    cwd = raw_cwd.rstrip("/\\") if isinstance(raw_cwd, str) else ""
     normalized_cwd = cwd.replace("\\", "/")
     for name, prefix in sorted(mappings, key=lambda item: len(item[1]), reverse=True):
         normalized_prefix = prefix.replace("\\", "/").rstrip("/")
@@ -109,17 +113,18 @@ def infer_project(
             return name, "mapping"
     git = metadata.get("git")
     if isinstance(git, dict):
-        repository = str(git.get("repository_url") or git.get("repository") or "")
+        raw_repository = git.get("repository_url") or git.get("repository")
+        repository = raw_repository if isinstance(raw_repository, str) else ""
         slug = re.split(r"[/\\:]", repository.rstrip("/\\"))[-1]
         if slug.endswith(".git"):
             slug = slug[:-4]
-        if slug:
+        if _safe_dimension(slug, 255):
             return slug, "git"
     return OUTSIDE_PROJECT, "none"
 
 
 def extract_tools(payload: dict[str, Any]) -> list[tuple[str, str]]:
-    outer_name = str(payload.get("name", "unknown"))
+    outer_name = _safe_dimension(payload.get("name"), 512) or "unknown"
     if outer_name != "exec":
         return [(outer_name, outer_name)]
     raw_input = payload.get("input", "")
@@ -233,9 +238,16 @@ class CodexAdapter:
                         continue
                     event_count += 1
                     if event.get("type") == "session_meta":
-                        conversation_id = str(event.get("payload", {}).get("id", ""))
-                conversation_id = conversation_id or path.stem
-                candidate = (machine, path, event_count, digest.hexdigest())
+                        payload = event.get("payload")
+                        if not isinstance(payload, dict):
+                            malformed += 1
+                            continue
+                        candidate_id = _safe_dimension(payload.get("id"), 512)
+                        if candidate_id and not conversation_id:
+                            conversation_id = candidate_id
+                content_hash = digest.hexdigest()
+                conversation_id = conversation_id or f"content-{content_hash}"
+                candidate = (machine, path, event_count, content_hash)
                 previous = selected.get(conversation_id)
                 if previous is None:
                     selected[conversation_id] = candidate
@@ -273,7 +285,16 @@ class CodexAdapter:
             ),
             {},
         )
-        conversation_id = str(metadata.get("id") or path.stem)
+        conversation_id = next(
+            (
+                candidate_id
+                for event in events
+                if event.get("type") == "session_meta"
+                and isinstance((payload := event.get("payload")), dict)
+                and (candidate_id := _safe_dimension(payload.get("id"), 512))
+            ),
+            f"content-{digest}",
+        )
         record_id = f"codex:{conversation_id}"
         project, project_source = infer_project(metadata, mappings)
         timestamps = [
@@ -316,9 +337,7 @@ class CodexAdapter:
                     {
                         "id": f"{record_id}:compaction:{compaction_sequence}",
                         "conversation_id": record_id,
-                        "turn_id": (
-                            f"{record_id}:{active_turn_id}" if active_turn_id else None
-                        ),
+                        "turn_id": turns.get(active_turn_id or "", {}).get("id"),
                         "sequence": compaction_sequence,
                         "timestamp": _iso(timestamp),
                     }
@@ -343,7 +362,7 @@ class CodexAdapter:
                         _merge_present(settings_by_turn[active_turn_id], updates)
             if event_type == "turn_context":
                 active_turn_id = (
-                    str(payload.get("turn_id") or active_turn_id or "") or None
+                    _safe_dimension(payload.get("turn_id"), 512) or active_turn_id
                 )
                 active_model = (
                     _safe_dimension(payload.get("model"), 255) or active_model
@@ -362,7 +381,7 @@ class CodexAdapter:
                         or setting_defaults["collaboration_mode"],
                     }
             if event_type == "event_msg" and payload_type == "task_started":
-                active_turn_id = str(payload.get("turn_id") or "") or None
+                active_turn_id = _safe_dimension(payload.get("turn_id"), 512)
                 if active_turn_id:
                     settings = settings_by_turn.setdefault(
                         active_turn_id, dict(setting_defaults)
@@ -391,7 +410,9 @@ class CodexAdapter:
                 "task_complete",
                 "turn_aborted",
             }:
-                turn_id = str(payload.get("turn_id") or active_turn_id or "")
+                turn_id = (
+                    _safe_dimension(payload.get("turn_id"), 512) or active_turn_id or ""
+                )
                 if turn_id in turns:
                     turns[turn_id].update(
                         ended_at=_iso(timestamp),
@@ -412,12 +433,13 @@ class CodexAdapter:
                 work_sequence += 1
                 started_at_ms = _integer_or_none(payload.get("started_at_ms"))
                 completed_at_ms = _integer_or_none(payload.get("completed_at_ms"))
-                turn_id = str(payload.get("turn_id") or active_turn_id or "") or None
+                turn_id = _safe_dimension(payload.get("turn_id") or active_turn_id, 512)
+                turn = turns.get(turn_id or "")
                 snapshot.work_items.append(
                     {
                         "id": f"{record_id}:work:{work_sequence}",
                         "conversation_id": record_id,
-                        "turn_id": f"{record_id}:{turn_id}" if turn_id else None,
+                        "turn_id": turn["id"] if turn else None,
                         "sequence": work_sequence,
                         "kind": WORK_ITEM_KINDS.get(
                             str(item.get("type") or ""), "other"
@@ -438,18 +460,12 @@ class CodexAdapter:
                 if not isinstance(usage, dict):
                     continue
                 call_sequence += 1
-                tokens = {
-                    field: _nonnegative_integer(usage.get(field))
-                    for field in TOKEN_FIELDS
-                }
-                tokens.update(_derived_tokens(tokens))
-                for field, value in tokens.items():
-                    totals[field] += value
+                tokens = _usage_tokens(usage)
+                _accumulate_tokens(totals, tokens)
                 turn = turns.get(active_turn_id or "")
                 if turn:
                     turn["model_calls"] += 1
-                    for field, value in tokens.items():
-                        turn[field] += value
+                    _accumulate_tokens(turn, tokens)
                 snapshot.model_calls.append(
                     {
                         "id": f"{record_id}:model:{call_sequence}",
@@ -548,24 +564,62 @@ class CodexAdapter:
         )
 
 
-def _derived_tokens(tokens: dict[str, int]) -> dict[str, int]:
+def _usage_tokens(usage: dict[str, Any]) -> dict[str, int]:
+    raw = {field: _nonnegative_integer(usage.get(field)) for field in TOKEN_FIELDS}
+    cached = raw["cached_input_tokens"]
+    cache_write = min(raw["cache_write_input_tokens"], MAX_BIGINT - cached)
+    input_tokens = max(raw["input_tokens"], cached + cache_write)
+    uncached = input_tokens - cached - cache_write
+    remaining = MAX_BIGINT - input_tokens
+    reasoning = min(raw["reasoning_output_tokens"], remaining)
+    visible = min(
+        max(0, raw["output_tokens"] - raw["reasoning_output_tokens"]),
+        remaining - reasoning,
+    )
+    output_tokens = reasoning + visible
+    unattributed = min(
+        max(0, raw["total_tokens"] - input_tokens - output_tokens),
+        MAX_BIGINT - input_tokens - output_tokens,
+    )
     return {
-        "uncached_input_tokens": max(
-            0,
-            tokens["input_tokens"]
-            - tokens["cached_input_tokens"]
-            - tokens["cache_write_input_tokens"],
-        ),
-        "visible_output_tokens": max(
-            0, tokens["output_tokens"] - tokens["reasoning_output_tokens"]
-        ),
-        "unattributed_tokens": max(
-            0,
-            tokens["total_tokens"] - tokens["input_tokens"] - tokens["output_tokens"],
-        ),
+        "input_tokens": input_tokens,
+        "cached_input_tokens": cached,
+        "cache_write_input_tokens": cache_write,
+        "output_tokens": output_tokens,
+        "reasoning_output_tokens": reasoning,
+        "total_tokens": input_tokens + output_tokens + unattributed,
+        "uncached_input_tokens": uncached,
+        "visible_output_tokens": visible,
+        "unattributed_tokens": unattributed,
     }
 
 
+def _accumulate_tokens(target: dict[str, Any], value: dict[str, int]) -> None:
+    remaining = MAX_BIGINT
+    for field in (
+        "uncached_input_tokens",
+        "cached_input_tokens",
+        "cache_write_input_tokens",
+        "visible_output_tokens",
+        "reasoning_output_tokens",
+        "unattributed_tokens",
+    ):
+        amount = min(remaining, int(target[field]) + value[field])
+        target[field] = amount
+        remaining -= amount
+    target["input_tokens"] = (
+        target["uncached_input_tokens"]
+        + target["cached_input_tokens"]
+        + target["cache_write_input_tokens"]
+    )
+    target["output_tokens"] = (
+        target["visible_output_tokens"] + target["reasoning_output_tokens"]
+    )
+    target["total_tokens"] = (
+        target["input_tokens"] + target["output_tokens"] + target["unattributed_tokens"]
+    )
+
+
 def _safe_dimension(value: object, maximum: int) -> str | None:
     if not isinstance(value, str):
         return None
@@ -650,5 +704,9 @@ def _integer_or_none(value: object) -> int | None:
 
 
 def _nonnegative_integer(value: object) -> int:
-    parsed = _integer_or_none(value)
+    parsed = (
+        _integer_or_none(value)
+        if not isinstance(value, float) or value.is_integer()
+        else None
+    )
     return max(0, parsed or 0)

{cli_consumption-0.3.1 → cli_consumption-0.3.3}/src/cli_consumption/adapters/continue_cli.py

@@ -18,6 +18,9 @@ from cli_consumption.adapters._shared import (
     ProviderInputBudget,
     read_bounded_bytes,
 )
+from cli_consumption.adapters._shared import (
+    add_tokens as _add_tokens,
+)
 from cli_consumption.adapters._shared import (
     bounded_sum as _sum,
 )
@@ -96,6 +99,7 @@ class ContinueAdapter:
         active: int | None = None
         raw_calls: list[tuple[int | None, str, dict[str, int]]] = []
         tools: list[tuple[int | None, str]] = []
+        compactions: list[int | None] = []
         seen_tools: set[str] = set()
 
         for item in history:
@@ -126,6 +130,8 @@ class ContinueAdapter:
 
             if role != "assistant":
                 continue
+            if isinstance(item.get("conversationSummary"), str):
+                compactions.append(active)
             model = _message_model(item) or session_model or "unknown"
             usage = _usage(message.get("usage"))
             raw_calls.append((active, model, usage))
@@ -152,7 +158,7 @@ class ContinueAdapter:
         totals = empty_tokens()
         models: set[str] = set()
         for sequence, (turn_index, model, usage) in enumerate(raw_calls, 1):
-            tokens = _tokens(usage)
+            tokens = _tokens(usage, model)
             models.add(model)
             _add_tokens(totals, tokens)
             turn = turns[turn_index] if turn_index is not None else None
@@ -187,6 +193,18 @@ class ContinueAdapter:
                 }
             )
 
+        for sequence, turn_index in enumerate(compactions, 1):
+            turn = turns[turn_index] if turn_index is not None else None
+            snapshot.compaction_events.append(
+                {
+                    "id": f"{conversation_id}:compaction:{sequence}",
+                    "conversation_id": conversation_id,
+                    "turn_id": turn["id"] if turn else None,
+                    "sequence": sequence,
+                    "timestamp": None,
+                }
+            )
+
         for index, turn in enumerate(turns):
             snapshot.turns.append(turn)
             observed_models = turn_models[index]
@@ -224,7 +242,7 @@ class ContinueAdapter:
                 "iterations": len(turns),
                 "model_calls": len(raw_calls),
                 "tool_calls": len(tools),
-                "compactions": 0,
+                "compactions": len(compactions),
                 "event_count": source.event_count,
                 "content_hash": source.digest,
                 **totals,
@@ -318,11 +336,25 @@ def _residual_usage(
     }
 
 
-def _tokens(usage: dict[str, int]) -> dict[str, int]:
-    input_tokens = usage["prompt"]
-    cached = min(input_tokens, usage["cached"])
-    cache_write = min(max(0, input_tokens - cached), usage["cache_write"])
-    output_tokens = usage["completion"]
+def _tokens(usage: dict[str, int], model: str) -> dict[str, int]:
+    prompt = usage["prompt"]
+    provider = model.partition("/")[0].casefold() if "/" in model else ""
+    separate_cache = (
+        provider in {"anthropic", "bedrock"}
+        or usage["cache_write"] > 0
+        or usage["cached"] > prompt
+    )
+    if separate_cache:
+        uncached = prompt
+        cached = min(usage["cached"], MAX_BIGINT - uncached)
+        cache_write = min(usage["cache_write"], MAX_BIGINT - uncached - cached)
+        input_tokens = uncached + cached + cache_write
+    else:
+        input_tokens = prompt
+        cached = min(input_tokens, usage["cached"])
+        cache_write = min(max(0, input_tokens - cached), usage["cache_write"])
+        uncached = input_tokens - cached - cache_write
+    output_tokens = min(usage["completion"], MAX_BIGINT - input_tokens)
     reasoning = min(output_tokens, usage["reasoning"])
     return {
         "input_tokens": input_tokens,
@@ -331,7 +363,7 @@ def _tokens(usage: dict[str, int]) -> dict[str, int]:
         "output_tokens": output_tokens,
         "reasoning_output_tokens": reasoning,
         "total_tokens": _sum(input_tokens, output_tokens),
-        "uncached_input_tokens": input_tokens - cached - cache_write,
+        "uncached_input_tokens": uncached,
         "visible_output_tokens": output_tokens - reasoning,
         "unattributed_tokens": 0,
     }
@@ -368,11 +400,6 @@ def _has_visible_content(value: object) -> bool:
     )
 
 
-def _add_tokens(target: dict[str, Any], values: dict[str, int]) -> None:
-    for key in empty_tokens():
-        target[key] = _sum(target[key], values[key])
-
-
 def _counter(value: object) -> int:
     if isinstance(value, bool):
         return 0