agentshim 0.24.0__tar.gz → 0.24.1__tar.gz

{agentshim-0.24.0 → agentshim-0.24.1}/CHANGELOG.md

@@ -1,5 +1,78 @@
 # Changelog
 
+## 0.24.1 (2026-10-31)
+
+Fixes the usage numbers 0.24.0 got wrong. Recorded cost does not move for any single-result turn:
+every price, `price_for` and `cost_usd` are as in 0.24.0, and `tests/fixtures/recorded_costs.json`
+still proves it.
+
+### Fixed
+
+- **Claude transcripts undercounted output by about a third.** The CLI writes one line per content
+  block of a message, and the lines do not repeat the same usage: the prompt side is identical, but
+  `output_tokens` grows as the reply streams, from a few tokens on the first line to the whole reply
+  on the last. 0.24.0 kept the first line and dropped the rest. A message is now counted once, at
+  the most any line reported, and the records of a message add up to its final usage: the first
+  line yields the request (`turns=1`), each later line yields only the tokens it added (`turns=0`,
+  usually only output). Sum tokens over records, and count requests by summing `turns`. An advisor
+  call that shows only on a later line is counted too. A position saved by 0.24.0 has no such
+  counts, so it is refused and the file is read again from the start (`reset`), which also corrects
+  what 0.24.0 missed.
+- **`ProviderUsage.by_model` counted the first turn after a steer twice.** `modelUsage` is
+  cumulative over the CLI process, and the parser added each result's rows to the last. The rows of
+  a run are now those of its last result, which cover the run once, and the rows sum to
+  `total_cost_usd`. The same seam is fixed for the primary row's `turns` (the run's, not the last
+  result's) and for the one-hour share of cache writes (the run's, applied to the run's writes), so a
+  steered run whose results differ in cache TTL is no longer priced at the last result's share.
+  A single result is unchanged.
+- **A moved Codex session file was counted twice.** Codex archives a rollout by moving it from
+  `sessions` to `archived_sessions` under the same name. agentshim now owns that: a file with the
+  name of one already read is read on from the furthest position held for it (after the usual check
+  of the bytes before that position), and a copy that is shorter than what was read holds nothing
+  new. Keep one `known` mapping for all roots of a provider, as before. A caller no longer
+  de-duplicates anything. **Removed:** `TranscriptRoot.shared_sessions` and
+  `TranscriptFormat.shared_sessions`, which only told the caller to do this itself; a format sets
+  `session_file` (the name every copy of a session's file has) instead.
+- **A fetched LiteLLM document took known prices away.** It replaced the pinned table whole, so a
+  model LiteLLM dropped or renamed lost its estimate, and a row listed at zero before its price was
+  known turned a known price into $0. A document is now laid over the pinned table: a model it lacks
+  keeps its pinned rates, a row with no input or output price does not replace a pinned price, and a
+  speed tier with no price under a priced standard is dropped. A model only the document knows is
+  taken as listed, free or not. `RateRead.known_models` counts the result.
+- A `modelUsage` key with no name no longer raises out of the parser; its tokens are the row
+  `unknown`, which has no price, so the turn is `UNKNOWN_MODEL` as in 0.23.1. The CLI's
+  `<synthetic>` placeholder bills nothing and no longer makes a turn `UNKNOWN_MODEL`.
+- `ultrafast` is read as `UsageSpeed.ULTRAFAST` in a Claude result and in a transcript (it read as
+  none, and as standard). A speed agentshim has no name for is still `None` (read as standard).
+- `read_rates` and `refresh_rates` reject a naive `now` with `ValueError` before anything is
+  fetched. A clock set back is tolerated: a time stamped ahead of `now` by more than the floor has an
+  unknown age and counts as expired (it was fresh until the clock caught up), and an attempt more than
+  the floor away no longer holds off a refresh.
+- The Codex app-server transport skips the `by_model` row of a model name that is blank.
+
+### Changed
+
+- `read_transcript_usage(..., include_cwd=False)`: records carry `cwd` only when asked. It is an
+  absolute path on the user's machine, so a record a caller stores or sends holds no personal path by
+  default.
+- `ModelUsage.recorded_usd` is documented as what it was: the pinned table's standard-tier list
+  price, no bill. A row whose `speed` is fast or ultrafast was billed more; docs/pricing.md shows
+  how to price the tier with `rates_for`, `price_tokens` and `speed_premium_usd`. Recorded cost still
+  does not model speed tiers.
+- docs/transcripts.md states that `since` selects files, not records (filter on `timestamp`), and
+  that a line is held in memory whole while it is read.
+
+### Added
+
+- Tests built from data shaped like what the CLIs write, not idealised input:
+  `tests/unit/transcripts/shapes.py` (streamed Claude messages whose output grows, advisor calls
+  that show late, Codex session files that grow, move and are copied),
+  `tests/unit/stream_shapes.py` (a steered Claude process with cumulative `modelUsage`),
+  `tests/unit/rate_shapes.py` (partial, renamed and placeholder LiteLLM documents), and a synthetic
+  session file shaped like a real streamed message (`tests/fixtures/claude_session`). Properties over
+  them: records summed over any read split equal the final usage; a moved or copied session is
+  counted once; steered results equal one result of the whole run; no known price becomes 0.
+
 ## 0.24.0 (2026-10-30)
 
 Everything here is additive. Recorded cost does not move: every price, `price_for`, `cost_usd`,

{agentshim-0.24.0 → agentshim-0.24.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentshim
-Version: 0.24.0
+Version: 0.24.1
 Summary: Provider-agnostic coding-agent CLI shims
 License-Expression: MIT
 License-File: LICENSE

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/__init__.py

@@ -226,7 +226,7 @@ from .runtime import Agent
 from .session import Session, Turn, TurnTicket, map_mcp_servers
 from .transcripts import read_transcript_usage, transcript_providers, transcript_roots
 
-__version__ = "0.24.0"
+__version__ = "0.24.1"
 
 __all__ = [  # noqa: RUF022 - public API test uses Python's case-sensitive order
     "Agent",

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/core/transcripts.py

@@ -8,6 +8,8 @@ is each provider's :class:`TranscriptFormat`.
 
 agentshim keeps no state between calls. The caller remembers a
 :class:`FilePosition` per file and hands it back, and gets only what is new.
+The positions of every root of a provider go in one mapping: a session file
+that moves between roots is then read on from where the old path stopped.
 """
 
 from __future__ import annotations
@@ -40,28 +42,30 @@ class TranscriptRoot:
     Attributes:
         provider: The provider that wrote them.
         path: The directory, resolved from the environment (it may not exist).
-        shared_sessions: Whether one session's records can appear in more than
-            one file under this provider, because a session file moved after
-            it was read. A caller then counts each session from one file only.
     """
 
     provider: str
     path: Path
-    shared_sessions: bool = False
 
 
 @dataclass(frozen=True)
 class TranscriptRecord:
-    """One model request's usage, as a transcript recorded it.
+    """One model request's usage, or the part of it one line of a transcript added.
 
     Attributes:
-        timestamp: When the request was recorded (timezone-aware).
+        timestamp: When the request was recorded (timezone-aware). For the
+            tokens a later line of the file added, when that line was written.
         model: The model id as the transcript names it.
         session_id: The session it belongs to; a subagent's records carry
             their parent's session.
-        cwd: The working directory the session ran in, when the file says.
+        cwd: The working directory the session ran in: an absolute path on the
+            user's machine, so a personal one. ``None`` unless the caller asked
+            for it with ``include_cwd``, and then only when the file says.
         tokens: The request's tokens. ``input_tokens`` includes cache reads and
-            writes, as everywhere in agentshim.
+            writes, as everywhere in agentshim. A request is counted once, at
+            the most any line of the file reported: the records of one request
+            add up to its final usage, the first holding the request and its
+            ``turns``, and later ones (``turns`` 0) only what a later line added.
         speed: The service speed it was billed at.
         reported_usd: A cost the file states. ``None`` for every shipped
             format: a CLI's own figure is not trusted (see :mod:`agentshim.providers.claude.cost`).
@@ -153,7 +157,9 @@ class TranscriptScan:
     Attributes:
         files: The files read this call, in path order.
         unchanged: Files whose size and modification time match the known
-            position of a finished read, so nothing was read.
+            position of a finished read, and files that are earlier copies of
+            a session already read further under another path, so nothing was
+            read.
         skipped: Files too old to matter for the window.
         pending: Files not finished: the byte budget ran out first. Call again
             with the new positions until it is zero.
@@ -211,7 +217,11 @@ class TranscriptFormat:
         suffix: Which file names are transcripts.
         unreadable_suffixes: Transcript names agentshim cannot read (a
             compressed archive). They are reported in ``failed``, not skipped.
-        shared_sessions: See :attr:`TranscriptRoot.shared_sessions`.
+        session_file: The name every copy of one session's file has, for a
+            provider that moves or copies a file between its directories after
+            it was read. A file with the name of one already read is read on
+            from there, so the move adds nothing twice. ``None`` when a file
+            keeps its path.
     """
 
     subdirs: tuple[str, ...]
@@ -220,7 +230,7 @@ class TranscriptFormat:
     parser: Callable[[Path, bytes | None], TranscriptParser | None]
     suffix: str = ".jsonl"
     unreadable_suffixes: tuple[str, ...] = field(default=())
-    shared_sessions: bool = False
+    session_file: Callable[[Path], str] | None = None
 
     def __post_init__(self) -> None:
         """Reject a format that would read nothing, or read non-ASCII markers."""

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/core/usage.py

@@ -293,9 +293,14 @@ class ModelUsage:
         model: The model id as the provider reports it.
         tokens: This turn's tokens for this model. Only the primary model
             carries the turn's ``turns`` count.
-        recorded_usd: The pinned-table price of ``tokens``; ``None`` when the
-            table has no price for the model. This is the figure a turn's
-            recorded cost sums, never the CLI's own.
+        recorded_usd: The pinned-table price of ``tokens`` at the standard
+            tier; ``None`` when the table has no price for the model. This is
+            the figure a turn's recorded cost sums, never the CLI's own. It is
+            not a bill: the pinned table has no speed tiers, so a row whose
+            ``speed`` is fast or ultrafast was billed more than this. For the
+            price at the tier, use :func:`agentshim.price_tokens` with the
+            entry's rates at ``speed``, or add
+            :func:`agentshim.speed_premium_usd`.
         speed: The service speed the turn ran at, on the primary model only;
             ``None`` when the provider did not report one (read as standard).
         primary: Whether this is the model the turn was run with. Exactly one
@@ -332,7 +337,10 @@ class ModelUsage:
 def merge_model_usage(
     first: Sequence[ModelUsage], second: Sequence[ModelUsage]
 ) -> tuple[ModelUsage, ...]:
-    """Sum the per-model usage of two turns of one run into one.
+    """Sum the per-model usage of two separate turns into one.
+
+    This adds turns that each have their own rows. Rows a parser built from a
+    CLI's cumulative counts already cover the whole run and must not be added.
 
     Rows with the same model and speed add up, and a sum is unpriced if either
     part was. The primary row of *first* (else of *second*) stays the only

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/providers/claude/cost.py

@@ -22,12 +22,16 @@ from agentshim.core.rates import candidate_names
 from agentshim.core.usage import CostSource, ModelUsage, TokenUsage, UsageSpeed, normalized_usage
 from agentshim.pricing import price_for
 
+from .model_data import SYNTHETIC_MODEL
+
 if TYPE_CHECKING:
     from collections.abc import Iterable, Mapping, Sequence
 
     from agentshim.core.pricing import PricingTable
 
 PROVIDER_NAME = "claude"
+#: What a ``modelUsage`` key with no name is called: its tokens are real, but no price can match.
+UNNAMED_MODEL = "unknown"
 
 
 @dataclass(frozen=True)
@@ -60,6 +64,17 @@ class ModelCounts:
             canonical=self.canonical,
         )
 
+    def plus(self, other: ModelCounts) -> ModelCounts:
+        """These counts and *other*'s added together, naming the model as this one does."""
+        return ModelCounts(
+            input_tokens=self.input_tokens + other.input_tokens,
+            output_tokens=self.output_tokens + other.output_tokens,
+            cache_read_tokens=self.cache_read_tokens + other.cache_read_tokens,
+            cache_write_tokens=self.cache_write_tokens + other.cache_write_tokens,
+            reasoning_tokens=self.reasoning_tokens + other.reasoning_tokens,
+            canonical=self.canonical or other.canonical,
+        )
+
     @property
     def empty(self) -> bool:
         """Whether no token was counted."""
@@ -87,7 +102,12 @@ class TurnCost:
 
 
 def parse_model_usage(raw: object) -> dict[str, ModelCounts] | None:
-    """Read a ``result`` frame's ``modelUsage``; ``None`` when it is absent or malformed."""
+    """Read a ``result`` frame's ``modelUsage``; ``None`` when it is absent or malformed.
+
+    A key with no name (empty or blank) is a CLI fault, not a reason to fail the turn: its
+    tokens are kept under :data:`UNNAMED_MODEL`, which has no price, so the turn is
+    ``UNKNOWN_MODEL`` with no cost as it is for any other model agentshim cannot price.
+    """
     if not isinstance(raw, dict):
         return None
     parsed: dict[str, ModelCounts] = {}
@@ -96,7 +116,7 @@ def parse_model_usage(raw: object) -> dict[str, ModelCounts] | None:
             return None
         fields = cast("Mapping[str, Any]", counts)
         canonical = fields.get("canonicalModel")
-        parsed[model] = ModelCounts(
+        read = ModelCounts(
             input_tokens=_count(fields, "inputTokens"),
             output_tokens=_count(fields, "outputTokens"),
             cache_read_tokens=_count(fields, "cacheReadInputTokens"),
@@ -104,6 +124,8 @@ def parse_model_usage(raw: object) -> dict[str, ModelCounts] | None:
             reasoning_tokens=_count(fields, "thinkingTokens"),
             canonical=canonical if isinstance(canonical, str) else None,
         )
+        name = model if model.strip() else UNNAMED_MODEL
+        parsed[name] = parsed[name].plus(read) if name in parsed else read
     return parsed
 
 
@@ -118,13 +140,15 @@ def priced_turn(
 
     *one_hour_share* is the fraction of cache writes billed at the one-hour
     rate. A model with tokens in the turn and no price makes the whole turn
-    ``UNKNOWN_MODEL`` with no cost: a partial sum would read as complete.
+    ``UNKNOWN_MODEL`` with no cost: a partial sum would read as complete. The
+    CLI's own placeholder for a message it wrote itself bills nothing and has
+    no row.
     """
     total: float | None = 0.0
     rows: list[ModelUsage] = []
     for model, counts in model_usage.items():
         delta = counts.since(baseline.get(model))
-        if delta.empty:
+        if delta.empty or model == SYNTHETIC_MODEL:
             continue
         pricing = price_for(PROVIDER_NAME, model, table)
         if pricing is None and delta.canonical is not None:
@@ -181,9 +205,14 @@ def mark_primary(
 
 
 def report_speed(usage: Mapping[str, Any] | None) -> UsageSpeed | None:
-    """The speed a ``result`` frame's ``usage`` says the turn ran at; ``None`` if it says none."""
+    """The speed a ``usage`` mapping says the request ran at.
+
+    ``None`` when it says none, or names a speed agentshim has no
+    :class:`UsageSpeed` for: the caller then reads it as standard, and the
+    figure is a list price of the standard tier whatever was billed.
+    """
     speed = None if usage is None else usage.get("speed")
-    return {"standard": UsageSpeed.STANDARD, "fast": UsageSpeed.FAST}.get(str(speed))
+    return next((tier for tier in UsageSpeed if tier.value == speed), None)
 
 
 def _tokens(

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/providers/claude/model_data.py

@@ -93,6 +93,8 @@ PRICING_ENTRIES = tuple(_anthropic(entry) for entry in RATE_ENTRIES)
 #: Flat rates with speed tiers, for estimates; the same generated numbers as ``PRICING_ENTRIES``.
 RATES = RateTable(SNAPSHOT_DATE, {entry.name: entry for entry in RATE_ENTRIES})
 
+#: The ``model`` of a message the CLI wrote itself rather than a model call, which bills nothing.
+SYNTHETIC_MODEL = "<synthetic>"
 #: Names the CLI reports that are not a model with a price: its placeholder for a message that
 #: never reached the API, and the bare family names, which do not say which generation ran.
-UNPRICEABLE_MODELS = frozenset({"<synthetic>", "synthetic", "opus", "sonnet", "haiku", "fable"})
+UNPRICEABLE_MODELS = frozenset({SYNTHETIC_MODEL, "synthetic", "opus", "sonnet", "haiku", "fable"})

{agentshim-0.24.0 → agentshim-0.24.1}/agentshim/providers/claude/parser.py

@@ -26,7 +26,6 @@ from agentshim.core.usage import (
     ModelUsage,
     ProviderUsage,
     TokenUsage,
-    merge_model_usage,
     normalized_usage,
 )
 
@@ -41,6 +40,7 @@ from .events import (
     parse_frame,
 )
 from .failures import SCHEMA_RETRIES_SUBTYPE, classify_failure
+from .model_data import SYNTHETIC_MODEL
 from .rate_limits import RateLimitFrame
 
 if TYPE_CHECKING:
@@ -96,10 +96,6 @@ def fold_usage(usage: Mapping[str, Any] | None, turns: int = 0) -> TokenUsage:
     )
 
 
-#: The ``model`` of a message Claude Code wrote itself rather than a model.
-SYNTHETIC_MODEL = "<synthetic>"
-
-
 def context_tokens(usage: Mapping[str, Any]) -> int:
     """Tokens one model call leaves in the context: its whole prompt plus its output.
 
@@ -343,14 +339,14 @@ class ClaudeStreamParser:
         self._final_text = frame.text
         self.total_cost_usd = frame.total_cost_usd
         tokens = fold_usage(frame.usage, turns=frame.num_turns or 0)
-        self._price(frame, tokens)
-        self._structured = self._structured_payload(frame)
-        by_model = self._by_model
         if self._results:
-            # A steered message can start a follow-on turn inside this run; its
-            # ``result`` reports only its own tokens, so the run sums them.
+            # A steered message can start a follow-on turn inside this run. Its ``usage`` reports
+            # only its own tokens, so the run sums them; ``modelUsage`` and ``total_cost_usd``
+            # are cumulative over the process, so the cost and the per-model rows priced from
+            # them already cover every result of the run and are taken as they are, not added.
             tokens = self._usage.tokens + tokens
-            by_model = merge_model_usage(self._usage.by_model, by_model)
+        self._price(frame, tokens)
+        self._structured = self._structured_payload(frame)
         self._results += 1
         self._usage = ProviderUsage(
             tokens=tokens,
@@ -358,7 +354,7 @@ class ClaudeStreamParser:
             provider=PROVIDER_NAME,
             raw=frame.usage,
             cost_source=self._cost_source,
-            by_model=by_model,
+            by_model=self._by_model,
         )
         self._emit(UsageReport(self._usage, self._cost_usd))
         if frame.is_error:
@@ -369,7 +365,12 @@ class ClaudeStreamParser:
             self._emit(ProviderError(self._error))
 
     def _price(self, frame: ResultFrame, tokens: TokenUsage) -> None:
-        """Set the turn's cost from its per-model tokens, or from the CLI's when it names none."""
+        """Set the run's cost from its per-model tokens, or from the CLI's when it names none.
+
+        *tokens* are the run's so far, the same span the cumulative ``modelUsage`` covers: their
+        one-hour share of cache writes applies to those writes, and their turn count goes on the
+        primary row.
+        """
         if frame.model_usage is not None:
             self.model_usage_total = frame.model_usage
             writes = tokens.cache_write_input_tokens