wiretruth 0.2.0__tar.gz → 0.3.0__tar.gz

{wiretruth-0.2.0/src/wiretruth.egg-info → wiretruth-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wiretruth
-Version: 0.2.0
+Version: 0.3.0
 Summary: Replays real LLM provider responses through LLM SDKs and checks whether each SDK reports what actually happened.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/kartsan03/wiretruth
@@ -38,8 +38,10 @@ happened.
 [![PyPI](https://img.shields.io/pypi/v/wiretruth?label=PyPI)](https://pypi.org/project/wiretruth/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/kartsan03/wiretruth/blob/main/LICENSE)
 
-So far wiretruth tests one library, any-llm, on six APIs: OpenAI Chat Completions and Responses, Ollama's native
-and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic Messages. That is 24 scenarios and 95 cells.
+wiretruth tests any-llm, LiteLLM, pydantic-ai, the Vercel AI SDK and TanStack AI across six APIs: OpenAI Chat
+Completions and Responses, Ollama's native and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic
+Messages. That is 24 scenarios and 100 cells per library. pydantic-ai and the Vercel AI SDK support only the OpenAI-compatible Ollama
+endpoint; their native Ollama cells are UNSUPPORTED.
 The Anthropic responses are transcribed from Anthropic's documentation rather than recorded: each of those fixtures
 is marked `documented` and links the page it comes from. More providers and libraries follow, and a public
 results matrix arrives in v0.4. Until then, the grades are in
@@ -52,11 +54,21 @@ A model hits its output token limit halfway through a JSON object. The provider
 parse error, nothing tells you that a higher limit would fix it. If it returns the partial object as a success,
 your pipeline stores an incomplete record.
 
-wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through any-llm:
+wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through each library:
 
 | Library | What the caller gets | Grade |
 |---|---|---|
-| any-llm 1.33.0 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| any-llm 1.40.2 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| LiteLLM 1.105.5 | Returns the cut-off JSON as text in `message.content` with `finish_reason: "length"` and parses none of it. | ✓ PASS |
+| pydantic-ai 2.61.0 | Retries once, then raises `UnexpectedModelBehavior("Exceeded maximum output retries (1)")`. The `length` finish is only in the messages that `capture_run_messages()` keeps. | ~ LOSSY |
+| Vercel AI SDK 7.0.153 | Raises `NoObjectGeneratedError` with `finishReason: "length"` and the token usage. | ✓ PASS |
+
+pydantic-ai's grades for structured output use `NativeOutput`, the provider's own JSON mode, because that is how
+the fixtures were recorded. It is opt-in: by default pydantic-ai asks for a forced tool call instead on every API
+here except Anthropic, and no scenario covers that path yet. On `gpt-6-luna` that default request is rejected with the same 400
+that `request.tools-with-reasoning-chat` replays (function tools without `reasoning_effort: "none"`). The Vercel AI
+SDK sends a JSON schema by default on every API here except OpenAI-compatible Ollama, where it defaults to JSON-object
+mode; its shim opts into the documented `supportsStructuredOutputs` setting there.
 
 wiretruth runs the same recorded edge cases through every library it supports and grades each one against what the
 provider sent.
@@ -93,6 +105,7 @@ Each scenario lists its requirements. MUST requirements set the grade. SHOULD re
 partial text in the truncation scenarios, add a note when they aren't met. How grades are computed:
 [docs/grading.md](https://github.com/kartsan03/wiretruth/blob/main/docs/grading.md).
 Scenario catalog: [docs/scenarios.md](https://github.com/kartsan03/wiretruth/blob/main/docs/scenarios.md).
+Fixed-bug validation: [docs/validation.md](https://github.com/kartsan03/wiretruth/blob/main/docs/validation.md).
 
 ## Quickstart
 
@@ -101,12 +114,21 @@ You need Python 3.11 or newer and [uv](https://docs.astral.sh/uv/), which builds
 ```bash
 git clone https://github.com/kartsan03/wiretruth && cd wiretruth
 uv run wiretruth run --lib any-llm
+uv run wiretruth run --lib litellm
+uv run wiretruth run --lib pydantic-ai
+uv run wiretruth run --lib vercel-ai
+uv run wiretruth run --lib tanstack-ai
 ```
 
+The Vercel AI SDK and TanStack AI shims also need Node.js and npm. The runner builds their pinned dependencies
+automatically; see [Shims](shims/README.md) for supported providers and structured-output modes.
+
+The any-llm run reports (see [`results/any-llm/1.40.2.json`](https://github.com/kartsan03/wiretruth/blob/main/results/any-llm/1.40.2.json)):
+
 ```text
 scenario                                  anthropic      gemini   ollama   ollama-openai  openai-chat  openai-responses
 finish.stop-sequence                      ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS
-finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✗ WRONG        ✗ WRONG      ✓ PASS
+finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 request.max-tokens-field                                                                  ✓ PASS
 request.max-tokens-ollama-openai                                           ✗ WRONG
 request.no-injected-options                                       ✓ PASS
@@ -116,8 +138,8 @@ stream.missing-finish                     ~ LOSSY        ~ LOSSY  ~ LOSSY  ~ LOS
 stream.missing-finish-abort               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8                               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8-split                         ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls                      ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls-stream               ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls                      ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls-stream               ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 truncation.reasoning-only                 ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 truncation.structured-output              ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ~ LOSSY
 truncation.structured-output-forced-tool  – UNSUPPORTED
@@ -126,8 +148,8 @@ truncation.text-stream                    ✓ PASS         ✓ PASS   ✓ PASS
 truncation.text-stream-usage                             ✓ PASS                           ✓ PASS
 truncation.tool-call-args                 ✓ PASS                                          ✓ PASS       ✓ PASS
 truncation.tool-call-args-stream          ✓ PASS                                          ✓ PASS       ✓ PASS
-usage.cached-tokens                       ✓ PASS
-usage.reasoning-tokens                                   ✓ PASS
+usage.cached-tokens                       ✓ PASS                                          ✓ PASS       ✓ PASS
+usage.reasoning-tokens                    ✓ PASS         ✓ PASS                           ✓ PASS       ✓ PASS
 usage.stream-final-chunk                  ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS
 
 finish.stop-sequence gemini
@@ -146,18 +168,6 @@ finish.unknown-value ollama
   MUST finish-is-other: CONTRADICTED: finish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"
   SHOULD raw-finish-reported: CONTRADICTED: finish.raw="stop"; truth.raw_finish="wiretruth_unknown_reason"
 
-finish.unknown-value ollama-openai
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Octopuses, like many marine invertebrates, employ sophisticated methods to camouflage"
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
-finish.unknown-value openai-chat
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Io, Europa, and Ganymede."
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
 request.max-tokens-ollama-openai ollama-openai
   MUST limit-field: CONTRADICTED: request[0].json_body.max_tokens=absent; required=16
 
@@ -190,18 +200,16 @@ stream.utf8-split gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-5201471136765337120_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_120847", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-5201471136765337120_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_120848", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="3972c760-858b-4728-a05a-279fc6e6a649", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="4d3747a6-5e75-4637-b82e-f269adbba01e", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="77ecdd8a-566d-4ba1-b098-d7a3b3082999", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="3a4cba16-a57a-4657-b28a-8ac89eaea7d9", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 tools.parallel-calls-stream gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-1615951408272134819_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_89893", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-1615951408272134819_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_89896", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls-stream ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="8b75c529-6eb6-4a63-a375-4551bfe56f39", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="8e864d64-6145-4ca4-81ba-cbdbad0523e1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="fc700b93-a9ac-4907-8da1-a7b38a28658c", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="265e0a1f-ef64-4209-b6b9-e3b5c9e3a181", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 truncation.structured-output gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -212,7 +220,7 @@ truncation.structured-output openai-responses
   SHOULD usage-reported: NOT_OBSERVABLE: usage.input=null (absent); truth.usage.input=68; usage.output=null (absent); truth.usage.output=24
 
 truncation.structured-output-forced-tool anthropic
-  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.33.0/src/any_llm/providers/anthropic/utils.py#L628-L655)
+  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.40.2/src/any_llm/providers/anthropic/utils.py#L660-L686)
 
 truncation.text gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -223,27 +231,45 @@ truncation.text-stream gemini
 truncation.text-stream-usage gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
+usage.reasoning-tokens anthropic
+  SHOULD reasoning-reported: NOT_OBSERVABLE: usage.reasoning=null (absent); truth.usage.reasoning=312
+
 usage.stream-final-chunk gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
-any-llm 1.33.0: 95 cells, 78 PASS, 6 LOSSY, 10 WRONG, 1 UNSUPPORTED
-stderr: .wiretruth/runs/20261013T134918.993664Z
+any-llm 1.40.2: 100 cells, 87 PASS, 6 LOSSY, 6 WRONG, 1 UNSUPPORTED
+stderr: .wiretruth/runs/20261109T005301.682490Z
 ```
 
-No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads any-llm
-into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference` runs the
-built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this repository,
-so wiretruth runs from a clone.
+The LiteLLM run grades its 100 cells 86 PASS, 13 WRONG and 1 UNSUPPORTED. Every cell, with its evidence, is in
+[`results/litellm/1.105.5.json`](https://github.com/kartsan03/wiretruth/blob/main/results/litellm/1.105.5.json).
+The pydantic-ai run grades its 100 cells 63 PASS, 16 LOSSY, 6 WRONG and 15 UNSUPPORTED (native Ollama), in
+[`results/pydantic-ai/2.61.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/pydantic-ai/2.61.0.json).
+The Vercel AI SDK run grades its 100 cells 84 PASS and 16 UNSUPPORTED (native Ollama, and the Anthropic
+forced-tool fixture, since current Claude models get native structured output), in
+[`results/vercel-ai/7.0.153.json`](https://github.com/kartsan03/wiretruth/blob/main/results/vercel-ai/7.0.153.json).
+The TanStack AI run grades its 100 cells 48 PASS, 2 LOSSY, 2 WRONG and 48 UNSUPPORTED, in
+[`results/tanstack-ai/1.0.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/tanstack-ai/1.0.0.json).
+Its `chat()` always requests a streamed response, so the non-streamed fixtures don't apply to it.
+
+No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads the
+selected library into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference`
+runs the built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this
+repository, so wiretruth runs from a clone.
 
 Releases are also on [PyPI](https://pypi.org/project/wiretruth/) (`pip install wiretruth`). The package is the
-command only, so run it inside a checkout of the matching release tag (for example `v0.2.0`); on `main`, use
+command only, so run it inside a checkout of the matching release tag (for example `v0.3.0`); on `main`, use
 `uv run` as above.
 
 ## Add your library
 
 A shim is a small program. It reads a call spec as JSON on stdin, calls your library the way its docs show, and
 prints what the library reported as JSON on stdout. The
-[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py) is 150 lines. The
+[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py),
+[LiteLLM shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/litellm/shim.py),
+[pydantic-ai shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/pydantic-ai/shim.py),
+[Vercel AI SDK shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/vercel-ai/shim.mjs) and
+[TanStack AI shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/tanstack-ai/shim.mjs) are examples. The
 protocol is in [shims/README.md](https://github.com/kartsan03/wiretruth/blob/main/shims/README.md); to propose a
 library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issues/new?template=new-library.yml).
 
@@ -267,7 +293,8 @@ library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issue
 ## Author note
 
 I've contributed fixes to libraries that wiretruth tests or will test:
-[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389) and
+[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389),
+[mozilla-ai/any-llm#1433](https://github.com/mozilla-ai/any-llm/pull/1433) and
 [TanStack/ai#1548](https://github.com/TanStack/ai/pull/1548). Grades are computed by the code in this repository.
 If you think a grade is wrong, please
 [open a grade dispute](https://github.com/kartsan03/wiretruth/issues/new?template=grade-dispute.yml).

{wiretruth-0.2.0 → wiretruth-0.3.0}/README.md

@@ -9,8 +9,10 @@ happened.
 [![PyPI](https://img.shields.io/pypi/v/wiretruth?label=PyPI)](https://pypi.org/project/wiretruth/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/kartsan03/wiretruth/blob/main/LICENSE)
 
-So far wiretruth tests one library, any-llm, on six APIs: OpenAI Chat Completions and Responses, Ollama's native
-and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic Messages. That is 24 scenarios and 95 cells.
+wiretruth tests any-llm, LiteLLM, pydantic-ai, the Vercel AI SDK and TanStack AI across six APIs: OpenAI Chat
+Completions and Responses, Ollama's native and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic
+Messages. That is 24 scenarios and 100 cells per library. pydantic-ai and the Vercel AI SDK support only the OpenAI-compatible Ollama
+endpoint; their native Ollama cells are UNSUPPORTED.
 The Anthropic responses are transcribed from Anthropic's documentation rather than recorded: each of those fixtures
 is marked `documented` and links the page it comes from. More providers and libraries follow, and a public
 results matrix arrives in v0.4. Until then, the grades are in
@@ -23,11 +25,21 @@ A model hits its output token limit halfway through a JSON object. The provider
 parse error, nothing tells you that a higher limit would fix it. If it returns the partial object as a success,
 your pipeline stores an incomplete record.
 
-wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through any-llm:
+wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through each library:
 
 | Library | What the caller gets | Grade |
 |---|---|---|
-| any-llm 1.33.0 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| any-llm 1.40.2 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| LiteLLM 1.105.5 | Returns the cut-off JSON as text in `message.content` with `finish_reason: "length"` and parses none of it. | ✓ PASS |
+| pydantic-ai 2.61.0 | Retries once, then raises `UnexpectedModelBehavior("Exceeded maximum output retries (1)")`. The `length` finish is only in the messages that `capture_run_messages()` keeps. | ~ LOSSY |
+| Vercel AI SDK 7.0.153 | Raises `NoObjectGeneratedError` with `finishReason: "length"` and the token usage. | ✓ PASS |
+
+pydantic-ai's grades for structured output use `NativeOutput`, the provider's own JSON mode, because that is how
+the fixtures were recorded. It is opt-in: by default pydantic-ai asks for a forced tool call instead on every API
+here except Anthropic, and no scenario covers that path yet. On `gpt-6-luna` that default request is rejected with the same 400
+that `request.tools-with-reasoning-chat` replays (function tools without `reasoning_effort: "none"`). The Vercel AI
+SDK sends a JSON schema by default on every API here except OpenAI-compatible Ollama, where it defaults to JSON-object
+mode; its shim opts into the documented `supportsStructuredOutputs` setting there.
 
 wiretruth runs the same recorded edge cases through every library it supports and grades each one against what the
 provider sent.
@@ -64,6 +76,7 @@ Each scenario lists its requirements. MUST requirements set the grade. SHOULD re
 partial text in the truncation scenarios, add a note when they aren't met. How grades are computed:
 [docs/grading.md](https://github.com/kartsan03/wiretruth/blob/main/docs/grading.md).
 Scenario catalog: [docs/scenarios.md](https://github.com/kartsan03/wiretruth/blob/main/docs/scenarios.md).
+Fixed-bug validation: [docs/validation.md](https://github.com/kartsan03/wiretruth/blob/main/docs/validation.md).
 
 ## Quickstart
 
@@ -72,12 +85,21 @@ You need Python 3.11 or newer and [uv](https://docs.astral.sh/uv/), which builds
 ```bash
 git clone https://github.com/kartsan03/wiretruth && cd wiretruth
 uv run wiretruth run --lib any-llm
+uv run wiretruth run --lib litellm
+uv run wiretruth run --lib pydantic-ai
+uv run wiretruth run --lib vercel-ai
+uv run wiretruth run --lib tanstack-ai
 ```
 
+The Vercel AI SDK and TanStack AI shims also need Node.js and npm. The runner builds their pinned dependencies
+automatically; see [Shims](shims/README.md) for supported providers and structured-output modes.
+
+The any-llm run reports (see [`results/any-llm/1.40.2.json`](https://github.com/kartsan03/wiretruth/blob/main/results/any-llm/1.40.2.json)):
+
 ```text
 scenario                                  anthropic      gemini   ollama   ollama-openai  openai-chat  openai-responses
 finish.stop-sequence                      ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS
-finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✗ WRONG        ✗ WRONG      ✓ PASS
+finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 request.max-tokens-field                                                                  ✓ PASS
 request.max-tokens-ollama-openai                                           ✗ WRONG
 request.no-injected-options                                       ✓ PASS
@@ -87,8 +109,8 @@ stream.missing-finish                     ~ LOSSY        ~ LOSSY  ~ LOSSY  ~ LOS
 stream.missing-finish-abort               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8                               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8-split                         ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls                      ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls-stream               ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls                      ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls-stream               ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 truncation.reasoning-only                 ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 truncation.structured-output              ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ~ LOSSY
 truncation.structured-output-forced-tool  – UNSUPPORTED
@@ -97,8 +119,8 @@ truncation.text-stream                    ✓ PASS         ✓ PASS   ✓ PASS
 truncation.text-stream-usage                             ✓ PASS                           ✓ PASS
 truncation.tool-call-args                 ✓ PASS                                          ✓ PASS       ✓ PASS
 truncation.tool-call-args-stream          ✓ PASS                                          ✓ PASS       ✓ PASS
-usage.cached-tokens                       ✓ PASS
-usage.reasoning-tokens                                   ✓ PASS
+usage.cached-tokens                       ✓ PASS                                          ✓ PASS       ✓ PASS
+usage.reasoning-tokens                    ✓ PASS         ✓ PASS                           ✓ PASS       ✓ PASS
 usage.stream-final-chunk                  ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS
 
 finish.stop-sequence gemini
@@ -117,18 +139,6 @@ finish.unknown-value ollama
   MUST finish-is-other: CONTRADICTED: finish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"
   SHOULD raw-finish-reported: CONTRADICTED: finish.raw="stop"; truth.raw_finish="wiretruth_unknown_reason"
 
-finish.unknown-value ollama-openai
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Octopuses, like many marine invertebrates, employ sophisticated methods to camouflage"
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
-finish.unknown-value openai-chat
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Io, Europa, and Ganymede."
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
 request.max-tokens-ollama-openai ollama-openai
   MUST limit-field: CONTRADICTED: request[0].json_body.max_tokens=absent; required=16
 
@@ -161,18 +171,16 @@ stream.utf8-split gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-5201471136765337120_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_120847", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-5201471136765337120_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_120848", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="3972c760-858b-4728-a05a-279fc6e6a649", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="4d3747a6-5e75-4637-b82e-f269adbba01e", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="77ecdd8a-566d-4ba1-b098-d7a3b3082999", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="3a4cba16-a57a-4657-b28a-8ac89eaea7d9", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 tools.parallel-calls-stream gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-1615951408272134819_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_89893", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-1615951408272134819_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_89896", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls-stream ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="8b75c529-6eb6-4a63-a375-4551bfe56f39", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="8e864d64-6145-4ca4-81ba-cbdbad0523e1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="fc700b93-a9ac-4907-8da1-a7b38a28658c", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="265e0a1f-ef64-4209-b6b9-e3b5c9e3a181", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 truncation.structured-output gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -183,7 +191,7 @@ truncation.structured-output openai-responses
   SHOULD usage-reported: NOT_OBSERVABLE: usage.input=null (absent); truth.usage.input=68; usage.output=null (absent); truth.usage.output=24
 
 truncation.structured-output-forced-tool anthropic
-  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.33.0/src/any_llm/providers/anthropic/utils.py#L628-L655)
+  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.40.2/src/any_llm/providers/anthropic/utils.py#L660-L686)
 
 truncation.text gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -194,27 +202,45 @@ truncation.text-stream gemini
 truncation.text-stream-usage gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
+usage.reasoning-tokens anthropic
+  SHOULD reasoning-reported: NOT_OBSERVABLE: usage.reasoning=null (absent); truth.usage.reasoning=312
+
 usage.stream-final-chunk gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
-any-llm 1.33.0: 95 cells, 78 PASS, 6 LOSSY, 10 WRONG, 1 UNSUPPORTED
-stderr: .wiretruth/runs/20261013T134918.993664Z
+any-llm 1.40.2: 100 cells, 87 PASS, 6 LOSSY, 6 WRONG, 1 UNSUPPORTED
+stderr: .wiretruth/runs/20261109T005301.682490Z
 ```
 
-No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads any-llm
-into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference` runs the
-built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this repository,
-so wiretruth runs from a clone.
+The LiteLLM run grades its 100 cells 86 PASS, 13 WRONG and 1 UNSUPPORTED. Every cell, with its evidence, is in
+[`results/litellm/1.105.5.json`](https://github.com/kartsan03/wiretruth/blob/main/results/litellm/1.105.5.json).
+The pydantic-ai run grades its 100 cells 63 PASS, 16 LOSSY, 6 WRONG and 15 UNSUPPORTED (native Ollama), in
+[`results/pydantic-ai/2.61.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/pydantic-ai/2.61.0.json).
+The Vercel AI SDK run grades its 100 cells 84 PASS and 16 UNSUPPORTED (native Ollama, and the Anthropic
+forced-tool fixture, since current Claude models get native structured output), in
+[`results/vercel-ai/7.0.153.json`](https://github.com/kartsan03/wiretruth/blob/main/results/vercel-ai/7.0.153.json).
+The TanStack AI run grades its 100 cells 48 PASS, 2 LOSSY, 2 WRONG and 48 UNSUPPORTED, in
+[`results/tanstack-ai/1.0.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/tanstack-ai/1.0.0.json).
+Its `chat()` always requests a streamed response, so the non-streamed fixtures don't apply to it.
+
+No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads the
+selected library into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference`
+runs the built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this
+repository, so wiretruth runs from a clone.
 
 Releases are also on [PyPI](https://pypi.org/project/wiretruth/) (`pip install wiretruth`). The package is the
-command only, so run it inside a checkout of the matching release tag (for example `v0.2.0`); on `main`, use
+command only, so run it inside a checkout of the matching release tag (for example `v0.3.0`); on `main`, use
 `uv run` as above.
 
 ## Add your library
 
 A shim is a small program. It reads a call spec as JSON on stdin, calls your library the way its docs show, and
 prints what the library reported as JSON on stdout. The
-[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py) is 150 lines. The
+[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py),
+[LiteLLM shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/litellm/shim.py),
+[pydantic-ai shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/pydantic-ai/shim.py),
+[Vercel AI SDK shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/vercel-ai/shim.mjs) and
+[TanStack AI shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/tanstack-ai/shim.mjs) are examples. The
 protocol is in [shims/README.md](https://github.com/kartsan03/wiretruth/blob/main/shims/README.md); to propose a
 library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issues/new?template=new-library.yml).
 
@@ -238,7 +264,8 @@ library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issue
 ## Author note
 
 I've contributed fixes to libraries that wiretruth tests or will test:
-[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389) and
+[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389),
+[mozilla-ai/any-llm#1433](https://github.com/mozilla-ai/any-llm/pull/1433) and
 [TanStack/ai#1548](https://github.com/TanStack/ai/pull/1548). Grades are computed by the code in this repository.
 If you think a grade is wrong, please
 [open a grade dispute](https://github.com/kartsan03/wiretruth/issues/new?template=grade-dispute.yml).

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/__init__.py

@@ -1,3 +1,3 @@
 """wiretruth: replay real LLM provider responses through LLM SDKs and grade what they report."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/checks.py

@@ -291,7 +291,7 @@ def tool_calls_match_truth(
 def no_complete_tool_call_from_partial_args(
     obs: Observation, truth: Truth, req_log: Sequence[LoggedRequest], args: JsonObject
 ) -> CheckResult:
-    """Reject parsed calls returned without an error when their truth arguments were partial.
+    """Reject parsed calls without an error when truth arguments were partial or unmatched.
 
     A raised outcome, an error object or a primary ``error`` finish satisfies first. Match truth
     by id, falling back to position when no id matches. Unparsed arguments are visibly partial
@@ -305,7 +305,7 @@ def no_complete_tool_call_from_partial_args(
             (item for item in truth.tool_calls if item.id is not None and item.id == call.id),
             truth.tool_calls[index] if index < len(truth.tool_calls) else None,
         )
-        if call.arguments_parsed_ok and expected is not None and not expected.arguments_parsed_ok:
+        if call.arguments_parsed_ok and (expected is None or not expected.arguments_parsed_ok):
             return CheckResult(
                 "CONTRADICTED",
                 f"tool_calls[{index}] id={_show(call.id)} returned without error; "

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/cli.py

@@ -18,7 +18,9 @@ from wiretruth import __version__
 from wiretruth.catalog import render as render_catalog
 from wiretruth.envs import (
     RunError,
+    build_javascript_env,
     find_library,
+    installed_javascript_packages,
     installed_packages,
     list_envs,
     prune_envs,
@@ -147,8 +149,12 @@ def build_parser() -> argparse.ArgumentParser:
             )
         elif name == "envs":
             action = subparser.add_mutually_exclusive_group()
-            action.add_argument("--list", action="store_true", help="list environments (default)")
-            action.add_argument("--prune", action="store_true", help="remove stale environments")
+            action.add_argument(
+                "--list", action="store_true", help="list Python environments (default)"
+            )
+            action.add_argument(
+                "--prune", action="store_true", help="remove stale Python environments"
+            )
             action.add_argument(
                 "--build", metavar="NAME", help="build or reuse a library environment"
             )
@@ -317,6 +323,8 @@ def run_record(args: argparse.Namespace) -> int:
         if args.dry_run:
             for scenario, request in zip(scenarios, requests, strict=True):
                 print(f"# {scenario.id}")
+                if scenario.warm_up:
+                    print("# Warm-up: send this request twice; keep only the second response.")
                 print(describe_request(provider, request, provider.API_BASE))
             return 0
         key = api_key(provider, os.environ)
@@ -478,8 +486,13 @@ def run_shim(args: argparse.Namespace) -> int:
     if saved is not None:
         try:
             interpreter = resolve_interpreter(Path(), manifest)
-            if interpreter is not None:
-                changes = package_changes(saved.packages, installed_packages(interpreter))
+            if manifest.language == "javascript" or interpreter is not None:
+                packages = (
+                    installed_javascript_packages(shim_dir)
+                    if manifest.language == "javascript"
+                    else installed_packages(cast(str, interpreter))
+                )
+                changes = package_changes(saved.packages, packages)
                 if changes:
                     print(f"packages differ from {path}: {', '.join(changes)}")
         except (RunError, OSError) as error:
@@ -499,6 +512,9 @@ def run_envs(args: argparse.Namespace) -> int:
         if args.build is not None:
             shim_dir = find_library(Path(), args.build)
             manifest = load_shim_manifest(shim_dir / "wiretruth-shim.json")
+            if manifest.language == "javascript":
+                print(build_javascript_env(shim_dir))
+                return 0
             interpreter = resolve_interpreter(Path(), manifest)
             if interpreter is None:
                 raise RunError("only Python library environments can be built")

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/envs.py

@@ -1,7 +1,9 @@
-"""Cached Python environments for pinned library shims."""
+"""Cached environments for pinned library shims."""
 
+import hashlib
 import json
 import os
+import re
 import shutil
 import subprocess
 import sys
@@ -99,6 +101,44 @@ def installed_packages(interpreter: str) -> dict[str, str]:
     return parse_installed_packages(result.stdout)
 
 
+def build_javascript_env(shim_dir: Path) -> Path:
+    path = shim_dir.resolve() / "node_modules"
+    marker = path / MARKER
+    try:
+        digest = hashlib.sha256((shim_dir / "package-lock.json").read_bytes()).hexdigest()
+        if marker.is_file() and marker.read_bytes() == digest.encode():
+            return path
+        marker.unlink(missing_ok=True)
+        command = ["npm", "ci", "--ignore-scripts", "--prefix", str(shim_dir.resolve())]
+        result = subprocess.run(command, stdout=sys.stderr, check=False)
+        if result.returncode:
+            raise RunError(f"{' '.join(command)} exited with code {result.returncode}")
+        path.mkdir(exist_ok=True)
+        marker.write_text(digest)
+    except OSError as error:
+        raise RunError(f"JavaScript environment: {error}") from error
+    return path
+
+
+def installed_javascript_packages(shim_dir: Path) -> dict[str, str]:
+    path = shim_dir / "node_modules" / ".package-lock.json"
+    try:
+        packages = json.loads(path.read_text())["packages"]
+        if not isinstance(packages, dict):
+            raise ValueError("expected a packages object")
+        installed = {}
+        for name, entry in packages.items():
+            match = re.fullmatch(r"node_modules/((?:@[^/]+/)?[^/]+)", name)
+            if match is not None:
+                version = entry["version"]
+                if not isinstance(version, str) or not version:
+                    raise ValueError("expected a package version")
+                installed[match[1]] = version
+        return installed
+    except (OSError, ValueError, KeyError, TypeError) as error:
+        raise RunError(f"{path}: cannot read installed packages: {error}") from error
+
+
 def list_envs(root: Path) -> tuple[tuple[Path, bool], ...]:
     """List cached directories and whether their installed pins match a current manifest."""
     manifests = [load_shim_manifest(path) for path in (root / "shims").rglob("wiretruth-shim.json")]

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/formats.py

@@ -575,6 +575,7 @@ class Scenario:
     call_overrides: Mapping[str, Call]  # the full call, override applied, per overridden provider
     grading_notes: str | None
     references: tuple[str, ...]
+    warm_up: bool = False
 
     def call_for(self, provider: str) -> Call:
         return self.call_overrides.get(provider, self.call)
@@ -610,6 +611,7 @@ def load_scenario(path: Path | str) -> Scenario:
         ),
         grading_notes=_opt_str(doc, "grading_notes"),
         references=_strings(doc, "references"),
+        warm_up=cast(bool, _object(doc, "recording").get("warm_up", False)),
     )
 
 

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/grading.py

@@ -98,6 +98,26 @@ def _requirements(
     return tuple(results)
 
 
+def is_model_lookup(
+    provider: str, request: LoggedRequest, request_log: Sequence[LoggedRequest]
+) -> bool:
+    """Native Ollama's unmatched ``POST /api/show`` beside a ``/api/chat`` that reached its fixture.
+
+    Model discovery is not an inference call. It stays in the request log and its count, but
+    neither the mismatch check nor the timeout rule treats it as the inference request.
+    """
+    return (
+        provider == "ollama"
+        and request.method == "POST"
+        and request.path == "/api/show"
+        and request.exchange is None
+        and any(
+            item.method == "POST" and item.path == "/api/chat" and item.exchange is not None
+            for item in request_log
+        )
+    )
+
+
 def grade_cell(
     run: ShimRun,
     *,
@@ -117,7 +137,8 @@ def grade_cell(
     """
     applicable = _applicable(scenario, provider)
     received = bool(run.request_log)
-    served = received and run.request_log[-1].response_complete
+    inference = [r for r in run.request_log if not is_model_lookup(provider, r, run.request_log)]
+    served = bool(inference) and inference[-1].response_complete
     if run.timed_out:
         if served:
             return CellGrade("HANG", 1, (), ("timed out after the response was served",))

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/lint.py

@@ -38,7 +38,9 @@ from wiretruth.formats import (
     load_document,
     load_fixture,
     load_scenario,
+    load_shim_manifest,
     parse_json,
+    read_json,
 )
 from wiretruth.providers import PROVIDERS
 from wiretruth.secret_scan import scan_headers, scan_text
@@ -279,5 +281,17 @@ def lint(root: Path) -> Report:
                 message = f"{library[key]!r} doesn't match the file path: expected {expected!r}"
                 problem(path, f"$.library.{key}", message)
 
+    for path in sorted((root / "shims").glob("*/*/wiretruth-shim.json")):
+        report.files += 1
+        _scan_secrets(path, report)
+        manifest = _load(load_shim_manifest, path, report)
+        if manifest is not None and manifest.language == "javascript":
+            package_path = path.with_name("package.json")
+            report.files += 1
+            _scan_secrets(package_path, report)
+            package = _load(read_json, package_path, report)
+            if not isinstance(package, dict) or package.get("dependencies") != manifest.packages:
+                problem(path, "$.packages", "must equal package.json dependencies exactly")
+
     report.problems = sorted(map(_quiet, report.problems), key=lambda found: found.file)
     return report

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/recorder.py

@@ -451,6 +451,10 @@ def record_scenario(
     target = destination(
         root, provider.ID, scenario.id, replace=replace
     )  # before paying for a call
+    if scenario.warm_up:
+        warm_up = send(provider, request, base_url, key)
+        if not 200 <= warm_up.status < 300:
+            raise RecordError(f"warm-up returned HTTP {warm_up.status}; nothing was written")
     recorded_at = datetime.now(UTC)
     response = send(provider, request, base_url, key)
     raw = b"".join(response.pieces)
@@ -464,6 +468,14 @@ def record_scenario(
         truth = provider.extract_truth([build_exchange(exchange)], request.body)
     except ValueError as error:
         raise RecordError(f"can't compute the truth: {error}") from None
+    if scenario.warm_up:  # a warm-up exists to get a cache hit; a fixture without one tests nothing
+        usage = truth.get("usage")
+        cached = usage.get("cached_input") if isinstance(usage, dict) else None
+        if not isinstance(cached, int) or cached <= 0:
+            raise RecordError(
+                f"the response after the warm-up reports no cached input tokens "
+                f"(HTTP {response.status}, cached input {cached}); nothing was written"
+            )
     path = target.path
     fixture: JsonObject = {
         "format": "wiretruth.fixture/v1",

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/results.py

@@ -11,7 +11,13 @@ from typing import NamedTuple, cast
 
 from wiretruth import __version__
 from wiretruth.checks import Status
-from wiretruth.envs import RunError, installed_packages, resolve_interpreter
+from wiretruth.envs import (
+    RunError,
+    find_library,
+    installed_javascript_packages,
+    installed_packages,
+    resolve_interpreter,
+)
 from wiretruth.formats import (
     FormatError,
     JsonObject,
@@ -106,16 +112,20 @@ def collect_results(root: Path, report: RunReport, manifest: ShimManifest) -> Js
         if all((server, repository, run_id))
         else None
     )
-    interpreter = resolve_interpreter(root, manifest)
-    if interpreter is None:
-        raise RunError("--write-results needs a Python shim environment")
+    if manifest.language == "javascript":
+        packages = installed_javascript_packages(find_library(root, manifest.name))
+    else:
+        interpreter = resolve_interpreter(root, manifest)
+        if interpreter is None:
+            raise RunError("--write-results needs a Python shim environment")
+        packages = installed_packages(interpreter)
     return results_document(
         report,
         manifest,
         at=datetime.now(UTC),
         git_sha=git_sha,
         ci_run=ci_run,
-        packages=installed_packages(interpreter),
+        packages=packages,
     )
 
 

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/runner.py

@@ -18,7 +18,7 @@ from typing import cast, get_args
 
 from wiretruth.checks import Status
 from wiretruth.envs import RunError as RunError
-from wiretruth.envs import resolve_interpreter
+from wiretruth.envs import build_javascript_env, resolve_interpreter
 from wiretruth.formats import (
     CallSpec,
     Fixture,
@@ -33,7 +33,7 @@ from wiretruth.formats import (
     parse_json,
     read_json,
 )
-from wiretruth.grading import CellGrade, Grade, ShimRun, grade_cell
+from wiretruth.grading import CellGrade, Grade, ShimRun, grade_cell, is_model_lookup
 from wiretruth.providers import PROVIDERS
 from wiretruth.recorder import find_scenarios
 from wiretruth.replay import BASE_PATHS, HOST, LoggedRequest, ReplayServer
@@ -156,6 +156,8 @@ def process_to_run(
             invalid = (str(error).splitlines() or [type(error).__name__])[0][:300]
     mismatches = []
     for index, request in enumerate(request_log):
+        if is_model_lookup(provider, request, request_log):
+            continue
         if request.exchange is None:
             mismatches.append(f"request {index}: no fixture for {request.method} {request.path}")
         requested = PROVIDERS[provider].requested_stream(request.path, request.json_body)
@@ -263,6 +265,8 @@ def run_suite(
     shim_dir = shim_dir.resolve()
     manifest = load_shim_manifest(shim_dir / "wiretruth-shim.json")
     cells = find_cells(root, pattern, provider)
+    if manifest.language == "javascript":
+        build_javascript_env(shim_dir)
     interpreter = resolve_interpreter(root, manifest)
     stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S.%fZ")
     stderr_dir = root / ".wiretruth" / "runs" / stamp

{wiretruth-0.2.0 → wiretruth-0.3.0}/src/wiretruth/schemas/scenario.schema.json

@@ -52,6 +52,18 @@
       }
     },
     "grading_notes": { "$ref": "common.schema.json#/$defs/nonempty_string" },
+    "recording": {
+      "description": "Recorder options; these do not change the replayed call.",
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "warm_up": {
+          "description": "Send the identical request once first, discard its response, and record only the second exchange, which must report cached input tokens.",
+          "type": "boolean",
+          "default": false
+        }
+      }
+    },
     "references": {
       "type": "array",
       "items": { "$ref": "common.schema.json#/$defs/url" }

{wiretruth-0.2.0 → wiretruth-0.3.0/src/wiretruth.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: wiretruth
-Version: 0.2.0
+Version: 0.3.0
 Summary: Replays real LLM provider responses through LLM SDKs and checks whether each SDK reports what actually happened.
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/kartsan03/wiretruth
@@ -38,8 +38,10 @@ happened.
 [![PyPI](https://img.shields.io/pypi/v/wiretruth?label=PyPI)](https://pypi.org/project/wiretruth/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/kartsan03/wiretruth/blob/main/LICENSE)
 
-So far wiretruth tests one library, any-llm, on six APIs: OpenAI Chat Completions and Responses, Ollama's native
-and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic Messages. That is 24 scenarios and 95 cells.
+wiretruth tests any-llm, LiteLLM, pydantic-ai, the Vercel AI SDK and TanStack AI across six APIs: OpenAI Chat
+Completions and Responses, Ollama's native and OpenAI-compatible endpoints, Gemini GenerateContent and Anthropic
+Messages. That is 24 scenarios and 100 cells per library. pydantic-ai and the Vercel AI SDK support only the OpenAI-compatible Ollama
+endpoint; their native Ollama cells are UNSUPPORTED.
 The Anthropic responses are transcribed from Anthropic's documentation rather than recorded: each of those fixtures
 is marked `documented` and links the page it comes from. More providers and libraries follow, and a public
 results matrix arrives in v0.4. Until then, the grades are in
@@ -52,11 +54,21 @@ A model hits its output token limit halfway through a JSON object. The provider
 parse error, nothing tells you that a higher limit would fix it. If it returns the partial object as a success,
 your pipeline stores an incomplete record.
 
-wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through any-llm:
+wiretruth has this case recorded as `truncation.structured-output` on `openai-chat`. Replayed through each library:
 
 | Library | What the caller gets | Grade |
 |---|---|---|
-| any-llm 1.33.0 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| any-llm 1.40.2 | Raises `openai.LengthFinishReasonError`. Its `completion` has `finish_reason: "length"` and the token usage. | ✓ PASS |
+| LiteLLM 1.105.5 | Returns the cut-off JSON as text in `message.content` with `finish_reason: "length"` and parses none of it. | ✓ PASS |
+| pydantic-ai 2.61.0 | Retries once, then raises `UnexpectedModelBehavior("Exceeded maximum output retries (1)")`. The `length` finish is only in the messages that `capture_run_messages()` keeps. | ~ LOSSY |
+| Vercel AI SDK 7.0.153 | Raises `NoObjectGeneratedError` with `finishReason: "length"` and the token usage. | ✓ PASS |
+
+pydantic-ai's grades for structured output use `NativeOutput`, the provider's own JSON mode, because that is how
+the fixtures were recorded. It is opt-in: by default pydantic-ai asks for a forced tool call instead on every API
+here except Anthropic, and no scenario covers that path yet. On `gpt-6-luna` that default request is rejected with the same 400
+that `request.tools-with-reasoning-chat` replays (function tools without `reasoning_effort: "none"`). The Vercel AI
+SDK sends a JSON schema by default on every API here except OpenAI-compatible Ollama, where it defaults to JSON-object
+mode; its shim opts into the documented `supportsStructuredOutputs` setting there.
 
 wiretruth runs the same recorded edge cases through every library it supports and grades each one against what the
 provider sent.
@@ -93,6 +105,7 @@ Each scenario lists its requirements. MUST requirements set the grade. SHOULD re
 partial text in the truncation scenarios, add a note when they aren't met. How grades are computed:
 [docs/grading.md](https://github.com/kartsan03/wiretruth/blob/main/docs/grading.md).
 Scenario catalog: [docs/scenarios.md](https://github.com/kartsan03/wiretruth/blob/main/docs/scenarios.md).
+Fixed-bug validation: [docs/validation.md](https://github.com/kartsan03/wiretruth/blob/main/docs/validation.md).
 
 ## Quickstart
 
@@ -101,12 +114,21 @@ You need Python 3.11 or newer and [uv](https://docs.astral.sh/uv/), which builds
 ```bash
 git clone https://github.com/kartsan03/wiretruth && cd wiretruth
 uv run wiretruth run --lib any-llm
+uv run wiretruth run --lib litellm
+uv run wiretruth run --lib pydantic-ai
+uv run wiretruth run --lib vercel-ai
+uv run wiretruth run --lib tanstack-ai
 ```
 
+The Vercel AI SDK and TanStack AI shims also need Node.js and npm. The runner builds their pinned dependencies
+automatically; see [Shims](shims/README.md) for supported providers and structured-output modes.
+
+The any-llm run reports (see [`results/any-llm/1.40.2.json`](https://github.com/kartsan03/wiretruth/blob/main/results/any-llm/1.40.2.json)):
+
 ```text
 scenario                                  anthropic      gemini   ollama   ollama-openai  openai-chat  openai-responses
 finish.stop-sequence                      ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS
-finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✗ WRONG        ✗ WRONG      ✓ PASS
+finish.unknown-value                      ✗ WRONG        ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 request.max-tokens-field                                                                  ✓ PASS
 request.max-tokens-ollama-openai                                           ✗ WRONG
 request.no-injected-options                                       ✓ PASS
@@ -116,8 +138,8 @@ stream.missing-finish                     ~ LOSSY        ~ LOSSY  ~ LOSSY  ~ LOS
 stream.missing-finish-abort               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8                               ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 stream.utf8-split                         ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls                      ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
-tools.parallel-calls-stream               ✓ PASS         ✗ WRONG  ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls                      ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
+tools.parallel-calls-stream               ✓ PASS         ✓ PASS   ✗ WRONG  ✓ PASS         ✓ PASS       ✓ PASS
 truncation.reasoning-only                 ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ✓ PASS
 truncation.structured-output              ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS       ~ LOSSY
 truncation.structured-output-forced-tool  – UNSUPPORTED
@@ -126,8 +148,8 @@ truncation.text-stream                    ✓ PASS         ✓ PASS   ✓ PASS
 truncation.text-stream-usage                             ✓ PASS                           ✓ PASS
 truncation.tool-call-args                 ✓ PASS                                          ✓ PASS       ✓ PASS
 truncation.tool-call-args-stream          ✓ PASS                                          ✓ PASS       ✓ PASS
-usage.cached-tokens                       ✓ PASS
-usage.reasoning-tokens                                   ✓ PASS
+usage.cached-tokens                       ✓ PASS                                          ✓ PASS       ✓ PASS
+usage.reasoning-tokens                    ✓ PASS         ✓ PASS                           ✓ PASS       ✓ PASS
 usage.stream-final-chunk                  ✓ PASS         ✓ PASS   ✓ PASS   ✓ PASS         ✓ PASS
 
 finish.stop-sequence gemini
@@ -146,18 +168,6 @@ finish.unknown-value ollama
   MUST finish-is-other: CONTRADICTED: finish.value="stop" (primary: choices[0].finish_reason); truth.finish="other"; required finish="other"
   SHOULD raw-finish-reported: CONTRADICTED: finish.raw="stop"; truth.raw_finish="wiretruth_unknown_reason"
 
-finish.unknown-value ollama-openai
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Octopuses, like many marine invertebrates, employ sophisticated methods to camouflage"
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
-finish.unknown-value openai-chat
-  MUST response-returned: CONTRADICTED: outcome=raised; error.category="validation"; truth.error=null
-  MUST finish-is-other: NOT_OBSERVABLE: finish.value=null (absent); truth.finish="other"; required finish="other"
-  MUST text-reported: NOT_OBSERVABLE: text.value=null (absent); truth.text="Io, Europa, and Ganymede."
-  SHOULD raw-finish-reported: NOT_OBSERVABLE: finish.raw=null; truth.raw_finish="wiretruth_unknown_reason"
-
 request.max-tokens-ollama-openai ollama-openai
   MUST limit-field: CONTRADICTED: request[0].json_body.max_tokens=absent; required=16
 
@@ -190,18 +200,16 @@ stream.utf8-split gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-5201471136765337120_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_120847", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-5201471136765337120_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_120848", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="3972c760-858b-4728-a05a-279fc6e6a649", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="4d3747a6-5e75-4637-b82e-f269adbba01e", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="77ecdd8a-566d-4ba1-b098-d7a3b3082999", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_9epwsqu6", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="3a4cba16-a57a-4657-b28a-8ac89eaea7d9", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_bvzy1ra9", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 tools.parallel-calls-stream gemini
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="call_-1615951408272134819_0", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_89893", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="call_-1615951408272134819_1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_89896", name="get_weather", args={"city": "Zürich"}, parsed=true
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
 tools.parallel-calls-stream ollama
-  MUST calls-reported: CONTRADICTED: tool_calls[0] id="8b75c529-6eb6-4a63-a375-4551bfe56f39", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="8e864d64-6145-4ca4-81ba-cbdbad0523e1", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
+  MUST calls-reported: CONTRADICTED: tool_calls[0] id="fc700b93-a9ac-4907-8da1-a7b38a28658c", name="get_weather", args={"city": "São Paulo"}, parsed=true; truth id="call_8uaqob2r", name="get_weather", args={"city": "São Paulo"}, parsed=true; tool_calls[1] id="265e0a1f-ef64-4209-b6b9-e3b5c9e3a181", name="get_weather", args={"city": "Zürich"}, parsed=true; truth id="call_nyjtbrz4", name="get_weather", args={"city": "Zürich"}, parsed=true
 
 truncation.structured-output gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -212,7 +220,7 @@ truncation.structured-output openai-responses
   SHOULD usage-reported: NOT_OBSERVABLE: usage.input=null (absent); truth.usage.input=68; usage.output=null (absent); truth.usage.output=24
 
 truncation.structured-output-forced-tool anthropic
-  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.33.0/src/any_llm/providers/anthropic/utils.py#L628-L655)
+  any-llm completion maps typed response_format to native output_config.format and has no structured parser for an explicitly forced tool. (https://github.com/mozilla-ai/any-llm/blob/1.40.2/src/any_llm/providers/anthropic/utils.py#L660-L686)
 
 truncation.text gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
@@ -223,27 +231,45 @@ truncation.text-stream gemini
 truncation.text-stream-usage gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
+usage.reasoning-tokens anthropic
+  SHOULD reasoning-reported: NOT_OBSERVABLE: usage.reasoning=null (absent); truth.usage.reasoning=312
+
 usage.stream-final-chunk gemini
   shim: Gemini reasoning_effort none is unsupported by any-llm; omitted to use the model default.
 
-any-llm 1.33.0: 95 cells, 78 PASS, 6 LOSSY, 10 WRONG, 1 UNSUPPORTED
-stderr: .wiretruth/runs/20261013T134918.993664Z
+any-llm 1.40.2: 100 cells, 87 PASS, 6 LOSSY, 6 WRONG, 1 UNSUPPORTED
+stderr: .wiretruth/runs/20261109T005301.682490Z
 ```
 
-No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads any-llm
-into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference` runs the
-built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this repository,
-so wiretruth runs from a clone.
+The LiteLLM run grades its 100 cells 86 PASS, 13 WRONG and 1 UNSUPPORTED. Every cell, with its evidence, is in
+[`results/litellm/1.105.5.json`](https://github.com/kartsan03/wiretruth/blob/main/results/litellm/1.105.5.json).
+The pydantic-ai run grades its 100 cells 63 PASS, 16 LOSSY, 6 WRONG and 15 UNSUPPORTED (native Ollama), in
+[`results/pydantic-ai/2.61.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/pydantic-ai/2.61.0.json).
+The Vercel AI SDK run grades its 100 cells 84 PASS and 16 UNSUPPORTED (native Ollama, and the Anthropic
+forced-tool fixture, since current Claude models get native structured output), in
+[`results/vercel-ai/7.0.153.json`](https://github.com/kartsan03/wiretruth/blob/main/results/vercel-ai/7.0.153.json).
+The TanStack AI run grades its 100 cells 48 PASS, 2 LOSSY, 2 WRONG and 48 UNSUPPORTED, in
+[`results/tanstack-ai/1.0.0.json`](https://github.com/kartsan03/wiretruth/blob/main/results/tanstack-ai/1.0.0.json).
+Its `chat()` always requests a streamed response, so the non-streamed fixtures don't apply to it.
+
+No API keys are needed. Everything runs against stored responses on `127.0.0.1`. The first run downloads the
+selected library into `.wiretruth/envs/`; later runs work offline. `uv run wiretruth run --shim shims/reference`
+runs the built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this
+repository, so wiretruth runs from a clone.
 
 Releases are also on [PyPI](https://pypi.org/project/wiretruth/) (`pip install wiretruth`). The package is the
-command only, so run it inside a checkout of the matching release tag (for example `v0.2.0`); on `main`, use
+command only, so run it inside a checkout of the matching release tag (for example `v0.3.0`); on `main`, use
 `uv run` as above.
 
 ## Add your library
 
 A shim is a small program. It reads a call spec as JSON on stdin, calls your library the way its docs show, and
 prints what the library reported as JSON on stdout. The
-[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py) is 150 lines. The
+[any-llm shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/any-llm/shim.py),
+[LiteLLM shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/litellm/shim.py),
+[pydantic-ai shim](https://github.com/kartsan03/wiretruth/blob/main/shims/python/pydantic-ai/shim.py),
+[Vercel AI SDK shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/vercel-ai/shim.mjs) and
+[TanStack AI shim](https://github.com/kartsan03/wiretruth/blob/main/shims/js/tanstack-ai/shim.mjs) are examples. The
 protocol is in [shims/README.md](https://github.com/kartsan03/wiretruth/blob/main/shims/README.md); to propose a
 library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issues/new?template=new-library.yml).
 
@@ -267,7 +293,8 @@ library, [open a new-library issue](https://github.com/kartsan03/wiretruth/issue
 ## Author note
 
 I've contributed fixes to libraries that wiretruth tests or will test:
-[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389) and
+[mozilla-ai/any-llm#1389](https://github.com/mozilla-ai/any-llm/pull/1389),
+[mozilla-ai/any-llm#1433](https://github.com/mozilla-ai/any-llm/pull/1433) and
 [TanStack/ai#1548](https://github.com/TanStack/ai/pull/1548). Grades are computed by the code in this repository.
 If you think a grade is wrong, please
 [open a grade dispute](https://github.com/kartsan03/wiretruth/issues/new?template=grade-dispute.yml).