spantruth 0.3.0__tar.gz

spantruth-0.3.0/.context/CONTEXT_llm_telemetry_testing_landscape.md

@@ -0,0 +1,117 @@
+# The landscape around testing LLM telemetry values
+
+Researched on 2026-11-03. Every number has its source and date. Anything marked "unverified" was not
+confirmed at the source. Re-check a number before you quote it in public.
+
+## Prior art
+
+Nothing found on 2026-11-03 does all of these together: a fake provider, the real provider SDK, the
+production instrumentation packages, an OTLP receiver, and values compared with what the provider sent,
+including failure paths.
+
+| Project | What it checks | Difference from a value check |
+| --- | --- | --- |
+| [semantic-conventions-genai `reference/`](https://github.com/open-telemetry/semantic-conventions-genai/tree/main/reference) | Hand-written reference instrumentations (built on `opentelemetry-util-genai`) of about 40 client SDKs and frameworks, run against a deterministic mock server. The [conformance runner](https://github.com/open-telemetry/semantic-conventions-conformance) validates the captured telemetry with Weaver and generates a matrix of supported attributes. | The instrumentation is a reference, not the package users install. The checks are names, types and required attributes, not equality with the provider's numbers. |
+| [Weaver live-check](https://github.com/open-telemetry/weaver/blob/main/docs/usage.md) | Received OTLP against a semantic-convention registry, with OPA policies. `--fail-on` sets the exit level. `--diagnostic-format gh_workflow_command` writes workflow annotations. | Conformance to the registry, not to the wire. |
+| [pytest-genai-semconv](https://github.com/Krishnachaitanyakc/pytest-genai-semconv) (0.1.0, 2026-07-06) | A pytest plugin with in-memory span capture and `assert_chat_span(...)` helpers: presence, types, span kind. | Tests your own spans. No fake provider, no real SDK, no failure paths. |
+| [kartsan03/wiretruth](https://github.com/kartsan03/wiretruth) | What the provider SDK returns to the caller, from recorded responses. Owns the `wiretruth` name on PyPI (0.1.0 on 2026-10-11, 0.2.0 on 2026-10-13). | Checks the SDK, not the span. Its fixture format is replayable. |
+| [4ktLuffy/spanproof](https://github.com/4ktLuffy/spanproof) | Sentry SDK integrations, with fixtures validated by the provider SDK's types. | One vendor. |
+| [arthi-arumugam-git/wrong-numbers](https://github.com/arthi-arumugam-git/wrong-numbers) | A field guide to wrong numbers in eval, tracing and cost libraries. | Documentation, not a harness. |
+| [jkelly-dev1/llm-observability-stack](https://github.com/jkelly-dev1/llm-observability-stack) | A personal study. Cost computed from spans was 44% below the provider's usage, because the span copied an `input_tokens` field that excludes the cached prefix. | Not a tool. Evidence for the cache-token class. |
+
+The reference scenarios name their metrics `gen_ai.client.inference.usage.input_tokens` (a counter) and
+`gen_ai.client.inference.operation.input_tokens` (a histogram), with matching output, cache and
+reasoning names. Source: `reference/scenarios/anthropic/scenario.py` on the `main` branch, 2026-11-03.
+
+## The semantic conventions
+
+- The GenAI conventions moved to their own repository,
+  [semantic-conventions-genai](https://github.com/open-telemetry/semantic-conventions-genai), in
+  semconv v1.42. Every GenAI attribute in its registry has `stability: development`.
+- Token attributes: `gen_ai.usage.input_tokens` (SHOULD include cached tokens),
+  `gen_ai.usage.output_tokens` (SHOULD include reasoning tokens), `gen_ai.usage.cache_read.input_tokens`,
+  `gen_ai.usage.cache_write.input_tokens` and `gen_ai.usage.reasoning.output_tokens`.
+- The cache-write attribute was `cache_creation.input_tokens` first. The rename is live, and
+  instrumentations lag: [pydantic-ai#11779](https://github.com/pydantic/pydantic-ai/issues/11779)
+  (2026-10-28). The date of the rename is unverified.
+- Since v1.37.0 (2025-08-25), content goes in `gen_ai.input.messages`, `gen_ai.output.messages` and
+  `gen_ai.system_instructions`, or in the `gen_ai.client.inference.operation.details` event.
+
+## Who would run a value check
+
+pypistats "last month" and npm downloads, read on 2026-11-03. Download counts mix SDK use with
+instrumentation use, so they rank reach, not need.
+
+| Library | Downloads a month | Emits OTel GenAI spans |
+| --- | --- | --- |
+| LiteLLM | 92.2M | Through its OTel callback |
+| Langfuse SDK | 23.6M | OTel-based SDK |
+| MLflow | 19.9M | OTel export |
+| Pydantic Logfire | 13.0M | Yes |
+| opentelemetry-util-genai (official) | 4.91M | Yes |
+| OpenLLMetry anthropic and openai | 3.02M and 2.86M | Yes |
+| OpenInference openai and anthropic | 2.25M and 0.78M | Its own `llm.*` names |
+| OpenLIT | 278k | Yes |
+| Vercel AI SDK (`ai`, npm) | 118.4M | `experimental_telemetry`, mixed names (unverified) |
+
+Provider SDKs: openai 295.9M, anthropic 150.2M, google-genai 137.6M, groq 16.8M and mistralai 16.2M a
+month. On npm: `@anthropic-ai/sdk` 181.4M, `openai` 162.0M and `@google/genai` 91.6M.
+
+## Demand: the bug classes repeat
+
+Issues and pull requests created between 2026-05-03 and 2026-11-03, with their state on 2026-11-03.
+Four classes repeat across eight independent projects: cache tokens dropped or counted twice, reasoning
+tokens dropped, spans that never end, and one call reported on two spans.
+
+- Cache tokens: [openllmetry#4665](https://github.com/traceloop/openllmetry/issues/4665),
+  [openllmetry#4583](https://github.com/traceloop/openllmetry/issues/4583) (`prompt_tokens_details` is
+  None), [mlflow#26907](https://github.com/mlflow/mlflow/issues/26907) (final counts in `message_delta`
+  ignored), [mlflow#26817](https://github.com/mlflow/mlflow/issues/26817),
+  [litellm#49558](https://github.com/BerriAI/litellm/pull/49558),
+  [langfuse#19483](https://github.com/langfuse/langfuse/pull/19483) and
+  [pydantic-ai#11779](https://github.com/pydantic/pydantic-ai/issues/11779).
+- Reasoning tokens: [openllmetry#4660](https://github.com/traceloop/openllmetry/issues/4660) (streamed
+  chat), [litellm#49557](https://github.com/BerriAI/litellm/pull/49557) and
+  [python-contrib#4987](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/4987).
+- Spans that never end: [openllmetry#4608](https://github.com/traceloop/openllmetry/issues/4608),
+  [openllmetry#4597](https://github.com/traceloop/openllmetry/issues/4597),
+  [openllmetry#4650](https://github.com/traceloop/openllmetry/pull/4650) and
+  [openinference#4215](https://github.com/Arize-ai/openinference/pull/4215).
+- One call on two spans: [openllmetry#4609](https://github.com/traceloop/openllmetry/issues/4609).
+- Calls through other entry points record nothing:
+  [logfire#2549](https://github.com/pydantic/logfire/issues/2549) (`with_raw_response` and
+  `with_streaming_response`).
+- Responses API paths: [openinference#4185](https://github.com/Arize-ai/openinference/issues/4185) and
+  [logfire#2565](https://github.com/pydantic/logfire/pull/2565).
+
+## How conformance suites became the default
+
+From primary sources, read on 2026-11-03.
+
+- **[Bowtie](https://github.com/bowtie-json-schema/bowtie)** is the closest model. Each JSON Schema
+  implementation runs in a container with a small harness that speaks a JSON-lines protocol on stdin and
+  stdout. Harnesses live in the implementers' directories. Bowtie has a report site, shields.io endpoint
+  badges, a composite setup action that installs it with uv, and `bowtie summary --format markdown` for
+  the job summary. A skipped test carries an issue URL. Its author wrote most early harnesses himself.
+  It launched with 12 implementations and now has 24.
+- **test262** and **web-platform-tests** keep known failures in the implementer's repository and
+  publish results per implementation (test262.fyi, wpt.fyi).
+- **Kubernetes conformance** gives a trademark to certified products, and the trademark drives
+  adoption.
+- **Protocol Buffers conformance** uses a small "testee" per language over stdin and stdout, the same
+  idea as the Bowtie harness.
+
+Patterns that drove adoption: a tiny harness as the unit of contribution, the harness in the
+implementer's repository, a public result and a badge per implementation, known failures recorded with
+an issue link, a one-step CI path, a neutral home, and a funded maintainer who writes the first
+harnesses.
+
+Weaknesses seen: a frozen toolchain (Autobahn is pinned to Python 2), no stable test identity (Bowtie
+matches skips by description), governance that never became formal, and no documented dispute path.
+
+Launch posts alone rarely drove adoption: the promptfoo Show HN scored 14 points. Integrations in the
+places users already work mattered more.
+
+## Related Context
+
+None yet.

spantruth-0.3.0/.editorconfig

@@ -0,0 +1,12 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+indent_style = space
+indent_size = 4
+
+[*.{yml,yaml,json,toml,md,html,js,css}]
+indent_size = 2

spantruth-0.3.0/.github/CODEOWNERS

@@ -0,0 +1 @@
+* @chrikrah

spantruth-0.3.0/.github/ISSUE_TEMPLATE/bug.yml

@@ -0,0 +1,24 @@
+name: Bug in spantruth
+description: The harness itself misbehaves, for example a cell is inconclusive or a run fails.
+labels: ["bug"]
+body:
+  - type: textarea
+    id: what
+    attributes:
+      label: What happened
+      description: The command you ran, and its output.
+      render: shell
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: spantruth version or commit
+    validations:
+      required: true
+  - type: input
+    id: env
+    attributes:
+      label: Operating system and Python version
+    validations:
+      required: true

spantruth-0.3.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Question or idea
+    url: https://github.com/Plexito-de/spantruth/discussions
+    about: Ask a question or float an idea in Discussions.
+  - name: Security problem
+    url: https://github.com/Plexito-de/spantruth/security/advisories/new
+    about: Report a vulnerability privately. Do not open a public issue.

spantruth-0.3.0/.github/ISSUE_TEMPLATE/dispute.yml

@@ -0,0 +1,30 @@
+name: Dispute a finding
+description: A cell on the report is red or yellow and you think the check is wrong.
+labels: ["dispute"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thank you. Maintainers of the library under test are especially welcome here. We decide
+        every dispute on one question: does the cited rule say what the check claims?
+  - type: input
+    id: cell
+    attributes:
+      label: Subject and scenario
+      placeholder: openllmetry, anthropic/stream
+    validations:
+      required: true
+  - type: input
+    id: check
+    attributes:
+      label: Check
+      placeholder: output_tokens
+    validations:
+      required: true
+  - type: textarea
+    id: why
+    attributes:
+      label: Why the finding is wrong
+      description: Quote the spec, the provider documentation, or your library's documented behavior.
+    validations:
+      required: true

spantruth-0.3.0/.github/ISSUE_TEMPLATE/subject.yml

@@ -0,0 +1,24 @@
+name: Add a subject, provider or scenario
+description: Propose a library, a provider API, or a failure mode to cover.
+labels: ["enhancement"]
+body:
+  - type: dropdown
+    id: kind
+    attributes:
+      label: What to add
+      options: [A subject (instrumentation library), A provider, A scenario]
+    validations:
+      required: true
+  - type: textarea
+    id: what
+    attributes:
+      label: Details
+      description: Name, link, and for a scenario the provider behavior it reproduces, with a link to the provider documentation.
+    validations:
+      required: true
+  - type: checkboxes
+    id: help
+    attributes:
+      label: Contribution
+      options:
+        - label: I would like to open the pull request myself.

spantruth-0.3.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,15 @@
+## What this changes
+
+<!-- One or two sentences. Link the issue it closes. -->
+
+## How it was tested
+
+<!-- The commands you ran and their output. -->
+
+## Checklist
+
+- [ ] The title is a Conventional Commit, for example `feat(subjects): add logfire`.
+- [ ] A test covers the change, and it fails when the change is reverted.
+- [ ] `control-honest` passes, and `control-faulty` fails exactly its expected checks.
+- [ ] Every new red cell is explained: the line of code that causes it, or a note that I could not find it.
+- [ ] `CHANGELOG.md` is updated under Unreleased.

spantruth-0.3.0/.github/dependabot.yml

@@ -0,0 +1,20 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+    cooldown:
+      default-days: 14
+    groups:
+      actions:
+        patterns: ["*"]
+  - package-ecosystem: uv
+    directory: /
+    schedule:
+      interval: weekly
+    cooldown:
+      default-days: 14
+    groups:
+      python:
+        patterns: ["*"]

spantruth-0.3.0/.github/workflows/ci.yml

@@ -0,0 +1,106 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions: {}
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv sync --locked
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+
+  test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    strategy:
+      matrix:
+        python: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv sync --locked --python ${{ matrix.python }}
+      - run: uv run pytest -m "not slow"
+
+  controls:
+    # The falsification of the whole suite: both controls, end to end, in real subprocesses.
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv sync --locked
+      - run: uv run pytest -m slow
+
+  zizmor:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      security-events: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: zizmorcore/zizmor-action@85b2b9fb8e3325e7a23d7d0a9d7ba5ae0f999e3e # v0.7.1
+
+  action:
+    # The action as a user runs it. The last run compares with an empty expectation file, so it
+    # must fail with exit code 1: a gate that cannot fail has measured nothing. `uses: $/` is
+    # GitHub's self-repository syntax (July 2026): this repository at the workflow's own commit,
+    # independent of what a step left in the workspace. It is not a typo for `./`.
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: $/
+        with:
+          subject: control-honest
+          scenario: |
+            openai/basic
+            anthropic/stream-early-exit
+      - uses: $/
+        with:
+          subject: control-faulty
+          expect: subjects/control-faulty/expected.toml
+          scenario: |
+            openai/stream-no-usage
+            anthropic/cache-hit
+            anthropic/stream-early-exit
+      - run: touch "$RUNNER_TEMP/empty.toml"
+      - id: differs
+        uses: $/
+        continue-on-error: true
+        with:
+          subject: control-faulty
+          expect: ${{ runner.temp }}/empty.toml
+          scenario: openai/stream-no-usage
+          artifact-name: spantruth-control-faulty-differs
+      - name: The run against an empty expectation file failed with exit code 1
+        env:
+          OUTCOME: ${{ steps.differs.outcome }}
+          CODE: ${{ steps.differs.outputs.exit-code }}
+        run: test "$OUTCOME" = failure && test "$CODE" = 1

spantruth-0.3.0/.github/workflows/cla.yml

@@ -0,0 +1,26 @@
+name: CLA
+
+# pull_request_target is required so the check can comment on pull requests from forks.
+# The reusable workflow never checks out or runs code from the pull request.
+on:
+  pull_request_target: # zizmor: ignore[dangerous-triggers]
+    types: [opened, synchronize, reopened]
+  issue_comment:
+    types: [created]
+
+permissions: {}
+
+jobs:
+  cla:
+    if: >-
+      github.event_name == 'pull_request_target' ||
+      (github.event.issue.pull_request &&
+       (contains(github.event.comment.body, 'I have read the CLA Document') ||
+        contains(github.event.comment.body, 'recheck')))
+    permissions:
+      contents: write
+      pull-requests: write
+      statuses: write
+    uses: Plexito-de/.github/.github/workflows/cla.yml@main # zizmor: ignore[unpinned-uses] our own org's workflow
+    with:
+      allowlist: "chrikrah,dependabot[bot],github-actions[bot]"

spantruth-0.3.0/.github/workflows/release.yml

@@ -0,0 +1,53 @@
+name: Release
+
+# A release is a tag with notes and a wheel on PyPI. The wheel holds the harness, the CLI and the
+# subjects, so `uvx spantruth` runs without a clone.
+on:
+  push:
+    tags: ["v*"]
+
+permissions: {}
+
+jobs:
+  check:
+    # The tag, the package version and the wheel name must agree, or the action ref, the wheel and
+    # the results document would each claim a different version.
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv build
+      - name: The tag matches the package version
+        env:
+          TAG: ${{ github.ref_name }}
+        run: test -f "dist/spantruth-${TAG#v}-py3-none-any.whl"
+
+  github-release:
+    needs: check
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: softprops/action-gh-release@efb35369e0ad2afab669f228072c1b0d510eae64 # v3.0.3
+        with:
+          generate_release_notes: true
+
+  pypi:
+    needs: check
+    runs-on: ubuntu-latest
+    # Trusted publishing: PyPI trusts this workflow and this environment, so no token is stored.
+    environment: pypi
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3

spantruth-0.3.0/.github/workflows/report.yml

@@ -0,0 +1,61 @@
+name: Report
+
+# Runs every subject at its pinned versions and at its latest release, and publishes the
+# matrix to GitHub Pages.
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions: {}
+
+concurrency:
+  group: report
+  cancel-in-progress: false
+
+jobs:
+  run:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
+      - run: uv sync --locked
+      # Exit code 3 means the harness could not run a cell or build an environment, for example
+      # when a new upstream release fails to install. Publish the report anyway, so the gap shows
+      # on the page, and flag the run.
+      - name: Pinned versions
+        run: |
+          rc=0
+          uv run spantruth run --out site/pinned.json || rc=$?
+          if [ "$rc" -eq 3 ]; then echo "::warning::incomplete pinned run"; elif [ "$rc" -ne 0 ]; then exit "$rc"; fi
+      - name: Latest releases
+        run: |
+          rc=0
+          uv run spantruth run --latest --out site/latest.json || rc=$?
+          if [ "$rc" -eq 3 ]; then echo "::warning::incomplete latest run"; elif [ "$rc" -ne 0 ]; then exit "$rc"; fi
+      - name: Render
+        run: |
+          uv run spantruth report site/pinned.json --html site/index.html
+          uv run spantruth report site/latest.json --html site/latest.html --badges site/badges
+      - uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
+        with:
+          path: site
+
+  deploy:
+    needs: run
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deploy.outputs.page_url }}
+    steps:
+      - id: deploy
+        uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1

spantruth-0.3.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.venv/
+.spantruth/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+site/
+results/*
+!results/2026-11-03-pinned.json