spanlight 0.1.2__tar.gz → 0.2.1__tar.gz

{spanlight-0.1.2 → spanlight-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: spanlight
-Version: 0.1.2
+Version: 0.2.1
 Summary: Python SDK for Spanlight, open-source observability for LLM applications.
 Author: Dheeraj
 License-Expression: MIT
@@ -124,7 +124,8 @@ def answer(question: str) -> str:
 
 `wrap_openai` instruments `chat.completions.create` and `responses.create` on
 the client it is given. It works with `OpenAI`, `AsyncOpenAI` and the Azure
-variants. The SDK's own `.stream()` helpers are traced as well, because they
+variants; an Azure client is recorded with the provider `azure_openai`, which the
+server does not price at OpenAI's rates (add a price override for it). The SDK's own `.stream()` helpers are traced as well, because they
 call `create` internally.
 
 Streaming works without any changes to your code. Usage is recorded when
@@ -145,6 +146,56 @@ The span ends when the stream is exhausted or closed, and records
 time-to-first-token. If you point the OpenAI SDK at a compatible API, set the
 provider name explicitly: `wrap_openai(client, provider="openrouter")`.
 
+When a stream carries no usage, the span's output token count is left unknown
+(never guessed) and the `usage.unreported_reason` attribute says why:
+`include_usage_not_set` when a Chat Completions stream ended without a usage
+chunk, `no_final_usage` when a Responses stream ended without its terminal
+event, `client_disconnected` when your code closed it early, and
+`stream_error` when it failed. Such a call's cost shows as unknown.
+
+### Service tier and web search
+
+- The `service_tier` OpenAI answers with (on the response, or on the last
+  stream chunk or terminal event) is recorded as `standard`, `flex`, `priority`
+  or `batch`; `default`, `auto` and `scale` are `standard`. A value the SDK does
+  not know is kept in the `usage.service_tier_raw` attribute and the call is
+  priced at the standard rate.
+- For the Responses API, every completed `web_search_call` output item counts
+  as one billed web search; searches still in progress or failed are not
+  counted.
+- Chat Completions with `web_search_options` report sources as message
+  annotations, which are not a count of searches. No count is recorded and the
+  span gets `usage.fee_unknown = "openai_chat_web_search"`, so the search fee
+  shows as unknown rather than zero.
+- Requests that set `inference_geo` (other than `global`) or `speed: "fast"`
+  are priced at the standard rate and tagged with
+  `usage.pricing_modifier_unknown`, because those modifiers are not modelled.
+
+### Batch output
+
+OpenAI Batch results are a file you download, so record them explicitly once
+the batch has completed:
+
+```python
+batch = client.batches.retrieve("batch_67ccd2bed1ec8190b14f964abc054267")
+content = client.files.content(batch.output_file_id)
+recorded = spanlight.record_openai_batch_output(content.iter_lines(), batch_id=batch.id)
+```
+
+Pass the error file (`batch.error_file_id`) the same way to record failed
+requests. Each line becomes one `llm` span with the shape described under
+[Anthropic batch results](#batch-results): `service_tier` is `batch` (the tier
+the body reported is kept in `usage.service_tier_raw`), cached tokens come from
+the usage details, and a line whose `status_code` is not 200 or that carries an
+`error` becomes an error span without usage. A Responses body with
+`status: "failed"` becomes an error span that keeps the usage OpenAI reports
+(it is billed), and `status: "incomplete"` a normal span whose finish reason is
+`incomplete_details.reason`. Chat Completions and Responses
+bodies are both read. The function accepts any iterable of `str` or `bytes`
+lines (or the whole file as one string), returns the number of spans recorded,
+and never raises: unreadable lines are skipped. Use `provider=` for a
+compatible API and `spanlight=` for a client other than the global one.
+
 ## Anthropic
 
 ```python
@@ -173,6 +224,33 @@ with client.messages.stream(
 `wrap_anthropic` instruments `messages.create` (including `stream=True`) and
 `messages.stream` on both `Anthropic` and `AsyncAnthropic`.
 
+### Batch results
+
+Results read through `messages.batches.results` are recorded as you iterate
+them, on both `Anthropic` and `AsyncAnthropic`:
+
+```python
+for item in client.messages.batches.results("msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d"):
+    handle(item)  # each item is yielded unchanged
+```
+
+Each result becomes one `llm` span named `batch <model>`, in a trace named
+`batch <batch_id>` and tagged `spanlight:batch`:
+
+- A `succeeded` result carries the model, token usage, finish reason and the
+  response content, with `service_tier` set to `batch` (the Batches API bills
+  every request at the batch rate). The tier the response reported is kept in
+  the `usage.service_tier_raw` attribute.
+- `errored`, `canceled` and `expired` results are error spans without usage,
+  because they are not billed.
+- The request is not part of a result, so the span has no input
+  (`batch.request_available` is `false`). Its start and end are both the
+  moment the result was read (`timing.synthetic` is `true`).
+- The attributes `batch.id`, `batch.custom_id` and `batch.result_type` link
+  the span back to the batch.
+- Trace and span ids are derived from the batch id and `custom_id`, so reading
+  the same batch again updates the same spans instead of adding new ones.
+
 ## Prompts
 
 Keep prompts in Spanlight and fetch them by label, so a new version ships
@@ -290,6 +368,16 @@ host must be `https://` unless it is localhost. Each
 run makes real provider calls. In a GitHub workflow use the
 `.github/actions/eval-gate` Action instead of calling the module yourself.
 
+The run prints a Markdown report (also appended to the GitHub step summary): the
+run's items, errors, cost and latency, then one row per score with its mean, the
+95% confidence interval on the pass rate of a boolean score (for example
+`64–88%`; `—` for numeric and categorical scores, and from servers that predate
+intervals), the number of items, the threshold and the result. When a threshold
+lies inside a boolean score's interval, the report adds "Interval includes the
+threshold; the result is not conclusive at this sample size." The gate itself
+still compares the mean with the threshold, so the exit codes do not depend on
+the interval.
+
 ## What gets recorded
 
 For every LLM call:
@@ -308,6 +396,22 @@ For every LLM call:
 | `output`                      | The assistant message, including tool calls                                                                                                                               |
 | `status` / `status_message`   | `error` plus the provider's message when the call fails                                                                                                                   |
 
+When a provider call fails with an HTTP error, the span also carries the
+provider's error type and code (`provider.error.type`, `provider.error.code`),
+the wait it asked for in seconds (`provider.retry_after_s`, from `retry-after-ms`
+or a numeric `retry-after`), and `provider.ratelimit`: the response's rate-limit
+headers, filtered through a fixed allowlist of 29 names. Only `retry-after`,
+`retry-after-ms`, OpenAI's `x-ratelimit-*` headers and Anthropic's
+`anthropic-ratelimit-*` and `anthropic-priority-*` headers are kept, each value
+at most 64 printable ASCII characters. Every other response header, including
+`authorization`, `x-api-key`, `cookie`, `set-cookie` and Anthropic's
+organization and workspace ids, is dropped before the span leaves your process.
+Connection errors and timeouts add none of these fields.
+When the Spanlight gateway refused the call itself (its `X-Spanlight-Code`
+header names one of its own codes, such as `RATE_LIMITED` for your gateway key's
+limit), the span carries that code as `spanlight.gateway.error_code` instead, and
+none of the `provider.*` fields above.
+
 The Spanlight server also redacts common secret patterns from
 payloads on ingestion. Set `capture_payloads = false` on a project to drop
 inputs and outputs entirely. To keep payloads out of the SDK side for a
@@ -337,7 +441,7 @@ options. Use `tracer.observe`, `tracer.span` and
 | `flush_interval`   |                         | `2.0`                   | Seconds a span may wait before it is sent                                |
 | `batch_size`       |                         | `100`                   | Spans per request (max 1000)                                             |
 | `max_queue`        |                         | `10000`                 | Buffered spans. When the buffer is full, the oldest spans are dropped.   |
-| `gzip`             |                         | `False`                 | Gzip-compress request bodies                                             |
+| `gzip`             |                         | `True`                  | Gzip-compress request bodies                                             |
 | `timeout`          |                         | `10.0`                  | HTTP timeout per request, in seconds                                     |
 | `max_retries`      |                         | `5`                     | Retries for network errors, 429 and 5xx                                  |
 | `shutdown_timeout` |                         | `5.0`                   | Time budget for the flush at interpreter exit                            |
@@ -370,10 +474,14 @@ cached_tokens=, cache_write_tokens=, cache_write_1h_tokens=)`,
 exception that escapes the block marks the span as failed and is re-raised
 unchanged.
 
-### `update_trace(name=None, user_id=None, session_id=None, tags=None)`
+### `update_trace(name=None, user_id=None, session_id=None, tags=None, metadata=None)`
 
 Sets trace-level fields on the trace of the current span. You can call it
-anywhere inside the trace. Tags accumulate across calls.
+anywhere inside the trace. Tags accumulate across calls. Metadata keys
+accumulate too, and a later value replaces a key's earlier one:
+`spanlight.update_trace(metadata={"customer_id": "acme", "feature": "search"})`.
+Up to 32 keys matching `^[a-z0-9_.-]{1,64}$`, values of 1 to 256 characters
+(numbers and booleans become text); an invalid entry is dropped with a debug log.
 
 ### `flush(timeout=None) -> bool` and `shutdown(timeout=None)`
 
@@ -407,6 +515,14 @@ logged to the `spanlight` logger, as are any spans the server rejects. If the
 queue fills up, the oldest spans are dropped and counted
 (`spanlight.get_client().exporter.dropped_count`).
 
+Since 0.2.0, every batch of spans also carries a small `client` object with the
+SDK name and version, a random `exporter_id` (new in each process, including a
+forked child) and two running totals for that exporter: `dropped_total` (spans
+and scores dropped because the queue was full) and `failed_total` (spans and
+scores lost after retries or rejected by the server). The totals are cumulative,
+so a batch delivered twice does not count a loss twice. It holds no span
+content, and older Spanlight servers ignore it.
+
 To see what the SDK is doing:
 
 ```python

{spanlight-0.1.2 → spanlight-0.2.1}/README.md

@@ -97,7 +97,8 @@ def answer(question: str) -> str:
 
 `wrap_openai` instruments `chat.completions.create` and `responses.create` on
 the client it is given. It works with `OpenAI`, `AsyncOpenAI` and the Azure
-variants. The SDK's own `.stream()` helpers are traced as well, because they
+variants; an Azure client is recorded with the provider `azure_openai`, which the
+server does not price at OpenAI's rates (add a price override for it). The SDK's own `.stream()` helpers are traced as well, because they
 call `create` internally.
 
 Streaming works without any changes to your code. Usage is recorded when
@@ -118,6 +119,56 @@ The span ends when the stream is exhausted or closed, and records
 time-to-first-token. If you point the OpenAI SDK at a compatible API, set the
 provider name explicitly: `wrap_openai(client, provider="openrouter")`.
 
+When a stream carries no usage, the span's output token count is left unknown
+(never guessed) and the `usage.unreported_reason` attribute says why:
+`include_usage_not_set` when a Chat Completions stream ended without a usage
+chunk, `no_final_usage` when a Responses stream ended without its terminal
+event, `client_disconnected` when your code closed it early, and
+`stream_error` when it failed. Such a call's cost shows as unknown.
+
+### Service tier and web search
+
+- The `service_tier` OpenAI answers with (on the response, or on the last
+  stream chunk or terminal event) is recorded as `standard`, `flex`, `priority`
+  or `batch`; `default`, `auto` and `scale` are `standard`. A value the SDK does
+  not know is kept in the `usage.service_tier_raw` attribute and the call is
+  priced at the standard rate.
+- For the Responses API, every completed `web_search_call` output item counts
+  as one billed web search; searches still in progress or failed are not
+  counted.
+- Chat Completions with `web_search_options` report sources as message
+  annotations, which are not a count of searches. No count is recorded and the
+  span gets `usage.fee_unknown = "openai_chat_web_search"`, so the search fee
+  shows as unknown rather than zero.
+- Requests that set `inference_geo` (other than `global`) or `speed: "fast"`
+  are priced at the standard rate and tagged with
+  `usage.pricing_modifier_unknown`, because those modifiers are not modelled.
+
+### Batch output
+
+OpenAI Batch results are a file you download, so record them explicitly once
+the batch has completed:
+
+```python
+batch = client.batches.retrieve("batch_67ccd2bed1ec8190b14f964abc054267")
+content = client.files.content(batch.output_file_id)
+recorded = spanlight.record_openai_batch_output(content.iter_lines(), batch_id=batch.id)
+```
+
+Pass the error file (`batch.error_file_id`) the same way to record failed
+requests. Each line becomes one `llm` span with the shape described under
+[Anthropic batch results](#batch-results): `service_tier` is `batch` (the tier
+the body reported is kept in `usage.service_tier_raw`), cached tokens come from
+the usage details, and a line whose `status_code` is not 200 or that carries an
+`error` becomes an error span without usage. A Responses body with
+`status: "failed"` becomes an error span that keeps the usage OpenAI reports
+(it is billed), and `status: "incomplete"` a normal span whose finish reason is
+`incomplete_details.reason`. Chat Completions and Responses
+bodies are both read. The function accepts any iterable of `str` or `bytes`
+lines (or the whole file as one string), returns the number of spans recorded,
+and never raises: unreadable lines are skipped. Use `provider=` for a
+compatible API and `spanlight=` for a client other than the global one.
+
 ## Anthropic
 
 ```python
@@ -146,6 +197,33 @@ with client.messages.stream(
 `wrap_anthropic` instruments `messages.create` (including `stream=True`) and
 `messages.stream` on both `Anthropic` and `AsyncAnthropic`.
 
+### Batch results
+
+Results read through `messages.batches.results` are recorded as you iterate
+them, on both `Anthropic` and `AsyncAnthropic`:
+
+```python
+for item in client.messages.batches.results("msgbatch_01HkcTjaV5uDC8jWR4ZsDV8d"):
+    handle(item)  # each item is yielded unchanged
+```
+
+Each result becomes one `llm` span named `batch <model>`, in a trace named
+`batch <batch_id>` and tagged `spanlight:batch`:
+
+- A `succeeded` result carries the model, token usage, finish reason and the
+  response content, with `service_tier` set to `batch` (the Batches API bills
+  every request at the batch rate). The tier the response reported is kept in
+  the `usage.service_tier_raw` attribute.
+- `errored`, `canceled` and `expired` results are error spans without usage,
+  because they are not billed.
+- The request is not part of a result, so the span has no input
+  (`batch.request_available` is `false`). Its start and end are both the
+  moment the result was read (`timing.synthetic` is `true`).
+- The attributes `batch.id`, `batch.custom_id` and `batch.result_type` link
+  the span back to the batch.
+- Trace and span ids are derived from the batch id and `custom_id`, so reading
+  the same batch again updates the same spans instead of adding new ones.
+
 ## Prompts
 
 Keep prompts in Spanlight and fetch them by label, so a new version ships
@@ -263,6 +341,16 @@ host must be `https://` unless it is localhost. Each
 run makes real provider calls. In a GitHub workflow use the
 `.github/actions/eval-gate` Action instead of calling the module yourself.
 
+The run prints a Markdown report (also appended to the GitHub step summary): the
+run's items, errors, cost and latency, then one row per score with its mean, the
+95% confidence interval on the pass rate of a boolean score (for example
+`64–88%`; `—` for numeric and categorical scores, and from servers that predate
+intervals), the number of items, the threshold and the result. When a threshold
+lies inside a boolean score's interval, the report adds "Interval includes the
+threshold; the result is not conclusive at this sample size." The gate itself
+still compares the mean with the threshold, so the exit codes do not depend on
+the interval.
+
 ## What gets recorded
 
 For every LLM call:
@@ -281,6 +369,22 @@ For every LLM call:
 | `output`                      | The assistant message, including tool calls                                                                                                                               |
 | `status` / `status_message`   | `error` plus the provider's message when the call fails                                                                                                                   |
 
+When a provider call fails with an HTTP error, the span also carries the
+provider's error type and code (`provider.error.type`, `provider.error.code`),
+the wait it asked for in seconds (`provider.retry_after_s`, from `retry-after-ms`
+or a numeric `retry-after`), and `provider.ratelimit`: the response's rate-limit
+headers, filtered through a fixed allowlist of 29 names. Only `retry-after`,
+`retry-after-ms`, OpenAI's `x-ratelimit-*` headers and Anthropic's
+`anthropic-ratelimit-*` and `anthropic-priority-*` headers are kept, each value
+at most 64 printable ASCII characters. Every other response header, including
+`authorization`, `x-api-key`, `cookie`, `set-cookie` and Anthropic's
+organization and workspace ids, is dropped before the span leaves your process.
+Connection errors and timeouts add none of these fields.
+When the Spanlight gateway refused the call itself (its `X-Spanlight-Code`
+header names one of its own codes, such as `RATE_LIMITED` for your gateway key's
+limit), the span carries that code as `spanlight.gateway.error_code` instead, and
+none of the `provider.*` fields above.
+
 The Spanlight server also redacts common secret patterns from
 payloads on ingestion. Set `capture_payloads = false` on a project to drop
 inputs and outputs entirely. To keep payloads out of the SDK side for a
@@ -310,7 +414,7 @@ options. Use `tracer.observe`, `tracer.span` and
 | `flush_interval`   |                         | `2.0`                   | Seconds a span may wait before it is sent                                |
 | `batch_size`       |                         | `100`                   | Spans per request (max 1000)                                             |
 | `max_queue`        |                         | `10000`                 | Buffered spans. When the buffer is full, the oldest spans are dropped.   |
-| `gzip`             |                         | `False`                 | Gzip-compress request bodies                                             |
+| `gzip`             |                         | `True`                  | Gzip-compress request bodies                                             |
 | `timeout`          |                         | `10.0`                  | HTTP timeout per request, in seconds                                     |
 | `max_retries`      |                         | `5`                     | Retries for network errors, 429 and 5xx                                  |
 | `shutdown_timeout` |                         | `5.0`                   | Time budget for the flush at interpreter exit                            |
@@ -343,10 +447,14 @@ cached_tokens=, cache_write_tokens=, cache_write_1h_tokens=)`,
 exception that escapes the block marks the span as failed and is re-raised
 unchanged.
 
-### `update_trace(name=None, user_id=None, session_id=None, tags=None)`
+### `update_trace(name=None, user_id=None, session_id=None, tags=None, metadata=None)`
 
 Sets trace-level fields on the trace of the current span. You can call it
-anywhere inside the trace. Tags accumulate across calls.
+anywhere inside the trace. Tags accumulate across calls. Metadata keys
+accumulate too, and a later value replaces a key's earlier one:
+`spanlight.update_trace(metadata={"customer_id": "acme", "feature": "search"})`.
+Up to 32 keys matching `^[a-z0-9_.-]{1,64}$`, values of 1 to 256 characters
+(numbers and booleans become text); an invalid entry is dropped with a debug log.
 
 ### `flush(timeout=None) -> bool` and `shutdown(timeout=None)`
 
@@ -380,6 +488,14 @@ logged to the `spanlight` logger, as are any spans the server rejects. If the
 queue fills up, the oldest spans are dropped and counted
 (`spanlight.get_client().exporter.dropped_count`).
 
+Since 0.2.0, every batch of spans also carries a small `client` object with the
+SDK name and version, a random `exporter_id` (new in each process, including a
+forked child) and two running totals for that exporter: `dropped_total` (spans
+and scores dropped because the queue was full) and `failed_total` (spans and
+scores lost after retries or rejected by the server). The totals are cumulative,
+so a batch delivered twice does not count a loss twice. It holds no span
+content, and older Spanlight servers ignore it.
+
 To see what the SDK is doing:
 
 ```python

{spanlight-0.1.2 → spanlight-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "spanlight"
-version = "0.1.2"
+version = "0.2.1"
 description = "Python SDK for Spanlight, open-source observability for LLM applications."
 readme = "README.md"
 license = "MIT"

{spanlight-0.1.2 → spanlight-0.2.1}/src/spanlight/__init__.py

@@ -25,6 +25,7 @@ from ._prompts import CompiledPrompt, Prompt, PromptUnavailableError, no_prompt
 from ._span import Span, SpanKind, SpanStatus
 from ._templates import InvalidTemplateError, MissingVariablesError, TemplateTooLargeError
 from ._version import __version__
+from .integrations._openai_batch import record_openai_batch_output
 from .integrations.anthropic import wrap_anthropic
 from .integrations.openai import wrap_openai
 
@@ -50,6 +51,7 @@ __all__ = [
     "no_prompt",
     "observe",
     "prompts",
+    "record_openai_batch_output",
     "score",
     "shutdown",
     "span",

{spanlight-0.1.2 → spanlight-0.2.1}/src/spanlight/_client.py

@@ -21,7 +21,8 @@ from ._context import (
 from ._exporter import BatchExporter
 from ._prompt_client import PromptClient
 from ._scores import ScoreValue, build_score
-from ._span import Span, SpanKind, TraceState, normalize_kind
+from ._span import Span, SpanKind, normalize_kind
+from ._trace_state import TraceState
 
 logger = logging.getLogger("spanlight")
 
@@ -60,7 +61,8 @@ class Spanlight:
         flush_interval: Maximum seconds a span waits before it is sent.
         batch_size: Maximum spans per request (capped at 1000).
         max_queue: Maximum spans buffered in memory; oldest dropped first.
-        gzip: Compress request bodies.
+        gzip: Gzip-compress request bodies (default). Turn it off only for a proxy that
+            cannot forward a compressed body.
         timeout: HTTP timeout per request, in seconds.
         max_retries: Retries per batch for network errors, 429 and 5xx.
         shutdown_timeout: Seconds allowed for the final flush at exit.
@@ -79,7 +81,7 @@ class Spanlight:
         flush_interval: float = 2.0,
         batch_size: int = 100,
         max_queue: int = 10_000,
-        gzip: bool = False,
+        gzip: bool = True,
         timeout: float = 10.0,
         max_retries: int = 5,
         shutdown_timeout: float = 5.0,
@@ -275,9 +277,12 @@ class Spanlight:
         user_id: str | None = None,
         session_id: str | None = None,
         tags: Iterable[str] | None = None,
+        metadata: Mapping[str, object] | None = None,
     ) -> None:
         """Set fields on the current trace. See :func:`spanlight.update_trace`."""
-        update_trace(name=name, user_id=user_id, session_id=session_id, tags=tags)
+        update_trace(
+            name=name, user_id=user_id, session_id=session_id, tags=tags, metadata=metadata
+        )
 
     # ------------------------------------------------------------------ #
     # Scores
@@ -426,6 +431,7 @@ def update_trace(
     user_id: str | None = None,
     session_id: str | None = None,
     tags: Iterable[str] | None = None,
+    metadata: Mapping[str, object] | None = None,
 ) -> None:
     """Attach trace-level fields to the trace of the current span.
 
@@ -437,13 +443,17 @@ def update_trace(
         user_id: Your application's user identifier.
         session_id: Groups traces into a conversation/session.
         tags: Free-form labels for filtering.
+        metadata: Key-value pairs such as ``{"customer_id": "acme"}``; a later value
+            replaces a key's earlier one. Invalid entries are dropped with a debug log.
     """
     span = get_current_span()
     if span is None:
         logger.debug("update_trace() called outside of a span; ignoring")
         return
     try:
-        span.trace.update(name=name, user_id=user_id, session_id=session_id, tags=tags)
+        span.trace.update(
+            name=name, user_id=user_id, session_id=session_id, tags=tags, metadata=metadata
+        )
     except Exception:
         logger.debug("update_trace() failed", exc_info=True)
 

{spanlight-0.1.2 → spanlight-0.2.1}/src/spanlight/_decorator.py

@@ -16,6 +16,7 @@ SDK itself are swallowed so they cannot break the decorated function.
 
 from __future__ import annotations
 
+import asyncio
 import functools
 import inspect
 import logging
@@ -196,7 +197,10 @@ def _traced_generator(
             item = step()
         except StopIteration as stop:
             _finish(
-                span, output=collected.result(stop.value), capture_output=options.capture_output
+                span,
+                output=collected.result(stop.value),
+                capture_output=options.capture_output,
+                output_cut=collected.truncated,
             )
             return stop.value
         except BaseException as exc:
@@ -212,7 +216,12 @@ def _traced_generator(
             try:
                 generator.close()
             finally:
-                _finish(span, output=collected.result(None), capture_output=options.capture_output)
+                _finish(
+                    span,
+                    output=collected.result(None),
+                    capture_output=options.capture_output,
+                    output_cut=collected.truncated,
+                )
             raise
         except BaseException as exc:
             step = functools.partial(generator.throw, exc)
@@ -244,16 +253,33 @@ async def _traced_async_generator(
     """Async counterpart of :func:`_traced_generator`."""
     collected = _OutputCollector(enabled=options.capture_output)
     step: Callable[[], Any] = functools.partial(generator.asend, None)
+    # A cancellation thrown in at a `yield` means the consumer abandoned the generator: asyncio
+    # closes one left behind by `async for ... break` that way at loop shutdown. Like a `break`
+    # over a sync generator, that ends the span as ok, not failed.
+    abandoned: BaseException | None = None
     while True:
         previous = get_current_span()
         token = set_current_span(span)
         try:
             item = await step()
         except StopAsyncIteration:
-            _finish(span, output=collected.result(None), capture_output=options.capture_output)
+            _finish(
+                span,
+                output=collected.result(None),
+                capture_output=options.capture_output,
+                output_cut=collected.truncated,
+            )
             return
         except BaseException as exc:
-            _finish(span, error=exc)
+            if exc is abandoned:
+                _finish(
+                    span,
+                    output=collected.result(None),
+                    capture_output=options.capture_output,
+                    output_cut=collected.truncated,
+                )
+            else:
+                _finish(span, error=exc)
             raise
         finally:
             restore_current_span(token, previous)
@@ -265,9 +291,15 @@ async def _traced_async_generator(
             try:
                 await generator.aclose()
             finally:
-                _finish(span, output=collected.result(None), capture_output=options.capture_output)
+                _finish(
+                    span,
+                    output=collected.result(None),
+                    capture_output=options.capture_output,
+                    output_cut=collected.truncated,
+                )
             raise
         except BaseException as exc:
+            abandoned = exc if isinstance(exc, asyncio.CancelledError) else None
             step = functools.partial(generator.athrow, exc)
         else:
             step = functools.partial(generator.asend, sent)
@@ -305,6 +337,7 @@ def _finish(
     *,
     output: Any = None,
     capture_output: bool = False,
+    output_cut: bool = False,
     error: BaseException | None = None,
 ) -> None:
     """End ``span`` recording either its output or the error. Never raises."""
@@ -314,6 +347,8 @@ def _finish(
                 span.record_error(error)
         elif capture_output:
             span.set_output(output)
+            # The collector kept only the first MAX_ITEMS yielded items.
+            span._output_cut = span._output_cut or output_cut
         span.end()
     except Exception:
         logger.debug("observe: failed to finish span", exc_info=True)

{spanlight-0.1.2 → spanlight-0.2.1}/src/spanlight/_experiments_report.py

@@ -2,11 +2,13 @@
 
 from __future__ import annotations
 
+import math
 from decimal import ROUND_HALF_UP, Decimal, InvalidOperation
 from typing import Any
 
 _PLACES = Decimal("0.000001")
 _UNKNOWN = "—"
+_INCONCLUSIVE = "Interval includes the threshold; the result is not conclusive at this sample size."
 
 
 def fmt(value: Decimal) -> str:
@@ -22,6 +24,30 @@ def _mean(stat: Any) -> Decimal | None:
     return Decimal(str(mean))
 
 
+def _interval(stat: Any) -> tuple[Decimal, Decimal] | None:
+    """The 95 % interval on a boolean score's pass rate.
+
+    ``None`` for a numeric or categorical score, or from a server that predates intervals.
+    """
+    if not isinstance(stat, dict):
+        return None
+    try:
+        low, high = Decimal(str(stat["ci_low"])), Decimal(str(stat["ci_high"]))
+    except (KeyError, InvalidOperation):  # a missing key, or null
+        return None
+    if not (low.is_finite() and high.is_finite()):  # "NaN" or "Infinity" would raise in floor
+        return None
+    return low, high
+
+
+def _percents(interval: tuple[Decimal, Decimal] | None) -> str:
+    """``64\u201388%``: the low bound floored, the high bound ceiled, so the range contains it."""
+    if interval is None:
+        return _UNKNOWN
+    low, high = interval
+    return f"{math.floor(low * 100)}\u2013{math.ceil(high * 100)}%"
+
+
 def check_thresholds(totals: dict[str, Any], fail_below: dict[str, Decimal]) -> list[str]:
     """One line per score whose mean is below its threshold or that the run did not produce.
 
@@ -41,7 +67,10 @@ def check_thresholds(totals: dict[str, Any], fail_below: dict[str, Decimal]) ->
 
 def _score_rows(totals: dict[str, Any], fail_below: dict[str, Decimal]) -> list[str]:
     scores = totals.get("scores") or {}
-    rows = ["| Score | Mean | Items | Threshold | Result |", "| --- | --- | --- | --- | --- |"]
+    rows = [
+        "| Score | Mean | 95% CI | Items | Threshold | Result |",
+        "| --- | --- | --- | --- | --- | --- |",
+    ]
     for name in sorted({*scores, *fail_below}):
         stat = scores.get(name)
         mean = _mean(stat)
@@ -55,12 +84,27 @@ def _score_rows(totals: dict[str, Any], fail_below: dict[str, Decimal]) -> list[
             result = "FAIL"
         rows.append(
             f"| {name} | {fmt(mean) if mean is not None else _UNKNOWN} "
+            f"| {_percents(_interval(stat))} "
             f"| {count if count is not None else _UNKNOWN} "
             f"| {'>= ' + fmt(threshold) if threshold is not None else _UNKNOWN} | {result} |"
         )
     return rows
 
 
+def _inconclusive(totals: dict[str, Any], fail_below: dict[str, Decimal]) -> list[str]:
+    """One line per gated boolean score whose threshold lies inside its interval.
+
+    The gate still compares the mean; this only says the verdict could flip with more items.
+    """
+    scores = totals.get("scores") or {}
+    lines: list[str] = []
+    for name, threshold in sorted(fail_below.items()):
+        interval = _interval(scores.get(name))
+        if interval is not None and interval[0] <= threshold <= interval[1]:
+            lines.append(f"{name}: {_INCONCLUSIVE}")
+    return lines
+
+
 def _number(value: Any) -> str:
     return _UNKNOWN if value is None else str(value)
 
@@ -107,6 +151,8 @@ def render_markdown(
             *_score_rows(totals, fail_below),
             "",
         ]
+        if inconclusive := _inconclusive(totals, fail_below):
+            lines += [*inconclusive, ""]
     if failures:
         lines += ["Below threshold:", "", *[f"- {line}" for line in failures], ""]
     return "\n".join(lines)