selfevals 0.2.2__py3-none-any.whl

selfevals/.agents/skills/error-analysis/SKILL.md

@@ -0,0 +1,149 @@
+---
+name: error-analysis
+description: Analyze an experiment's failed traces and grow its failure-mode taxonomy. Use when a selfevals experiment has staged error analysis (or a human asks "why is this experiment failing?"). Drives the pull → open coding → axial coding → push → promote cycle. selfevals owns the data and the contract; you provide the intelligence (the coding), it never calls an LLM itself.
+---
+
+# Error Analysis (open + axial coding)
+
+You are the intelligence half of selfevals's error-analysis loop. selfevals has
+already run the experiment, tagged the deterministic failures, and (if the YAML
+opted in and the trigger fired) staged a bundle. Your job is to read the failed
+traces and **grow the failure-mode taxonomy** so the next experiment can target
+specific, named modes.
+
+This is the established **open coding → axial coding** method (Hamel Husain &
+Shreya Shankar's error analysis for LLM apps). Apply it faithfully — not an
+ad-hoc clustering.
+
+The hard boundary: **selfevals owns data + contract + persistence; you own the
+coding.** You never edit the database directly — everything flows through
+`analyze pull` / `analyze push`. selfevals never calls an LLM; you do all the
+reading and judgement.
+
+## 0. Preflight
+
+- Confirm the `selfevals` CLI is available: `selfevals --help`. If the project
+  uses `uv`, prefix commands with `uv run`.
+- Identify the target **workspace id** and **experiment id**. If the user gave
+  a spec path, the workspace is the spec's `workspace:` key; the experiment id
+  is printed when the experiment was created/run.
+- You will need the db path the project uses (the CLI's `--db` flag, default per
+  the project). Reuse whatever the human/other commands already use.
+
+## 1. Pull the bundle
+
+```bash
+selfevals --db <db> analyze pull <workspace_id> <experiment_id> > bundle.json
+```
+
+`bundle.json` contains:
+- `taxonomy`: the **live failure modes** (id, slug, title, definition, status).
+  This is what you classify against. Treat OFFICIAL modes as the stable
+  vocabulary; CANDIDATE modes are proposals awaiting a human.
+- `traces`: each failed trace with `grade` (label, score, any
+  `deterministic_modes` already tagged, optional `judge_reason`), the
+  `transcript` (the real conversation), and `first_error_span` (selfevals's
+  guess at where it first went wrong).
+
+If `traces` is empty, the run was healthy or nothing was staged — report that
+and stop. Don't invent failures.
+
+## 2. Open coding — one note per first failure
+
+For **each** failed trace, in order:
+1. Read the `transcript` and `first_error_span`. Find the **first** thing that
+   went wrong (Hamel's rule: code the first failure, not the downstream cascade).
+2. Write a single, concrete, one-line note describing *what* failed — behavioral,
+   not a fix. Good: "cited a price the catalog never contained." Bad: "should
+   validate prices" (that's a fix, not an observation).
+3. If a `deterministic_modes` tag already fully explains the failure, you may
+   skip writing a new note — **unless** the residue suggests a deeper mode the
+   deterministic rule missed. Don't re-discover what's already tagged.
+
+Keep notes verbatim-grounded: capture a short `quote` from the transcript that
+evidences the failure. You'll attach it to the assignment.
+
+## 3. Axial coding — classify against the LIVE taxonomy
+
+This is the discipline that makes the taxonomy stable ("discover once, classify
+thereafter"). For each note:
+
+- **Does it match an existing mode's `definition`?** → assign that mode by
+  `mode_id`. Prefer an existing mode whenever the definition genuinely fits.
+- **No existing mode fits?** → propose a `new_mode_slug` with a **testable
+  definition** (a sentence a different person could apply to a new trace and
+  agree on). Lower-case, snake_case slug.
+- **Never rename or redefine an existing mode.** If an official mode's
+  definition is wrong, note it for the human — do not silently fork it.
+
+Each trace gets **exactly one** of `mode_id` *or* `new_mode_slug` (XOR). selfevals
+rejects an assignment that sets both or neither.
+
+## 4. Saturation check
+
+Track new modes as you go. When ~20 consecutive traces produce **no** new mode,
+you've reached saturation — the taxonomy now covers this run. Note it; you can
+stop proposing and just classify the remainder.
+
+## 5. Optional: hypotheses
+
+For the dominant modes, you may add a `hypotheses` entry: a testable statement
+("tightening the price-grounding instruction will reduce `invented_price`") with
+optional `suggested_parameters`. selfevals stores these; the proposer consults
+them next iteration. It does **not** run them automatically.
+
+## 6. Push the result
+
+Emit an `AnalysisResult` JSON and push it:
+
+```bash
+selfevals --db <db> analyze push <workspace_id> <experiment_id> --by "agent:<your-name>" < result.json
+```
+
+`result.json` shape:
+
+```json
+{
+  "proposed_modes": [
+    {"slug": "invented_price", "title": "Invented price",
+     "definition": "Agent states a price not present in the catalog context."}
+  ],
+  "assignments": [
+    {"trace_id": "trc_…", "mode_id": "fm_…", "quote": "…", "open_note": "…"},
+    {"trace_id": "trc_…", "new_mode_slug": "invented_price", "quote": "$499", "open_note": "…"}
+  ],
+  "hypotheses": [
+    {"targets_mode_slug": "invented_price",
+     "statement": "Add an explicit 'only cite catalog prices' instruction.",
+     "suggested_parameters": {"prompt_section": "grounding"}}
+  ]
+}
+```
+
+selfevals validates the whole result **before** writing (transactional intent),
+enforces the XOR and classify-don't-rename invariants, creates candidate modes
+idempotently (re-proposing an existing slug does not duplicate it), and stamps
+`mode_id` onto each trace's grader results. It prints a summary.
+
+## 7. Hand back to the human
+
+Print which candidates are strongest — frequency (how many traces) plus your
+confidence — so the human can batch-promote:
+
+```bash
+selfevals --db <db> failuremode list <workspace_id> --status candidate
+selfevals --db <db> failuremode promote <workspace_id> <fm_id>
+```
+
+Promotion (candidate → official) is a **human gate** by design. Never promote on
+your own. Your output is a recommendation, not a decision.
+
+## What you must not do
+
+- Do not call any database or storage API directly. Only `analyze pull/push`
+  and `failuremode *`.
+- Do not rename, redefine, or merge existing modes (merging is a human
+  `failuremode merge`).
+- Do not promote candidates.
+- Do not invent failures for healthy traces or pad the taxonomy to look busy —
+  a smaller, sharper taxonomy is the goal.

selfevals/__init__.py

@@ -0,0 +1,19 @@
+"""selfevals — self-improving evals framework for AI agents."""
+
+from selfevals.sdk import (
+    InitResult,
+    SelfEvalsAlreadyInitialized,
+    init,
+    is_initialized,
+    shutdown,
+)
+from selfevals.version import __version__
+
+__all__ = [
+    "InitResult",
+    "SelfEvalsAlreadyInitialized",
+    "__version__",
+    "init",
+    "is_initialized",
+    "shutdown",
+]

selfevals/_errors.py

@@ -0,0 +1,44 @@
+"""Shared exception hierarchy.
+
+Two-tier model:
+
+- `SelfEvalsError`     — base for all selfevals-raised exceptions. Catch
+  this at the outermost boundary to distinguish selfevals failures from
+  truly internal Python errors (e.g. an OS-level disk-full).
+- `SelfEvalsUserError` — base for *user-correctable* failures: bad YAML,
+  missing dataset, unknown grader, unreachable HTTP endpoint, locked
+  database. These flow up to the CLI which prints a single-line
+  `error: ...` and exits 2 (no stacktrace).
+
+Everything else (assertion violations, programmer bugs, unexpected
+Pydantic shapes that escape our friendly wrappers) keeps its stack
+trace and yields exit 1 from the CLI dispatcher.
+
+The CLI's `CommandError` is kept as a thin alias of `SelfEvalsUserError`
+for source compatibility with the rest of the package; both terms refer
+to the same class so `except CommandError` and `except SelfEvalsUserError`
+behave identically.
+"""
+
+from __future__ import annotations
+
+
+class SelfEvalsError(Exception):
+    """Root of selfevals's exception hierarchy."""
+
+
+class SelfEvalsUserError(SelfEvalsError):
+    """Raised for failures the user can fix without reading a traceback.
+
+    The message must be self-contained: it is printed verbatim as
+    `error: <message>` and the user gets no other context. Include the
+    file path, the offending field, and (ideally) one concrete hint
+    about how to fix it.
+    """
+
+    def __init__(self, message: str, *, hint: str | None = None) -> None:
+        self.hint = hint
+        if hint:
+            super().__init__(f"{message}\n  hint: {hint}")
+        else:
+            super().__init__(message)

selfevals/_internal/hashing.py

@@ -0,0 +1,23 @@
+"""Content hashing for snapshot identity and pointer integrity."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Any
+
+
+def content_hash(payload: Any) -> str:
+    """Return a stable `sha256:...` hash of a JSON-serializable payload.
+
+    Keys are sorted and separators are canonical to make the hash reproducible
+    regardless of dict insertion order.
+    """
+    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode()
+    digest = hashlib.sha256(encoded).hexdigest()
+    return f"sha256:{digest}"
+
+
+def bytes_hash(data: bytes) -> str:
+    """Return `sha256:...` for raw bytes (used for stored blobs)."""
+    return f"sha256:{hashlib.sha256(data).hexdigest()}"

selfevals/_internal/ids.py

@@ -0,0 +1,65 @@
+"""ULID generation (stdlib only).
+
+A ULID is a 128-bit identifier: 48-bit big-endian millisecond timestamp +
+80 bits of cryptographic randomness, encoded as 26 chars of Crockford Base32.
+Lexicographically sortable by creation time.
+
+Spec: https://github.com/ulid/spec
+"""
+
+from __future__ import annotations
+
+import re
+import secrets
+import time
+
+_ALPHABET = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"  # Crockford Base32 (no I L O U)
+_ALPHABET_SET = frozenset(_ALPHABET)
+_ULID_RE = re.compile(r"^[0-9A-HJKMNP-TV-Z]{26}$")
+
+
+def new_ulid() -> str:
+    """Generate a new ULID string (26 chars, uppercase Crockford Base32)."""
+    timestamp_ms = int(time.time() * 1000) & ((1 << 48) - 1)
+    randomness = secrets.randbits(80)
+    value = (timestamp_ms << 80) | randomness
+    return _encode(value)
+
+
+def _encode(value: int) -> str:
+    chars: list[str] = []
+    for _ in range(26):
+        chars.append(_ALPHABET[value & 0x1F])
+        value >>= 5
+    return "".join(reversed(chars))
+
+
+def is_ulid(value: str) -> bool:
+    """Return True if `value` is a syntactically valid ULID string."""
+    return bool(_ULID_RE.match(value))
+
+
+def new_prefixed_id(prefix: str) -> str:
+    """Generate a prefixed identifier (e.g. `ws_01H...`).
+
+    Prefix must be 2-6 lowercase ASCII letters; separator is underscore.
+    """
+    if (
+        not 2 <= len(prefix) <= 6
+        or not prefix.isascii()
+        or not prefix.isalpha()
+        or not prefix.islower()
+    ):
+        raise ValueError(f"invalid id prefix: {prefix!r}")
+    return f"{prefix}_{new_ulid()}"
+
+
+_PREFIXED_RE = re.compile(r"^[a-z]{2,6}_[0-9A-HJKMNP-TV-Z]{26}$")
+
+
+def is_prefixed_id(value: str, prefix: str | None = None) -> bool:
+    if not _PREFIXED_RE.match(value):
+        return False
+    if prefix is None:
+        return True
+    return value.startswith(f"{prefix}_")

selfevals/_internal/time.py

@@ -0,0 +1,17 @@
+"""Timezone-aware UTC helpers. All timestamps in selfevals are tz-aware UTC."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+
+def utc_now() -> datetime:
+    """Return current time as a tz-aware UTC datetime."""
+    return datetime.now(UTC)
+
+
+def ensure_utc(dt: datetime) -> datetime:
+    """Coerce a datetime to UTC; reject naive datetimes."""
+    if dt.tzinfo is None:
+        raise ValueError("naive datetime is not allowed; provide tz-aware UTC")
+    return dt.astimezone(UTC)

selfevals/analysis/__init__.py

@@ -0,0 +1,23 @@
+"""Error-analysis handshake: build bundles, ingest results.
+
+selfevals owns the data, the contract, the persistence, and the verification.
+The intelligence (open/axial coding) lives in an external coding agent that
+honours the `AnalysisBundle` / `AnalysisResult` contract defined in `schemas`.
+selfevals never calls an LLM here. See docs/spec/error_analysis_design.md.
+"""
+
+from __future__ import annotations
+
+from selfevals.analysis.bundle import build_bundle
+from selfevals.analysis.ingest import IngestSummary, ingest_result
+from selfevals.analysis.schemas import AnalysisBundle, AnalysisResult
+from selfevals.analysis.staging import AnalysisStagingRecord
+
+__all__ = [
+    "AnalysisBundle",
+    "AnalysisResult",
+    "AnalysisStagingRecord",
+    "IngestSummary",
+    "build_bundle",
+    "ingest_result",
+]

selfevals/analysis/bundle.py

@@ -0,0 +1,162 @@
+"""Build an AnalysisBundle from a workspace's persisted traces.
+
+`build_bundle` gathers the failed traces of an experiment (optionally one
+iteration), projects each into the wire shape an external agent needs to do
+open/axial coding, and attaches the live taxonomy the agent must classify
+against. Pure read — it never mutates storage. See design §4.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING
+
+from selfevals.analysis.schemas import (
+    AnalysisBundle,
+    BundleErrorSpan,
+    BundleGrade,
+    BundleMessage,
+    BundleTrace,
+    TaxonomyEntry,
+)
+from selfevals.schemas.failure_mode import FailureMode
+from selfevals.schemas.trace import ErrorSpan, LLMCallSpan, ToolCallSpan, Trace
+
+if TYPE_CHECKING:
+    from selfevals.storage.sqlite import SQLiteStorage
+
+# Labels that count as "needs coding". PASS/SKIPPED are dropped.
+_FAILED_LABELS = {"fail", "error", "partial"}
+
+
+def _transcript(trace: Trace) -> list[BundleMessage]:
+    """Recover messages from the inline OTel-imported form.
+
+    The importer stores reconstructed messages under provider_metadata
+    (`selfevals.messages_in` / `selfevals.messages_out`) on each LLM span.
+    We concatenate them across LLM spans in span order so the agent reads the
+    real conversation, not pointers.
+    """
+    out: list[BundleMessage] = []
+    for span in trace.spans:
+        if not isinstance(span, LLMCallSpan):
+            continue
+        for key in ("selfevals.messages_in", "selfevals.messages_out"):
+            raw = span.provider_metadata.get(key)
+            if not isinstance(raw, list):
+                continue
+            for msg in raw:
+                if isinstance(msg, dict) and "role" in msg and "content" in msg:
+                    out.append(BundleMessage(role=str(msg["role"]), content=str(msg["content"])))
+    return out
+
+
+def _first_error_span(trace: Trace) -> BundleErrorSpan | None:
+    """The first failure in the trace — Hamel's "code the first failure" rule.
+
+    Prefers an explicit ErrorSpan; falls back to the first errored tool call.
+    """
+    for span in trace.spans:
+        if isinstance(span, ErrorSpan):
+            return BundleErrorSpan(kind=str(span.kind), name=span.name, error=span.message)
+        if isinstance(span, ToolCallSpan) and span.error:
+            return BundleErrorSpan(kind=str(span.kind), name=span.name, error=span.error)
+    return None
+
+
+def _grade(trace: Trace) -> BundleGrade:
+    """Collapse the trace's grader results into one bundle grade.
+
+    Worst label wins; deterministic failure-mode tags and any judge reason are
+    surfaced so the agent focuses on the untagged residue.
+    """
+    severity = {"error": 4, "fail": 3, "partial": 2, "skipped": 1, "pass": 0}
+    label = "pass"
+    score: float | None = None
+    modes: list[str] = []
+    reason: str | None = None
+    for gr in trace.grader_results:
+        if severity.get(gr.label, 0) >= severity.get(label, 0):
+            label = gr.label
+            score = gr.score
+        modes.extend(gr.failure_modes)
+        # The judge reason is payload-routed; we pass the pointer through as a
+        # hint when present (resolving it is the object store's job, optional).
+        if reason is None and gr.reason_pointer:
+            reason = gr.reason_pointer
+    # De-dup modes preserving order.
+    seen: set[str] = set()
+    deduped: list[str] = []
+    for m in modes:
+        if m not in seen:
+            seen.add(m)
+            deduped.append(m)
+    return BundleGrade(label=label, score=score, deterministic_modes=deduped, judge_reason=reason)
+
+
+def _is_failed(trace: Trace) -> bool:
+    if trace.final_state.status != "completed":
+        return True
+    return any(gr.label in _FAILED_LABELS for gr in trace.grader_results)
+
+
+def build_bundle(
+    storage: SQLiteStorage,
+    *,
+    workspace_id: str,
+    experiment_id: str,
+    iteration: int | None = None,
+    only_failed: bool = True,
+) -> AnalysisBundle:
+    """Assemble the bundle. Traces are matched by experiment (and iteration)
+    via the run metadata stored in each trace's payload."""
+    clauses = [
+        "workspace_id = ?",
+        "entity_type = 'Trace'",
+        "json_extract(payload, '$.run.experiment_id') = ?",
+    ]
+    params: list[object] = [workspace_id, experiment_id]
+    if iteration is not None:
+        clauses.append("json_extract(payload, '$.run.iteration') = ?")
+        params.append(iteration)
+    sql = "SELECT payload FROM entities WHERE " + " AND ".join(clauses)
+    rows = storage.connection.execute(sql, tuple(params)).fetchall()
+
+    traces = [Trace.model_validate(json.loads(p)) for (p,) in rows]
+
+    bundle_traces: list[BundleTrace] = []
+    for trace in traces:
+        if only_failed and not _is_failed(trace):
+            continue
+        bundle_traces.append(
+            BundleTrace(
+                trace_id=trace.id,
+                run_id=trace.run.run_id,
+                thread_id=trace.run.thread_id,
+                eval_case_id=trace.run.eval_case_id,
+                grade=_grade(trace),
+                transcript=_transcript(trace),
+                first_error_span=_first_error_span(trace),
+            )
+        )
+
+    with storage.open(workspace_id) as scope:
+        taxonomy = [
+            TaxonomyEntry(
+                id=fm.id,
+                slug=fm.slug,
+                title=fm.title,
+                definition=fm.definition,
+                status=str(fm.status),
+            )
+            for fm in scope.list_entities(FailureMode)
+            if isinstance(fm, FailureMode) and str(fm.status) != "retired"
+        ]
+
+    return AnalysisBundle(
+        workspace_id=workspace_id,
+        experiment_id=experiment_id,
+        iteration=iteration,
+        taxonomy=taxonomy,
+        traces=bundle_traces,
+    )

selfevals/analysis/hypothesis.py

@@ -0,0 +1,26 @@
+"""HypothesisRecord — a testable change targeting a failure mode.
+
+Produced by error analysis (the `hypotheses` block of an AnalysisResult) and
+stored as a workspace entity linked to the experiment. The proposer consults
+these to target a specific mode in the next iteration; selfevals does not run
+them automatically. See docs/spec/error_analysis_design.md §7.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from pydantic import Field
+
+from selfevals.schemas._base import BaseEntity, NonEmptyStr
+
+
+class HypothesisRecord(BaseEntity):
+    _id_prefix: ClassVar[str] = "hyp"
+
+    experiment_id: NonEmptyStr
+    targets_mode_slug: NonEmptyStr
+    statement: NonEmptyStr
+    suggested_parameters: dict[str, Any] = Field(default_factory=dict)
+    consumed_by_iteration: int | None = Field(default=None, ge=0)
+    """Set once a proposer has used this hypothesis, so it isn't re-applied."""