tracelens 0.1.0__tar.gz

tracelens-0.1.0/.dockerignore

@@ -0,0 +1,41 @@
+# Git
+.git
+.gitignore
+
+# Python
+__pycache__
+*.py[cod]
+*$py.class
+*.egg-info
+.eggs
+dist
+build
+*.egg
+
+# Virtual environments
+.venv
+venv
+ENV
+
+# IDE
+.idea
+.vscode
+*.swp
+*.swo
+
+# Testing artifacts
+.pytest_cache
+.coverage
+coverage/
+htmlcov/
+.mypy_cache
+
+# Documentation
+docs/_build
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Claude
+.claude/

tracelens-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,52 @@
+---
+name: Bug report
+about: Report a defect with a minimal reproduction
+title: "[bug] "
+labels: bug
+assignees: ''
+---
+
+## What happened
+
+<!-- A clear, concise description of the bug. -->
+
+## What you expected
+
+<!-- What did you expect to happen instead? -->
+
+## Minimal reproduction
+
+<!--
+Smallest possible code or eval-set snippet that reproduces the issue.
+If it's CI-related, paste the relevant workflow + run URL.
+-->
+
+```python
+# repro
+```
+
+## Environment
+
+- tracelens version: `pip show tracelens | grep Version`
+- Python version: `python --version`
+- OS: (macOS 14 / Ubuntu 22.04 / Windows 11 / ...)
+- Install method: `pip` / `uv` / git clone
+- Optional extras installed: `[llm]` / `[http]` / `[dev]` / none
+
+## Logs / traceback
+
+<details>
+<summary>Full traceback</summary>
+
+```
+<paste here>
+```
+
+</details>
+
+## Anything else?
+
+<!--
+e.g., does it reproduce with `num_runs=1`? Does it reproduce on a fresh venv?
+Does the same DecisionSpec fingerprint appear across runs?
+-->

tracelens-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,40 @@
+---
+name: Feature request
+about: Propose a new capability or API change
+title: "[feat] "
+labels: enhancement
+assignees: ''
+---
+
+## The agent / eval scenario you're trying to support
+
+<!--
+Start with the use case, not the API.
+What kind of agent? What grading dimension? What guarantee do you need?
+-->
+
+## What's missing today
+
+<!-- Why can't you express this with the current tracelens surface? -->
+
+## Proposed API or behavior
+
+<!--
+Sketch the smallest API change you can imagine.
+Code snippets > prose.
+-->
+
+```python
+# proposed usage
+```
+
+## Alternatives you considered
+
+<!-- e.g. subclassing existing graders, writing a custom adapter, doing it out-of-band. -->
+
+## Compatibility considerations
+
+- [ ] Does this require a new `DecisionSpec` field? (If yes, fingerprint impact?)
+- [ ] Does this change baseline / regression semantics?
+- [ ] Does this add a new optional dependency?
+- [ ] Could this be a separate package that depends on tracelens instead of living in core?

tracelens-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,41 @@
+<!--
+Thanks for contributing to tracelens!
+Please keep one logical change per PR. Split branches that touch
+multiple layers (e.g. adapter + statistics) into separate PRs.
+-->
+
+## Summary
+
+<!-- One paragraph: what does this PR change, from the user's perspective? -->
+
+## Motivation
+
+<!--
+Why is this change needed? What problem does it solve?
+What alternatives did you consider and why did you reject them?
+-->
+
+## Type of change
+
+- [ ] Bug fix
+- [ ] New feature (non-breaking)
+- [ ] Breaking change (please flag in CHANGELOG under [Unreleased] → Changed)
+- [ ] Docs / examples / benchmarks
+- [ ] Internal refactor (no user-visible change)
+
+## Verification
+
+- [ ] `pytest -q` passes locally
+- [ ] `ruff check src/ tests/` passes
+- [ ] `mypy src/tracelens/` passes (strict)
+- [ ] New / changed code is covered by tests
+- [ ] If touching `DecisionSpec`, baseline, or regression logic: backwards-compat note added to `CHANGELOG.md`
+- [ ] If adding to `src/tracelens/__init__.py`: this symbol is intended as stable public API
+
+## Linked issues
+
+<!-- Closes #123 / Refs #456 -->
+
+## Notes for the reviewer
+
+<!-- Anything subtle, controversial, or worth flagging. -->

tracelens-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,157 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+# Cancel in-flight runs for the same branch/PR — keeps CI fast.
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  # ------------------------------------------------------------------
+  # core: install WITHOUT the [llm] extra and confirm the test suite
+  # still passes. tracelens has no optional-dep-gated tests after the
+  # LiteLLM wrapper was removed; this job exists to verify the core
+  # import graph stays clean under a minimal install.
+  # ------------------------------------------------------------------
+  core:
+    name: core (py${{ matrix.python }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+          cache: pip
+      - name: Install (core + dev)
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,http]"
+      - name: Verify core-only import
+        run: |
+          python -c "import tracelens; print(f'{len(tracelens.__all__)} public exports')"
+      - name: Run tests
+        run: pytest -q --no-header
+
+  # ------------------------------------------------------------------
+  # with-llm: install the [llm] convenience bundle so subclasses of
+  # LLMProvider that happen to import openai/anthropic at module load
+  # keep working. Same test suite as core — nothing conditionally
+  # depends on these SDKs today.
+  # ------------------------------------------------------------------
+  with-llm:
+    name: with-llm (py3.13)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+      - name: Install (core + dev + llm + http)
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,llm,http]"
+      - name: Run tests
+        run: pytest -q --no-header
+
+  # ------------------------------------------------------------------
+  # examples-smoke: end-to-end smoke test of the packaged example.
+  # Proves the CLI entry point is wired correctly and that a realistic
+  # user workflow (load tasks -> run agent -> grade -> report) works
+  # against the built package.
+  # ------------------------------------------------------------------
+  examples-smoke:
+    name: examples-smoke (py3.13)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[http]"
+      - name: Run bundled example
+        run: python examples/run_eval.py
+      - name: Verify CLI entry point
+        run: tracelens --help
+
+  # ------------------------------------------------------------------
+  # lint: ruff check on source + tests.
+  # ------------------------------------------------------------------
+  lint:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+      - name: Install ruff
+        run: pip install "ruff>=0.1"
+      - name: ruff check
+        run: ruff check src/ tests/
+
+  # ------------------------------------------------------------------
+  # typecheck: strict mypy check for the public typed package promise.
+  # ------------------------------------------------------------------
+  typecheck:
+    name: typecheck
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+      - name: mypy
+        run: mypy src/tracelens/
+
+  # ------------------------------------------------------------------
+  # build: prove that the source distribution and wheel are publishable
+  # on every PR and main commit. The release workflow reuses the same
+  # build path before uploading to PyPI.
+  # ------------------------------------------------------------------
+  build:
+    name: build artifacts
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+      - name: Build package
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install build twine
+          python -m build
+          python -m twine check dist/*
+      - name: Smoke install wheel
+        run: |
+          python -m venv /tmp/tracelens-smoke
+          /tmp/tracelens-smoke/bin/python -m pip install dist/*.whl
+          /tmp/tracelens-smoke/bin/python -c "import tracelens; print(tracelens.__version__)"
+          /tmp/tracelens-smoke/bin/tracelens --help

tracelens-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,66 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*.*.*"
+
+permissions:
+  contents: read
+
+jobs:
+  publish:
+    name: publish to PyPI
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+
+      - name: Build package
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install build twine
+          python -m build
+          python -m twine check dist/*
+
+      - name: Verify tag matches package version
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          python - <<'PY'
+          import email.parser
+          import os
+          import zipfile
+          from pathlib import Path
+
+          tag = os.environ["GITHUB_REF_NAME"].removeprefix("v")
+          wheel = next(Path("dist").glob("tracelens-*.whl"))
+          with zipfile.ZipFile(wheel) as zf:
+              metadata_name = next(
+                  name for name in zf.namelist()
+                  if name.endswith(".dist-info/METADATA")
+              )
+              metadata = email.parser.Parser().parsestr(
+                  zf.read(metadata_name).decode()
+              )
+          version = metadata["Version"]
+          if version != tag:
+              raise SystemExit(
+                  f"tag v{tag} does not match built package version {version}"
+              )
+          print(f"release version verified: {version}")
+          PY
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/

tracelens-0.1.0/.gitignore

@@ -0,0 +1,88 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+.env
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+coverage/
+.tox/
+.nox/
+.cache/
+
+# Type checking
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Lint caches
+.ruff_cache/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+.project
+.pydevproject
+.settings/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Claude / harness state
+.claude/
+.harness-mem/
+.mcp.json
+
+# Worktrees
+.worktrees/
+
+# Jupyter
+.ipynb_checkpoints/
+*.ipynb
+
+# Build
+*.manifest
+*.spec
+
+# Logs
+*.log
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Generated reports
+examples/report.html
+report.html
+
+# Benchmark artifacts — regenerate via benchmarks/*/README.md
+benchmarks/**/baseline_*.json
+benchmarks/**/run_*.json

tracelens-0.1.0/CHANGELOG.md

@@ -0,0 +1,164 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
+aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) —
+but while `0.x`, minor bumps may contain breaking changes. Treat the
+top-level `tracelens.*` imports as the stable surface; submodule paths may move.
+
+## [Unreleased]
+
+Everything under this heading is queued for the next tag-driven release. Two
+themes:
+
+1. **Productization pre-v1.0** — turn the engine into something an external
+   user can actually install, understand, and trust.
+2. **Infra-noise differentiator** — implement Anthropic's "Quantifying
+   infrastructure noise in agentic coding evals" recommendations
+   (Feb 2026) as first-class framework features.
+
+### Added
+
+- **`DecisionSpec.InfraConfig`.** Resource limits (CPU/memory guaranteed
+  vs. hard-limit), time budget, concurrency level, sandbox provider, and
+  harness version are now first-class experimental variables that feed
+  into the `DecisionSpec` SHA-256 fingerprint. Observational fields
+  (hostname, container ID, wall-clock start) are recorded but
+  intentionally excluded from the fingerprint, so two runs with identical
+  configs on different hosts collide to the same fingerprint. Based on
+  Anthropic's recommendation that "resource configuration ... [be]
+  documented and controlled with the same rigor as prompt format or
+  sampling temperature."
+- **`TrialStatus.INFRA_ERROR` + `InfraError` exception.** The runner now
+  classifies `MemoryError`, `ConnectionError`, and adapter-raised
+  `InfraError` as infrastructure failures, distinct from task-level
+  `FAILED` status. `TrialBatch.infra_error_rate` and `infra_error_count`
+  aggregate at the suite level.
+- **`RegressionDetector.compare_with_specs()`.** New method that diffs
+  the two runs' `InfraConfig`s. When they differ AND a regression's
+  absolute delta is below the 3pp noise band (Anthropic's threshold),
+  the regression is marked `within_noise_band=True` and filtered out
+  of `blocking_regressions`. Default `should_block_ci()` honors that
+  filter so CI doesn't gate on ambiguous noise.
+- **`ReportGenerator` surfaces infra metrics.** `infra_error_rate`
+  appears in markdown, CI summary, and HTML outputs when non-zero.
+  Regression reports with an infra-config mismatch get an explicit
+  warning block plus a "blocking / total" breakdown in the CI line.
+- **Flagship benchmark pack** at `benchmarks/high-stakes-autonomous/`.
+  Six tasks (safety GATE + resource-sensitive capability) plus a
+  runnable script that reproduces Anthropic's finding end-to-end:
+  same agent, same tasks, 100% → 50% pass rate purely from a memory
+  budget change, correctly attributed to infrastructure rather than
+  capability. Ships with a walkthrough README.
+- **Three runnable examples** under `examples/`:
+  - `http_agent_eval.py` — `HTTPAPIAdapter` + `JsonSchemaGrader` against
+    an in-process stdlib HTTP server. Swap the URL for your real
+    service.
+  - `contract_eval.py` — `BehaviorContract.to_graders()` producing a
+    mixed GATE/WARN/TRACK grader suite from one declaration.
+  - `noise_aware_regression.py` — 98-line version of the infra-noise
+    differentiator: two runs, different memory budgets, fingerprint
+    mismatch, noise-aware regression report.
+- **4-job GitHub Actions CI workflow** at `.github/workflows/ci.yml`:
+  `core` (Python 3.11 / 3.12 / 3.13), `with-llm`, `examples-smoke`, and
+  `lint`. Concurrency group cancels in-flight runs on re-push.
+- **Tag-driven release workflow** at `.github/workflows/release.yml`.
+  Versions come from `vX.Y.Z` git tags via `hatch-vcs`; the workflow builds,
+  validates, checks the tag/version match, and publishes with PyPI trusted
+  publishing.
+- **OSS launch guidance**: `docs/releasing.md`, `docs/scenarios.md`, and
+  `examples/README.md`.
+- **Curated public API** at `tracelens/*` (83 symbols). Top-level imports
+  are the stable surface; submodules may move.
+- **OSS hygiene files**: `LICENSE` (MIT), `CONTRIBUTING.md`,
+  `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), `SECURITY.md`,
+  `CHANGELOG.md` (this file), GitHub issue templates, and a pull
+  request template.
+- **Typed package marker**: `src/tracelens/py.typed` now ships in
+  wheels so the `Typing :: Typed` classifier reflects the package
+  artifact.
+
+### Changed
+
+- **pyproject.toml metadata** — real author + project URLs (repo,
+  issues, changelog), richer classifiers, descriptive keywords. Moved
+  `[project.urls]` below `dependencies` to fix a hatchling editable-
+  install break.
+- **Public installation docs** — first-path README, installation, and
+  CI/CD guidance now describe public GitHub installs instead of private
+  repository authentication.
+- **CI type gate** — the workflow now runs strict mypy as a required
+  job, matching the contributor verification checklist.
+- **sdist contents** — maintainer-local `CLAUDE.md` is excluded from
+  source distributions.
+- **Ruff baseline** — applied autofixes to 67 pre-existing findings
+  (unused imports, `datetime.utcnow()` → `datetime.now(UTC)`, f-string
+  cleanup). Configured `ignore = ["E501", "UP042"]` for the remaining
+  23 long lines and 7 `str, Enum` sites as deferred cleanup.
+- **Factory policy change for LLM providers** — `create_provider()`
+  now only supports `"in-memory"`; passing any other alias raises
+  `ValueError` with guidance to subclass `LLMProvider` directly. This
+  replaces the shipped LiteLLM wrapper with a documented subclassing
+  pattern; tracelens no longer tries to own the provider integration
+  layer.
+
+### Removed
+
+- **`WorkflowTask` / `WorkflowRunner` / `WorkflowAdapter`** (392 LOC +
+  tests). Codex found that `workflow_runner.py:158-194` computed per-
+  step grading outcomes then silently dropped them on the floor. No
+  example, benchmark, CLI, or README referenced the workflow layer.
+  Breaking change for the top-level API, but no known downstream user.
+- **`LiteLLMProvider`** (`src/tracelens/llm/litellm_provider.py`, ~40
+  LOC). A thin wrapper over `litellm.acompletion` with no distinct
+  value over the vendor SDK. Users now subclass `LLMProvider`
+  directly; the factory module docstring shows the pattern. The
+  `[llm]` extra retains `openai` + `anthropic` as a convenience
+  bundle, minus `litellm`.
+- **`BehaviorContract.tool_param_constraints`** and
+  **`BehaviorContract.max_cost_usd`**. Both declared, both unreferenced
+  in `to_graders()`. Surface area for placeholders that never did
+  anything.
+- **Dead `if False` block + hardcoded `v0.1.0` strings** in
+  `ReportGenerator.render_html`. Version string now sourced from
+  `tracelens._version.__version__`.
+
+### Fixed
+
+- **Litellm-dependent tests FAIL vs. SKIP.** Guarded behind a
+  `pytest.mark.skipif(find_spec("litellm") is None)` marker so the
+  core-only CI job passes cleanly without the optional dep. Now
+  obsolete after the LiteLLMProvider removal, but preserved the
+  pattern for any future optional-dep-gated tests.
+- **Degenerate regression samples no longer emit SciPy precision-loss
+  warnings.** Zero-variance current samples now short-circuit the
+  one-sample t-test and preserve the same significance outcome without
+  leaking runtime warnings during Python 3.13 test runs.
+- **Core installs no longer require the `[http]` extra at import time.**
+  `HTTPAPIAdapter` still raises a targeted install hint when used
+  without `httpx`, but `import tracelens` and `tracelens --help` now work
+  from a base wheel install.
+
+### Notes on backward compatibility
+
+This is a `0.x` release, so breaking changes are allowed per the
+version scheme noted at the top. The visible breaks are:
+
+- `from tracelens import WorkflowTask` (and siblings) will now raise
+  `ImportError`. No known downstream user imports these, but flagging
+  the break in case you had one.
+- `from tracelens.llm.litellm_provider import LiteLLMProvider` is gone;
+  write a `LLMProvider` subclass against your vendor SDK instead.
+  The pattern is in the `tracelens.llm.factory` module docstring.
+- `BehaviorContract(tool_param_constraints=...)` or `max_cost_usd=...`
+  will now raise `pydantic.ValidationError`. Neither had any effect
+  before, so removing the kwarg is a no-op for real usage.
+
+## [0.1.0] - Initial alpha
+
+Initial scaffold shipped prior to this branch — `Task`, `Trial`,
+`Grader`, `Transcript`, `AgentAdapter`, `EvaluationRunner`,
+`BaselineManager`, `RegressionDetector`, `ReportGenerator`,
+`DecisionSpec`, pass@k / pass^k / bootstrap CI statistics,
+`tracelens run` CLI, etc. See git history before branch point for
+the full list.