aimeval 0.6.0__tar.gz

aimeval-0.6.0/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.4
+Name: aimeval
+Version: 0.6.0
+Summary: AIMEval Python SDK — Vision-Text Evaluation Platform
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.25
+Requires-Dist: pydantic>=2.0
+Requires-Dist: tqdm>=4.65
+Provides-Extra: cli
+Requires-Dist: typer>=0.12; extra == "cli"
+Requires-Dist: rich>=13.7; extra == "cli"
+Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "cli"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: pytest-httpx>=0.30; extra == "dev"
+Requires-Dist: pyyaml>=6; extra == "dev"
+Provides-Extra: otel
+Requires-Dist: opentelemetry-api>=1.20; extra == "otel"
+Requires-Dist: opentelemetry-sdk>=1.20; extra == "otel"

aimeval-0.6.0/README.md

@@ -0,0 +1,330 @@
+# aimeval — AIMEval Python SDK
+
+The Python SDK + CLI for [AIMEval](https://aimeval.com) — the
+vision-language evaluation platform for e-commerce product listings.
+
+```python
+import aimeval
+from aimeval.metrics import VisualFaithfulness, Hallucination
+
+# One-call CI gate. Returns when the run finishes; assert_test raises
+# (with the per-metric breakdown) if any scorer fell below threshold.
+result = aimeval.Eval(
+    name="nightly",
+    model="m_42", dataset="d_77", metrics="c_5", gate="g_strict",
+    scorers=[
+        VisualFaithfulness(threshold=0.80),
+        Hallucination(threshold=0.10),     # ← lower is better, SDK knows
+    ],
+)
+aimeval.assert_test(result)
+```
+
+> **The IDs above are placeholders.** `m_42` / `d_77` / `c_5` / `g_strict`
+> stand in for real IDs — on AIMEval every object ID is a UUID
+> (`a1b2c3d4-…`). Get yours from the dashboard, the CLI
+> (`aimeval model list` · `dataset list` · `collection list` · `gate list`),
+> or in code:
+>
+> ```python
+> from aimeval import AIMEval
+> c = AIMEval()
+> model_id   = c.models.list().data[0].id        # '9133edaa-f0d4-46c1-…'
+> dataset_id = c.datasets.list().data[0].id
+> coll_id    = c.collections.list().data[0].id
+> ```
+
+## Install
+
+```bash
+pip install aimeval               # base: typed responses + async + retry
+pip install 'aimeval[cli]'        # adds the `aimeval` CLI + Rich panels
+pip install 'aimeval[otel]'       # OpenTelemetry export for @aimeval.trace
+```
+
+Python 3.10+.
+
+## 60-second quickstart
+
+```bash
+pip install 'aimeval[cli]'
+
+aimeval init my-project --github-action
+cd my-project
+cp .env.example .env && $EDITOR .env       # AIMEVAL_API_KEY=…
+aimeval doctor                              # verify env + auth + resources
+$EDITOR evals/ci.py                         # plug in your real IDs
+python evals/ci.py
+```
+
+Done. The scaffolded `ci.py` exits 0 on pass, 2 on review, 1 on fail — the
+GitHub Action wires those into PR checks automatically.
+
+## What's in the box
+
+| Surface | What you get |
+|---|---|
+| **`client.runs / models / datasets / gates / collections / prompts`** | Full CRUD + lifecycle for every product object |
+| **`client.runs.create(…, gate=, prompt=, baseline_run=)`** + **`apply_gate`** / **`promote`** / **`unpromote`** / **`export_results`** | Wires the v7 evaluation flow end-to-end |
+| **`aimeval.Eval(name, model, dataset, metrics, scorers=[...])`** + **`assert_test`** | One-call CI harness with direction-aware threshold checking |
+| **`aimeval.metrics`** (15 typed metric handles) | `VisualFaithfulness`, `Hallucination`, `BrandSupport`, `PIIExposure`, … — each encodes its direction so `passed(score)` can't be inverted |
+| **`@aimeval.trace`** + **`aimeval.span(…)`** + **`TraceCollector`** | Local span capture, parent/child nesting, sync/async/gen; no fake trace endpoint |
+| **`aimeval.save_trace`** / **`load_trace`** / **`replay_trace`** | JSONL persistence + replay through a candidate model (prod-to-eval) |
+| **`aimeval.enable_otel()`** | Mirror spans into Datadog / Honeycomb / Phoenix / Langfuse via OpenInference convention |
+| **`client.webhooks.*`** + **`aimeval.verify_webhook(...)`** | CRUD + HMAC-SHA256 verifier (Stripe-style: constant-time + 5-min replay window) |
+| **`client.runs.estimate_cost(model=, samples=)`** | Honest cost preview from your project's own run history — no stale price tables |
+| **`client.regression_sets.from_run(run_id, n=20)`** | Pin the worst-scoring samples for re-test on every commit |
+| **`client.audit_log()`** / **`client.usage()`** | Settings observability (compliance + quota dashboards) |
+| **`client.beta.replay.*`** | Experimental surface (OpenAI/Anthropic pattern; pin SDK + read CHANGELOG before bumping) |
+| **`client.compare`** / **`compare_history`** / **`search`** / **`metrics_catalog`** / **`wizard_bootstrap`** | Top-level convenience methods |
+| **CLI** — `aimeval init / doctor / evaluate / run / annotations / gate / model / search / compare` | Every workflow scriptable from the terminal |
+
+Every async resource has a sync twin on `AIMEval`, and vice versa on `AsyncAIMEval`.
+
+## Configure
+
+The SDK reads from the environment by default:
+
+```bash
+export AIMEVAL_API_KEY="aime_sk_..."
+export AIMEVAL_BASE_URL="https://app.aimeval.com"
+
+# Optional:
+export AIMEVAL_LOG=info                  # logfmt request logs on stderr
+export AIMEVAL_OTEL=1                    # mirror @trace spans to OTel
+export AIMEVAL_WEBHOOK_SECRET=...        # for verify_webhook
+export AIMEVAL_MODEL_AUTH=...            # Custom Endpoint test_connection
+```
+
+Production-shaped API keys (`aime_sk_live_*`) passed inline (instead of via
+env var) raise `AIMEvalSecurityWarning` so a leaked key surfaces before it
+lands in shell history or a committed notebook. Test/sandbox keys are
+silent.
+
+`api_key` is **masked** in `__repr__`, in error messages, in CLI output,
+and in the dev logger handler.
+
+## Selected recipes
+
+### CI gate that blocks merges on regression
+
+```python
+# evals/ci.py — scaffolded by `aimeval init`
+import aimeval
+from aimeval.metrics import VisualFaithfulness, Hallucination, PIIExposure
+
+result = aimeval.Eval(
+    name=f"ci-{os.environ['AIMEVAL_CI_COMMIT'][:7]}",
+    model="m_gpt4o", dataset="d_amazon", metrics="c_standard",
+    gate="g_strict",
+    idempotency_key=os.environ["AIMEVAL_CI_COMMIT"],  # auto-generated if omitted
+    scorers=[
+        VisualFaithfulness(threshold=0.80),
+        Hallucination(threshold=0.10),
+        PIIExposure(threshold=0.05),
+    ],
+)
+aimeval.assert_test(result)
+```
+
+Pair with the bundled GitHub Action:
+
+```yaml
+# .github/workflows/aimeval-ci.yml
+- uses: rybalena/aimeval-backend/.github/actions/aimeval-eval@main
+  with:
+    script: ./evals/ci.py
+    api-key: ${{ secrets.AIMEVAL_API_KEY }}
+    aimeval-version: "0.6.0"
+```
+
+### pytest plugin (auto-loaded via the `pytest11` entry point)
+
+```python
+@pytest.mark.aimeval_eval
+def test_amazon_nightly():
+    result = aimeval.Eval(name="t", model="m", dataset="d", metrics="c",
+                          scorers=[VisualFaithfulness(threshold=0.8)])
+    aimeval.assert_test(result)
+```
+
+Failures render a dedicated "AIMEval eval failure" section with the
+per-metric breakdown — no traceback noise.
+
+### Instrument production, replay as an eval
+
+```python
+@aimeval.trace
+def describe(image_url: str) -> str:
+    return my_vlm(image_url)
+
+# In prod: capture every call
+with aimeval.TraceCollector() as spans:
+    for url in queue:
+        describe(url)
+aimeval.save_trace(spans, "captures/2026-06-07.jsonl")
+
+# Later: replay through a candidate model
+results = aimeval.replay_trace(
+    "captures/2026-06-07.jsonl",
+    task=lambda inp: candidate_vlm(inp["image_url"]),
+)
+```
+
+### OpenTelemetry mirror — Datadog / Honeycomb / Phoenix / Langfuse
+
+```python
+aimeval.enable_otel()       # or AIMEVAL_OTEL=1
+# Spans now also go to whichever OTLP collector OTEL_EXPORTER_OTLP_* points at.
+# Attributes follow the OpenInference semantic convention so Phoenix /
+# Langfuse / Patronus / LangSmith / Opik / Weave all consume them natively.
+```
+
+### Webhook signature verification
+
+```python
+from aimeval import verify_webhook, WebhookVerificationError
+
+# In your FastAPI / Flask handler:
+try:
+    verify_webhook(
+        secret=os.environ["AIMEVAL_WEBHOOK_SECRET"],
+        body=request.body,                                  # raw bytes!
+        signature=request.headers["X-AIMEval-Signature"],
+        timestamp=request.headers["X-AIMEval-Timestamp"],
+    )
+except WebhookVerificationError as exc:
+    return Response(str(exc), status=400)
+```
+
+Constant-time HMAC + 5-minute default tolerance window. Same protections
+Stripe / GitHub ship.
+
+### Honest cost preview (no stale vendor price tables)
+
+```python
+est = client.runs.estimate_cost(model="m_gpt4o", samples=5000)
+print(est)
+# CostEstimate(model='m_gpt4o', samples=5000, mean=$90.00,
+#              range=$75.00–$110.00, based_on_runs=8)
+```
+
+Averages `cost_usd / samples_processed` across the project's most recent
+completed runs of the given model. With no history yet: returns
+`known=False` plus a note ("run sample_limit=5 first") instead of
+fabricating a number.
+
+### Enterprise: corporate proxy / mTLS
+
+```python
+import httpx
+proxied = httpx.Client(
+    proxies="http://corp-proxy:8080",
+    verify="/etc/corp/ca.pem",
+)
+client = AIMEval(api_key="...", base_url="...", http_client=proxied)
+```
+
+The SDK installs auth + UA + `base_url` **on top** of your client — no
+second client is constructed behind the scenes.
+
+### Per-call overrides + raw response
+
+```python
+# OpenAI/Anthropic with_options pattern — immutable copy, never mutates self.
+slow_run = client.with_options(timeout=300, max_retries=10).runs.create(...)
+
+# Read request_id + headers without losing the typed surface:
+resp = client.with_raw_response.runs.retrieve("run_abc")
+print(resp.headers["x-request-id"])
+run = resp.parse()       # same dict a normal call would return
+```
+
+Every error also carries `exc.request_id` + `exc.response` so support
+tickets are diagnosable.
+
+## CLI
+
+```bash
+aimeval --install-completion bash         # shell tab-completion
+
+aimeval init my-project [--github-action] # scaffold evals/ + .env.example
+aimeval doctor                             # env + DNS + auth + resources
+aimeval evaluate --name x --model m_… --dataset d_… --collection c_…
+
+aimeval run apply-gate / promote / unpromote / report / export-results
+aimeval run estimate-cost --model M --samples 1000
+
+aimeval annotations bootstrap / list / label / flag
+aimeval gate duplicate / check-compatibility / metric-registry
+aimeval model test-connection --model M           # AIMEVAL_MODEL_AUTH from env
+
+aimeval search "amazon-nightly"
+aimeval compare run_baseline run_candidate
+```
+
+Global flags: `-o json|table|markdown`, `-q` (quiet), `-p PROFILE`,
+`--api-key`, `--base-url`.
+
+## Async
+
+Every method has an `async` twin:
+
+```python
+import asyncio
+from aimeval import AsyncAIMEval
+
+async def main():
+    async with AsyncAIMEval() as client:
+        results = await asyncio.gather(*[
+            client.runs.estimate_cost(model="m1", samples=n) for n in (100, 1000, 10000)
+        ])
+        for r in results:
+            print(r)
+
+asyncio.run(main())
+```
+
+## Debugging — `AIMEVAL_LOG`
+
+```bash
+$ AIMEVAL_LOG=info python evals/ci.py
+07:31:39.575 aimeval INFO op=http method=GET path=/runs/page \
+  status=200 request_id=req_abc duration_ms=42
+07:31:40.211 aimeval WARNING op=http method=POST path=/runs \
+  status=429 request_id=req_def attempt=1
+```
+
+logfmt — `grep request_id=req_…` works straight from `cat`/`less` without
+JSON parsing. Promotes to WARNING on 4xx/5xx and on every retry attempt.
+Zero hot-path cost when the env var is unset.
+
+## Error taxonomy
+
+```python
+from aimeval import (
+    AIMEvalError, APIConnectionError, APITimeoutError,
+    AuthenticationError, PermissionError_, NotFoundError, ConflictError,
+    BadRequestError, RateLimitError, APIServerError,
+    WebhookVerificationError, EvalAssertionError,
+)
+```
+
+Each maps to an HTTP status range. Retries (408 / 429 / 502 / 503 / 504)
+are automatic with exponential backoff + `Retry-After` honoured. Every
+error carries `exc.request_id` + `exc.response`.
+
+## Versioning
+
+This SDK follows the backend's API contract. Minor versions
+(`0.5.x → 0.6.0`) introduce new endpoint wrappers and ergonomic surface;
+breaking changes only happen at `0.x → 1.0`. `client.beta.*` is exempt —
+that's the unstable surface by contract.
+
+Current: **v0.6.0** — see [CHANGELOG.md](CHANGELOG.md).
+
+## More
+
+- Recipes: [EXAMPLES.md](EXAMPLES.md)
+- Backend API reference: https://app.aimeval.com/api/docs
+- Source: https://github.com/rybalena/aimeval-backend

aimeval-0.6.0/aimeval/__init__.py

@@ -0,0 +1,191 @@
+"""
+AIMEval Python SDK — programmatic access for CI/CD pipelines.
+
+Quick start::
+
+    from aimeval import AIMEval
+
+    client = AIMEval(api_key="aime_sk_...", base_url="https://app.aimeval.com")
+
+    # End-to-end workflow with a live progress bar (TTY only):
+    run = client.evaluate(
+        name="nightly-regression",
+        model=MODEL_ID,
+        dataset=DATASET_ID,
+        metrics=COLLECTION_ID,
+    )
+    sys.exit(0 if run.passed else 1)
+
+    # Or step by step (industry-standard resource namespaces):
+    run = client.runs.create(name="...", model=..., dataset=..., metrics=...)
+    run = client.runs.wait(run.id, progress=True)
+
+    # Auto-pagination — iterate every page transparently:
+    for run in client.runs.list(status="completed"):
+        print(run.id, run.score)
+"""
+from aimeval.client import AIMEval, AsyncAIMEval
+from aimeval._otel import disable_otel, enable_otel, is_available as otel_is_available, otel_enabled
+from aimeval import events
+from aimeval._trace import Span, TraceCollector, records, span, trace
+from aimeval._trace_io import (
+    ReplayResult,
+    load_trace,
+    load_trace_iter,
+    replay_trace,
+    save_trace,
+)
+from aimeval._webhook_verify import (
+    WebhookVerificationError,
+    construct_event,
+    verify_webhook,
+)
+from aimeval._eval import (
+    Eval,
+    EvalAssertionError,
+    EvalResult,
+    EvalScore,
+    assert_test,
+    eval_from_run,
+)
+from aimeval import metrics
+from aimeval._exceptions import (
+    AIMEvalError,
+    AIMEvalSecurityWarning,
+    APIConnectionError,
+    APIServerError,
+    APITimeoutError,
+    AuthenticationError,
+    BadRequestError,
+    ConflictError,
+    NotFoundError,
+    RateLimitError,
+)
+from aimeval._streaming import (
+    AsyncRunSSEStream,
+    AsyncRunStream,
+    RunEvent,
+    RunSSEStream,
+    RunStream,
+)
+from aimeval._types import (
+    Annotation,
+    AnnotationsBootstrapResult,
+    AuditEntry,
+    CompareHistoryItem,
+    Comparison,
+    ConnectionTestResult,
+    CostEstimate,
+    Dataset,
+    GateCompatibility,
+    MetricCollection,
+    MetricDistribution,
+    MetricRegistry,
+    MetricRegistryEntry,
+    MetricSummary,
+    Model,
+    Prompt,
+    PromptDiff,
+    PromptTestResult,
+    PromptTestRun,
+    PromptVersion,
+    QualityGate,
+    RegressionSet,
+    Resource,
+    Run,
+    SearchHit,
+    SearchResults,
+    UploadStatus,
+    UploadTicket,
+    Usage,
+    Webhook,
+    WizardBootstrap,
+)
+
+
+__version__ = "0.6.0"
+
+__all__ = [
+    # Clients
+    "AIMEval",
+    "AsyncAIMEval",
+    "__version__",
+    # Tracing (local span capture → eval datasets; optional OTel mirror)
+    "trace",
+    "span",
+    "TraceCollector",
+    "Span",
+    "records",
+    "events",
+    # Trace recording + replay (prod-to-eval pattern)
+    "save_trace",
+    "load_trace",
+    "load_trace_iter",
+    "replay_trace",
+    "ReplayResult",
+    "enable_otel",
+    "disable_otel",
+    "otel_enabled",
+    "otel_is_available",
+    # Eval harness + metrics (CI assertion surface)
+    "Eval",
+    "eval_from_run",
+    "assert_test",
+    "EvalResult",
+    "EvalScore",
+    "EvalAssertionError",
+    "metrics",
+    # Errors
+    "AIMEvalError",
+    "AIMEvalSecurityWarning",
+    "APIConnectionError",
+    "APIServerError",
+    "APITimeoutError",
+    "AuthenticationError",
+    "BadRequestError",
+    "ConflictError",
+    "NotFoundError",
+    "RateLimitError",
+    # Response types
+    "Annotation",
+    "AnnotationsBootstrapResult",
+    "AuditEntry",
+    "CompareHistoryItem",
+    "Comparison",
+    "ConnectionTestResult",
+    "CostEstimate",
+    "Dataset",
+    "GateCompatibility",
+    "MetricCollection",
+    "MetricDistribution",
+    "MetricRegistry",
+    "MetricRegistryEntry",
+    "MetricSummary",
+    "Model",
+    "Prompt",
+    "PromptDiff",
+    "PromptTestResult",
+    "PromptTestRun",
+    "PromptVersion",
+    "QualityGate",
+    "RegressionSet",
+    "Resource",
+    "Run",
+    "SearchHit",
+    "SearchResults",
+    "UploadStatus",
+    "UploadTicket",
+    "Usage",
+    "Webhook",
+    "WizardBootstrap",
+    # Webhook verifier (security helper)
+    "WebhookVerificationError",
+    "construct_event",
+    "verify_webhook",
+    # Streaming
+    "AsyncRunSSEStream",
+    "AsyncRunStream",
+    "RunEvent",
+    "RunSSEStream",
+    "RunStream",
+]

aimeval-0.6.0/aimeval/_beta.py

@@ -0,0 +1,61 @@
+"""``client.beta.*`` — experimental surface area.
+
+Industry pattern from OpenAI / Anthropic: features that haven't earned
+SemVer protection yet live under a dedicated ``beta`` namespace. This
+gives them three properties at once:
+
+  1. **Visible**: a glance at ``client.beta.<something>`` tells the
+     reader "this isn't stable yet — pin your SDK and read the
+     CHANGELOG before bumping".
+  2. **Safe to remove / rename** without bumping the major version of
+     the SDK. The ``beta`` package itself is the contract.
+  3. **Cheap to ship**: graduation to the stable surface (e.g.
+     ``client.replay``) is a one-line re-export when the API settles.
+
+Currently in beta:
+
+  - :attr:`Beta.replay` — trace recording + replay helpers
+    (:func:`aimeval.save_trace` / :func:`aimeval.load_trace` /
+    :func:`aimeval.replay_trace`). Wrapped under ``client.beta.replay``
+    so the helpers feel discoverable from autocomplete even when
+    callers don't import them directly.
+"""
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from aimeval import _trace_io
+
+if TYPE_CHECKING:  # pragma: no cover
+    from aimeval.client import AIMEval, AsyncAIMEval
+
+
+class _Replay:
+    """Beta wrapper around :mod:`aimeval._trace_io`.
+
+    Stays opt-in via ``client.beta.replay.save_trace(...)`` for the
+    same reason every premium SDK quarantines unstable surface:
+    discoverable via autocomplete, but the rename / removal is on
+    a separate timeline from the stable client.
+    """
+
+    def __init__(self, _client: "AIMEval | AsyncAIMEval"):
+        # Client kept on the instance only so a future beta feature
+        # that *does* hit the network has something to call. The replay
+        # helpers themselves are pure offline.
+        self._client = _client
+
+    save_trace = staticmethod(_trace_io.save_trace)
+    load_trace = staticmethod(_trace_io.load_trace)
+    load_trace_iter = staticmethod(_trace_io.load_trace_iter)
+    replay_trace = staticmethod(_trace_io.replay_trace)
+
+
+class Beta:
+    """Namespace for experimental features. See module docstring."""
+
+    def __init__(self, client: "AIMEval | AsyncAIMEval"):
+        self.replay = _Replay(client)
+
+
+__all__ = ["Beta"]