spanforge 2.0.1__tar.gz → 2.0.3__tar.gz

{spanforge-2.0.1 → spanforge-2.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spanforge
-Version: 2.0.1
+Version: 2.0.3
 Summary: SpanForge — AI lifecycle and governance platform (RFC-0001 SPANFORGE)
 Project-URL: Homepage, https://github.com/veerarag1973/spanforge
 Project-URL: Documentation, https://github.com/veerarag1973/spanforge/blob/main/docs/index.md

spanforge-2.0.3/docs/api/config.md

@@ -0,0 +1,119 @@
+# spanforge.config — Global configuration
+
+> **Module:** `spanforge.config`
+
+`spanforge.config` provides the global configuration singleton, the
+`configure()` entry point, and environment-variable interpolation utilities.
+
+---
+
+## Quick example
+
+```python
+from spanforge import configure
+from spanforge.config import get_config
+
+configure(
+    preset="production",
+    exporter="otlp",
+    endpoint="http://collector:4318",
+)
+cfg = get_config()
+print(cfg.exporter)  # "otlp"
+```
+
+---
+
+## API
+
+### `SpanForgeConfig`
+
+Mutable dataclass holding all SDK configuration fields.  Key fields:
+
+| Field | Default | Env var | Description |
+|-------|---------|---------|-------------|
+| `exporter` | `"console"` | `SPANFORGE_EXPORTER` | Backend: `console`, `jsonl`, `otlp`, `webhook`, `datadog`, `grafana_loki` |
+| `endpoint` | `None` | `SPANFORGE_ENDPOINT` | Exporter destination (file path or URL) |
+| `org_id` | `None` | `SPANFORGE_ORG_ID` | Organisation identifier |
+| `service_name` | `"unknown-service"` | `SPANFORGE_SERVICE_NAME` | Human-readable service name |
+| `env` | `"production"` | `SPANFORGE_ENV` | Deployment environment tag |
+| `service_version` | `"0.0.0"` | `SPANFORGE_SERVICE_VERSION` | SemVer string |
+| `signing_key` | `None` | `SPANFORGE_SIGNING_KEY` | Base64-encoded HMAC key |
+| `sample_rate` | `1.0` | `SPANFORGE_SAMPLE_RATE` | Fraction of traces to emit (0.0–1.0) |
+| `on_export_error` | `"warn"` | `SPANFORGE_ON_EXPORT_ERROR` | `"warn"` / `"raise"` / `"drop"` |
+| `enable_trace_store` | `False` | `SPANFORGE_ENABLE_TRACE_STORE` | In-process ring buffer |
+| `no_egress` | `False` | `SPANFORGE_NO_EGRESS` | Block all network exporters |
+| `compliance_sampling` | `True` | `SPANFORGE_COMPLIANCE_SAMPLING` | Always-record compliance events |
+| `consent_enforcement` | `False` | `SPANFORGE_CONSENT_ENFORCEMENT` | T.R.U.S.T. consent checks |
+| `hitl_enabled` | `False` | `SPANFORGE_HITL_ENABLED` | Human-in-the-loop review queue |
+
+See the source docstring for the full list of fields.
+
+---
+
+### `configure(**kwargs)`
+
+```python
+def configure(**kwargs: Any) -> None: ...
+```
+
+Mutate the global `SpanForgeConfig` singleton.  Accepts any field name as a
+keyword argument.  Unknown keys raise `ValueError`.
+
+**Presets:** Pass `preset="<name>"` to apply a sensible defaults bundle
+before other kwargs:
+
+| Preset | Exporter | Sample rate | Notes |
+|--------|----------|-------------|-------|
+| `development` | `console` | 1.0 | Trace store on, private endpoints allowed |
+| `testing` | `console` | 1.0 | `on_export_error="raise"` |
+| `staging` | `console` | 0.5 | Always-sample errors |
+| `production` | `otlp` | 0.1 | Batch 512, flush 5 s |
+| `otel_passthrough` | `otel_bridge` | 1.0 | Compliance sampling on |
+
+---
+
+### `get_config()`
+
+```python
+def get_config() -> SpanForgeConfig: ...
+```
+
+Return the live configuration singleton.  Modifications to the returned
+object affect all subsequent SDK operations.
+
+---
+
+### `interpolate_env()`
+
+> **Added in:** 2.0.3
+
+```python
+def interpolate_env(data: Any) -> Any: ...
+```
+
+Recursively replace `${VAR}` and `${VAR:default}` patterns in *data*.
+
+Walks strings, dicts (values only), and lists depth-first.  Non-string
+leaves are returned unchanged.
+
+| Pattern | Behaviour |
+|---------|-----------|
+| `${FOO}` | Replaced with `os.environ["FOO"]`; left as-is if unset. |
+| `${FOO:bar}` | Replaced with `os.environ["FOO"]` when set, `"bar"` otherwise. |
+
+**Example:**
+
+```python
+import os
+from spanforge.config import interpolate_env
+
+os.environ["MODEL"] = "gpt-4o"
+result = interpolate_env({
+    "model": "${MODEL}",
+    "endpoint": "${ENDPOINT:https://api.openai.com/v1}",
+})
+# {"model": "gpt-4o", "endpoint": "https://api.openai.com/v1"}
+```
+
+> **Note:** Dict keys are **not** interpolated — only values.

spanforge-2.0.3/docs/api/eval.md

@@ -0,0 +1,211 @@
+# spanforge.eval — Evaluation framework
+
+> **Module:** `spanforge.eval`
+
+`spanforge.eval` provides lightweight instrumentation for attaching quality
+scores to active spans and emitting them as RFC-0001 `llm.eval.*` events.
+It ships built-in scorers, a batch runner, a mean-based regression detector,
+and a plug-in scorer ABC for the entry-point ecosystem.
+
+---
+
+## Quick example
+
+```python
+from spanforge.eval import record_eval_score, EvalScore
+
+score = record_eval_score(
+    metric="faithfulness",
+    value=0.87,
+    span_id="abcdef0123456789",
+    trace_id="abcdef0123456789abcdef0123456789",
+    label="pass",
+    metadata={"evaluator": "ragas"},
+)
+```
+
+---
+
+## API
+
+### `record_eval_score()`
+
+```python
+def record_eval_score(
+    metric: str,
+    value: float,
+    *,
+    span_id: str | None = None,
+    trace_id: str | None = None,
+    label: str | None = None,
+    metadata: dict[str, Any] | None = None,
+) -> EvalScore: ...
+```
+
+Record a score and emit it as an `llm.eval.score.recorded` event via the
+configured SpanForge exporter.
+
+| Parameter | Description |
+|-----------|-------------|
+| `metric` | Name of the quality metric (e.g. `"faithfulness"`). |
+| `value` | Numeric score value (any float). |
+| `span_id` | Optional parent span ID (16 lowercase hex chars). |
+| `trace_id` | Optional trace ID (32 lowercase hex chars). |
+| `label` | Optional human-readable label (`"pass"` / `"fail"` / etc.). |
+| `metadata` | Optional free-form dict with evaluator details. |
+
+---
+
+### `EvalScore`
+
+```python
+@dataclass
+class EvalScore:
+    metric: str
+    value: float
+    span_id: str | None = None
+    trace_id: str | None = None
+    label: str | None = None
+    metadata: dict[str, Any] | None = None
+    timestamp: float = ...  # auto-set
+```
+
+A single quality measurement attached to a span or agent run.
+
+| Method | Description |
+|--------|-------------|
+| `to_dict()` | Serialise to a plain dict. |
+| `from_dict(data)` | Class method — deserialise from a dict. |
+
+---
+
+### `EvalScorer` (Protocol)
+
+```python
+@runtime_checkable
+class EvalScorer(Protocol):
+    @property
+    def metric_name(self) -> str: ...
+    def score(self, example: dict[str, Any]) -> EvalScore: ...
+```
+
+Protocol for scorers compatible with `EvalRunner`.  Each scorer receives a
+single example dict (with at least an `"output"` key) and returns an
+`EvalScore`.
+
+---
+
+### `EvalRunner`
+
+```python
+class EvalRunner:
+    def __init__(
+        self,
+        scorers: list[EvalScorer] | None = None,
+        *,
+        emit: bool = True,
+    ) -> None: ...
+
+    def add_scorer(self, scorer: EvalScorer) -> None: ...
+    def run(self, dataset: list[dict[str, Any]]) -> EvalReport: ...
+```
+
+Run one or more scorers over a dataset.  When `emit=True` (default) each
+score is also emitted via `record_eval_score()`.
+
+---
+
+### `EvalReport`
+
+```python
+@dataclass
+class EvalReport:
+    scores: list[EvalScore]
+    dataset: list[dict[str, Any]]
+```
+
+| Method | Description |
+|--------|-------------|
+| `summary()` | Return `{metric: mean_value}` dict. |
+| `print_summary()` | Print a human-readable table to stdout. |
+
+---
+
+### `RegressionDetector`
+
+```python
+class RegressionDetector:
+    def __init__(
+        self,
+        baseline: dict[str, float] | None = None,
+        *,
+        threshold_pct: float = 5.0,
+        emit: bool = True,
+    ) -> None: ...
+
+    def set_baseline(self, metric: str, value: float) -> None: ...
+    def check(self, report: EvalReport) -> list[dict[str, Any]]: ...
+```
+
+Mean-based regression detection.  When the mean score for a metric drops
+below `baseline_mean * (1 - threshold_pct / 100)`, an
+`llm.eval.regression.detected` event is emitted.
+
+> **Note:** For per-case pass/fail regression detection, see
+> [`spanforge.regression`](regression.md).
+
+---
+
+## Built-in scorers
+
+| Scorer | `metric_name` | Description |
+|--------|---------------|-------------|
+| `FaithfulnessScorer` | `faithfulness` | Token-overlap proxy between `output` and `context`. |
+| `RefusalDetectionScorer` | `refusal_detection` | Matches common refusal phrases (returns 1.0 on refusal). |
+| `PIILeakageScorer` | `pii_leakage` | Delegates to `spanforge.redact.scan_payload()`; returns 1.0 on PII detection. |
+
+---
+
+### `BehaviourScorer` (ABC)
+
+> **Added in:** 2.0.3
+
+```python
+class BehaviourScorer(ABC):
+    name: str = "base"
+
+    @abstractmethod
+    def score(self, case: Any, response: str) -> tuple[float, str]: ...
+```
+
+Abstract base class for plug-in behaviour scorers registered via the
+`spanforge.scorers` entry-point group.  Unlike `EvalScorer` (which scores
+full `dict` examples), `BehaviourScorer` targets named test-case workflows
+where the scorer receives a structured test case object and the raw model
+response.
+
+| Attribute / Method | Description |
+|--------------------|-------------|
+| `name` | Unique identifier for the scorer (override in subclasses). |
+| `score(case, response)` | Return `(score, reason)` where score ∈ [0.0, 1.0]. |
+
+**Registration:**
+
+```toml
+[project.entry-points."spanforge.scorers"]
+toxicity = "my_package.scorers:ToxicityScorer"
+```
+
+**Example:**
+
+```python
+from spanforge.eval import BehaviourScorer
+
+class ToxicityScorer(BehaviourScorer):
+    name = "toxicity"
+
+    def score(self, case, response: str) -> tuple[float, str]:
+        if any(w in response.lower() for w in ("hate", "kill")):
+            return 0.0, "toxic content detected"
+        return 1.0, "no toxicity detected"
+```

spanforge-2.0.3/docs/api/http.md

@@ -0,0 +1,108 @@
+# spanforge.http — OpenAI-compatible HTTP client
+
+> **Module:** `spanforge.http`  
+> **Added in:** 2.0.3
+
+`spanforge.http` provides a zero-dependency, synchronous HTTP client for
+OpenAI-compatible chat-completion APIs.  It is intentionally minimal — no
+`httpx`, no `requests`, no `aiohttp` — just the stdlib `urllib.request`.
+
+---
+
+## Quick example
+
+```python
+from spanforge.http import chat_completion
+
+resp = chat_completion(
+    endpoint="https://api.openai.com/v1",
+    model="gpt-4o",
+    messages=[{"role": "user", "content": "What is the capital of France?"}],
+    api_key="sk-...",
+)
+
+if resp.ok:
+    print(resp.text)          # "Paris"
+    print(resp.total_tokens)  # 23
+else:
+    print("Error:", resp.error)
+```
+
+---
+
+## API
+
+### `chat_completion()`
+
+```python
+def chat_completion(
+    endpoint: str,
+    model: str,
+    messages: list[dict],
+    *,
+    api_key: str | None = None,
+    timeout: float = 30.0,
+    max_retries: int = 3,
+    extra_body: dict | None = None,
+) -> ChatCompletionResponse: ...
+```
+
+Send a single chat-completion request and return a `ChatCompletionResponse`.
+
+| Parameter | Description |
+|-----------|-------------|
+| `endpoint` | Base URL, e.g. `"https://api.openai.com/v1"`. The path `/chat/completions` is appended automatically. |
+| `model` | Model name, e.g. `"gpt-4o"`. |
+| `messages` | List of chat message dicts (`role` + `content`). |
+| `api_key` | Bearer token. Falls back to the `OPENAI_API_KEY` environment variable. |
+| `timeout` | Socket timeout in seconds (default 30). |
+| `max_retries` | Maximum number of retry attempts on `429 / 5xx` or network errors. Exponential back-off: `min(2**attempt, 8)` seconds. Default 3. |
+| `extra_body` | Additional fields merged into the request JSON body (e.g. `temperature`, `stream`). |
+
+**Returns:** `ChatCompletionResponse`
+
+---
+
+### `ChatCompletionResponse`
+
+```python
+@dataclass(frozen=True)
+class ChatCompletionResponse:
+    text: str
+    latency_ms: float
+    error: str | None = None
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    total_tokens: int = 0
+```
+
+| Field | Description |
+|-------|-------------|
+| `text` | The generated text from the first choice, or `""` on error. |
+| `latency_ms` | Wall-clock round-trip time in milliseconds. |
+| `error` | Error description string, or `None` on success. |
+| `prompt_tokens` | Tokens used in the prompt (from the `usage` field). |
+| `completion_tokens` | Tokens generated in the completion. |
+| `total_tokens` | `prompt_tokens + completion_tokens`. |
+| `ok` *(property)* | `True` when `error is None`. |
+
+---
+
+## Retry behaviour
+
+Retries are performed on:
+- HTTP status codes `429`, `500`, `502`, `503`, `504`
+- `urllib.error.URLError` (network-level failures)
+- `OSError` (connection refused, timeouts)
+
+Back-off delay: `min(2**attempt, 8)` seconds (max 8 s).  Non-retryable
+status codes (e.g. `400`, `401`, `404`) are returned immediately as an
+error response.
+
+---
+
+## Environment variables
+
+| Variable | Effect |
+|----------|--------|
+| `OPENAI_API_KEY` | Default API key when `api_key` is not supplied. |

{spanforge-2.0.1 → spanforge-2.0.3}/docs/api/index.md

@@ -31,6 +31,14 @@ exported at the top-level package under `spanforge`.
 - [models](models.md)
 - [cache](cache.md)
 - [lint](lint.md)
+- [eval](eval.md)
+- [config](config.md)
+- [http](http.md)
+- [io](io.md)
+- [plugins](plugins.md)
+- [schema](schema.md)
+- [regression](regression.md)
+- [stats](stats.md)
 
 ## Module summary
 
@@ -66,3 +74,12 @@ exported at the top-level package under `spanforge`.
 | `spanforge.model_registry` | `ModelRegistryEntry`, model governance lifecycle (`registered` / `deprecated` / `retired`), attestation integration |
 | `spanforge.explain` | `ExplainabilityRecord`, decision explainability (`generated`), EU AI Act Art. 13 / NIST MAP 1.1 mapping |
 | `spanforge.lint` | `run_checks()`, `LintError`, AO001–AO005 checks, `SpanForgeChecker` flake8 plugin, `python -m spanforge.lint` CLI |
+| `spanforge.eval` | `record_eval_score()`, `EvalScore`, `EvalRunner`, `EvalReport`, `RegressionDetector` (mean-based), `BehaviourScorer` ABC, built-in scorers |
+| `spanforge.config` | `SpanForgeConfig`, `configure()`, `get_config()`, `interpolate_env()` — global configuration and env-var interpolation |
+| `spanforge.http` | `chat_completion()`, `ChatCompletionResponse` — zero-dependency OpenAI-compatible HTTP client with exponential-backoff retry |
+| `spanforge.io` | `write_jsonl()`, `read_jsonl()`, `append_jsonl()`, `write_events()`, `read_events()` — synchronous JSONL read/write utilities |
+| `spanforge.plugins` | `discover(group)` — Python-version-aware entry-point plugin discovery (3.9 / 3.10 / 3.12+) |
+| `spanforge.schema` | `validate()`, `validate_strict()`, `SchemaValidationError` — lightweight zero-dependency JSON Schema validator |
+| `spanforge.regression` | `RegressionDetector`, `RegressionReport`, `compare()` — pass/fail and score-drop regression detection |
+| `spanforge.stats` | `percentile()`, `latency_summary()` — latency statistics with linear-interpolation percentiles |
+| `spanforge._ansi` | `color()`, `strip_ansi()`, ANSI color constants — terminal colour helpers with `NO_COLOR` / non-TTY support |

spanforge-2.0.3/docs/api/io.md

@@ -0,0 +1,128 @@
+# spanforge.io — Synchronous JSONL utilities
+
+> **Module:** `spanforge.io`  
+> **Added in:** 2.0.3
+
+`spanforge.io` provides reliable, synchronous JSONL read/write helpers.
+They are a simpler, dependency-free alternative to the async `JSONLExporter`
+and `EventStream.from_file()` pattern — ideal for scripts, tests, and
+offline pipelines.
+
+---
+
+## Quick example
+
+```python
+from spanforge.io import write_events, read_events
+
+# Write eval results as event envelopes
+write_events(
+    [{"case_id": "tc-001", "score": 0.95}, {"case_id": "tc-002", "score": 0.72}],
+    "results.jsonl",
+    event_type="llm.eval.done",
+    source="my-eval-runner@1.0",
+)
+
+# Read them back, filtered by event type
+payloads = read_events("results.jsonl", event_type="llm.eval.done")
+# → [{"case_id": "tc-001", "score": 0.95}, ...]
+```
+
+---
+
+## API
+
+### `write_jsonl()`
+
+```python
+def write_jsonl(
+    records: Iterable[dict],
+    path: str | Path,
+    *,
+    mode: str = "w",
+) -> int: ...
+```
+
+Serialise each record as a JSON line and write to *path*.  Parent directories
+are created automatically.
+
+| Parameter | Description |
+|-----------|-------------|
+| `records` | Any iterable of `dict` objects (including generators). |
+| `path` | Destination file path. |
+| `mode` | `"w"` (overwrite, default) or `"a"` (append). |
+
+**Returns:** Number of records written.
+
+---
+
+### `read_jsonl()`
+
+```python
+def read_jsonl(
+    path: str | Path,
+    *,
+    event_type: str | None = None,
+    skip_errors: bool = True,
+) -> list[dict]: ...
+```
+
+Read a JSONL file and return a list of `dict` records.
+
+| Parameter | Description |
+|-----------|-------------|
+| `path` | Source file path. Raises `FileNotFoundError` if absent. |
+| `event_type` | If set, only records with `"event_type" == event_type` are returned. |
+| `skip_errors` | When `True` (default), malformed lines are silently skipped. Set to `False` to raise `json.JSONDecodeError` on the first bad line. |
+
+Non-`dict` JSON values (arrays, scalars) are always skipped.
+
+---
+
+### `append_jsonl()`
+
+```python
+def append_jsonl(record: dict, path: str | Path) -> None: ...
+```
+
+Append a single record to *path*, creating the file if it does not exist.
+Equivalent to `write_jsonl([record], path, mode="a")`.
+
+---
+
+### `write_events()`
+
+```python
+def write_events(
+    payloads: Iterable[dict],
+    path: str | Path,
+    *,
+    event_type: str,
+    source: str = "spanforge",
+    mode: str = "w",
+) -> int: ...
+```
+
+Wrap each payload in a spanforge event envelope and write to *path*.
+
+Envelope format:
+
+```json
+{"event_type": "<event_type>", "source": "<source>", "payload": { ... }}
+```
+
+---
+
+### `read_events()`
+
+```python
+def read_events(
+    path: str | Path,
+    *,
+    event_type: str,
+) -> list[dict]: ...
+```
+
+Read event envelopes from *path* and return the unwrapped `payload` objects
+where `"event_type"` matches.  Lines that do not carry a `"payload"` field
+are silently skipped.

spanforge-2.0.3/docs/api/plugins.md

@@ -0,0 +1,70 @@
+# spanforge.plugins — Entry-point plugin discovery
+
+> **Module:** `spanforge.plugins`  
+> **Added in:** 2.0.3
+
+`spanforge.plugins` provides a single `discover()` helper that loads
+third-party plugins registered via Python packaging entry-points.  It
+handles the API split across Python 3.9, 3.10, and 3.12+ transparently
+and silently skips broken plugins.
+
+---
+
+## Quick example
+
+```python
+from spanforge.plugins import discover
+
+# Load all scorers registered under the "spanforge.scorers" group
+scorers = discover("spanforge.scorers")
+
+for scorer_cls in scorers:
+    scorer = scorer_cls()
+    print(scorer.name)
+```
+
+---
+
+## API
+
+### `discover()`
+
+```python
+def discover(group: str) -> list[Any]: ...
+```
+
+Return a list of objects (classes, functions, or instances) loaded from all
+entry points registered under *group*.
+
+| Parameter | Description |
+|-----------|-------------|
+| `group` | The entry-point group name, e.g. `"spanforge.scorers"`. |
+
+**Returns:** A list of loaded objects.  Broken entry points (those whose
+`ep.load()` raises) are silently omitted.  An empty list is returned if the
+group has no registered entry points or if `importlib.metadata` is
+unavailable.
+
+---
+
+## Registering a plugin
+
+Add an entry point to your package's `pyproject.toml`:
+
+```toml
+[project.entry-points."spanforge.scorers"]
+my_scorer = "my_package.scorers:MyScorer"
+```
+
+After `pip install`-ing your package, `discover("spanforge.scorers")` will
+include `MyScorer` in the returned list.
+
+---
+
+## Python version compatibility
+
+| Python | `entry_points()` API used |
+|--------|--------------------------|
+| 3.12+ | `entry_points(group=group)` |
+| 3.10–3.11 | `entry_points().select(group=group)` |
+| 3.9 | `entry_points().get(group, [])` (dict style) |