provy-sdk 0.5.0__tar.gz

provy_sdk-0.5.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.4
+Name: provy-sdk
+Version: 0.5.0
+Summary: Python SDK for Provy: prove your AI agents actually worked
+Author: Provy
+License: MIT
+Project-URL: Documentation, https://provy.ai/knowledge/index
+Project-URL: Issues, https://github.com/amitgarg73/provy-sdk/issues
+Project-URL: Homepage, https://provy.ai
+Project-URL: Repository, https://github.com/amitgarg73/provy-sdk
+Keywords: observability,llm,agents,opentelemetry,evals,outcome-assurance
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28
+Requires-Dist: pytz>=2023.3
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: judge
+Requires-Dist: anthropic>=0.25.0; extra == "judge"
+Provides-Extra: engine
+Requires-Dist: supabase>=2.0.0; extra == "engine"
+Provides-Extra: otel
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "otel"
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "otel"
+Provides-Extra: all
+Requires-Dist: anthropic>=0.25.0; extra == "all"
+Requires-Dist: supabase>=2.0.0; extra == "all"
+Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "all"
+Requires-Dist: opentelemetry-api>=1.20.0; extra == "all"
+
+# provy-sdk
+
+Python SDK for [Provy](https://provy.ai): prove your AI agents actually worked.
+
+Send every session, agent step and evaluation with an ingest key. No database credentials and no
+Provy-side configuration — close a session and it appears on your dashboard.
+
+> **A note on names.** You may see `argus` inside: environment variables such as `ARGUS_INGEST_KEY`
+> and `ARGUS_URL` still work, and OpenTelemetry attributes are still `argus.*`. Argus is the original
+> codename, and those names are kept for wire compatibility so existing integrations keep working.
+> Everything you actually type is `provy`.
+
+## Reliability
+
+Telemetry that silently disappears is worse than none, so this client is built not to lose spans:
+
+- **Retries** transient failures with backoff, honouring `Retry-After`. A timeout or a 503 delays
+  your data rather than destroying it.
+- **Buffers and batches** spans, flushing on a background thread and again at process exit, so a
+  short script cannot end with telemetry still in memory.
+- **Never silent.** Anything dropped is counted and logged, and the counts are readable at
+  `client.buffer_stats`.
+- **Never raises into your agent.** A failed send is our problem, not a crash in your pipeline.
+
+---
+
+## Install
+
+```bash
+pip install provy-sdk
+```
+```
+
+The base install is the ingest client only (just `requests`). Optional extras:
+
+| Extra | Adds | For |
+|---|---|---|
+| `provy-sdk[otel]` | OpenTelemetry SDK | streaming existing OTel spans via `ProvyExporter` |
+| `provy-sdk[judge]` | `anthropic` | running the LLM-as-judge in your own pipeline (circuit breakers) |
+| `provy-sdk[engine]` | `supabase` | the legacy direct-to-database path (prefer the ingest API instead) |
+
+---
+
+## Connect
+
+Get an ingest key from Provy: **Agent Fleets → your fleet → Reveal key**. Set it in your environment:
+
+```bash
+export PROVY_API_KEY=argus_...   # key values are still prefixed argus_
+# optional, defaults to the hosted app:
+export PROVY_URL=https://provy.ai
+```
+
+That key authenticates your fleet. It is the only credential you need.
+
+---
+
+## Quickstart — direct ingest
+
+```python
+from provy import ProvyClient
+
+provy = ProvyClient()  # reads PROVY_API_KEY from the environment
+
+session_id = provy.open_session("premarket")
+
+provy.trace(
+    session_id = session_id,
+    agent      = "research",
+    step_type  = "agent_step",          # llm_call | tool_call | agent_step | decision | error
+    outcome    = "Generated AAPL thesis",
+    latency_ms = 1240,
+    tokens_in  = 800,
+    tokens_out = 150,
+)
+
+provy.close_session(session_id, result_summary="Trade plan ready")
+```
+
+Open **Sessions** in Provy — your run appears within seconds.
+
+The decorator form auto-traces a function:
+
+```python
+@provy.trace_fn(agent="research", step_type="agent_step")
+def run_research(ticker):
+    ...
+
+run_research("AAPL", session_id=session_id)
+```
+
+---
+
+## Already on OpenTelemetry?
+
+If your pipeline emits OTel spans (LangChain, CrewAI, AutoGen, LlamaIndex, or raw OTel), attach the exporter and stream them — no per-step calls:
+
+```bash
+pip install "provy-sdk[otel]"
+```
+
+```python
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from provy import ProvyExporter
+
+provider = TracerProvider()
+provider.add_span_processor(BatchSpanProcessor(ProvyExporter(api_key="provy_...")))
+```
+
+Provy auto-detects the convention (OpenInference, OpenLLMetry/Traceloop, Langfuse, OTel GenAI) and builds the session from your spans.
+
+---
+
+## Quality scoring
+
+By default Provy runs the LLM-as-judge **server-side** on the traces you send — no SDK code, no key of yours. Configure criteria in **Eval Manager** and scores appear on the Quality page.
+
+Run the judge **in your own pipeline** only when you want the verdict before an output is used (circuit breakers):
+
+```bash
+pip install "provy-sdk[judge]"   # adds anthropic
+```
+
+```python
+from provy import evaluate_session_outputs
+
+evaluate_session_outputs(session_id, {"research": research_output_text})
+```
+
+Needs `ANTHROPIC_API_KEY` in your environment. Same judge core as the server side.
+
+---
+
+## Business outcomes
+
+For metrics you compute yourself (no LLM), write them with `write_eval()` — they land in **Outcomes**:
+
+```python
+from provy import write_eval
+
+write_eval(
+    session_id = session_id,
+    eval_name  = "approval_rate",
+    agent      = "risk",
+    score      = 0.6,
+    passed     = True,
+    threshold  = 0.2,
+    reasoning  = "3 of 5 proposals approved",
+)
+```
+
+---
+
+## API reference
+
+### `ProvyClient(ingest_key=None, base_url=None)`
+Reads `PROVY_API_KEY` / `PROVY_URL` from the environment when arguments are omitted (legacy `ARGUS_INGEST_KEY` / `ARGUS_URL` still work).
+
+| Method | When to call |
+|---|---|
+| `open_session(session_type, external_id=None, metadata=None)` | start of a run; returns `session_id` |
+| `trace(session_id, agent, step_type, outcome, ...)` | each step; returns the span id |
+| `close_session(session_id, status="completed", result_summary=None, terminal_reason=None)` | end of the run |
+| `trace_fn(agent, step_type="agent_step")` | decorator that auto-traces a function |
+
+### `ProvyExporter(api_key, endpoint=None)`
+OTel `SpanExporter`. Attach to any `TracerProvider`. Needs the `otel` extra.
+
+### `evaluate_session_outputs(session_id, agent_outputs)`
+Client-side LLM-as-judge. Needs the `judge` extra and `ANTHROPIC_API_KEY`.
+
+### `write_eval(session_id, eval_name, agent, score, passed, threshold, reasoning, layer=5)`
+Writes one business-outcome eval row.
+
+> **Legacy:** `TraceLogger` (direct database writes via the `engine` extra) predates the ingest API. New pipelines should use `ProvyClient`. `TraceLogger` remains for existing internal pipelines.
+
+---
+
+## Examples
+
+- `examples/otel_quickstart.py` — stream OTel spans to Provy
+- `examples/github-actions-otel.yml` — run a pipeline in GitHub Actions and stream to Provy
+
+---
+
+## Support
+
+Open an issue at [github.com/amitgarg73/provy-sdk](https://github.com/amitgarg73/provy-sdk/issues).

provy_sdk-0.5.0/README.md

@@ -0,0 +1,189 @@
+# provy-sdk
+
+Python SDK for [Provy](https://provy.ai): prove your AI agents actually worked.
+
+Send every session, agent step and evaluation with an ingest key. No database credentials and no
+Provy-side configuration — close a session and it appears on your dashboard.
+
+> **A note on names.** You may see `argus` inside: environment variables such as `ARGUS_INGEST_KEY`
+> and `ARGUS_URL` still work, and OpenTelemetry attributes are still `argus.*`. Argus is the original
+> codename, and those names are kept for wire compatibility so existing integrations keep working.
+> Everything you actually type is `provy`.
+
+## Reliability
+
+Telemetry that silently disappears is worse than none, so this client is built not to lose spans:
+
+- **Retries** transient failures with backoff, honouring `Retry-After`. A timeout or a 503 delays
+  your data rather than destroying it.
+- **Buffers and batches** spans, flushing on a background thread and again at process exit, so a
+  short script cannot end with telemetry still in memory.
+- **Never silent.** Anything dropped is counted and logged, and the counts are readable at
+  `client.buffer_stats`.
+- **Never raises into your agent.** A failed send is our problem, not a crash in your pipeline.
+
+---
+
+## Install
+
+```bash
+pip install provy-sdk
+```
+```
+
+The base install is the ingest client only (just `requests`). Optional extras:
+
+| Extra | Adds | For |
+|---|---|---|
+| `provy-sdk[otel]` | OpenTelemetry SDK | streaming existing OTel spans via `ProvyExporter` |
+| `provy-sdk[judge]` | `anthropic` | running the LLM-as-judge in your own pipeline (circuit breakers) |
+| `provy-sdk[engine]` | `supabase` | the legacy direct-to-database path (prefer the ingest API instead) |
+
+---
+
+## Connect
+
+Get an ingest key from Provy: **Agent Fleets → your fleet → Reveal key**. Set it in your environment:
+
+```bash
+export PROVY_API_KEY=argus_...   # key values are still prefixed argus_
+# optional, defaults to the hosted app:
+export PROVY_URL=https://provy.ai
+```
+
+That key authenticates your fleet. It is the only credential you need.
+
+---
+
+## Quickstart — direct ingest
+
+```python
+from provy import ProvyClient
+
+provy = ProvyClient()  # reads PROVY_API_KEY from the environment
+
+session_id = provy.open_session("premarket")
+
+provy.trace(
+    session_id = session_id,
+    agent      = "research",
+    step_type  = "agent_step",          # llm_call | tool_call | agent_step | decision | error
+    outcome    = "Generated AAPL thesis",
+    latency_ms = 1240,
+    tokens_in  = 800,
+    tokens_out = 150,
+)
+
+provy.close_session(session_id, result_summary="Trade plan ready")
+```
+
+Open **Sessions** in Provy — your run appears within seconds.
+
+The decorator form auto-traces a function:
+
+```python
+@provy.trace_fn(agent="research", step_type="agent_step")
+def run_research(ticker):
+    ...
+
+run_research("AAPL", session_id=session_id)
+```
+
+---
+
+## Already on OpenTelemetry?
+
+If your pipeline emits OTel spans (LangChain, CrewAI, AutoGen, LlamaIndex, or raw OTel), attach the exporter and stream them — no per-step calls:
+
+```bash
+pip install "provy-sdk[otel]"
+```
+
+```python
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from provy import ProvyExporter
+
+provider = TracerProvider()
+provider.add_span_processor(BatchSpanProcessor(ProvyExporter(api_key="provy_...")))
+```
+
+Provy auto-detects the convention (OpenInference, OpenLLMetry/Traceloop, Langfuse, OTel GenAI) and builds the session from your spans.
+
+---
+
+## Quality scoring
+
+By default Provy runs the LLM-as-judge **server-side** on the traces you send — no SDK code, no key of yours. Configure criteria in **Eval Manager** and scores appear on the Quality page.
+
+Run the judge **in your own pipeline** only when you want the verdict before an output is used (circuit breakers):
+
+```bash
+pip install "provy-sdk[judge]"   # adds anthropic
+```
+
+```python
+from provy import evaluate_session_outputs
+
+evaluate_session_outputs(session_id, {"research": research_output_text})
+```
+
+Needs `ANTHROPIC_API_KEY` in your environment. Same judge core as the server side.
+
+---
+
+## Business outcomes
+
+For metrics you compute yourself (no LLM), write them with `write_eval()` — they land in **Outcomes**:
+
+```python
+from provy import write_eval
+
+write_eval(
+    session_id = session_id,
+    eval_name  = "approval_rate",
+    agent      = "risk",
+    score      = 0.6,
+    passed     = True,
+    threshold  = 0.2,
+    reasoning  = "3 of 5 proposals approved",
+)
+```
+
+---
+
+## API reference
+
+### `ProvyClient(ingest_key=None, base_url=None)`
+Reads `PROVY_API_KEY` / `PROVY_URL` from the environment when arguments are omitted (legacy `ARGUS_INGEST_KEY` / `ARGUS_URL` still work).
+
+| Method | When to call |
+|---|---|
+| `open_session(session_type, external_id=None, metadata=None)` | start of a run; returns `session_id` |
+| `trace(session_id, agent, step_type, outcome, ...)` | each step; returns the span id |
+| `close_session(session_id, status="completed", result_summary=None, terminal_reason=None)` | end of the run |
+| `trace_fn(agent, step_type="agent_step")` | decorator that auto-traces a function |
+
+### `ProvyExporter(api_key, endpoint=None)`
+OTel `SpanExporter`. Attach to any `TracerProvider`. Needs the `otel` extra.
+
+### `evaluate_session_outputs(session_id, agent_outputs)`
+Client-side LLM-as-judge. Needs the `judge` extra and `ANTHROPIC_API_KEY`.
+
+### `write_eval(session_id, eval_name, agent, score, passed, threshold, reasoning, layer=5)`
+Writes one business-outcome eval row.
+
+> **Legacy:** `TraceLogger` (direct database writes via the `engine` extra) predates the ingest API. New pipelines should use `ProvyClient`. `TraceLogger` remains for existing internal pipelines.
+
+---
+
+## Examples
+
+- `examples/otel_quickstart.py` — stream OTel spans to Provy
+- `examples/github-actions-otel.yml` — run a pipeline in GitHub Actions and stream to Provy
+
+---
+
+## Support
+
+Open an issue at [github.com/amitgarg73/provy-sdk](https://github.com/amitgarg73/provy-sdk/issues).

provy_sdk-0.5.0/provy/__init__.py

@@ -0,0 +1,95 @@
+"""
+Provy SDK.
+
+Importing `provy` is lightweight: it pulls in only the ingest client (REST + OTel
+exporter), which needs `requests`. The optional pieces load on first use and tell
+you which extra to install if it is missing:
+
+  - the LLM-as-judge          → pip install "provy-sdk[judge]"   (anthropic)
+  - the local eval/RCA engine → pip install "provy-sdk[engine]"  (supabase; legacy
+                                direct-DB path — prefer the ingest API)
+
+So a tenant who just wants to send traces installs the base package and nothing else.
+"""
+from __future__ import annotations
+
+import importlib
+from typing import Any
+
+# ── Eager, lightweight (ingest client — requests only) ──────────────────────────
+from provy.client import ProvyClient, ProvyExporter
+from provy.session import TraceLogger
+from provy.evals import write_eval
+
+__version__ = "0.5.0"
+
+# ── Lazy, heavy (loaded on first access; mapped to the extra that provides them) ──
+#   name -> (module, attribute, extra)
+_LAZY: dict[str, tuple[str, str, str]] = {
+    # LLM-as-judge (anthropic)
+    "evaluate_session_outputs":  ("provy.judge", "evaluate_session_outputs", "judge"),
+    # local eval + pattern + RCA engine (supabase, legacy direct-DB)
+    "EvalResult":                ("provy.engine", "EvalResult", "engine"),
+    "Incident":                  ("provy.engine", "Incident", "engine"),
+    "run_evals_from_config":     ("provy.engine", "run_evals_from_config", "engine"),
+    "run_all_detectors":         ("provy.engine", "run_all_detectors", "engine"),
+    "run_quality_detectors":     ("provy.engine", "run_quality_detectors", "engine"),
+    "run_evals_and_persist":     ("provy.engine", "run_evals_and_persist", "engine"),
+    "run_detectors_and_persist": ("provy.engine", "run_detectors_and_persist", "engine"),
+    "compute_shadow_cb_fires":   ("provy.engine", "compute_shadow_cb_fires", "engine"),
+    "build_annotated_call_stack":("provy.engine", "build_annotated_call_stack", "engine"),
+    "generate_fix_suggestion":   ("provy.engine", "generate_fix_suggestion", "engine"),
+    "summarize_incident":        ("provy.engine", "summarize_incident", "engine"),
+    "load_pipeline_config":      ("provy.engine", "load_pipeline_config", "engine"),
+    "load_eval_configs":         ("provy.engine", "load_eval_configs", "engine"),
+    "load_pipeline_agents":      ("provy.engine", "load_pipeline_agents", "engine"),
+    "register_eval":             ("provy.engine", "register_eval", "engine"),
+    "get_registry":              ("provy.engine", "get_registry", "engine"),
+}
+
+
+def __getattr__(name: str) -> Any:  # PEP 562 — module-level lazy attributes
+    target = _LAZY.get(name)
+    if target is None:
+        raise AttributeError(f"module 'provy' has no attribute {name!r}")
+    module, attr, extra = target
+    try:
+        mod = importlib.import_module(module)
+    except ImportError as exc:
+        raise ImportError(
+            f"{name!r} needs the '{extra}' extra. Install it with: "
+            f'pip install "provy-sdk[{extra}]"'
+        ) from exc
+    return getattr(mod, attr)
+
+
+def __dir__() -> list[str]:
+    return sorted(__all__)
+
+
+__all__ = [
+    # ingest client (base)
+    "ProvyClient",
+    "ProvyExporter",
+    "TraceLogger",
+    "write_eval",
+    # LLM-as-judge (extra: judge)
+    "evaluate_session_outputs",
+    # local engine (extra: engine)
+    "EvalResult",
+    "Incident",
+    "run_evals_from_config",
+    "run_all_detectors",
+    "run_quality_detectors",
+    "run_evals_and_persist",
+    "run_detectors_and_persist",
+    "compute_shadow_cb_fires",
+    "build_annotated_call_stack",
+    "generate_fix_suggestion",
+    "summarize_incident",
+    "load_pipeline_config",
+    "load_eval_configs",
+    "load_pipeline_agents",
+    "register_eval",
+    "get_registry",
+]