failproofai 1.0.1-beta.1 → 1.0.2-beta.0

package/.next/standalone/sdk/python/docs/README.md

@@ -0,0 +1,249 @@
+# failproofai-sdk — integration guide
+
+Plug your agent into Failproof AI. One call, no call-site changes, no ids
+threaded through your code.
+
+```bash
+pip install 'failproofai-sdk[langgraph]'
+```
+
+```python
+import failproofai_sdk
+
+failproofai_sdk.instrument()          # auto-detects the frameworks you imported
+
+with failproofai_sdk.session():
+    graph.invoke(...)                 # recorded
+```
+
+That is the whole integration.
+
+---
+
+## pick your framework
+
+| framework | guide | runnable code |
+|---|---|---|
+| LangChain / LangGraph | [langgraph/](langgraph/) | [langgraph/examples/](langgraph/examples/) |
+| CrewAI | [crewai/](crewai/) | [crewai/examples/](crewai/examples/) |
+| LlamaIndex | [llama_index/](llama_index/) | [llama_index/examples/](llama_index/examples/) |
+| Pydantic AI | [pydantic_ai/](pydantic_ai/) | [pydantic_ai/examples/](pydantic_ai/examples/) |
+| **no framework** (your own agent) | [manual/](manual/) | [manual/examples/](manual/examples/) |
+
+Using something else — AutoGen, Haystack, Semantic Kernel, your own loop? Read
+[manual/](manual/). It is the same fidelity, it just costs you the call sites,
+and it explains why AutoGen has no adapter.
+
+---
+
+## what you get
+
+Here is a real trace, printed by `langgraph/examples/quickstart.py`. This is
+captured output, not an illustration:
+
+```
+━━ failproof_ai · langgraph quickstart
+   session  faf3a02d64464e34809eae95fa244230
+   events   14 · agents 1 · types 8
+
+     №    offset  event             detail
+   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+     1   +0.000s  agent_start       LangGraph
+     2   +0.001s    hook_triggered  agent
+     3   +0.002s      model_request  gpt-5.6-terra
+     4   +3.023s      model_response  gpt-5.6-terra · 21 out-tok
+     5   +3.024s    hook_completed  agent
+     6   +3.024s    hook_triggered  tools
+     7   +3.025s      tool_use      word_count
+     8   +3.025s      tool_result   word_count · ok
+     9   +3.025s    hook_completed  tools
+    10   +3.026s    hook_triggered  agent
+    11   +3.027s      model_request  gpt-5.6-terra
+    12   +5.717s      model_response  gpt-5.6-terra · 5 out-tok
+    13   +5.720s    hook_completed  agent
+    14   +5.721s  agent_end         LangGraph · success
+```
+
+One `graph.invoke()` from the outside. Fourteen events with timings, token
+counts and the tool call, from the inside.
+
+---
+
+## the three calls
+
+### 1. `configure()` — optional
+
+```python
+failproofai_sdk.configure(
+    environment="production",   # label on every event; default "dev"
+    flush_interval=0.5,         # seconds between disk flushes
+    base_dir=None,              # spool root; default ~/.failproofai/custom-agents
+)
+```
+
+Defaults are usually right. Set `environment` so you can tell prod from staging
+on the dashboard.
+
+### 2. `instrument()` — the one that does the work
+
+```python
+failproofai_sdk.instrument()               # every framework already imported
+failproofai_sdk.instrument("crewai")       # exactly one
+failproofai_sdk.uninstrument()             # put everything back
+```
+
+Auto-detection reads `sys.modules` — a framework you have installed but are not
+using is never imported on your behalf.
+
+### 3. `session()` — group one run
+
+```python
+with failproofai_sdk.session():
+    ...
+```
+
+A session is one run. Without it, each top-level call becomes its own session,
+which is almost never what you want.
+
+---
+
+## the fifteen event types
+
+| group | events | what they mean |
+|---|---|---|
+| agents | `agent_start` `agent_end` | a unit of work begins / ends |
+| | `agent_pause` `agent_resume` | it is blocked / unblocked (feeds paused time) |
+| models | `model_request` `model_response` | one LLM round trip, with tokens |
+| tools | `tool_use` `tool_result` | one tool call, with args and output |
+| hooks | `hook_triggered` `hook_completed` | a node / step / task boundary |
+| humans | `human_wait` `human_input` | you asked a person / they answered |
+| | `human_pause` `human_interrupt` | a person paused / stopped the agent |
+| failures | `error` | something went wrong |
+
+**Events come in pairs.** The closing event carries a `duration_ms` measured
+from the matching opening one. An opening event with no close renders as a span
+that never finishes.
+
+---
+
+## what each framework emits
+
+Measured from the real runs in this directory, not declared.
+
+| event | langgraph | crewai | llama_index | pydantic ai | your own |
+|---|:--:|:--:|:--:|:--:|:--:|
+| `agent_start` / `agent_end` | yes | yes | yes | yes | you |
+| `model_request` / `model_response` | yes | yes | yes | yes | you |
+| `tool_use` / `tool_result` | yes | yes | yes | yes | you |
+| `hook_triggered` / `hook_completed` | node | task | step | — | you |
+| `error` | yes | yes | yes | yes | automatic |
+| `human_wait` / `human_input` | yes | yes | yes | — | you |
+| `agent_pause` / `agent_resume` | yes | yes | yes | — | you |
+| `human_pause` / `human_interrupt` | — | — | — | — | you |
+
+A dash means the framework has no such concept, not that it is missing.
+Pydantic AI has no node boundary and no built-in human pause, so there is
+nothing to map. `human_pause` and `human_interrupt` describe a *person* acting
+on the agent — a stop button — which no framework signals; emit those yourself.
+
+---
+
+## running the examples
+
+Every example in this tree was executed against a live model before it shipped.
+
+```bash
+pip install 'failproofai-sdk[langgraph]'
+export OPENAI_API_KEY=sk-...
+export FPAI_MODEL=gpt-4o-mini            # optional, this is the default
+
+python docs/langgraph/examples/quickstart.py
+```
+
+Each one prints the event stream it produced when it finishes. That printer
+lives in [`_shared/`](_shared/) and is **cosmetics only** — the two bootstrap
+lines at the top of each example exist just to import it. Delete them and the
+example still instruments correctly.
+
+---
+
+## instrumentation cannot break your agent
+
+Every callback goes through a wrapper whose only job is to re-raise. Your call
+sits in exactly one `try`; everything the SDK does happens outside it.
+
+- a hook that raises is logged once, with its traceback, and your call is
+  unaffected;
+- a hook that raises three times at the same site is disabled for the rest of
+  the process, with one error line saying so.
+
+That firewall is the right default in production and the wrong one when you are
+debugging. To make failures loud instead:
+
+```bash
+export FAILPROOFAI_SDK_STRICT=1
+```
+
+Without it you can only ever prove "it did not crash", never "it swallowed the
+right thing".
+
+---
+
+## zero dependencies
+
+`import failproofai_sdk` pulls in nothing outside the standard library. That is
+enforced, not asserted: one test installs the built wheel with `--no-deps`,
+another launches a fresh interpreter and proves no framework reaches
+`sys.modules`.
+
+The extras install the **framework**, not the adapter — the adapter code always
+ships in the base wheel and is imported only when `instrument()` asks for it.
+Most people already have the framework and never need an extra.
+
+| extra | installs |
+|---|---|
+| `failproofai-sdk[langchain]` | `langchain-core>=1.4.7,<2` |
+| `failproofai-sdk[langgraph]` | `langgraph>=1.2,<2` |
+| `failproofai-sdk[crewai]` | `crewai>=1.13,<2` |
+| `failproofai-sdk[llamaindex]` | `llama-index-core>=0.14.23,<0.15` |
+| `failproofai-sdk[pydantic-ai]` | `pydantic-ai-slim>=2.0,<3` |
+
+There is deliberately no `[all]` — an extra that installs four agent frameworks
+at once is a resolver problem handed to somebody who wanted a telemetry library.
+
+---
+
+## verifying it works
+
+Look at the dashboard, or query the store.
+
+**Do not read the spool directory to check.** When `failproofaid` is running it
+collects and deletes each batch file within milliseconds of it appearing, so a
+read races the collector and returns only what has not shipped yet — which looks
+exactly like an adapter that emitted nothing but its closing events. This is a
+real thing that happened while writing these docs.
+
+To inspect events in-process instead, tap the writer:
+
+```python
+captured = []
+_original = failproofai_sdk._writer.submit
+failproofai_sdk._writer.submit = lambda e: (captured.append(e), _original(e))[1]
+```
+
+Every example here does exactly that.
+
+---
+
+## troubleshooting
+
+| symptom | most likely cause |
+|---|---|
+| no events at all | `instrument()` never ran, or ran after you built the agent (Pydantic AI) |
+| events, but no session | no `with failproofai_sdk.session():` — each call became its own session |
+| `TypeError: ... missing session_id` | you emitted from a new thread; wrap it in `failproofai_sdk.propagate()` |
+| spans that never finish | an opening event with no closing one — emit the pairs |
+| null token counts on LlamaIndex | streaming has no usage; see [llama_index/](llama_index/) |
+| `agent_id` full of uuids | keep it low-cardinality — a role or node name, never a run id |
+| duplicated events on LangChain | you registered a second callback handler as well as `instrument()` |
+| adapter seems to do nothing | set `FAILPROOFAI_SDK_STRICT=1` and run again |

package/.next/standalone/sdk/python/docs/_shared/README.md

@@ -0,0 +1,43 @@
+# _shared
+
+Cosmetics for the runnable examples. **Nothing here is part of the SDK**, and no
+example needs it to instrument correctly.
+
+Two helpers:
+
+- `model()` — reads `FPAI_MODEL` / `MODEL`, defaults to `gpt-4o-mini`, so one
+  export drives every example.
+- `trace(session_id)` — prints the event stream the run produced.
+
+`trace()` is why the examples are worth running rather than reading. An adapter
+that "works" is one whose event stream you can see, so every example ends by
+printing its own instead of asserting in a comment that events happened.
+
+## why it taps the writer instead of reading the spool
+
+When `failproofaid` is running it collects and **deletes** each batch file
+within milliseconds of it appearing. A spool read therefore races the collector
+and returns only whatever has not shipped yet — which looks exactly like an
+adapter that emitted nothing but its closing events.
+
+That is not hypothetical: it is what happened the first time these examples were
+run, and it read as three adapter bugs that did not exist.
+
+So `capture()` wraps `failproofai_sdk._writer.submit` and appends every entry to
+a list first. Same dict that goes to disk, race-free, and it works whether or not
+a daemon is running. `banner()` calls it, so every example is tapped from its
+first line.
+
+## removing it
+
+Each example starts with two bootstrap lines that exist only to import this
+package:
+
+```python
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+...
+from _shared import banner, model, trace
+```
+
+Delete those and the `banner()` / `trace()` calls, and the example still
+instruments correctly. It just stops printing.

package/.next/standalone/sdk/python/docs/_shared/__init__.py

@@ -0,0 +1,195 @@
+"""Cosmetics shared by the examples. No SDK behaviour lives here.
+
+Two helpers, both deliberately boring:
+
+* `model()` — the model id, from `FPAI_MODEL` / `MODEL`, so one export drives
+  every example instead of a dozen edits.
+* `trace()` — reads the spool back and prints what the run actually recorded.
+
+`trace()` is the point of these examples. An adapter that "works" is one whose
+event stream you can read, so every example ends by printing its own trace
+rather than asserting in a comment that events were produced. It reads the same
+JSONL the daemon collects, filtered to the session just run.
+
+Nothing here is required to use the SDK. Delete this folder and the examples
+still instrument correctly; they just stop printing.
+"""
+from __future__ import annotations
+
+import json
+import os
+import pathlib
+import sys
+
+import failproofai_sdk
+
+# ── brand ────────────────────────────────────────────────────────────────────
+# Exact hex from the design system: pink #e4587d, mint #66d1b5, ink #d8d6d2.
+# Truecolor, and only when stdout is a terminal — a piped log gets clean text.
+_TTY = sys.stdout.isatty()
+
+
+def _c(code: str, text: str) -> str:
+    return f"\033[{code}m{text}\033[0m" if _TTY else text
+
+
+def pink(t: str) -> str:
+    return _c("38;2;228;88;125", t)
+
+
+def mint(t: str) -> str:
+    return _c("38;2;102;209;181", t)
+
+
+def dim(t: str) -> str:
+    return _c("38;2;120;120;130", t)
+
+
+def bold(t: str) -> str:
+    return _c("1", t)
+
+
+RULE = "━" * 68
+
+
+def model() -> str:
+    """The model id every example runs against."""
+    return os.environ.get("FPAI_MODEL") or os.environ.get("MODEL") or "gpt-4o-mini"
+
+
+def banner(title: str, subtitle: str = "") -> None:
+    """Print the example's header and start capturing events."""
+    capture()
+    print()
+    print(pink("━━ ") + bold(title.lower()))
+    if subtitle:
+        print(dim("   " + subtitle))
+    print()
+
+
+# ── capturing what the run emitted ───────────────────────────────────────────
+# NOT by reading the spool. When `failproofaid` is running it collects and
+# DELETES each batch file within milliseconds of it appearing, so a spool read
+# races the daemon and returns whatever happens to be left — which looks exactly
+# like an adapter that only emitted its closing events. (That is a real thing
+# that happened while writing these examples.)
+#
+# So the tap sits on the writer instead: the same dict that goes to disk is
+# appended to a list first. Race-free, and it works whether or not a daemon is
+# running.
+_CAPTURED: list[dict] = []
+_TAPPED = False
+
+
+def capture() -> None:
+    """Start recording every event this process emits. Idempotent."""
+    global _TAPPED
+    if _TAPPED:
+        return
+    writer = failproofai_sdk._writer
+    original = writer.submit
+
+    def _tee(entry: dict) -> None:
+        _CAPTURED.append(entry)
+        return original(entry)
+
+    writer.submit = _tee  # instance attribute; the real method is untouched
+    _TAPPED = True
+
+
+def events(session_id: str | None = None) -> list[dict]:
+    """Everything captured, optionally narrowed to one session."""
+    rows = [r for r in _CAPTURED if session_id is None or r.get("session_id") == session_id]
+    return sorted(rows, key=lambda r: r.get("timestamp", ""))
+
+
+_LEAF = {
+    "tool_use": "tool_result",
+    "model_request": "model_response",
+    "hook_triggered": "hook_completed",
+    "agent_start": "agent_end",
+    "human_wait": "human_input",
+    "agent_pause": "agent_resume",
+}
+_CLOSERS = set(_LEAF.values())
+
+
+def _detail(row: dict) -> str:
+    t = row["type"]
+    if t in ("tool_use", "tool_result"):
+        bits = [row.get("tool_name") or "?"]
+        if row.get("error"):
+            bits.append(pink("error"))
+        elif t == "tool_result":
+            bits.append(mint("ok"))
+        return " · ".join(bits)
+    if t in ("model_request", "model_response"):
+        bits = [row.get("model") or "?"]
+        tok = row.get("output_tokens")
+        if tok:
+            bits.append(f"{tok} out-tok")
+        if row.get("error"):
+            bits.append(pink("error"))
+        return " · ".join(bits)
+    if t in ("agent_start", "agent_end"):
+        bits = [row.get("agent_id") or "?"]
+        if row.get("parent_id"):
+            bits.append(dim("under " + row["parent_id"]))
+        if row.get("outcome"):
+            good = row["outcome"] == "success"
+            bits.append((mint if good else pink)(row["outcome"]))
+        return " · ".join(bits)
+    if t in ("hook_triggered", "hook_completed"):
+        return " · ".join(x for x in (row.get("hook_name"), row.get("status")) if x)
+    if t == "error":
+        return pink(str(row.get("message", ""))[:48])
+    for key in ("prompt", "reason", "response", "summary", "goal"):
+        if row.get(key):
+            return dim(str(row[key])[:48])
+    return ""
+
+
+def trace(session_id: str, *, title: str = "trace") -> list[dict]:
+    """Print the event stream this run produced. Returns the rows."""
+    rows = events(session_id)
+    if not rows:
+        print(pink("  no events — is the spool configured?"))
+        return rows
+
+    kinds = sorted({r["type"] for r in rows})
+    agents = sorted({r.get("agent_id") for r in rows if r.get("agent_id")})
+    t0 = rows[0]["timestamp"]
+
+    print(pink("━━ ") + bold(f"failproof_ai · {title}"))
+    print(dim(f"   session  {session_id}"))
+    print(dim(f"   events   {len(rows)} · agents {len(agents)} · types {len(kinds)}"))
+    print()
+    print(dim(f"   {'№':>3}  {'offset':>8}  {'event':<16}  detail"))
+    print(dim("   " + RULE))
+
+    depth = 0
+    for i, row in enumerate(rows, 1):
+        t = row["type"]
+        if t in _CLOSERS:
+            depth = max(depth - 1, 0)
+        indent = "  " * depth
+        offset = _offset(t0, row["timestamp"])
+        name = f"{indent}{t}"
+        print(f"   {i:>3}  {dim(offset):>8}  {name:<16}  {_detail(row)}")
+        if t in _LEAF:
+            depth += 1
+    print(dim("   " + RULE))
+    print(dim("   " + " ".join(kinds)))
+    print()
+    return rows
+
+
+def _offset(first: str, ts: str) -> str:
+    from datetime import datetime
+
+    try:
+        a = datetime.fromisoformat(first.replace("Z", "+00:00"))
+        b = datetime.fromisoformat(ts.replace("Z", "+00:00"))
+        return f"+{(b - a).total_seconds():.3f}s"
+    except Exception:
+        return ""