continuous-intelligence-layer 0.1.0__py3-none-any.whl

continuous_intelligence_layer-0.1.0.dist-info/METADATA

@@ -0,0 +1,633 @@
+Metadata-Version: 2.4
+Name: continuous-intelligence-layer
+Version: 0.1.0
+Summary: Multi-Agent Intelligent Validation & RCA Framework
+Project-URL: Homepage, https://github.com/Jugal-lachhwani/Multi_agent_intelligence_layer
+Project-URL: Repository, https://github.com/Jugal-lachhwani/Multi_agent_intelligence_layer
+Author-email: dmlabs <dmlabs6@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,langgraph,llm,observability,opentelemetry,rca,tracing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.11
+Requires-Dist: anthropic>=0.34.0
+Requires-Dist: bcrypt>=4.0.0
+Requires-Dist: colorama>=0.4.6
+Requires-Dist: fastapi>=0.110.0
+Requires-Dist: google-genai>=1.0.0
+Requires-Dist: langchain-community>=0.2.0
+Requires-Dist: langchain-openai>=0.1.0
+Requires-Dist: langchain>=0.2.0
+Requires-Dist: langgraph>=0.2.0
+Requires-Dist: networkx>=3.3
+Requires-Dist: openai>=1.30.0
+Requires-Dist: openinference-instrumentation-langchain>=0.1.0
+Requires-Dist: openinference-semantic-conventions>=0.1.0
+Requires-Dist: opentelemetry-api>=1.25.0
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.25.0
+Requires-Dist: opentelemetry-sdk>=1.25.0
+Requires-Dist: passlib[bcrypt]>=1.7.4
+Requires-Dist: pydantic>=2.7.0
+Requires-Dist: pyjwt>=2.8.0
+Requires-Dist: pymongo>=4.8.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: python-multipart>=0.0.9
+Requires-Dist: requests>=2.31.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: tenacity>=8.3.0
+Requires-Dist: uvicorn>=0.29.0
+Provides-Extra: all
+Requires-Dist: openinference-instrumentation-anthropic>=1.0.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-crewai>=1.0.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-llama-index>=3.0.0; extra == 'all'
+Requires-Dist: openinference-instrumentation-openai>=0.1.19; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: openinference-instrumentation-anthropic>=1.0.0; extra == 'anthropic'
+Provides-Extra: crewai
+Requires-Dist: openinference-instrumentation-crewai>=1.0.0; extra == 'crewai'
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == 'dev'
+Requires-Dist: ipykernel; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: twine>=5.0.0; extra == 'dev'
+Provides-Extra: llamaindex
+Requires-Dist: openinference-instrumentation-llama-index>=3.0.0; extra == 'llamaindex'
+Provides-Extra: openai
+Requires-Dist: openinference-instrumentation-openai>=0.1.19; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# Multi-Agent Intelligence Layer
+
+**A zero-infrastructure, OpenTelemetry-based observability and validation layer for LangGraph / Agentic AI systems.**
+
+This project automatically instruments any LangChain/LangGraph agent, captures full execution traces (nodes, LLMs, tools, tokens, latency), builds an interactive **Execution Graph** in MongoDB, and runs a comprehensive **Validation and Root Cause Analysis (RCA) Engine** over the results.
+
+Trace data never touches MongoDB directly from the agent's machine — the SDK
+POSTs the built graph to the API over HTTP (`POST /traces/ingest`), using the
+same `api_key` already used for project sync. Only the API process holds
+`MONGODB_URI`; agent machines need nothing but an `api_key` and network
+access to the API.
+
+---
+
+## 🏗️ Complete Architecture
+
+```text
+       Your LangGraph Agent
+                │
+                ▼
+  Continuous Intelligence Layer SDK (Zero-config OpenTelemetry + OpenInference)
+                │
+                ├──▶ JSONL Exporter (Local trace backup)
+                │
+                └──▶ GraphSpanExporter (Buffers spans & builds graph)
+                           │
+                           ▼
+              POST /traces/ingest  (x-api-key auth)
+                           │
+                           ▼
+                    FastAPI Backend
+                           │
+                           ▼
+                  MongoDB (Atlas or self-hosted)
+      (executions / nodes / evaluations / rca_reports)
+                           │
+                           ▼
+             Evaluation Runner Pipeline
+    ┌──────────────────────┼──────────────────────┐
+    ▼                      ▼                      ▼
+Input Evaluator     Output Evaluator    Tool/Agent Evaluator
+    │                      │                      │
+    └──────────────────────┼──────────────────────┘
+                           ▼
+                 Validation Results
+                 (Saved to JSONL & MongoDB)
+                           │
+                           ▼
+             Root Cause Analysis (RCA) Engine
+               (Analyzes failures & saves to MongoDB)
+```
+
+---
+
+## 🚀 Quick Start
+
+### 1. Prerequisites
+- Python 3.11+
+- A MongoDB instance — the easiest path is a free [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) cluster (no local install/Docker needed); a local `mongod` also works.
+  `MONGODB_URI` is only needed by the **API server** process — it can live
+  fully private (Atlas network access rules / same VPC as the API) since
+  agent machines running `continuous_intelligence_layer.<framework>.init()` never connect to it
+  directly; they only need an `api_key` and HTTP access to the API.
+
+### 2. Setup
+
+**Just want to instrument an agent against a hosted/existing AgentOPS API?**
+Install the published SDK directly — no repo clone needed:
+
+```bash
+pip install continuous-intelligence-layer
+
+# Or with a specific framework's instrumentor extra:
+pip install "continuous-intelligence-layer[crewai]"
+pip install "continuous-intelligence-layer[openai]"
+pip install "continuous-intelligence-layer[anthropic]"
+```
+
+**Running the full stack (API + UI + MongoDB) yourself?** Clone the repo:
+
+```bash
+# Clone the repository
+git clone https://github.com/Jugal-lachhwani/Multi_agent_intelligence_layer.git
+cd Multi_agent_intelligence_layer
+
+# Create a virtual environment and install
+python -m venv .venv
+source .venv/bin/activate
+pip install -e .
+
+# Setup environment variables (requires OpenAI API Key for evaluation & RCA,
+# and a MongoDB connection string)
+cp .env.example .env
+# Edit .env and set:
+#   OPENAI_API_KEY="sk-..."
+#   MONGODB_URI="mongodb+srv://<user>:<password>@<cluster>.mongodb.net/?retryWrites=true&w=majority"
+```
+
+### 3. MongoDB
+
+No separate service to start — the app connects directly to whatever `MONGODB_URI` in `.env` points at (Atlas cluster or local `mongod`). Collections and indexes are created automatically the first time the API starts or `seed_db.py` runs.
+
+### 4. Start the AgentOPS UI and API
+
+The AgentOPS layer includes a full observability dashboard to view traces, token usage, and evaluation results.
+
+1. **Start the FastAPI Backend:**
+   ```bash
+   # Seed a default demo user + project into MongoDB (safe to re-run — idempotent)
+   python seed_db.py
+
+   # Start the API server (runs on http://localhost:8080)
+   uvicorn api.main:app --reload --port 8080
+   ```
+
+2. **Start the React Frontend:**
+   ```bash
+   # In a new terminal window
+   cd ui
+   npm install
+   npm run dev
+   ```
+   Open **[http://localhost:5173/projects/proj_default_123](http://localhost:5173/projects/proj_default_123)** in your browser to view the dashboard!
+
+### 5. Run the Pipeline
+
+1. **Run the Demo Agent:**
+   This executes the multi-agent LangGraph demo and POSTs the built graph to
+   the API's `/traces/ingest` endpoint, which is the only thing that writes
+   it to MongoDB.
+   ```bash
+   AGENTOPS_API_KEY="ao_..." python examples/multi_agent_demo.py "Explain quantum computing simply"
+   ```
+   `AGENTOPS_API_KEY` should be an API key from a registered user (see `POST /register` then `POST /api_key`, or use the one printed by `seed_db.py`). The demo agent's machine needs this key and the API's `base_url` (`http://localhost:8080` by default) — it does **not** need `MONGODB_URI`.
+
+2. **Run the Evaluator and RCA Engine:**
+   This evaluates the execution, saves validation results, and generates a Root Cause Analysis report.
+   *(Make sure you grab the `execution_id` from the output of the demo script).*
+   ```bash
+   python -c "
+   from evaluators.runner import EvaluationRunner
+   # Replace with your actual execution ID
+   runner = EvaluationRunner('YOUR_EXECUTION_ID_HERE')
+   runner.run()
+   "
+   ```
+
+### 6. View the Results in MongoDB (Advanced)
+
+The dashboard UI is the intended way to inspect traces, but the raw documents
+are also queryable directly — via [MongoDB Atlas's Data Explorer](https://www.mongodb.com/docs/atlas/atlas-ui/data-explorer/)
+in the browser (if using Atlas), or the `mongosh` shell / any GUI client
+(Compass, Studio 3T) pointed at your `MONGODB_URI`.
+
+1. **View the Execution Graph** (nodes for one execution):
+   ```js
+   db.nodes.find({ execution_id: "YOUR_EXECUTION_ID" })
+   db.executions.findOne({ _id: "YOUR_EXECUTION_ID" })   // summary + edges
+   ```
+2. **View Evaluation Verdicts:**
+   ```js
+   db.evaluations.find(
+     { execution_id: "YOUR_EXECUTION_ID", confidence: { $gt: 0.3 } },
+     { node_name: 1, evaluator: 1, status: 1, severity: 1, reason: 1 }
+   ).sort({ timestamp: 1 })
+   ```
+3. **View the Root Cause Analysis (RCA) Report:**
+   ```js
+   db.rca_reports.findOne(
+     { _id: "YOUR_EXECUTION_ID" },
+     { overall_status: 1, root_cause: 1, recommendations: 1 }
+   )
+   ```
+
+---
+
+## 📊 Example Evaluation & RCA Output
+
+Below is an example of the Evaluation Engine successfully detecting a "silent failure" where a search tool returned irrelevant information, causing downstream agents to struggle.
+
+### Node Executions & Metrics
+*(Extracted from the MongoDB `nodes` collection)*
+
+| Output | Status | Latency_ms |
+|--------|--------|------------|
+| `[Supervisor] I’ll first use the Research Agent...` | OK | 2387.188 |
+| `[web_search] Results for 'Explain quantum computing simply': Recent AI agent research highlights...` | OK | 0.340 |
+| `[document_retriever] Retrieved 1 document... Document: 'General Overview'` | OK | 0.344 |
+| `[Research Agent] - The web results are not relevant to explaining quantum computing...` | OK | 2699.448 |
+| `[Writer Agent] # Quantum Computing, Simply Explained...` (Truncated mid-sentence) | OK | 6346.446 |
+
+### Evaluation Verdicts
+
+| Node Name | Evaluator | Status | Reason |
+|-----------|-----------|--------|--------|
+| `supervisor` | `OutputEvaluator` | **PASS** | "The output is well-formed, preserves the user query, and provides an appropriate orchestration plan..." |
+| `web_search` | `OutputEvaluator` | **FAIL** (High) | "The output is completely unrelated to the request to explain quantum computing simply and instead discusses AI-agent architectures." |
+| `document_retriever` | `ToolAgentEvaluator` | **FAIL** (High) | "The returned document contains no substantive quantum-computing information. The result is effectively empty." |
+| `research_agent` | `OutputEvaluator` | **WARNING** (Med) | "Honestly reports that retrieved sources are inadequate... However, it does not provide substantive research for a downstream writer." |
+| `writer_agent` | `InputEvaluator` | **WARNING** (Med) | "The supplied research is largely irrelevant... which could confuse the writer." |
+| `LangGraph` | `OutputEvaluator` | **FAIL** (Med) | "The response is relevant and generally accurate, but the writer's explanation is visibly truncated mid-sentence..." |
+
+### Root Cause Analysis (RCA) Report
+
+```json
+{
+  "overall_status": "PARTIAL",
+  "root_cause": "The primary root cause was a research-tool grounding failure: the web_search node returned semantically unrelated AI-agent content for a quantum-computing query, and the system accepted that output instead of detecting and recovering from the relevance mismatch.",
+  "confidence": 0.91,
+  "propagation_chain": [
+    "LangGraph",
+    "supervisor",
+    "research_agent",
+    "web_search",
+    "document_retriever",
+    "research_agent",
+    "writer_agent",
+    "LangGraph"
+  ],
+  "contributing_factors": [
+    "The document_retriever node also returned non-substantive content (only a generic document label), leaving the research agent without usable grounded material.",
+    "No tool-output relevance, topic-consistency, or minimum-content validation gate stopped unrelated or empty retrieval results before they reached downstream agents.",
+    "The research agent appears to have continued despite both retrieval sources being unusable, rather than retrying with a reformulated query, selecting an alternate source, or explicitly reporting insufficient evidence.",
+    "The writer agent likely relied on general model knowledge because the provided research context was weak or invalid, reducing traceability and reliability even though the visible content was broadly relevant.",
+    "The final LangGraph output was truncated mid-sentence, indicating missing output-length budgeting, incomplete-response detection, or response serialization/stream-finalization safeguards.",
+    "The initially empty top-level query field was noted as mildly ambiguous; although the supervisor correctly used the message content, this creates avoidable risk for downstream components that rely on the query state field."
+  ],
+  "recommendations": [
+    "Add a mandatory semantic relevance check after web_search and document_retriever. Require returned content to match the requested topic using query-document similarity thresholds, keyword/entity overlap, and an LLM relevance classifier.",
+    "Reject and retry tool results when they are unrelated, empty, generic, below a minimum substantive-content threshold, or lack extractable facts. For this request, a result focused on AI agents should trigger an automatic retry with a quantum-computing-specific query.",
+    "Implement a retrieval fallback policy: reformulate the query, use a second search provider or corpus, request multiple results, and stop only after obtaining at least one validated source containing relevant explanatory material.",
+    "Require the research agent to emit structured evidence with source snippets, topic labels, and a research-quality status. Prevent handoff to the writer when evidence quality is below threshold unless the writer is explicitly instructed to answer from general knowledge.",
+    "Add a supervisor-level quality gate that checks whether research findings answer the original user question before routing to the writer agent.",
+    "Fix final-response truncation by enforcing token/output budgets, reserving completion tokens for conclusions, validating that the final response ends cleanly, and retrying or continuing generation when termination occurs mid-sentence.",
+    "Populate and normalize the top-level query field from the user message at graph entry so all nodes consume a single canonical request representation.",
+    "Track tool relevance failures as explicit failed or degraded node states rather than reporting zero failed nodes when validation shows unusable tool outputs."
+  ]
+}
+```
+
+---
+
+## 🛠️ Usage in Your Own Code
+
+The SDK is designed to be **plug-and-play**. You don't need to change any of your LangGraph nodes or edges. Just add two lines to the very top of your main script:
+
+```python
+from continuous_intelligence_layer.langgraph import init  # or .crewai / .openai
+
+# Initialize before importing LangGraph/LangChain
+init(
+    session_id="my-conversation-123",  # Groups multiple runs together
+    store="mongodb"                    # Saves the execution graph to MongoDB
+    # run_evaluations=True is the default — evaluations + RCA run automatically
+    # right after the trace is ingested. Pass run_evaluations=False if you'd
+    # rather trigger them yourself later (via the API or the UI's "Run
+    # Evaluations" button).
+)
+
+# ... run your normal LangGraph code here ...
+```
+
+---
+
+## 📂 Project Structure
+
+- **`continuous_intelligence_layer/`**: The core SDK. One subpackage per agent framework (`langgraph/`, `crewai/`, `openai/`), each with its own `init()` that monkey-patches that framework to automatically emit OTel spans. Framework-agnostic pieces (JSONL exporter, `GraphSpanExporter`, which POSTs the built graph to the API's `/traces/ingest` endpoint with `x-api-key` auth instead of writing to MongoDB directly) live in the shared internal `_core/` module.
+- **`ui/`**: React-based frontend dashboard for viewing execution traces, token costs, and evaluation results.
+- **`api/`**: FastAPI backend that serves traces and evaluations from MongoDB to the frontend UI, and is the **only** process that writes trace data to MongoDB — via `POST /traces/ingest`, called by the SDK. Also owns `users`/`projects` auth data (also in MongoDB).
+- **`evaluators/`**: The Evaluation Pipeline containing `InputEvaluator`, `OutputEvaluator`, and `ToolAgentEvaluator`.
+- **`rca_engine/`**: Synthesizes the evaluation results across the graph to generate the final Root Cause Analysis and fixes.
+- **`graph_builder/`**: Converts raw OpenTelemetry spans into structured `ExecutionNode` and `ExecutionEdge` Pydantic models. `MongoStore` (idempotent upserts, one document per node) is used only from within the API process, not from the SDK.
+- **`examples/`**: Contains `multi_agent_demo.py`, a realistic implementation of a hierarchical agent system used for testing the SDK.
+
+---
+
+Models_used = 
+- examples/multi_agent_demo.py (demo agent) → gpt-5.6-luna (reads VALIDATOR_MODEL, not AGENT_MODEL as .env.example implies)
+- examples/support_triage_multi_agent.py (demo agent) → not set in .env, so it falls back to its own default, gpt-4o-mini (reads SUPPORT_AGENT_MODEL)
+- evaluators/ (Input/Output/ToolAgent, via base_evaluator.py) → gpt-5.6-luna (reads VALIDATOR_MODEL)
+- rca_engine/ → gpt-5.6-terra (reads RCA_MODEL)"
+
+## 🔎 In-Depth Feature Reference (Built System)
+
+This section documents what is **actually implemented** in each module of the
+built system, verified by a full code read-through (audit date: 2026-08-03).
+It excludes `examples/` — those are demo agents used to exercise the SDK, not
+part of the product itself. For the full bug/fix history behind these
+findings, see [`ISSUES.md`](./ISSUES.md).
+
+### `continuous_intelligence_layer/` — SDK
+
+- Each framework subpackage exposes its own **`init(api_key, service_name, session_id=None, execution_id=None, store="mongodb", write_jsonl=False, base_url="http://localhost:8080", verbose=True, run_evaluations=True)`**
+  — `continuous_intelligence_layer.langgraph.init()` (covers LangGraph and
+  plain LangChain, since both share the same OpenInference instrumentor),
+  `continuous_intelligence_layer.crewai.init()`, and
+  `continuous_intelligence_layer.openai.init()`. The framework is selected by
+  which subpackage you import, not by a `frameworks=` argument. Each is
+  idempotent within a process (a second call is a no-op) and provides
+  `reset()` for test suites that need to re-init.
+- Verifies `api_key` against the backend at startup and auto-registers the
+  project via `POST /projects/sync` — `service_name` becomes the project name.
+  The returned `project_id`, together with `api_key` and `base_url`, is
+  threaded into `GraphSpanExporter` so it can authenticate the later
+  `/traces/ingest` call — the SDK never needs `MONGODB_URI`.
+- **`run_evaluations` (default `True`)** is threaded into `GraphSpanExporter`
+  and sent as a `run_evaluations` query param on its `/traces/ingest` POST.
+  The SDK itself never runs evaluations or waits on them — it just tells the
+  API whether to kick them off. Pass `run_evaluations=False` to keep the
+  pre-existing manual-only flow (evaluations only run when triggered via the
+  API or the UI's "Run Evaluations" button).
+- Resource attributes attached to every span: `service.name`, `project.id`,
+  `execution.id`, `session.id`, `sdk.name`, `sdk.version`.
+- Each subpackage's `instrumentation.py::instrument()` activates that
+  framework's single OpenInference instrumentor (`LangChainInstrumentor`,
+  `CrewAIInstrumentor`, or `OpenAIInstrumentor`, guarded against
+  double-patching); `uninstrument()` restores original callbacks. A missing
+  `openinference-instrumentation-*` package degrades gracefully with a fix-it
+  message instead of crashing.
+- **Span → record extraction** (`ExecutionSpanExporter._extract`) captures:
+  trace/span/parent IDs, latency (ms), node-type classification (from
+  OpenInference span kind, with a keyword-based fallback on span name),
+  status/error (from OTel span events), token usage (prompt/completion/total),
+  prompt/response (both legacy and current OpenInference attribute keys),
+  tool name/description/retrieved docs, and any remaining attributes as
+  free-form `metadata`.
+- **Two export paths**: `ExecutionSpanExporter` writes durable JSONL
+  (`SimpleSpanProcessor`, synchronous per-span); `GraphSpanExporter` buffers
+  spans **in memory** and, at `TracerProvider` shutdown, builds the graph and
+  `POST`s it as JSON to `{base_url}/traces/ingest` with an `x-api-key` header
+  (see Known Gaps below) — it never imports or connects to `MongoStore`.
+- `_core/utils.py` (re-exported from every framework subpackage) provides
+  `load_traces()`, `print_trace_summary()` (terminal pretty-printer with
+  per-span status icons and an aggregate footer), and `spans_to_dataframe()`
+  (pandas export).
+
+### `graph_builder/` — DAG construction & persistence
+
+- **Data model** (`models.py`): `ExecutionNode` (~20 fields — identity,
+  `node_type` enum, LLM/tool-specific fields, `TokenUsage`, timestamps,
+  `metadata`, `status`, `validation_results`), `ExecutionEdge`
+  (`source_id`/`target_id`/`edge_type`), `ExecutionGraph` (nodes + edges, with
+  helpers: `get_node`, `get_children`, `get_root_nodes`, `get_nodes_by_type`,
+  `get_failed_nodes`, `summary()`).
+- **Builder algorithm** (`builder.py::TraceToGraphBuilder.build`): sorts spans
+  by timestamp, maps each span to an `ExecutionNode` keyed by `span_id`,
+  creates a `CALLS` edge from `parent_span_id → node_id` when the parent span
+  is present in the same batch, then groups nodes by `parent_id` and chains
+  each sibling group into a timestamp-ordered `NEXT` edge list. Only `CALLS`
+  and `NEXT` are ever produced — `RETURNS`/`USES`/`READS`/`WRITES` exist as
+  enum values but are never emitted.
+- **MongoStore** (`mongo_store.py`): only ever instantiated inside the API
+  process now — `save_graph()` is called from `POST /traces/ingest`, not
+  from the SDK. Idempotent `update_one(..., upsert=True)`
+  writes, one document per node in a `nodes` collection (keyed by `node_id`)
+  — deliberately **not** one document per execution embedding a `nodes`
+  array, since that risks hitting MongoDB's 16MB per-document limit on large
+  executions or long LLM outputs. A lightweight `executions` header document
+  (status, latency, node count, timestamps, embedded `edges` array, root
+  input/output snippets) is written alongside every save, so the trace-list
+  view never has to fetch full node bodies. Indexes on `execution_id`,
+  `project_id`, `session_id`, and compound indexes for the KPI/trace-list
+  query patterns are created once at startup. Read helpers: `get_graph`,
+  `get_session_graph`, `get_llm_spans`, `get_tool_spans`, `get_failed_nodes`,
+  `get_all_execution_ids`, `get_all_session_ids`, plus `get_trace_summaries`/
+  `get_trace_detail` (purpose-built to avoid the old per-row full-graph
+  fetch). Also exposes the write path used by evaluators/RCA:
+  `write_evaluation_result` (one document per evaluation in an `evaluations`
+  collection, referencing `node_id`/`project_id` as plain fields rather than
+  a graph edge), `update_node_evaluation` (flattened eval props on the node
+  document for fast filtering, plus denormalizing `has_evaluation`/`status`
+  onto the execution header), `write_rca_result` (one document per
+  `execution_id` in `rca_reports`), plus `clear_execution` / `clear_session`
+  maintenance methods (wrapped in a MongoDB multi-document transaction so the
+  node/evaluation/RCA/header deletes for one execution are all-or-nothing).
+
+### `evaluators/` — Evaluation Engine
+
+Three evaluators, orchestrated by `EvaluationRunner`, all sharing one
+`BaseEvaluator` (OpenAI `gpt-4o-mini` by default via `VALIDATOR_MODEL`,
+JSON-mode responses, no retry logic — any failure degrades to a generic
+`FAIL`/low-confidence result rather than crashing the run):
+
+- **`InputEvaluator`** — runs on every node. Checks completeness, context
+  relevance, well-formedness, prompt-injection detection, and ambiguity.
+  Detected injection **always** forces `status=FAIL`, `severity=CRITICAL`,
+  overriding whatever the LLM itself returned.
+- **`OutputEvaluator`** — runs on every node. Auto-detects structured
+  (dict/JSON) vs. unstructured (text) output. Structured checks: relevance,
+  structural correctness, internal consistency, handoff validity, schema
+  violations. Unstructured checks: relevance, completeness, hallucination
+  (with specific unsupported-claim evidence), toxicity. Detected toxicity
+  always forces `status=FAIL`, `severity=CRITICAL`.
+- **`ToolAgentEvaluator`** — runs only on Tool/Retriever nodes, and on
+  Agent/Router nodes that have at least one tool-type child. Tool-node mode
+  grades input/output quality given the calling agent's context; agent-node
+  mode grades whether tool selection was appropriate (input/output quality of
+  the tool call itself is intentionally left to the tool-node evaluation).
+- **`EvaluationRunner.run()`**: loads the graph from MongoDB, evaluates every
+  node with all 3 evaluators sequentially (no parallelism), builds a
+  `NodeEvaluationSummary` per node (overall status + worst severity rollup),
+  persists each result to MongoDB immediately, appends everything to a JSONL
+  log, then automatically invokes the RCA Engine and persists its report.
+  **Not idempotent**: because `EvaluationResult.evaluation_id` is a fresh
+  random UUID each run, re-running the pipeline for the same `execution_id`
+  creates duplicate `evaluations` documents (the flattened eval props on the
+  node document and the `rca_reports` document *are* overwritten cleanly on
+  rerun, since those are keyed by stable ids).
+- **Triggering**: `EvaluationRunner` itself has no idea whether it was invoked
+  manually or automatically — both paths call the exact same constructor/`run()`.
+  It's invoked from two call sites: `POST .../run-evaluations` (`api/routers/traces.py`,
+  synchronous, request/response cycle) and, since `init(run_evaluations=True)`
+  became the SDK default, from a `BackgroundTasks` job scheduled inside
+  `POST /traces/ingest` right after a trace is saved. The one-shot guard on
+  the manual endpoint (`count_evaluations(...) > 0` → 400) is what keeps the
+  two triggers from racing into duplicate writes: once auto-evaluation has
+  run for a trace, a later manual "Run Evaluations" click just 400s instead
+  of re-running.
+
+### `rca_engine/` — Root Cause Analysis
+
+- **`RCAEngine.analyze(graph)`** takes an in-memory `ExecutionGraph` whose
+  nodes already have `validation_results` attached (done by the runner, not
+  a separate DB fetch). One LLM call (`gpt-4o` by default via `RCA_MODEL`)
+  produces `overall_status`, `root_cause`, `confidence`, `propagation_chain`,
+  `contributing_factors`, and `recommendations`.
+- **`evidence` is deterministic, not LLM-generated** — it's built directly
+  from every `FAIL`-status evaluation entry in code, which keeps root-cause
+  claims grounded in actual evaluator findings rather than model invention.
+- Any LLM/parsing failure degrades to a fixed fallback `RCAResult`
+  (`overall_status` derived from whether any failures exist) instead of
+  raising — the evaluation run is never blocked by an RCA failure.
+- Persisted as a standalone document in `rca_reports`, keyed by
+  `execution_id` (safe to re-run, always overwrites in place via upsert).
+- **`IncidentReportGenerator`** (`incident_report.py`) is a separate,
+  fully-working Markdown report exporter (status badge, overview table, ASCII
+  propagation-chain diagram, evidence table, node summary table) — it is
+  **not currently wired into the automatic evaluation pipeline** (no caller
+  outside `examples/`), so it exists as a ready-to-use feature rather than
+  dead code.
+
+### LLM Models Actually Configured
+
+All LLM calls in this project go through the OpenAI SDK (`OPENAI_API_KEY`).
+There is no single global "the LLM" — four independent env vars pick the
+model per component:
+
+| Component | Env var | Code default (`.env.example`) | Currently set in `.env` |
+|---|---|---|---|
+| `evaluators/base_evaluator.py` (all 3 evaluators) | `VALIDATOR_MODEL` | `gpt-4o-mini` | `gpt-5.6-luna` |
+| `rca_engine/rca_engine.py` | `RCA_MODEL` | `gpt-4o` | `gpt-5.6-terra` |
+| `examples/multi_agent_demo.py` (demo agent's own LLM) | `VALIDATOR_MODEL` *(reuses the evaluator's var — see note)* | `gpt-4o-mini` | `gpt-5.6-luna` |
+| `examples/support_triage_multi_agent.py` (demo agent's own LLM) | `SUPPORT_AGENT_MODEL` | `gpt-4o-mini` | *(not set in `.env` — falls back to default)* |
+
+- `.env.example` documents an `AGENT_MODEL` var as controlling "the demo
+  agent", but `examples/multi_agent_demo.py` actually reads `VALIDATOR_MODEL`,
+  not `AGENT_MODEL` — so that demo agent currently shares a model with the
+  evaluators rather than having its own knob, and `AGENT_MODEL` is unused/dead.
+- `gpt-5.6-luna` / `gpt-5.6-terra` are custom/internal aliases currently set
+  in the live `.env`, not OpenAI's public model names — swap them for real
+  model IDs (e.g. `gpt-4o-mini` / `gpt-4o`) if pointing at the standard
+  OpenAI API.
+
+### `api/` — FastAPI Backend
+
+- **Auth**: two schemes — JWT bearer tokens (7-day expiry, `HS256`, bcrypt
+  password hashing) for the dashboard, and a static per-user API key
+  (`ao_<32 hex chars>`) for agent-SDK ingestion. `SECRET_KEY` is required from
+  the environment with no insecure fallback.
+- **Endpoints**:
+  - Auth: `POST /register`, `POST /token`, `GET /me`, `POST /api_key`.
+  - Projects: `GET/POST /projects`, `POST /projects/sync` (idempotent
+    find-or-create per `(user, service_name)` — the mechanism behind the
+    "service_name = project" design), `DELETE /projects/{id}`.
+  - Traces: `GET /projects/{project_id}/traces` (per-project rollup: status,
+    latency, evaluation presence), `GET /projects/{project_id}/traces/{execution_id}`
+    (full `{graph, evaluations, rca}` payload), `POST /projects/{project_id}/traces/{execution_id}/run-evaluations`
+    (synchronous — runs `EvaluationRunner` **inline** in the request/response
+    cycle; guarded to only run once per execution, with no reset/re-run path —
+    this is the manual trigger, still available regardless of whether
+    auto-evaluation already ran).
+  - Ingest: `POST /traces/ingest` — the SDK's only write path into MongoDB.
+    Accepts a serialized `ExecutionGraph` plus a `run_evaluations` query param
+    (default `True`, set by the SDK from `init(run_evaluations=...)`).
+    Authenticates via the same `x-api-key` header `/projects/sync` uses,
+    verifies the payload's `project_id` belongs to that api key's user (403
+    otherwise), re-stamps `project_id` onto every node server-side (never
+    trusts the client's per-node values), then calls `MongoStore.save_graph()`.
+    This is what lets `MONGODB_URI` stay private to the API process. When
+    `run_evaluations` is true, it then schedules `EvaluationRunner(...).run()`
+    (which triggers RCA too) as a FastAPI `BackgroundTasks` job **after** the
+    response is sent, so the SDK's shutdown-time POST isn't held open for the
+    LLM-bound evaluation/RCA pipeline. A background failure is only logged
+    server-side (`print`) — there's no request left to report it to — so a
+    trace can still be evaluated later via the manual `run-evaluations`
+    endpoint if the background run fails.
+- **Multi-tenancy**: every project/trace endpoint scopes by `user_id` and
+  `project_id` (both plain MongoDB query filters now — no FK/graph traversal
+  involved), closing the cross-tenant leakage class of bug described in
+  `ISSUES.md` #1.
+- **Data model**: a single MongoDB database (`agentops` by default) holds
+  everything — `users` (username, bcrypt hash, nullable API key, unique
+  indexes), `projects` (`_id` = slugified name, `user_id` field, compound
+  unique index on `(user_id, name)`), and `executions`/`nodes`/`evaluations`/
+  `rca_reports` for all execution/evaluation/RCA data. No SQLite, no second
+  database to run.
+
+### `ui/` — React Dashboard
+
+- **5 pages**: `Login`, `Register`, `Dashboard` (project grid + API key
+  management), `ProjectView` (trace list table), `TraceView` (the primary
+  trace-inspection surface).
+- `Dashboard` and `ProjectView` auto-refresh every 3 seconds and surface
+  run/failure counts per project.
+- **`TraceView`**: renders the span hierarchy as a hand-rolled recursive tree
+  component (not a graph library — `reactflow` is an installed dependency but
+  currently unused/unrendered anywhere). Each row shows a colored evaluation-
+  status dot (red/amber/green) and, for LLM nodes, the model name and an
+  estimated per-call cost (client-side, against a small hardcoded pricing
+  table). The right pane has Details / Evaluations / RCA Report tabs; the RCA
+  summary banner is trace-level and stays visible regardless of the selected
+  node. The "Run Evaluations" button calls the backend synchronously and
+  disappears once evaluations exist for the trace (no re-run affordance in
+  the UI either) — with `init(run_evaluations=True)` (the SDK default), the
+  button will typically already be hidden by the time a trace is opened,
+  since evaluation runs automatically in the background right after ingest.
+- Several controls are currently **decorative placeholders** with no backing
+  logic: the ProjectView search bar, its "With errors"/"Tags" filter buttons,
+  its row checkboxes, and two Sidebar nav items ("Test suites", "Experiments").
+
+### Known Gaps & Next-Phase Backlog
+
+Captured here so they can be triaged deliberately rather than rediscovered.
+See `ISSUES.md` for the full audit trail of what's already been fixed.
+
+- **Non-idempotent evaluation writes** — re-running evaluations for an
+  execution creates duplicate `evaluations` documents; the API also only
+  allows running evaluations once per trace, with no delete/reset endpoint.
+- **Auto-triggered evaluation has no retry/observability** — the
+  `BackgroundTasks` job `POST /traces/ingest` schedules when
+  `run_evaluations=True` (the default) has no retry, no dead-letter queue,
+  and no status surfaced anywhere if it fails — a failure is only a `print`
+  in the API server's own logs. The trace itself still ingests successfully
+  either way; a user who notices missing evaluations can always fall back to
+  the manual "Run Evaluations" button/endpoint.
+- **Trace data is only sent at process shutdown** (buffer-then-flush in
+  `GraphSpanExporter`, now a single `POST /traces/ingest` instead of a direct
+  Mongo write) — long-running/persistent agent processes show nothing in the
+  UI until they exit. Unchanged by the Neo4j → MongoDB migration or the
+  move to routing writes through the API; would need a real per-span write
+  path to fix.
+- **Fan-out/fan-in topology is flattened** into a single linear `NEXT` chain
+  (see `builder.py` note in `CLAUDE.md`) — a supervisor's parallel branches
+  aren't distinguishable from a genuinely linear pipeline today.
+- **No pagination anywhere** — API list endpoints and every UI list view load
+  the entire dataset in one shot. (The trace-list endpoint's old N+1 —
+  fetching the full graph per execution just to compute a summary row — was
+  fixed as part of the MongoDB migration: a lightweight `executions` header
+  document now carries the summary fields directly, so listing traces no
+  longer touches node bodies at all. Pagination itself is still not
+  implemented.)
+- **Permissive CORS** (`allow_origins=["*"]` with `allow_credentials=True`),
+  no JWT revocation/logout flow, and a single non-expiring API key per user.
+- **No true graph visualization yet** — `reactflow` is installed but unused;
+  `TraceView` uses a recursive-tree approximation instead of a node-link view.
+
+---
+