datagol-agent-harness 0.2.0__tar.gz

datagol_agent_harness-0.2.0/PKG-INFO

@@ -0,0 +1,809 @@
+Metadata-Version: 2.4
+Name: datagol-agent-harness
+Version: 0.2.0
+Summary: A lightweight, production-grade LLM agent toolkit in Python: streaming, MCP, lazy skills, memory, and sandboxing.
+Author-email: Jyotish Bora <jyotish.bora@datagol.ai>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/inncretech/datagol-agent-harness
+Project-URL: Repository, https://github.com/inncretech/datagol-agent-harness
+Project-URL: Issues, https://github.com/inncretech/datagol-agent-harness/issues
+Project-URL: Documentation, https://github.com/inncretech/datagol-agent-harness#readme
+Keywords: agent,ai-agents,llm,anthropic,claude,openai,mcp,model-context-protocol,sandboxing,langsmith,multi-agent,streaming
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: anthropic>=0.83.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: httpx>=0.27.0
+Provides-Extra: langsmith
+Requires-Dist: langsmith>=0.1.0; extra == "langsmith"
+Provides-Extra: openai
+Requires-Dist: openai>=1.50.0; extra == "openai"
+Requires-Dist: tiktoken>=0.7.0; extra == "openai"
+Provides-Extra: mcp
+Requires-Dist: mcp[cli]>=1.0.0; extra == "mcp"
+Provides-Extra: docker
+Requires-Dist: docker>=7.0.0; extra == "docker"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110.0; extra == "server"
+Requires-Dist: uvicorn[standard]>=0.27.0; extra == "server"
+Requires-Dist: python-dotenv>=1.0.0; extra == "server"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Provides-Extra: all
+Requires-Dist: langsmith>=0.1.0; extra == "all"
+Requires-Dist: openai>=1.50.0; extra == "all"
+Requires-Dist: tiktoken>=0.7.0; extra == "all"
+Requires-Dist: mcp[cli]>=1.0.0; extra == "all"
+Requires-Dist: docker>=7.0.0; extra == "all"
+Requires-Dist: fastapi>=0.110.0; extra == "all"
+Requires-Dist: uvicorn[standard]>=0.27.0; extra == "all"
+Requires-Dist: python-dotenv>=1.0.0; extra == "all"
+
+# DataGOL Agent Harness
+
+[![PyPI version](https://img.shields.io/pypi/v/datagol-agent-harness.svg)](https://pypi.org/project/datagol-agent-harness/)
+[![Python versions](https://img.shields.io/pypi/pyversions/datagol-agent-harness.svg)](https://pypi.org/project/datagol-agent-harness/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+
+**A provider-agnostic, library-first Python toolkit for building production-grade LLM agents.**
+
+Rather than locking developers into rigid state-machine graphs or opaque persona prompts, `datagol-agent-harness` gives you a composable set of building blocks: real-time streaming, typed tool registration with schema inference, lazy-loaded skills, multi-agent delegation, 4-tier memory, sandboxed code execution, Model Context Protocol (MCP) tool bridges, native LangSmith tracing, and evaluation suites.
+
+### Why DataGOL Agent Harness?
+
+- **No Graph Boilerplate:** Write simple async Python functions instead of complex state graphs.
+- **Provider-Agnostic:** First-class support for Anthropic (Claude 3.5/3.7/Sonnet/Opus) and OpenAI (GPT-4o/GPT-5).
+- **Streaming-First:** First-class typed event stream (`run_stream`) for WebSocket and SSE frontends.
+- **Lazy Instruction Packs (Skills):** Load specialized guidelines only when needed via YAML-frontmatter `SKILL.md` packs.
+- **Enterprise Guardrails & Sandboxing:** Execute untrusted code safely via Process, Docker, or macOS Seatbelt isolation with granular budget and iteration caps.
+- **MCP Native:** Connect to any Model Context Protocol server (stdio subprocess or remote SSE) in 3 lines of code.
+- **Full Observability & Evals:** Zero-overhead lifecycle hooks, middleware transforms, native LangSmith tracing, and benchmark evaluations.
+
+---
+
+- [Setup](#setup)
+- [1. Creating an agent](#1-creating-an-agent)
+- [2. Registering tools](#2-registering-tools)
+- [3. Streaming agents](#3-streaming-agents)
+- [4. Multi-agent orchestration](#4-multi-agent-orchestration)
+- [5. Skills](#5-skills)
+- [6. Memory](#6-memory)
+- [7. Permissions and guardrails](#7-permissions-and-guardrails)
+- [8. Hooks and middleware](#8-hooks-and-middleware)
+- [9. MCP servers](#9-mcp-servers)
+- [10. Session persistence](#10-session-persistence)
+- [11. Shipping a web app](#11-shipping-a-web-app)
+- [12. Evaluations with LangSmith](#12-evaluations-with-langsmith)
+- [API reference (quick)](#api-reference-quick)
+
+---
+
+## Setup
+
+```bash
+# From PyPI
+pip install datagol-agent-harness
+
+# With optional extras (LangSmith, OpenAI, MCP, Docker, or everything)
+pip install "datagol-agent-harness[langsmith]"
+pip install "datagol-agent-harness[openai]"
+pip install "datagol-agent-harness[all]"
+
+# For local development (editable)
+pip install -e ".[all]"
+```
+
+Set your key (a root `.env` is auto-loaded by the `examples` package):
+
+```bash
+ANTHROPIC_API_KEY=sk-ant-...
+# or, for OpenAI-backed agents:
+OPENAI_API_KEY=sk-...
+```
+
+Run examples as modules from the repo root (not as plain scripts):
+
+```bash
+python -m examples.simple_chat
+```
+
+---
+
+## 1. Creating an agent
+
+`Agent` is the core agentic loop: LLM reasoning → tool execution → repeat,
+until the model stops with `end_turn`.
+
+```python
+import asyncio
+from datagol_agent_harness import Agent, AgentConfig
+
+agent = Agent(
+    config=AgentConfig(
+        model="claude-sonnet-4-6",      # or "gpt-5"
+        provider="anthropic",            # or "openai"
+        system_prompt="You are a helpful assistant.",
+        max_tokens=8192,
+        max_iterations=50,               # loop guardrail
+        temperature=0.0,
+    )
+)
+
+print(asyncio.run(agent.run("What is 17 + 25?")))
+```
+
+`AgentConfig` fields (all optional, these are the defaults):
+
+| Field | Default | Purpose |
+|---|---|---|
+| `model` | `"claude-sonnet-4-6"` | Model id passed to the provider |
+| `provider` | `"anthropic"` | `"anthropic"` or `"openai"` |
+| `max_tokens` | `8192` | Per-response token cap |
+| `max_iterations` | `50` | Max loop iterations before `MaxIterationsError` |
+| `system_prompt` | `"You are a helpful assistant."` | System prompt |
+| `temperature` | `0.0` | Sampling temperature |
+| `max_result_chars` | `12000` | Tool-result eviction threshold in memory |
+
+You can also inject your own pieces — everything is a constructor parameter:
+
+```python
+from datagol_agent_harness import (
+    Agent, AgentConfig, ConversationMemory, HookManager,
+    MiddlewarePipeline, PermissionManager, ToolRegistry,
+)
+
+agent = Agent(
+    config=AgentConfig(system_prompt="..."),
+    tools=ToolRegistry(),               # your own registry
+    memory=ConversationMemory(),        # your own memory
+    permissions=PermissionManager(),    # your own permission policy
+    hooks=HookManager(),                # lifecycle hooks
+    middleware=MiddlewarePipeline(),    # request/response transforms
+)
+```
+
+### Conversation loop
+
+`agent.run()` keeps the conversation in memory, so calling it repeatedly is a
+multi-turn chat:
+
+```python
+async def chat(agent):
+    while True:
+        text = input("You: ").strip()
+        if text in ("", "quit"):
+            break
+        print("Agent:", await agent.run(text))
+```
+
+---
+
+## 2. Registering tools
+
+A tool is any Python function. The registry converts its signature, type
+hints, and docstring into a JSON schema for the model, and dispatches calls
+back to the function.
+
+### Decorator registration
+
+```python
+from datagol_agent_harness import PermissionLevel
+
+@agent.tools.register(permission=PermissionLevel.ALLOW)
+def add(a: int, b: int) -> int:
+    """Add two numbers."""          # becomes the tool description
+    return a + b
+
+@agent.tools.register(permission=PermissionLevel.ASK)
+async def read_file(path: str) -> str:
+    """Read a file from disk.
+
+    Args:
+        path: Absolute path to the file.   # becomes the param description
+    """
+    with open(path) as f:
+        return f.read()
+```
+
+Rules of thumb:
+
+- The **first line of the docstring** is the tool description the model sees.
+- An `Args:` section documents parameters.
+- Type hints drive the JSON schema: `str`, `int`, `float`, `bool`, `list[X]`,
+  `dict`. Optional params (with defaults) are not required.
+- **Pitfall:** avoid `list[str] | None` style unions — the schema generator
+  can't express them and will fall back to `"type": "string"`. Give optional
+  params a plain default and type instead, or use `register_with_schema`
+  (below) for exact control.
+- Sync and async handlers both work. Errors never crash the agent — they are
+  returned to the model as `is_error` tool results.
+
+### Explicit schema registration
+
+When you need a schema the signature can't express (nested objects, arrays
+with item types, `oneOf`-style choices):
+
+```python
+async def propose_items(query: str = "", items=None, intro_text: str = "") -> str:
+    if (query and items) or (not query and not items):
+        return "Error: provide exactly one of `query` or `items`."
+    resolved = [query] if query else items
+    return f"Presented {len(resolved)} item(s)."
+
+agent.tools.register_with_schema(
+    name="propose_items",
+    description="Propose items, via query (user's words) or items (your strings).",
+    input_schema={
+        "type": "object",
+        "properties": {
+            "query": {"type": "string", "description": "User's words, untouched."},
+            "items": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Item strings you wrote yourself.",
+            },
+            "intro_text": {"type": "string"},
+        },
+    },
+    handler=propose_items,
+    permission=PermissionLevel.ALLOW,
+)
+```
+
+### Built-in tools
+
+`datagol_agent_harness.builtin` ships ready-made tool sets:
+
+```python
+from datagol_agent_harness.builtin import register_all_tools
+
+register_all_tools(agent.tools)
+# read_file, write_file, list_directory, run_bash, fetch_url, memory tools...
+```
+
+Or individually: `register_filesystem_tools`, `register_bash_tools`,
+`register_web_tools`, `register_memory_tools`.
+
+### Inspecting and executing
+
+```python
+agent.tools.list_tools()                 # ['add', 'read_file', ...]
+agent.tools.get_tool_params()            # Anthropic-format tool schemas
+result = await agent.tools.execute(call) # manual dispatch (ToolCall -> ToolResult)
+```
+
+---
+
+## 3. Streaming agents
+
+`StreamingAgent` yields typed events as the model generates, instead of
+returning one final string. It mirrors `Agent`'s constructor.
+
+```python
+import asyncio
+from datagol_agent_harness import AgentConfig, StreamingAgent, StreamEventType
+
+agent = StreamingAgent(config=AgentConfig(system_prompt="You are helpful."))
+
+async def main():
+    async for event in agent.run_stream("Tell me a story"):
+        if event.type == StreamEventType.TEXT_DELTA:
+            print(event.data, end="", flush=True)
+        elif event.type == StreamEventType.TOOL_CALL_START:
+            print(f"\n> calling {event.data.name}({event.data.input})")
+        elif event.type == StreamEventType.TOOL_RESULT:
+            if event.data.is_error:
+                print(f"  tool error: {event.data.content}")
+        elif event.type == StreamEventType.TURN_COMPLETE:
+            print()
+
+asyncio.run(main())
+```
+
+Event types: `TEXT_DELTA`, `TEXT_COMPLETE`, `TOOL_CALL_START`,
+`TOOL_CALL_COMPLETE`, `TOOL_RESULT`, `THINKING_DELTA`, `TURN_COMPLETE`,
+`ERROR`.
+
+Usage stats accumulate on the guardrails engine:
+
+```python
+agent.guardrails.usage_summary
+# {'iterations': 3, 'input_tokens': 5120, 'output_tokens': 640, 'estimated_cost': '$0.0250'}
+```
+
+---
+
+## 4. Multi-agent orchestration
+
+There is no special "multi-agent" class — a sub-agent is just an `Agent`
+wrapped in a tool. The orchestrator's LLM decides when to delegate; the tool
+runs the specialist to completion and returns its text as the tool result.
+
+```python
+import asyncio
+from datagol_agent_harness import Agent, AgentConfig, PermissionLevel
+from datagol_agent_harness.builtin.web import register_web_tools
+
+
+async def run_research_agent(query: str) -> str:
+    """A specialist with its own prompt, tools, and iteration budget."""
+    researcher = Agent(
+        config=AgentConfig(
+            system_prompt="You are a research specialist. Cite sources.",
+            max_iterations=10,
+        )
+    )
+    register_web_tools(researcher.tools)
+    researcher.permissions.set_permission("fetch_url", PermissionLevel.ALLOW)
+    return await researcher.run(query)
+
+
+orchestrator = Agent(
+    config=AgentConfig(
+        system_prompt=(
+            "You manage a team of specialists. Delegate research to "
+            "delegate_research; synthesize results for the user."
+        ),
+        max_iterations=20,
+    )
+)
+
+@orchestrator.tools.register(permission=PermissionLevel.ALLOW)
+async def delegate_research(query: str) -> str:
+    """Delegate a research question to the research specialist.
+
+    Args:
+        query: The research question.
+    """
+    return await run_research_agent(query)
+
+
+print(asyncio.run(orchestrator.run("Compare SQLite and DuckDB for analytics.")))
+```
+
+Patterns that work well:
+
+- **Give each specialist a tight system prompt and a small tool set.** The
+  orchestrator's prompt should list its delegation tools and when to use them.
+- **Cap `max_iterations` per specialist** so a runaway sub-agent can't burn
+  the budget.
+- Sub-agents are created per call (stateless). If a specialist needs
+  continuity, keep one instance and reuse it across calls.
+- Sub-agents can themselves delegate — the pattern composes.
+
+A full working version with three specialists lives in
+`examples/multi_agent.py` (`python -m examples.multi_agent`).
+
+---
+
+## 5. Skills
+
+A skill is a markdown file with YAML frontmatter — a way to ship reusable
+instruction packs that load **lazily**: the agent only sees each skill's
+name + description in its system prompt, and pulls the full body into context
+by calling the `Skill` tool when a request matches.
+
+### Authoring a skill
+
+`skills/commit-message/SKILL.md`:
+
+```markdown
+---
+name: commit-message
+description: Write a conventional commit message for a described change.
+---
+
+You write commit messages. Rules:
+- Conventional Commits format: type(scope): subject
+- Subject in imperative mood, <= 72 chars, no trailing period
+- Add a body only when the "why" isn't obvious from the subject
+```
+
+A path can be a folder containing `SKILL.md`, or a markdown file directly.
+
+### Loading skills into an agent
+
+```python
+from datagol_agent_harness import Agent, AgentConfig
+
+agent = Agent(
+    config=AgentConfig(
+        system_prompt=(
+            "You are an engineering assistant. When a request matches a "
+            "skill in <available-skills>, call the Skill tool first to load "
+            "its instructions, then follow them."
+        ),
+    ),
+    skills=["./skills/commit-message", "./skills/code-review"],
+)
+```
+
+The `skills=` argument accepts a list of paths or a `SkillManager`:
+
+```python
+from datagol_agent_harness import SkillManager
+
+manager = SkillManager.from_paths(["./skills/commit-message"])
+agent = Agent(config=AgentConfig(system_prompt="..."), skills=manager)
+
+for s in agent.skills.list():
+    print(s.name, "-", s.description)
+```
+
+### Reacting to skill invocation
+
+A `SKILL_INVOKED` hook fires when the model loads a skill:
+
+```python
+from datagol_agent_harness import HookContext, HookEvent
+
+async def on_skill(ctx: HookContext):
+    if ctx.data.get("found"):
+        print(f"loaded skill: {ctx.data['skill']} ({ctx.data['body_chars']} chars)")
+
+agent.hooks.on(HookEvent.SKILL_INVOKED, on_skill)
+```
+
+Full demo: `examples/skills_agent.py`, with sample skills under
+`examples/skills/`.
+
+---
+
+## 6. Memory
+
+Three layers, all optional:
+
+### Conversation memory (default)
+
+Every agent has a `ConversationMemory` that stores the message list and trims
+itself when approaching the context limit. You rarely touch it directly, but
+you can:
+
+```python
+agent.memory.get_messages()        # current message list
+agent.memory.set_messages(saved)   # restore
+```
+
+### Agent memory — markdown notes on disk
+
+`AgentMemory` persists titled, tagged notes as markdown files. You expose it
+to the model through tools you write:
+
+```python
+from datagol_agent_harness import AgentMemory, PermissionLevel
+
+memory = AgentMemory(storage_dir=".agent_memory/agent")
+
+@agent.tools.register(permission=PermissionLevel.ALLOW)
+async def save_memory(title: str, content: str, tags: str = "") -> str:
+    """Save a note to persistent memory.
+
+    Args:
+        title: Short title for the memory.
+        content: The information to remember.
+        tags: Comma-separated tags.
+    """
+    tag_list = [t.strip() for t in tags.split(",") if t.strip()]
+    filename = memory.save(title, content, tags=tag_list)
+    return f"Saved '{title}' -> {filename}"
+
+@agent.tools.register(permission=PermissionLevel.ALLOW)
+async def search_memory(query: str = "") -> str:
+    """Search saved memories by keyword."""
+    results = memory.search(query=query)
+    return "\n".join(f"- {r['title']} [{r['filename']}]" for r in results) or "None found."
+```
+
+### Long-term memory — structured facts
+
+`LongTermMemory` stores categorized facts with ids:
+
+```python
+from datagol_agent_harness import LongTermMemory
+
+long_term = LongTermMemory(storage_dir=".agent_memory/long_term")
+fact_id = long_term.save("User prefers dark mode", category="preference")
+long_term.search(query="dark mode")
+long_term.update(fact_id, "User prefers light mode")
+```
+
+Tell the agent about these tools in the system prompt and when to use them
+("proactively save useful information"). A complete two-layer example is in
+`examples/memory_agent.py`.
+
+---
+
+## 7. Permissions and guardrails
+
+### Permission levels
+
+Every tool has a level: `ALLOW` (run silently), `ASK` (prompt the user
+y/n/always), or `DENY` (never run). The default at registration is `ASK`.
+
+```python
+from datagol_agent_harness import PermissionLevel
+
+# At registration
+@agent.tools.register(permission=PermissionLevel.ALLOW)
+def add(a: int, b: int) -> int: ...
+
+# Or override later
+agent.permissions.set_permission("run_bash", PermissionLevel.ASK)
+agent.permissions.grant_session("read_file")   # pre-approve for this session
+```
+
+`ASK` prompts on stdin in terminal apps. In a server context, register tools
+as `ALLOW` (or build your own `PermissionManager` subclass that asks over
+your transport).
+
+### Guardrails
+
+```python
+from datagol_agent_harness import GuardrailsEngine, MaxIterationsError, CostLimitError
+
+agent.guardrails.max_cost_dollars = 0.50   # raises CostLimitError past this
+agent.guardrails.usage_summary             # tokens + estimated cost so far
+agent.guardrails.reset()                   # reset counters
+```
+
+`max_iterations` comes from `AgentConfig` and raises `MaxIterationsError`.
+
+---
+
+## 8. Hooks and middleware
+
+### Hooks — observe the lifecycle
+
+```python
+from datagol_agent_harness import HookContext, HookEvent
+
+@agent.hooks.before_tool
+async def log_call(ctx: HookContext):
+    tc = ctx.data["tool_call"]
+    print(f"-> {tc.name}({tc.input})")
+
+@agent.hooks.after_tool
+async def log_result(ctx: HookContext):
+    result = ctx.data["result"]
+    if result.is_error:
+        print(f"tool failed: {result.content}")
+
+agent.hooks.on(HookEvent.LLM_RESPONSE, lambda ctx: print(ctx.data["stop_reason"]))
+```
+
+Events: `AGENT_START`, `AGENT_END`, `LOOP_ITERATION_START`,
+`LOOP_ITERATION_END`, `LLM_REQUEST`, `LLM_RESPONSE`, `TOOL_CALL_START`,
+`TOOL_CALL_END`, `SKILL_INVOKED`, `SANDBOX_EXEC`, `CHECKPOINT`, `ERROR`.
+
+### Middleware — transform requests and results
+
+Subclass `Middleware` to mutate messages, tool calls, or results as they flow:
+
+```python
+from datagol_agent_harness import Middleware
+
+class RedactSecrets(Middleware):
+    async def before_tool_execution(self, tool_call):
+        if "api_key" in str(tool_call.input):
+            tool_call.input = {k: ("***" if "key" in k else v)
+                               for k, v in tool_call.input.items()}
+        return tool_call
+
+agent.middleware.add(RedactSecrets())
+```
+
+Overridable stages: `before_llm_call`, `after_llm_call`,
+`before_tool_execution`, `after_tool_execution`.
+
+---
+
+## 9. MCP servers
+
+`MCPManager` connects to Model Context Protocol servers (stdio subprocess or
+remote SSE) and registers their tools into your agent's registry, so MCP
+tools look exactly like native tools to the model.
+
+```python
+from datagol_agent_harness import Agent, AgentConfig, MCPManager
+
+mcp = MCPManager()
+await mcp.connect(
+    "filesystem",
+    command="npx",
+    args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp"],
+)
+# or a remote server:
+# await mcp.connect("remote", url="http://localhost:8000/sse")
+
+agent = Agent(config=AgentConfig(system_prompt="..."), mcp=mcp)
+mcp.register_tools(agent.tools)
+
+print(await agent.run("What's in /tmp?"))
+await mcp.disconnect_all()
+```
+
+See `examples/mcp_agent.py` for the interactive version.
+
+---
+
+## 10. Session persistence
+
+Save and restore an agent's conversation:
+
+```python
+session_id = await agent.save_session()            # -> writes .agent_sessions/
+
+# later, or in another process:
+from datagol_agent_harness import Agent
+restored = await Agent.load_session(session_id, config=AgentConfig(system_prompt="..."))
+await restored.run("Where were we?")
+```
+
+For managed multi-session services (checkpointing, pause/resume, expiry),
+see `AgentRuntime` in `datagol_agent_harness/runtime.py`.
+
+---
+
+## 11. Shipping a web app
+
+The harness is transport-agnostic. The reference implementation in
+`examples/web_app` demonstrates:
+
+1. **Streaming agent execution** translated to Server-Sent Events (SSE) or WebSockets:
+
+```python
+from fastapi import FastAPI
+from fastapi.responses import StreamingResponse
+from datagol_agent_harness import AgentConfig, StreamingAgent, StreamEventType
+import json
+
+app = FastAPI()
+agent = StreamingAgent(config=AgentConfig(system_prompt="You are helpful."))
+
+@app.post("/api/stream")
+async def stream(req: dict):
+    async def event_generator():
+        async for event in agent.run_stream(req["message"]):
+            if event.type == StreamEventType.TEXT_DELTA:
+                yield f"data: {json.dumps({'type': 'text_delta', 'content': event.data})}\n\n"
+            elif event.type == StreamEventType.TOOL_CALL_START:
+                yield f"data: {json.dumps({'type': 'tool_call', 'name': event.data.name})}\n\n"
+        yield "data: [DONE]\n\n"
+
+    return StreamingResponse(event_generator(), media_type="text/event-stream")
+```
+
+2. **Register tools as `PermissionLevel.ALLOW`** — in a server context, tools should be pre-approved or gated via an authorization middleware rather than interactive stdin prompts.
+3. **Built-in UI and MCP integrations** — the web app serves a complete browser UI that displays tool calls, token usage, MCP servers, and dynamic skills.
+
+Run the full example from the repo root:
+
+```bash
+uvicorn examples.web_app.server:app --port 8000 --reload
+# open http://localhost:8000
+```
+
+---
+
+## 12. Evaluations with LangSmith
+
+The harness includes first-class evaluation capabilities powered by the
+**LangSmith evaluation framework** (`evaluate` / `aevaluate`). You can benchmark
+tool selection, skill routing, multi-agent delegation, and guardrails either
+locally (offline, zero-cost) or in the LangSmith Web UI.
+
+### Running an evaluation
+
+```python
+from datagol_agent_harness import Agent, AgentConfig, PermissionLevel
+from datagol_agent_harness.evals import (
+    build_example,
+    default_evaluators,
+    evaluate_agent,
+    tool_selection_evaluator,
+    contains_evaluator,
+)
+
+# 1. Define your agent factory or instance
+def make_agent(inputs):
+    agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))
+    @agent.tools.register(permission=PermissionLevel.ALLOW)
+    def calculate(expression: str) -> str:
+        return str(eval(expression, {"__builtins__": None}, {}))
+    return agent
+
+# 2. Define test cases or use built-in suites ("tool_calling", "skills", "all")
+dataset = [
+    build_example(
+        inputs={"prompt": "What is 144 / 12?"},
+        outputs={
+            "expected_tools": ["calculate"],
+            "contains_all": ["12"],
+            "max_allowed_iterations": 3,
+        },
+    )
+]
+
+# 3. Evaluate (offline for fast CI/CD tests, or live to sync to LangSmith)
+summary = evaluate_agent(
+    agent=make_agent,
+    dataset=dataset,
+    evaluators=[tool_selection_evaluator, contains_evaluator],
+    experiment_prefix="math-agent-benchmark",
+    offline=True,   # set False or omit when LANGSMITH_API_KEY is present
+)
+
+print(f"Pass rate: {summary.pass_rate * 100:.1f}%")
+```
+
+### Running evals from the CLI
+
+```bash
+# Run tool calling benchmarks offline (no API keys required)
+python -m datagol_agent_harness.evals.cli --suite tool_calling --offline
+
+# Run against Anthropic and upload results + traces to LangSmith
+export LANGSMITH_API_KEY="lsv2_pt_..."
+python -m datagol_agent_harness.evals.cli --suite skills --model claude-sonnet-4-6 --upload
+
+# Run all benchmark suites with concurrency
+python -m datagol_agent_harness.evals.cli --suite all --concurrency 2
+```
+
+When uploaded, the CLI prints a clickable LangSmith URL to inspect row-level scores,
+side-by-side prompt diffs, and the complete nested execution tree for every turn.
+
+---
+
+## API reference (quick)
+
+| Class / function | Module | Purpose |
+|---|---|---|
+| `Agent` | `datagol_agent_harness` | Core agentic loop (`await agent.run(msg)`) |
+| `StreamingAgent` | `datagol_agent_harness` | Event-streaming variant (`run_stream`) |
+| `AgentConfig` | `datagol_agent_harness` | Model, provider, limits, prompt |
+| `ToolRegistry` | `datagol_agent_harness` | `register`, `register_with_schema`, `execute` |
+| `PermissionLevel` | `datagol_agent_harness` | `ALLOW` / `ASK` / `DENY` |
+| `PermissionManager` | `datagol_agent_harness` | Per-tool overrides, session grants |
+| `GuardrailsEngine` | `datagol_agent_harness` | Iteration/cost limits, usage stats |
+| `ConversationMemory` | `datagol_agent_harness` | Message list with auto-trimming |
+| `AgentMemory` / `LongTermMemory` | `datagol_agent_harness` | Disk-backed notes / facts |
+| `PersistentMemory` | `datagol_agent_harness` | Session save/load |
+| `HookManager` / `HookEvent` | `datagol_agent_harness` | Lifecycle hooks |
+| `Middleware` / `MiddlewarePipeline` | `datagol_agent_harness` | Request/result transforms |
+| `SkillManager` | `datagol_agent_harness` | Lazy skill loading |
+| `MCPManager` | `datagol_agent_harness` | MCP server connections |
+| `Sandbox` | `datagol_agent_harness` | Sandboxed code execution |
+| `AgentRuntime` | `datagol_agent_harness` | Managed sessions, checkpoints |
+| `Extension` / `LangSmithExtension` | `datagol_agent_harness` | Pluggable runtime extensions / LangSmith tracing |
+| `evaluate_agent` | `datagol_agent_harness.evals` | LangSmith evaluation runner |
+| `AgentTarget` | `datagol_agent_harness.evals` | Target adapter with telemetry & trace linking |
+| `default_evaluators` | `datagol_agent_harness.evals` | Standard suite of evaluators |
+| `register_all_tools` | `datagol_agent_harness.builtin` | Filesystem, bash, web, memory tools |
+
+## Runnable examples
+
+| Example | Shows |
+|---|---|
+| `python -m examples.simple_chat` | Streaming interactive chat + basic tools |
+| `python -m examples.multi_agent` | Orchestrator + specialist agents |
+| `python -m examples.langsmith_tracing` | LangSmith lifecycle tracing + multi-agent nesting |
+| `python -m examples.run_evals` | Agent evaluation suite with LangSmith |
+| `python -m examples.skills_agent` | Lazy skill loading |
+| `python -m examples.memory_agent` | Two-layer persistent memory |
+| `python -m examples.mcp_agent` | MCP tool integration |
+| `python -m examples.coding_agent` | Full coding assistant |
+| `python -m examples.sandboxed_coder` | Sandboxed execution |
+| `uvicorn examples.web_app.server:app --port 8000` | Web app with streaming UI, MCP, and skills |