nodloop 0.8.0__tar.gz → 0.10.0__tar.gz

{nodloop-0.8.0 → nodloop-0.10.0}/.gitignore

@@ -17,7 +17,7 @@ __pycache__/
 /.claude/nodloop/
 .claude/settings.local.json
 CLAUDE.local.md
-/tmp
+/tmp/
 
 # environment
 .env*

nodloop-0.10.0/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.5
+Name: nodloop
+Version: 0.10.0
+Summary: Record AI runs and the verdicts people give on them, and hand the knowledge they approve to the next run in the same place
+Project-URL: Homepage, https://github.com/jeon-jihyeon/nodloop
+Project-URL: Repository, https://github.com/jeon-jihyeon/nodloop
+License-Expression: MIT
+Keywords: agents,ai,feedback,human-in-the-loop,langchain,mcp,memory
+Requires-Python: >=3.10
+Requires-Dist: mcp>=2.0
+Provides-Extra: claude-agent
+Requires-Dist: claude-agent-sdk>=0.1; extra == 'claude-agent'
+Provides-Extra: langchain
+Requires-Dist: langchain>=1.0; extra == 'langchain'
+Provides-Extra: openai-agents
+Requires-Dist: openai-agents>=0.3.3; extra == 'openai-agents'
+Description-Content-Type: text/markdown
+
+# nodloop for Python
+
+Record what your agent answered, take a person's verdict on it, and hand the corrections they approve to the agent's next run in the same situation. This is the Python client of [nodloop](https://github.com/jeon-jihyeon/nodloop).
+
+```
+pip install nodloop
+pip install "nodloop[langchain]"      # or openai-agents, claude-agent
+```
+
+The client starts a local `nodloop mcp` over stdio, so no server runs. It uses `NODLOOP_BIN`, then the binary the Claude Code plugin keeps under `~/.nodloop/bin`, then `nodloop` on PATH, and otherwise downloads the release of its own version from GitHub and checks it against the release checksums. It requires Python 3.10 or newer and is tested on Linux with Python 3.12.
+
+## The loop
+
+```python
+from nodloop import Client
+
+acme = {"tenant": ["acme"], "task": ["refund"]}
+
+async with Client() as c:
+    knowledge = await c.knowledge("support-bot", acme)
+    prompt = f"{system_prompt}\n\n{knowledge.text}"  # empty text when nothing is approved
+    answer = await my_agent(prompt)
+    run = await c.record("support-bot", answer, labels=acme, applied=knowledge.items)
+
+    # later, when a person reads the answer
+    await c.judge(run, "reject", "ann", reason="the refund window was missing", reason_code="scope")
+    candidate = await c.propose("judgment", "Quote the refund window before the steps", from_run=run)
+```
+
+The person approves the candidate in a terminal of their own with `nodloop knowledge approve <id> --version <n>`, or the client approves it through a nodloop server under an approver key the person holds, added with `--person`. Over stdio `approve` records a proposal and changes nothing, since a program proves no person. From then on every `knowledge("support-bot", acme)` carries the item, and a run of another tenant never does. `judge` takes `approve`, `edit` with `edited=` holding the corrected output, `reject` with what was wrong, or `withdraw` to take back the verdict before it, and always the name of the reviewer.
+
+| Method | Returns |
+|---|---|
+| `knowledge(producer, labels, holdout_key)` | `Knowledge` with `items`, the `text` to put in the prompt and `withheld`. With a `holdout_key` such as `<service>:<hour>`, the `holdout` share of config.json gets no items and `withheld` instead |
+| `record(producer, output, *, labels, applied, subject, withheld, holdout_key, model)` | the run id a verdict cites. Pass `knowledge.withheld` as `withheld` and the `holdout_key` the run passed to `knowledge`. `report effect` counts only runs recorded with the key and counts a run without it as `undrawn`. `withheld` takes `Ref` or `Item` values. `model` names the one model the producer runs, which the OTLP export sends as `gen_ai.request.model`. Leave it out when the producer may switch models |
+| `judge(run, verdict, reviewer, reason, reason_code, edited)` | nothing. `reviewer` is required, so a verdict without a name never reads as a person's |
+| `propose(kind, content, from_run, producer, labels, new_labels, item_id, veto)` | `Candidate` with `id`, `version`, `status` and `overlaps`, the current items of the same kind one run may carry with it |
+| `approve(item_id, version, approver, labels=)` | Under a key of a nodloop server that a person holds, approves as proposed, or for the runs carrying `labels` when given, and `{}` for every run of the producer. Anywhere else it records a proposal the person is asked about and the status stays. `Approval` with `id`, `version`, `status`, `approver`, `veto`, `proposal` when a proposal was recorded, and `export_error` and `folder_error` when a step after the approval failed |
+| `check_call(tool, arguments, producer)` | `Decision` with `action` of allow, block or ask, `veto` and `reason` |
+| `outcome(run, result, confirmed_cause, note, reviewer)` | nothing. `result` is a `Result`, confirmed, refuted or inconclusive: what a real check found, apart from the verdict on the output. Over stdio `reviewer` is required |
+| `report(name)` | `Report` with `report`, the report as JSON, and `since`, the RFC3339 start of the runs it read or empty for every run. `name` is a `ReportName`: `loop`, `extract`, `critic`, `effect`, `health` or `replay`. `critic` is `{critics, reactions}`: the rows of the draft critics and of the reaction classifiers |
+
+A tool that refuses, such as a proposal from a run nobody corrected, raises `ToolError` with the reason.
+
+Every adapter takes an optional `holdout_key`, passes it to `knowledge` and `record` and records what came back as applied or withheld.
+
+## A stream producer with a holdout
+
+A producer that runs on a schedule, such as an anomaly check over each window of a stream, draws the holdout on a key it keeps for the run. Use `<service>:<time block of an hour or more>` so the windows of one block land in the same arm and a lesson seen in one window does not leak into a withheld window next to it.
+
+```python
+from datetime import datetime, timezone
+
+block = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")  # one hour
+orders = {"service": ["orders"]}
+knowledge = await c.knowledge("lag-check", orders, holdout_key=f"orders:{block}")
+answer = await check(window, knowledge.text)
+run = await c.record(
+    "lag-check", answer, labels=orders, applied=knowledge.items, withheld=knowledge.withheld, holdout_key=f"orders:{block}"
+)
+
+# later, when the cause was checked
+await c.outcome(run, "confirmed", confirmed_cause="partition 7 was hot", reviewer="ann")
+effect = (await c.report("effect")).report
+```
+
+The share comes from `holdout` in config.json. With `url` the share is the one in the config.json of the nodloop server, read when the server starts.
+
+## Adapters
+
+LangChain `create_agent` middleware fetches the items once per run and puts them in the system message, records the final answer and refuses a vetoed tool call:
+
+```python
+from nodloop.langchain import NodloopMiddleware
+
+mw = NodloopMiddleware(c, "support-bot", acme)
+agent = create_agent(model, tools, middleware=[mw])
+await agent.ainvoke({"messages": [...]})
+await c.judge(mw.run, "approve", "ann")
+```
+
+OpenAI Agents SDK hooks and a tool input guardrail:
+
+```python
+from nodloop.openai_agents import NodloopHooks, veto_guardrail
+
+hooks = NodloopHooks(c, "support-bot", acme)
+agent = Agent(name="support", instructions=hooks.instructions("You help with refunds"), tools=[...])
+await Runner.run(agent, "refund?", hooks=hooks)
+```
+
+Pass `veto_guardrail(c, "support-bot")` in a tool's `tool_input_guardrails`.
+
+Claude Agent SDK hooks, the same loop the Claude Code plugin runs:
+
+```python
+from nodloop.claude_agent import NodloopHooks
+
+hooks = NodloopHooks(c, "support-bot", acme, holdout_key=conversation_id)
+options = ClaudeAgentOptions(hooks=hooks.hooks())
+```
+
+Use one adapter instance per run or conversation, since it keeps the items it handed out and the id of the run it recorded.
+
+The Claude Agent hooks draw once per conversation since the key stays the same for every turn, so a whole conversation is in one arm. `report effect` counts every turn of one key as one draw unit, so the turns of a conversation widen its intervals instead of passing for independent samples.
+
+## A shared server
+
+`Client(url="https://nodloop.example/mcp", key="nl_...")` connects to `nodloop-server serve` instead of a local binary. The key decides the tenant and what the client may do, and an approval through it is recorded under the key's name.
+
+## More
+
+The [nodloop README](https://github.com/jeon-jihyeon/nodloop#readme) covers labels, vetoes, reports and the server. MIT licensed.

nodloop-0.10.0/README.md

@@ -0,0 +1,113 @@
+# nodloop for Python
+
+Record what your agent answered, take a person's verdict on it, and hand the corrections they approve to the agent's next run in the same situation. This is the Python client of [nodloop](https://github.com/jeon-jihyeon/nodloop).
+
+```
+pip install nodloop
+pip install "nodloop[langchain]"      # or openai-agents, claude-agent
+```
+
+The client starts a local `nodloop mcp` over stdio, so no server runs. It uses `NODLOOP_BIN`, then the binary the Claude Code plugin keeps under `~/.nodloop/bin`, then `nodloop` on PATH, and otherwise downloads the release of its own version from GitHub and checks it against the release checksums. It requires Python 3.10 or newer and is tested on Linux with Python 3.12.
+
+## The loop
+
+```python
+from nodloop import Client
+
+acme = {"tenant": ["acme"], "task": ["refund"]}
+
+async with Client() as c:
+    knowledge = await c.knowledge("support-bot", acme)
+    prompt = f"{system_prompt}\n\n{knowledge.text}"  # empty text when nothing is approved
+    answer = await my_agent(prompt)
+    run = await c.record("support-bot", answer, labels=acme, applied=knowledge.items)
+
+    # later, when a person reads the answer
+    await c.judge(run, "reject", "ann", reason="the refund window was missing", reason_code="scope")
+    candidate = await c.propose("judgment", "Quote the refund window before the steps", from_run=run)
+```
+
+The person approves the candidate in a terminal of their own with `nodloop knowledge approve <id> --version <n>`, or the client approves it through a nodloop server under an approver key the person holds, added with `--person`. Over stdio `approve` records a proposal and changes nothing, since a program proves no person. From then on every `knowledge("support-bot", acme)` carries the item, and a run of another tenant never does. `judge` takes `approve`, `edit` with `edited=` holding the corrected output, `reject` with what was wrong, or `withdraw` to take back the verdict before it, and always the name of the reviewer.
+
+| Method | Returns |
+|---|---|
+| `knowledge(producer, labels, holdout_key)` | `Knowledge` with `items`, the `text` to put in the prompt and `withheld`. With a `holdout_key` such as `<service>:<hour>`, the `holdout` share of config.json gets no items and `withheld` instead |
+| `record(producer, output, *, labels, applied, subject, withheld, holdout_key, model)` | the run id a verdict cites. Pass `knowledge.withheld` as `withheld` and the `holdout_key` the run passed to `knowledge`. `report effect` counts only runs recorded with the key and counts a run without it as `undrawn`. `withheld` takes `Ref` or `Item` values. `model` names the one model the producer runs, which the OTLP export sends as `gen_ai.request.model`. Leave it out when the producer may switch models |
+| `judge(run, verdict, reviewer, reason, reason_code, edited)` | nothing. `reviewer` is required, so a verdict without a name never reads as a person's |
+| `propose(kind, content, from_run, producer, labels, new_labels, item_id, veto)` | `Candidate` with `id`, `version`, `status` and `overlaps`, the current items of the same kind one run may carry with it |
+| `approve(item_id, version, approver, labels=)` | Under a key of a nodloop server that a person holds, approves as proposed, or for the runs carrying `labels` when given, and `{}` for every run of the producer. Anywhere else it records a proposal the person is asked about and the status stays. `Approval` with `id`, `version`, `status`, `approver`, `veto`, `proposal` when a proposal was recorded, and `export_error` and `folder_error` when a step after the approval failed |
+| `check_call(tool, arguments, producer)` | `Decision` with `action` of allow, block or ask, `veto` and `reason` |
+| `outcome(run, result, confirmed_cause, note, reviewer)` | nothing. `result` is a `Result`, confirmed, refuted or inconclusive: what a real check found, apart from the verdict on the output. Over stdio `reviewer` is required |
+| `report(name)` | `Report` with `report`, the report as JSON, and `since`, the RFC3339 start of the runs it read or empty for every run. `name` is a `ReportName`: `loop`, `extract`, `critic`, `effect`, `health` or `replay`. `critic` is `{critics, reactions}`: the rows of the draft critics and of the reaction classifiers |
+
+A tool that refuses, such as a proposal from a run nobody corrected, raises `ToolError` with the reason.
+
+Every adapter takes an optional `holdout_key`, passes it to `knowledge` and `record` and records what came back as applied or withheld.
+
+## A stream producer with a holdout
+
+A producer that runs on a schedule, such as an anomaly check over each window of a stream, draws the holdout on a key it keeps for the run. Use `<service>:<time block of an hour or more>` so the windows of one block land in the same arm and a lesson seen in one window does not leak into a withheld window next to it.
+
+```python
+from datetime import datetime, timezone
+
+block = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")  # one hour
+orders = {"service": ["orders"]}
+knowledge = await c.knowledge("lag-check", orders, holdout_key=f"orders:{block}")
+answer = await check(window, knowledge.text)
+run = await c.record(
+    "lag-check", answer, labels=orders, applied=knowledge.items, withheld=knowledge.withheld, holdout_key=f"orders:{block}"
+)
+
+# later, when the cause was checked
+await c.outcome(run, "confirmed", confirmed_cause="partition 7 was hot", reviewer="ann")
+effect = (await c.report("effect")).report
+```
+
+The share comes from `holdout` in config.json. With `url` the share is the one in the config.json of the nodloop server, read when the server starts.
+
+## Adapters
+
+LangChain `create_agent` middleware fetches the items once per run and puts them in the system message, records the final answer and refuses a vetoed tool call:
+
+```python
+from nodloop.langchain import NodloopMiddleware
+
+mw = NodloopMiddleware(c, "support-bot", acme)
+agent = create_agent(model, tools, middleware=[mw])
+await agent.ainvoke({"messages": [...]})
+await c.judge(mw.run, "approve", "ann")
+```
+
+OpenAI Agents SDK hooks and a tool input guardrail:
+
+```python
+from nodloop.openai_agents import NodloopHooks, veto_guardrail
+
+hooks = NodloopHooks(c, "support-bot", acme)
+agent = Agent(name="support", instructions=hooks.instructions("You help with refunds"), tools=[...])
+await Runner.run(agent, "refund?", hooks=hooks)
+```
+
+Pass `veto_guardrail(c, "support-bot")` in a tool's `tool_input_guardrails`.
+
+Claude Agent SDK hooks, the same loop the Claude Code plugin runs:
+
+```python
+from nodloop.claude_agent import NodloopHooks
+
+hooks = NodloopHooks(c, "support-bot", acme, holdout_key=conversation_id)
+options = ClaudeAgentOptions(hooks=hooks.hooks())
+```
+
+Use one adapter instance per run or conversation, since it keeps the items it handed out and the id of the run it recorded.
+
+The Claude Agent hooks draw once per conversation since the key stays the same for every turn, so a whole conversation is in one arm. `report effect` counts every turn of one key as one draw unit, so the turns of a conversation widen its intervals instead of passing for independent samples.
+
+## A shared server
+
+`Client(url="https://nodloop.example/mcp", key="nl_...")` connects to `nodloop-server serve` instead of a local binary. The key decides the tenant and what the client may do, and an approval through it is recorded under the key's name.
+
+## More
+
+The [nodloop README](https://github.com/jeon-jihyeon/nodloop#readme) covers labels, vetoes, reports and the server. MIT licensed.

{nodloop-0.8.0 → nodloop-0.10.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "nodloop"
-version = "0.8.0"
+version = "0.10.0"
 description = "Record AI runs and the verdicts people give on them, and hand the knowledge they approve to the next run in the same place"
 readme = "README.md"
 license = "MIT"

nodloop-0.10.0/src/nodloop/__init__.py

@@ -0,0 +1,38 @@
+"""Record AI runs and the verdicts people give on them, and hand the knowledge they approve to the next run in the same place"""
+
+from .client import (
+    Approval,
+    Candidate,
+    Client,
+    Decision,
+    Intervention,
+    InterventionAction,
+    Item,
+    Knowledge,
+    Page,
+    Ref,
+    Report,
+    ReportName,
+    Result,
+    ToolError,
+)
+from .records import AnyRecord, RecordKind
+
+__all__ = [
+    "AnyRecord",
+    "Approval",
+    "Candidate",
+    "Client",
+    "Decision",
+    "Intervention",
+    "InterventionAction",
+    "Item",
+    "Knowledge",
+    "Page",
+    "RecordKind",
+    "Ref",
+    "Report",
+    "ReportName",
+    "Result",
+    "ToolError",
+]

{nodloop-0.8.0 → nodloop-0.10.0}/src/nodloop/claude_agent.py

@@ -7,11 +7,12 @@
 
 from __future__ import annotations
 
+import time
 from typing import Any
 
 from claude_agent_sdk import HookMatcher
 
-from .client import Client, Item
+from .client import Client, Item, Ref
 
 Labels = dict[str, list[str]]
 
@@ -22,13 +23,20 @@ class NodloopHooks:
     Use one instance per conversation because it keeps the items of the turn and the id of the last run
     """
 
-    def __init__(self, client: Client, producer: str, labels: Labels | None = None) -> None:
+    def __init__(self, client: Client, producer: str, labels: Labels | None = None, holdout_key: str = "") -> None:
         self.client = client
         self.producer = producer
         self.labels = labels or {}
+        # Passed to knowledge so a share of conversations gets no item and report effect compares them
+        # The key stays the same for every turn so a whole conversation falls in one arm
+        # report effect counts the turns of one key as one draw unit so they weigh as a cluster and not as independent runs
+        self.holdout_key = holdout_key
         self.applied: list[Item] = []
+        self.withheld: list[Ref] = []
         # The run the last answer was recorded as so a verdict can cite it
         self.run: str = ""
+        # When the prompt of the turn came so the run records how long the turn took
+        self._started = time.monotonic()
 
     def hooks(self) -> dict[str, list[HookMatcher]]:
         return {
@@ -38,8 +46,9 @@ class NodloopHooks:
         }
 
     async def prompt(self, input_data: dict[str, Any], tool_use_id: str | None, context: Any) -> dict[str, Any]:
-        knowledge = await self.client.knowledge(self.producer, self.labels)
-        self.applied = knowledge.items
+        self._started = time.monotonic()
+        knowledge = await self.client.knowledge(self.producer, self.labels, self.holdout_key)
+        self.applied, self.withheld = knowledge.items, knowledge.withheld
         if not knowledge.text:
             return {}
         return {"hookSpecificOutput": {"hookEventName": "UserPromptSubmit", "additionalContext": knowledge.text}}
@@ -49,7 +58,15 @@ class NodloopHooks:
         # A turn without one records nothing
         answer = input_data.get("last_assistant_message") or ""
         if answer.strip():
-            self.run = await self.client.record(self.producer, answer, labels=self.labels, applied=self.applied)
+            self.run = await self.client.record(
+                self.producer,
+                answer,
+                labels=self.labels,
+                applied=self.applied,
+                withheld=self.withheld,
+                holdout_key=self.holdout_key,
+                duration_ms=int((time.monotonic() - self._started) * 1000),
+            )
         return {}
 
     async def pre_tool_use(self, input_data: dict[str, Any], tool_use_id: str | None, context: Any) -> dict[str, Any]: