@quantod/qq 0.3.6 → 1.0.0

package/dist/resources/guide.md

@@ -0,0 +1,491 @@
+# QQ — Complete Reference
+
+QQ is a persistent, crash-safe pipeline for multi-stage agentic workflows. It is a **state machine with concurrency guarantees**: stages are states, items move through them, and `claim → release` is an atomic transition only one agent can hold at a time. State survives session restarts and crashes.
+
+Use QQ when work fans out: web crawling, bulk API enrichment, research pipelines (find → filter → enrich), batch transforms, any task where multiple items move through multiple stages, possibly with concurrent agents working in parallel.
+
+---
+
+## Mental Model
+
+A **pipeline** holds **items**. Each item lives in exactly one **stage** at a time. Items are pushed once and never deleted — they move between stages, accumulating payload as they go.
+
+Stages are just labels. The pipeline has no built-in notion of "first" or "last" — you define the topology in your PIPELINES.md.
+
+Items have:
+- `id` — the deduplication key. Pushing an item with an existing id fails.
+- `stage` — the current state
+- `payload` — item data; append-only string by default; YAML by convention.
+- `seq` — the claim token, returned by `claim`, required to `release`
+- `priority` — float, higher = claimed first (default 0.0)
+
+---
+
+## The Core Loop
+
+```
+make_pipeline        → create a pipeline, get its name, once per setup
+push (N times)       → load items into a stage
+claim                → atomically lock one item (returns seq + payload)
+  ... do work ...
+release              → unlock item, move to next stage, append payload
+status               → monitor progress
+unstick              → recover stuck/lost items
+```
+
+**Orchestrator** creates the pipeline, pushes items, spawns subagents, monitors with `status`, calls `batch_read` to collect results, calls `unstick` after.
+
+**Subagent** claims one item, does its work, releases to the next stage, repeats until "no items to claim" — then stops and reports back.
+
+---
+
+## Orchestrator / Subagent Pattern
+
+Orchestrator example:
+```
+make_pipeline → "0529_a3f9c2"   (store this as Q)
+push Q/pending "url-1" payload: "url: https://a.com"
+push Q/pending "url-2" payload: "url: https://b.com"
+push Q/pending "url-3" payload: "url: https://c.com"
+```
+
+Spawn subagents with instructions like:
+> Claim an item from pipeline `0529_a3f9c2` stage `pending`. Open the URL. Extract name, price, availability. Release to `done` appending the fields. If the page fails, release to `failed` appending `error: <reason>`. Repeat until "No items to claim". Report back.
+
+Monitor with `status` — done when `pending: [0, 0]` and nothing claimed.
+
+---
+
+## MCP Tools — Full Reference
+
+### make_pipeline
+
+Creates a pipeline. Returns its name.
+
+Parameters:
+- `name` (string, optional) — custom name. Auto-generated (`MMDD_xxxxxx`) if omitted. Fails if name already taken.
+- `description` (string, optional) — plain-text description of the pipeline's purpose.
+
+Returns:
+```yaml
+pipeline: 0529_a3f9c2
+```
+
+---
+
+### push
+
+Pushes an item into a pipeline stage. Fails if an item with the same `id` already exists in the pipeline (deduplication).
+
+Parameters:
+- `pipeline` (string) — pipeline name
+- `stage` (string, optional) — destination stage. Supports sub-paths: `enriched/shortlisted`. Defaults to empty string if omitted.
+- `id` (string, optional) — stable unique identifier. Omit to use auto-generated seq number.
+- `payload` (string, optional) — payload string
+- `payloadFormat` (`yaml` | `json` | `text`, optional, default `yaml`) — `yaml`: validate as YAML. `json`: validate as JSON and minify to one line (appended payloads form valid JSONL). `text`: no validation.
+- `priority` (number, optional) — float, default 0.0. Higher = claimed first. Only set when meaningful.
+
+Returns:
+```yaml
+id: url-1
+```
+
+**ID strategy**: use the item's natural stable identity from its origin system — URL, database row id, API object id. Omit id only when items have no upstream identity.
+
+---
+
+### claim
+
+Atomically claims the highest-priority lowest-seq unclaimed item from a stage. Returns the item with its full payload.
+
+Parameters:
+- `pipeline` (string)
+- `stage` (string) — supports glob wildcards: `enriched/*`, `*`. Omit to claim from any stage.
+- `id` (string, optional) — claim a specific item by id. Fails if not found or already claimed.
+- `random` (boolean, optional) — claim a random unclaimed item instead of priority+seq order
+
+Returns:
+```yaml
+id: url-1
+stage: pending
+seq: 3
+priority: 0
+claimed: true
+created: 1748476800000
+last_modified: 1748476800000
+payload: |
+  url: https://example.com/product/42
+```
+
+Returns `error: no items to claim` when nothing is available. **Stop and report back when this happens** — do not retry.
+
+Errors (MCP YAML / SDK JSON):
+- `error: no items to claim` — stage is empty or all items are already claimed. Stop and report back.
+- `error: item <id> not found` — only when claiming by specific `id`
+- `error: item <id> already claimed` — only when claiming by specific `id`
+
+**Save the `seq` value** — it is your claim token, required to call `release`.
+
+---
+
+### release
+
+Releases a claimed item. Optionally moves it to a new stage and appends/replaces payload.
+
+Parameters:
+- `pipeline` (string)
+- `seq` (number) — the claim token from `claim`
+- `target` (string, optional) — destination stage. **Always specify when moving the item.** Omit to release back to the same stage.
+- `payload` (string, optional) — payload string to append (or replace)
+- `payloadFormat` (`yaml` | `json` | `text`, optional, default `yaml`) — same as `push`
+- `replace` (boolean, optional) — replace payload entirely instead of appending. Use when treating payload as a structured object.
+- `priority` (number, optional) — update the item's priority
+
+Returns:
+```yaml
+ok: true
+```
+
+Fails if the claim expired (item was unstuck while you worked) or item is not claimed.
+
+**Payload is append-only history by default.** Each stage appends its output — resulting duplicate YAML fields are intentional. Use `replace: true` only when treating payload as a structured object you own.
+
+---
+
+### status
+
+Returns the pipeline description and item counts per stage.
+
+Parameters:
+- `pipeline` (string) — pipeline name
+
+Returns:
+```yaml
+description: Scrape product pages and extract pricing
+stats:
+  '*': [23, 3]
+  done: [7, 0]
+  failed: [6, 0]
+  pending: [10, 3]
+```
+
+Each value is `[total, claimed]` — total item count and how many are currently claimed (being processed). `*` is the pipeline-level aggregate across all stages. For nested stages (`enriched/shortlisted`), intermediate paths are included as aggregates of their children. `description` is `null` if none was set.
+
+---
+
+### batch_read
+
+Reads items without claiming them. For inspection, data collection, and monitoring.
+
+Parameters:
+- `pipeline` (string)
+- `stage` (string, optional) — supports globs. Defaults to `*`
+- `includePayload` (boolean, optional) — include payload in results
+- `claimed` (boolean, optional) — filter by claimed status. Default: unclaimed
+- `ids` (string[], optional)
+- `createdAfter`, `createdBefore`, `modifiedAfter` (number, epoch ms)
+- `limit`, `offset` (number)
+
+Returns a YAML list of items. `payload` is omitted unless `includePayload: true`. Returns `[]` when nothing matches.
+
+```yaml
+- id: url-1
+  stage: done
+  claimed: false
+  seq: 5
+  priority: 0
+  created: 1748476800000
+  last_modified: 1748476801000
+- id: url-2
+  stage: done
+  claimed: false
+  seq: 7
+  priority: 0
+  created: 1748476800001
+  last_modified: 1748476802000
+```
+
+---
+
+### unstick
+
+Releases all stuck claimed items back to their stage. Use when agents have crashed and left items locked.
+
+Parameters:
+- `pipeline` (string)
+- `stage` (string, optional) — limit to a specific stage. Defaults to `*`
+- `ids`, `createdAfter`, `createdBefore`, `modifiedAfter` — optional filters
+
+Returns:
+```yaml
+unstuck: 3
+```
+
+---
+
+### delete_pipeline
+
+Deletes a pipeline and all its items. Irreversible. Only call if explicitly asked.
+
+Parameters:
+- `pipeline` (string)
+
+Returns:
+```yaml
+ok: true
+```
+
+---
+
+### get_sdk
+
+Returns a self-contained SDK snippet (JavaScript or Python) that scripts can use to interact with QQ's HTTP bridge from anywhere — including the Cowork VM or a browser's HTTP-origin tab.
+
+Parameters:
+- `language` ("javascript" | "python")
+
+Returns a code snippet to inline at the top of your script. The snippet uses only stdlib — no installs. It has the host IP, port, and token baked in for this MCP process's lifetime.
+
+**Call once per task.** Re-call only if a script throws `BridgeError: Connection failed` — that means this MCP server restarted and credentials changed.
+
+**JavaScript signatures:**
+```
+QQ.push(pipeline, stage, id?, payload?, opts?)       → Promise<{ id: string } | { error: string }>
+QQ.claim(pipeline, stage, opts?)                     → Promise<Item | { error: string }>
+QQ.release(pipeline, seq, opts?)                     → Promise<{ ok: true } | { error: string }>
+QQ.batch_read(pipeline, stage, opts?)                → Promise<Item[] | { error: string }>
+QQ.status(pipeline)                                  → Promise<{ description: string|null, stats: Record<string, [number,number]> } | { error: string }>
+```
+
+- `push` opts: `priority` (number), `payloadFormat` (`"yaml"` | `"json"` | `"text"`)
+- `claim` opts: `id` (string — claim a specific item), `random` (boolean)
+- `release` opts: `target` (string), `payload` (string), `replace` (boolean), `priority` (number), `payloadFormat`
+- `batch_read` opts: `includePayload` (boolean), `claimed` (boolean), `ids` (string, comma-separated), `createdAfter`, `createdBefore`, `modifiedAfter` (epoch ms), `limit`, `offset` (number)
+
+All functions return `{ error: string }` on failure — never throw (except `BridgeError` for network connection failures). Check `result.error` before using the result.
+
+`claim` returns `{ error: "no items to claim" }` when nothing is available.
+
+Item shape (returned by `claim` and `batch_read`):
+```
+{ id: string, stage: string, seq: number, priority: number, claimed: boolean,
+  created: number, last_modified: number, payload?: string | null }
+```
+
+`claim` always includes `payload`. `batch_read` omits `payload` unless `includePayload: true`.
+
+**Python signatures:**
+```
+qq_push(pipeline, stage, id=None, payload=None, **opts)  → { 'id': str } | { 'error': str }
+qq_claim(pipeline, stage, **opts)                        → dict | { 'error': str }
+qq_release(pipeline, seq, **opts)                        → { 'ok': True } | { 'error': str }
+qq_batch_read(pipeline, stage, **opts)                   → list[dict] | { 'error': str }
+qq_status(pipeline)                                      → { 'description': str|None, 'stats': dict } | { 'error': str }
+```
+
+All Python functions return `{ 'error': str }` on failure — never raise (except `BridgeError` for connection failures).
+
+---
+
+### load_file
+
+Loads multiple items from a file on the host filesystem into a pipeline. Each record in the file becomes one pipeline item.
+
+**Supported formats:** JSONL (`.jsonl`, `.ndjson`), JSON array (`.json`), YAML array (`.yaml`, `.yml`), CSV (`.csv`). Format is detected from the file extension.
+
+**`path`** supports `$VAR` environment variable expansion — e.g. `$HOME/Downloads/data.jsonl`.
+
+Parameters:
+- `path` (string) — path to the file. Supports `$VAR` expansion.
+- `pipeline` (string) — pipeline to push records into
+- `stage` (string, optional) — default stage for records that do not specify their own. Defaults to empty string.
+- `deleteAfter` (boolean, optional) — delete the file after loading. Default: false.
+
+**Record format** (all fields optional):
+```yaml
+id: item-1
+stage: pending
+payload: "any string"
+priority: 0.0
+```
+- `id` — omit for auto-generated id
+- `stage` — overrides the `stage` parameter for this record
+- `payload` — treated as an opaque string; non-string values are YAML-serialized
+- `priority` — float, default 0.0
+
+**Returns** one line per record:
+```
+item-1: ok
+item-2: duplicate
+item-3: error: <message>
+```
+
+File-level parse errors (bad JSON, malformed CSV, etc.) abort the entire operation. Per-record push failures are reported individually without aborting remaining records.
+
+---
+
+## SDK Usage
+
+Use the SDK when you need to run code that processes pipeline items programmatically — loops, transforms, bulk operations. It is faster and more efficient than calling MCP tools one at a time for batch work.
+
+### When to use SDK vs MCP tools
+
+- **MCP tools**: orchestration decisions, one-off operations, monitoring, anything requiring Claude's judgment
+- **SDK (via script)**: processing many items in a loop, transforming data, bulk pushes, anything that is pure computation
+
+### Typical workflow
+
+```javascript
+// 1. Call get_sdk once to get the snippet
+// 2. Write a script, inline the snippet at the top
+// 3. Run with: node script.js   or   python script.py
+
+// --- JavaScript example ---
+// [paste get_sdk output here]
+
+const pipeline = '0529_a3f9c2';
+
+// Bulk push — payload is a YAML string
+for (const url of urls) {
+  await QQ.push(pipeline, 'pending', url, `url: ${url}`);
+}
+
+// Claim loop — check result.error, not null
+while (true) {
+  const item = await QQ.claim(pipeline, 'pending');
+  if (item.error) break;                             // "no items to claim" or other error
+  const result = await processItem(item.payload);
+  await QQ.release(pipeline, item.seq, { target: 'done', payload: result });
+}
+
+// Collect results with payloads
+const done = await QQ.batch_read(pipeline, 'done', { includePayload: true });
+for (const item of done) {
+  console.log(item.id, item.payload);
+}
+
+// Monitor progress
+const s = await QQ.status(pipeline);
+console.log(`total=${s.stats['*'][0]} pending=${s.stats.pending?.[0] ?? 0}`);
+```
+
+### Error routing
+
+Release to `failed` on error, `done` on success — never leave items claimed when your work throws:
+
+```javascript
+while (true) {
+  const item = await QQ.claim(pipeline, 'pending');
+  if (item.error) break;
+  try {
+    const result = await processItem(item.payload);
+    await QQ.release(pipeline, item.seq, { target: 'done', payload: `result: ${result}` });
+  } catch (e) {
+    await QQ.release(pipeline, item.seq, { target: 'failed', payload: `error: ${e.message}` });
+  }
+}
+```
+
+### Structured JSON payloads
+
+Use `payloadFormat: "json"` when payload is a JSON object. Appended payloads form valid JSONL; `replace: true` treats it as a single mutable object:
+
+```javascript
+// Push with JSON payload
+await QQ.push(pipeline, 'pending', 'item-1', JSON.stringify({ url: 'https://a.com', score: 0 }),
+  { payloadFormat: 'json' });
+
+// Update a structured payload: parse → modify → replace
+const item = await QQ.claim(pipeline, 'pending');
+if (!item.error) {
+  const data = JSON.parse(item.payload);
+  data.score = await scoreItem(data.url);
+  await QQ.release(pipeline, item.seq, {
+    target: 'done',
+    payload: JSON.stringify(data),
+    payloadFormat: 'json',
+    replace: true,
+  });
+}
+```
+
+### Python example
+
+```python
+# [paste get_sdk output here]
+
+pipeline = "0529_a3f9c2"
+
+# Bulk push
+for url in urls:
+    qq_push(pipeline, "pending", url, f"url: {url}")
+
+# Claim loop
+while True:
+    item = qq_claim(pipeline, "pending")
+    if "error" in item:  # "no items to claim" or other error
+        break
+    result = process_item(item["payload"])
+    qq_release(pipeline, item["seq"], target="done", payload=result)
+
+items = qq_batch_read(pipeline, "done", includePayload=True)
+for item in items:
+    print(item["id"], item["payload"])
+
+# Monitor
+s = qq_status(pipeline)
+print(f"total={s['stats']['*'][0]}")
+```
+
+### BridgeError reconnect
+
+If a script throws `BridgeError: Connection failed`:
+1. Call `get_sdk` again to get fresh credentials
+2. Update the snippet in your script
+3. Re-run the script
+
+This happens when the QQ MCP server restarted (e.g. Claude Desktop was restarted).
+
+---
+
+## YAML Payload Conventions
+
+Payload is a YAML string appended to existing content by default. The result is a YAML document where fields may appear multiple times — this is intentional. Each stage's output appears in sequence, creating a trace of the item's journey.
+
+```yaml
+# After push:
+url: https://example.com/product/42
+
+# After first subagent appends:
+url: https://example.com/product/42
+name: Widget Pro
+price: 29.99
+available: true
+
+# After second subagent appends (review):
+url: https://example.com/product/42
+name: Widget Pro
+price: 29.99
+available: true
+review_reason: price seems anomalous
+```
+
+Use `replace: true` only when you own the payload as a structured object (e.g., accumulating messages in a chat pipeline). To update a structured payload: claim, parse with yaml.load, modify, release with replace.
+
+---
+
+## Globs, Filters, Priority
+
+**Stage globs**: `*` matches any single segment, `enriched/*` matches sub-stages. Use in `claim` and `batch_read` to work across multiple stages.
+
+**Priority**: float, default 0.0. Higher value = claimed first within the same stage. Use only when meaningful (e.g., score, confidence). Most pipelines don't need priority.
+
+**Filters in batch_read / unstick**: `claimed`, `ids`, `createdAfter`, `createdBefore`, `modifiedAfter`, `limit`, `offset`.
+
+---
+
+## Pipeline Design
+
+Before creating a new pipeline, read the `pipeline_design` resource. It walks through the design process, the PIPELINES.md template, a filled example, and a pre-implementation checklist.
+
+Document every pipeline in `PIPELINES.md` before pushing a single item.

package/dist/resources/pipeline_design.md

@@ -0,0 +1,192 @@
+# Pipeline Design Guide
+
+## The Iron Law
+
+```
+NO PIPELINE IMPLEMENTATION WITHOUT A PIPELINES.md ENTRY FIRST
+```
+
+Design the pipeline on paper before pushing a single item. Undocumented pipelines drift — stages accumulate, payload grows inconsistently, and future agents guess wrong about what each stage means.
+
+**Violating the letter of this process is violating the spirit of it.**
+
+---
+
+## When to Use This Guide
+
+Follow this guide whenever you are:
+- Creating a new pipeline
+- Adding stages to an existing pipeline
+- Changing what payload a stage produces
+
+**When working with an existing pipeline:** read `PIPELINES.md` first. It should contain everything you need. If it's missing or incomplete, update it before proceeding.
+
+---
+
+## Clarifying with the User
+
+Pipeline design usually happens with a human partner. Before filling in the template, ask the user to clarify what you don't know. **One question per message.** Prefer multiple-choice when the options are known.
+
+The five Phase 1 questions are your agenda — work through them in dialogue, not in isolation.
+
+**Good questions to ask:**
+- "What are the sources of items — where do they come from, and what identifies each one uniquely?"
+- "What should happen when an item fails — retry, discard, or route to a review stage?"
+- "Is there a natural stable id for each item (URL, database id), or should the pipeline assign one?"
+- "Are any stages optional, or does every item pass through all of them?"
+- "Who runs the orchestrator and when — on demand, scheduled, or triggered by an event?"
+
+**When to skip asking:**
+- The user has already provided enough detail to answer all five Phase 1 questions
+- You are working from a spec or design doc that covers the pipeline completely
+- The pipeline is a trivial two-stage transform with no ambiguous transitions
+
+If you skip asking, state why — don't silently proceed.
+
+---
+
+## Phase 1 — Design
+
+Answer all five questions before writing anything.
+
+### 1. What does this pipeline do?
+
+One sentence. If you need more than one sentence, the scope is unclear — break it down or narrow it.
+
+### 2. What are the stages?
+
+List every stage by name. For each: what does it mean for an item to be in this stage?
+
+Stages are not steps in a script — they are **states** an item can be in. Name them as nouns or past participles, not verbs.
+
+| Good stage names | Bad stage names |
+|-----------------|-----------------|
+| `pending`, `fetched`, `done` | `fetch`, `process`, `handle` |
+| `review/approved`, `review/rejected` | `reviewing`, `to_review` |
+
+Sub-stages (`review/approved`, `enriched/shortlisted`) are valid — use them when a stage has distinct sub-states that agents need to distinguish.
+
+### 3. What are the valid transitions?
+
+For each transition: which stage does it come from, which stage does it go to, and what did the agent do to earn the move?
+
+**Terminal stages** are stages with no outbound transitions — `done`, `failed`, `discarded`. Be explicit about which stages are terminal.
+
+**Loops** (releasing back to the same stage) are valid: use them for retry logic or chat-style accumulation. Document them explicitly.
+
+### 4. What is the item ID — and why?
+
+The `id` is the deduplication primitive. An item can only be pushed once per pipeline per id. Choose it carefully.
+
+| Situation | Use |
+|-----------|-----|
+| Items come from an external system | The external system's stable identifier (URL, database row id, API object id) |
+| Items have no upstream identifier | Omit `id` — the pipeline assigns a sequence number |
+| Items must be globally unique across runs | A hash or composite key (e.g., `sha256(url + date)`) |
+
+Never use a generated UUID as the id unless items genuinely have no stable identity.
+
+### 5. What does the payload accumulate?
+
+Payload is append-only history by default. Each stage appends its output.
+
+For each field that will appear in the payload:
+- Which stage introduces it?
+- What type/shape is it?
+- Is it ever replaced (use `replace: true`) rather than appended?
+
+---
+
+## Red Flags — Stop and Re-Design
+
+| Thought | Problem |
+|---------|---------|
+| "I'll figure out the stages as I go" | Undefined stages produce inconsistent item states |
+| "Payload structure is flexible" | Future agents can't reliably parse a payload with no schema |
+| "I'll use auto-generated IDs for now" | If items have a natural identity, you'll get duplicates across runs |
+| "I'll document after it works" | You won't, and future agents will guess wrong |
+| "The pipeline is simple, it doesn't need docs" | Undocumented simple pipelines become undocumented complex ones |
+| "I'll add the review stage later if we need it" | Retrofit stages invalidate existing items' stage semantics |
+| "Error handling is just releasing to `failed`" | What happens to items in `failed`? Document it. |
+
+---
+
+## Phase 2 — Document
+
+Write the PIPELINES.md entry using the template below. Every field is required.
+
+PIPELINES.md lives in the project root. One file documents all pipelines in the project.
+
+### PIPELINES.md Template
+
+```markdown
+## [Pipeline Name]
+
+**Description:** One sentence.
+
+**Purpose:** 2-3 sentences. Why does this pipeline exist? What triggers it?
+
+**Pipeline ID:** `[name or auto-generated pattern]`
+
+---
+
+### ID Strategy
+[What value is used as the item id and why.]
+
+---
+
+### Payload Schema
+
+| Field | Type | Added by stage | Description |
+|-------|------|----------------|-------------|
+
+---
+
+### Stages
+
+| Stage | Meaning | Terminal? |
+|-------|---------|-----------|
+
+---
+
+### Transitions
+
+| From | To | Agent action | Payload appended | Notes |
+|------|----|-------------|-----------------|-------|
+
+---
+
+### State Diagram
+
+```mermaid
+stateDiagram-v2
+    [*] --> pending
+    pending --> done : processed
+    done --> [*]
+```
+
+---
+
+### Process Notes
+
+**Orchestrator responsibilities:**
+**Subagent responsibilities:**
+**Retry logic:**
+**Error handling:**
+**Known edge cases:**
+```
+
+---
+
+## Phase 3 — Verify
+
+- [ ] Every stage has a defined meaning in the Stages table
+- [ ] Every valid transition appears in both the Transitions table and the Mermaid diagram
+- [ ] Terminal stages are identified
+- [ ] ID strategy is explicit and justified
+- [ ] Payload schema shows which stage introduces each field
+- [ ] Retry logic is documented
+- [ ] Orchestrator and subagent responsibilities are separated and complete
+- [ ] Process notes cover what happens to `failed` items
+
+If any item is unchecked: complete it before implementing.

package/dist/sdk-templates/javascript.d.ts

@@ -0,0 +1,2 @@
+export declare function renderJavaScriptSdk(host: string, port: number, token: string): string;
+//# sourceMappingURL=javascript.d.ts.map

package/dist/sdk-templates/javascript.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"javascript.d.ts","sourceRoot":"","sources":["../../src/sdk-templates/javascript.ts"],"names":[],"mappings":"AAAA,wBAAgB,mBAAmB,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAmBrF"}