@quantod/qq 1.0.8 → 1.0.9

package/dist/resources/pipeline_design.md

@@ -53,6 +53,8 @@ Answer all five questions before writing anything.
 
 One sentence. If you need more than one sentence, the scope is unclear — break it down or narrow it.
 
+The description should include the project name — it's the primary way to identify which project a pipeline belongs to. Keep it short: `[ProjectName] — [what it does]`.
+
 **Pipelines are global** — all projects share the same store. Always prefix the pipeline name with the current project identifier (e.g. `myapp_`, `proj_`) to avoid collisions. Use `make_pipeline` with an explicit `name` like `myapp_scrape_products`. Auto-generated names (`MMDD_xxxxxx`) are safe for one-off tasks but not for named, reusable pipelines.
 
 ### 2. What are the stages?
@@ -70,7 +72,9 @@ Sub-stages (`review/approved`, `enriched/shortlisted`) are valid — use them wh
 
 ### 3. What are the valid transitions?
 
-For each transition: which stage does it come from, which stage does it go to, and what did the agent do to earn the move?
+For each transition: from stage, to stage, the condition on the source item that triggers it, and the action the agent takes.
+
+Columns: **from**, **to**, **condition** (what must be true of the item), **action** (what the agent does), **notes**.
 
 **Terminal stages** are stages with no outbound transitions — `done`, `failed`, `discarded`. Be explicit about which stages are terminal.
 
@@ -80,11 +84,13 @@ For each transition: which stage does it come from, which stage does it go to, a
 
 The `id` is the deduplication primitive. An item can only be pushed once per pipeline per id. Choose it carefully.
 
+**Prefer extracted IDs over URLs.** URLs are unstable — the same resource may appear at a mobile URL, a REST API endpoint, a CDN path, or a redirect. Extract the platform's native object ID from the URL and use that instead. A numeric or slug ID survives URL structure changes; a URL does not.
+
 | Situation | Use |
 |-----------|-----|
-| Items come from an external system | The external system's stable identifier (URL, database row id, API object id) |
+| Items come from an external system | The platform's native object ID (e.g. post ID, product SKU) — extract from URL if needed |
 | Items have no upstream identifier | Omit `id` — the pipeline assigns a sequence number |
-| Items must be globally unique across runs | A hash or composite key (e.g., `sha256(url + date)`) |
+| Items must be globally unique across runs | A hash or composite key (e.g., `sha256(source_id + date)`) |
 
 Never use a generated UUID as the id unless items genuinely have no stable identity.
 
@@ -92,10 +98,18 @@ Never use a generated UUID as the id unless items genuinely have no stable ident
 
 Payload is append-only history by default. Each stage appends its output.
 
-For each field that will appear in the payload:
-- Which stage introduces it?
-- What type/shape is it?
-- Is it ever replaced (use `replace: true`) rather than appended?
+Document the payload as a YAML example that mirrors the actual structure. Use inline comments for fields that need explanation. No need to enumerate which stage adds each field — the structure itself is the schema.
+
+```yaml
+# Example payload schema
+id: "12345"
+title: "Product name"
+price: 29.99
+tags: [electronics, sale]
+summary: "One-paragraph AI summary"  # optional, added after enrichment
+```
+
+Is any field ever replaced rather than appended? Note it with a comment (`# replaced, not appended`).
 
 ---
 
@@ -124,7 +138,7 @@ PIPELINES.md lives in the project root. One file documents all pipelines in the
 ```markdown
 ## [Pipeline Name]
 
-**Description:** One sentence.
+**Description:** `[ProjectName] — [what it does]`
 
 **Purpose:** 2-3 sentences. Why does this pipeline exist? What triggers it?
 
@@ -139,22 +153,23 @@ PIPELINES.md lives in the project root. One file documents all pipelines in the
 
 ### Payload Schema
 
-| Field | Type | Added by stage | Description |
-|-------|------|----------------|-------------|
+```yaml
+# YAML example mirroring the actual payload structure
+```
 
 ---
 
 ### Stages
 
-| Stage | Meaning | Terminal? |
-|-------|---------|-----------|
+| Stage | Meaning |
+|-------|---------|
 
 ---
 
 ### Transitions
 
-| From | To | Agent action | Payload appended | Notes |
-|------|----|-------------|-----------------|-------|
+| From | To | Condition | Action | Notes |
+|------|----|-----------|--------|-------|
 
 ---
 
@@ -185,8 +200,8 @@ stateDiagram-v2
 - [ ] Every stage has a defined meaning in the Stages table
 - [ ] Every valid transition appears in both the Transitions table and the Mermaid diagram
 - [ ] Terminal stages are identified
-- [ ] ID strategy is explicit and justified
-- [ ] Payload schema shows which stage introduces each field
+- [ ] ID strategy is explicit and justified — stable object ID preferred over URL
+- [ ] Payload schema is a YAML example that reflects the actual structure
 - [ ] Retry logic is documented
 - [ ] Orchestrator and subagent responsibilities are separated and complete
 - [ ] Process notes cover what happens to `failed` items

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@quantod/qq",
-  "version": "1.0.8",
+  "version": "1.0.9",
   "description": "Persistent queue for multi-stage Claude agent pipelines",
   "license": "UNLICENSED",
   "engines": {

package/src/resources/pipeline_design.md

@@ -53,6 +53,8 @@ Answer all five questions before writing anything.
 
 One sentence. If you need more than one sentence, the scope is unclear — break it down or narrow it.
 
+The description should include the project name — it's the primary way to identify which project a pipeline belongs to. Keep it short: `[ProjectName] — [what it does]`.
+
 **Pipelines are global** — all projects share the same store. Always prefix the pipeline name with the current project identifier (e.g. `myapp_`, `proj_`) to avoid collisions. Use `make_pipeline` with an explicit `name` like `myapp_scrape_products`. Auto-generated names (`MMDD_xxxxxx`) are safe for one-off tasks but not for named, reusable pipelines.
 
 ### 2. What are the stages?
@@ -70,7 +72,9 @@ Sub-stages (`review/approved`, `enriched/shortlisted`) are valid — use them wh
 
 ### 3. What are the valid transitions?
 
-For each transition: which stage does it come from, which stage does it go to, and what did the agent do to earn the move?
+For each transition: from stage, to stage, the condition on the source item that triggers it, and the action the agent takes.
+
+Columns: **from**, **to**, **condition** (what must be true of the item), **action** (what the agent does), **notes**.
 
 **Terminal stages** are stages with no outbound transitions — `done`, `failed`, `discarded`. Be explicit about which stages are terminal.
 
@@ -80,11 +84,13 @@ For each transition: which stage does it come from, which stage does it go to, a
 
 The `id` is the deduplication primitive. An item can only be pushed once per pipeline per id. Choose it carefully.
 
+**Prefer extracted IDs over URLs.** URLs are unstable — the same resource may appear at a mobile URL, a REST API endpoint, a CDN path, or a redirect. Extract the platform's native object ID from the URL and use that instead. A numeric or slug ID survives URL structure changes; a URL does not.
+
 | Situation | Use |
 |-----------|-----|
-| Items come from an external system | The external system's stable identifier (URL, database row id, API object id) |
+| Items come from an external system | The platform's native object ID (e.g. post ID, product SKU) — extract from URL if needed |
 | Items have no upstream identifier | Omit `id` — the pipeline assigns a sequence number |
-| Items must be globally unique across runs | A hash or composite key (e.g., `sha256(url + date)`) |
+| Items must be globally unique across runs | A hash or composite key (e.g., `sha256(source_id + date)`) |
 
 Never use a generated UUID as the id unless items genuinely have no stable identity.
 
@@ -92,10 +98,18 @@ Never use a generated UUID as the id unless items genuinely have no stable ident
 
 Payload is append-only history by default. Each stage appends its output.
 
-For each field that will appear in the payload:
-- Which stage introduces it?
-- What type/shape is it?
-- Is it ever replaced (use `replace: true`) rather than appended?
+Document the payload as a YAML example that mirrors the actual structure. Use inline comments for fields that need explanation. No need to enumerate which stage adds each field — the structure itself is the schema.
+
+```yaml
+# Example payload schema
+id: "12345"
+title: "Product name"
+price: 29.99
+tags: [electronics, sale]
+summary: "One-paragraph AI summary"  # optional, added after enrichment
+```
+
+Is any field ever replaced rather than appended? Note it with a comment (`# replaced, not appended`).
 
 ---
 
@@ -124,7 +138,7 @@ PIPELINES.md lives in the project root. One file documents all pipelines in the
 ```markdown
 ## [Pipeline Name]
 
-**Description:** One sentence.
+**Description:** `[ProjectName] — [what it does]`
 
 **Purpose:** 2-3 sentences. Why does this pipeline exist? What triggers it?
 
@@ -139,22 +153,23 @@ PIPELINES.md lives in the project root. One file documents all pipelines in the
 
 ### Payload Schema
 
-| Field | Type | Added by stage | Description |
-|-------|------|----------------|-------------|
+```yaml
+# YAML example mirroring the actual payload structure
+```
 
 ---
 
 ### Stages
 
-| Stage | Meaning | Terminal? |
-|-------|---------|-----------|
+| Stage | Meaning |
+|-------|---------|
 
 ---
 
 ### Transitions
 
-| From | To | Agent action | Payload appended | Notes |
-|------|----|-------------|-----------------|-------|
+| From | To | Condition | Action | Notes |
+|------|----|-----------|--------|-------|
 
 ---
 
@@ -185,8 +200,8 @@ stateDiagram-v2
 - [ ] Every stage has a defined meaning in the Stages table
 - [ ] Every valid transition appears in both the Transitions table and the Mermaid diagram
 - [ ] Terminal stages are identified
-- [ ] ID strategy is explicit and justified
-- [ ] Payload schema shows which stage introduces each field
+- [ ] ID strategy is explicit and justified — stable object ID preferred over URL
+- [ ] Payload schema is a YAML example that reflects the actual structure
 - [ ] Retry logic is documented
 - [ ] Orchestrator and subagent responsibilities are separated and complete
 - [ ] Process notes cover what happens to `failed` items