@quantod/qq 1.1.18 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -21,7 +21,7 @@ Stages are just labels. The pipeline has no built-in notion of "first" or "last"
21
21
  Items have:
22
22
  - `id` — the deduplication key. Pushing an item with an existing id fails.
23
23
  - `stage` — the current state
24
- - `payload` — item data; append-only string by default; YAML by convention.
24
+ - `payload` — item data; a structured JSON object (or array). Merged with json_patch by default; replaced when `replace: true`.
25
25
  - `seq` — the claim token, returned by `claim`, required to `release`
26
26
  - `priority` — float, higher = claimed first (default 0.0)
27
27
 
@@ -34,12 +34,12 @@ make_pipeline → create a pipeline, get its name, once per setup
34
34
  push (N times) → load items into a stage
35
35
  claim → atomically lock one item (returns seq + payload)
36
36
  ... process the claimed item ...
37
- release → unlock item, move to next stage, append payload
37
+ release → unlock item, move to next stage, merge payload
38
38
  status → monitor progress
39
39
  unstick → recover stuck/lost items
40
40
  ```
41
41
 
42
- **Orchestrator** creates the pipeline, pushes items, spawns subagents, monitors with `status`, calls `batch_read` to collect results, calls `unstick` after.
42
+ **Orchestrator** creates the pipeline, pushes items, spawns subagents, monitors with `status`, calls `query` to collect results, calls `unstick` after.
43
43
 
44
44
  **Subagent** claims one item, processes it, releases to the next stage, repeats until "no items to claim" — then stops and reports back. Never process an item before claiming it: another agent could claim and modify it concurrently.
45
45
 
@@ -50,9 +50,9 @@ unstick → recover stuck/lost items
50
50
  Orchestrator example:
51
51
  ```
52
52
  make_pipeline → "0529_a3f9c2" (store this as Q)
53
- push Q/pending "url-1" payload: "url: https://a.com"
54
- push Q/pending "url-2" payload: "url: https://b.com"
55
- push Q/pending "url-3" payload: "url: https://c.com"
53
+ push Q/pending "url-1" payload: {url: "https://a.com"}
54
+ push Q/pending "url-2" payload: {url: "https://b.com"}
55
+ push Q/pending "url-3" payload: {url: "https://c.com"}
56
56
  ```
57
57
 
58
58
  Spawn subagents with instructions like:
@@ -98,8 +98,7 @@ Parameters:
98
98
  - `pipeline` (string) — pipeline name
99
99
  - `stage` (string, optional) — destination stage. Supports sub-paths: `enriched/shortlisted`. Defaults to empty string if omitted.
100
100
  - `id` (string, optional) — stable unique identifier. Omit to use auto-generated seq number.
101
- - `payload` (string, optional) — payload string
102
- - `payloadFormat` (`yaml` | `json` | `text`, optional, default `yaml`) — `yaml`: validate as YAML. `json`: validate as JSON and minify to one line (appended payloads form valid JSONL). `text`: no validation.
101
+ - `payload` (object, optional) — structured payload object. Stored as SQLite JSONB. Any JSON-serializable object or array.
103
102
  - `priority` (number, optional) — float, default 0.0. Higher = claimed first. Only set when meaningful.
104
103
 
105
104
  Returns:
@@ -111,8 +110,6 @@ id: url-1
111
110
 
112
111
  Errors:
113
112
  - `error: item <id> already in the pipeline` — duplicate id; item already exists
114
- - `error: payload is not valid YAML: <detail>` — `payloadFormat` is `yaml` (default) and payload fails YAML parse
115
- - `error: payload is not valid JSON: <detail>` — `payloadFormat` is `json` and payload fails JSON parse
116
113
 
117
114
  ---
118
115
 
@@ -135,7 +132,7 @@ priority: 0
135
132
  claimed: true
136
133
  created: 2025-05-28T14:30:00.000Z
137
134
  last_modified: 2025-05-28T14:30:00.000Z
138
- payload: |
135
+ payload:
139
136
  url: https://example.com/product/42
140
137
  ```
141
138
 
@@ -156,9 +153,8 @@ Parameters:
156
153
  - `pipeline` (string)
157
154
  - `seq` (number) — the claim token from `claim`
158
155
  - `target` (string, optional) — destination stage. **Always specify when moving the item.** Omit to release back to the same stage.
159
- - `payload` (string, optional) — payload string to append (or replace)
160
- - `payloadFormat` (`yaml` | `json` | `text`, optional, default `yaml`) same as `push`
161
- - `replace` (boolean, optional) — replace payload entirely instead of appending. Use when treating payload as a structured object.
156
+ - `payload` (object, optional) — structured payload object to merge into the existing payload
157
+ - `replace` (boolean, optional) replace payload entirely instead of merging. Use when you want to discard the previous payload.
162
158
  - `priority` (number, optional) — update the item's priority
163
159
 
164
160
  Returns:
@@ -169,10 +165,8 @@ ok: true
169
165
  Errors:
170
166
  - `error: claim on seq <n> expired` — seq no longer exists; discard your work
171
167
  - `error: item seq <n> not claimed` — items should be claimed before releasing - likely bug
172
- - `error: payload is not valid YAML: <detail>` — `payloadFormat` is `yaml` (default) and payload fails YAML parse
173
- - `error: payload is not valid JSON: <detail>` — `payloadFormat` is `json` and payload fails JSON parse
174
168
 
175
- **Payload is append-only history by default.** Each stage appends its output resulting duplicate YAML fields are intentional. Use `replace: true` only when treating payload as a structured object you own.
169
+ **Payload merge uses json_patch (RFC 7396) by default.** When you pass a `payload` object, it is merged into the existing payload using RFC 7396 semantics: keys present in the patch are set on the target (recursively for nested objects); keys set to `null` in the patch are **removed** from the target; keys absent from the patch are left unchanged. Omitting `payload` leaves the existing payload unchanged. Use `replace: true` to discard the existing payload entirely.
176
170
 
177
171
  ---
178
172
 
@@ -187,34 +181,38 @@ Returns:
187
181
  ```yaml
188
182
  description: Scrape product pages and extract pricing
189
183
  stats:
190
- '*': [23, 3]
184
+ /*: [23, 3]
191
185
  done: [7, 0]
186
+ enriched: [3, 1]
187
+ enriched/*: [10, 2]
188
+ enriched/shortlisted: [7, 1]
192
189
  failed: [6, 0]
193
190
  pending: [10, 3]
194
191
  ```
195
192
 
196
- Each value is `[total, claimed]` — total item count and how many are currently claimed (being processed). `*` is the pipeline-level aggregate across all stages. For nested stages (`enriched/shortlisted`), intermediate paths are included as aggregates of their children. `description` is `null` if none was set.
193
+ Each value is `[total, claimed]` — total item count and how many are currently claimed (being processed). `/*` is the pipeline-level aggregate across all stages. A key ending in `*` is an aggregate it sums all items in that prefix and its sub-stages.
197
194
 
198
195
  Errors:
199
196
  - `error: pipeline <name> not found`
200
197
 
201
198
  ---
202
199
 
203
- ### batch_read
200
+ ### query
204
201
 
205
202
  Reads items without claiming them. For inspection, data collection, and monitoring.
206
203
 
207
204
  Parameters:
208
205
  - `pipeline` (string)
209
206
  - `stage` (string, optional) — supports globs. Defaults to `*`
210
- - `includePayload` (boolean, optional) — include payload in results
211
- - `claimed` (boolean, optional) — filter by claimed status. Default: unclaimed
207
+ - `select` (`"*"` | string[], optional) — which payload fields to return. `"*"` returns the full payload; an array of dot-notation paths (e.g. `["status", "meta.author"]`) returns only those fields; omit to return no payload.
208
+ - `filter` (object, optional) — MongoDB-style filter on payload fields. Keys are dot-notation paths into the payload. Values are either a literal (equality match) or an operator object. Supported operators: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in` (array), `$exists` (boolean). Example: `{ "status": "active", "score": { "$gt": 80 } }`.
209
+ - `claimed` (boolean, optional) — filter by claimed status
212
210
  - `ids` (string[], optional)
213
211
  - `createdAfter`, `createdBefore`, `modifiedAfter` (string | number) — epoch ms, ISO 8601 (`"2025-05-28"`, `"2025-05-28T14:30:00Z"`), or relative (`"-2h"`, `"-1d"`, `"-30m"`, `"-1w"`)
214
212
  - `limit`, `offset` (number)
215
213
  - `strategy` (`priority` | `fifo` | `lifo` | `random`, optional, default `fifo`) — sort order. Same semantics as `claim`: `priority` highest first then lowest seq, `fifo` lowest seq first, `lifo` highest seq first, `random` random order.
216
214
 
217
- Returns a YAML list of items. `payload` is omitted unless `includePayload: true`. Returns `[]` when nothing matches.
215
+ Returns a YAML list of items. `payload` is included only when `select` is provided. Returns `[]` when nothing matches.
218
216
 
219
217
  ```yaml
220
218
  - id: url-1
@@ -224,6 +222,9 @@ Returns a YAML list of items. `payload` is omitted unless `includePayload: true`
224
222
  priority: 0
225
223
  created: 2025-05-28T14:30:00.000Z
226
224
  last_modified: 2025-05-28T14:30:01.000Z
225
+ payload:
226
+ url: https://example.com
227
+ score: 92
227
228
  - id: url-2
228
229
  stage: done
229
230
  claimed: false
@@ -231,6 +232,9 @@ Returns a YAML list of items. `payload` is omitted unless `includePayload: true`
231
232
  priority: 0
232
233
  created: 2025-05-28T14:30:00.001Z
233
234
  last_modified: 2025-05-28T14:30:02.000Z
235
+ payload:
236
+ url: https://other.com
237
+ score: 47
234
238
  ```
235
239
 
236
240
  ---
@@ -281,31 +285,32 @@ Returns a code snippet to inline at the top of your script. The snippet uses onl
281
285
  QQ.push(pipeline, stage, id?, payload?, opts?) → Promise<{ id: string } | { error: string }>
282
286
  QQ.claim(pipeline, stage, opts?) → Promise<Item | { error: string }>
283
287
  QQ.release(pipeline, seq, opts?) → Promise<{ ok: true } | { error: string }>
284
- QQ.batch_read(pipeline, stage, opts?) → Promise<Item[] | { error: string }>
288
+ QQ.query(pipeline, stage, opts?) → Promise<Item[] | { error: string }>
285
289
  QQ.status(pipeline) → Promise<{ description: string|null, stats: Record<string, [number,number]> } | { error: string }>
286
290
  ```
287
291
 
288
- - `push` opts: `priority` (number), `payloadFormat` (`"yaml"` | `"json"` | `"text"`)
292
+ - `push` opts: `priority` (number)
289
293
  - `claim` opts: `id` (string — claim a specific item), `strategy` (`"priority"` | `"fifo"` | `"lifo"` | `"random"`)
290
- - `release` opts: `target` (string), `payload` (string), `replace` (boolean), `priority` (number), `payloadFormat`
291
- - `batch_read` opts: `includePayload` (boolean), `claimed` (boolean), `ids` (string, comma-separated), `createdAfter`, `createdBefore`, `modifiedAfter` (epoch ms, ISO 8601, or relative e.g. `"-2h"`), `limit`, `offset` (number), `strategy` (`"priority"` | `"fifo"` | `"lifo"` | `"random"`)
294
+ - `release` opts: `target` (string), `payload` (object), `replace` (boolean), `priority` (number)
295
+ - `query` opts: `select` (`"*"` or string[] of dot-notation paths), `filter` (object — MongoDB-style payload filter), `claimed` (boolean), `ids` (string, comma-separated), `createdAfter`, `createdBefore`, `modifiedAfter` (epoch ms, ISO 8601, or relative e.g. `"-2h"`), `limit`, `offset` (number), `strategy` (`"priority"` | `"fifo"` | `"lifo"` | `"random"`)
292
296
 
293
297
  All functions return `{ error: string }` on failure — never throw (except `BridgeError` for network connection failures).
294
298
 
295
- Item shape (returned by `claim` and `batch_read`):
299
+ Item shape (returned by `claim` and `query`):
296
300
  ```
297
301
  { id: string, stage: string, seq: number, priority: number, claimed: boolean,
298
- created: number, last_modified: number, payload?: string | null }
302
+ created: number, last_modified: number, payload?: object | null }
299
303
  ```
300
304
 
301
- `claim` always includes `payload`. `batch_read` omits `payload` unless `includePayload: true`.
305
+ `claim` always includes `payload`. `query` omits `payload` unless `select` is provided.
306
+ `payload` is a parsed JavaScript object (or null) — never a string.
302
307
 
303
308
  **Python signatures:**
304
309
  ```
305
310
  qq_push(pipeline, stage, id=None, payload=None, **opts) → { 'id': str } | { 'error': str }
306
311
  qq_claim(pipeline, stage, **opts) → dict | { 'error': str }
307
312
  qq_release(pipeline, seq, **opts) → { 'ok': True } | { 'error': str }
308
- qq_batch_read(pipeline, stage, **opts) → list[dict] | { 'error': str }
313
+ qq_query(pipeline, stage, **opts) → list[dict] | { 'error': str }
309
314
  qq_status(pipeline) → { 'description': str|None, 'stats': dict } | { 'error': str }
310
315
  ```
311
316
 
@@ -313,9 +318,9 @@ All Python functions return `{ 'error': str }` on failure — never raise (excep
313
318
 
314
319
  ---
315
320
 
316
- ### load_file
321
+ ### upload_file
317
322
 
318
- Loads multiple items from a file on the host filesystem into a pipeline. Each record in the file becomes one pipeline item.
323
+ Uploads multiple items from a file on the host filesystem into a pipeline. Each record in the file becomes one pipeline item.
319
324
 
320
325
  **Supported formats:** JSONL (`.jsonl`, `.ndjson`), JSON array (`.json`), YAML array (`.yaml`, `.yml`), CSV (`.csv`). Format is detected from the file extension.
321
326
 
@@ -326,7 +331,7 @@ Parameters:
326
331
  - `pipeline` (string) — pipeline to push records into
327
332
  - `stage` (string, optional) — default stage for records that do not specify their own. Defaults to empty string.
328
333
  - `deleteAfter` (boolean, optional) — delete the file after loading. Default: true.
329
- - `duplicates` (`ignore` | `append` | `replace`, optional, default `ignore`) — how to handle records whose `id` already exists in the pipeline. `ignore`: skip silently (existing item unchanged). `append`: append the new record's payload to the existing item's payload and bump `seq` (invalidates any live claim). `replace`: replace the existing item's payload entirely and bump `seq`.
334
+ - `duplicates` (`ignore` | `append` | `replace`, optional, default `ignore`) — how to handle records whose `id` already exists in the pipeline. `ignore`: skip silently (existing item unchanged). `append`: merge the new record's payload into the existing payload using json_patch, bump `seq` (invalidates any live claim). `replace`: replace the existing item's payload entirely and bump `seq`.
330
335
 
331
336
  **⚠ Avoid `append` and `replace` in production pipelines.** Both modes write to items outside the normal `claim → process → release` cycle, bypassing QQ's concurrency control. Any agent currently holding a claim on an affected item will find its `seq` token expired and lose its work. `replace` is especially destructive — it silently discards the existing payload with no recovery path. The correct way to update an item's payload is to `claim` it, process it, and `release` it with new content. Use `append`/`replace` only for offline data preparation before a pipeline has active workers.
332
337
 
@@ -334,14 +339,18 @@ Parameters:
334
339
  ```yaml
335
340
  id: item-1
336
341
  stage: pending
337
- payload: "any string"
342
+ payload:
343
+ url: https://example.com
344
+ score: 42
338
345
  priority: 0.0
339
346
  ```
340
347
  - `id` — omit for auto-generated id
341
348
  - `stage` — overrides the `stage` parameter for this record
342
- - `payload` — treated as an opaque string; non-string values are YAML-serialized
349
+ - `payload` — structured object. String values are parsed as YAML for backward compatibility.
343
350
  - `priority` — float, default 0.0
344
351
 
352
+ **CSV format**: for CSV files, all columns except `id`, `stage`, and `priority` become the payload object. A CSV row `id,url,title` with values `a,https://x.com,Hello` produces payload `{url: "https://x.com", title: "Hello"}`.
353
+
345
354
  **Returns** YAML with a summary of counts and a per-item outcome list:
346
355
  ```yaml
347
356
  summary:
@@ -378,24 +387,22 @@ Errors:
378
387
 
379
388
  ---
380
389
 
381
- ### batch_write
390
+ ### upload_data
382
391
 
383
- Same as `load_file` but accepts data inline instead of reading from a file. Use when the records are already in memory or generated on the fly.
392
+ Same as `upload_file` but accepts data inline instead of reading from a file. Use when the records are already in memory or generated on the fly.
384
393
 
385
394
  Parameters:
386
395
  - `data` (string) — the raw record data
387
396
  - `format` (`jsonl` | `json` | `yaml` | `csv`) — data format (explicit, no file extension to infer from)
388
397
  - `pipeline` (string)
389
398
  - `stage` (string, optional) — default stage. Defaults to empty string.
390
- - `duplicates` (`ignore` | `append` | `replace`, optional, default `ignore`) — same semantics as `load_file`. The same warning applies: prefer `claim → process → release` over `append`/`replace` whenever workers may be active.
399
+ - `duplicates` (`ignore` | `append` | `replace`, optional, default `ignore`) — same semantics as `upload_file`. The same warning applies: prefer `claim → process → release` over `append`/`replace` whenever workers may be active.
391
400
 
392
- Returns the same shape as `load_file` (`summary` + `items`), without the `deleteAfter` behaviour.
401
+ Returns the same shape as `upload_file` (`summary` + `items`), without the `deleteAfter` behaviour.
393
402
 
394
403
  Errors:
395
404
  - `error: <message>` — malformed data. Entire operation aborted; no records are pushed.
396
405
 
397
- ---
398
-
399
406
  ## SDK Usage
400
407
 
401
408
  Use the SDK when you need to run code that processes pipeline items programmatically — loops, transforms, bulk operations. It is faster and more efficient than calling MCP tools one at a time for batch work.
@@ -417,28 +424,34 @@ Use the SDK when you need to run code that processes pipeline items programmatic
417
424
 
418
425
  const pipeline = '0529_a3f9c2';
419
426
 
420
- // Bulk push — payload is a YAML string
427
+ // Bulk push — payload is a plain object
421
428
  for (const url of urls) {
422
- await QQ.push(pipeline, 'pending', url, `url: ${url}`);
429
+ await QQ.push(pipeline, 'pending', url, { url });
423
430
  }
424
431
 
425
432
  // Claim loop — check result.error, not null
426
433
  while (true) {
427
434
  const item = await QQ.claim(pipeline, 'pending');
428
435
  if (item.error) break; // "no items to claim" or other error
429
- const result = await processItem(item.payload);
430
- await QQ.release(pipeline, item.seq, { target: 'done', payload: result });
436
+ const result = await processItem(item.payload); // item.payload is already a parsed object
437
+ await QQ.release(pipeline, item.seq, { target: 'done', payload: { result } });
431
438
  }
432
439
 
433
440
  // Collect results with payloads
434
- const done = await QQ.batch_read(pipeline, 'done', { includePayload: true });
441
+ const done = await QQ.query(pipeline, 'done', { select: '*' });
435
442
  for (const item of done) {
436
- console.log(item.id, item.payload);
443
+ console.log(item.id, item.payload); // item.payload is a parsed object
437
444
  }
438
445
 
446
+ // Filter by payload fields and select only specific fields
447
+ const highScores = await QQ.query(pipeline, 'done', {
448
+ filter: { score: { $gt: 80 }, status: 'active' },
449
+ select: ['url', 'score'],
450
+ });
451
+
439
452
  // Monitor progress
440
453
  const s = await QQ.status(pipeline);
441
- console.log(`total=${s.stats['*'][0]} pending=${s.stats.pending?.[0] ?? 0}`);
454
+ console.log(`total=${s.stats['/*'][0]} pending=${s.stats.pending?.[0] ?? 0}`);
442
455
  ```
443
456
 
444
457
  ### Error routing
@@ -451,34 +464,32 @@ while (true) {
451
464
  if (item.error) break;
452
465
  try {
453
466
  const result = await processItem(item.payload);
454
- await QQ.release(pipeline, item.seq, { target: 'done', payload: `result: ${result}` });
467
+ await QQ.release(pipeline, item.seq, { target: 'done', payload: { result } });
455
468
  } catch (e) {
456
- await QQ.release(pipeline, item.seq, { target: 'failed', payload: `error: ${e.message}` });
469
+ await QQ.release(pipeline, item.seq, { target: 'failed', payload: { error: e.message } });
457
470
  }
458
471
  }
459
472
  ```
460
473
 
461
- ### Structured JSON payloads
474
+ ### Working with structured payloads
462
475
 
463
- Use `payloadFormat: "json"` when payload is a JSON object. Appended payloads form valid JSONL; `replace: true` treats it as a single mutable object:
476
+ Payloads are plain objects. `release` merges them by default (json_patch); `replace: true` overwrites:
464
477
 
465
478
  ```javascript
466
- // Push with JSON payload
467
- await QQ.push(pipeline, 'pending', 'item-1', JSON.stringify({ url: 'https://a.com', score: 0 }),
468
- { payloadFormat: 'json' });
479
+ // Push with a structured payload object
480
+ await QQ.push(pipeline, 'pending', 'item-1', { url: 'https://a.com', score: 0 });
469
481
 
470
- // Update a structured payload: parse modify → replace
482
+ // Claim returns payload as a parsed object no JSON.parse needed
471
483
  const item = await QQ.claim(pipeline, 'pending');
472
484
  if (!item.error) {
473
- const data = JSON.parse(item.payload);
474
- data.score = await scoreItem(data.url);
475
- await QQ.release(pipeline, item.seq, {
476
- target: 'done',
477
- payload: JSON.stringify(data),
478
- payloadFormat: 'json',
479
- replace: true,
480
- });
485
+ const score = await scoreItem(item.payload.url);
486
+ // Merge new fields into existing payload (json_patch)
487
+ await QQ.release(pipeline, item.seq, { target: 'done', payload: { score } });
488
+ // Result: { url: 'https://a.com', score: <value> }
481
489
  }
490
+
491
+ // Replace entire payload (discard existing)
492
+ await QQ.release(pipeline, item.seq, { payload: { rebuilt: true }, replace: true });
482
493
  ```
483
494
 
484
495
  ### Python example
@@ -488,25 +499,30 @@ if (!item.error) {
488
499
 
489
500
  pipeline = "0529_a3f9c2"
490
501
 
491
- # Bulk push
502
+ # Bulk push — payload is a dict
492
503
  for url in urls:
493
- qq_push(pipeline, "pending", url, f"url: {url}")
504
+ qq_push(pipeline, "pending", url, {"url": url})
494
505
 
495
- # Claim loop
506
+ # Claim loop — item["payload"] is already a parsed dict
496
507
  while True:
497
508
  item = qq_claim(pipeline, "pending")
498
509
  if "error" in item: # "no items to claim" or other error
499
510
  break
500
- result = process_item(item["payload"])
501
- qq_release(pipeline, item["seq"], target="done", payload=result)
511
+ result = process_item(item["payload"]) # payload is a dict
512
+ qq_release(pipeline, item["seq"], target="done", payload={"result": result})
502
513
 
503
- items = qq_batch_read(pipeline, "done", includePayload=True)
514
+ items = qq_query(pipeline, "done", select="*")
504
515
  for item in items:
505
- print(item["id"], item["payload"])
516
+ print(item["id"], item["payload"]) # payload is a dict
517
+
518
+ # Filter by payload fields; select only specific fields
519
+ high_scores = qq_query(pipeline, "done",
520
+ filter={"score": {"$gt": 80}, "status": "active"},
521
+ select=["url", "score"])
506
522
 
507
523
  # Monitor
508
524
  s = qq_status(pipeline)
509
- print(f"total={s['stats']['*'][0]}")
525
+ print(f"total={s['stats']['/*'][0]}")
510
526
  ```
511
527
 
512
528
  ### BridgeError reconnect
@@ -520,39 +536,41 @@ This happens when the QQ MCP server restarted (e.g. Claude Desktop was restarted
520
536
 
521
537
  ---
522
538
 
523
- ## YAML Payload Conventions
539
+ ## Structured Payload and Merge Semantics
524
540
 
525
- Payload is a YAML string appended to existing content by default. The result is a YAML document where fields may appear multiple times — this is intentional. Each stage's output appears in sequence, creating a trace of the item's journey.
541
+ Payloads are structured JSON objects (or arrays), stored as SQLite JSONB. Plain text is not allowed.
526
542
 
527
- ```yaml
528
- # After push:
529
- url: https://example.com/product/42
530
-
531
- # After first subagent appends:
532
- url: https://example.com/product/42
533
- name: Widget Pro
534
- price: 29.99
535
- available: true
536
-
537
- # After second subagent appends (review):
538
- url: https://example.com/product/42
539
- name: Widget Pro
540
- price: 29.99
541
- available: true
542
- review_reason: price seems anomalous
543
+ **Default merge — json_patch (RFC 7396):** When `release` or `upload_data append` receives a payload, it is merged into the existing payload using RFC 7396 semantics:
544
+ - Keys in the patch with a non-null value are set on the target (recursively for nested objects)
545
+ - Keys in the patch set to `null` are **removed** from the target
546
+ - Keys absent from the patch are left unchanged
547
+
548
+ ```
549
+ # Existing: { url: "https://x.com", score: 0, tags: ["a"], draft: true }
550
+ # Patch: { score: 42, extra: "new", draft: null }
551
+ # Result: { url: "https://x.com", score: 42, tags: ["a"], extra: "new" }
552
+ # (draft removed because patch set it to null)
543
553
  ```
544
554
 
545
- Use `replace: true` only when you own the payload as a structured object (e.g., accumulating messages in a chat pipeline). To update a structured payload: claim, parse with yaml.load, modify, release with replace.
555
+ **Replace:** Pass `replace: true` to discard the existing payload entirely and store only the new value.
556
+
557
+ **No payload:** Omit `payload` from `release` to leave the existing payload unchanged (useful when only moving stage or updating priority).
558
+
559
+ **Null payload:** An item pushed with no `payload` argument stores SQL NULL — `claim` and `query` return it as `null`.
560
+
561
+ **CLI and SDK string input:** The CLI (`--payload`) and `payloadFormat` option accept YAML or JSON strings that are parsed before storage. The `text` format has been removed — all inputs must be valid YAML or JSON.
546
562
 
547
563
  ---
548
564
 
549
565
  ## Globs, Filters, Priority
550
566
 
551
- **Stage globs**: `*` matches any single segment, `enriched/*` matches sub-stages. Use in `claim` and `batch_read` to work across multiple stages.
567
+ **Stage globs**: `*` matches any single segment, `enriched/*` matches sub-stages. Use in `claim` and `query` to work across multiple stages.
552
568
 
553
569
  **Priority**: float, default 0.0. Higher value = claimed first within the same stage. Use only when meaningful (e.g., score, confidence). Most pipelines don't need priority.
554
570
 
555
- **Filters in batch_read / unstick**: `claimed`, `ids`, `createdAfter`, `createdBefore`, `modifiedAfter` (epoch ms, ISO 8601, or relative: `"-2h"`, `"-1d"`, `"-30m"`, `"-1w"`), `limit`, `offset`. **batch_read** also accepts `strategy` (`priority` | `fifo` | `lifo` | `random`, default `fifo`).
571
+ **Filters in query / unstick**: `claimed`, `ids`, `createdAfter`, `createdBefore`, `modifiedAfter` (epoch ms, ISO 8601, or relative: `"-2h"`, `"-1d"`, `"-30m"`, `"-1w"`), `limit`, `offset`. **query** also accepts `strategy` (`priority` | `fifo` | `lifo` | `random`, default `fifo`), `filter` (MongoDB-style payload filter), and `select` (`"*"` or array of dot-notation field paths).
572
+
573
+ **Payload filter operators** (`filter` in `query`): `{ "field": value }` (equality), `{ "field": { "$gt": n } }`, `$gte`, `$lt`, `$lte`, `$ne`, `{ "field": { "$in": [a, b] } }`, `{ "field": { "$exists": true } }`. Nested fields use dot notation: `"meta.author"`. All conditions are ANDed together.
556
574
 
557
575
  ---
558
576
 
@@ -16,7 +16,7 @@ const QQ = (() => {
16
16
  push: (pipeline, stage, id, payload, opts = {}) => _req('POST', \`/pipelines/\${pipeline}/push\`, { stage, id, payload, ...opts }),
17
17
  claim: (pipeline, stage, opts = {}) => _req('POST', \`/pipelines/\${pipeline}/claim\`, { stage, ...opts }),
18
18
  release: (pipeline, seq, opts = {}) => _req('POST', \`/pipelines/\${pipeline}/release/\${seq}\`, opts),
19
- batch_read: (pipeline, stage, opts = {}) => _req('GET', \`/pipelines/\${pipeline}/items?\${new URLSearchParams({ stage, ...opts })}\`),
19
+ query: (pipeline, stage, { filter, select, ...opts } = {}) => { const p = { stage, ...opts }; if (filter != null) p.filter = JSON.stringify(filter); if (select != null) p.select = Array.isArray(select) ? select.join(',') : select; return _req('GET', \`/pipelines/\${pipeline}/items?\${new URLSearchParams(p)}\`); },
20
20
  status: (pipeline) => _req('GET', \`/pipelines/\${pipeline}/status\`),
21
21
  };
22
22
  })();`;
@@ -1 +1 @@
1
- {"version":3,"file":"python.d.ts","sourceRoot":"","sources":["../../src/sdk-templates/python.ts"],"names":[],"mappings":"AAAA,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CA0BjF"}
1
+ {"version":3,"file":"python.d.ts","sourceRoot":"","sources":["../../src/sdk-templates/python.ts"],"names":[],"mappings":"AAAA,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CA6BjF"}
@@ -22,8 +22,11 @@ def qq_claim(pipeline, stage, **opts):
22
22
  return _bridge_req('POST', f'/pipelines/{pipeline}/claim', {'stage': stage, **opts})
23
23
  def qq_release(pipeline, seq, **opts):
24
24
  return _bridge_req('POST', f'/pipelines/{pipeline}/release/{seq}', opts)
25
- def qq_batch_read(pipeline, stage, **opts):
26
- qs = urllib.parse.urlencode({'stage': stage, **opts})
25
+ def qq_query(pipeline, stage, filter=None, select=None, **opts):
26
+ params = {'stage': stage, **opts}
27
+ if filter is not None: params['filter'] = _json.dumps(filter)
28
+ if select is not None: params['select'] = ','.join(select) if isinstance(select, list) else select
29
+ qs = urllib.parse.urlencode(params)
27
30
  return _bridge_req('GET', f'/pipelines/{pipeline}/items?{qs}')
28
31
  def qq_status(pipeline):
29
32
  return _bridge_req('GET', f'/pipelines/{pipeline}/status')`;
@@ -1 +1 @@
1
- {"version":3,"file":"python.js","sourceRoot":"","sources":["../../src/sdk-templates/python.ts"],"names":[],"mappings":";;AAAA,0CA0BC;AA1BD,SAAgB,eAAe,CAAC,IAAY,EAAE,IAAY,EAAE,KAAa;IACvE,OAAO;;wBAEe,IAAI,IAAI,IAAI;8CACU,KAAK;;;;;;;;;;;;;;;;;;;;;+DAqBY,CAAC;AAChE,CAAC"}
1
+ {"version":3,"file":"python.js","sourceRoot":"","sources":["../../src/sdk-templates/python.ts"],"names":[],"mappings":";;AAAA,0CA6BC;AA7BD,SAAgB,eAAe,CAAC,IAAY,EAAE,IAAY,EAAE,KAAa;IACvE,OAAO;;wBAEe,IAAI,IAAI,IAAI;8CACU,KAAK;;;;;;;;;;;;;;;;;;;;;;;;+DAwBY,CAAC;AAChE,CAAC"}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@quantod/qq",
3
- "version": "1.1.18",
3
+ "version": "1.2.0",
4
4
  "description": "Persistent queue for multi-stage Claude agent pipelines",
5
5
  "license": "All Rights Reserved",
6
6
  "engines": {