@trawlme/cli 2.1.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -2
- package/dist/commands/create.d.ts +2 -1
- package/dist/commands/create.js +4 -4
- package/dist/commands/doctor.js +2 -1
- package/dist/commands/ping.js +3 -1
- package/dist/commands/scraps.d.ts +1 -0
- package/dist/commands/scraps.js +133 -139
- package/dist/index.js +6 -3
- package/dist/lib/format.d.ts +7 -0
- package/dist/lib/format.js +19 -0
- package/dist/lib/spinner.d.ts +18 -0
- package/dist/lib/spinner.js +17 -0
- package/docs/agent-quickstart.md +8 -5
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -46,6 +46,8 @@ trawl data <id> [--json] [--fresh] [--errors] Get scrap data (last persisted ru
|
|
|
46
46
|
trawl history <id> [--json] [-n <limit>] List past runs for a scrap (newest first)
|
|
47
47
|
trawl run-info <hid> [--json] Show details of a single run
|
|
48
48
|
trawl trigger <id> [--watch] [--wait] [--json] Launch a scrap as a background worker (returns immediately)
|
|
49
|
+
trawl update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
50
|
+
Update an existing scrap
|
|
49
51
|
trawl whoami [--json] Show the authenticated user's identity
|
|
50
52
|
trawl ping [--json] Health/version handshake against the Trawl API
|
|
51
53
|
```
|
|
@@ -54,7 +56,7 @@ trawl ping [--json] Health/version handshake against
|
|
|
54
56
|
|
|
55
57
|
`create`/`whoami`/`ping` are fully non-interactive — all three read auth only from `TRAWL_TOKEN`/the stored login token, never prompt. `trawl create` runs the AI wizard server-side (`POST /api/ai/wizard`): generate scrap code from `--prompt` via LLM, persist the scrap, trigger its FIRST run, and auto-fix on failure (default on — `--no-autofix` disables it, sending `autoFix:false`). `success` is an honest outcome of that first run, not "did the HTTP call succeed" — a failed first run is still a 200 response (the scrap was still created; auto-fix, when enabled, retries in the background), and the CLI exits 1 in that case (both human and `--json` modes) even though `--json` always prints the raw payload verbatim. The call legitimately takes 30–250s+ server-side (AI generation + a real run), same long-run timeout as `run`/`data --fresh`/`trigger --wait` below. `trawl whoami`/`trawl ping` mirror the MCP `trawl_whoami`/`trawl_health_ping` tools as closely as the REST surface allows (`GET /api/users/me` / `GET /api/health`) — `ping`'s `--json` payload is admin-enriched (version/uptime/db) and just `{"status":"ok"}` for anyone else.
|
|
56
58
|
|
|
57
|
-
> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl
|
|
59
|
+
> **`create` is NOT idempotent, and every wizard-created scrap runs on a DAILY cron by default.** A client-side timeout (exit `5`, a `NetworkError`) does not mean the wizard failed server-side — scrap creation + the first run keep going after the CLI gives up waiting, so the scrap may already exist. Run `trawl list` and look for a matching URL/title **before** retrying — a blind retry creates a DUPLICATE scrap and burns AI-generation quota a second time for the same goal. Separately, the scrap the wizard creates is scheduled to re-run every day at 07:00 UTC (`cron: "0 7 * * *"`, hardcoded server-side, unrelated to `--no-autofix`) — each of those recurring runs consumes execute quota like any other run. Review the generated scrap, then change or disable the schedule with `trawl update <id> --cron <expr>` (or `--no-cron` to disable it). Because the call can legitimately run 250s+, also confirm `TRAWL_TIMEOUT` isn't set to something tighter than `create` needs — the env var always wins over `create`'s own 300s default (see [Environment variables](#environment-variables)), so a value set for another purpose (e.g. a tight CI smoke-test budget) silently clamps `create` too; unset it or raise it before running `create`.
|
|
58
60
|
|
|
59
61
|
- `list` has a short alias, `ls` (matches `trawl --help`'s `list|ls`).
|
|
60
62
|
- `history` lists past runs (newest first); `run-info <hid>` shows details of a single run from that history.
|
|
@@ -68,7 +70,6 @@ trawl ping [--json] Health/version handshake against
|
|
|
68
70
|
|
|
69
71
|
```
|
|
70
72
|
trawl scraps create -t <title> [-u <url>] [-r <request>] [-d <description>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
71
|
-
trawl scraps update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <tier0|tier1|tier2|tier3|tier4>] [--force-tier <tier0|tier1|tier2|tier3|tier4>] [--json]
|
|
72
73
|
trawl scraps watch <id> [--json]
|
|
73
74
|
trawl scraps doctor <id> [--json] [--autofix]
|
|
74
75
|
trawl scraps autofix <id> [--json]
|
|
@@ -30,7 +30,8 @@ export interface WizardResponse {
|
|
|
30
30
|
* 'UTC'` — unconditionally, regardless of `--prompt`/`--no-autofix`.
|
|
31
31
|
* Present on the scrap object returned here (the wizard controller
|
|
32
32
|
* passes the created scrap straight through, no stripping) — change or
|
|
33
|
-
* disable it with `trawl
|
|
33
|
+
* disable it with `trawl update <id> --cron <expr>` / `--no-cron` (#120
|
|
34
|
+
* — `update` is now a promoted top-level verb, same as `get`/`run`/…).
|
|
34
35
|
*/
|
|
35
36
|
cron?: string | null;
|
|
36
37
|
cronTimezone?: string;
|
package/dist/commands/create.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { Command } from 'commander';
|
|
2
2
|
import chalk from 'chalk';
|
|
3
|
-
import {
|
|
3
|
+
import { spin } from '../lib/spinner.js';
|
|
4
4
|
import { api, LONG_RUN_TIMEOUT_MS, NetworkError } from '../lib/api.js';
|
|
5
5
|
import { json } from '../lib/format.js';
|
|
6
6
|
import { requireUrl, requireString } from '../lib/validate.js';
|
|
@@ -53,8 +53,8 @@ function scheduleLabel(scrap) {
|
|
|
53
53
|
export const create = new Command('create')
|
|
54
54
|
.description('Create a persistent, self-healing scrap from a URL + a goal (AI-generated)')
|
|
55
55
|
.argument('[url]', 'Target public URL (http/https)')
|
|
56
|
-
.requiredOption('--prompt <goal>', 'What to extract/scrape, in plain language')
|
|
57
|
-
.option('--url <url>', 'Target public URL — alias of the positional argument
|
|
56
|
+
.requiredOption('--prompt <goal>', 'What to extract/scrape, in plain language (required)')
|
|
57
|
+
.option('--url <url>', 'Target public URL — alias of the positional argument')
|
|
58
58
|
.option('--no-autofix', 'Disable AI auto-fix on first-run failure (default: on)')
|
|
59
59
|
.option('--json', 'Output the raw API payload')
|
|
60
60
|
.action(async (rawUrl, opts) => {
|
|
@@ -95,7 +95,7 @@ export const create = new Command('create')
|
|
|
95
95
|
}
|
|
96
96
|
else {
|
|
97
97
|
try {
|
|
98
|
-
data = await
|
|
98
|
+
data = await spin(call, {
|
|
99
99
|
text: `Creating a scrap from ${url}…`,
|
|
100
100
|
// No verdict symbol here (#106-F3) — ora's success only means "the
|
|
101
101
|
// HTTP call didn't throw", not "the first run succeeded". The real
|
package/dist/commands/doctor.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { api } from '../lib/api.js';
|
|
2
2
|
import chalk from 'chalk';
|
|
3
|
+
import { formatDate } from '../lib/format.js';
|
|
3
4
|
/**
|
|
4
5
|
* Known anti-bot vendors (+ auth) that the worker's `blockType` field may name.
|
|
5
6
|
* Ordered by first-match; `blockType` is a freeform worker string, not an enum
|
|
@@ -142,7 +143,7 @@ export function formatDoctor(scrapTitle, run, fix = null, scrapId) {
|
|
|
142
143
|
}
|
|
143
144
|
// Timestamp
|
|
144
145
|
if (run.createdAt) {
|
|
145
|
-
lines.push(chalk.dim(' Run at: ') +
|
|
146
|
+
lines.push(chalk.dim(' Run at: ') + formatDate(run.createdAt));
|
|
146
147
|
}
|
|
147
148
|
// Snapshot hint
|
|
148
149
|
if (run.errorSnapshot?.html || run.statusDetail === 'empty' || run.statusDetail === 'error') {
|
package/dist/commands/ping.js
CHANGED
|
@@ -17,7 +17,9 @@ export const ping = new Command('ping')
|
|
|
17
17
|
json(data);
|
|
18
18
|
return;
|
|
19
19
|
}
|
|
20
|
-
|
|
20
|
+
// #119 — label it as the API's version, not a bare `v0.4.0` that reads as
|
|
21
|
+
// "the platform is v0.4" (it's the server package version field).
|
|
22
|
+
const versionSuffix = data.version ? ` (api v${data.version})` : '';
|
|
21
23
|
if (data.status === 'ok') {
|
|
22
24
|
console.log(chalk.green('✓ OK') + versionSuffix);
|
|
23
25
|
}
|
|
@@ -140,6 +140,7 @@ export declare function pollRunProgress(id: string, before: BeforeRunState | und
|
|
|
140
140
|
}): Promise<void>;
|
|
141
141
|
export declare function attachListCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
142
142
|
export declare function attachGetCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
143
|
+
export declare function attachUpdateCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
143
144
|
export declare function attachRunCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
144
145
|
export declare function attachDataCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
|
145
146
|
export declare function attachHistoryCommand(parent: Command, attachOpts?: AttachOptions): Command;
|
package/dist/commands/scraps.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
import { Command } from 'commander';
|
|
2
2
|
import chalk from 'chalk';
|
|
3
|
-
import {
|
|
3
|
+
import { spin } from '../lib/spinner.js';
|
|
4
4
|
import { api, LONG_RUN_TIMEOUT_MS } from '../lib/api.js';
|
|
5
|
-
import { table, json } from '../lib/format.js';
|
|
5
|
+
import { table, json, formatDate } from '../lib/format.js';
|
|
6
6
|
import { promptPassword } from '../lib/prompt.js';
|
|
7
7
|
import { validateObjectId } from '../lib/validate.js';
|
|
8
8
|
import { classifyError, reportError, UsageError, RefusalError } from '../lib/errors.js';
|
|
@@ -42,16 +42,7 @@ function lastStatus(scrap) {
|
|
|
42
42
|
return last.status === true ? 'success' : 'failure';
|
|
43
43
|
}
|
|
44
44
|
function lastRun(scrap) {
|
|
45
|
-
|
|
46
|
-
if (!last?.createdAt)
|
|
47
|
-
return '—';
|
|
48
|
-
const d = new Date(last.createdAt);
|
|
49
|
-
const dd = String(d.getDate()).padStart(2, '0');
|
|
50
|
-
const mm = String(d.getMonth() + 1).padStart(2, '0');
|
|
51
|
-
const yy = String(d.getFullYear()).slice(2);
|
|
52
|
-
const hh = String(d.getHours()).padStart(2, '0');
|
|
53
|
-
const min = String(d.getMinutes()).padStart(2, '0');
|
|
54
|
-
return `${dd}/${mm}/${yy} ${hh}:${min}`;
|
|
45
|
+
return formatDate(scrap.history?.[0]?.createdAt); // #119 — unified DD/MM/YY HH:mm
|
|
55
46
|
}
|
|
56
47
|
function statusIcon(status) {
|
|
57
48
|
if (status === 'success')
|
|
@@ -389,7 +380,7 @@ export function attachListCommand(parent, attachOpts = {}) {
|
|
|
389
380
|
}
|
|
390
381
|
let data;
|
|
391
382
|
try {
|
|
392
|
-
data = await
|
|
383
|
+
data = await spin(async () => {
|
|
393
384
|
if (page !== undefined) {
|
|
394
385
|
// Single-page mode: explicit page requested, no loop
|
|
395
386
|
return api.get(`/api/scraps?perPage=50&page=${page}`);
|
|
@@ -445,7 +436,7 @@ export function attachListCommand(parent, attachOpts = {}) {
|
|
|
445
436
|
cron: s.cron || '—',
|
|
446
437
|
status: statusIcon(lastStatus(s)),
|
|
447
438
|
'last run': lastRun(s),
|
|
448
|
-
updated:
|
|
439
|
+
updated: formatDate(s.updatedAt),
|
|
449
440
|
}));
|
|
450
441
|
table(tableRows, ['id', 'title', 'cron', 'status', 'last run', 'updated']);
|
|
451
442
|
// Print footer when --limit truncates
|
|
@@ -471,7 +462,7 @@ export function attachGetCommand(parent, attachOpts = {}) {
|
|
|
471
462
|
console.log(chalk.dim(` Cron: `) + (data.cron || '—'));
|
|
472
463
|
console.log(chalk.dim(` Status: `) + statusIcon(lastStatus(data)));
|
|
473
464
|
console.log(chalk.dim(` Last run: `) + lastRun(data));
|
|
474
|
-
console.log(chalk.dim(` Updated: `) +
|
|
465
|
+
console.log(chalk.dim(` Updated: `) + formatDate(data.updatedAt));
|
|
475
466
|
});
|
|
476
467
|
}
|
|
477
468
|
attachGetCommand(scraps, { hidden: true });
|
|
@@ -566,7 +557,7 @@ scraps
|
|
|
566
557
|
usageError(`Invalid --tier "${opts.tier}" (allowed: ${VALID_TIERS.join(', ')})`, { json: opts.json });
|
|
567
558
|
return;
|
|
568
559
|
}
|
|
569
|
-
const data = await
|
|
560
|
+
const data = await spin(() => api.post('/api/scraps', {
|
|
570
561
|
title: opts.title,
|
|
571
562
|
...(opts.url && { url: opts.url }),
|
|
572
563
|
request: opts.request || '',
|
|
@@ -592,129 +583,132 @@ scraps
|
|
|
592
583
|
if (refused)
|
|
593
584
|
process.exitCode = 1;
|
|
594
585
|
});
|
|
595
|
-
// update
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
if (opts.forceTier !== undefined && !VALID_TIERS.includes(opts.forceTier)) {
|
|
621
|
-
usageError(`Invalid --force-tier "${opts.forceTier}" (allowed: ${VALID_TIERS.join(', ')})`, { json: opts.json });
|
|
622
|
-
return;
|
|
623
|
-
}
|
|
624
|
-
const body = {};
|
|
625
|
-
if (opts.title !== undefined)
|
|
626
|
-
body.title = opts.title;
|
|
627
|
-
if (opts.url !== undefined)
|
|
628
|
-
body.url = opts.url;
|
|
629
|
-
if (opts.request !== undefined)
|
|
630
|
-
body.request = opts.request;
|
|
631
|
-
if (opts.description !== undefined)
|
|
632
|
-
body.description = opts.description;
|
|
633
|
-
if (opts.cron === false)
|
|
634
|
-
body.cron = null;
|
|
635
|
-
else if (typeof opts.cron === 'string')
|
|
636
|
-
body.cron = opts.cron === '' ? null : opts.cron;
|
|
637
|
-
if (opts.alert === false)
|
|
638
|
-
body.alert = null;
|
|
639
|
-
else if (typeof opts.alert === 'string')
|
|
640
|
-
body.alert = opts.alert === '' ? null : opts.alert;
|
|
641
|
-
if (opts.autofix === true)
|
|
642
|
-
body.autoFix = true;
|
|
643
|
-
else if (opts.autofix === false)
|
|
644
|
-
body.autoFix = false;
|
|
645
|
-
if (opts.params !== undefined || opts.paramsFile !== undefined) {
|
|
646
|
-
let raw;
|
|
647
|
-
if (opts.paramsFile) {
|
|
648
|
-
const { readFileSync } = await import('fs');
|
|
649
|
-
raw = readFileSync(opts.paramsFile, 'utf8');
|
|
586
|
+
// update — promoted to a top-level verb (#108/#120)
|
|
587
|
+
export function attachUpdateCommand(parent, attachOpts = {}) {
|
|
588
|
+
return parent
|
|
589
|
+
.command('update <id>', attachOpts)
|
|
590
|
+
.description('Update an existing scrap')
|
|
591
|
+
.option('-t, --title <title>', 'New title')
|
|
592
|
+
.option('-u, --url <url>', 'New target URL')
|
|
593
|
+
.option('-r, --request <request>', 'New request')
|
|
594
|
+
.option('-d, --description <text>', 'New description')
|
|
595
|
+
.option('--cron <expression>', 'Cron expression (empty string to disable)')
|
|
596
|
+
.option('--no-cron', 'Disable cron (set to null)')
|
|
597
|
+
.option('--alert <email>', 'Failure alert email (empty string to clear)')
|
|
598
|
+
.option('--no-alert', 'Disable failure alert email (set to null)')
|
|
599
|
+
.option('--autofix', 'Enable AI Fix (auto-recovery on selector breakage)')
|
|
600
|
+
.option('--no-autofix', 'Disable AI Fix')
|
|
601
|
+
.option('-p, --params <json>', 'Runtime params as JSON array of objects (e.g. \'[{"TRAWL.paramName":"value"}]\')')
|
|
602
|
+
.option('--params-file <path>', 'Runtime params from a JSON file')
|
|
603
|
+
.option('--tier <tier>', `Force proxy tier (${VALID_TIERS.join('|')})`)
|
|
604
|
+
.option('--force-tier <tier>', `Raise the proxy-tier ceiling PAST the auto-cap (${VALID_TIERS.join('|')}) — history-gated: may be refused or cost more`)
|
|
605
|
+
.option('--json', 'Output as JSON')
|
|
606
|
+
.action(async (id, opts) => {
|
|
607
|
+
validateObjectId(id);
|
|
608
|
+
if (opts.tier !== undefined && !VALID_TIERS.includes(opts.tier)) {
|
|
609
|
+
usageError(`Invalid --tier "${opts.tier}" (allowed: ${VALID_TIERS.join(', ')})`, { json: opts.json });
|
|
610
|
+
return;
|
|
650
611
|
}
|
|
651
|
-
|
|
652
|
-
|
|
612
|
+
if (opts.forceTier !== undefined && !VALID_TIERS.includes(opts.forceTier)) {
|
|
613
|
+
usageError(`Invalid --force-tier "${opts.forceTier}" (allowed: ${VALID_TIERS.join(', ')})`, { json: opts.json });
|
|
614
|
+
return;
|
|
653
615
|
}
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
616
|
+
const body = {};
|
|
617
|
+
if (opts.title !== undefined)
|
|
618
|
+
body.title = opts.title;
|
|
619
|
+
if (opts.url !== undefined)
|
|
620
|
+
body.url = opts.url;
|
|
621
|
+
if (opts.request !== undefined)
|
|
622
|
+
body.request = opts.request;
|
|
623
|
+
if (opts.description !== undefined)
|
|
624
|
+
body.description = opts.description;
|
|
625
|
+
if (opts.cron === false)
|
|
626
|
+
body.cron = null;
|
|
627
|
+
else if (typeof opts.cron === 'string')
|
|
628
|
+
body.cron = opts.cron === '' ? null : opts.cron;
|
|
629
|
+
if (opts.alert === false)
|
|
630
|
+
body.alert = null;
|
|
631
|
+
else if (typeof opts.alert === 'string')
|
|
632
|
+
body.alert = opts.alert === '' ? null : opts.alert;
|
|
633
|
+
if (opts.autofix === true)
|
|
634
|
+
body.autoFix = true;
|
|
635
|
+
else if (opts.autofix === false)
|
|
636
|
+
body.autoFix = false;
|
|
637
|
+
if (opts.params !== undefined || opts.paramsFile !== undefined) {
|
|
638
|
+
let raw;
|
|
639
|
+
if (opts.paramsFile) {
|
|
640
|
+
const { readFileSync } = await import('fs');
|
|
641
|
+
raw = readFileSync(opts.paramsFile, 'utf8');
|
|
642
|
+
}
|
|
643
|
+
else {
|
|
644
|
+
raw = opts.params;
|
|
645
|
+
}
|
|
646
|
+
let parsed;
|
|
647
|
+
try {
|
|
648
|
+
parsed = JSON.parse(raw);
|
|
649
|
+
}
|
|
650
|
+
catch (e) {
|
|
651
|
+
usageError(`Invalid JSON for --params: ${e.message}`, { json: opts.json });
|
|
652
|
+
return;
|
|
653
|
+
}
|
|
654
|
+
if (!Array.isArray(parsed)) {
|
|
655
|
+
usageError('--params must be a JSON array of objects', { json: opts.json });
|
|
656
|
+
return;
|
|
657
|
+
}
|
|
658
|
+
body.params = parsed;
|
|
657
659
|
}
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
660
|
+
if (opts.tier !== undefined)
|
|
661
|
+
body.proxyTier = opts.tier;
|
|
662
|
+
if (opts.forceTier !== undefined) {
|
|
663
|
+
// Raise the ceiling; also start the run at that tier unless --tier says otherwise.
|
|
664
|
+
body.proxyMaxTier = opts.forceTier;
|
|
665
|
+
if (opts.tier === undefined)
|
|
666
|
+
body.proxyTier = opts.forceTier;
|
|
661
667
|
}
|
|
662
|
-
if (
|
|
663
|
-
|
|
668
|
+
if (Object.keys(body).length === 0) {
|
|
669
|
+
if (opts.json) {
|
|
670
|
+
process.exitCode = reportError(new UsageError('Nothing to update. Provide at least one option.'), { json: true });
|
|
671
|
+
return;
|
|
672
|
+
}
|
|
673
|
+
console.log(chalk.yellow('Nothing to update. Provide at least one option.'));
|
|
664
674
|
return;
|
|
665
675
|
}
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
//
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
676
|
+
const data = await spin(() => api.put(`/api/scraps/${id}`, body), {
|
|
677
|
+
text: 'Updating scrap…',
|
|
678
|
+
successText: (d) => `Scrap updated: ${chalk.bold(d._id)}`,
|
|
679
|
+
});
|
|
680
|
+
// #1559 / #86 findings 4b/5 — surface the effective tier + clamp/refuse
|
|
681
|
+
// reason (fixes the silent-clamp: the server may persist a lower tier
|
|
682
|
+
// than requested), and NEVER echo the requested value as applied when
|
|
683
|
+
// the server doesn't confirm it (old-server fallback below).
|
|
684
|
+
const tierWasRequested = opts.tier !== undefined || opts.forceTier !== undefined;
|
|
685
|
+
warnIfUnconfirmedTier(data, tierWasRequested, id);
|
|
686
|
+
const refused = Boolean(data._tierOverride?.refused);
|
|
677
687
|
if (opts.json) {
|
|
678
|
-
|
|
688
|
+
if (refused) {
|
|
689
|
+
process.exitCode = reportTierRefusal(data, true);
|
|
690
|
+
return;
|
|
691
|
+
}
|
|
692
|
+
json(withTierUnconfirmed(data, tierWasRequested));
|
|
679
693
|
return;
|
|
680
694
|
}
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
|
|
691
|
-
// the server doesn't confirm it (old-server fallback below).
|
|
692
|
-
const tierWasRequested = opts.tier !== undefined || opts.forceTier !== undefined;
|
|
693
|
-
warnIfUnconfirmedTier(data, tierWasRequested, id);
|
|
694
|
-
const refused = Boolean(data._tierOverride?.refused);
|
|
695
|
-
if (opts.json) {
|
|
696
|
-
if (refused) {
|
|
697
|
-
process.exitCode = reportTierRefusal(data, true);
|
|
698
|
-
return;
|
|
695
|
+
const shown = data;
|
|
696
|
+
for (const key of Object.keys(body)) {
|
|
697
|
+
// Tier keys are rendered exclusively by renderTierOverrideHuman /
|
|
698
|
+
// warnIfUnconfirmedTier above — never echo them here, whether or not
|
|
699
|
+
// _tierOverride came back (an old-server echo of the REQUESTED value
|
|
700
|
+
// is exactly the silent-clamp lie #1559 fixed).
|
|
701
|
+
if (key === 'proxyTier' || key === 'proxyMaxTier')
|
|
702
|
+
continue;
|
|
703
|
+
const src = key in shown ? shown[key] : body[key];
|
|
704
|
+
console.log(chalk.dim(` ${key}: `) + String(src ?? '—'));
|
|
699
705
|
}
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
// warnIfUnconfirmedTier above — never echo them here, whether or not
|
|
707
|
-
// _tierOverride came back (an old-server echo of the REQUESTED value
|
|
708
|
-
// is exactly the silent-clamp lie #1559 fixed).
|
|
709
|
-
if (key === 'proxyTier' || key === 'proxyMaxTier')
|
|
710
|
-
continue;
|
|
711
|
-
const src = key in shown ? shown[key] : body[key];
|
|
712
|
-
console.log(chalk.dim(` ${key}: `) + String(src ?? '—'));
|
|
713
|
-
}
|
|
714
|
-
renderTierOverrideHuman(data);
|
|
715
|
-
if (refused)
|
|
716
|
-
process.exitCode = 1;
|
|
717
|
-
});
|
|
706
|
+
renderTierOverrideHuman(data);
|
|
707
|
+
if (refused)
|
|
708
|
+
process.exitCode = 1;
|
|
709
|
+
});
|
|
710
|
+
}
|
|
711
|
+
attachUpdateCommand(scraps, { hidden: true });
|
|
718
712
|
// run — promoted to a top-level verb (#108)
|
|
719
713
|
export function attachRunCommand(parent, attachOpts = {}) {
|
|
720
714
|
return parent
|
|
@@ -735,7 +729,7 @@ export function attachRunCommand(parent, attachOpts = {}) {
|
|
|
735
729
|
// all, mirroring `trawl create`'s own --json handling.
|
|
736
730
|
const data = opts.json
|
|
737
731
|
? await call()
|
|
738
|
-
: await
|
|
732
|
+
: await spin(call, { text: 'Launching scrap…', successText: 'Scrap launched' });
|
|
739
733
|
if (opts.json)
|
|
740
734
|
json(data);
|
|
741
735
|
if (opts.watch) {
|
|
@@ -830,7 +824,7 @@ export function attachDataCommand(parent, attachOpts = {}) {
|
|
|
830
824
|
if (opts.fresh) {
|
|
831
825
|
// #91 P0 — same long-run endpoint as `run` (30-250s server-side);
|
|
832
826
|
// the 30s default was aborting it mid-flight.
|
|
833
|
-
const loaded = await
|
|
827
|
+
const loaded = await spin(() => api.get(`/api/scraps/load/${id}`, { timeoutMs: LONG_RUN_TIMEOUT_MS }), {
|
|
834
828
|
text: 'Launching a fresh scrap run (consumes execute quota)…',
|
|
835
829
|
successText: 'Fresh run complete',
|
|
836
830
|
});
|
|
@@ -1050,7 +1044,7 @@ scraps
|
|
|
1050
1044
|
json({ deleted: true, id });
|
|
1051
1045
|
return;
|
|
1052
1046
|
}
|
|
1053
|
-
await
|
|
1047
|
+
await spin(call, { text: 'Deleting…', successText: 'Scrap deleted' });
|
|
1054
1048
|
});
|
|
1055
1049
|
// banner
|
|
1056
1050
|
scraps
|
|
@@ -1092,7 +1086,7 @@ scraps
|
|
|
1092
1086
|
json(data);
|
|
1093
1087
|
return;
|
|
1094
1088
|
}
|
|
1095
|
-
await
|
|
1089
|
+
await spin(call, {
|
|
1096
1090
|
text: 'Uploading banner…',
|
|
1097
1091
|
successText: `Banner uploaded for scrap ${chalk.bold(id)}`,
|
|
1098
1092
|
});
|
|
@@ -1135,7 +1129,7 @@ export function attachTriggerCommand(parent, attachOpts = {}) {
|
|
|
1135
1129
|
// #107 — under --json the stdout path stays pure: no spinner channel.
|
|
1136
1130
|
const data = opts.json
|
|
1137
1131
|
? await call()
|
|
1138
|
-
: await
|
|
1132
|
+
: await spin(call, {
|
|
1139
1133
|
text: opts.wait ? 'Running worker…' : 'Triggering worker…',
|
|
1140
1134
|
successText: opts.wait ? 'Worker run complete' : 'Worker triggered',
|
|
1141
1135
|
});
|
|
@@ -1214,7 +1208,7 @@ account
|
|
|
1214
1208
|
json(data);
|
|
1215
1209
|
return;
|
|
1216
1210
|
}
|
|
1217
|
-
const data = await
|
|
1211
|
+
const data = await spin(call, { text: 'Saving credentials…', successText: 'Credentials saved' });
|
|
1218
1212
|
const acc = data.account;
|
|
1219
1213
|
console.log(chalk.dim(' Credentials: ') + (acc.hasCredentials ? chalk.green('✓ configured') : chalk.dim('not set')));
|
|
1220
1214
|
console.log(chalk.dim(' Session: ') + (acc.hasSession ? chalk.green('active') : chalk.dim('none')));
|
|
@@ -1244,7 +1238,7 @@ account
|
|
|
1244
1238
|
json({ deleted: true, id });
|
|
1245
1239
|
return;
|
|
1246
1240
|
}
|
|
1247
|
-
await
|
|
1241
|
+
await spin(call, {
|
|
1248
1242
|
text: 'Deleting credentials…',
|
|
1249
1243
|
successText: 'Account credentials deleted',
|
|
1250
1244
|
});
|
|
@@ -1262,7 +1256,7 @@ account
|
|
|
1262
1256
|
json({ cleared: true, id });
|
|
1263
1257
|
return;
|
|
1264
1258
|
}
|
|
1265
|
-
await
|
|
1259
|
+
await spin(call, {
|
|
1266
1260
|
text: 'Clearing session…',
|
|
1267
1261
|
successText: 'Session cleared',
|
|
1268
1262
|
});
|
|
@@ -1311,7 +1305,7 @@ accountSession
|
|
|
1311
1305
|
json(data);
|
|
1312
1306
|
return;
|
|
1313
1307
|
}
|
|
1314
|
-
const data = await
|
|
1308
|
+
const data = await spin(call, {
|
|
1315
1309
|
text: 'Uploading session cookies…',
|
|
1316
1310
|
successText: `Session cookies saved for scrap ${chalk.bold(id)}`,
|
|
1317
1311
|
});
|
|
@@ -1325,7 +1319,7 @@ account
|
|
|
1325
1319
|
.option('--json', 'Output as JSON')
|
|
1326
1320
|
.action(async (id, opts) => {
|
|
1327
1321
|
validateObjectId(id);
|
|
1328
|
-
const data = await
|
|
1322
|
+
const data = await spin(() => api.get(`/api/scraps/${id}`), 'Fetching scrap…');
|
|
1329
1323
|
const acc = data.account;
|
|
1330
1324
|
if (opts.json) {
|
|
1331
1325
|
// #86 finding 12 — `json` is already statically imported at the top of
|
package/dist/index.js
CHANGED
|
@@ -4,7 +4,7 @@ import { readFileSync, realpathSync } from 'node:fs';
|
|
|
4
4
|
import { fileURLToPath } from 'node:url';
|
|
5
5
|
import { dirname, join } from 'node:path';
|
|
6
6
|
import { login, logout } from './commands/login.js';
|
|
7
|
-
import { scraps, attachListCommand, attachGetCommand, attachRunCommand, attachDataCommand, attachHistoryCommand, attachRunInfoCommand, attachTriggerCommand, } from './commands/scraps.js';
|
|
7
|
+
import { scraps, attachListCommand, attachGetCommand, attachRunCommand, attachDataCommand, attachHistoryCommand, attachRunInfoCommand, attachTriggerCommand, attachUpdateCommand, } from './commands/scraps.js';
|
|
8
8
|
import { skills } from './commands/skills.js';
|
|
9
9
|
import { telemetry } from './commands/telemetry.js';
|
|
10
10
|
import { token } from './commands/token.js';
|
|
@@ -108,11 +108,14 @@ export function createProgram() {
|
|
|
108
108
|
attachHistoryCommand(program);
|
|
109
109
|
attachRunInfoCommand(program);
|
|
110
110
|
attachTriggerCommand(program);
|
|
111
|
+
attachUpdateCommand(program);
|
|
111
112
|
program.addCommand(whoami);
|
|
112
113
|
program.addCommand(ping);
|
|
113
114
|
// Management (#108) — human/CI surface, grouped below. `scraps` still
|
|
114
|
-
// holds every pre-#108 management command (create/
|
|
115
|
-
// watch/account.*/session.*/doctor/autofix/snapshot) exactly as before
|
|
115
|
+
// holds every pre-#108 management command (create/delete/banner/
|
|
116
|
+
// watch/account.*/session.*/doctor/autofix/snapshot) exactly as before —
|
|
117
|
+
// `update` is now a promoted core verb (#120), still reachable (hidden)
|
|
118
|
+
// under `scraps` for backward compatibility.
|
|
116
119
|
program.commandsGroup(MANAGEMENT_GROUP);
|
|
117
120
|
program.addCommand(scraps);
|
|
118
121
|
program.addCommand(skills);
|
package/dist/lib/format.d.ts
CHANGED
|
@@ -1,2 +1,9 @@
|
|
|
1
1
|
export declare function table(rows: Record<string, unknown>[], columns: string[]): void;
|
|
2
2
|
export declare function json(data: unknown): void;
|
|
3
|
+
/**
|
|
4
|
+
* One date format across the whole CLI: `DD/MM/YY HH:mm` (local time).
|
|
5
|
+
* #119 — `list` mixed `DD/MM/YY` (last-run col) with `M/D/YYYY,
|
|
6
|
+
* h:mm:ss AM/PM` (toLocaleString), ambiguous day/month side by side.
|
|
7
|
+
* Returns `—` for a missing/invalid date.
|
|
8
|
+
*/
|
|
9
|
+
export declare function formatDate(value: string | number | Date | null | undefined): string;
|
package/dist/lib/format.js
CHANGED
|
@@ -23,3 +23,22 @@ export function table(rows, columns) {
|
|
|
23
23
|
export function json(data) {
|
|
24
24
|
console.log(JSON.stringify(data, null, 2));
|
|
25
25
|
}
|
|
26
|
+
/**
|
|
27
|
+
* One date format across the whole CLI: `DD/MM/YY HH:mm` (local time).
|
|
28
|
+
* #119 — `list` mixed `DD/MM/YY` (last-run col) with `M/D/YYYY,
|
|
29
|
+
* h:mm:ss AM/PM` (toLocaleString), ambiguous day/month side by side.
|
|
30
|
+
* Returns `—` for a missing/invalid date.
|
|
31
|
+
*/
|
|
32
|
+
export function formatDate(value) {
|
|
33
|
+
if (value === null || value === undefined)
|
|
34
|
+
return '—';
|
|
35
|
+
const d = new Date(value);
|
|
36
|
+
if (Number.isNaN(d.getTime()))
|
|
37
|
+
return '—';
|
|
38
|
+
const dd = String(d.getDate()).padStart(2, '0');
|
|
39
|
+
const mm = String(d.getMonth() + 1).padStart(2, '0');
|
|
40
|
+
const yy = String(d.getFullYear()).slice(2);
|
|
41
|
+
const hh = String(d.getHours()).padStart(2, '0');
|
|
42
|
+
const min = String(d.getMinutes()).padStart(2, '0');
|
|
43
|
+
return `${dd}/${mm}/${yy} ${hh}:${min}`;
|
|
44
|
+
}
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
type Action<T> = PromiseLike<T> | (() => PromiseLike<T>);
|
|
2
|
+
type SpinOptions<T> = string | {
|
|
3
|
+
text?: string;
|
|
4
|
+
successText?: string | ((result: T) => string);
|
|
5
|
+
failText?: string | ((error: Error) => string);
|
|
6
|
+
[key: string]: unknown;
|
|
7
|
+
};
|
|
8
|
+
/**
|
|
9
|
+
* Drop-in for `oraPromise` that emits NOTHING when stderr is not a TTY.
|
|
10
|
+
*
|
|
11
|
+
* #119 — under a pipe / non-TTY, `oraPromise` still printed the start text AND
|
|
12
|
+
* a persisted `✔ …` line, so `trawl list | cat` showed the spinner caption
|
|
13
|
+
* twice. An agent/CI never wants spinner chrome; here we just run the action.
|
|
14
|
+
* When stderr IS a TTY, behaviour is identical to `oraPromise` (spinner writes
|
|
15
|
+
* to stderr, stdout stays clean either way).
|
|
16
|
+
*/
|
|
17
|
+
export declare function spin<T>(action: Action<T>, options?: SpinOptions<T>): Promise<T>;
|
|
18
|
+
export {};
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import { oraPromise } from 'ora';
|
|
2
|
+
/**
|
|
3
|
+
* Drop-in for `oraPromise` that emits NOTHING when stderr is not a TTY.
|
|
4
|
+
*
|
|
5
|
+
* #119 — under a pipe / non-TTY, `oraPromise` still printed the start text AND
|
|
6
|
+
* a persisted `✔ …` line, so `trawl list | cat` showed the spinner caption
|
|
7
|
+
* twice. An agent/CI never wants spinner chrome; here we just run the action.
|
|
8
|
+
* When stderr IS a TTY, behaviour is identical to `oraPromise` (spinner writes
|
|
9
|
+
* to stderr, stdout stays clean either way).
|
|
10
|
+
*/
|
|
11
|
+
export function spin(action, options) {
|
|
12
|
+
if (!process.stderr.isTTY) {
|
|
13
|
+
return Promise.resolve(typeof action === 'function' ? action() : action);
|
|
14
|
+
}
|
|
15
|
+
// oraPromise's own overloads accept (action, string) and (action, options).
|
|
16
|
+
return oraPromise(action, options);
|
|
17
|
+
}
|
package/docs/agent-quickstart.md
CHANGED
|
@@ -20,7 +20,7 @@ agent should never need them.)
|
|
|
20
20
|
|
|
21
21
|
## Core commands (agent + human)
|
|
22
22
|
|
|
23
|
-
These
|
|
23
|
+
These eleven commands are the CLI's agent+human surface — `--json` is
|
|
24
24
|
first-class on every one, and none of them ever blocks on a prompt (see
|
|
25
25
|
[Non-interactive contract](#non-interactive-contract) below):
|
|
26
26
|
|
|
@@ -34,6 +34,8 @@ trawl data <id> [--json] [--fresh] [--errors] Get scrap data (last persisted ru
|
|
|
34
34
|
trawl history <id> [--json] [-n <limit>] List past runs for a scrap
|
|
35
35
|
trawl run-info <hid> [--json] Show details of a single run
|
|
36
36
|
trawl trigger <id> [--watch] [--wait] [--json] Launch a scrap as a background worker
|
|
37
|
+
trawl update <id> [-t <title>] [-u <url>] [-r <request>] [-d <description>] [--cron <expr>|--no-cron] [--alert <email>|--no-alert] [--autofix|--no-autofix] [-p <json>|--params-file <path>] [--tier <t>] [--force-tier <t>] [--json]
|
|
38
|
+
Update an existing scrap
|
|
37
39
|
trawl whoami [--json] Show the authenticated user's identity
|
|
38
40
|
trawl ping [--json] Health/version handshake against the Trawl API
|
|
39
41
|
```
|
|
@@ -59,7 +61,7 @@ default.
|
|
|
59
61
|
> scheduled to re-run every day at 07:00 UTC by default
|
|
60
62
|
> (`cron: "0 7 * * *"`, unrelated to `--no-autofix`) — each recurring run
|
|
61
63
|
> consumes execute quota. Disable or change it once you've reviewed the
|
|
62
|
-
> scrap: `trawl
|
|
64
|
+
> scrap: `trawl update <id> --no-cron` (or `--cron <expr>`). Finally,
|
|
63
65
|
> if `TRAWL_TIMEOUT` is set globally for a tighter budget than 300s, it
|
|
64
66
|
> clamps `create`'s ceiling too (env always wins) — unset it or raise it
|
|
65
67
|
> before calling `create`.
|
|
@@ -69,9 +71,10 @@ default.
|
|
|
69
71
|
> a hidden alias. Prefer the bare top-level form above; it's what
|
|
70
72
|
> `trawl --help` now shows.
|
|
71
73
|
|
|
72
|
-
For the full flag reference (tier overrides on `scraps create
|
|
73
|
-
update` — the core `create` verb
|
|
74
|
-
`--
|
|
74
|
+
For the full flag reference (tier overrides on `scraps create` (management)
|
|
75
|
+
and the core `update` verb above — note the core `create` verb has no
|
|
76
|
+
`--tier` of its own, unlike `scraps create`, the `--watch` polling mechanics,
|
|
77
|
+
retention/regression semantics on `data`, …)
|
|
75
78
|
see the README's [Core commands](../README.md#core-commands-agent--human) section
|
|
76
79
|
— this doc intentionally stays minimal.
|
|
77
80
|
|