@gobing-ai/knowledge-kit 0.1.13 → 0.1.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/dist/index.js +796 -176
- package/package.json +1 -1
- package/plugins/generations/content-gen/dist/index.js +2 -2
- package/plugins/generations/core-facts-gen/dist/index.js +2 -2
- package/plugins/generations/daily-article-gen/dist/index.js +2 -2
- package/plugins/generations/dailynews-gen/dist/index.js +72 -9
- package/plugins/generations/dailynews-gen/src/index.ts +23 -1
- package/plugins/generations/dailynews-gen/src/script-plan.ts +103 -5
- package/plugins/generations/episode-plan-gen/dist/index.js +2 -2
- package/plugins/generations/gemini-tts/dist/index.js +2 -2
- package/plugins/generations/image-gen/dist/index.js +2 -2
- package/plugins/generations/news-report-gen/dist/index.js +2 -2
- package/plugins/generations/voice-gen/dist/index.js +2 -2
- package/plugins/generations/voxcpm-tts/dist/index.js +2 -2
- package/plugins/ingestions/aihot-ingest/dist/index.js +2 -2
- package/plugins/ingestions/horizon-ingest/dist/index.js +2 -2
- package/plugins/ingestions/last30days-ingest/dist/index.js +2 -2
- package/plugins/ingestions/web-search/dist/index.js +2 -2
- package/plugins/kk/plugin.json +1 -1
- package/plugins/kk/scripts/daily-scheduled-run.ts +53 -5
- package/plugins/kk/scripts/daily-stages.ts +476 -12
- package/plugins/kk/scripts/kk-workflow-stages.ts +20 -4
- package/plugins/kk/skills/content-judge/SKILL.md +5 -4
- package/plugins/kk/skills/content-judge/references/rubrics.md +14 -0
- package/plugins/kk/skills/news-editorial/SKILL.md +122 -0
- package/plugins/kk/skills/news-editorial/references/selection.md +47 -0
- package/plugins/kk/skills/news-editorial/references/style-contract.zh.md +99 -0
- package/plugins/kk/skills/publish/SKILL.md +3 -3
- package/plugins/kk/skills/web-automation/SKILL.md +229 -77
- package/plugins/kk/skills/web-automation/references/browser-launch.md +127 -36
- package/plugins/kk/workflows/kk-daily-ai-voice2.yaml +4 -6
- package/plugins/kk/workflows/kk-daily-ai-voice3.yaml +183 -222
- package/plugins/kk/workflows/kk-daily-publish3.yaml +16 -8
- package/plugins/publishings/bilibili-pub/dist/index.js +5 -3
- package/plugins/publishings/bilibili-pub/dist/profile-check.js +3 -1
- package/plugins/publishings/cnblogs-pub/dist/index.js +5 -3
- package/plugins/publishings/cnblogs-pub/dist/profile-check.js +3 -1
- package/plugins/publishings/csdn-pub/dist/index.js +5 -3
- package/plugins/publishings/csdn-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/emdash-pub/dist/index.js +2 -2
- package/plugins/publishings/infoq-pub/dist/index.js +5 -3
- package/plugins/publishings/infoq-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/jike-pub/dist/index.js +5 -3
- package/plugins/publishings/jike-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/juejin-pub/dist/index.js +5 -3
- package/plugins/publishings/juejin-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/linkedin-pub/dist/index.js +5 -3
- package/plugins/publishings/linkedin-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/medium-pub/dist/index.js +2 -2
- package/plugins/publishings/podcast-pub/dist/index.js +147 -4
- package/plugins/publishings/podcast-pub/src/index.ts +71 -2
- package/plugins/publishings/podcast-pub/src/map.ts +3 -0
- package/plugins/publishings/podcast-pub/src/transcript.ts +195 -0
- package/plugins/publishings/qiita-pub/dist/index.js +2 -2
- package/plugins/publishings/shipinhao-pub/dist/index.js +5 -3
- package/plugins/publishings/shipinhao-pub/dist/profile-check.js +3 -1
- package/plugins/publishings/sspai-pub/dist/index.js +5 -3
- package/plugins/publishings/sspai-pub/dist/profile-check.js +3 -1
- package/plugins/publishings/substack-pub/dist/index.js +5 -3
- package/plugins/publishings/substack-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/surfdash-pub/dist/index.js +2 -2
- package/plugins/publishings/threads-pub/dist/index.js +2 -2
- package/plugins/publishings/wechatmp-pub/dist/index.js +52 -33
- package/plugins/publishings/wechatmp-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/wechatmp-pub/src/flow.ts +70 -33
- package/plugins/publishings/weibo-pub/dist/index.js +21 -6
- package/plugins/publishings/weibo-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/weibo-pub/src/index.ts +31 -7
- package/plugins/publishings/x-pub/dist/index.js +5 -3
- package/plugins/publishings/x-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/xhs-pub/dist/index.js +5 -3
- package/plugins/publishings/xhs-pub/dist/profile-check.js +5 -3
- package/plugins/publishings/xiaoyuzhou-pub/dist/index.js +5 -3
- package/plugins/publishings/xiaoyuzhou-pub/dist/profile-check.js +3 -1
- package/plugins/publishings/zenn-pub/dist/index.js +2 -2
- package/plugins/kk/scripts/daily-voice-style.zh.md +0 -70
|
@@ -1,101 +1,253 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: web-automation
|
|
3
3
|
description: >-
|
|
4
|
-
Discover, enhance, and automate web tasks with Playwright
|
|
5
|
-
over CDP
|
|
4
|
+
Discover, enhance, and automate web tasks with Playwright, kk bot, or agent-browser
|
|
5
|
+
over CDP — "walk this console and record evidence", "figure out this site's
|
|
6
6
|
UI", "automate this web flow", "reuse my cached login on <channel>", "check
|
|
7
|
-
whether my session on <channel> is still valid", "
|
|
8
|
-
selectors", or when a spur workflow `agent.run` step targets the
|
|
9
|
-
web-automation skill.
|
|
10
|
-
plugins; orchestration is
|
|
11
|
-
transition-flow (see docs/design/web-automation-engine.md).
|
|
12
|
-
requires: [cli:spur, local-fs]
|
|
7
|
+
whether my session on <channel> is still valid", "run a browser collection plan",
|
|
8
|
+
"fix this channel's selectors", or when a spur workflow `agent.run` step targets the
|
|
9
|
+
web-automation skill. Coordinates with the kk bot session lifecycle and collection
|
|
10
|
+
substrate. Owns method and evidence; transport stays in kk plugins; orchestration is
|
|
11
|
+
a @gobing-ai/ts-dual-workflow-engine transition-flow (see docs/design/web-automation-engine.md).
|
|
12
|
+
requires: [cli:kk, cli:spur, local-fs]
|
|
13
13
|
---
|
|
14
14
|
|
|
15
15
|
# web-automation — discover, enhance, automate with a real browser
|
|
16
16
|
|
|
17
17
|
## Purpose
|
|
18
18
|
|
|
19
|
-
Three
|
|
20
|
-
cached login profile**, record **evidence** (screenshots, DOM signals, JSON),
|
|
19
|
+
Three operational lenses over one unified browser automation architecture: drive a **real Chromium on the named bot's
|
|
20
|
+
cached login profile**, record **verifiable evidence** (screenshots, DOM signals, JSON responses),
|
|
21
21
|
and never report an outcome that was not observed.
|
|
22
22
|
|
|
23
|
-
| Operation | When | Output |
|
|
24
|
-
| --- | --- | --- |
|
|
25
|
-
| **discover** |
|
|
26
|
-
| **enhance** |
|
|
27
|
-
| **automate** |
|
|
23
|
+
| Operation | When | Tier & Substrate | Output |
|
|
24
|
+
| --- | --- | --- | --- |
|
|
25
|
+
| **discover** | New/changed site; "figure out this UI", inspect selectors/shadow roots | **Tier 3** (`kk bot open` + CDP/Playwright) | Evidence dir (`frames/`, `signals.json`) + drift list (`docs/design/<channel>-ui-discovery.md`) |
|
|
26
|
+
| **enhance** | Selectors drifted; a plugin or collection plan fails on a live console | **Tier 2 / 3** (`kk bot exec`, probe script) | Root-cause fix in plugin + updated discovery doc in same commit |
|
|
27
|
+
| **automate** | Repeatable web task (data collection, form fill, draft save, health audit) | **Tier 1 / 2** (`kk bot exec`, `runBotCollection`) | Bounded Result + captured artifacts (draft-safe by default, fail-closed) |
|
|
28
28
|
|
|
29
|
-
This skill owns **method**. It does not own transport (that
|
|
30
|
-
|
|
31
|
-
`spur workflow` runs `@gobing-ai/ts-dual-workflow-engine`; see
|
|
32
|
-
`docs/design/web-automation-engine.md` for the layering and the YAML).
|
|
29
|
+
This skill owns **method**. It does not own transport (that belongs to `kind: publish` / `ingestion` plugins) or orchestration
|
|
30
|
+
(that is the engine — `spur workflow` running `@gobing-ai/ts-dual-workflow-engine`; see `docs/design/web-automation-engine.md`).
|
|
33
31
|
|
|
34
|
-
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## The 3-Tier Web Automation Model
|
|
35
|
+
|
|
36
|
+
To keep workflows clean, safe, and robust against anti-bot friction, all web automation tasks follow a strict 3-tier hierarchy:
|
|
37
|
+
|
|
38
|
+
```text
|
|
39
|
+
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
40
|
+
│ Tier 1: Session & Probe Gateway (kk bot CLI) │
|
|
41
|
+
│ • Profile inventory & expiry estimates: kk bot list │
|
|
42
|
+
│ • Headed login & transactional staging: kk bot login <channel> │
|
|
43
|
+
│ • Single-request authenticated queries: kk bot exec <name> <url> │
|
|
44
|
+
│ • Local credential wipe: kk bot logout <name> │
|
|
45
|
+
└──────────────────────────────────────────────────────────────────────────────┘
|
|
46
|
+
│
|
|
47
|
+
▼
|
|
48
|
+
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
49
|
+
│ Tier 2: Declarative Headless Automation (kk bot plan / bot-collection) │
|
|
50
|
+
│ • Canonical CLI runner: kk bot plan <bot> <plan.json> [-o <dir>] [-m <sec>]│
|
|
51
|
+
│ • Bounded multi-step JSON plans (navigate, wait, click, scroll, extract) │
|
|
52
|
+
│ • Declarative network response captures (responses: [{pathPrefix}]) │
|
|
53
|
+
│ • Single profile lease + single headless context launch │
|
|
54
|
+
│ • Fail-closed errors: authentication (auth walls) vs empty vs browser │
|
|
55
|
+
│ • Programmatic runner: runBotCollection in @gobing-ai/publish-harness │
|
|
56
|
+
│ • Low-level probe tool: bun packages/publish-harness/scripts/bot-collection-probe.ts │
|
|
57
|
+
└──────────────────────────────────────────────────────────────────────────────┘
|
|
58
|
+
│
|
|
59
|
+
▼
|
|
60
|
+
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
61
|
+
│ Tier 3: Deep Interactive Automation (Playwright / CDP under withProfileLease)│
|
|
62
|
+
│ • Headed visual inspection & interaction: kk bot open <name> [--login] │
|
|
63
|
+
│ • File uploads (setInputFiles) & shadow-root coordinate clicks via CDP │
|
|
64
|
+
│ • Live DOM accessibility dumps & interactive selector reverse-engineering │
|
|
65
|
+
│ • Strict lease enforcement: withProfileLease prevents concurrent corruption │
|
|
66
|
+
└──────────────────────────────────────────────────────────────────────────────┘
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## Tier 1: Session & Profile Lifecycle (`kk bot`)
|
|
72
|
+
|
|
73
|
+
All browser sessions live under the fixed default root:
|
|
74
|
+
|
|
75
|
+
```text
|
|
76
|
+
~/.config/kk/browser-profiles/<name> # persistent profiles (xhs, infoq, juejin, substack, x, wechatmp, etc.)
|
|
77
|
+
~/.config/kk/browser-transactions/ # crash-safe staging and atomic promotion for logins
|
|
78
|
+
~/.config/kk/browser-leases/ # shared lease locks preventing concurrent profile corruption
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### CLI Command Reference
|
|
35
82
|
|
|
36
|
-
|
|
37
|
-
|
|
83
|
+
| Command | Purpose | Input / Options | Behavior |
|
|
84
|
+
| --- | --- | --- | --- |
|
|
85
|
+
| `kk bot list` | Inventory | None | Prints tab-separated columns: `name`, `cached`, `defaultUrl`, `identity`, `lastLogin`, `expires`. Browser-free. |
|
|
86
|
+
| `kk bot login <channel>` / `<url> --save <name>` | Fresh authentication | Channel name or URL (`--save <name>`) | Opens headed in isolated staging; saves adapter-verified evidence or waits for operator confirmation; promotes atomically. |
|
|
87
|
+
| `kk bot open <name> [--login]` | Inspection | Bot name (`--login` optional) | Opens saved default URL headed until operator closes browser. `--login` re-authenticates first. |
|
|
88
|
+
| `kk bot logout <name>` | Wipe credentials | Bot name | Deletes local profile directory behind shared lease. Idempotent. |
|
|
89
|
+
| `kk bot exec <name> <url> [opts]` | Authenticated HTTP | `-X`, `-H`, `-d`, `-o`, `-i`, `-L`, `-f`, `-m` | One headless persistent launch; sends request with profile's cached cookies; writes body as raw bytes. |
|
|
90
|
+
| `kk bot plan <name> <plan.json> [opts]` | Declarative Collection | `-o, --output <dir>`, `-m, --max-time <sec>` | Headless persistent launch under shared lease; runs declarative plan steps and captures; writes artifacts or prints JSON result. |
|
|
38
91
|
|
|
92
|
+
### Session Preflight & Health Check
|
|
93
|
+
|
|
94
|
+
Before triggering automated runs, perform a fast probe. Age-guessing timestamps is banned — probe, don't predict:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
# 1. Check local profile inventory
|
|
98
|
+
kk bot list
|
|
99
|
+
|
|
100
|
+
# 2. Lightweight authenticated query probe (returns status + headers)
|
|
101
|
+
kk bot exec <channel> <console-url> -i -m 15 -o /tmp/probe.txt
|
|
102
|
+
|
|
103
|
+
# 3. If expired or unauthenticated (HTTP 401/403 or redirect to login):
|
|
104
|
+
kk bot login <channel>
|
|
39
105
|
```
|
|
40
|
-
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Tier 2: Declarative Headless Automation (`bot-collection`)
|
|
110
|
+
|
|
111
|
+
For data collection, scraping, site audits, and structured form workflows, **do not write ad-hoc Playwright scripts**.
|
|
112
|
+
Use the declarative `@gobing-ai/publish-harness/bot-collection` substrate:
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
import { runBotCollection } from '@gobing-ai/publish-harness/bot-collection';
|
|
116
|
+
|
|
117
|
+
const result = await runBotCollection('wechatmp', {
|
|
118
|
+
steps: [
|
|
119
|
+
{ id: 'nav-home', kind: 'navigate', url: 'https://mp.weixin.qq.com/' },
|
|
120
|
+
{ id: 'wait-dashboard', kind: 'wait', selector: '.weui-desktop-layout', timeoutMs: 15_000 },
|
|
121
|
+
{ id: 'extract-stats', kind: 'extract', selector: '.weui-desktop-mass__item', format: 'text' },
|
|
122
|
+
],
|
|
123
|
+
responses: [
|
|
124
|
+
{ id: 'appmsg-list', pathPrefix: '/cgi-bin/appmsg' },
|
|
125
|
+
],
|
|
126
|
+
timeoutMs: 30_000,
|
|
127
|
+
});
|
|
41
128
|
```
|
|
42
129
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
130
|
+
### Plan Grammar (`BotCollectionPlan`)
|
|
131
|
+
|
|
132
|
+
A plan is a frozen JSON object with `steps[]`, optional `responses[]` captures, and an optional whole-run `timeoutMs`:
|
|
133
|
+
|
|
134
|
+
- **`navigate`**: `{ id, kind: 'navigate', url, waitUntil?: 'load'|'domcontentloaded'|'networkidle'|'commit' }`
|
|
135
|
+
- **`wait`**: `{ id, kind: 'wait', selector?, timeoutMs? }`
|
|
136
|
+
- **`click`**: `{ id, kind: 'click', selector, timeoutMs? }`
|
|
137
|
+
- **`scroll`**: `{ id, kind: 'scroll', deltaY, selector?, timeoutMs? }`
|
|
138
|
+
- **`extract`**: `{ id, kind: 'extract', selector, format: 'text'|'html'|'count', timeoutMs? }`
|
|
139
|
+
- **`request`**: `{ id, kind: 'request', url, method?: 'GET'|'POST', headers?, data? }`
|
|
140
|
+
- **`responses`**: `[{ id, origin?, pathPrefix, optional?: boolean }]`
|
|
47
141
|
|
|
48
|
-
|
|
142
|
+
### Fail-Closed Integrity Rules
|
|
143
|
+
|
|
144
|
+
- **`authentication`**: Fails immediately when an explicit 401/403 occurs on a request, navigation, or required response, or when the final navigation URL path matches `/login|signin|sign-in|auth|sso/i`. Action: alert operator to run `kk bot login <channel>`.
|
|
145
|
+
- **`empty`**: Fails immediately when a declared non-optional capture receives no response, or an `extract`/`request` step yields an empty body. Prevents client-side rendered SPA shells from masquerading as a successful quiet run.
|
|
146
|
+
- **`browser`**: Transport faults, timeouts, navigation drops, and DOM exceptions.
|
|
147
|
+
|
|
148
|
+
### Running Plans via CLI (`kk bot plan`)
|
|
149
|
+
|
|
150
|
+
Execute a collection plan directly from the terminal or a workflow script:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
# Run plan and stream JSON result to stdout
|
|
154
|
+
kk bot plan <channel> <path/to/plan.json> -m 60
|
|
155
|
+
|
|
156
|
+
# Run plan and write 0600 artifact files + result.json / error.json into output dir
|
|
157
|
+
kk bot plan <channel> <path/to/plan.json> -o <path/to/evidence-dir> -m 60
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Exit codes: `0` = success, `1` = operational failure (`authentication`, `empty`, or `browser`), `2` = usage error (invalid JSON, missing profile, invalid timeout).
|
|
161
|
+
|
|
162
|
+
Low-level debugging can also use the repository probe script:
|
|
49
163
|
|
|
50
164
|
```bash
|
|
51
|
-
|
|
165
|
+
bun packages/publish-harness/scripts/bot-collection-probe.ts \
|
|
166
|
+
--profile <name> \
|
|
167
|
+
--plan <path/to/plan.json> \
|
|
168
|
+
--out <path/to/evidence-dir>
|
|
52
169
|
```
|
|
53
170
|
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
`
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## Tier 3: Deep Interactive Automation (Playwright / CDP)
|
|
174
|
+
|
|
175
|
+
When an activity requires capabilities outside Tier 2 — specifically:
|
|
176
|
+
1. **Interactive UI discovery** (navigating unknown layouts, recording screenshots).
|
|
177
|
+
2. **Shadow-root component piercing** (e.g. XHS creator console shadow boundaries).
|
|
178
|
+
3. **File uploads** (attaching images/videos via `page.setInputFiles`).
|
|
179
|
+
4. **Canvas / complex coordinate dragging**.
|
|
180
|
+
|
|
181
|
+
You MUST wrap the run inside the shared profile lease to prevent profile corruption:
|
|
182
|
+
|
|
183
|
+
```ts
|
|
184
|
+
import { chromium } from 'playwright';
|
|
185
|
+
import { withProfileLease } from '@gobing-ai/publish-harness';
|
|
186
|
+
|
|
187
|
+
const profileName = 'xhs';
|
|
188
|
+
const profileDir = `${process.env.HOME}/.config/kk/browser-profiles/${profileName}`;
|
|
189
|
+
const port = process.env.KK_WEB_CDP ?? '9222';
|
|
190
|
+
|
|
191
|
+
await withProfileLease(profileName, async () => {
|
|
192
|
+
const ctx = await chromium.launchPersistentContext(profileDir, {
|
|
193
|
+
headless: process.env.KK_WEB_HEADLESS === 'true',
|
|
194
|
+
viewport: null,
|
|
195
|
+
args: [
|
|
196
|
+
`--remote-debugging-port=${port}`,
|
|
197
|
+
'--disable-blink-features=AutomationControlled',
|
|
198
|
+
],
|
|
199
|
+
});
|
|
200
|
+
try {
|
|
201
|
+
const page = ctx.pages()[0] ?? (await ctx.newPage());
|
|
202
|
+
// Drive page actions, upload files, or attach agent-browser via CDP...
|
|
203
|
+
} finally {
|
|
204
|
+
await ctx.close();
|
|
205
|
+
}
|
|
206
|
+
});
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## Playbook: Common Automation Scenarios
|
|
212
|
+
|
|
213
|
+
### Scenario A: Site Discovery & Reverse-Engineering
|
|
214
|
+
1. Check cached session: `kk bot list`.
|
|
215
|
+
2. Open target page headed: `kk bot open <channel>`.
|
|
216
|
+
3. Inspect DOM, capture layout, dump accessibility tree, take screenshots.
|
|
217
|
+
4. Document selectors, shadow roots, and observed signals into `docs/design/<channel>-ui-discovery.md`.
|
|
218
|
+
5. Close window; `kk bot open` outputs `<name>\tclosed`.
|
|
219
|
+
|
|
220
|
+
### Scenario B: Lightweight Authenticated Scraping
|
|
221
|
+
1. Send direct authenticated request:
|
|
222
|
+
```bash
|
|
223
|
+
kk bot exec <channel> "https://target.com/api/data" -H "Accept: application/json" -o data.json
|
|
224
|
+
```
|
|
225
|
+
2. If the endpoint returns client-rendered HTML shell rather than JSON, escalate to **Scenario C**.
|
|
226
|
+
|
|
227
|
+
### Scenario C: Declarative Headless Data Collection
|
|
228
|
+
1. Draft a `plan.json` with required navigation, waits, and response captures.
|
|
229
|
+
2. Probe run:
|
|
230
|
+
```bash
|
|
231
|
+
bun packages/publish-harness/scripts/bot-collection-probe.ts --profile <channel> --plan plan.json --out .spur/run/evidence
|
|
232
|
+
```
|
|
233
|
+
3. Verify `result.json` shows all artifacts captured with non-empty bodies.
|
|
234
|
+
4. Wire into ingestion plugin or workflow.
|
|
235
|
+
|
|
236
|
+
### Scenario D: Broken Selector Repair (`enhance`)
|
|
237
|
+
1. Replay failing plan or run `kk bot exec`.
|
|
238
|
+
2. Inspect `error.json` / diagnostics (is it `authentication`, `empty`, or `browser`?).
|
|
239
|
+
3. If selector changed, launch `kk bot open <channel>`, observe the updated DOM element.
|
|
240
|
+
4. Update selector in shared helper, re-verify with probe script, commit alongside discovery doc.
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## Safety Rails (Non-Negotiable)
|
|
245
|
+
|
|
246
|
+
- **Draft-Safe by Default**: Never click publish/send/submit/delete unless the task explicitly authorizes live actions.
|
|
247
|
+
- **Lease Invariant**: Every browser context touching `~/.config/kk/browser-profiles/<name>` must be acquired through `kk bot`, `runBotCollection`, or `withProfileLease`. Direct un-leased `launchPersistentContext` is banned.
|
|
248
|
+
- **Fail Closed on Quiet Failures**: Missing data and auth walls must fail loud (`authentication` or `empty`). Never treat an empty SPA shell or an auth redirect as a normal quiet response.
|
|
249
|
+
- **Headed vs Headless**:
|
|
250
|
+
- Headless: Session probes (`kk bot list`), authenticated queries (`kk bot exec`), declarative collection (`runBotCollection`).
|
|
251
|
+
- Headed: First-time logins (`kk bot login`), interactive discovery (`kk bot open`), and bot-walled platforms (XHS).
|
|
252
|
+
- **Secrets Invariant**: Zero credentials in repo, code, or logs. All state lives exclusively in `~/.config/kk/` outside Git.
|
|
253
|
+
- **Confidence-Verified Verdicts**: Every workflow outcome verdict must declare `confidence: HIGH | MEDIUM | LOW`. Missing or out-of-enum confidence halts execution.
|
|
@@ -1,71 +1,160 @@
|
|
|
1
1
|
# Browser launch & cached-login recipes
|
|
2
2
|
|
|
3
|
-
Everything here is measured on the kk channels (E92 discovery,
|
|
4
|
-
The
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
Everything here is measured on the kk channels (E92 discovery, A22 named bots, and 0346 collection runner).
|
|
4
|
+
The canonical tool for named session management is `kk bot`; the underlying libraries are `@gobing-ai/publish-harness`
|
|
5
|
+
and `@gobing-ai/publish-harness/bot-collection`.
|
|
6
|
+
|
|
7
|
+
## Named bot CLI suite (`kk bot`)
|
|
8
|
+
|
|
9
|
+
The profile root `~/.config/kk/browser-profiles/<name>` is managed safely with transactional staging and leases:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
kk bot list # List cached profiles, last login, and estimated expiry
|
|
13
|
+
kk bot login <channel> # Headed interactive login (delegates to matched channel adapter)
|
|
14
|
+
kk bot login <url> --save <name> # Generic headed login (operator terminal confirmation before save)
|
|
15
|
+
kk bot open <name> [--login] # Open saved page headed until window closes (--login re-authenticates)
|
|
16
|
+
kk bot logout <name> # Delete local profile directory behind the shared lease
|
|
17
|
+
kk bot exec <name> <url> [opts] # Single authenticated HTTP request with cached cookies (curl subset)
|
|
18
|
+
kk bot plan <name> <plan.json> [opts] # Execute declarative collection plan under lease (-o, -m)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Rendered collection runner substrate (task 0346)
|
|
7
22
|
|
|
8
|
-
|
|
23
|
+
For automated structured web data collection without raw browser scripting, use `@gobing-ai/publish-harness/bot-collection`.
|
|
24
|
+
It executes a declarative JSON plan under a single profile lease and a single headless persistent context:
|
|
25
|
+
|
|
26
|
+
```ts
|
|
27
|
+
import { runBotCollection } from '@gobing-ai/publish-harness/bot-collection';
|
|
28
|
+
|
|
29
|
+
const result = await runBotCollection('xhs', {
|
|
30
|
+
steps: [
|
|
31
|
+
{ id: 'home', kind: 'navigate', url: 'https://creator.xiaohongshu.com/' },
|
|
32
|
+
{ id: 'wait-nav', kind: 'wait', selector: '.creator-menu' },
|
|
33
|
+
{ id: 'click-menu', kind: 'click', selector: '.creator-menu >> text="Publish"' },
|
|
34
|
+
{ id: 'extract-data', kind: 'extract', selector: '.data-card', format: 'text' },
|
|
35
|
+
],
|
|
36
|
+
responses: [
|
|
37
|
+
{ id: 'creator-info', pathPrefix: '/api/galaxy/creator/info' },
|
|
38
|
+
],
|
|
39
|
+
timeoutMs: 45_000,
|
|
40
|
+
});
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
### Plan step grammar (`BotCollectionStep`)
|
|
44
|
+
|
|
45
|
+
- `{ id, kind: 'navigate', url, waitUntil?: 'load'|'domcontentloaded'|'networkidle'|'commit' }`
|
|
46
|
+
- `{ id, kind: 'wait', selector?, timeoutMs? }`
|
|
47
|
+
- `{ id, kind: 'click', selector, timeoutMs? }`
|
|
48
|
+
- `{ id, kind: 'scroll', deltaY, selector?, timeoutMs? }`
|
|
49
|
+
- `{ id, kind: 'extract', selector, format: 'text'|'html'|'count', timeoutMs? }`
|
|
50
|
+
- `{ id, kind: 'request', url, method?: 'GET'|'POST', headers?, data? }`
|
|
51
|
+
- Capture: `responses: [{ id, origin?, pathPrefix, optional?: boolean }]`
|
|
52
|
+
|
|
53
|
+
### Fail-closed error contract
|
|
54
|
+
|
|
55
|
+
- `authentication`: Explicit 401/403 (request, navigation, or required response) or final navigation URL matching `/login|signin|sign-in|auth|sso/i`.
|
|
56
|
+
- `empty`: A declared non-optional capture matched no response, or an extract/request step yielded an empty body. Prevents client-side rendered SPA shells from masquerading as a quiet day.
|
|
57
|
+
- `browser`: Network/transport error, timeout, or uncaught DOM exception.
|
|
58
|
+
|
|
59
|
+
### CLI Execution (`kk bot plan`) and Probe Script
|
|
60
|
+
|
|
61
|
+
Run collection plans directly with the CLI:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
# Output JSON result to stdout
|
|
65
|
+
kk bot plan <channel> <plan.json> -m 60
|
|
66
|
+
|
|
67
|
+
# Write individual 0600 artifacts and result.json / error.json into directory
|
|
68
|
+
kk bot plan <channel> <plan.json> -o <output-dir> -m 60
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Low-level debugging can also use `bot-collection-probe.ts`:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
bun packages/publish-harness/scripts/bot-collection-probe.ts \
|
|
75
|
+
--profile <name> \
|
|
76
|
+
--plan <plan.json> \
|
|
77
|
+
--out <output-dir>
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Outputs artifact files with 0600 permissions, `result.json` (metadata without bodies), or `error.json` on failure.
|
|
81
|
+
|
|
82
|
+
## Direct Playwright launcher (persistent context + lease + CDP + stealth)
|
|
83
|
+
|
|
84
|
+
When interactive discovery or custom CDP scripts are needed, always wrap execution with the profile lease:
|
|
9
85
|
|
|
10
86
|
```ts
|
|
11
87
|
import { chromium } from 'playwright';
|
|
88
|
+
import { withProfileLease } from '@gobing-ai/publish-harness';
|
|
12
89
|
|
|
13
|
-
const
|
|
90
|
+
const profileName = process.env.KK_WEB_BOT ?? '<channel>';
|
|
91
|
+
const profileDir = `${process.env.HOME}/.config/kk/browser-profiles/${profileName}`;
|
|
14
92
|
const port = process.env.KK_WEB_CDP ?? '9222';
|
|
15
93
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
94
|
+
await withProfileLease(profileName, async () => {
|
|
95
|
+
const ctx = await chromium.launchPersistentContext(profileDir, {
|
|
96
|
+
headless: process.env.KK_WEB_HEADLESS === 'true', // default headed
|
|
97
|
+
viewport: null,
|
|
98
|
+
args: [
|
|
99
|
+
`--remote-debugging-port=${port}`, // agent-browser / external CDP clients attach here
|
|
100
|
+
'--disable-blink-features=AutomationControlled', // required at OAuth/bot-wall gates
|
|
101
|
+
],
|
|
102
|
+
});
|
|
103
|
+
const page = ctx.pages()[0] ?? (await ctx.newPage());
|
|
104
|
+
console.log(JSON.stringify({ cdpPort: port, profile: profileDir, url: page.url() }));
|
|
105
|
+
process.on('SIGTERM', async () => { await ctx.close(); process.exit(0); });
|
|
106
|
+
await new Promise(() => {}); // stay up for CDP drivers
|
|
23
107
|
});
|
|
24
|
-
const page = ctx.pages()[0] ?? (await ctx.newPage());
|
|
25
|
-
console.log(JSON.stringify({ cdpPort: port, profile: profileDir, url: page.url() }));
|
|
26
|
-
process.on('SIGTERM', async () => { await ctx.close(); process.exit(0); });
|
|
27
|
-
await new Promise(() => {}); // stay up for CDP drivers
|
|
28
108
|
```
|
|
29
109
|
|
|
30
110
|
- The **profile dir is the credential cache** — cookies + storage survive runs;
|
|
31
111
|
nothing else is persisted, no tokens are printed.
|
|
32
112
|
- `agent-browser` (or any CDP client) attaches to the printed port; plain
|
|
33
113
|
Playwright scripts can also import the same process's `page`.
|
|
34
|
-
-
|
|
114
|
+
- **Profile lease required**: `withProfileLease` ensures single-process access.
|
|
115
|
+
Two concurrent launches on one profile corrupt it.
|
|
35
116
|
|
|
36
117
|
## Headless vs headed
|
|
37
118
|
|
|
38
119
|
| Situation | Mode | Why |
|
|
39
120
|
| --- | --- | --- |
|
|
40
|
-
| Session probe / preflight | headless OK | read-only, cheap |
|
|
121
|
+
| Session probe / preflight (`kk bot list`, `check`) | headless OK | read-only, cheap |
|
|
122
|
+
| `kk bot exec` single-request fetch | headless | persistent context attaches cookies automatically |
|
|
123
|
+
| `runBotCollection` automated collection | headless | bounded steps and required response captures |
|
|
41
124
|
| Automated drive on a proven session | headless OK | measured: infoq/juejin/substack/x/wechatmp drives work headless post-login |
|
|
42
|
-
| `
|
|
43
|
-
|
|
|
125
|
+
| `kk bot login` / fresh authentication | **headed** | operator completes QR/OAuth/password by hand |
|
|
126
|
+
| `kk bot open` inspection | **headed** | operator views page and DOM |
|
|
127
|
+
| xhs any publish/drive | **headed** | bot-walls headless loads (measured); shadow-root clicks need CDP anyway |
|
|
44
128
|
| Any step that might trip a bot wall first time | **headed** | watch it once, record, then automate |
|
|
45
129
|
|
|
46
130
|
## Profile matrix (as shipped)
|
|
47
131
|
|
|
48
|
-
| Channel | Profile dir | Notes |
|
|
49
|
-
| --- | --- | --- |
|
|
50
|
-
| xhs | `browser-profiles/xhs` | headed always; shadow-root controls via CDP box models |
|
|
51
|
-
| infoq | `browser-profiles/infoq` | draft save = `/draft/<id>` URL |
|
|
52
|
-
| juejin | `browser-profiles/juejin` | draft id from post-save URL |
|
|
53
|
-
| substack | `browser-profiles/substack` | robinmin.substack.com; post id in `/publish/post/<id>` |
|
|
54
|
-
| x | `browser-profiles/x` | article edit URL carries draft id |
|
|
55
|
-
| wechatmp | `browser-profiles/wechatmp` | QR login; draft id from draft/add response + editor URL |
|
|
132
|
+
| Channel | Profile dir | Login command | Notes |
|
|
133
|
+
| --- | --- | --- | --- |
|
|
134
|
+
| xhs | `browser-profiles/xhs` | `kk bot login xhs` | headed always; shadow-root controls via CDP box models |
|
|
135
|
+
| infoq | `browser-profiles/infoq` | `kk bot login infoq` | draft save = `/draft/<id>` URL |
|
|
136
|
+
| juejin | `browser-profiles/juejin` | `kk bot login juejin` | draft id from post-save URL |
|
|
137
|
+
| substack | `browser-profiles/substack` | `kk bot login substack` | robinmin.substack.com; post id in `/publish/post/<id>` |
|
|
138
|
+
| x | `browser-profiles/x` | `kk bot login x` | article edit URL carries draft id |
|
|
139
|
+
| wechatmp | `browser-profiles/wechatmp` | `kk bot login wechatmp` | QR login; draft id from draft/add response + editor URL |
|
|
140
|
+
| weibo | `browser-profiles/weibo` | `kk bot login weibo` | 头条文章 editor |
|
|
56
141
|
|
|
57
|
-
New channel ⇒ one-time headed `
|
|
142
|
+
New channel ⇒ one-time headed `kk bot login <url> --save <name>` or `kk bot login <channel>`, then it becomes the cache.
|
|
58
143
|
|
|
59
144
|
## Session probe (cached-auth validity)
|
|
60
145
|
|
|
61
|
-
Don't age-guess — probe a signed-in-only selector
|
|
146
|
+
Don't age-guess — probe a signed-in-only selector or run the plugin's probe:
|
|
62
147
|
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
148
|
+
```bash
|
|
149
|
+
# Fast metadata check
|
|
150
|
+
kk bot list
|
|
151
|
+
|
|
152
|
+
# Lightweight authenticated request test
|
|
153
|
+
kk bot exec <name> <console-url>
|
|
67
154
|
|
|
68
|
-
|
|
155
|
+
# Reopen or re-authenticate
|
|
156
|
+
kk bot open <name> --login
|
|
157
|
+
```
|
|
69
158
|
|
|
70
159
|
## storageState vs profile dir
|
|
71
160
|
|
|
@@ -84,3 +173,5 @@ assumed.
|
|
|
84
173
|
timeout (e.g. wechatmp `.weui-desktop-toast` ≤ 15 s) — loud on miss.
|
|
85
174
|
- Always `ctx.close()` on SIGTERM/exit paths; orphaned Chromiums hold the
|
|
86
175
|
profile lock and the next run fails confusingly.
|
|
176
|
+
- In collection plans (`runBotCollection`), differentiate `authentication` (action: run `kk bot login`),
|
|
177
|
+
`empty` (action: check site selector/API path drift), and `browser` (action: investigate transport/timeout).
|
|
@@ -50,12 +50,9 @@ vars:
|
|
|
50
50
|
cover_fallback: ""
|
|
51
51
|
# 0219 R3: image-gen vendor fallback for the XHS render; "" restores strict fail-fast.
|
|
52
52
|
xhs_fallback: "true"
|
|
53
|
-
# 2026-
|
|
54
|
-
#
|
|
55
|
-
|
|
56
|
-
# Flip back to "true" to resume the card plan/render/draft series; `xhs_live` stays a separate
|
|
57
|
-
# launch-time switch and nothing else in the run depends on this target.
|
|
58
|
-
xhs_enabled: "false"
|
|
53
|
+
# 2026-11-09 operator decision: turned on by default for 上传图文 (cards) and 发播客 (episode note);
|
|
54
|
+
# `xhs_live` stays a separate launch-time switch (draft by default) and nothing else in the run depends on this target.
|
|
55
|
+
xhs_enabled: "true"
|
|
59
56
|
xhs_cards: "4"
|
|
60
57
|
xhs_style: "xhs-knowledge"
|
|
61
58
|
xhs_provider: "minimax"
|
|
@@ -391,6 +388,7 @@ nodes:
|
|
|
391
388
|
input: &article-write-input |
|
|
392
389
|
Read ${vars.work_dir}/21-plan/${vars.run_date}_03_plan_plan.json — metadata.docs (Doc[]) is the translated plan; don't edit it.
|
|
393
390
|
Write vivid simplified-Chinese tech news for 咸话咸说之每日AI乱炖 (column 每日AI乱炖, host Robin). No title line. Open with exactly「这里是由Robin为你带来的咸话咸说之每日AI乱炖」。Never「我是咸话咸说」or an AI-production footer. Each ## section: what happened, why it matters, one concrete detail — a human reporter, not a telegram. Inline [来源:<label>](<sourceUri>). Close by tying the items together.
|
|
391
|
+
标题纪律(站点 digest 契约,不满足则 refine 拒绝发布):每个 ## 标题必须以主体起头,第一个逗号/冒号之前的那段不超过 14 个汉字(英文专名与数字各算 1 个字位),且是一个名词短语——如「TSMC 把存储搬进逻辑芯片」「OpenAI 新证明」「Qwen3.5 纯汇编推理」。禁止以动词或「上周/一条/再」这类虚词起头的整句式标题;冒号之后可自由展开。
|
|
394
392
|
选题边界:只写技术内容;某条主线若属政治/外交或贸易管制/军事/公共政策争议/社会公共事件,跳过该条并在文末加一行「> 注:<N> 条条目因选题边界未写入:<id>(<原因>)、<id>(<原因>)」。
|
|
395
393
|
事实纪律:每个事实取自该条计划内容,不虚构、不引他处;数字与主体不得放大;引语只能归属其原句主语;保留「据报道/称/承诺」等限定词;导语/收束不加计划外程度词;材料不足就少写或不写,不凑数。
|
|
396
394
|
Narrative register — smooth transitions, no list-like bulleting, no scaffold labels (「可做角度」「背景」「潜在影响」), no hedging (「读者宜另行核实」「仍待验证」). 「」 quotes.
|