@agentrhq/webcmd 0.1.2 → 0.2.0

package/skills/webcmd-adapter-author/references/jsdom-fixture-pattern.md

@@ -1,196 +1,129 @@
-# JSDOM-against-frozen-fixture pattern (for in-browser DOM extractors)
-
-## When this pattern applies
-
-You're writing an adapter where the data extraction happens **inside the
-live browser** via `page.evaluate(...)` — not in Node-side post-processing.
-Typical signal: the adapter has a function literal stringified into
-`page.evaluate('(' + fn.toString() + ')()')`, walking `document.querySelector`
-and other DOM APIs.
-
-These extractors are invisible to mocked `page.evaluate` unit tests — those
-tests feed pre-baked results to the func, so the real DOM walk never runs.
-PR #1312 found two such silent in-browser bugs in dianping that only
-surfaced on live verify:
-
-1. shop title fallback split on ASCII `[]` while the page renders
-   full-width `【】`, so `name` was always empty.
-2. `headText.replace(/\s+/g, ' ')` collapsed rating "4.8" with reviews
-   "21241条", and a head-wide `/\d+条/` regex captured `4.821241` → 5.
-
-If either category looks plausible for your site, freeze a representative
-HTML snapshot and replay it through JSDOM in a unit test.
-
-## File layout
-
-- Test file: `clis/<site>/<site>.test.js` (alongside the adapter file)
-- Fixture file: `clis/<site>/__fixtures__/<command>.html`
-
-Reference implementation: `clis/dianping/__fixtures__/{shop,search}.html`
-(see PR #1313 for the original test + PR #1318 for the whitespace-strip
-follow-up that this doc grew out of).
-
-## Creating the HTML fixture
-
-The whole point is to commit a **representative snapshot of the live
-page's DOM** — so the JSDOM unit test exercises the real selector paths
-the live extractor walks.
-
-### Mandatory steps (in order)
-
-1. **Capture** the page's HTML from a live verify run:
-   ```bash
-   webcmd browser open https://www.example.com/<page>
-   # In another shell, dump page.content():
-   webcmd browser eval 'document.documentElement.outerHTML' \
-     > /tmp/raw-<command>.html
-   ```
-
-2. **Strip noise blocks** that JSDOM doesn't need and that change every
-   page load (so committed fixtures wouldn't survive a re-capture diff
-   anyway):
-   - All `<script>...</script>` content
-   - All `<style>...</style>` content
-   - All `<iframe>...</iframe>` content
-   - All `<!-- ... -->` HTML comments
-   - All `<link rel="preload" ...>` / tracking pixels
-
-3. **Replace `<img src="...">`** with a placeholder
-   (`src="placeholder.png"`) — real CDN URLs leak account-scoped tokens
-   and are noise.
-
-4. **Trim to the minimum subtree** that exercises the extractor and
-   triggers the bug you're guarding against. For dianping shop, that
-   was `.shop-head` + `.desc-info` + `.review-title`; for search,
-   3 of 15 result `<li>` cards (rank 1, 2, 3).
-
-5. **MANDATORY whitespace normalization step** — strip all
-   whitespace-only lines:
-   ```bash
-   awk 'NF>0' /tmp/raw-<command>.html > clis/<site>/__fixtures__/<command>.html
-   ```
-   JSDOM's HTML parser is whitespace-tolerant; blank lines have zero
-   semantic effect on the test, but they bloat the committed diff and
-   obscure the meaningful DOM subtree from reviewers.
-   **Skipping this step is the most common silent quality regression
-   in fixture creation.** PR #1318 cleaned 239 leftover blank lines
-   from dianping's two fixtures (84.6% / 54.8% of file content) and
-   the JSDOM tests still passed unchanged.
-
-6. **Commit** at `clis/<site>/__fixtures__/<command>.html`.
-
-### Anti-patterns to avoid
-
-- ❌ "Strip script/style content" but leave the surrounding newlines.
-  Removing inline script body without collapsing the now-blank line is
-  the source of the noise — Step 5 exists specifically to clean this up.
-- ❌ Trim to minimum subtree, skip Step 5. The fixture works for the
-  test but reviewers see hundreds of blank lines.
-- ❌ Pretty-print the **mega-line** (e.g. `<div>...</div>` collapsed
-  onto one giant line by the source page). Some bugs depend on
-  text-node adjacency without intervening whitespace (e.g. `4.8` and
-  `21241条` immediately adjacent → `headText` fusion bug). Pretty-print
-  inserts whitespace that masks the very condition you're testing for.
-  Step 5 only deletes empty lines — never re-flow content.
-- ❌ Re-capture from live and overwrite the committed fixture without
-  re-running Steps 2-5. The fixture is a **frozen** snapshot; if the
-  page layout changes, that's a separate decision (update test
-  expectations + re-trim + re-strip).
-
-## Writing the JSDOM unit test
+# JSDOM Fixture Pattern
+
+Use this when a browser adapter extracts DOM inside `page.evaluate` and a mocked-evaluate unit test is too weak to catch silent parser bugs.
+
+## When To Use
+
+Use a JSDOM fixture when:
+
+- DOM parsing is complex.
+- The page mixes repeated labels, counts, or nested cards.
+- A previous bug passed verify but extracted the wrong value.
+- The parser depends on sibling relationships or scoped selectors.
+
+Do not add JSDOM fixtures for simple JSON adapters.
+
+## Fixture Location
+
+Commit intentional review fixtures under:
+
+```text
+clis/<site>/__fixtures__/<command>.html
+```
+
+Temporary debug dumps still belong only in:
+
+```text
+~/.webcmd/sites/<site>/fixtures/
+/tmp/
+```
+
+## Five-Step Workflow
+
+### 1. Capture Minimal HTML
+
+Use the browser to capture the specific DOM region, not the entire page.
+
+```bash
+webcmd browser eval "document.querySelector('<root-selector>')?.outerHTML"
+```
+
+Save only the required HTML for the parser.
+
+### 2. Tighten Blank Lines
+
+Run the mandatory blank-line tightening before committing:
+
+```bash
+awk 'NF>0' clis/<site>/__fixtures__/<command>.html > /tmp/<command>.html
+mv /tmp/<command>.html clis/<site>/__fixtures__/<command>.html
+```
+
+This prevents fixture bloat and makes diffs readable.
+
+### 3. Extract Parser Logic
+
+Move DOM parsing into a helper that can run both in `page.evaluate` and in JSDOM.
+
+Keep helper input explicit:
+
+```js
+export function parseCards(root) {
+  return [...root.querySelectorAll('[data-testid="card"]')].map((card) => ({
+    title: card.querySelector('a')?.textContent?.trim() || '',
+  }));
+}
+```
+
+### 4. Add JSDOM Test
 
 ```js
-import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import fs from 'node:fs';
+import path from 'node:path';
 import { JSDOM } from 'jsdom';
-import { readFileSync } from 'node:fs';
-import { join, dirname } from 'node:path';
-import { fileURLToPath } from 'node:url';
-import { extractShopFields } from './shop.js';
-
-const __dirname = dirname(fileURLToPath(import.meta.url));
-const SHOP_FIXTURE = readFileSync(join(__dirname, '__fixtures__/shop.html'), 'utf8');
-
-describe('shop adapter — extractor against frozen HTML fixture', () => {
-    let originalDocument;
-    let originalLocation;
-
-    beforeEach(() => {
-        originalDocument = globalThis.document;
-        originalLocation = globalThis.location;
-    });
-
-    afterEach(() => {
-        globalThis.document = originalDocument;
-        globalThis.location = originalLocation;
-    });
-
-    function loadFixture(html, url) {
-        const dom = new JSDOM(html, { url });
-        globalThis.document = dom.window.document;
-        globalThis.location = dom.window.location;
-        return dom;
-    }
-
-    it('extracts the canonical fields and avoids known silent bugs', () => {
-        loadFixture(SHOP_FIXTURE, 'https://www.example.com/shop/123');
-
-        const data = extractShopFields();
-
-        expect(data.ok).toBe(true);
-        expect(data.name).toBe('...');
-        // Add explicit regression guards for each known silent bug.
-        expect(data.reviewsRaw).toBe('...');  // not the fused "<rating><reviews>" form
-    });
+import { describe, expect, it } from 'vitest';
+import { parseCards } from './parse.js';
+
+describe('parseCards', () => {
+  it('extracts scoped card values from fixture HTML', () => {
+    const html = fs.readFileSync(path.join(import.meta.dirname, '__fixtures__/search.html'), 'utf8');
+    const dom = new JSDOM(html);
+
+    expect(parseCards(dom.window.document)).toMatchObject([
+      { title: 'Expected first title' },
+    ]);
+  });
 });
 ```
 
-For this to work, the adapter's extractor must be a **top-level
-function** that uses bare `document` / `location` (not `window.document`),
-so the same code is exercised by:
+### 5. Reverse-Validate
 
-- live browser: injected via `${extractFn.toString()}` into
-  `page.evaluate`
-- JSDOM unit test: with `globalThis.document` swapped
+Temporarily break the parser and confirm the test fails for the intended reason. Then restore the correct parser and confirm it passes.
 
-If your adapter currently has the extractor as an IIFE inside a
-template literal, refactor to a top-level `export function` first.
-Reference: `clis/dianping/{shop,search}.js` extracts `extractShopFields()`
-and `extractSearchRows()` with bare `document`/`location`.
+This proves the test is guarding the real bug rather than only exercising code.
 
-## Reverse-validation (mandatory before claiming the test catches the bug)
+## Scoped Selector Rule
 
-A test that "passes 18/18" doesn't prove it would have caught the original
-bug — only that it agrees with the current implementation. Before
-trusting a regression guard:
+Every selector inside a repeated component must be scoped to that component root.
 
-1. Make a backup of the adapter source.
-2. Reintroduce the buggy variant of the relevant extractor.
-3. Run the test. It MUST fail with an assertion that points at the
-   silent bug.
-4. Restore from backup.
+Bad:
 
-For dianping bug #2 (rating/reviews fusion):
 ```js
-// BUGGY VARIANT — replaces the .reviews selector path
-const buggyMatch = headText.match(/(\d+)条/);
-let reviewsRaw = buggyMatch ? buggyMatch[0] : '';
+document.querySelector('[data-testid="like"]')
 ```
 
-If after Step 5 of fixture creation the test still fails on this buggy
-variant with `expected '21241条' to be '21241条'` actually receiving
-`'821241条'` (the fused digits), the regression guard is intact. If the
-test still passes with the buggy variant, the fixture is too stripped /
-normalization went too far / the assertion is too loose — go back and
-tighten.
+Good:
+
+```js
+card.querySelector('[data-testid="like"]')
+```
+
+If a partial fixture represents a reusable component, write its scope root at the top of the fixture or test.
+
+## Common Bugs This Catches
+
+- Page-level first match selects the first card instead of the target card.
+- Head text from adjacent elements fuses numbers into a wrong count.
+- Locale-specific labels disappear or change.
+- Hidden template content is parsed as visible content.
+- A missing selector silently becomes an empty string.
 
-This is the same discipline as `--write-fixture` Step 10 in the main
-runbook (verify fixture catches what it should), applied to the JSDOM
-HTML fixture instead of the response JSON fixture.
+## Review Bar
 
-## See also
+Committed JSDOM fixtures are not temporary dumps. They are review artifacts. Keep them:
 
-- `references/adapter-template.md` — basic adapter file structure
-- `references/output-design.md` — column naming for the post-extract mapping
-- `references/success-rate-pitfalls.md` — broader "verify can pass while
-  data is silently wrong" catalog; the mocked-page.evaluate gap that
-  motivates this whole pattern is one entry there
+- minimal
+- readable
+- free of private data
+- blank-line tightened
+- paired with reverse-validation

package/skills/webcmd-adapter-author/references/output-design.md

@@ -1,149 +1,71 @@
 # Output Design
 
-adapter 的 `columns` 不是随便列。要让下游（用户、其他 adapter 合并、agent 后续分析）都能直接读。
+Design output before writing the final adapter mapping. Good output is stable, scan-friendly, and consistent with neighboring adapters.
 
----
+## Column Order
 
-## 核心约定
+Use this order:
 
-### 1. 命名
+1. Identifier columns: `title`, `name`, `symbol`, `id`, `url`.
+2. Primary business values: score, price, rank, counts, status, dates.
+3. Supporting fields: author, source, category, tags, metadata.
+4. Debug or raw fields only when they are user-relevant.
 
-**camelCase，全英文**：
+## Naming
 
-| 好 | 差 |
-|----|---|
-| `marketCap` | `market_cap` / `市值` / `MarketCap` |
-| `change24hPct` | `change_percentage_24h` / `涨跌幅24h` / `changePct_24h` |
-| `bondCode` | `bond_code` / `BOND_CODE` |
-| `pubTime` | `publish_time` / `pubdate` |
+- Use `camelCase`.
+- Prefer user-facing meaning over source field code.
+- Align with existing adapters for the same concept.
+- Do not include units in names unless needed to avoid ambiguity.
+- Avoid vague names like `value`, `data`, `info`, `num`, or `type` unless the domain makes them precise.
 
-原则：
-- 缩写：`pct`（百分比）/ `pe`（市盈率）/ `pb`（市净率）/ `ytm`（到期收益率）/ `id`
-- 时间后缀：`Time`（具体时刻）/ `Date`（日期）/ `Ts`（unix 秒）
-- 百分比一律 `Pct` 结尾，数值是"已乘 100"形式（`2.5` 表示 2.5%，不是 `0.025`）
-- 数量后缀：`Count`（整数计数）/ `Total`（累计值）
+## Types
 
-### 2. 类型
+Use the narrowest honest type:
 
-| 字段类 | JS 类型 | 格式 |
-|--------|---------|------|
-| 价格 / 金额 | `number` | 原始小数，别除 1000 别取整 |
-| 百分比 | `number` | 已 × 100（`2.5` = 2.5%） |
-| 计数 / rank | `number` | 正整数 |
-| 代码 / id / symbol | `string` | 股票代码 `'600000'` 保留前导 0 |
-| 名称 / 标题 | `string` | 去首尾空白 |
-| 时间 | `string` ISO（`'2024-01-15T10:30:00Z'`）或 `number` unix 秒 | 不要本地字符串 `'2024/1/15'` |
-| 布尔 | `boolean` | 不用 `0/1` |
-| URL | `string` | 绝对路径，相对路径要拼 host |
+| Data | Type |
+| --- | --- |
+| text | `string` |
+| count | `number` |
+| price | `number` plus `currency` when needed |
+| percent | `number` with documented scale |
+| date/time | ISO string |
+| URL | absolute string |
+| optional value | `string|null` or `number|null` only when truly nullable |
 
-**特殊**：
-- 缺失用 `null`，不用 `0` / `''`（0 和空字符串有业务含义，别混）
-- 枚举 → string，不要 int 代号（`'listed'` 比 `0` 清楚）
+Do not hide extraction failure by making every field nullable.
 
-### 3. 顺序
+## Percentages And Units
 
-固定三段：
+State the scale once in comments or notes:
 
-```
-[识别列 ...] [业务数字 ...] [metadata ...]
-```
+- `2.5` means 2.5 percent.
+- `0.025` means ratio form.
 
-**识别列**（前 1-3 列）：`rank / symbol / code / bondCode / name / title / id`
+Choose one scale and keep it consistent.
 
-**业务数字**（中间）：价格、涨跌幅、成交量、市值等业务语义
+For compact units, normalize when possible:
 
-**metadata**（最后 1-3 列）：`pubTime / updateTime / source / url`
+- `1.2K` -> `1200`
+- `3.4M` -> `3400000`
+- currency values should include `currency` if the site can mix currencies
 
-### 4. 必有列（按类型）
+## Empty Results
 
-| adapter 类型 | 必须包含 |
-|-------------|---------|
-| 排行 / 列表 | `rank` + 识别列 + 业务数字 |
-| 时序 / K 线 | `date`（或 `ts`）+ 数值 |
-| 详情（单对象） | 识别列 + 业务字段 |
-| 新闻 / 公告 | `title` + `pubTime` + `url` |
+Empty result is valid only when the site clearly returned no rows for the user's query. Otherwise throw a typed error.
 
-### 5. 控量
+Examples:
 
-**单条 ≤ 15 列**。超了就得考虑：
-- 拆成多个 adapter（列表版 + 详情版）
-- 次要字段合进 `extras: {...}` 对象
-- 只有少数用户关心的字段默认隐藏（靠参数开关）
+- Valid empty: search query has no matches and page says no results.
+- Error: selector changed, auth expired, endpoint returned HTML, or field path failed.
 
----
+## Output Verification
 
-## 对齐邻居 adapter
+Before declaring the adapter ready:
 
-写新 adapter 前先看同站点现有 adapter 怎么命名：
-
-```bash
-grep -h "columns:" clis/<site>/*.js
-```
-
-复用同类列名。比如 `clis/eastmoney/convertible.js` 用 `bondCode / bondName / stockCode / stockName`，新写 eastmoney 某个涉及股票代码的 adapter 就沿用 `stockCode / stockName`，不要发明 `securityId / securityName`。
-
----
-
-## 常见错误
-
-| 错 | 对 |
-|----|---|
-| `columns: ['id', 'name', 'data.price']`（点路径） | 把 `data.price` 在 func 里打平成 `price` |
-| `{date: '2024-01-15 10:30'}`（空格 + 非 ISO） | `'2024-01-15T10:30:00Z'` 或 `Date.toISOString()` |
-| `{pct: '2.5%'}`（字符串 + 单位） | `{changePct: 2.5}` 纯数字 |
-| `{volume: '1.2万'}` | `{volume: 12000}` |
-| `{code: 600000}`（整数丢前导 0） | `{code: '600000'}` string，或 `String(code).padStart(6, '0')` |
-| columns 和 func 返回的 keys 对不上 | 列出的每个 key 必须在返回对象里，顺序也一致 |
-
----
-
-## description 字段
-
-adapter `description` 是用户第一眼看到的，写清楚：
-
-1. **数据是什么**：`"A 股涨幅排行"` vs `"大盘指数分时"`
-2. **默认行为**：`"默认按涨幅排序，前 20 条"`
-3. **重要参数**：`"支持 market 参数切换沪深/北证"`
-
-不要：
-- `"get data"` — 废话
-- `"查询 xxx 数据"` — 也是废话
-- 塞完整 URL / 字段代号列表 — 留给 help
-
-一行 30 字左右够了。
-
----
-
-## args 命名
-
-- `limit` 而不是 `count / num / n / size`
-- `sort` 而不是 `sort_by / order_by / sortKey`
-- `market` 而不是 `exchange / platform / type`
-- `symbol` / `code` / `query` 根据业务选，保持和邻居 adapter 一致
-- 布尔参数用 `enableX / includeX`，默认 false 免得用户改变认知负担
-
-`help` 文案给**所有合法值**，别让用户猜：
-
-```javascript
-{ name: 'sort', type: 'string', default: 'turnover', help: '排序：turnover / change / drop / price / premium' }
-```
-
----
-
-## 示例对比
-
-**差的**：
-
-```javascript
-columns: ['股票代码', 'name', 'PRICE', 'change%', 'vol', 'time']
-```
-
-问题：中英混、大小写乱、百分号字符串、缩写不统一。
-
-**好的**：
-
-```javascript
-columns: ['rank', 'stockCode', 'stockName', 'price', 'changePct', 'volume', 'updateTime']
-```
-
-识别列在前，metadata 在后，命名统一 camelCase。
+- Check `columns` exactly match row object keys, including order.
+- Compare one or more rows against the visible page.
+- Verify numeric units and percentage scale.
+- Verify dates and timezones.
+- Run `webcmd browser verify <site>/<name>`.
+- Add or update fixture assertions for `columns`, `types`, `patterns`, `notEmpty`, and `rowCount`.