amenbo 0.1.4 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +188 -0
- package/README.md +13 -1
- package/dist/cache.d.ts +31 -1
- package/dist/cache.js +67 -7
- package/dist/cache.js.map +1 -1
- package/dist/cli.d.ts +3 -1
- package/dist/cli.js +59 -4
- package/dist/cli.js.map +1 -1
- package/dist/core.d.ts +6 -0
- package/dist/core.js +125 -38
- package/dist/core.js.map +1 -1
- package/dist/diff.d.ts +1 -1
- package/dist/diff.js +29 -9
- package/dist/diff.js.map +1 -1
- package/dist/errors.d.ts +11 -0
- package/dist/errors.js +13 -0
- package/dist/errors.js.map +1 -1
- package/dist/extract/dataSources.js +17 -4
- package/dist/extract/dataSources.js.map +1 -1
- package/dist/extract/outline.js +15 -1
- package/dist/extract/outline.js.map +1 -1
- package/dist/extract/pdf.js +2 -2
- package/dist/extract/pruning.d.ts +4 -2
- package/dist/extract/pruning.js +36 -7
- package/dist/extract/pruning.js.map +1 -1
- package/dist/extract/qualityScore.d.ts +3 -2
- package/dist/extract/qualityScore.js +8 -3
- package/dist/extract/qualityScore.js.map +1 -1
- package/dist/fetcher/browser.js +15 -1
- package/dist/fetcher/browser.js.map +1 -1
- package/dist/fetcher/http.d.ts +12 -1
- package/dist/fetcher/http.js +15 -4
- package/dist/fetcher/http.js.map +1 -1
- package/dist/fetcher/index.d.ts +6 -0
- package/dist/fetcher/index.js +18 -13
- package/dist/fetcher/index.js.map +1 -1
- package/dist/installBrowser.d.ts +11 -0
- package/dist/installBrowser.js +30 -0
- package/dist/installBrowser.js.map +1 -0
- package/dist/links.d.ts +12 -0
- package/dist/links.js +32 -17
- package/dist/links.js.map +1 -1
- package/dist/politeness.d.ts +23 -3
- package/dist/politeness.js +27 -6
- package/dist/politeness.js.map +1 -1
- package/dist/screenshot.d.ts +2 -0
- package/dist/screenshot.js +14 -4
- package/dist/screenshot.js.map +1 -1
- package/dist/server.d.ts +2 -0
- package/dist/server.js +67 -26
- package/dist/server.js.map +1 -1
- package/package.json +3 -6
package/README.en.md
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
# amenbo 🐜💧
|
|
2
|
+
|
|
3
|
+
English | [日本語](./README.md)
|
|
4
|
+
|
|
5
|
+
**Skims the web without making waves** — a Japanese-web-native MCP server for low-impact, token-efficient web collection.
|
|
6
|
+
|
|
7
|
+
amenbo (アメンボ / water strider) is an [MCP](https://modelcontextprotocol.io) server for coding agents such as Claude Code and Codex. Like the insect gliding across water without making ripples, it collects information from the web **with minimal load on target sites and minimal tokens** — optimized especially for **Japanese websites**. In shell environments without an MCP client, it also works as a CLI sharing the same core ([Using as a CLI](#using-as-a-cli)).
|
|
8
|
+
|
|
9
|
+
## Why amenbo
|
|
10
|
+
|
|
11
|
+
Most general-purpose scraping tools assume the English-speaking web. On Japanese sites they tend to miss content in the following ways — amenbo addresses each of them:
|
|
12
|
+
|
|
13
|
+
- **Loosely structured sites**: even on Japanese sites full of nested divs and table layouts, the main content area is inferred from rendered geometry (visual placement)
|
|
14
|
+
- **Mojibake**: automatic detection of Shift_JIS / EUC-JP / ISO-2022-JP
|
|
15
|
+
- **Furigana**: strips `<ruby>` reading annotations to prevent duplicated body text
|
|
16
|
+
- **Information rendered as images**: pages built around image-based pricing tables or banners automatically fall back to screenshots when text extraction is poor
|
|
17
|
+
- **Major Japanese sites**: dedicated adapters for Qiita / Zenn / note / Hatena Blog / Yahoo! News / PR TIMES / Japanese Wikipedia
|
|
18
|
+
|
|
19
|
+
For a measured comparison against similar tools (official fetch MCP / Jina Reader / Playwright MCP / PixelRAG pixelshot), see the article "[エージェントのWeb取得、ツール次第でトークンが5000倍違った話](https://zenn.dev/rererr_engineer/articles/e571e5b6eb1d53)" (Japanese). The benchmark harness and raw logs are in [`bench/`](./bench/).
|
|
20
|
+
|
|
21
|
+
## How it saves tokens
|
|
22
|
+
|
|
23
|
+
- **Progressive disclosure**: `mode: outline` first returns only the heading tree with a token estimate per section; fetch just the sections you need via `section`. Long pages are never dumped wholesale into context
|
|
24
|
+
- **CJK-aware content pruning**: removes nav / ads / footers using punctuation density, character-class ratios, and link density
|
|
25
|
+
- **Diff responses**: when refetching a URL, returns `unchanged` if nothing changed, or only the changed sections
|
|
26
|
+
- **Automatic Markdown/screenshot switching**: only pages with a low quality score become screenshots, avoiding the round trip of reading broken Markdown and refetching
|
|
27
|
+
- **CJK token estimation**: Japanese text costs more tokens per character than English, so page-split budgets are computed with per-character-class coefficients
|
|
28
|
+
|
|
29
|
+
## Low impact on target sites
|
|
30
|
+
|
|
31
|
+
- **Two-tier fetching**: plain HTTP GET first; only pages that need JS rendering escalate to headless Chromium, so most fetches never launch a browser
|
|
32
|
+
- **Polite crawling**: respects robots.txt and Crawl-Delay; serialized access per domain at 1 req/sec by default. Link discovery prefers sitemap / RSS instead of crawling pages
|
|
33
|
+
- **Honest User-Agent**: identifies itself as a bot. **No anti-bot circumvention is implemented**
|
|
34
|
+
- **Caching**: revalidates with ETag / If-Modified-Since to avoid needless refetches
|
|
35
|
+
|
|
36
|
+
## Installation
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
npm install -g amenbo
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
That's enough for plain Markdown fetching (`fetch` / `links`). If you also want browser-based fetching — SPA escalation or screenshots — run this once (a ~170MB download the first time):
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npx -y amenbo install-browser
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Or for development:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
git clone https://github.com/Rererr/amenbo.git
|
|
52
|
+
cd amenbo
|
|
53
|
+
npm install
|
|
54
|
+
npm run build
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Registering with Claude Code
|
|
58
|
+
|
|
59
|
+
Add to `.mcp.json` (or your global config):
|
|
60
|
+
|
|
61
|
+
```json
|
|
62
|
+
{
|
|
63
|
+
"mcpServers": {
|
|
64
|
+
"amenbo": {
|
|
65
|
+
"command": "amenbo"
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
For a local build, use `"command": "node", "args": ["/path/to/amenbo/dist/server.js"]`.
|
|
72
|
+
|
|
73
|
+
### Teaching your agent how to use it (recommended prompt)
|
|
74
|
+
|
|
75
|
+
Tool definitions alone don't convey usage conventions like "fetch with progressive disclosure." Paste the following into `CLAUDE.md` or `AGENTS.md` so your agent uses amenbo efficiently:
|
|
76
|
+
|
|
77
|
+
```markdown
|
|
78
|
+
## Use amenbo for web fetching
|
|
79
|
+
|
|
80
|
+
- Fetch pages with `fetch` (default mode `auto`). For pages that look long or where only part
|
|
81
|
+
is needed, first check headings and per-section token counts with `mode: "outline"`,
|
|
82
|
+
then fetch only the needed sections via `section`
|
|
83
|
+
- Getting `unchanged` / `diff` when refetching the same URL is normal (no change / changed
|
|
84
|
+
sections only). Use `force_full: true` only when the full text is required
|
|
85
|
+
- When looking for pages within a site, don't guess URLs — enumerate them with `links`
|
|
86
|
+
(narrow down with `filter`)
|
|
87
|
+
- In environments with a shell, for long pages you only want to keyword-search or for bulk
|
|
88
|
+
collection of multiple pages, dump via the CLI with `amenbo fetch <url> > page.md` and
|
|
89
|
+
grep / partially read it (keep the body text out of context).
|
|
90
|
+
For pages where you want to judge while seeing the structure, MCP's outline → section flow fits better
|
|
91
|
+
- Works on non-Japanese sites too (progressive disclosure, caching, and low-impact fetching are
|
|
92
|
+
language-independent). However, content extraction is tuned for Japanese, so if body text
|
|
93
|
+
looks missing on a non-Japanese page, retry with a `selector` or `mode: "screenshot"`
|
|
94
|
+
- If visual information is the goal (pricing tables, layout), use `screenshot`.
|
|
95
|
+
`scale: 0.5` or so reduces image tokens
|
|
96
|
+
- Fetch failures due to robots.txt denial or anti-bot measures are by design (no circumvention).
|
|
97
|
+
Report failures to the user as-is
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Using as a CLI
|
|
101
|
+
|
|
102
|
+
`amenbo` also works as a CLI sharing the same core (fetching, cache, politeness, extraction logic) as the MCP server. Running it with no arguments, or as `amenbo serve`, starts the MCP server as before (`"command": "amenbo"` in `.mcp.json` keeps working), so existing MCP registrations are unaffected.
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
# Fetch a page as Markdown (to stdout)
|
|
106
|
+
amenbo fetch https://example.com/
|
|
107
|
+
|
|
108
|
+
# For long pages, check just the headings and token counts first
|
|
109
|
+
amenbo fetch https://example.com/ --mode outline
|
|
110
|
+
|
|
111
|
+
# Dump the output to a file for grep / partial reads (head/sed)
|
|
112
|
+
amenbo fetch https://example.com/ > page.md
|
|
113
|
+
grep -A3 "pricing" page.md
|
|
114
|
+
|
|
115
|
+
# Enumerate links within a site (sitemap/RSS-first)
|
|
116
|
+
amenbo links https://example.com/ --filter "blog/*"
|
|
117
|
+
|
|
118
|
+
# Screenshot (tiled PNGs are saved to --out-dir; their paths are listed on stdout)
|
|
119
|
+
amenbo screenshot https://example.com/ --viewport-only --scale 0.5 --out-dir ./shots
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
See `amenbo <fetch|links|screenshot> --help` for details on each subcommand.
|
|
123
|
+
|
|
124
|
+
**When to use MCP vs. CLI**:
|
|
125
|
+
|
|
126
|
+
- **MCP**: the primary path for agents. The browser (Chromium) stays warm within the process, and images such as screenshots can be returned directly into the conversation. Also reaches agents on hosts without a shell, like claude.ai
|
|
127
|
+
- **CLI**: suited to shell scripts, CI, debugging, dumping output to a file for `grep`/partial reads, or use from MCP-incapable agents/toolchains. One command = one process, so the browser launches each time
|
|
128
|
+
|
|
129
|
+
Cache, diff responses (`unchanged`/`diff`), and rate-control state (robots.txt / per-domain serialized access) are shared between the MCP server and the CLI (both use the same `~/.cache/amenbo`). Cross-process rate control is best-effort, however: per-domain serialization is strictly guaranteed only within each process, so when the MCP server and multiple CLI runs hit the same domain simultaneously, the minimum interval may occasionally be undershot.
|
|
130
|
+
|
|
131
|
+
## Tools
|
|
132
|
+
|
|
133
|
+
### `fetch` — fetch a page
|
|
134
|
+
| Parameter | Description |
|
|
135
|
+
|---|---|
|
|
136
|
+
| `url` | Target URL (http/https only; PDF supported) |
|
|
137
|
+
| `mode` | `auto` (default; switches Markdown/screenshot by quality score) / `markdown` / `outline` (heading summary) / `screenshot` |
|
|
138
|
+
| `selector` | CSS selector to narrow down the body content |
|
|
139
|
+
| `section` | Section ID obtained from outline; returns only that section's Markdown |
|
|
140
|
+
| `page` | Page number (default 1) |
|
|
141
|
+
| `max_tokens` | Approximate token cap per page (default 8000) |
|
|
142
|
+
| `force_full` | true disables diff responses and boilerplate removal, always returning the full text |
|
|
143
|
+
|
|
144
|
+
### `links` — enumerate links
|
|
145
|
+
| Parameter | Description |
|
|
146
|
+
|---|---|
|
|
147
|
+
| `url` | Starting URL |
|
|
148
|
+
| `filter` | Substring match on URL/link text, or a glob using `*` |
|
|
149
|
+
|
|
150
|
+
Discovery order: sitemap → RSS/Atom → in-page links.
|
|
151
|
+
|
|
152
|
+
### `screenshot` — capture a screenshot
|
|
153
|
+
| Parameter | Description |
|
|
154
|
+
|---|---|
|
|
155
|
+
| `url` | Target URL (http/https only) |
|
|
156
|
+
| `fullPage` | Default true. false captures only the first viewport |
|
|
157
|
+
| `width` | Tile width in px (default 1280) |
|
|
158
|
+
| `scale` | Resolution scale 0.5–1.0 (default 1.0). Smaller reduces image tokens |
|
|
159
|
+
|
|
160
|
+
## Environment variables
|
|
161
|
+
|
|
162
|
+
| Variable | Default | Description |
|
|
163
|
+
|---|---|---|
|
|
164
|
+
| `AMENBO_CACHE_DIR` | `~/.cache/amenbo` | Where the cache (SQLite + PNG) is stored |
|
|
165
|
+
| `AMENBO_CACHE_TTL_MS` | `900000` (15 min) | Cache TTL |
|
|
166
|
+
| `AMENBO_MAX_BODY_BYTES` | `20971520` (20MB) | Maximum fetched body size |
|
|
167
|
+
|
|
168
|
+
## Security
|
|
169
|
+
|
|
170
|
+
- **SSRF protection**: rejects non-http/https schemes (`file:`, `ftp:`, etc.). Rejects DNS-resolved destinations that are private / loopback / link-local / reserved addresses. As a DNS rebinding (TOCTOU) countermeasure, actual connections are pinned to the validated IP
|
|
171
|
+
- **Body size cap**: prevents OOM from oversized responses
|
|
172
|
+
|
|
173
|
+
## Known limitations
|
|
174
|
+
|
|
175
|
+
- **No HTTP proxy support**: `HTTP_PROXY` / `HTTPS_PROXY` environment variables are not honored. As an SSRF countermeasure, connections are pinned to a validated IP (DNS rebinding protection), which is incompatible with delegating name resolution to an upstream proxy. Networks that require an upstream proxy are currently unsupported
|
|
176
|
+
- **No anti-bot circumvention**: fetch failures due to robots.txt denial or bot countermeasures are by design and reported as-is (see [Low impact on target sites](#low-impact-on-target-sites))
|
|
177
|
+
|
|
178
|
+
## Development
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
npm run typecheck # strict type checking
|
|
182
|
+
npm test # vitest
|
|
183
|
+
npm run build # build into dist/
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## License
|
|
187
|
+
|
|
188
|
+
[MIT](./LICENSE)
|
package/README.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# amenbo 🐜💧
|
|
2
2
|
|
|
3
|
+
[English](./README.en.md) | 日本語
|
|
4
|
+
|
|
3
5
|
**Skims the web without making waves** — a Japanese-web-native MCP server for low-impact, token-efficient web collection.
|
|
4
6
|
|
|
5
7
|
amenbo(アメンボ / water strider)は、Claude Code や Codex のようなコーディングエージェント向けの [MCP](https://modelcontextprotocol.io) サーバーです。水面に波を立てずに滑る虫のように、**収集先に負荷をかけず、少ないトークンで** Web から情報を集めます。とりわけ**日本語サイト**に最適化しています。MCP クライアントを持たないシェル環境からは、同じコアを共有する CLI としても使えます([CLIとして使う](#cliとして使う)参照)。
|
|
@@ -35,7 +37,12 @@ amenbo(アメンボ / water strider)は、Claude Code や Codex のようなコ
|
|
|
35
37
|
|
|
36
38
|
```bash
|
|
37
39
|
npm install -g amenbo
|
|
38
|
-
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Markdown 取得(通常の `fetch` / `links`)はこれだけで動きます。JS 描画が必要な SPA への昇格やスクリーンショットなど、ブラウザ(Chromium)経由の取得を使う場合のみ、初回に一度だけ実行してください(約 170MB のダウンロード):
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npx -y amenbo install-browser
|
|
39
46
|
```
|
|
40
47
|
|
|
41
48
|
または開発用途:
|
|
@@ -158,6 +165,11 @@ sitemap → RSS/Atom → ページ内リンクの順で探索します。
|
|
|
158
165
|
- **SSRF 対策**:http/https 以外のスキーム(`file:`, `ftp:` 等)を拒否。DNS 解決した接続先が private / loopback / link-local / 予約アドレスなら拒否。DNS rebinding(TOCTOU)対策として実接続を検証済み IP に固定します
|
|
159
166
|
- **ボディサイズ上限**:巨大レスポンスによる OOM を防止
|
|
160
167
|
|
|
168
|
+
## 既知の制限
|
|
169
|
+
|
|
170
|
+
- **HTTP プロキシ非対応**:`HTTP_PROXY` / `HTTPS_PROXY` 等の環境変数は尊重しません。SSRF 対策として接続先を検証済み IP に固定する設計(DNS rebinding 対策)と、プロキシへ名前解決を委ねる方式が両立しないためです。上流プロキシ必須のネットワークでは現状ご利用いただけません
|
|
171
|
+
- **anti-bot 回避は実装しません**:robots.txt 拒否やボット対策による取得失敗は仕様です。失敗はそのまま報告します([収集先への低負荷](#収集先への低負荷)参照)
|
|
172
|
+
|
|
161
173
|
## 開発
|
|
162
174
|
|
|
163
175
|
```bash
|
package/dist/cache.d.ts
CHANGED
|
@@ -17,6 +17,7 @@ export interface CacheWriteInput {
|
|
|
17
17
|
markdown: string;
|
|
18
18
|
metadata: Record<string, unknown>;
|
|
19
19
|
}
|
|
20
|
+
export declare function resolveCacheDir(): string;
|
|
20
21
|
export declare function hashContent(content: string): string;
|
|
21
22
|
/** スクリーンショットのキャッシュキー(url+撮影パラメータの組)を計算する。 */
|
|
22
23
|
export declare function computeScreenshotCacheKey(url: string, width: number, scale: number, fullPage: boolean): string;
|
|
@@ -38,6 +39,7 @@ export declare class PageCache {
|
|
|
38
39
|
private readonly cacheDir;
|
|
39
40
|
private readonly ttlMs;
|
|
40
41
|
private readonly now;
|
|
42
|
+
private closed;
|
|
41
43
|
constructor(options?: {
|
|
42
44
|
dbPath?: string;
|
|
43
45
|
cacheDir?: string;
|
|
@@ -57,7 +59,16 @@ export declare class PageCache {
|
|
|
57
59
|
}): boolean;
|
|
58
60
|
/** 保存済みスクリーンショットエントリを取得する(TTL判定は isFresh を使う)。 */
|
|
59
61
|
getScreenshot(cacheKey: string): ScreenshotCacheEntry | undefined;
|
|
60
|
-
/**
|
|
62
|
+
/**
|
|
63
|
+
* スクリーンショットのタイル群をファイル保存し、パスをSQLiteへ記録する。
|
|
64
|
+
*
|
|
65
|
+
* レビュー指摘対応(Medium): 同一cacheKeyへの再撮影でタイル数が減った場合(旧5枚→新3枚)、
|
|
66
|
+
* 新しいtile_pathsで上書きされるDB行からは旧tile-3.png/tile-4.pngを参照できなくなり、
|
|
67
|
+
* pruneExpiredもそれらを認識できず孤立ファイルとしてディスクに残り続けていた。
|
|
68
|
+
* tileDirはcacheKey単位のディレクトリ(他のcacheKeyのタイルと混在しない)のため、
|
|
69
|
+
* 書き込み前にディレクトリごと削除してから作り直すことで、枚数が変化しても
|
|
70
|
+
* 旧タイルを確実に一掃する(初回撮影時はディレクトリが存在しないためforce:trueで無視する)。
|
|
71
|
+
*/
|
|
61
72
|
setScreenshot(input: ScreenshotWriteInput): ScreenshotCacheEntry;
|
|
62
73
|
/** 新規取得結果を保存(既存があれば上書き)する。 */
|
|
63
74
|
set(input: CacheWriteInput): CacheEntry;
|
|
@@ -86,5 +97,24 @@ export declare class PageCache {
|
|
|
86
97
|
* store JSDoc参照)。せめて時刻が巻き戻らないよう、SQL側でMAXを取って単調増加を保証する。
|
|
87
98
|
*/
|
|
88
99
|
setHostLastRequestAt(host: string, at: number): void;
|
|
100
|
+
/**
|
|
101
|
+
* レビュー指摘対応: オリジン(例 "https://example.com")のrobots.txtキャッシュを返す
|
|
102
|
+
* (未記録ならnull)。PolitenessManagerのrobotsStoreオプションとして注入し、プロセス間で
|
|
103
|
+
* robots.txt取得結果を共有するために使う。TTL判定(robotsTtlMs)は呼び出し元(PolitenessManager)
|
|
104
|
+
* がfetchedAtを見て行う(page cacheのttlMsとは別の時間軸のため、ここでは判定しない)。
|
|
105
|
+
* 空bodyのrobots.txt(=制限なし)も正しい取得結果として保存されるため、
|
|
106
|
+
* 「未記録(null)」と「空bodyの記録あり」は区別される。
|
|
107
|
+
*/
|
|
108
|
+
getRobotsCache(origin: string): {
|
|
109
|
+
body: string;
|
|
110
|
+
fetchedAt: number;
|
|
111
|
+
} | null;
|
|
112
|
+
/** オリジンのrobots.txt取得結果(本文+取得時刻)を記録する。既存があれば上書きする。 */
|
|
113
|
+
setRobotsCache(origin: string, body: string, fetchedAt: number): void;
|
|
114
|
+
/**
|
|
115
|
+
* 冪等なclose。better-sqlite3のclose()は閉鎖済みでも安全だったが、node:sqliteは
|
|
116
|
+
* "database is not open" を投げるため、CLIのfinallyとprocess 'exit'ハンドラの両方から
|
|
117
|
+
* 呼ばれる二重close経路(core.ts参照)を自前のフラグで吸収する。
|
|
118
|
+
*/
|
|
89
119
|
close(): void;
|
|
90
120
|
}
|
package/dist/cache.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* cache.ts —
|
|
2
|
+
* cache.ts — node:sqlite(DatabaseSync)によるURL単位のローカルキャッシュ。
|
|
3
3
|
*
|
|
4
4
|
* 保存場所: $AMENBO_CACHE_DIR または ~/.cache/amenbo/cache.sqlite
|
|
5
5
|
* URL毎にETag/Last-Modified/本文ハッシュ/変換済みMarkdown/取得時刻を保存する。
|
|
@@ -21,17 +21,26 @@
|
|
|
21
21
|
* 短時間に連続実行すると同一ドメインへの最小間隔が守られない。host_requests テーブルへ
|
|
22
22
|
* 最終リクエスト時刻を永続化し、PolitenessManagerのstoreオプションとして注入することで
|
|
23
23
|
* MCPサーバー/CLIの複数プロセス間で共有する(再起動直後の連続アクセスも守れる副次効果あり)。
|
|
24
|
-
*
|
|
24
|
+
* 同じ理由でrobots.txtの取得結果もrobots_cache テーブルへ永続化し、PolitenessManagerの
|
|
25
|
+
* robotsStoreオプションとして注入する(CLIでの複数ページ一括収集時にコマンド毎の
|
|
26
|
+
* robots.txt再取得を防ぐ)。journal_mode=WAL は複数プロセスからの同時アクセスを想定して設定している。
|
|
27
|
+
*
|
|
28
|
+
* node:sqliteの制約: better-sqlite3はNumber.MAX_SAFE_INTEGERを超えるINTEGER列を読むと
|
|
29
|
+
* 例外を投げたが、node:sqliteは黙ってbigintを返す(上流の挙動差)。本ファイルの整数列
|
|
30
|
+
* (fetched_at/last_request_at)は全てDate.now()由来でMAX_SAFE_INTEGERを超えないため、
|
|
31
|
+
* number型へのキャストのみで安全に扱える。
|
|
25
32
|
*/
|
|
26
33
|
import { createHash } from "node:crypto";
|
|
27
34
|
import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
|
|
28
35
|
import { homedir } from "node:os";
|
|
29
36
|
import { join } from "node:path";
|
|
30
|
-
import
|
|
37
|
+
import { DatabaseSync } from "node:sqlite";
|
|
31
38
|
const DEFAULT_TTL_MS = 15 * 60 * 1000; // 15分
|
|
32
39
|
/** Phase 4テンプレート学習: 定型ブロック判定に使う「直近ページ数」の既定値。 */
|
|
33
40
|
const DEFAULT_TEMPLATE_RECENT_PAGES = 3;
|
|
34
|
-
|
|
41
|
+
// レビュー指摘対応: cli.ts(fetch画像の保存先)がキャッシュディレクトリ解決ロジックを
|
|
42
|
+
// 再利用できるようexportする(AMENBO_CACHE_DIR未指定時の既定値~/.cache/amenboとのずれを防ぐ)。
|
|
43
|
+
export function resolveCacheDir() {
|
|
35
44
|
const dir = process.env.AMENBO_CACHE_DIR ?? join(homedir(), ".cache", "amenbo");
|
|
36
45
|
if (!existsSync(dir)) {
|
|
37
46
|
mkdirSync(dir, { recursive: true });
|
|
@@ -50,15 +59,16 @@ export class PageCache {
|
|
|
50
59
|
cacheDir;
|
|
51
60
|
ttlMs;
|
|
52
61
|
now;
|
|
62
|
+
closed = false;
|
|
53
63
|
constructor(options = {}) {
|
|
54
64
|
this.cacheDir = options.cacheDir ?? resolveCacheDir();
|
|
55
65
|
if (!existsSync(this.cacheDir)) {
|
|
56
66
|
mkdirSync(this.cacheDir, { recursive: true });
|
|
57
67
|
}
|
|
58
|
-
this.db = new
|
|
68
|
+
this.db = new DatabaseSync(options.dbPath ?? join(this.cacheDir, "cache.sqlite"));
|
|
59
69
|
this.ttlMs = options.ttlMs ?? DEFAULT_TTL_MS;
|
|
60
70
|
this.now = options.now ?? Date.now;
|
|
61
|
-
this.db.
|
|
71
|
+
this.db.exec("PRAGMA journal_mode = WAL");
|
|
62
72
|
this.db.exec(`
|
|
63
73
|
CREATE TABLE IF NOT EXISTS pages (
|
|
64
74
|
url TEXT PRIMARY KEY,
|
|
@@ -94,6 +104,16 @@ export class PageCache {
|
|
|
94
104
|
host TEXT PRIMARY KEY,
|
|
95
105
|
last_request_at INTEGER NOT NULL
|
|
96
106
|
)
|
|
107
|
+
`);
|
|
108
|
+
// レビュー指摘対応: robots.txtの取得結果をhost_requestsと同じ考え方でプロセス間共有する。
|
|
109
|
+
// CLIバルク収集(1コマンド=1プロセス)で、コマンド毎にrobots.txtを再取得してしまう
|
|
110
|
+
// 低負荷原則違反への対応(PolitenessManagerのrobotsStoreオプションとして注入する)。
|
|
111
|
+
this.db.exec(`
|
|
112
|
+
CREATE TABLE IF NOT EXISTS robots_cache (
|
|
113
|
+
origin TEXT PRIMARY KEY,
|
|
114
|
+
body TEXT NOT NULL,
|
|
115
|
+
fetched_at INTEGER NOT NULL
|
|
116
|
+
)
|
|
97
117
|
`);
|
|
98
118
|
// M3: TTL超過削除・サイズ上限が無くディスクが無制限に増加する問題への対応。
|
|
99
119
|
// 起動(コンストラクタ)のたびにTTL超過エントリを掃除する。
|
|
@@ -156,9 +176,19 @@ export class PageCache {
|
|
|
156
176
|
fetchedAt: row.fetched_at,
|
|
157
177
|
};
|
|
158
178
|
}
|
|
159
|
-
/**
|
|
179
|
+
/**
|
|
180
|
+
* スクリーンショットのタイル群をファイル保存し、パスをSQLiteへ記録する。
|
|
181
|
+
*
|
|
182
|
+
* レビュー指摘対応(Medium): 同一cacheKeyへの再撮影でタイル数が減った場合(旧5枚→新3枚)、
|
|
183
|
+
* 新しいtile_pathsで上書きされるDB行からは旧tile-3.png/tile-4.pngを参照できなくなり、
|
|
184
|
+
* pruneExpiredもそれらを認識できず孤立ファイルとしてディスクに残り続けていた。
|
|
185
|
+
* tileDirはcacheKey単位のディレクトリ(他のcacheKeyのタイルと混在しない)のため、
|
|
186
|
+
* 書き込み前にディレクトリごと削除してから作り直すことで、枚数が変化しても
|
|
187
|
+
* 旧タイルを確実に一掃する(初回撮影時はディレクトリが存在しないためforce:trueで無視する)。
|
|
188
|
+
*/
|
|
160
189
|
setScreenshot(input) {
|
|
161
190
|
const tileDir = join(this.cacheDir, "screenshots", input.cacheKey);
|
|
191
|
+
rmSync(tileDir, { recursive: true, force: true });
|
|
162
192
|
mkdirSync(tileDir, { recursive: true });
|
|
163
193
|
const tilePaths = input.tiles.map((buffer, index) => {
|
|
164
194
|
const filePath = join(tileDir, `tile-${index}.png`);
|
|
@@ -279,7 +309,37 @@ export class PageCache {
|
|
|
279
309
|
ON CONFLICT(host) DO UPDATE SET last_request_at = MAX(last_request_at, excluded.last_request_at)`)
|
|
280
310
|
.run({ host, lastRequestAt: at });
|
|
281
311
|
}
|
|
312
|
+
/**
|
|
313
|
+
* レビュー指摘対応: オリジン(例 "https://example.com")のrobots.txtキャッシュを返す
|
|
314
|
+
* (未記録ならnull)。PolitenessManagerのrobotsStoreオプションとして注入し、プロセス間で
|
|
315
|
+
* robots.txt取得結果を共有するために使う。TTL判定(robotsTtlMs)は呼び出し元(PolitenessManager)
|
|
316
|
+
* がfetchedAtを見て行う(page cacheのttlMsとは別の時間軸のため、ここでは判定しない)。
|
|
317
|
+
* 空bodyのrobots.txt(=制限なし)も正しい取得結果として保存されるため、
|
|
318
|
+
* 「未記録(null)」と「空bodyの記録あり」は区別される。
|
|
319
|
+
*/
|
|
320
|
+
getRobotsCache(origin) {
|
|
321
|
+
const row = this.db.prepare("SELECT * FROM robots_cache WHERE origin = ?").get(origin);
|
|
322
|
+
return row ? { body: row.body, fetchedAt: row.fetched_at } : null;
|
|
323
|
+
}
|
|
324
|
+
/** オリジンのrobots.txt取得結果(本文+取得時刻)を記録する。既存があれば上書きする。 */
|
|
325
|
+
setRobotsCache(origin, body, fetchedAt) {
|
|
326
|
+
this.db
|
|
327
|
+
.prepare(`INSERT INTO robots_cache (origin, body, fetched_at)
|
|
328
|
+
VALUES (@origin, @body, @fetchedAt)
|
|
329
|
+
ON CONFLICT(origin) DO UPDATE SET
|
|
330
|
+
body = excluded.body,
|
|
331
|
+
fetched_at = excluded.fetched_at`)
|
|
332
|
+
.run({ origin, body, fetchedAt });
|
|
333
|
+
}
|
|
334
|
+
/**
|
|
335
|
+
* 冪等なclose。better-sqlite3のclose()は閉鎖済みでも安全だったが、node:sqliteは
|
|
336
|
+
* "database is not open" を投げるため、CLIのfinallyとprocess 'exit'ハンドラの両方から
|
|
337
|
+
* 呼ばれる二重close経路(core.ts参照)を自前のフラグで吸収する。
|
|
338
|
+
*/
|
|
282
339
|
close() {
|
|
340
|
+
if (this.closed)
|
|
341
|
+
return;
|
|
342
|
+
this.closed = true;
|
|
283
343
|
this.db.close();
|
|
284
344
|
}
|
|
285
345
|
}
|
package/dist/cache.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cache.js","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"cache.js","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA+BG;AACH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAwB3C,MAAM,cAAc,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,CAAC,MAAM;AAC7C,gDAAgD;AAChD,MAAM,6BAA6B,GAAG,CAAC,CAAC;AAExC,mDAAmD;AACnD,oEAAoE;AACpE,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,gBAAgB,IAAI,IAAI,CAAC,OAAO,EAAE,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAC;IAChF,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACrB,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACtC,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAsCD,MAAM,UAAU,WAAW,CAAC,OAAe;IACzC,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AAC5D,CAAC;AAED,6CAA6C;AAC7C,MAAM,UAAU,yBAAyB,CAAC,GAAW,EAAE,KAAa,EAAE,KAAa,EAAE,QAAiB;IACpG,OAAO,WAAW,CAAC,GAAG,GAAG,MAAM,KAAK,MAAM,KAAK,SAAS,QAAQ,EAAE,CAAC,CAAC;AACtE,CAAC;AAiBD,MAAM,OAAO,SAAS;IACH,EAAE,CAAe;IACjB,QAAQ,CAAS;IACjB,KAAK,CAAS;IACd,GAAG,CAAe;IAC3B,MAAM,GAAG,KAAK,CAAC;IAEvB,YAAY,OAAO,GAA+E,EAAE;QAClG,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,eAAe,EAAE,CAAC;QACtD,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/B,SAAS,CAAC,IAAI,CAAC,QAAQ,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAChD,CAAC;QACD,IAAI,CAAC,EAAE,GAAG,IAAI,YAAY,CAAC,OAAO,CAAC,MAAM,IAAI,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,cAAc,CAAC,CAAC,CAAC;QAClF,IAAI,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,IAAI,cAAc,CAAC;QAC7C,IAAI,CAAC,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC;QACnC,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC;QAC1C,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;KAUZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;KAQZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;KAQZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,yGAAyG,CAAC,CAAC;QACxH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;KAKZ,CAAC,CAAC;QACH,2DAA2D;QAC3D,mDAAmD;QACnD,0DAA0D;QAC1D,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;KAMZ,CAAC,CAAC;QAEH,2CAA2C;QAC3C,iCAAiC;QACjC,IAAI,CAAC,YAAY,EAAE,CAAC;IACtB,CAAC;IAED;;;OAGG;IACK,YAAY;QAClB,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC;QAEvC,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,wCAAwC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACtE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAC7E,2DAA2D;QAC3D,sDAAsD;QACtD,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,qDAAqD,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAEnF,MAAM,kBAAkB,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,gDAAgD,CAAC,CAAC,GAAG,CAAC,MAAM,CAA+B,CAAC;QACvI,KAAK,MAAM,GAAG,IAAI,kBAAkB,EAAE,CAAC;YACrC,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,CAAa,CAAC;YACzD,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE,CAAC;gBACjC,IAAI,CAAC;oBACH,MAAM,CAAC,QAAQ,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;gBACpC,CAAC;gBAAC,MAAM,CAAC;oBACP,oCAAoC;gBACtC,CAAC;YACH,CAAC;QACH,CAAC;QACD,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,8CAA8C,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;IAC9E,CAAC;IAED,iDAAiD;IACjD,GAAG,CAAC,GAAW;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,mCAAmC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAwB,CAAC;QACjG,IAAI,CAAC,GAAG;YAAE,OAAO,SAAS,CAAC;QAC3B,OAAO;YACL,GAAG,EAAE,GAAG,CAAC,GAAG;YACZ,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,YAAY,EAAE,GAAG,CAAC,aAAa;YAC/B,WAAW,EAAE,GAAG,CAAC,YAAY;YAC7B,QAAQ,EAAE,GAAG,CAAC,QAAQ;YACtB,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAA4B;YAC7D,SAAS,EAAE,GAAG,CAAC,UAAU;SAC1B,CAAC;IACJ,CAAC;IAED,qDAAqD;IACrD,OAAO,CAAC,KAA4B;QAClC,OAAO,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC;IACnD,CAAC;IAED,kDAAkD;IAClD,aAAa,CAAC,QAAgB;QAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC,CAAC,GAAG,CAAC,QAAQ,CAA8B,CAAC;QACxH,IAAI,CAAC,GAAG;YAAE,OAAO,SAAS,CAAC;QAC3B,OAAO;YACL,QAAQ,EAAE,GAAG,CAAC,SAAS;YACvB,GAAG,EAAE,GAAG,CAAC,GAAG;YACZ,SAAS,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,CAAa;YACjD,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAA4B;YAC7D,SAAS,EAAE,GAAG,CAAC,UAAU;SAC1B,CAAC;IACJ,CAAC;IAED;;;;;;;;;OASG;IACH,aAAa,CAAC,KAA2B;QACvC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,aAAa,EAAE,KAAK,CAAC,QAAQ,CAAC,CAAC;QACnE,MAAM,CAAC,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;QAClD,SAAS,CAAC,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QACxC,MAAM,SAAS,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE;YAClD,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,EAAE,QAAQ,KAAK,MAAM,CAAC,CAAC;YACpD,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAC;YAChC,OAAO,QAAQ,CAAC;QAClB,CAAC,CAAC,CAAC;QAEH,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,IAAI,CAAC,EAAE;aACJ,OAAO,CACN;;;;;;4CAMoC,CACrC;aACA,GAAG,CAAC;YACH,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC;YACpC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,CAAC;YACxC,SAAS;SACV,CAAC,CAAC;QAEL,OAAO,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE,SAAS,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,SAAS,EAAE,CAAC;IACtG,CAAC;IAED,8BAA8B;IAC9B,GAAG,CAAC,KAAsB;QACxB,MAAM,WAAW,GAAG,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;QAChD,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,IAAI,CAAC,EAAE;aACJ,OAAO,CACN;;;;;;;;4CAQoC,CACrC;aACA,GAAG,CAAC;YACH,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,YAAY,EAAE,KAAK,CAAC,YAAY;YAChC,WAAW;YACX,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,CAAC;YACxC,SAAS;SACV,CAAC,CAAC;QACL,OAAO;YACL,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,YAAY,EAAE,KAAK,CAAC,YAAY;YAChC,WAAW;YACX,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,SAAS;SACV,CAAC;IACJ,CAAC;IAED,2CAA2C;IAC3C,KAAK,CAAC,GAAW;QACf,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,GAAG,CAAC,CAAC;IACxF,CAAC;IAED,2DAA2D;IAC3D,sBAAsB,CAAC,MAAc,EAAE,GAAW,EAAE,WAAqB;QACvE,IAAI,CAAC,EAAE;aACJ,OAAO,CACN;;;;gDAIwC,CACzC;aACA,GAAG,CAAC,EAAE,MAAM,EAAE,GAAG,EAAE,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,EAAE,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,EAAE,CAAC,CAAC;IAC3F,CAAC;IAED;;;;OAIG;IACH,sBAAsB,CAAC,MAAc,EAAE,WAAW,GAAW,6BAA6B;QACxF,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE;aACjB,OAAO,CAAC,8EAA8E,CAAC;aACvF,GAAG,CAAC,MAAM,EAAE,WAAW,CAA+B,CAAC;QAE1D,IAAI,IAAI,CAAC,MAAM,GAAG,WAAW;YAAE,OAAO,IAAI,GAAG,EAAE,CAAC;QAEhD,MAAM,eAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAClD,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;YACvB,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,YAAY,CAAa,CAAC;YACxD,KAAK,MAAM,IAAI,IAAI,MAAM,EAAE,CAAC;gBAC1B,eAAe,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,eAAe,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YAClE,CAAC;QACH,CAAC;QAED,MAAM,cAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACzC,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,eAAe,EAAE,CAAC;YAC5C,IAAI,KAAK,IAAI,WAAW;gBAAE,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACrD,CAAC;QACD,OAAO,cAAc,CAAC;IACxB,CAAC;IAED;;;OAGG;IACH,oBAAoB,CAAC,IAAY;QAC/B,MAAM,GAAG,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,4CAA4C,CAAC,CAAC,GAAG,CAAC,IAAI,CAA+B,CAAC;QAClH,OAAO,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,IAAI,CAAC;IAC1C,CAAC;IAED;;;;;;;;OAQG;IACH,oBAAoB,CAAC,IAAY,EAAE,EAAU;QAC3C,IAAI,CAAC,EAAE;aACJ,OAAO,CACN;;0GAEkG,CACnG;aACA,GAAG,CAAC,EAAE,IAAI,EAAE,aAAa,EAAE,EAAE,EAAE,CAAC,CAAC;IACtC,CAAC;IAED;;;;;;;OAOG;IACH,cAAc,CAAC,MAAc;QAC3B,MAAM,GAAG,GAAG,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC,CAAC,GAAG,CAAC,MAAM,CAA+B,CAAC;QACrH,OAAO,GAAG,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,GAAG,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IACpE,CAAC;IAED,qDAAqD;IACrD,cAAc,CAAC,MAAc,EAAE,IAAY,EAAE,SAAiB;QAC5D,IAAI,CAAC,EAAE;aACJ,OAAO,CACN;;;;4CAIoC,CACrC;aACA,GAAG,CAAC,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC,CAAC;IACtC,CAAC;IAED;;;;OAIG;IACH,KAAK;QACH,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;IAClB,CAAC;CACF"}
|
package/dist/cli.d.ts
CHANGED
|
@@ -11,7 +11,9 @@ export type ParsedCommand = {
|
|
|
11
11
|
kind: "version";
|
|
12
12
|
} | {
|
|
13
13
|
kind: "help";
|
|
14
|
-
topic?: "fetch" | "links" | "screenshot";
|
|
14
|
+
topic?: "fetch" | "links" | "screenshot" | "install-browser";
|
|
15
|
+
} | {
|
|
16
|
+
kind: "install-browser";
|
|
15
17
|
} | ParsedFetchCommand | ParsedLinksCommand | ParsedScreenshotCommand;
|
|
16
18
|
export interface ParsedFetchCommand {
|
|
17
19
|
kind: "fetch";
|
package/dist/cli.js
CHANGED
|
@@ -8,6 +8,8 @@
|
|
|
8
8
|
* - `amenbo fetch/links/screenshot <url> [options]` は対応するcore.tsのハンドラを呼び、
|
|
9
9
|
* TextBlockはそのまま標準出力へ、ImageBlock(スクリーンショット/スキャンPDF画像)は
|
|
10
10
|
* --out-dir配下へファイル保存しパスを列挙する(エージェント/ユーザーがそのまま開けるように)。
|
|
11
|
+
* - `amenbo install-browser` はChromium遅延化(postinstall廃止)対応で追加したコマンド。
|
|
12
|
+
* core.tsは経由せず、installBrowser.tsが同梱playwrightのCLIをspawnする。
|
|
11
13
|
* - 引数パース(parseCliArgs)は純関数として切り出し、ユニットテストできるようにしている。
|
|
12
14
|
* - キャッシュ・差分応答・レート制御の状態はMCPサーバーと同じ ~/.cache/amenbo (cache.ts)を
|
|
13
15
|
* 共有する。CLIは1コマンド=1プロセスのため、politenessのドメイン毎レート制御は
|
|
@@ -17,10 +19,13 @@ import { mkdirSync, realpathSync, writeFileSync } from "node:fs";
|
|
|
17
19
|
import { join } from "node:path";
|
|
18
20
|
import { parseArgs } from "node:util";
|
|
19
21
|
import { fileURLToPath } from "node:url";
|
|
22
|
+
import { resolveCacheDir } from "./cache.js";
|
|
20
23
|
import { cache, formatLinksResponse, handleFetchTool, handleScreenshotTool, politeness, resolvePackageVersion, } from "./core.js";
|
|
21
24
|
import { AmenboError } from "./errors.js";
|
|
22
25
|
import { closeBrowser } from "./fetcher/browser.js";
|
|
26
|
+
import { installBrowser } from "./installBrowser.js";
|
|
23
27
|
import { discoverLinks } from "./links.js";
|
|
28
|
+
import { MAX_SCALE, MIN_SCALE } from "./screenshot.js";
|
|
24
29
|
import { runServer } from "./server.js";
|
|
25
30
|
// ---- 引数パース(純関数。ユニットテスト対象) ----
|
|
26
31
|
/** 引数の使い方が不正な場合に投げる(cli.ts側でusageを表示しexit code 2にする)。 */
|
|
@@ -41,6 +46,18 @@ function parsePositiveFloat(raw, flag) {
|
|
|
41
46
|
}
|
|
42
47
|
return n;
|
|
43
48
|
}
|
|
49
|
+
/**
|
|
50
|
+
* レビュー指摘対応(Low): --scaleはparsePositiveFloatを通すだけで[0.5,1.0]範囲を検証しておらず、
|
|
51
|
+
* `--scale 3.0`のような範囲外値がscreenshot.ts clampScaleで黙って1.0にクランプされていた。
|
|
52
|
+
* MCP側はzod .min(0.5).max(1.0)で明示的にエラーになるため、CLIも同じ強度の検証で揃える。
|
|
53
|
+
*/
|
|
54
|
+
function parseScreenshotScale(raw) {
|
|
55
|
+
const n = parsePositiveFloat(raw, "--scale");
|
|
56
|
+
if (n < MIN_SCALE || n > MAX_SCALE) {
|
|
57
|
+
throw new CliUsageError(`--scaleは${MIN_SCALE}〜${MAX_SCALE}の範囲で指定してください: ${raw}`);
|
|
58
|
+
}
|
|
59
|
+
return n;
|
|
60
|
+
}
|
|
44
61
|
function requireUrl(positionals, command) {
|
|
45
62
|
const url = positionals[0];
|
|
46
63
|
if (!url) {
|
|
@@ -122,10 +139,19 @@ function parseScreenshotArgs(args) {
|
|
|
122
139
|
url,
|
|
123
140
|
...(values["viewport-only"] ? { viewportOnly: true } : {}),
|
|
124
141
|
...(values.width !== undefined ? { width: parsePositiveInt(values.width, "--width") } : {}),
|
|
125
|
-
...(values.scale !== undefined ? { scale:
|
|
142
|
+
...(values.scale !== undefined ? { scale: parseScreenshotScale(values.scale) } : {}),
|
|
126
143
|
...(values["out-dir"] !== undefined ? { outDir: values["out-dir"] } : {}),
|
|
127
144
|
};
|
|
128
145
|
}
|
|
146
|
+
function parseInstallBrowserArgs(args) {
|
|
147
|
+
const { values, positionals } = safeParseArgs(args, "install-browser", { help: { type: "boolean", short: "h" } });
|
|
148
|
+
if (values.help)
|
|
149
|
+
return { kind: "help", topic: "install-browser" };
|
|
150
|
+
if (positionals.length > 0) {
|
|
151
|
+
throw new CliUsageError(`install-browser: 不明な引数です: ${positionals[0]}`);
|
|
152
|
+
}
|
|
153
|
+
return { kind: "install-browser" };
|
|
154
|
+
}
|
|
129
155
|
/** CLI引数(process.argv.slice(2)相当)をコマンドへパースする純関数。 */
|
|
130
156
|
export function parseCliArgs(argv) {
|
|
131
157
|
if (argv.length === 0)
|
|
@@ -143,6 +169,8 @@ export function parseCliArgs(argv) {
|
|
|
143
169
|
return parseLinksArgs(rest);
|
|
144
170
|
if (first === "screenshot")
|
|
145
171
|
return parseScreenshotArgs(rest);
|
|
172
|
+
if (first === "install-browser")
|
|
173
|
+
return parseInstallBrowserArgs(rest);
|
|
146
174
|
throw new CliUsageError(`不明なコマンドです: ${first}`);
|
|
147
175
|
}
|
|
148
176
|
// ---- ヘルプ/usage文言 ----
|
|
@@ -155,6 +183,7 @@ function usageText() {
|
|
|
155
183
|
" fetch <url> [options] ページをMarkdownとして取得する",
|
|
156
184
|
" links <url> [options] ページ内のリンクを列挙する(sitemap/RSS優先)",
|
|
157
185
|
" screenshot <url> [options] ページのスクリーンショットを撮影する",
|
|
186
|
+
" install-browser スクリーンショット/SPA取得に必要なChromiumをインストールする(初回のみ)",
|
|
158
187
|
"",
|
|
159
188
|
"Options:",
|
|
160
189
|
" -h, --help ヘルプを表示する",
|
|
@@ -174,7 +203,7 @@ function fetchHelpText() {
|
|
|
174
203
|
" --page <n> ページ番号(既定 1)",
|
|
175
204
|
" --max-tokens <n> 1ページの概算トークン上限(既定 8000)",
|
|
176
205
|
" --force-full 差分応答(unchanged/diff)と定型ブロック除去を無効化し常に全文を返す",
|
|
177
|
-
" --out-dir <dir> screenshot切替時の画像保存先(既定
|
|
206
|
+
" --out-dir <dir> screenshot切替時の画像保存先(既定 $AMENBO_CACHE_DIR/screenshots-cli/、未設定時は ~/.cache/amenbo/screenshots-cli/)",
|
|
178
207
|
].join("\n");
|
|
179
208
|
}
|
|
180
209
|
function linksHelpText() {
|
|
@@ -187,10 +216,20 @@ function screenshotHelpText() {
|
|
|
187
216
|
"Options:",
|
|
188
217
|
" --viewport-only 最初のビューポート分のみ撮影する(既定はfullPage)",
|
|
189
218
|
" --width <n> タイル幅px(既定 1280)",
|
|
190
|
-
" --scale <x> 解像度スケール(
|
|
219
|
+
" --scale <x> 解像度スケール(0.5〜1.0、既定 1.0)。小さいほど画像トークンが減る",
|
|
191
220
|
" --out-dir <dir> 画像保存先(既定 カレントディレクトリ)",
|
|
192
221
|
].join("\n");
|
|
193
222
|
}
|
|
223
|
+
function installBrowserHelpText() {
|
|
224
|
+
return [
|
|
225
|
+
"amenbo install-browser",
|
|
226
|
+
"",
|
|
227
|
+
"スクリーンショット撮影やSPA(JavaScriptレンダリング必須)ページの取得に必要なChromiumを",
|
|
228
|
+
"インストールする(初回のみ、約170MBのダウンロード)。通常のHTTP取得(fetch/linksの大半)は",
|
|
229
|
+
"この操作なしで動作する。amenboに同梱されたバージョンのplaywright CLIを使うため、",
|
|
230
|
+
"バージョン不一致は起こらない。",
|
|
231
|
+
].join("\n");
|
|
232
|
+
}
|
|
194
233
|
function helpText(topic) {
|
|
195
234
|
if (topic === "fetch")
|
|
196
235
|
return fetchHelpText();
|
|
@@ -198,9 +237,22 @@ function helpText(topic) {
|
|
|
198
237
|
return linksHelpText();
|
|
199
238
|
if (topic === "screenshot")
|
|
200
239
|
return screenshotHelpText();
|
|
240
|
+
if (topic === "install-browser")
|
|
241
|
+
return installBrowserHelpText();
|
|
201
242
|
return usageText();
|
|
202
243
|
}
|
|
203
244
|
// ---- 出力(TextBlockはstdout、ImageBlockはファイル保存) ----
|
|
245
|
+
/**
|
|
246
|
+
* レビュー指摘対応(Medium・ユーザー確定仕様): `amenbo fetch`(mode auto)が品質低下でscreenshotへ
|
|
247
|
+
* 自動切替した場合の画像既定保存先。以前はoutDir未指定時にprocess.cwd()を使っており、
|
|
248
|
+
* リポジトリ直下等カレントディレクトリを黙って汚していた。cache.tsのキャッシュディレクトリ
|
|
249
|
+
* 解決ロジック(AMENBO_CACHE_DIR ?? ~/.cache/amenbo)を再利用し、その配下のscreenshots-cli/へ
|
|
250
|
+
* 保存する(実撮影キャッシュのscreenshots/とは別ディレクトリ)。screenshotサブコマンドは
|
|
251
|
+
* 明示的な視覚確認用途のため対象外(--out-dir未指定時は従来通りcwd)。
|
|
252
|
+
*/
|
|
253
|
+
function resolveFetchScreenshotDir() {
|
|
254
|
+
return join(resolveCacheDir(), "screenshots-cli");
|
|
255
|
+
}
|
|
204
256
|
/** ファイル名として安全なホスト名(取れなければ"page")。amenbo-<hostname>-<連番>.pngの命名に使う。 */
|
|
205
257
|
function hostnameForFilename(url) {
|
|
206
258
|
try {
|
|
@@ -246,7 +298,7 @@ async function dispatch(command) {
|
|
|
246
298
|
...(command.forceFull !== undefined ? { force_full: command.forceFull } : {}),
|
|
247
299
|
};
|
|
248
300
|
const blocks = await handleFetchTool(input);
|
|
249
|
-
writeBlocks(blocks, command.url, command.outDir ??
|
|
301
|
+
writeBlocks(blocks, command.url, command.outDir ?? resolveFetchScreenshotDir());
|
|
250
302
|
return;
|
|
251
303
|
}
|
|
252
304
|
if (command.kind === "links") {
|
|
@@ -291,6 +343,9 @@ export async function run(argv) {
|
|
|
291
343
|
return 0;
|
|
292
344
|
}
|
|
293
345
|
try {
|
|
346
|
+
if (command.kind === "install-browser") {
|
|
347
|
+
return await installBrowser();
|
|
348
|
+
}
|
|
294
349
|
await dispatch(command);
|
|
295
350
|
return 0;
|
|
296
351
|
}
|