amenbo 0.5.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +8 -6
- package/README.md +8 -6
- package/dist/cache.js +4 -0
- package/dist/cache.js.map +1 -1
- package/dist/cli.js +7 -2
- package/dist/cli.js.map +1 -1
- package/dist/core.js +171 -42
- package/dist/core.js.map +1 -1
- package/dist/extract/pdf.d.ts +7 -1
- package/dist/extract/pdf.js +24 -6
- package/dist/extract/pdf.js.map +1 -1
- package/dist/fetcher/browser.d.ts +10 -3
- package/dist/fetcher/browser.js +13 -6
- package/dist/fetcher/browser.js.map +1 -1
- package/dist/fetcher/http.d.ts +12 -14
- package/dist/fetcher/http.js +95 -24
- package/dist/fetcher/http.js.map +1 -1
- package/dist/fetcher/index.d.ts +4 -1
- package/dist/fetcher/index.js +23 -2
- package/dist/fetcher/index.js.map +1 -1
- package/dist/fetcher/ssrfProxy.d.ts +1 -1
- package/dist/fetcher/ssrfProxy.js +5 -1
- package/dist/fetcher/ssrfProxy.js.map +1 -1
- package/dist/formatting.d.ts +9 -1
- package/dist/formatting.js +28 -2
- package/dist/formatting.js.map +1 -1
- package/dist/politeness.d.ts +2 -0
- package/dist/politeness.js +11 -3
- package/dist/politeness.js.map +1 -1
- package/dist/screenshot.d.ts +2 -0
- package/dist/screenshot.js +16 -8
- package/dist/screenshot.js.map +1 -1
- package/dist/server.d.ts +10 -2
- package/dist/server.js +137 -107
- package/dist/server.js.map +1 -1
- package/dist/tokens.d.ts +20 -7
- package/dist/tokens.js +74 -23
- package/dist/tokens.js.map +1 -1
- package/package.json +4 -2
package/README.en.md
CHANGED
|
@@ -20,7 +20,7 @@ Most general-purpose scraping tools assume the English-speaking web. On Japanese
|
|
|
20
20
|
- **Lost headings**: on pages that wrap headings with edit-link containers (wiki-style sites), content extraction can strip the entire heading structure; amenbo detects headings whose section content survived and restores them in place (keeping outline / section progressive disclosure reliable)
|
|
21
21
|
- **Major Japanese sites**: dedicated adapters for Qiita / Zenn / note / Hatena Blog / Yahoo! News / PR TIMES
|
|
22
22
|
|
|
23
|
-
For a measured comparison against similar tools (official fetch MCP / Jina Reader / Playwright MCP / PixelRAG pixelshot), see the article "[エージェントのWeb取得、ツール次第でトークンが5000倍違った話](https://zenn.dev/rererr_engineer/articles/e571e5b6eb1d53)" (Japanese). The benchmark harness and raw logs are in [`bench/`](
|
|
23
|
+
For a measured comparison against similar tools (official fetch MCP / Jina Reader / Playwright MCP / PixelRAG pixelshot), see the article "[エージェントのWeb取得、ツール次第でトークンが5000倍違った話](https://zenn.dev/rererr_engineer/articles/e571e5b6eb1d53)" (Japanese). The benchmark harness and raw logs are in [`bench/`](https://github.com/Rererr/amenbo/blob/main/bench/).
|
|
24
24
|
|
|
25
25
|
## How it saves tokens
|
|
26
26
|
|
|
@@ -28,12 +28,12 @@ For a measured comparison against similar tools (official fetch MCP / Jina Reade
|
|
|
28
28
|
- **CJK-aware content pruning**: removes nav / ads / footers using punctuation density, character-class ratios, and link density
|
|
29
29
|
- **Diff responses**: when refetching a URL, returns `unchanged` if nothing changed, or only the changed sections
|
|
30
30
|
- **Automatic Markdown/screenshot switching**: only pages with a low quality score become screenshots, avoiding the round trip of reading broken Markdown and refetching
|
|
31
|
-
- **
|
|
31
|
+
- **Script-aware token estimation**: Japanese, Korean, Cyrillic, emoji and the like cost more tokens per character than English, so page-split budgets are computed with per-character-class coefficients calibrated against a real tokenizer
|
|
32
32
|
|
|
33
33
|
## Low impact on target sites
|
|
34
34
|
|
|
35
35
|
- **Two-tier fetching**: plain HTTP GET first; only pages that need JS rendering escalate to headless Chromium, so most fetches never launch a browser
|
|
36
|
-
- **Polite crawling**: respects robots.txt and Crawl-Delay; serialized access per domain at 1 req/sec by default. Link discovery prefers sitemap / RSS instead of crawling pages
|
|
36
|
+
- **Polite crawling**: respects robots.txt and Crawl-Delay; serialized access per domain at 1 req/sec by default (fetching robots.txt itself counts as one of those requests). Link discovery prefers sitemap / RSS instead of crawling pages
|
|
37
37
|
- **Honest User-Agent**: identifies itself as a bot. **No anti-bot circumvention is implemented**
|
|
38
38
|
- **Caching**: revalidates with ETag / If-Modified-Since to avoid needless refetches
|
|
39
39
|
|
|
@@ -92,6 +92,8 @@ For other clients (Cursor / Cline, etc.), add the following entry to the client'
|
|
|
92
92
|
|
|
93
93
|
To avoid a global install, use `"command": "npx", "args": ["-y", "amenbo"]`. For a local build, use `"command": "node", "args": ["/path/to/amenbo/dist/server.js"]`.
|
|
94
94
|
|
|
95
|
+
Over stdio, amenbo answers both MCP 2026-07-28 (stateless core) and the 2025-era revisions, so the configuration above works whichever revision your client speaks.
|
|
96
|
+
|
|
95
97
|
### Teaching your agent how to use it (recommended prompt)
|
|
96
98
|
|
|
97
99
|
Tool definitions alone don't convey usage conventions like "fetch with progressive disclosure." Paste the following into `CLAUDE.md` or `AGENTS.md` so your agent uses amenbo efficiently:
|
|
@@ -103,7 +105,7 @@ Tool definitions alone don't convey usage conventions like "fetch with progressi
|
|
|
103
105
|
is needed, first check headings and per-section token counts with `mode: "outline"`,
|
|
104
106
|
then fetch only the needed sections via `section`
|
|
105
107
|
- Getting `unchanged` / `diff` when refetching the same URL is normal (no change / changed
|
|
106
|
-
sections only). Use `force_full: true` only when the
|
|
108
|
+
sections only). Use `force_full: true` only when you need the whole content again instead of a diff
|
|
107
109
|
- When looking for pages within a site, don't guess URLs — enumerate them with `links`
|
|
108
110
|
(narrow down with `filter`)
|
|
109
111
|
- In environments with a shell, for long pages you only want to keyword-search or for bulk
|
|
@@ -163,7 +165,7 @@ Cache, diff responses (`unchanged`/`diff`), and rate-control state (robots.txt /
|
|
|
163
165
|
| `section` | Section ID obtained from outline; returns only that section's Markdown (if the section has ancestor headings, the response includes `section_path`, a breadcrumb joined by ` › `) |
|
|
164
166
|
| `page` | Page number (default 1) |
|
|
165
167
|
| `max_tokens` | Approximate token cap per page (default 8000) |
|
|
166
|
-
| `force_full` | true disables diff responses and boilerplate removal
|
|
168
|
+
| `force_full` | true disables diff responses and boilerplate removal (pagination by `max_tokens` still applies) |
|
|
167
169
|
|
|
168
170
|
### `links` — enumerate links
|
|
169
171
|
| Parameter | Description |
|
|
@@ -211,7 +213,7 @@ npm run build # build into dist/
|
|
|
211
213
|
|
|
212
214
|
If you reference this software in an article or in research, please cite the Zenodo DOI. The `10.5281/zenodo.21553636` shown in the badge above is the **Concept DOI**, shared by all versions and always resolving to the latest one. To cite a specific version, take that version's own DOI from the [Zenodo record](https://zenodo.org/records/21553637).
|
|
213
215
|
|
|
214
|
-
Machine-readable citation metadata lives in [CITATION.cff](
|
|
216
|
+
Machine-readable citation metadata lives in [CITATION.cff](https://github.com/Rererr/amenbo/blob/main/CITATION.cff) (GitHub's "Cite this repository" generates BibTeX / APA from it).
|
|
215
217
|
|
|
216
218
|
## License
|
|
217
219
|
|
package/README.md
CHANGED
|
@@ -20,7 +20,7 @@ amenbo(アメンボ / water strider)は、Claude Code や Codex のようなコ
|
|
|
20
20
|
- **見出しの消失**:見出しが編集リンク付きラッパーに包まれたページ(Wiki系など)では本文抽出時に見出し構造が丸ごと失われることがあり、本文が残っている節の見出しを検出して元の位置へ復元します(outline / section の段階開示が安定)
|
|
21
21
|
- **国内主要サイト**:Qiita / Zenn / note / はてなブログ / Yahoo!ニュース / PR TIMES に専用アダプタ
|
|
22
22
|
|
|
23
|
-
類似ツール(公式 fetch MCP / Jina Reader / Playwright MCP / PixelRAG pixelshot)との実測比較は、記事「[エージェントのWeb取得、ツール次第でトークンが5000倍違った話](https://zenn.dev/rererr_engineer/articles/e571e5b6eb1d53)」を参照してください。ハーネスと生ログは [`bench/`](
|
|
23
|
+
類似ツール(公式 fetch MCP / Jina Reader / Playwright MCP / PixelRAG pixelshot)との実測比較は、記事「[エージェントのWeb取得、ツール次第でトークンが5000倍違った話](https://zenn.dev/rererr_engineer/articles/e571e5b6eb1d53)」を参照してください。ハーネスと生ログは [`bench/`](https://github.com/Rererr/amenbo/blob/main/bench/) にあります。
|
|
24
24
|
|
|
25
25
|
## トークンを節約する仕組み
|
|
26
26
|
|
|
@@ -28,12 +28,12 @@ amenbo(アメンボ / water strider)は、Claude Code や Codex のようなコ
|
|
|
28
28
|
- **CJK 対応の本文プルーニング**:句読点密度、文字種比率、リンク密度でナビ/広告/フッターを除去
|
|
29
29
|
- **差分応答**:一度取得した URL の再取得時、変更が無ければ `unchanged`、あれば変更された節だけを返します
|
|
30
30
|
- **自動 Markdown/画像切替**:品質スコアが低いページだけスクリーンショットにし、壊れた Markdown を読ませて取り直す往復を避けます
|
|
31
|
-
-
|
|
31
|
+
- **文字種別のトークン見積り**:日本語・韓国語・キリル文字・絵文字などは英語よりトークン単価が重いため、文字クラス別の係数(実測で校正)でページ分割の予算を計算します
|
|
32
32
|
|
|
33
33
|
## 収集先への低負荷
|
|
34
34
|
|
|
35
35
|
- **二段フェッチ**:まず素の HTTP GET。JS 描画が必要なページだけ headless Chromium に昇格するので、大半の取得でブラウザを起動しません
|
|
36
|
-
- **礼儀正しいクローラ**:robots.txt と Crawl-Delay を尊重、同一ドメインへは直列 + 既定 1 req
|
|
36
|
+
- **礼儀正しいクローラ**:robots.txt と Crawl-Delay を尊重、同一ドメインへは直列 + 既定 1 req/秒(robots.txt の取得もこの1リクエストとして数えます)。リンク列挙は sitemap / RSS を優先しページを舐めません
|
|
37
37
|
- **正直な User-Agent**:ボットであることを明示します。**anti-bot 回避は実装しません**
|
|
38
38
|
- **キャッシュ**:ETag / If-Modified-Since で再検証し、無駄な再取得を避けます
|
|
39
39
|
|
|
@@ -92,6 +92,8 @@ code --add-mcp '{"name":"amenbo","command":"amenbo"}'
|
|
|
92
92
|
|
|
93
93
|
グローバルインストールを避ける場合は `"command": "npx", "args": ["-y", "amenbo"]`、ローカルビルドを使う場合は `"command": "node", "args": ["/path/to/amenbo/dist/server.js"]` を指定してください。
|
|
94
94
|
|
|
95
|
+
stdio 経由で MCP の 2026-07-28(ステートレスコア)と 2025 系の両方に応答します。クライアントがどちらの版を話すかに関わらず、上記の設定のまま繋がります。
|
|
96
|
+
|
|
95
97
|
### エージェントに使い方を教える(推奨プロンプト)
|
|
96
98
|
|
|
97
99
|
ツール定義だけでは「段階開示で取る」といった使い方の作法までは伝わりません。以下を `CLAUDE.md` や `AGENTS.md` にコピペすると、エージェントが amenbo を効率よく使うようになります。
|
|
@@ -102,7 +104,7 @@ code --add-mcp '{"name":"amenbo","command":"amenbo"}'
|
|
|
102
104
|
- ページ取得は `fetch`(mode 既定 `auto`)。長そうなページや一部しか要らないページは、
|
|
103
105
|
まず `mode: "outline"` で見出しと各節のトークン量を確認し、必要な節だけ `section` 指定で取得する
|
|
104
106
|
- 同じ URL の再取得で `unchanged` / `diff` が返るのは正常(変更なし / 変更節のみ)。
|
|
105
|
-
|
|
107
|
+
差分ではなく内容全体をもう一度受け取りたいときだけ `force_full: true` を使う
|
|
106
108
|
- サイト内のページを探すときは URL を推測せず `links`(`filter` で絞り込み)で列挙する
|
|
107
109
|
- シェルが使える環境で、キーワードで探したいだけの長いページや複数ページの一括収集は、
|
|
108
110
|
CLI で `amenbo fetch <url> > page.md` に落として grep / 部分読みする(本文をコンテキストに入れない)。
|
|
@@ -158,7 +160,7 @@ amenbo screenshot https://example.com/ --viewport-only --scale 0.5 --out-dir ./s
|
|
|
158
160
|
| `section` | outline で得た section ID。その節の Markdown のみ返す(祖先見出しがあれば応答に `section_path`(` › ` 区切りのパンくず)を付与) |
|
|
159
161
|
| `page` | ページ番号(既定 1) |
|
|
160
162
|
| `max_tokens` | 1 ページの概算トークン上限(既定 8000) |
|
|
161
|
-
| `force_full` | true
|
|
163
|
+
| `force_full` | true で差分応答・定型ブロック除去を無効化する(`max_tokens` によるページ分割は従来通り働く) |
|
|
162
164
|
|
|
163
165
|
### `links` でリンクを列挙
|
|
164
166
|
| パラメータ | 説明 |
|
|
@@ -206,7 +208,7 @@ npm run build # dist/ へビルド
|
|
|
206
208
|
|
|
207
209
|
記事や研究で参照する場合は Zenodo の DOI を使ってください。上のバッジの `10.5281/zenodo.21553636` は**全バージョン共通の Concept DOI** で、常に最新版へ解決されます。特定の版を指す場合は、その版の DOI を [Zenodo のレコード](https://zenodo.org/records/21553637)から取得してください。
|
|
208
210
|
|
|
209
|
-
機械可読な引用情報は [CITATION.cff](
|
|
211
|
+
機械可読な引用情報は [CITATION.cff](https://github.com/Rererr/amenbo/blob/main/CITATION.cff) にあります(GitHub の "Cite this repository" から BibTeX / APA を生成できます)。
|
|
210
212
|
|
|
211
213
|
## ライセンス
|
|
212
214
|
|
package/dist/cache.js
CHANGED
|
@@ -141,6 +141,7 @@ export class PageCache {
|
|
|
141
141
|
deletePagesExpired: this.db.prepare("DELETE FROM pages WHERE fetched_at < ?"),
|
|
142
142
|
deleteDomainPagesExpired: this.db.prepare("DELETE FROM domain_pages WHERE fetched_at < ?"),
|
|
143
143
|
deleteHostRequestsExpired: this.db.prepare("DELETE FROM host_requests WHERE last_request_at < ?"),
|
|
144
|
+
deleteRobotsExpired: this.db.prepare("DELETE FROM robots_cache WHERE fetched_at < ?"),
|
|
144
145
|
selectScreenshotsExpired: this.db.prepare("SELECT * FROM screenshots WHERE fetched_at < ?"),
|
|
145
146
|
deleteScreenshotsExpired: this.db.prepare("DELETE FROM screenshots WHERE fetched_at < ?"),
|
|
146
147
|
getPage: this.db.prepare("SELECT * FROM pages WHERE url = ?"),
|
|
@@ -217,6 +218,9 @@ export class PageCache {
|
|
|
217
218
|
// code-reviewer指摘: host_requestsもM3の対象から漏れていた(訪問ホスト毎に1行増える
|
|
218
219
|
// 無制限増加テーブルという点でdomain_pages等と同じ性質を持つため、他テーブルと一貫させる)。
|
|
219
220
|
this.statements.deleteHostRequestsExpired.run(cutoff);
|
|
221
|
+
// robots_cacheも同じ性質(オリジン毎に1行)なのに掃除対象から漏れていた。
|
|
222
|
+
// 消えてもPolitenessManager側が取得し直すだけなので、他テーブルと同じTTLで揃える。
|
|
223
|
+
this.statements.deleteRobotsExpired.run(cutoff);
|
|
220
224
|
const expiredScreenshots = this.statements.selectScreenshotsExpired.all(cutoff);
|
|
221
225
|
for (const row of expiredScreenshots) {
|
|
222
226
|
const tilePaths = JSON.parse(row.tile_paths);
|
package/dist/cache.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cache.js","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAmCG;AACH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,YAAY,EAAsB,MAAM,aAAa,CAAC;AAwB/D,MAAM,cAAc,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,CAAC,MAAM;AAC7C,gDAAgD;AAChD,MAAM,6BAA6B,GAAG,CAAC,CAAC;AACxC,kCAAkC;AAClC,MAAM,mBAAmB,GAAG,gBAAgB,CAAC;AAE7C,mDAAmD;AACnD,oEAAoE;AACpE,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,gBAAgB,IAAI,IAAI,CAAC,OAAO,EAAE,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAC;IAChF,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACrB,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACtC,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;
|
|
1
|
+
{"version":3,"file":"cache.js","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAmCG;AACH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACvE,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,YAAY,EAAsB,MAAM,aAAa,CAAC;AAwB/D,MAAM,cAAc,GAAG,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,CAAC,MAAM;AAC7C,gDAAgD;AAChD,MAAM,6BAA6B,GAAG,CAAC,CAAC;AACxC,kCAAkC;AAClC,MAAM,mBAAmB,GAAG,gBAAgB,CAAC;AAE7C,mDAAmD;AACnD,oEAAoE;AACpE,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,gBAAgB,IAAI,IAAI,CAAC,OAAO,EAAE,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAC;IAChF,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACrB,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACtC,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AA6DD,MAAM,UAAU,WAAW,CAAC,OAAe;IACzC,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AAC5D,CAAC;AAED,6CAA6C;AAC7C,MAAM,UAAU,yBAAyB,CAAC,GAAW,EAAE,KAAa,EAAE,KAAa,EAAE,QAAiB;IACpG,OAAO,WAAW,CAAC,GAAG,GAAG,MAAM,KAAK,MAAM,KAAK,SAAS,QAAQ,EAAE,CAAC,CAAC;AACtE,CAAC;AAiBD,MAAM,OAAO,SAAS;IACH,EAAE,CAAe;IACjB,QAAQ,CAAS;IACjB,KAAK,CAAS;IACd,GAAG,CAAe;IAClB,UAAU,CAAqB;IACxC,MAAM,GAAG,KAAK,CAAC;IAEvB,YAAY,OAAO,GAA+E,EAAE;QAClG,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,eAAe,EAAE,CAAC;QACtD,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/B,SAAS,CAAC,IAAI,CAAC,QAAQ,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QAChD,CAAC;QACD,IAAI,CAAC,EAAE,GAAG,IAAI,YAAY,CAAC,OAAO,CAAC,MAAM,IAAI,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,cAAc,CAAC,CAAC,CAAC;QAClF,IAAI,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,IAAI,cAAc,CAAC;QAC7C,IAAI,CAAC,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,IAAI,CAAC,GAAG,CAAC;QACnC,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC;QAC1C,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;KAUZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;KAQZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;;;KAQZ,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC,yGAAyG,CAAC,CAAC;QACxH,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;KAKZ,CAAC,CAAC;QACH,2DAA2D;QAC3D,mDAAmD;QACnD,0DAA0D;QAC1D,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;;KAMZ,CAAC,CAAC;QACH,mDAAmD;QACnD,IAAI,CAAC,EAAE,CAAC,IAAI,CAAC;;;;;KAKZ,CAAC,CAAC;QAEH,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,iBAAiB,EAAE,CAAC;QAE3C,2CAA2C;QAC3C,kDAAkD;QAClD,4CAA4C;QAC5C,IAAI,CAAC,iBAAiB,EAAE,CAAC;IAC3B,CAAC;IAED,6CAA6C;IACrC,iBAAiB;QACvB,OAAO;YACL,kBAAkB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,wCAAwC,CAAC;YAC7E,wBAAwB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC;YAC1F,yBAAyB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,qDAAqD,CAAC;YACjG,mBAAmB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC;YACrF,wBAAwB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,gDAAgD,CAAC;YAC3F,wBAAwB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,8CAA8C,CAAC;YACzF,OAAO,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,mCAAmC,CAAC;YAC7D,OAAO,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CACtB;;;;;;;;4CAQoC,CACrC;YACD,SAAS,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC;YAC3E,aAAa,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,+CAA+C,CAAC;YAC/E,aAAa,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAC5B;;;;;;4CAMoC,CACrC;YACD,gBAAgB,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAC/B;;;;gDAIwC,CACzC;YACD,cAAc,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,8EAA8E,CAAC;YAC/G,cAAc,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,4CAA4C,CAAC;YAC7E,cAAc,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAC7B;;0GAEkG,CACnG;YACD,SAAS,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,6CAA6C,CAAC;YACzE,SAAS,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CACxB;;;;4CAIoC,CACrC;YACD,OAAO,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,sCAAsC,CAAC;YAChE,OAAO,EAAE,IAAI,CAAC,EAAE,CAAC,OAAO,CACtB;+DACuD,CACxD;SACF,CAAC;IACJ,CAAC;IAED;;;;;;;;;OASG;IACK,iBAAiB;QACvB,MAAM,YAAY,GAAG,IAAI,CAAC,YAAY,CAAC,mBAAmB,CAAC,CAAC;QAC5D,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACtB,IAAI,YAAY,KAAK,IAAI,IAAI,EAAE,GAAG,YAAY,GAAG,IAAI,CAAC,KAAK;YAAE,OAAO;QACpE,IAAI,CAAC,YAAY,EAAE,CAAC;QACpB,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,GAAG,EAAE,mBAAmB,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,CAAC;IACvE,CAAC;IAED,qCAAqC;IAC7B,YAAY,CAAC,GAAW;QAC9B,MAAM,GAAG,GAAG,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAkC,CAAC;QAC9E,OAAO,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;IACxC,CAAC;IAED;;;OAGG;IACK,YAAY;QAClB,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC;QAEvC,IAAI,CAAC,UAAU,CAAC,kBAAkB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAC/C,IAAI,CAAC,UAAU,CAAC,wBAAwB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACrD,2DAA2D;QAC3D,sDAAsD;QACtD,IAAI,CAAC,UAAU,CAAC,yBAAyB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QACtD,6CAA6C;QAC7C,qDAAqD;QACrD,IAAI,CAAC,UAAU,CAAC,mBAAmB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;QAEhD,MAAM,kBAAkB,GAAG,IAAI,CAAC,UAAU,CAAC,wBAAwB,CAAC,GAAG,CAAC,MAAM,CAA+B,CAAC;QAC9G,KAAK,MAAM,GAAG,IAAI,kBAAkB,EAAE,CAAC;YACrC,MAAM,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,CAAa,CAAC;YACzD,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE,CAAC;gBACjC,IAAI,CAAC;oBACH,MAAM,CAAC,QAAQ,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;gBACpC,CAAC;gBAAC,MAAM,CAAC;oBACP,oCAAoC;gBACtC,CAAC;YACH,CAAC;QACH,CAAC;QACD,IAAI,CAAC,UAAU,CAAC,wBAAwB,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;IACvD,CAAC;IAED,iDAAiD;IACjD,GAAG,CAAC,GAAW;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAwB,CAAC;QACpE,IAAI,CAAC,GAAG;YAAE,OAAO,SAAS,CAAC;QAC3B,OAAO;YACL,GAAG,EAAE,GAAG,CAAC,GAAG;YACZ,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,YAAY,EAAE,GAAG,CAAC,aAAa;YAC/B,WAAW,EAAE,GAAG,CAAC,YAAY;YAC7B,QAAQ,EAAE,GAAG,CAAC,QAAQ;YACtB,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAA4B;YAC7D,SAAS,EAAE,GAAG,CAAC,UAAU;SAC1B,CAAC;IACJ,CAAC;IAED,qDAAqD;IACrD,OAAO,CAAC,KAA4B;QAClC,OAAO,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,SAAS,GAAG,IAAI,CAAC,KAAK,CAAC;IACnD,CAAC;IAED,kDAAkD;IAClD,aAAa,CAAC,QAAgB;QAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,UAAU,CAAC,aAAa,CAAC,GAAG,CAAC,QAAQ,CAA8B,CAAC;QACrF,IAAI,CAAC,GAAG;YAAE,OAAO,SAAS,CAAC;QAC3B,OAAO;YACL,QAAQ,EAAE,GAAG,CAAC,SAAS;YACvB,GAAG,EAAE,GAAG,CAAC,GAAG;YACZ,SAAS,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,UAAU,CAAa;YACjD,QAAQ,EAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAA4B;YAC7D,SAAS,EAAE,GAAG,CAAC,UAAU;SAC1B,CAAC;IACJ,CAAC;IAED;;;;;;;;;OASG;IACH,aAAa,CAAC,KAA2B;QACvC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,aAAa,EAAE,KAAK,CAAC,QAAQ,CAAC,CAAC;QACnE,MAAM,CAAC,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;QAClD,SAAS,CAAC,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QACxC,MAAM,SAAS,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE;YAClD,MAAM,QAAQ,GAAG,IAAI,CAAC,OAAO,EAAE,QAAQ,KAAK,MAAM,CAAC,CAAC;YACpD,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAC;YAChC,OAAO,QAAQ,CAAC;QAClB,CAAC,CAAC,CAAC;QAEH,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,IAAI,CAAC,UAAU,CAAC,aAAa,CAAC,GAAG,CAAC;YAChC,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC;YACpC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,CAAC;YACxC,SAAS;SACV,CAAC,CAAC;QAEH,OAAO,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,GAAG,EAAE,KAAK,CAAC,GAAG,EAAE,SAAS,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,SAAS,EAAE,CAAC;IACtG,CAAC;IAED,8BAA8B;IAC9B,GAAG,CAAC,KAAsB;QACxB,MAAM,WAAW,GAAG,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;QAChD,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QAC7B,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC,GAAG,CAAC;YAC1B,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,YAAY,EAAE,KAAK,CAAC,YAAY;YAChC,WAAW;YACX,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,QAAQ,CAAC;YACxC,SAAS;SACV,CAAC,CAAC;QACH,OAAO;YACL,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,YAAY,EAAE,KAAK,CAAC,YAAY;YAChC,WAAW;YACX,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,QAAQ,EAAE,KAAK,CAAC,QAAQ;YACxB,SAAS;SACV,CAAC;IACJ,CAAC;IAED,2CAA2C;IAC3C,KAAK,CAAC,GAAW;QACf,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,EAAE,EAAE,GAAG,CAAC,CAAC;IACjD,CAAC;IAED,2DAA2D;IAC3D,sBAAsB,CAAC,MAAc,EAAE,GAAW,EAAE,WAAqB;QACvE,IAAI,CAAC,UAAU,CAAC,gBAAgB,CAAC,GAAG,CAAC,EAAE,MAAM,EAAE,GAAG,EAAE,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,EAAE,WAAW,EAAE,IAAI,CAAC,SAAS,CAAC,WAAW,CAAC,EAAE,CAAC,CAAC;IACzH,CAAC;IAED;;;;OAIG;IACH,sBAAsB,CAAC,MAAc,EAAE,WAAW,GAAW,6BAA6B;QACxF,MAAM,IAAI,GAAG,IAAI,CAAC,UAAU,CAAC,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,WAAW,CAA+B,CAAC;QAEnG,IAAI,IAAI,CAAC,MAAM,GAAG,WAAW;YAAE,OAAO,IAAI,GAAG,EAAE,CAAC;QAEhD,MAAM,eAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAClD,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;YACvB,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,YAAY,CAAa,CAAC;YACxD,KAAK,MAAM,IAAI,IAAI,MAAM,EAAE,CAAC;gBAC1B,eAAe,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,eAAe,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YAClE,CAAC;QACH,CAAC;QAED,MAAM,cAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACzC,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,eAAe,EAAE,CAAC;YAC5C,IAAI,KAAK,IAAI,WAAW;gBAAE,cAAc,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACrD,CAAC;QACD,OAAO,cAAc,CAAC;IACxB,CAAC;IAED;;;OAGG;IACH,oBAAoB,CAAC,IAAY;QAC/B,MAAM,GAAG,GAAG,IAAI,CAAC,UAAU,CAAC,cAAc,CAAC,GAAG,CAAC,IAAI,CAA+B,CAAC;QACnF,OAAO,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,IAAI,CAAC;IAC1C,CAAC;IAED;;;;;;;;OAQG;IACH,oBAAoB,CAAC,IAAY,EAAE,EAAU;QAC3C,IAAI,CAAC,UAAU,CAAC,cAAc,CAAC,GAAG,CAAC,EAAE,IAAI,EAAE,aAAa,EAAE,EAAE,EAAE,CAAC,CAAC;IAClE,CAAC;IAED;;;;;;;OAOG;IACH,cAAc,CAAC,MAAc;QAC3B,MAAM,GAAG,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,GAAG,CAAC,MAAM,CAA+B,CAAC;QAChF,OAAO,GAAG,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,EAAE,SAAS,EAAE,GAAG,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IACpE,CAAC;IAED,qDAAqD;IACrD,cAAc,CAAC,MAAc,EAAE,IAAY,EAAE,SAAiB;QAC5D,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC,CAAC;IAC7D,CAAC;IAED;;;;OAIG;IACH,KAAK;QACH,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;IAClB,CAAC;CACF"}
|
package/dist/cli.js
CHANGED
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
* 共有する。CLIは1コマンド=1プロセスのため、politenessのドメイン毎レート制御は
|
|
16
16
|
* core.ts側でcacheをstoreとして注入したPolitenessManagerがプロセス間永続化する。
|
|
17
17
|
*/
|
|
18
|
+
import { createHash } from "node:crypto";
|
|
18
19
|
import { mkdirSync, realpathSync, writeFileSync } from "node:fs";
|
|
19
20
|
import { join } from "node:path";
|
|
20
21
|
import { parseArgs } from "node:util";
|
|
@@ -254,7 +255,7 @@ function helpText(topic) {
|
|
|
254
255
|
function resolveFetchScreenshotDir() {
|
|
255
256
|
return join(resolveCacheDir(), "screenshots-cli");
|
|
256
257
|
}
|
|
257
|
-
/** ファイル名として安全なホスト名(取れなければ"page")。amenbo-<hostname
|
|
258
|
+
/** ファイル名として安全なホスト名(取れなければ"page")。amenbo-<hostname>-<URLハッシュ>-<連番>.pngの命名に使う。 */
|
|
258
259
|
function hostnameForFilename(url) {
|
|
259
260
|
try {
|
|
260
261
|
return new URL(url).hostname.replace(/[^A-Za-z0-9.-]/g, "_") || "page";
|
|
@@ -266,6 +267,10 @@ function hostnameForFilename(url) {
|
|
|
266
267
|
/** TextBlockは標準出力へ、ImageBlockはoutDir配下へ保存しパスを列挙する。テスト用export。 */
|
|
267
268
|
export function writeBlocks(blocks, url, outDir) {
|
|
268
269
|
const hostname = hostnameForFilename(url);
|
|
270
|
+
// 同じホストの別ページを同じ--out-dirへ撮ると、連番だけでは同名になり前回の画像を
|
|
271
|
+
// 黙って上書きしてしまう。URLごとに短いハッシュを挟んで衝突を避ける
|
|
272
|
+
// (同じURLの撮り直しは同じ名前になり、これまで通り上書きされる)。
|
|
273
|
+
const urlDigest = createHash("sha256").update(url).digest("hex").slice(0, 12);
|
|
269
274
|
const savedPaths = [];
|
|
270
275
|
let sequence = 0;
|
|
271
276
|
for (const block of blocks) {
|
|
@@ -278,7 +283,7 @@ export function writeBlocks(blocks, url, outDir) {
|
|
|
278
283
|
mkdirSync(outDir, { recursive: true });
|
|
279
284
|
}
|
|
280
285
|
const extension = block.mimeType === "image/png" ? "png" : "bin";
|
|
281
|
-
const filePath = join(outDir, `amenbo-${hostname}-${sequence}.${extension}`);
|
|
286
|
+
const filePath = join(outDir, `amenbo-${hostname}-${urlDigest}-${sequence}.${extension}`);
|
|
282
287
|
writeFileSync(filePath, Buffer.from(block.data, "base64"));
|
|
283
288
|
savedPaths.push(filePath);
|
|
284
289
|
}
|
package/dist/cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA;;;;;;;;;;;;;;;GAeG;AACH,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACjE,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAC7C,OAAO,EACL,KAAK,EACL,eAAe,EACf,oBAAoB,EACpB,UAAU,EACV,qBAAqB,GAKtB,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,cAAc,EAAE,MAAM,qBAAqB,CAAC;AACrD,OAAO,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAC3C,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AACvD,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAExC,iCAAiC;AAEjC,yDAAyD;AACzD,MAAM,OAAO,aAAc,SAAQ,KAAK;CAAG;AAE3C,MAAM,WAAW,GAAG,CAAC,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,YAAY,CAAU,CAAC;AAuC3E,SAAS,gBAAgB,CAAC,GAAW,EAAE,IAAY;IACjD,MAAM,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IACtB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QACnC,MAAM,IAAI,aAAa,CAAC,GAAG,IAAI,mBAAmB,GAAG,EAAE,CAAC,CAAC;IAC3D,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAS,kBAAkB,CAAC,GAAW,EAAE,IAAY;IACnD,MAAM,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IACtB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAClC,MAAM,IAAI,aAAa,CAAC,GAAG,IAAI,mBAAmB,GAAG,EAAE,CAAC,CAAC;IAC3D,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED;;;;GAIG;AACH,SAAS,oBAAoB,CAAC,GAAW;IACvC,MAAM,CAAC,GAAG,kBAAkB,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;IAC7C,IAAI,CAAC,GAAG,SAAS,IAAI,CAAC,GAAG,SAAS,EAAE,CAAC;QACnC,MAAM,IAAI,aAAa,CAAC,WAAW,SAAS,IAAI,SAAS,iBAAiB,GAAG,EAAE,CAAC,CAAC;IACnF,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAS,UAAU,CAAC,WAAqB,EAAE,OAAe;IACxD,MAAM,GAAG,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IAC3B,IAAI,CAAC,GAAG,EAAE,CAAC;QACT,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,gBAAgB,CAAC,CAAC;IACtD,CAAC;IACD,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,cAAc,WAAW,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IACpE,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,4DAA4D;AAC5D,SAAS,aAAa,CACpB,IAAc,EACd,OAAe,EACf,OAAU;IAEV,IAAI,CAAC;QACH,wDAAwD;QACxD,0CAA0C;QAC1C,MAAM,MAAM,GAAG,SAAS,CAAC,EAAE,IAAI,EAAE,gBAAgB,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,CAAC;QACpE,OAAO,MAAsI,CAAC;IAChJ,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,KAAK,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACnG,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,IAAc;IACpC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,OAAO,EAAE;QAC3D,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACxB,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC5B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC3B,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACxB,YAAY,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAChC,YAAY,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE;QACjC,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAEzD,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC;IAE7C,IAAI,MAAM,CAAC,IAAI,KAAK,SAAS,IAAI,CAAE,WAAiC,CAAC,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;QAC3F,MAAM,IAAI,aAAa,CAAC,iBAAiB,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,mBAAmB,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IAClG,CAAC;IAED,OAAO;QACL,IAAI,EAAE,OAAO;QACb,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,IAAoB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC3E,GAAG,CAAC,MAAM,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvE,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpE,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,gBAAgB,CAAC,MAAM,CAAC,IAAI,EAAE,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvF,GAAG,CAAC,MAAM,CAAC,YAAY,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,gBAAgB,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,cAAc,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpH,GAAG,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpD,GAAG,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC1E,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CAAC,IAAc;IACpC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,OAAO,EAAE;QAC3D,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC1B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAEzD,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC;IAC7C,OAAO;QACL,IAAI,EAAE,OAAO;QACb,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAClE,CAAC;AACJ,CAAC;AAED,SAAS,mBAAmB,CAAC,IAAc;IACzC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,YAAY,EAAE;QAChE,eAAe,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE;QACpC,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACzB,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACzB,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,YAAY,EAAE,CAAC;IAE9D,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,YAAY,CAAC,CAAC;IAClD,OAAO;QACL,IAAI,EAAE,YAAY;QAClB,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC1D,GAAG,CAAC,MAAM,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,gBAAgB,CAAC,MAAM,CAAC,KAAK,EAAE,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC3F,GAAG,CAAC,MAAM,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,oBAAoB,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpF,GAAG,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC1E,CAAC;AACJ,CAAC;AAED,SAAS,uBAAuB,CAAC,IAAc;IAC7C,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,iBAAiB,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,EAAE,CAAC,CAAC;IAElH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,iBAAiB,EAAE,CAAC;IACnE,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,aAAa,CAAC,6BAA6B,WAAW,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IACzE,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,iBAAiB,EAAE,CAAC;AACrC,CAAC;AAED,oDAAoD;AACpD,MAAM,UAAU,YAAY,CAAC,IAAc;IACzC,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;IAEhD,MAAM,CAAC,KAAK,EAAE,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC;IAE9B,IAAI,KAAK,KAAK,WAAW,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC;IACxE,IAAI,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;IAClE,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;IAChD,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,cAAc,CAAC,IAAI,CAAC,CAAC;IACnD,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,cAAc,CAAC,IAAI,CAAC,CAAC;IACnD,IAAI,KAAK,KAAK,YAAY;QAAE,OAAO,mBAAmB,CAAC,IAAI,CAAC,CAAC;IAC7D,IAAI,KAAK,KAAK,iBAAiB;QAAE,OAAO,uBAAuB,CAAC,IAAI,CAAC,CAAC;IAEtE,MAAM,IAAI,aAAa,CAAC,cAAc,KAAK,EAAE,CAAC,CAAC;AACjD,CAAC;AAED,wBAAwB;AAExB,SAAS,SAAS;IAChB,OAAO;QACL,4BAA4B;QAC5B,EAAE;QACF,WAAW;QACX,8DAA8D;QAC9D,sDAAsD;QACtD,+DAA+D;QAC/D,qDAAqD;QACrD,6EAA6E;QAC7E,EAAE;QACF,UAAU;QACV,6BAA6B;QAC7B,+BAA+B;QAC/B,EAAE;QACF,gDAAgD;KACjD,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,aAAa;IACpB,OAAO;QACL,8BAA8B;QAC9B,EAAE;QACF,UAAU;QACV,8DAA8D;QAC9D,8DAA8D;QAC9D,qFAAqF;QACrF,2DAA2D;QAC3D,uEAAuE;QACvE,0FAA0F;QAC1F,kJAAkJ;KACnJ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,aAAa;IACpB,OAAO,CAAC,8BAA8B,EAAE,EAAE,EAAE,UAAU,EAAE,sDAAsD,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7H,CAAC;AAED,SAAS,kBAAkB;IACzB,OAAO;QACL,mCAAmC;QACnC,EAAE;QACF,UAAU;QACV,qDAAqD;QACrD,uCAAuC;QACvC,8DAA8D;QAC9D,4CAA4C;KAC7C,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,sBAAsB;IAC7B,OAAO;QACL,wBAAwB;QACxB,EAAE;QACF,wDAAwD;QACxD,yDAAyD;QACzD,oDAAoD;QACpD,iBAAiB;KAClB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,QAAQ,CAAC,KAAuE;IACvF,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,aAAa,EAAE,CAAC;IAC9C,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,aAAa,EAAE,CAAC;IAC9C,IAAI,KAAK,KAAK,YAAY;QAAE,OAAO,kBAAkB,EAAE,CAAC;IACxD,IAAI,KAAK,KAAK,iBAAiB;QAAE,OAAO,sBAAsB,EAAE,CAAC;IACjE,OAAO,SAAS,EAAE,CAAC;AACrB,CAAC;AAED,mDAAmD;AAEnD;;;;;;;GAOG;AACH,SAAS,yBAAyB;IAChC,OAAO,IAAI,CAAC,eAAe,EAAE,EAAE,iBAAiB,CAAC,CAAC;AACpD,CAAC;AAED,sEAAsE;AACtE,SAAS,mBAAmB,CAAC,GAAW;IACtC,IAAI,CAAC;QACH,OAAO,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,iBAAiB,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC;IACzE,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,MAAM,CAAC;IAChB,CAAC;AACH,CAAC;AAED,iEAAiE;AACjE,MAAM,UAAU,WAAW,CAAC,MAAqC,EAAE,GAAW,EAAE,MAAc;IAC5F,MAAM,QAAQ,GAAG,mBAAmB,CAAC,GAAG,CAAC,CAAC;IAC1C,MAAM,UAAU,GAAa,EAAE,CAAC;IAChC,IAAI,QAAQ,GAAG,CAAC,CAAC;IAEjB,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QAC3B,IAAI,KAAK,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;YAC1B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,IAAI,IAAI,CAAC,CAAC;YACxC,SAAS;QACX,CAAC;QACD,QAAQ,EAAE,CAAC;QACX,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC5B,SAAS,CAAC,MAAM,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QACzC,CAAC;QACD,MAAM,SAAS,GAAG,KAAK,CAAC,QAAQ,KAAK,WAAW,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;QACjE,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,EAAE,UAAU,QAAQ,IAAI,QAAQ,IAAI,SAAS,EAAE,CAAC,CAAC;QAC7E,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,CAAC;QAC3D,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC5B,CAAC;IAED,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC1B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,iBAAiB,UAAU,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,KAAK,QAAQ,EAAE,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACtG,CAAC;AACH,CAAC;AAED,eAAe;AAEf,KAAK,UAAU,QAAQ,CAAC,OAA0E;IAChG,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,MAAM,KAAK,GAAmB;YAC5B,GAAG,EAAE,OAAO,CAAC,GAAG;YAChB,GAAG,CAAC,OAAO,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7D,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACzE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACtE,GAAG,CAAC,OAAO,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7D,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7E,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAC9E,CAAC;QACF,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,KAAK,CAAC,CAAC;QAC5C,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,MAAM,IAAI,yBAAyB,EAAE,CAAC,CAAC;QAChF,OAAO;IACT,CAAC;IAED,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,MAAM,MAAM,GAAG,MAAM,aAAa,CAAC,OAAO,CAAC,GAAG,EAAE,UAAU,EAAE,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QAC5H,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,mBAAmB,CAAC,OAAO,CAAC,GAAG,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC;QACtE,OAAO;IACT,CAAC;IAED,MAAM,KAAK,GAAwB;QACjC,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,QAAQ,EAAE,CAAC,OAAO,CAAC,YAAY;QAC/B,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACjE,CAAC;IACF,MAAM,MAAM,GAAG,MAAM,oBAAoB,CAAC,KAAK,CAAC,CAAC;IACjD,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,MAAM,IAAI,OAAO,CAAC,GAAG,EAAE,CAAC,CAAC;AACpE,CAAC;AAED,sDAAsD;AACtD,MAAM,CAAC,KAAK,UAAU,GAAG,CAAC,IAAc;IACtC,IAAI,OAAsB,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC/B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,aAAa,EAAE,CAAC;YACnC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,OAAO,OAAO,SAAS,EAAE,IAAI,CAAC,CAAC;YAC7D,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,KAAK,CAAC;IACd,CAAC;IAED,IAAI,OAAO,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;QAC5B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACrD,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,OAAO,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;QAC/B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,qBAAqB,EAAE,IAAI,CAAC,CAAC;QACrD,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,4DAA4D;QAC5D,mCAAmC;QACnC,MAAM,SAAS,EAAE,CAAC;QAClB,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,CAAC;QACH,IAAI,OAAO,CAAC,IAAI,KAAK,iBAAiB,EAAE,CAAC;YACvC,OAAO,MAAM,cAAc,EAAE,CAAC;QAChC,CAAC;QACD,MAAM,QAAQ,CAAC,OAAO,CAAC,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,WAAW,EAAE,CAAC;YACjC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,OAAO,IAAI,CAAC,CAAC;QAC7C,CAAC;aAAM,CAAC;YACN,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACvB,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;YAAS,CAAC;QACT,iDAAiD;QACjD,sDAAsD;QACtD,MAAM,YAAY,EAAE,CAAC;QACrB,KAAK,CAAC,KAAK,EAAE,CAAC;IAChB,CAAC;AACH,CAAC;AAED;;;;GAIG;AACH,SAAS,kBAAkB;IACzB,OAAO,OAAO,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,QAAQ,IAAI,aAAa,CAAC,OAAO,IAAI,CAAC,GAAG,CAAC,KAAK,YAAY,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;AACjH,CAAC;AAED,IAAI,kBAAkB,EAAE,EAAE,CAAC;IACzB,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;SACvB,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE;QACjB,OAAO,CAAC,QAAQ,GAAG,QAAQ,CAAC;IAC9B,CAAC,CAAC;SACD,KAAK,CAAC,CAAC,KAAc,EAAE,EAAE;QACxB,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACrB,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;IACvB,CAAC,CAAC,CAAC;AACP,CAAC"}
|
|
1
|
+
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA;;;;;;;;;;;;;;;GAeG;AACH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACjE,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,SAAS,EAAE,MAAM,WAAW,CAAC;AACtC,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAC7C,OAAO,EACL,KAAK,EACL,eAAe,EACf,oBAAoB,EACpB,UAAU,EACV,qBAAqB,GAKtB,MAAM,WAAW,CAAC;AACnB,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACpD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,cAAc,EAAE,MAAM,qBAAqB,CAAC;AACrD,OAAO,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAC3C,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AACvD,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAExC,iCAAiC;AAEjC,yDAAyD;AACzD,MAAM,OAAO,aAAc,SAAQ,KAAK;CAAG;AAE3C,MAAM,WAAW,GAAG,CAAC,MAAM,EAAE,UAAU,EAAE,SAAS,EAAE,YAAY,CAAU,CAAC;AAuC3E,SAAS,gBAAgB,CAAC,GAAW,EAAE,IAAY;IACjD,MAAM,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IACtB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QACnC,MAAM,IAAI,aAAa,CAAC,GAAG,IAAI,mBAAmB,GAAG,EAAE,CAAC,CAAC;IAC3D,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAS,kBAAkB,CAAC,GAAW,EAAE,IAAY;IACnD,MAAM,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;IACtB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAClC,MAAM,IAAI,aAAa,CAAC,GAAG,IAAI,mBAAmB,GAAG,EAAE,CAAC,CAAC;IAC3D,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED;;;;GAIG;AACH,SAAS,oBAAoB,CAAC,GAAW;IACvC,MAAM,CAAC,GAAG,kBAAkB,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;IAC7C,IAAI,CAAC,GAAG,SAAS,IAAI,CAAC,GAAG,SAAS,EAAE,CAAC;QACnC,MAAM,IAAI,aAAa,CAAC,WAAW,SAAS,IAAI,SAAS,iBAAiB,GAAG,EAAE,CAAC,CAAC;IACnF,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAS,UAAU,CAAC,WAAqB,EAAE,OAAe;IACxD,MAAM,GAAG,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC;IAC3B,IAAI,CAAC,GAAG,EAAE,CAAC;QACT,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,gBAAgB,CAAC,CAAC;IACtD,CAAC;IACD,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,cAAc,WAAW,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IACpE,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,4DAA4D;AAC5D,SAAS,aAAa,CACpB,IAAc,EACd,OAAe,EACf,OAAU;IAEV,IAAI,CAAC;QACH,wDAAwD;QACxD,0CAA0C;QAC1C,MAAM,MAAM,GAAG,SAAS,CAAC,EAAE,IAAI,EAAE,gBAAgB,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,CAAC;QACpE,OAAO,MAAsI,CAAC;IAChJ,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,aAAa,CAAC,GAAG,OAAO,KAAK,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACnG,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,IAAc;IACpC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,OAAO,EAAE;QAC3D,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACxB,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC5B,OAAO,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC3B,IAAI,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACxB,YAAY,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAChC,YAAY,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE;QACjC,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAEzD,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC;IAE7C,IAAI,MAAM,CAAC,IAAI,KAAK,SAAS,IAAI,CAAE,WAAiC,CAAC,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC;QAC3F,MAAM,IAAI,aAAa,CAAC,iBAAiB,WAAW,CAAC,IAAI,CAAC,GAAG,CAAC,mBAAmB,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IAClG,CAAC;IAED,OAAO;QACL,IAAI,EAAE,OAAO;QACb,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,IAAoB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC3E,GAAG,CAAC,MAAM,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvE,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpE,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,gBAAgB,CAAC,MAAM,CAAC,IAAI,EAAE,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvF,GAAG,CAAC,MAAM,CAAC,YAAY,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,gBAAgB,CAAC,MAAM,CAAC,YAAY,CAAC,EAAE,cAAc,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpH,GAAG,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpD,GAAG,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC1E,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CAAC,IAAc;IACpC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,OAAO,EAAE;QAC3D,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC1B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;IAEzD,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,OAAO,CAAC,CAAC;IAC7C,OAAO;QACL,IAAI,EAAE,OAAO;QACb,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAClE,CAAC;AACJ,CAAC;AAED,SAAS,mBAAmB,CAAC,IAAc;IACzC,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,YAAY,EAAE;QAChE,eAAe,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE;QACpC,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACzB,KAAK,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QACzB,SAAS,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE;KACtC,CAAC,CAAC;IAEH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,YAAY,EAAE,CAAC;IAE9D,MAAM,GAAG,GAAG,UAAU,CAAC,WAAW,EAAE,YAAY,CAAC,CAAC;IAClD,OAAO;QACL,IAAI,EAAE,YAAY;QAClB,GAAG;QACH,GAAG,CAAC,MAAM,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC1D,GAAG,CAAC,MAAM,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,gBAAgB,CAAC,MAAM,CAAC,KAAK,EAAE,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC3F,GAAG,CAAC,MAAM,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,oBAAoB,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACpF,GAAG,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC1E,CAAC;AACJ,CAAC;AAED,SAAS,uBAAuB,CAAC,IAAc;IAC7C,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,aAAa,CAAC,IAAI,EAAE,iBAAiB,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,GAAG,EAAE,EAAE,CAAC,CAAC;IAElH,IAAI,MAAM,CAAC,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,iBAAiB,EAAE,CAAC;IACnE,IAAI,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3B,MAAM,IAAI,aAAa,CAAC,6BAA6B,WAAW,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IACzE,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,iBAAiB,EAAE,CAAC;AACrC,CAAC;AAED,oDAAoD;AACpD,MAAM,UAAU,YAAY,CAAC,IAAc;IACzC,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;IAEhD,MAAM,CAAC,KAAK,EAAE,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC;IAE9B,IAAI,KAAK,KAAK,WAAW,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,SAAS,EAAE,CAAC;IACxE,IAAI,KAAK,KAAK,QAAQ,IAAI,KAAK,KAAK,IAAI;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;IAClE,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC;IAChD,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,cAAc,CAAC,IAAI,CAAC,CAAC;IACnD,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,cAAc,CAAC,IAAI,CAAC,CAAC;IACnD,IAAI,KAAK,KAAK,YAAY;QAAE,OAAO,mBAAmB,CAAC,IAAI,CAAC,CAAC;IAC7D,IAAI,KAAK,KAAK,iBAAiB;QAAE,OAAO,uBAAuB,CAAC,IAAI,CAAC,CAAC;IAEtE,MAAM,IAAI,aAAa,CAAC,cAAc,KAAK,EAAE,CAAC,CAAC;AACjD,CAAC;AAED,wBAAwB;AAExB,SAAS,SAAS;IAChB,OAAO;QACL,4BAA4B;QAC5B,EAAE;QACF,WAAW;QACX,8DAA8D;QAC9D,sDAAsD;QACtD,+DAA+D;QAC/D,qDAAqD;QACrD,6EAA6E;QAC7E,EAAE;QACF,UAAU;QACV,6BAA6B;QAC7B,+BAA+B;QAC/B,EAAE;QACF,gDAAgD;KACjD,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,aAAa;IACpB,OAAO;QACL,8BAA8B;QAC9B,EAAE;QACF,UAAU;QACV,8DAA8D;QAC9D,8DAA8D;QAC9D,qFAAqF;QACrF,2DAA2D;QAC3D,uEAAuE;QACvE,0FAA0F;QAC1F,kJAAkJ;KACnJ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,aAAa;IACpB,OAAO,CAAC,8BAA8B,EAAE,EAAE,EAAE,UAAU,EAAE,sDAAsD,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7H,CAAC;AAED,SAAS,kBAAkB;IACzB,OAAO;QACL,mCAAmC;QACnC,EAAE;QACF,UAAU;QACV,qDAAqD;QACrD,uCAAuC;QACvC,8DAA8D;QAC9D,4CAA4C;KAC7C,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,sBAAsB;IAC7B,OAAO;QACL,wBAAwB;QACxB,EAAE;QACF,wDAAwD;QACxD,yDAAyD;QACzD,oDAAoD;QACpD,iBAAiB;KAClB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACf,CAAC;AAED,SAAS,QAAQ,CAAC,KAAuE;IACvF,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,aAAa,EAAE,CAAC;IAC9C,IAAI,KAAK,KAAK,OAAO;QAAE,OAAO,aAAa,EAAE,CAAC;IAC9C,IAAI,KAAK,KAAK,YAAY;QAAE,OAAO,kBAAkB,EAAE,CAAC;IACxD,IAAI,KAAK,KAAK,iBAAiB;QAAE,OAAO,sBAAsB,EAAE,CAAC;IACjE,OAAO,SAAS,EAAE,CAAC;AACrB,CAAC;AAED,mDAAmD;AAEnD;;;;;;;GAOG;AACH,SAAS,yBAAyB;IAChC,OAAO,IAAI,CAAC,eAAe,EAAE,EAAE,iBAAiB,CAAC,CAAC;AACpD,CAAC;AAED,gFAAgF;AAChF,SAAS,mBAAmB,CAAC,GAAW;IACtC,IAAI,CAAC;QACH,OAAO,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,iBAAiB,EAAE,GAAG,CAAC,IAAI,MAAM,CAAC;IACzE,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,MAAM,CAAC;IAChB,CAAC;AACH,CAAC;AAED,iEAAiE;AACjE,MAAM,UAAU,WAAW,CAAC,MAAqC,EAAE,GAAW,EAAE,MAAc;IAC5F,MAAM,QAAQ,GAAG,mBAAmB,CAAC,GAAG,CAAC,CAAC;IAC1C,+CAA+C;IAC/C,qCAAqC;IACrC,qCAAqC;IACrC,MAAM,SAAS,GAAG,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IAC9E,MAAM,UAAU,GAAa,EAAE,CAAC;IAChC,IAAI,QAAQ,GAAG,CAAC,CAAC;IAEjB,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QAC3B,IAAI,KAAK,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;YAC1B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,IAAI,IAAI,CAAC,CAAC;YACxC,SAAS;QACX,CAAC;QACD,QAAQ,EAAE,CAAC;QACX,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC5B,SAAS,CAAC,MAAM,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QACzC,CAAC;QACD,MAAM,SAAS,GAAG,KAAK,CAAC,QAAQ,KAAK,WAAW,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC;QACjE,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,EAAE,UAAU,QAAQ,IAAI,SAAS,IAAI,QAAQ,IAAI,SAAS,EAAE,CAAC,CAAC;QAC1F,aAAa,CAAC,QAAQ,EAAE,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,QAAQ,CAAC,CAAC,CAAC;QAC3D,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC5B,CAAC;IAED,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC1B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,iBAAiB,UAAU,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,KAAK,QAAQ,EAAE,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACtG,CAAC;AACH,CAAC;AAED,eAAe;AAEf,KAAK,UAAU,QAAQ,CAAC,OAA0E;IAChG,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,MAAM,KAAK,GAAmB;YAC5B,GAAG,EAAE,OAAO,CAAC,GAAG;YAChB,GAAG,CAAC,OAAO,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7D,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACzE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACtE,GAAG,CAAC,OAAO,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7D,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC7E,GAAG,CAAC,OAAO,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAC9E,CAAC;QACF,MAAM,MAAM,GAAG,MAAM,eAAe,CAAC,KAAK,CAAC,CAAC;QAC5C,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,MAAM,IAAI,yBAAyB,EAAE,CAAC,CAAC;QAChF,OAAO;IACT,CAAC;IAED,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,MAAM,MAAM,GAAG,MAAM,aAAa,CAAC,OAAO,CAAC,GAAG,EAAE,UAAU,EAAE,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QAC5H,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,mBAAmB,CAAC,OAAO,CAAC,GAAG,EAAE,MAAM,CAAC,IAAI,CAAC,CAAC;QACtE,OAAO;IACT,CAAC;IAED,MAAM,KAAK,GAAwB;QACjC,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,QAAQ,EAAE,CAAC,OAAO,CAAC,YAAY;QAC/B,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAChE,GAAG,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACjE,CAAC;IACF,MAAM,MAAM,GAAG,MAAM,oBAAoB,CAAC,KAAK,CAAC,CAAC;IACjD,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,MAAM,IAAI,OAAO,CAAC,GAAG,EAAE,CAAC,CAAC;AACpE,CAAC;AAED,sDAAsD;AACtD,MAAM,CAAC,KAAK,UAAU,GAAG,CAAC,IAAc;IACtC,IAAI,OAAsB,CAAC;IAC3B,IAAI,CAAC;QACH,OAAO,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC/B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,aAAa,EAAE,CAAC;YACnC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,OAAO,OAAO,SAAS,EAAE,IAAI,CAAC,CAAC;YAC7D,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,KAAK,CAAC;IACd,CAAC;IAED,IAAI,OAAO,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;QAC5B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACrD,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,OAAO,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;QAC/B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,qBAAqB,EAAE,IAAI,CAAC,CAAC;QACrD,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,OAAO,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;QAC7B,4DAA4D;QAC5D,mCAAmC;QACnC,MAAM,SAAS,EAAE,CAAC;QAClB,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,CAAC;QACH,IAAI,OAAO,CAAC,IAAI,KAAK,iBAAiB,EAAE,CAAC;YACvC,OAAO,MAAM,cAAc,EAAE,CAAC;QAChC,CAAC;QACD,MAAM,QAAQ,CAAC,OAAO,CAAC,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,WAAW,EAAE,CAAC;YACjC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC,OAAO,IAAI,CAAC,CAAC;QAC7C,CAAC;aAAM,CAAC;YACN,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACvB,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;YAAS,CAAC;QACT,iDAAiD;QACjD,sDAAsD;QACtD,MAAM,YAAY,EAAE,CAAC;QACrB,KAAK,CAAC,KAAK,EAAE,CAAC;IAChB,CAAC;AACH,CAAC;AAED;;;;GAIG;AACH,SAAS,kBAAkB;IACzB,OAAO,OAAO,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,QAAQ,IAAI,aAAa,CAAC,OAAO,IAAI,CAAC,GAAG,CAAC,KAAK,YAAY,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;AACjH,CAAC;AAED,IAAI,kBAAkB,EAAE,EAAE,CAAC;IACzB,GAAG,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;SACvB,IAAI,CAAC,CAAC,QAAQ,EAAE,EAAE;QACjB,OAAO,CAAC,QAAQ,GAAG,QAAQ,CAAC;IAC9B,CAAC,CAAC;SACD,KAAK,CAAC,CAAC,KAAc,EAAE,EAAE;QACxB,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QACrB,OAAO,CAAC,QAAQ,GAAG,CAAC,CAAC;IACvB,CAAC,CAAC,CAAC;AACP,CAAC"}
|
package/dist/core.js
CHANGED
|
@@ -22,7 +22,7 @@ import { evaluateQuality } from "./extract/qualityScore.js";
|
|
|
22
22
|
import { closeBrowser } from "./fetcher/browser.js";
|
|
23
23
|
import { fetchPage } from "./fetcher/index.js";
|
|
24
24
|
import { assertHttpScheme, httpGetBinary } from "./fetcher/http.js";
|
|
25
|
-
import { buildScreenshotContent, formatDiffResponse, formatHandoffResponse, formatMarkdownResponse, formatOutlineResponse, formatPdfResponseFromCache, formatPdfTextResponse, formatUnchangedResponse, } from "./formatting.js";
|
|
25
|
+
import { buildScreenshotContent, formatDiffResponse, formatHandoffResponse, formatMarkdownResponse, formatOutlineResponse, formatPdfImageResponse, formatPdfResponseFromCache, formatPdfTextResponse, formatUnchangedResponse, } from "./formatting.js";
|
|
26
26
|
import { PolitenessManager } from "./politeness.js";
|
|
27
27
|
import { captureTiledScreenshot, DEFAULT_TILE_WIDTH } from "./screenshot.js";
|
|
28
28
|
import { computeBlockHashes, removeTemplateBlocks } from "./templateLearning.js";
|
|
@@ -105,6 +105,17 @@ export function registerCoreShutdownHandlers() {
|
|
|
105
105
|
const DEFAULT_MAX_TOKENS = 8000;
|
|
106
106
|
const DEFAULT_PAGE = 1;
|
|
107
107
|
const DEFAULT_SCREENSHOT_SCALE = 1.0;
|
|
108
|
+
/**
|
|
109
|
+
* そのURLがPDFとしてキャッシュ済みか(pdfPageCountはPDF経路だけが書くメタデータ)。
|
|
110
|
+
*
|
|
111
|
+
* 「解析中でまだキャッシュに現れていない間に届いた同一URLの要求が、HTML経路へ入って
|
|
112
|
+
* 取得をやり直す」という窓は無い。pdf.jsの解析はマクロタスクを一切譲らないため
|
|
113
|
+
* (1000ページのPDFの解析192msの間、5ms間隔のタイマーが0回発火)、後続要求が処理されるのは
|
|
114
|
+
* 解析が終わってキャッシュが書かれた後になる。
|
|
115
|
+
*/
|
|
116
|
+
function isCachedAsPdf(url) {
|
|
117
|
+
return typeof cache.get(url)?.metadata.pdfPageCount === "number";
|
|
118
|
+
}
|
|
108
119
|
/** URLのhostnameを安全に取り出す(不正URLならnull)。Phase 4テンプレート学習のドメインキーに使う。 */
|
|
109
120
|
function safeHostname(url) {
|
|
110
121
|
try {
|
|
@@ -127,9 +138,14 @@ async function fetchAndExtract(url, selector, conditionalHeaders, onProgress) {
|
|
|
127
138
|
// 確認する(guardedFetch内で着地先オリジンが変わった場合のみ呼ばれる。同一オリジン内
|
|
128
139
|
// リダイレクトや初回URLは呼び出し元のpoliteness.guardで確認済みのため二重チェックしない)。
|
|
129
140
|
const checkRobots = (targetUrl) => politeness.checkRobotsAllowed(targetUrl);
|
|
130
|
-
|
|
141
|
+
// content-typeでしかPDFと分からないURLは、プレビュー読み(256KB)で返してもらうと
|
|
142
|
+
// 専用経路が同じURLをもう一度全体取得することになる。この段階で全体を読み切ってもらい、
|
|
143
|
+
// 取得を1回で済ませる(handleFetchToolのhandoff分岐がこのバイト列をそのまま使う)。
|
|
144
|
+
const fullReadMaxBytes = (contentType, finalUrl) => looksLikePdf(finalUrl, contentType) ? DEFAULT_PDF_MAX_BYTES : null;
|
|
145
|
+
const baseOptions = { onProgress, checkRobots, fullReadMaxBytes };
|
|
146
|
+
const fetchResult = await fetchPage(url, conditionalHeaders ? { ...baseOptions, headers: conditionalHeaders } : baseOptions);
|
|
131
147
|
if ("notModified" in fetchResult) {
|
|
132
|
-
return { status: "not_modified" };
|
|
148
|
+
return { status: "not_modified", finalUrl: fetchResult.finalUrl };
|
|
133
149
|
}
|
|
134
150
|
// 機能B: HTML/PDF以外のコンテンツはハンドオフ応答(メタデータ+プレビュー+curl誘導)の対象。
|
|
135
151
|
// Markdown抽出・品質判定・キャッシュ保存は行わない。
|
|
@@ -214,6 +230,37 @@ function pageFromCacheEntry(url, cached, cacheStatus) {
|
|
|
214
230
|
* ため、outline→sectionの推奨フロー等でキャッシュヒットするたびに待機・robots再判定という
|
|
215
231
|
* 自己ペナルティが発生していた不具合の修正。selector指定時は必ずfetchするため常にguardする。
|
|
216
232
|
*/
|
|
233
|
+
/**
|
|
234
|
+
* 同じキーの処理を1本にまとめる。politenessの直列キューは取得の「開始間隔」を空けるだけで
|
|
235
|
+
* 取得自体は並行するため、同じURLへの同時呼び出しが先方へ2回出ていた。あわせて、
|
|
236
|
+
* 先に始まった取得が後から完了して新しいキャッシュを古い内容で上書きする競合も塞ぐ。
|
|
237
|
+
*
|
|
238
|
+
* 進捗通知は相乗りした呼び出しにも配る。まとめた先頭の呼び出しにprogressTokenが無く、
|
|
239
|
+
* 後から相乗りした呼び出しにだけある場合でも、通知が届かなくならないようにするため。
|
|
240
|
+
*/
|
|
241
|
+
class SharedOperations {
|
|
242
|
+
running = new Map();
|
|
243
|
+
run(key, onProgress, operation) {
|
|
244
|
+
const existing = this.running.get(key);
|
|
245
|
+
if (existing) {
|
|
246
|
+
if (onProgress)
|
|
247
|
+
existing.listeners.add(onProgress);
|
|
248
|
+
return existing.promise;
|
|
249
|
+
}
|
|
250
|
+
const listeners = new Set();
|
|
251
|
+
if (onProgress)
|
|
252
|
+
listeners.add(onProgress);
|
|
253
|
+
const notify = (message) => {
|
|
254
|
+
for (const listener of listeners)
|
|
255
|
+
listener(message);
|
|
256
|
+
};
|
|
257
|
+
const promise = operation(notify).finally(() => this.running.delete(key));
|
|
258
|
+
this.running.set(key, { promise, listeners });
|
|
259
|
+
return promise;
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
/** selector指定時はキャッシュを使わず結果もselector毎に変わるため、まとめる対象にしない。 */
|
|
263
|
+
const pageOperations = new SharedOperations();
|
|
217
264
|
async function resolvePage(url, selector, onProgress) {
|
|
218
265
|
if (selector) {
|
|
219
266
|
await politeness.guard(url, onProgress);
|
|
@@ -243,20 +290,34 @@ async function resolvePage(url, selector, onProgress) {
|
|
|
243
290
|
if (cached && cache.isFresh(cached)) {
|
|
244
291
|
return pageFromCacheEntry(url, cached, "fresh");
|
|
245
292
|
}
|
|
293
|
+
return pageOperations.run(url, onProgress, (notify) => fetchAndResolvePage(url, cached, notify));
|
|
294
|
+
}
|
|
295
|
+
/** resolvePageのうち、実際にサーバーへ取りにいく部分(同一URLでは1本にまとめられる)。 */
|
|
296
|
+
async function fetchAndResolvePage(url, cached, onProgress) {
|
|
246
297
|
// 条件付きGET(304)も含め実際にサーバーへ通信するため、ここでguardする
|
|
247
|
-
// (キャッシュfresh応答で早期returnした場合はguard
|
|
298
|
+
// (キャッシュfresh応答で早期returnした場合はguardしない=呼び出し元のコメント参照)。
|
|
248
299
|
await politeness.guard(url, onProgress);
|
|
249
300
|
const conditionalHeaders = {};
|
|
250
301
|
if (cached?.etag)
|
|
251
302
|
conditionalHeaders["If-None-Match"] = cached.etag;
|
|
252
303
|
if (cached?.lastModified)
|
|
253
304
|
conditionalHeaders["If-Modified-Since"] = cached.lastModified;
|
|
254
|
-
|
|
305
|
+
let result = await fetchAndExtract(url, undefined, Object.keys(conditionalHeaders).length > 0 ? conditionalHeaders : undefined, onProgress);
|
|
255
306
|
if (result.status === "not_modified") {
|
|
256
307
|
if (!cached)
|
|
257
308
|
throw new UnexpectedNotModifiedError(url);
|
|
258
|
-
|
|
259
|
-
|
|
309
|
+
// ETag/Last-Modifiedはリソース間で一意である必要がないため、リダイレクト先が
|
|
310
|
+
// 別のURLへ変わっていても304が返りうる。その304は手元のキャッシュとは別の表現に対する
|
|
311
|
+
// 応答なので、そのまま返すと古い本文を新しいURLの内容として渡してしまう。
|
|
312
|
+
const cachedFinalUrl = cached.metadata.finalUrl ?? url;
|
|
313
|
+
if (result.finalUrl === cachedFinalUrl) {
|
|
314
|
+
cache.touch(url);
|
|
315
|
+
return pageFromCacheEntry(url, cached, "revalidated");
|
|
316
|
+
}
|
|
317
|
+
await politeness.guard(url, onProgress);
|
|
318
|
+
result = await fetchAndExtract(url, undefined, undefined, onProgress);
|
|
319
|
+
if (result.status === "not_modified")
|
|
320
|
+
throw new UnexpectedNotModifiedError(url);
|
|
260
321
|
}
|
|
261
322
|
if (result.status === "handoff") {
|
|
262
323
|
return { kind: "handoff", handoff: result.handoff };
|
|
@@ -344,7 +405,11 @@ async function resolveScreenshot(url, options) {
|
|
|
344
405
|
// guardとは別枠。handleFetchTool冒頭の共通guardは撤去済み)。キャッシュfresh返却時は通らない。
|
|
345
406
|
await politeness.guard(url, options.onProgress);
|
|
346
407
|
options.onProgress?.("スクリーンショットを撮影しています…");
|
|
347
|
-
const captured = await captureTiledScreenshot(url,
|
|
408
|
+
const captured = await captureTiledScreenshot(url, {
|
|
409
|
+
...options,
|
|
410
|
+
// ブラウザ層が別オリジンへリダイレクトした場合もrobots.txtを確認する(HTTP層と同じ扱い)。
|
|
411
|
+
checkRobots: (targetUrl) => politeness.checkRobotsAllowed(targetUrl),
|
|
412
|
+
});
|
|
348
413
|
cache.setScreenshot({
|
|
349
414
|
cacheKey,
|
|
350
415
|
url: captured.finalUrl,
|
|
@@ -360,16 +425,54 @@ async function resolveScreenshot(url, options) {
|
|
|
360
425
|
truncated: captured.truncated,
|
|
361
426
|
};
|
|
362
427
|
}
|
|
363
|
-
|
|
428
|
+
/**
|
|
429
|
+
* 取得中のPDF。キーはURLのみにする。page/max_tokensは応答整形の差でしかなく、
|
|
430
|
+
* そこまでキーに含めると同じPDFを条件違いで別々に取得・解析し、同じキャッシュ行を
|
|
431
|
+
* 競合して更新することになるため。
|
|
432
|
+
*/
|
|
433
|
+
const pdfOperations = new SharedOperations();
|
|
364
434
|
/** PDFはキャッシュ層(§3-3等)を流用しつつ独立した経路で扱う。テキスト層があればMarkdown、無ければ画像タイル。 */
|
|
365
|
-
async function handlePdfFetch(url, page, maxTokens, onProgress) {
|
|
366
|
-
|
|
367
|
-
|
|
435
|
+
async function handlePdfFetch(url, page, maxTokens, onProgress, prefetched) {
|
|
436
|
+
if (!prefetched) {
|
|
437
|
+
const cached = cache.get(url);
|
|
438
|
+
if (cached && cache.isFresh(cached)) {
|
|
439
|
+
return [{ type: "text", text: formatPdfResponseFromCache(url, cached, maxTokens, page) }];
|
|
440
|
+
}
|
|
441
|
+
}
|
|
442
|
+
const document = await pdfOperations.run(url, onProgress, (notify) => loadPdfDocument(url, notify, prefetched));
|
|
443
|
+
if (document.kind === "cached") {
|
|
444
|
+
const cached = cache.get(url);
|
|
445
|
+
if (!cached)
|
|
446
|
+
throw new UnexpectedNotModifiedError(url);
|
|
447
|
+
// PDF応答フォーマットのcacheStatus型は"fresh"|"miss"のみのため、304再検証も
|
|
448
|
+
// "fresh"として表示する(HTML応答の"revalidated"相当だが、既存フォーマットは変更しない)。
|
|
368
449
|
return [{ type: "text", text: formatPdfResponseFromCache(url, cached, maxTokens, page) }];
|
|
369
450
|
}
|
|
451
|
+
if (document.kind === "text") {
|
|
452
|
+
const paginated = paginateMarkdown(document.markdown, maxTokens, page);
|
|
453
|
+
return [
|
|
454
|
+
{
|
|
455
|
+
type: "text",
|
|
456
|
+
text: formatPdfTextResponse(document.title, document.finalUrl, "miss", document.pageCount, paginated, document.extractedPageCount),
|
|
457
|
+
},
|
|
458
|
+
];
|
|
459
|
+
}
|
|
460
|
+
return [
|
|
461
|
+
{
|
|
462
|
+
type: "text",
|
|
463
|
+
text: formatPdfImageResponse(document.title, document.finalUrl, document.pageCount, document.extractedPageCount, document.images.length),
|
|
464
|
+
},
|
|
465
|
+
...document.images.map((image) => ({ type: "image", data: image.png.toString("base64"), mimeType: "image/png" })),
|
|
466
|
+
];
|
|
467
|
+
}
|
|
468
|
+
/** PDFの取得(必要なら)と解析。同一URLではこれが1本にまとめられる。 */
|
|
469
|
+
async function loadPdfDocument(url, notify, prefetched) {
|
|
470
|
+
if (prefetched)
|
|
471
|
+
return parsePdfDocument(url, prefetched, notify);
|
|
472
|
+
const cached = cache.get(url);
|
|
370
473
|
// レビュー指摘対応: キャッシュfresh応答時は実ネットワークアクセスが発生しないため
|
|
371
474
|
// guardしない(resolvePageと同じ方針)。実際にPDFを取得しに行く直前でのみguardする。
|
|
372
|
-
await politeness.guard(url,
|
|
475
|
+
await politeness.guard(url, notify);
|
|
373
476
|
// 低負荷原則対応: TTL失効後もキャッシュ(etag/last-modified)があれば条件付きGETで再検証する。
|
|
374
477
|
// HTML経路(resolvePage)は既にこの流儀で304時のフルDL+再パースを回避しているのに対し、
|
|
375
478
|
// PDF経路はTTL失効後は常にフルDL+全ページ再パースを行っており、20MB級の官公庁PDFで
|
|
@@ -379,48 +482,66 @@ async function handlePdfFetch(url, page, maxTokens, onProgress) {
|
|
|
379
482
|
conditionalHeaders["If-None-Match"] = cached.etag;
|
|
380
483
|
if (cached?.lastModified)
|
|
381
484
|
conditionalHeaders["If-Modified-Since"] = cached.lastModified;
|
|
382
|
-
const
|
|
485
|
+
const checkRobots = (targetUrl) => politeness.checkRobotsAllowed(targetUrl);
|
|
486
|
+
let binary = await httpGetBinary(url, {
|
|
383
487
|
maxBytes: DEFAULT_PDF_MAX_BYTES,
|
|
384
|
-
checkRobots
|
|
488
|
+
checkRobots,
|
|
385
489
|
...(Object.keys(conditionalHeaders).length > 0 ? { headers: conditionalHeaders } : {}),
|
|
386
490
|
});
|
|
387
491
|
if (binary.status === 304) {
|
|
388
492
|
if (!cached)
|
|
389
493
|
throw new UnexpectedNotModifiedError(url);
|
|
390
|
-
|
|
391
|
-
//
|
|
392
|
-
|
|
393
|
-
|
|
494
|
+
// HTML経路と同じ理由: リダイレクト先が変わっていても304は返りうるため、
|
|
495
|
+
// 着地先が一致するときだけキャッシュ済みの本文を使う(resolvePageのコメント参照)。
|
|
496
|
+
const cachedFinalUrl = cached.metadata.finalUrl ?? url;
|
|
497
|
+
if (binary.finalUrl === cachedFinalUrl) {
|
|
498
|
+
cache.touch(url);
|
|
499
|
+
return { kind: "cached" };
|
|
500
|
+
}
|
|
501
|
+
await politeness.guard(url, notify);
|
|
502
|
+
binary = await httpGetBinary(url, { maxBytes: DEFAULT_PDF_MAX_BYTES, checkRobots });
|
|
503
|
+
if (binary.status === 304)
|
|
504
|
+
throw new UnexpectedNotModifiedError(url);
|
|
394
505
|
}
|
|
395
|
-
|
|
396
|
-
|
|
506
|
+
return parsePdfDocument(url, { bytes: binary.bytes, finalUrl: binary.finalUrl, etag: binary.headers.get("etag"), lastModified: binary.headers.get("last-modified") }, notify);
|
|
507
|
+
}
|
|
508
|
+
/** 取得済みPDFバイト列を解析し、テキスト層があればキャッシュへ書き込む。 */
|
|
509
|
+
async function parsePdfDocument(url, source, notify) {
|
|
510
|
+
notify("PDFを解析しています…");
|
|
511
|
+
const textResult = await extractPdfText(source.bytes);
|
|
397
512
|
if (textResult.hasTextLayer) {
|
|
398
513
|
const markdown = markdownFromPdfText(textResult);
|
|
399
514
|
cache.set({
|
|
400
515
|
url,
|
|
401
|
-
etag:
|
|
402
|
-
lastModified:
|
|
516
|
+
etag: source.etag,
|
|
517
|
+
lastModified: source.lastModified,
|
|
403
518
|
markdown,
|
|
404
|
-
metadata: {
|
|
519
|
+
metadata: {
|
|
520
|
+
title: textResult.title,
|
|
521
|
+
finalUrl: source.finalUrl,
|
|
522
|
+
pdfPageCount: textResult.pageCount,
|
|
523
|
+
pdfExtractedPageCount: textResult.extractedPageCount,
|
|
524
|
+
},
|
|
405
525
|
});
|
|
406
|
-
|
|
407
|
-
|
|
526
|
+
return {
|
|
527
|
+
kind: "text",
|
|
528
|
+
title: textResult.title,
|
|
529
|
+
finalUrl: source.finalUrl,
|
|
530
|
+
markdown,
|
|
531
|
+
pageCount: textResult.pageCount,
|
|
532
|
+
extractedPageCount: textResult.extractedPageCount,
|
|
533
|
+
};
|
|
408
534
|
}
|
|
409
535
|
// テキスト層が実質無い(スキャンPDF): 先頭ページから画像タイルとして返す(キャッシュ非対象)
|
|
410
|
-
const images = await renderPdfPages(
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
].join("\n");
|
|
420
|
-
return [
|
|
421
|
-
{ type: "text", text: header },
|
|
422
|
-
...images.map((image) => ({ type: "image", data: image.png.toString("base64"), mimeType: "image/png" })),
|
|
423
|
-
];
|
|
536
|
+
const images = await renderPdfPages(source.bytes);
|
|
537
|
+
return {
|
|
538
|
+
kind: "images",
|
|
539
|
+
title: textResult.title,
|
|
540
|
+
finalUrl: source.finalUrl,
|
|
541
|
+
pageCount: textResult.pageCount,
|
|
542
|
+
extractedPageCount: textResult.extractedPageCount,
|
|
543
|
+
images,
|
|
544
|
+
};
|
|
424
545
|
}
|
|
425
546
|
// テスト用export(既存スタイルに合わせた最小のテスト可能化。politeness.guard呼び出し回数の検証用)。
|
|
426
547
|
export async function handleFetchTool(input) {
|
|
@@ -437,7 +558,9 @@ export async function handleFetchTool(input) {
|
|
|
437
558
|
// 各経路(resolvePage/handlePdfFetch/resolveScreenshot)の内部で、必要な箇所にのみ
|
|
438
559
|
// 個別に行う設計へ変更した(このファイル内の各所コメント参照)。
|
|
439
560
|
assertHttpScheme(input.url);
|
|
440
|
-
|
|
561
|
+
// URL拡張子に現れないPDF(`/download?id=123`型)は初回取得後にPDFとしてキャッシュされる。
|
|
562
|
+
// その記録も入口判定に使わないと、2回目以降だけHTMLページの応答形式で返ってしまう。
|
|
563
|
+
if (looksLikePdf(input.url, null) || isCachedAsPdf(input.url)) {
|
|
441
564
|
return handlePdfFetch(input.url, page, maxTokens, input.onProgress);
|
|
442
565
|
}
|
|
443
566
|
// 公開品質バグ修正: mode: screenshot(section未指定時)はMarkdown抽出結果を一切使わないため、
|
|
@@ -467,7 +590,13 @@ export async function handleFetchTool(input) {
|
|
|
467
590
|
if ("kind" in resolvedOrHandoff) {
|
|
468
591
|
const { handoff } = resolvedOrHandoff;
|
|
469
592
|
if (looksLikePdf(handoff.finalUrl, handoff.contentType)) {
|
|
470
|
-
|
|
593
|
+
// fullReadMaxBytesで本体を読み切っているため、ここで取得し直さない。
|
|
594
|
+
return handlePdfFetch(input.url, page, maxTokens, input.onProgress, {
|
|
595
|
+
bytes: handoff.bytes,
|
|
596
|
+
finalUrl: handoff.finalUrl,
|
|
597
|
+
etag: handoff.etag,
|
|
598
|
+
lastModified: handoff.lastModified,
|
|
599
|
+
});
|
|
471
600
|
}
|
|
472
601
|
return [{ type: "text", text: formatHandoffResponse(handoff, maxTokens) }];
|
|
473
602
|
}
|