@askalf/deepdive 0.1.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (72) hide show
  1. package/README.md +177 -52
  2. package/dist/agent.d.ts +48 -2
  3. package/dist/agent.d.ts.map +1 -1
  4. package/dist/agent.js +178 -65
  5. package/dist/agent.js.map +1 -1
  6. package/dist/cache.d.ts +16 -0
  7. package/dist/cache.d.ts.map +1 -0
  8. package/dist/cache.js +62 -0
  9. package/dist/cache.js.map +1 -0
  10. package/dist/cli.d.ts +1 -0
  11. package/dist/cli.d.ts.map +1 -1
  12. package/dist/cli.js +164 -18
  13. package/dist/cli.js.map +1 -1
  14. package/dist/concurrency.d.ts +2 -0
  15. package/dist/concurrency.d.ts.map +1 -0
  16. package/dist/concurrency.js +38 -0
  17. package/dist/concurrency.js.map +1 -0
  18. package/dist/config.d.ts +20 -0
  19. package/dist/config.d.ts.map +1 -1
  20. package/dist/config.js +64 -3
  21. package/dist/config.js.map +1 -1
  22. package/dist/doctor.d.ts +44 -0
  23. package/dist/doctor.d.ts.map +1 -0
  24. package/dist/doctor.js +533 -0
  25. package/dist/doctor.js.map +1 -0
  26. package/dist/index.d.ts +9 -3
  27. package/dist/index.d.ts.map +1 -1
  28. package/dist/index.js +9 -3
  29. package/dist/index.js.map +1 -1
  30. package/dist/llm-stream.d.ts +27 -0
  31. package/dist/llm-stream.d.ts.map +1 -0
  32. package/dist/llm-stream.js +173 -0
  33. package/dist/llm-stream.js.map +1 -0
  34. package/dist/llm.d.ts +10 -0
  35. package/dist/llm.d.ts.map +1 -1
  36. package/dist/llm.js +96 -20
  37. package/dist/llm.js.map +1 -1
  38. package/dist/plan.d.ts +7 -0
  39. package/dist/plan.d.ts.map +1 -1
  40. package/dist/plan.js +51 -0
  41. package/dist/plan.js.map +1 -1
  42. package/dist/retry.d.ts +18 -0
  43. package/dist/retry.d.ts.map +1 -0
  44. package/dist/retry.js +70 -0
  45. package/dist/retry.js.map +1 -0
  46. package/dist/robots.d.ts +26 -0
  47. package/dist/robots.d.ts.map +1 -0
  48. package/dist/robots.js +183 -0
  49. package/dist/robots.js.map +1 -0
  50. package/dist/search/duckduckgo.d.ts +2 -0
  51. package/dist/search/duckduckgo.d.ts.map +1 -1
  52. package/dist/search/duckduckgo.js +38 -13
  53. package/dist/search/duckduckgo.js.map +1 -1
  54. package/dist/search/exa.d.ts +17 -0
  55. package/dist/search/exa.d.ts.map +1 -0
  56. package/dist/search/exa.js +62 -0
  57. package/dist/search/exa.js.map +1 -0
  58. package/dist/search/searxng.d.ts.map +1 -1
  59. package/dist/search/searxng.js +2 -1
  60. package/dist/search/searxng.js.map +1 -1
  61. package/dist/search.d.ts.map +1 -1
  62. package/dist/search.js +9 -1
  63. package/dist/search.js.map +1 -1
  64. package/dist/synthesize.d.ts +1 -1
  65. package/dist/synthesize.d.ts.map +1 -1
  66. package/dist/synthesize.js +11 -2
  67. package/dist/synthesize.js.map +1 -1
  68. package/dist/url-util.d.ts +4 -0
  69. package/dist/url-util.d.ts.map +1 -0
  70. package/dist/url-util.js +24 -0
  71. package/dist/url-util.js.map +1 -0
  72. package/package.json +3 -2
package/README.md CHANGED
@@ -1,103 +1,223 @@
1
1
  <p align="center">
2
2
  <h1 align="center">deepdive</h1>
3
- <p align="center"><strong>A local research agent. One command, cited answer.</strong><br>Decomposes your question into sub-queries, runs web searches, fetches pages through a real headless browser, and hands everything to an LLM that writes a cited markdown report. Every LLM call goes through your own router default target is <a href="https://github.com/askalf/dario">dario</a> at <code>localhost:3456</code>, so synthesis runs on your Claude Max / Pro subscription, your own OpenAI key, or any local model. Any Anthropic-compat endpoint works.</p>
3
+ <p align="center"><strong>Your machine. Your LLM subscription. Your search backend. Your cited report.</strong><br>A local research agent: ask a question, it plans sub-queries, searches the web, reads the pages in a real browser, iterates with a critic loop until the answer stops having gaps, and writes you a cited markdown report. Nothing leaves your laptop except the searches you run and the URLs the planner picked to read. Every LLM call routes through <a href="https://github.com/askalf/dario">dario</a> (or any Anthropic-compat endpoint), so the 30-to-60 calls a deep query burns bill against the Claude Max subscription you're already paying for not a hosted tool stacking its margin on top of Anthropic's API pricing.</p>
4
4
  </p>
5
5
 
6
6
  <p align="center"><em>Zero hosted dependencies. MIT. Independent, unofficial, third-party — see <a href="DISCLAIMER.md">DISCLAIMER.md</a>.</em></p>
7
7
 
8
+ <p align="center">
9
+ <a href="https://www.npmjs.com/package/@askalf/deepdive"><img src="https://img.shields.io/npm/v/@askalf/deepdive?color=blue" alt="npm version"></a>
10
+ <a href="https://github.com/askalf/deepdive/actions/workflows/ci.yml"><img src="https://github.com/askalf/deepdive/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
11
+ <a href="https://github.com/askalf/deepdive/blob/master/LICENSE"><img src="https://img.shields.io/npm/l/@askalf/deepdive" alt="License"></a>
12
+ </p>
13
+
14
+ ---
15
+
16
+ ## What you keep
17
+
18
+ Every hosted research tool — Perplexity, OpenAI Deep Research, Gemini Deep Research — solves a real problem: one question → plan → search → read → cited answer. They also quietly take four decisions away from you:
19
+
20
+ **Your data.** The question, the sub-queries the planner invents, every URL the agent chose to read — all of it goes to the vendor's servers. Often to their analytics pipeline. Sometimes to their ad-targeting pipeline. With deepdive, none of that exists. The planner runs in your Node process. The searches hit whichever backend you point at (DuckDuckGo by default, zero keys required; SearXNG, Brave, Tavily, or Exa if you'd rather). The only outbound connections from your machine are: your chosen LLM endpoint, your chosen search endpoint, and the specific URLs the planner decided to read. No telemetry, no analytics, no data retention. Inspectable: `lsof -i` during a run.
21
+
22
+ **Your model.** Hosted tools pick for you — Perplexity routes through their own blend, OpenAI uses GPT-5, Gemini uses 2.5 Pro. deepdive runs whatever model your endpoint exposes. Default is `claude-sonnet-4-6` for a good quality/cost balance; switch to `claude-opus-4-7` for reasoning-heavy questions; point `--base-url` at a LiteLLM or vLLM instance and run a local model. Same one-line flag either way.
23
+
24
+ **Your search backend.** Hosted tools use their own search index and won't tell you its exact shape. deepdive swaps between DuckDuckGo HTML (default, no key), self-hosted SearXNG, Brave Search API, Tavily, or Exa with one flag. Adding a new adapter is ~30 lines of TypeScript.
25
+
26
+ **Your depth.** Hosted tools cap how far the agent will dig because unbounded research eats their unit economics. deepdive's `--deep` flag keeps iterating with a critic LLM — review draft → name the gaps → search for them → re-synthesize — until the critic says the answer is complete or you hit `--deep=N` rounds. You decide where the ceiling is.
27
+
28
+ ## What you stop paying for
29
+
30
+ Most people reading this already pay Anthropic for Claude Max ($100–200/mo). A hosted research tool asks for another $20/mo subscription — on top of a subscription you already have — so some vendor's servers can run LLM calls that your Max plan would have covered for free.
31
+
32
+ Here's the math for one deep query — a question that needs the critic loop to finish well, roughly 50k–200k tokens across planner + synthesis + critique + re-synthesis:
33
+
34
+ | How you run it | Per-query cost | Per-month cost at 10 queries | Data stays local? |
35
+ |---|---|---|---|
36
+ | Per-token API (`claude-opus-4-7`) | **~$2–$8** | **~$20–$80** | Your infra, your call |
37
+ | Per-token API (`claude-sonnet-4-6`) | **~$0.30–$1.20** | **~$3–$12** | Your infra, your call |
38
+ | Perplexity Pro | Capped depth, fixed tier | **$20/mo** | ❌ Perplexity + upstream |
39
+ | OpenAI Deep Research (ChatGPT Plus) | Capped usage, fixed tier | **$20/mo** | ❌ OpenAI + upstream |
40
+ | Gemini Deep Research (AI Advanced) | Capped usage, fixed tier | **$20/mo** | ❌ Google + upstream |
41
+ | **deepdive + dario + Claude Max** | **$0 per query** | **$0** (included in Max) | **✅ your machine** |
42
+
43
+ The cost-arbitrage argument is: the deep-research workload is *exactly* the shape Claude Max was priced for — 50k–200k tokens per question, sustained, bursty. Running it through a second subscription that marks up LLM calls on top of LLM calls is paying twice for something you already bought. dario unlocks the subscription; deepdive is the tool that uses it for this workload.
44
+
8
45
  ---
9
46
 
10
47
  ## 60 seconds
11
48
 
12
49
  ```bash
13
- # 1. Have dario running (or any Anthropic-compat endpoint at a local URL).
14
- # See: https://github.com/askalf/dario
15
- dario proxy # http://localhost:3456, routes to Claude Max / OpenAI / etc.
50
+ # 1. Start dario (your local LLM router https://github.com/askalf/dario).
51
+ dario proxy # http://localhost:3456, routes to Claude Max, OpenAI, etc.
16
52
 
17
53
  # 2. Install deepdive.
18
54
  npm install -g @askalf/deepdive
19
- npx playwright install chromium # first run only, ~300 MB
55
+ npx playwright install chromium # first run only, ~300 MB
20
56
 
21
57
  # 3. Ask.
22
- deepdive "how does claude's rate limiter work"
58
+ deepdive "how does claude's rate limiter work" --deep --verbose --out=report.md
23
59
  ```
24
60
 
25
- Output is cited markdown printed to stdout. Pipe it, save it with `--out=report.md`, or stream progress with `--verbose`:
61
+ `--deep` turns on the critic loop (2 extra rounds by default). `--verbose` streams every plan / search / fetch / critique step to stderr so you can watch the agent think. `--out` writes the cited markdown to a file in addition to stdout.
26
62
 
27
- ```bash
28
- deepdive "how does claude's rate limiter work" --verbose --out=rate-limiter.md
63
+ ---
64
+
65
+ ## What the output looks like
66
+
67
+ A real deepdive run produces a file like this (excerpt):
68
+
69
+ ```markdown
70
+ # how does claude's rate limiter work
71
+
72
+ Claude's rate limiter uses a two-tier rolling window: a **5-hour bucket**
73
+ and a **7-day bucket**, both scoped per OAuth session [1][2]. Requests
74
+ are billed against the short-term bucket first; when that window is
75
+ exhausted, the `representative-claim` response header switches from
76
+ `five_hour` to `seven_day` and Anthropic begins charging against the
77
+ longer bucket — still subscription billing, not API overage [3].
78
+
79
+ Exhausting both buckets triggers `overage` state, at which point per-token
80
+ Extra Usage charges apply if the account has enabled them; otherwise the
81
+ request gets a 429 [3][4]. The `anthropic-ratelimit-unified-*` response
82
+ headers expose the current utilization at request-response time so
83
+ clients can pool-balance intelligently [5].
84
+
85
+ ...
86
+
87
+ ## Sources
88
+
89
+ 1. [Anthropic rate-limit headers docs](https://docs.anthropic.com/...) — fetched 2026-04-22
90
+ 2. [Claude Code /usage command reference](https://code.claude.com/...) — fetched 2026-04-22
91
+ 3. [Discussion #32 — seven_day claim explainer](https://github.com/...) — fetched 2026-04-22
92
+ 4. [Anthropic overage billing documentation](https://docs.anthropic.com/...) — fetched 2026-04-22
93
+ 5. [dario pool-mode implementation notes](https://github.com/...) — fetched 2026-04-22
29
94
  ```
30
95
 
31
- Under the hood:
32
- 1. **Plan.** LLM decomposes your question into 3–5 searchable sub-queries.
33
- 2. **Search.** DuckDuckGo HTML by default (no API key). Pluggable: `--search=searxng|brave|tavily` with your own endpoint or key.
34
- 3. **Fetch.** Playwright-driven Chromium renders each result page (JS-rendered SPAs included).
35
- 4. **Extract.** Boilerplate stripped, main content capped to a word budget.
36
- 5. **Synthesize.** LLM writes the answer with inline `[N]` citations referencing the source list.
96
+ Citations are numbered and inline. The source table at the end records the exact URL and fetch timestamp for every source, so you can verify any claim the model made.
37
97
 
38
98
  ---
39
99
 
40
- ## Why this exists
100
+ ## The `--deep` loop
101
+
102
+ Single-pass mode (no `--deep` flag) is what other local "research CLI" tools do: plan → search → fetch → synthesize → done. Good enough for simple factual lookups. Not great for "explain how X actually works."
103
+
104
+ `--deep` adds a critic:
105
+
106
+ ```
107
+ plan ──▶ search ──▶ fetch ──▶ synthesize ──▶ critic ─┐
108
+
109
+ ◀───── "here's what's missing, try: q4, q5" ──┘
110
+
111
+
112
+ search ──▶ fetch ──▶ synthesize ──▶ critic ──▶ ...
113
+ ```
114
+
115
+ The critic reads its own draft, flags gaps ("the draft didn't source the 429 header format"), and proposes up to three follow-up queries. The loop re-runs with those queries, re-synthesizes from every source gathered so far, and passes to the critic again. When the critic says the draft is complete — or when you hit `--deep=N` rounds — the loop stops.
41
116
 
42
- Every hosted research tool (Perplexity, OpenAI Deep Research, Gemini Deep Research) sends your queries to someone else's server, charges per query, and gives you no say in which model synthesizes the answer or which sources get read. deepdive is the self-hosted alternative: your machine, your LLM subscription, your model choice, your search backend.
117
+ Bare `--deep` = 2 extra rounds. `--deep=5` = up to 5. `--deep=0` is explicit single-pass.
43
118
 
44
- Pair it with [dario](https://github.com/askalf/dario) and every research query routes through your Claude Max / Pro subscription instead of per-token API pricing — a single deep query can be 50k–200k tokens, which is exactly the workload subscription billing was built for.
119
+ **Why this is the whole point.** The critic loop is the axis hosted tools cap on. Per-query unit economics force them to ship a fixed depth if they let you run a 5-round loop, some users would and their margins would collapse. On your own subscription, the only cap is the one you set on the command line.
45
120
 
46
121
  ---
47
122
 
48
- ## LLM routing (default: dario)
123
+ ## Common flags
49
124
 
50
- deepdive speaks the Anthropic Messages API. The default target is `http://localhost:3456` (dario's default port), with `x-api-key: dario`:
125
+ Run `deepdive --help` for the full list. The ones you'll reach for:
51
126
 
52
- ```bash
53
- deepdive "…" --base-url=http://localhost:3456 --api-key=dario --model=claude-sonnet-4-6
54
- ```
127
+ | Flag | Default | Why |
128
+ |---|---|---|
129
+ | `--deep[=<n>]` | off (bare = 2) | Turn on the critic loop. This is the headline feature. |
130
+ | `--model=<name>` | `claude-sonnet-4-6` | Try `claude-opus-4-7` on reasoning-heavy questions. |
131
+ | `--search=<adapter>` | `duckduckgo` | `searxng` for privacy, `brave` for quality, `tavily` or `exa` for research-tuned results. |
132
+ | `--max-sources=<n>` | `12` per round | Upper bound. Deep mode accumulates across rounds, capped each round. |
133
+ | `--concurrency=<n>` | `4` | Parallel fetches. Bump on a fast connection. |
134
+ | `--json` | markdown | Emit `{question, plan, rounds, sources, answer, usage}` for piping. |
135
+ | `--out=<path>` | — | Save to file. |
136
+ | `--verbose`, `-v` | — | Stream plan / search / fetch / critique events to stderr. |
55
137
 
56
- Any Anthropic-compat endpoint works. If you already have Claude Code credentials and dario installed, there is nothing else to configure.
138
+ Every flag mirrors a `DEEPDIVE_*` env var. CLI flags win over env.
57
139
 
58
140
  ---
59
141
 
60
142
  ## Search adapters
61
143
 
144
+ One adapter per backend. Default (DuckDuckGo) needs no key.
145
+
62
146
  | Adapter | Flag | Needs | Notes |
63
147
  |---|---|---|---|
64
- | DuckDuckGo HTML | `--search=duckduckgo` (default) | nothing | Scrapes `html.duckduckgo.com`. No key, no account. Quality is fine for v1; if DDG changes their HTML, the parser may need an update. |
65
- | SearXNG | `--search=searxng` | `DEEPDIVE_SEARXNG_URL` | Self-hosted metasearch. Best quality/privacy combo. |
66
- | Brave Search | `--search=brave` | `DEEPDIVE_BRAVE_KEY` | Brave Search API. Paid, but high quality. |
67
- | Tavily | `--search=tavily` | `DEEPDIVE_TAVILY_KEY` | Research-tuned API. Returns pre-extracted content, which deepdive re-fetches for consistency. |
148
+ | DuckDuckGo HTML | `--search=duckduckgo` (default) | nothing | Scrapes `html.duckduckgo.com`. Good enough for most questions. |
149
+ | SearXNG | `--search=searxng` | `DEEPDIVE_SEARXNG_URL` | Self-hosted metasearch. Best privacy. |
150
+ | Brave Search | `--search=brave` | `DEEPDIVE_BRAVE_KEY` | Paid, high quality. |
151
+ | Tavily | `--search=tavily` | `DEEPDIVE_TAVILY_KEY` | Research-tuned. Returns pre-extracted content; deepdive re-fetches anyway for consistency. |
152
+ | Exa | `--search=exa` | `DEEPDIVE_EXA_KEY` | Neural search tuned for long, intent-rich queries — a fit for the kind of sub-queries the planner and critic loop generate. Highlights only; deepdive re-fetches the page for full content. |
68
153
 
69
- Adding a new adapter is ~30 lines implement `SearchAdapter` in `src/search/*.ts` and register in `src/search.ts`.
154
+ Adding a new adapter is ~30 lines: implement `SearchAdapter` in `src/search/*.ts`, register in `src/search.ts`.
70
155
 
71
156
  ---
72
157
 
73
- ## Flags
158
+ ## `deepdive doctor`
74
159
 
75
- Run `deepdive --help` for the full list. The ones you'll actually use:
160
+ One command, aggregated health report. Paste the output when filing issues.
76
161
 
77
- | Flag | Default | Description |
78
- |---|---|---|
79
- | `--base-url=<url>` | `http://localhost:3456` | LLM endpoint (dario / Anthropic / any compat URL) |
80
- | `--api-key=<key>` | `dario` | LLM key |
81
- | `--model=<name>` | `claude-sonnet-4-6` | Model for both planning and synthesis |
82
- | `--search=<adapter>` | `duckduckgo` | Search backend (see table above) |
83
- | `--max-sources=<n>` | `12` | Total pages fetched per query |
84
- | `--results-per-query=<n>` | `5` | Candidates pulled per sub-query |
85
- | `--max-words-per-source=<n>` | `2000` | Per-source content cap before synthesis |
86
- | `--timeout-ms=<ms>` | `30000` | Per-fetch timeout |
87
- | `--out=<path>` | — | Also write markdown to file |
88
- | `--verbose`, `-v` | — | Stream progress events to stderr |
89
-
90
- All flags mirror to `DEEPDIVE_*` env vars (e.g. `DEEPDIVE_MODEL`, `DEEPDIVE_MAX_SOURCES`). CLI flags win over env vars.
162
+ ```bash
163
+ $ deepdive doctor
164
+ deepdive doctor v0.3.0
165
+
166
+ # environment
167
+ OK Node v22.21.1
168
+ --- Platform win32 x64
169
+ --- deepdive v0.3.0
170
+
171
+ # cache
172
+ --- dir ~/.deepdive/cache
173
+ OK writable yes
174
+ --- entries 42 files · 18.3 MB
175
+ --- oldest 3h ago
176
+
177
+ # llm
178
+ --- base URL http://localhost:3456
179
+ --- model claude-sonnet-4-6
180
+ OK reachable 200 in 142ms
181
+ OK probe max_tokens=1 · in=3 out=1
182
+
183
+ # search
184
+ --- adapter duckduckgo
185
+ OK probe 4 results in 380ms
186
+
187
+ # browser
188
+ OK playwright module loaded
189
+ OK chromium launch + close in 244ms
190
+
191
+ Summary: 13 checks · 8 ok · 0 warn · 0 fail
192
+ ```
193
+
194
+ Exit code is 1 if anything's broken, 0 otherwise. `--json` for structured output.
195
+
196
+ ---
197
+
198
+ ## Caching
199
+
200
+ Every successful fetch goes to `~/.deepdive/cache/<sha256>.json` with a 1-hour TTL. A re-run of the same question — or a follow-up run that re-fetches overlapping URLs — never re-opens Chromium for sources it already has. Iteration during question refinement is free.
201
+
202
+ Disable with `--no-cache` or `DEEPDIVE_NO_CACHE=1`. Change the dir with `DEEPDIVE_CACHE_DIR`. Change the TTL with `--cache-ttl-ms` or `DEEPDIVE_CACHE_TTL_MS`.
91
203
 
92
204
  ---
93
205
 
94
206
  ## Library mode
95
207
 
96
208
  ```ts
97
- import { runAgent, resolveSearchAdapter, resolveConfig } from "@askalf/deepdive";
98
-
99
- const config = resolveConfig({}, process.env);
209
+ import {
210
+ runAgent,
211
+ resolveSearchAdapter,
212
+ resolveConfig,
213
+ createCache,
214
+ } from "@askalf/deepdive";
215
+
216
+ const config = resolveConfig({ deepRounds: 2 }, process.env);
100
217
  const search = await resolveSearchAdapter(config.searchAdapter, process.env);
218
+ const cache = config.cache.enabled
219
+ ? createCache({ dir: config.cache.dir, ttlMs: config.cache.ttlMs })
220
+ : undefined;
101
221
 
102
222
  const result = await runAgent("how does claude's rate limiter work", {
103
223
  llm: config.llm,
@@ -106,23 +226,28 @@ const result = await runAgent("how does claude's rate limiter work", {
106
226
  resultsPerQuery: config.resultsPerQuery,
107
227
  maxSources: config.maxSources,
108
228
  maxWordsPerSource: config.maxWordsPerSource,
229
+ deepRounds: config.deepRounds,
230
+ concurrency: config.concurrency,
231
+ cache,
109
232
  onEvent: (e) => console.error(e),
110
233
  });
111
234
 
112
235
  console.log(result.markdown);
236
+ console.log(`rounds: ${result.usage.rounds} · sources: ${result.usage.kept} · cache hits: ${result.usage.cacheHits}`);
113
237
  ```
114
238
 
239
+ All event types, the round-trace structure, and the browser-factory injection point are exported for programmatic use. See `src/index.ts`.
240
+
115
241
  ---
116
242
 
117
243
  ## Trust and transparency
118
244
 
119
245
  | Signal | Status |
120
246
  |---|---|
121
- | **Source** | One TypeScript package, small enough to audit in an evening |
122
247
  | **Runtime dependencies** | One — `playwright`. No hosted services, no telemetry. |
123
- | **Credentials** | API keys live in env vars or CLI flags; deepdive never persists them |
124
- | **Network scope** | LLM endpoint (your choice), search backend (your choice), and the actual URLs your LLM picked to read |
125
- | **Telemetry** | None. Zero analytics, tracking, or data collection. |
248
+ | **Credentials** | API keys live in env vars or CLI flags; deepdive never persists them. Cache files store fetched page content only, never auth. |
249
+ | **Network scope** | LLM endpoint (your choice), search backend (your choice), and the actual URLs your planner picked to read. No other outbound traffic. Verify with `lsof -i` during a run. |
250
+ | **Telemetry** | None. Zero analytics, tracking, or data collection. Deliberately, not aspirationally. |
126
251
  | **License** | MIT |
127
252
 
128
253
  See [DISCLAIMER.md](DISCLAIMER.md) for the full AS IS / no-affiliation / user-responsibility terms.
package/dist/agent.d.ts CHANGED
@@ -1,8 +1,15 @@
1
1
  import type { LLMConfig } from "./llm.js";
2
2
  import type { SearchAdapter } from "./search.js";
3
- import { type Plan } from "./plan.js";
4
- import { type BrowserOptions } from "./browser.js";
3
+ import { type Plan, type Critique } from "./plan.js";
4
+ import { type BrowserOptions, type FetchedPage } from "./browser.js";
5
5
  import { type Source } from "./citations.js";
6
+ import type { PageCache } from "./cache.js";
7
+ import { type RobotsCache } from "./robots.js";
8
+ export interface BrowserLike {
9
+ start(): Promise<void>;
10
+ fetch(url: string): Promise<FetchedPage>;
11
+ close(): Promise<void>;
12
+ }
6
13
  export interface AgentConfig {
7
14
  llm: LLMConfig;
8
15
  search: SearchAdapter;
@@ -10,7 +17,15 @@ export interface AgentConfig {
10
17
  resultsPerQuery: number;
11
18
  maxSources: number;
12
19
  maxWordsPerSource: number;
20
+ deepRounds: number;
21
+ concurrency: number;
22
+ cache?: PageCache;
23
+ browserFactory?: (opts: BrowserOptions) => BrowserLike;
24
+ respectRobots?: boolean;
25
+ robotsUserAgent?: string;
26
+ robotsCache?: RobotsCache;
13
27
  onEvent?: (event: AgentEvent) => void;
28
+ onSynthesizeToken?: (chunk: string, round: number) => void;
14
29
  }
15
30
  export type AgentEvent = {
16
31
  type: "plan.start";
@@ -18,6 +33,10 @@ export type AgentEvent = {
18
33
  } | {
19
34
  type: "plan.done";
20
35
  plan: Plan;
36
+ } | {
37
+ type: "round.start";
38
+ round: number;
39
+ queries: string[];
21
40
  } | {
22
41
  type: "search.start";
23
42
  query: string;
@@ -28,27 +47,54 @@ export type AgentEvent = {
28
47
  } | {
29
48
  type: "fetch.start";
30
49
  url: string;
50
+ cached: boolean;
31
51
  } | {
32
52
  type: "fetch.done";
33
53
  url: string;
34
54
  ok: boolean;
35
55
  status: number;
36
56
  words: number;
57
+ cached: boolean;
58
+ } | {
59
+ type: "fetch.skipped";
60
+ url: string;
61
+ reason: "robots";
37
62
  } | {
38
63
  type: "synthesize.start";
39
64
  sourceCount: number;
65
+ round: number;
40
66
  } | {
41
67
  type: "synthesize.done";
68
+ round: number;
69
+ } | {
70
+ type: "critique.start";
71
+ round: number;
72
+ } | {
73
+ type: "critique.done";
74
+ round: number;
75
+ critique: Critique;
42
76
  };
77
+ export interface RoundTrace {
78
+ round: number;
79
+ queries: string[];
80
+ candidatesFound: number;
81
+ fetched: number;
82
+ kept: number;
83
+ critique?: Critique;
84
+ }
43
85
  export interface AgentResult {
44
86
  question: string;
45
87
  plan: Plan;
46
88
  sources: Source[];
89
+ answer: string;
47
90
  markdown: string;
91
+ rounds: RoundTrace[];
48
92
  usage: {
49
93
  queries: number;
50
94
  fetched: number;
51
95
  kept: number;
96
+ rounds: number;
97
+ cacheHits: number;
52
98
  };
53
99
  }
54
100
  export declare function runAgent(question: string, config: AgentConfig, signal?: AbortSignal): Promise<AgentResult>;
@@ -1 +1 @@
1
- {"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"AAMA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC1C,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,OAAO,EAAe,KAAK,IAAI,EAAE,MAAM,WAAW,CAAC;AACnD,OAAO,EAAkB,KAAK,cAAc,EAAoB,MAAM,cAAc,CAAC;AAErF,OAAO,EAA0C,KAAK,MAAM,EAAE,MAAM,gBAAgB,CAAC;AAGrF,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,SAAS,CAAC;IACf,MAAM,EAAE,aAAa,CAAC;IACtB,OAAO,EAAE,cAAc,CAAC;IACxB,eAAe,EAAE,MAAM,CAAC;IACxB,UAAU,EAAE,MAAM,CAAC;IACnB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAC;CACvC;AAED,MAAM,MAAM,UAAU,GAClB;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,QAAQ,EAAE,MAAM,CAAA;CAAE,GACxC;IAAE,IAAI,EAAE,WAAW,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GACjC;IAAE,IAAI,EAAE,cAAc,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACvC;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACrD;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,GAAG,EAAE,MAAM,CAAA;CAAE,GACpC;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,OAAO,CAAC;IAAC,MAAM,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAC/E;IAAE,IAAI,EAAE,kBAAkB,CAAC;IAAC,WAAW,EAAE,MAAM,CAAA;CAAE,GACjD;IAAE,IAAI,EAAE,iBAAiB,CAAA;CAAE,CAAC;AAEhC,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAC;IACjB,IAAI,EAAE,IAAI,CAAC;IACX,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;CAC3D;AAED,wBAAsB,QAAQ,CAC5B,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,WAAW,EACnB,MAAM,CAAC,EAAE,WAAW,GACnB,OAAO,CAAC,WAAW,CAAC,CA2FtB"}
1
+ {"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC1C,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,OAAO,EAAyB,KAAK,IAAI,EAAE,KAAK,QAAQ,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAkB,KAAK,cAAc,EAAE,KAAK,WAAW,EAAE,MAAM,cAAc,CAAC;AAErF,OAAO,EAA0C,KAAK,MAAM,EAAE,MAAM,gBAAgB,CAAC;AAErF,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AAE5C,OAAO,EAGL,KAAK,WAAW,EACjB,MAAM,aAAa,CAAC;AAIrB,MAAM,WAAW,WAAW;IAC1B,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;IACvB,KAAK,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IACzC,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CACxB;AAED,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,SAAS,CAAC;IACf,MAAM,EAAE,aAAa,CAAC;IACtB,OAAO,EAAE,cAAc,CAAC;IACxB,eAAe,EAAE,MAAM,CAAC;IACxB,UAAU,EAAE,MAAM,CAAC;IACnB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,UAAU,EAAE,MAAM,CAAC;IACnB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,CAAC,EAAE,SAAS,CAAC;IAClB,cAAc,CAAC,EAAE,CAAC,IAAI,EAAE,cAAc,KAAK,WAAW,CAAC;IAIvD,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,WAAW,CAAC,EAAE,WAAW,CAAC;IAC1B,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAC;IAKtC,iBAAiB,CAAC,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,KAAK,IAAI,CAAC;CAC5D;AAED,MAAM,MAAM,UAAU,GAClB;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,QAAQ,EAAE,MAAM,CAAA;CAAE,GACxC;IAAE,IAAI,EAAE,WAAW,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GACjC;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,EAAE,CAAA;CAAE,GACzD;IAAE,IAAI,EAAE,cAAc,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACvC;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACrD;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,GACrD;IACE,IAAI,EAAE,YAAY,CAAC;IACnB,GAAG,EAAE,MAAM,CAAC;IACZ,EAAE,EAAE,OAAO,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,OAAO,CAAC;CACjB,GACD;IAAE,IAAI,EAAE,eAAe,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,QAAQ,CAAA;CAAE,GACxD;IAAE,IAAI,EAAE,kBAAkB,CAAC;IAAC,WAAW,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAChE;IAAE,IAAI,EAAE,iBAAiB,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAC1C;IAAE,IAAI,EAAE,gBAAgB,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACzC;IAAE,IAAI,EAAE,eAAe,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,QAAQ,CAAA;CAAE,CAAC;AAEjE,MAAM,WAAW,UAAU;IACzB,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,eAAe,EAAE,MAAM,CAAC;IACxB,OAAO,EAAE,MAAM,CAAC;IAChB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB;AAED,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAC;IACjB,IAAI,EAAE,IAAI,CAAC;IACX,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,UAAU,EAAE,CAAC;IACrB,KAAK,EAAE;QACL,OAAO,EAAE,MAAM,CAAC;QAChB,OAAO,EAAE,MAAM,CAAC;QAChB,IAAI,EAAE,MAAM,CAAC;QACb,MAAM,EAAE,MAAM,CAAC;QACf,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;CACH;AAID,wBAAsB,QAAQ,CAC5B,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,WAAW,EACnB,MAAM,CAAC,EAAE,WAAW,GACnB,OAAO,CAAC,WAAW,CAAC,CA8JtB"}