@askalf/deepdive 0.1.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +177 -52
- package/dist/agent.d.ts +48 -2
- package/dist/agent.d.ts.map +1 -1
- package/dist/agent.js +178 -65
- package/dist/agent.js.map +1 -1
- package/dist/cache.d.ts +16 -0
- package/dist/cache.d.ts.map +1 -0
- package/dist/cache.js +62 -0
- package/dist/cache.js.map +1 -0
- package/dist/cli.d.ts +1 -0
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +164 -18
- package/dist/cli.js.map +1 -1
- package/dist/concurrency.d.ts +2 -0
- package/dist/concurrency.d.ts.map +1 -0
- package/dist/concurrency.js +38 -0
- package/dist/concurrency.js.map +1 -0
- package/dist/config.d.ts +20 -0
- package/dist/config.d.ts.map +1 -1
- package/dist/config.js +64 -3
- package/dist/config.js.map +1 -1
- package/dist/doctor.d.ts +44 -0
- package/dist/doctor.d.ts.map +1 -0
- package/dist/doctor.js +533 -0
- package/dist/doctor.js.map +1 -0
- package/dist/index.d.ts +9 -3
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +9 -3
- package/dist/index.js.map +1 -1
- package/dist/llm-stream.d.ts +27 -0
- package/dist/llm-stream.d.ts.map +1 -0
- package/dist/llm-stream.js +173 -0
- package/dist/llm-stream.js.map +1 -0
- package/dist/llm.d.ts +10 -0
- package/dist/llm.d.ts.map +1 -1
- package/dist/llm.js +96 -20
- package/dist/llm.js.map +1 -1
- package/dist/plan.d.ts +7 -0
- package/dist/plan.d.ts.map +1 -1
- package/dist/plan.js +51 -0
- package/dist/plan.js.map +1 -1
- package/dist/retry.d.ts +18 -0
- package/dist/retry.d.ts.map +1 -0
- package/dist/retry.js +70 -0
- package/dist/retry.js.map +1 -0
- package/dist/robots.d.ts +26 -0
- package/dist/robots.d.ts.map +1 -0
- package/dist/robots.js +183 -0
- package/dist/robots.js.map +1 -0
- package/dist/search/duckduckgo.d.ts +2 -0
- package/dist/search/duckduckgo.d.ts.map +1 -1
- package/dist/search/duckduckgo.js +38 -13
- package/dist/search/duckduckgo.js.map +1 -1
- package/dist/search/exa.d.ts +17 -0
- package/dist/search/exa.d.ts.map +1 -0
- package/dist/search/exa.js +62 -0
- package/dist/search/exa.js.map +1 -0
- package/dist/search/searxng.d.ts.map +1 -1
- package/dist/search/searxng.js +2 -1
- package/dist/search/searxng.js.map +1 -1
- package/dist/search.d.ts.map +1 -1
- package/dist/search.js +9 -1
- package/dist/search.js.map +1 -1
- package/dist/synthesize.d.ts +1 -1
- package/dist/synthesize.d.ts.map +1 -1
- package/dist/synthesize.js +11 -2
- package/dist/synthesize.js.map +1 -1
- package/dist/url-util.d.ts +4 -0
- package/dist/url-util.d.ts.map +1 -0
- package/dist/url-util.js +24 -0
- package/dist/url-util.js.map +1 -0
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -1,103 +1,223 @@
|
|
|
1
1
|
<p align="center">
|
|
2
2
|
<h1 align="center">deepdive</h1>
|
|
3
|
-
<p align="center"><strong>
|
|
3
|
+
<p align="center"><strong>Your machine. Your LLM subscription. Your search backend. Your cited report.</strong><br>A local research agent: ask a question, it plans sub-queries, searches the web, reads the pages in a real browser, iterates with a critic loop until the answer stops having gaps, and writes you a cited markdown report. Nothing leaves your laptop except the searches you run and the URLs the planner picked to read. Every LLM call routes through <a href="https://github.com/askalf/dario">dario</a> (or any Anthropic-compat endpoint), so the 30-to-60 calls a deep query burns bill against the Claude Max subscription you're already paying for — not a hosted tool stacking its margin on top of Anthropic's API pricing.</p>
|
|
4
4
|
</p>
|
|
5
5
|
|
|
6
6
|
<p align="center"><em>Zero hosted dependencies. MIT. Independent, unofficial, third-party — see <a href="DISCLAIMER.md">DISCLAIMER.md</a>.</em></p>
|
|
7
7
|
|
|
8
|
+
<p align="center">
|
|
9
|
+
<a href="https://www.npmjs.com/package/@askalf/deepdive"><img src="https://img.shields.io/npm/v/@askalf/deepdive?color=blue" alt="npm version"></a>
|
|
10
|
+
<a href="https://github.com/askalf/deepdive/actions/workflows/ci.yml"><img src="https://github.com/askalf/deepdive/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
|
11
|
+
<a href="https://github.com/askalf/deepdive/blob/master/LICENSE"><img src="https://img.shields.io/npm/l/@askalf/deepdive" alt="License"></a>
|
|
12
|
+
</p>
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## What you keep
|
|
17
|
+
|
|
18
|
+
Every hosted research tool — Perplexity, OpenAI Deep Research, Gemini Deep Research — solves a real problem: one question → plan → search → read → cited answer. They also quietly take four decisions away from you:
|
|
19
|
+
|
|
20
|
+
**Your data.** The question, the sub-queries the planner invents, every URL the agent chose to read — all of it goes to the vendor's servers. Often to their analytics pipeline. Sometimes to their ad-targeting pipeline. With deepdive, none of that exists. The planner runs in your Node process. The searches hit whichever backend you point at (DuckDuckGo by default, zero keys required; SearXNG, Brave, Tavily, or Exa if you'd rather). The only outbound connections from your machine are: your chosen LLM endpoint, your chosen search endpoint, and the specific URLs the planner decided to read. No telemetry, no analytics, no data retention. Inspectable: `lsof -i` during a run.
|
|
21
|
+
|
|
22
|
+
**Your model.** Hosted tools pick for you — Perplexity routes through their own blend, OpenAI uses GPT-5, Gemini uses 2.5 Pro. deepdive runs whatever model your endpoint exposes. Default is `claude-sonnet-4-6` for a good quality/cost balance; switch to `claude-opus-4-7` for reasoning-heavy questions; point `--base-url` at a LiteLLM or vLLM instance and run a local model. Same one-line flag either way.
|
|
23
|
+
|
|
24
|
+
**Your search backend.** Hosted tools use their own search index and won't tell you its exact shape. deepdive swaps between DuckDuckGo HTML (default, no key), self-hosted SearXNG, Brave Search API, Tavily, or Exa with one flag. Adding a new adapter is ~30 lines of TypeScript.
|
|
25
|
+
|
|
26
|
+
**Your depth.** Hosted tools cap how far the agent will dig because unbounded research eats their unit economics. deepdive's `--deep` flag keeps iterating with a critic LLM — review draft → name the gaps → search for them → re-synthesize — until the critic says the answer is complete or you hit `--deep=N` rounds. You decide where the ceiling is.
|
|
27
|
+
|
|
28
|
+
## What you stop paying for
|
|
29
|
+
|
|
30
|
+
Most people reading this already pay Anthropic for Claude Max ($100–200/mo). A hosted research tool asks for another $20/mo subscription — on top of a subscription you already have — so some vendor's servers can run LLM calls that your Max plan would have covered for free.
|
|
31
|
+
|
|
32
|
+
Here's the math for one deep query — a question that needs the critic loop to finish well, roughly 50k–200k tokens across planner + synthesis + critique + re-synthesis:
|
|
33
|
+
|
|
34
|
+
| How you run it | Per-query cost | Per-month cost at 10 queries | Data stays local? |
|
|
35
|
+
|---|---|---|---|
|
|
36
|
+
| Per-token API (`claude-opus-4-7`) | **~$2–$8** | **~$20–$80** | Your infra, your call |
|
|
37
|
+
| Per-token API (`claude-sonnet-4-6`) | **~$0.30–$1.20** | **~$3–$12** | Your infra, your call |
|
|
38
|
+
| Perplexity Pro | Capped depth, fixed tier | **$20/mo** | ❌ Perplexity + upstream |
|
|
39
|
+
| OpenAI Deep Research (ChatGPT Plus) | Capped usage, fixed tier | **$20/mo** | ❌ OpenAI + upstream |
|
|
40
|
+
| Gemini Deep Research (AI Advanced) | Capped usage, fixed tier | **$20/mo** | ❌ Google + upstream |
|
|
41
|
+
| **deepdive + dario + Claude Max** | **$0 per query** | **$0** (included in Max) | **✅ your machine** |
|
|
42
|
+
|
|
43
|
+
The cost-arbitrage argument is: the deep-research workload is *exactly* the shape Claude Max was priced for — 50k–200k tokens per question, sustained, bursty. Running it through a second subscription that marks up LLM calls on top of LLM calls is paying twice for something you already bought. dario unlocks the subscription; deepdive is the tool that uses it for this workload.
|
|
44
|
+
|
|
8
45
|
---
|
|
9
46
|
|
|
10
47
|
## 60 seconds
|
|
11
48
|
|
|
12
49
|
```bash
|
|
13
|
-
# 1.
|
|
14
|
-
#
|
|
15
|
-
dario proxy # http://localhost:3456, routes to Claude Max / OpenAI / etc.
|
|
50
|
+
# 1. Start dario (your local LLM router — https://github.com/askalf/dario).
|
|
51
|
+
dario proxy # http://localhost:3456, routes to Claude Max, OpenAI, etc.
|
|
16
52
|
|
|
17
53
|
# 2. Install deepdive.
|
|
18
54
|
npm install -g @askalf/deepdive
|
|
19
|
-
npx playwright install chromium
|
|
55
|
+
npx playwright install chromium # first run only, ~300 MB
|
|
20
56
|
|
|
21
57
|
# 3. Ask.
|
|
22
|
-
deepdive "how does claude's rate limiter work"
|
|
58
|
+
deepdive "how does claude's rate limiter work" --deep --verbose --out=report.md
|
|
23
59
|
```
|
|
24
60
|
|
|
25
|
-
|
|
61
|
+
`--deep` turns on the critic loop (2 extra rounds by default). `--verbose` streams every plan / search / fetch / critique step to stderr so you can watch the agent think. `--out` writes the cited markdown to a file in addition to stdout.
|
|
26
62
|
|
|
27
|
-
|
|
28
|
-
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## What the output looks like
|
|
66
|
+
|
|
67
|
+
A real deepdive run produces a file like this (excerpt):
|
|
68
|
+
|
|
69
|
+
```markdown
|
|
70
|
+
# how does claude's rate limiter work
|
|
71
|
+
|
|
72
|
+
Claude's rate limiter uses a two-tier rolling window: a **5-hour bucket**
|
|
73
|
+
and a **7-day bucket**, both scoped per OAuth session [1][2]. Requests
|
|
74
|
+
are billed against the short-term bucket first; when that window is
|
|
75
|
+
exhausted, the `representative-claim` response header switches from
|
|
76
|
+
`five_hour` to `seven_day` and Anthropic begins charging against the
|
|
77
|
+
longer bucket — still subscription billing, not API overage [3].
|
|
78
|
+
|
|
79
|
+
Exhausting both buckets triggers `overage` state, at which point per-token
|
|
80
|
+
Extra Usage charges apply if the account has enabled them; otherwise the
|
|
81
|
+
request gets a 429 [3][4]. The `anthropic-ratelimit-unified-*` response
|
|
82
|
+
headers expose the current utilization at request-response time so
|
|
83
|
+
clients can pool-balance intelligently [5].
|
|
84
|
+
|
|
85
|
+
...
|
|
86
|
+
|
|
87
|
+
## Sources
|
|
88
|
+
|
|
89
|
+
1. [Anthropic rate-limit headers docs](https://docs.anthropic.com/...) — fetched 2026-04-22
|
|
90
|
+
2. [Claude Code /usage command reference](https://code.claude.com/...) — fetched 2026-04-22
|
|
91
|
+
3. [Discussion #32 — seven_day claim explainer](https://github.com/...) — fetched 2026-04-22
|
|
92
|
+
4. [Anthropic overage billing documentation](https://docs.anthropic.com/...) — fetched 2026-04-22
|
|
93
|
+
5. [dario pool-mode implementation notes](https://github.com/...) — fetched 2026-04-22
|
|
29
94
|
```
|
|
30
95
|
|
|
31
|
-
|
|
32
|
-
1. **Plan.** LLM decomposes your question into 3–5 searchable sub-queries.
|
|
33
|
-
2. **Search.** DuckDuckGo HTML by default (no API key). Pluggable: `--search=searxng|brave|tavily` with your own endpoint or key.
|
|
34
|
-
3. **Fetch.** Playwright-driven Chromium renders each result page (JS-rendered SPAs included).
|
|
35
|
-
4. **Extract.** Boilerplate stripped, main content capped to a word budget.
|
|
36
|
-
5. **Synthesize.** LLM writes the answer with inline `[N]` citations referencing the source list.
|
|
96
|
+
Citations are numbered and inline. The source table at the end records the exact URL and fetch timestamp for every source, so you can verify any claim the model made.
|
|
37
97
|
|
|
38
98
|
---
|
|
39
99
|
|
|
40
|
-
##
|
|
100
|
+
## The `--deep` loop
|
|
101
|
+
|
|
102
|
+
Single-pass mode (no `--deep` flag) is what other local "research CLI" tools do: plan → search → fetch → synthesize → done. Good enough for simple factual lookups. Not great for "explain how X actually works."
|
|
103
|
+
|
|
104
|
+
`--deep` adds a critic:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
plan ──▶ search ──▶ fetch ──▶ synthesize ──▶ critic ─┐
|
|
108
|
+
│
|
|
109
|
+
◀───── "here's what's missing, try: q4, q5" ──┘
|
|
110
|
+
│
|
|
111
|
+
▼
|
|
112
|
+
search ──▶ fetch ──▶ synthesize ──▶ critic ──▶ ...
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The critic reads its own draft, flags gaps ("the draft didn't source the 429 header format"), and proposes up to three follow-up queries. The loop re-runs with those queries, re-synthesizes from every source gathered so far, and passes to the critic again. When the critic says the draft is complete — or when you hit `--deep=N` rounds — the loop stops.
|
|
41
116
|
|
|
42
|
-
|
|
117
|
+
Bare `--deep` = 2 extra rounds. `--deep=5` = up to 5. `--deep=0` is explicit single-pass.
|
|
43
118
|
|
|
44
|
-
|
|
119
|
+
**Why this is the whole point.** The critic loop is the axis hosted tools cap on. Per-query unit economics force them to ship a fixed depth — if they let you run a 5-round loop, some users would and their margins would collapse. On your own subscription, the only cap is the one you set on the command line.
|
|
45
120
|
|
|
46
121
|
---
|
|
47
122
|
|
|
48
|
-
##
|
|
123
|
+
## Common flags
|
|
49
124
|
|
|
50
|
-
deepdive
|
|
125
|
+
Run `deepdive --help` for the full list. The ones you'll reach for:
|
|
51
126
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
127
|
+
| Flag | Default | Why |
|
|
128
|
+
|---|---|---|
|
|
129
|
+
| `--deep[=<n>]` | off (bare = 2) | Turn on the critic loop. This is the headline feature. |
|
|
130
|
+
| `--model=<name>` | `claude-sonnet-4-6` | Try `claude-opus-4-7` on reasoning-heavy questions. |
|
|
131
|
+
| `--search=<adapter>` | `duckduckgo` | `searxng` for privacy, `brave` for quality, `tavily` or `exa` for research-tuned results. |
|
|
132
|
+
| `--max-sources=<n>` | `12` per round | Upper bound. Deep mode accumulates across rounds, capped each round. |
|
|
133
|
+
| `--concurrency=<n>` | `4` | Parallel fetches. Bump on a fast connection. |
|
|
134
|
+
| `--json` | markdown | Emit `{question, plan, rounds, sources, answer, usage}` for piping. |
|
|
135
|
+
| `--out=<path>` | — | Save to file. |
|
|
136
|
+
| `--verbose`, `-v` | — | Stream plan / search / fetch / critique events to stderr. |
|
|
55
137
|
|
|
56
|
-
|
|
138
|
+
Every flag mirrors a `DEEPDIVE_*` env var. CLI flags win over env.
|
|
57
139
|
|
|
58
140
|
---
|
|
59
141
|
|
|
60
142
|
## Search adapters
|
|
61
143
|
|
|
144
|
+
One adapter per backend. Default (DuckDuckGo) needs no key.
|
|
145
|
+
|
|
62
146
|
| Adapter | Flag | Needs | Notes |
|
|
63
147
|
|---|---|---|---|
|
|
64
|
-
| DuckDuckGo HTML | `--search=duckduckgo` (default) | nothing | Scrapes `html.duckduckgo.com`.
|
|
65
|
-
| SearXNG | `--search=searxng` | `DEEPDIVE_SEARXNG_URL` | Self-hosted metasearch. Best
|
|
66
|
-
| Brave Search | `--search=brave` | `DEEPDIVE_BRAVE_KEY` |
|
|
67
|
-
| Tavily | `--search=tavily` | `DEEPDIVE_TAVILY_KEY` | Research-tuned
|
|
148
|
+
| DuckDuckGo HTML | `--search=duckduckgo` (default) | nothing | Scrapes `html.duckduckgo.com`. Good enough for most questions. |
|
|
149
|
+
| SearXNG | `--search=searxng` | `DEEPDIVE_SEARXNG_URL` | Self-hosted metasearch. Best privacy. |
|
|
150
|
+
| Brave Search | `--search=brave` | `DEEPDIVE_BRAVE_KEY` | Paid, high quality. |
|
|
151
|
+
| Tavily | `--search=tavily` | `DEEPDIVE_TAVILY_KEY` | Research-tuned. Returns pre-extracted content; deepdive re-fetches anyway for consistency. |
|
|
152
|
+
| Exa | `--search=exa` | `DEEPDIVE_EXA_KEY` | Neural search tuned for long, intent-rich queries — a fit for the kind of sub-queries the planner and critic loop generate. Highlights only; deepdive re-fetches the page for full content. |
|
|
68
153
|
|
|
69
|
-
Adding a new adapter is ~30 lines
|
|
154
|
+
Adding a new adapter is ~30 lines: implement `SearchAdapter` in `src/search/*.ts`, register in `src/search.ts`.
|
|
70
155
|
|
|
71
156
|
---
|
|
72
157
|
|
|
73
|
-
##
|
|
158
|
+
## `deepdive doctor`
|
|
74
159
|
|
|
75
|
-
|
|
160
|
+
One command, aggregated health report. Paste the output when filing issues.
|
|
76
161
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
162
|
+
```bash
|
|
163
|
+
$ deepdive doctor
|
|
164
|
+
deepdive doctor — v0.3.0
|
|
165
|
+
|
|
166
|
+
# environment
|
|
167
|
+
OK Node v22.21.1
|
|
168
|
+
--- Platform win32 x64
|
|
169
|
+
--- deepdive v0.3.0
|
|
170
|
+
|
|
171
|
+
# cache
|
|
172
|
+
--- dir ~/.deepdive/cache
|
|
173
|
+
OK writable yes
|
|
174
|
+
--- entries 42 files · 18.3 MB
|
|
175
|
+
--- oldest 3h ago
|
|
176
|
+
|
|
177
|
+
# llm
|
|
178
|
+
--- base URL http://localhost:3456
|
|
179
|
+
--- model claude-sonnet-4-6
|
|
180
|
+
OK reachable 200 in 142ms
|
|
181
|
+
OK probe max_tokens=1 · in=3 out=1
|
|
182
|
+
|
|
183
|
+
# search
|
|
184
|
+
--- adapter duckduckgo
|
|
185
|
+
OK probe 4 results in 380ms
|
|
186
|
+
|
|
187
|
+
# browser
|
|
188
|
+
OK playwright module loaded
|
|
189
|
+
OK chromium launch + close in 244ms
|
|
190
|
+
|
|
191
|
+
Summary: 13 checks · 8 ok · 0 warn · 0 fail
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Exit code is 1 if anything's broken, 0 otherwise. `--json` for structured output.
|
|
195
|
+
|
|
196
|
+
---
|
|
197
|
+
|
|
198
|
+
## Caching
|
|
199
|
+
|
|
200
|
+
Every successful fetch goes to `~/.deepdive/cache/<sha256>.json` with a 1-hour TTL. A re-run of the same question — or a follow-up run that re-fetches overlapping URLs — never re-opens Chromium for sources it already has. Iteration during question refinement is free.
|
|
201
|
+
|
|
202
|
+
Disable with `--no-cache` or `DEEPDIVE_NO_CACHE=1`. Change the dir with `DEEPDIVE_CACHE_DIR`. Change the TTL with `--cache-ttl-ms` or `DEEPDIVE_CACHE_TTL_MS`.
|
|
91
203
|
|
|
92
204
|
---
|
|
93
205
|
|
|
94
206
|
## Library mode
|
|
95
207
|
|
|
96
208
|
```ts
|
|
97
|
-
import {
|
|
98
|
-
|
|
99
|
-
|
|
209
|
+
import {
|
|
210
|
+
runAgent,
|
|
211
|
+
resolveSearchAdapter,
|
|
212
|
+
resolveConfig,
|
|
213
|
+
createCache,
|
|
214
|
+
} from "@askalf/deepdive";
|
|
215
|
+
|
|
216
|
+
const config = resolveConfig({ deepRounds: 2 }, process.env);
|
|
100
217
|
const search = await resolveSearchAdapter(config.searchAdapter, process.env);
|
|
218
|
+
const cache = config.cache.enabled
|
|
219
|
+
? createCache({ dir: config.cache.dir, ttlMs: config.cache.ttlMs })
|
|
220
|
+
: undefined;
|
|
101
221
|
|
|
102
222
|
const result = await runAgent("how does claude's rate limiter work", {
|
|
103
223
|
llm: config.llm,
|
|
@@ -106,23 +226,28 @@ const result = await runAgent("how does claude's rate limiter work", {
|
|
|
106
226
|
resultsPerQuery: config.resultsPerQuery,
|
|
107
227
|
maxSources: config.maxSources,
|
|
108
228
|
maxWordsPerSource: config.maxWordsPerSource,
|
|
229
|
+
deepRounds: config.deepRounds,
|
|
230
|
+
concurrency: config.concurrency,
|
|
231
|
+
cache,
|
|
109
232
|
onEvent: (e) => console.error(e),
|
|
110
233
|
});
|
|
111
234
|
|
|
112
235
|
console.log(result.markdown);
|
|
236
|
+
console.log(`rounds: ${result.usage.rounds} · sources: ${result.usage.kept} · cache hits: ${result.usage.cacheHits}`);
|
|
113
237
|
```
|
|
114
238
|
|
|
239
|
+
All event types, the round-trace structure, and the browser-factory injection point are exported for programmatic use. See `src/index.ts`.
|
|
240
|
+
|
|
115
241
|
---
|
|
116
242
|
|
|
117
243
|
## Trust and transparency
|
|
118
244
|
|
|
119
245
|
| Signal | Status |
|
|
120
246
|
|---|---|
|
|
121
|
-
| **Source** | One TypeScript package, small enough to audit in an evening |
|
|
122
247
|
| **Runtime dependencies** | One — `playwright`. No hosted services, no telemetry. |
|
|
123
|
-
| **Credentials** | API keys live in env vars or CLI flags; deepdive never persists them |
|
|
124
|
-
| **Network scope** | LLM endpoint (your choice), search backend (your choice), and the actual URLs your
|
|
125
|
-
| **Telemetry** | None. Zero analytics, tracking, or data collection. |
|
|
248
|
+
| **Credentials** | API keys live in env vars or CLI flags; deepdive never persists them. Cache files store fetched page content only, never auth. |
|
|
249
|
+
| **Network scope** | LLM endpoint (your choice), search backend (your choice), and the actual URLs your planner picked to read. No other outbound traffic. Verify with `lsof -i` during a run. |
|
|
250
|
+
| **Telemetry** | None. Zero analytics, tracking, or data collection. Deliberately, not aspirationally. |
|
|
126
251
|
| **License** | MIT |
|
|
127
252
|
|
|
128
253
|
See [DISCLAIMER.md](DISCLAIMER.md) for the full AS IS / no-affiliation / user-responsibility terms.
|
package/dist/agent.d.ts
CHANGED
|
@@ -1,8 +1,15 @@
|
|
|
1
1
|
import type { LLMConfig } from "./llm.js";
|
|
2
2
|
import type { SearchAdapter } from "./search.js";
|
|
3
|
-
import { type Plan } from "./plan.js";
|
|
4
|
-
import { type BrowserOptions } from "./browser.js";
|
|
3
|
+
import { type Plan, type Critique } from "./plan.js";
|
|
4
|
+
import { type BrowserOptions, type FetchedPage } from "./browser.js";
|
|
5
5
|
import { type Source } from "./citations.js";
|
|
6
|
+
import type { PageCache } from "./cache.js";
|
|
7
|
+
import { type RobotsCache } from "./robots.js";
|
|
8
|
+
export interface BrowserLike {
|
|
9
|
+
start(): Promise<void>;
|
|
10
|
+
fetch(url: string): Promise<FetchedPage>;
|
|
11
|
+
close(): Promise<void>;
|
|
12
|
+
}
|
|
6
13
|
export interface AgentConfig {
|
|
7
14
|
llm: LLMConfig;
|
|
8
15
|
search: SearchAdapter;
|
|
@@ -10,7 +17,15 @@ export interface AgentConfig {
|
|
|
10
17
|
resultsPerQuery: number;
|
|
11
18
|
maxSources: number;
|
|
12
19
|
maxWordsPerSource: number;
|
|
20
|
+
deepRounds: number;
|
|
21
|
+
concurrency: number;
|
|
22
|
+
cache?: PageCache;
|
|
23
|
+
browserFactory?: (opts: BrowserOptions) => BrowserLike;
|
|
24
|
+
respectRobots?: boolean;
|
|
25
|
+
robotsUserAgent?: string;
|
|
26
|
+
robotsCache?: RobotsCache;
|
|
13
27
|
onEvent?: (event: AgentEvent) => void;
|
|
28
|
+
onSynthesizeToken?: (chunk: string, round: number) => void;
|
|
14
29
|
}
|
|
15
30
|
export type AgentEvent = {
|
|
16
31
|
type: "plan.start";
|
|
@@ -18,6 +33,10 @@ export type AgentEvent = {
|
|
|
18
33
|
} | {
|
|
19
34
|
type: "plan.done";
|
|
20
35
|
plan: Plan;
|
|
36
|
+
} | {
|
|
37
|
+
type: "round.start";
|
|
38
|
+
round: number;
|
|
39
|
+
queries: string[];
|
|
21
40
|
} | {
|
|
22
41
|
type: "search.start";
|
|
23
42
|
query: string;
|
|
@@ -28,27 +47,54 @@ export type AgentEvent = {
|
|
|
28
47
|
} | {
|
|
29
48
|
type: "fetch.start";
|
|
30
49
|
url: string;
|
|
50
|
+
cached: boolean;
|
|
31
51
|
} | {
|
|
32
52
|
type: "fetch.done";
|
|
33
53
|
url: string;
|
|
34
54
|
ok: boolean;
|
|
35
55
|
status: number;
|
|
36
56
|
words: number;
|
|
57
|
+
cached: boolean;
|
|
58
|
+
} | {
|
|
59
|
+
type: "fetch.skipped";
|
|
60
|
+
url: string;
|
|
61
|
+
reason: "robots";
|
|
37
62
|
} | {
|
|
38
63
|
type: "synthesize.start";
|
|
39
64
|
sourceCount: number;
|
|
65
|
+
round: number;
|
|
40
66
|
} | {
|
|
41
67
|
type: "synthesize.done";
|
|
68
|
+
round: number;
|
|
69
|
+
} | {
|
|
70
|
+
type: "critique.start";
|
|
71
|
+
round: number;
|
|
72
|
+
} | {
|
|
73
|
+
type: "critique.done";
|
|
74
|
+
round: number;
|
|
75
|
+
critique: Critique;
|
|
42
76
|
};
|
|
77
|
+
export interface RoundTrace {
|
|
78
|
+
round: number;
|
|
79
|
+
queries: string[];
|
|
80
|
+
candidatesFound: number;
|
|
81
|
+
fetched: number;
|
|
82
|
+
kept: number;
|
|
83
|
+
critique?: Critique;
|
|
84
|
+
}
|
|
43
85
|
export interface AgentResult {
|
|
44
86
|
question: string;
|
|
45
87
|
plan: Plan;
|
|
46
88
|
sources: Source[];
|
|
89
|
+
answer: string;
|
|
47
90
|
markdown: string;
|
|
91
|
+
rounds: RoundTrace[];
|
|
48
92
|
usage: {
|
|
49
93
|
queries: number;
|
|
50
94
|
fetched: number;
|
|
51
95
|
kept: number;
|
|
96
|
+
rounds: number;
|
|
97
|
+
cacheHits: number;
|
|
52
98
|
};
|
|
53
99
|
}
|
|
54
100
|
export declare function runAgent(question: string, config: AgentConfig, signal?: AbortSignal): Promise<AgentResult>;
|
package/dist/agent.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"agent.d.ts","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC1C,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,OAAO,EAAyB,KAAK,IAAI,EAAE,KAAK,QAAQ,EAAE,MAAM,WAAW,CAAC;AAC5E,OAAO,EAAkB,KAAK,cAAc,EAAE,KAAK,WAAW,EAAE,MAAM,cAAc,CAAC;AAErF,OAAO,EAA0C,KAAK,MAAM,EAAE,MAAM,gBAAgB,CAAC;AAErF,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AAE5C,OAAO,EAGL,KAAK,WAAW,EACjB,MAAM,aAAa,CAAC;AAIrB,MAAM,WAAW,WAAW;IAC1B,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;IACvB,KAAK,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IACzC,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CACxB;AAED,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,SAAS,CAAC;IACf,MAAM,EAAE,aAAa,CAAC;IACtB,OAAO,EAAE,cAAc,CAAC;IACxB,eAAe,EAAE,MAAM,CAAC;IACxB,UAAU,EAAE,MAAM,CAAC;IACnB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,UAAU,EAAE,MAAM,CAAC;IACnB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,CAAC,EAAE,SAAS,CAAC;IAClB,cAAc,CAAC,EAAE,CAAC,IAAI,EAAE,cAAc,KAAK,WAAW,CAAC;IAIvD,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,WAAW,CAAC,EAAE,WAAW,CAAC;IAC1B,OAAO,CAAC,EAAE,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAC;IAKtC,iBAAiB,CAAC,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,KAAK,IAAI,CAAC;CAC5D;AAED,MAAM,MAAM,UAAU,GAClB;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,QAAQ,EAAE,MAAM,CAAA;CAAE,GACxC;IAAE,IAAI,EAAE,WAAW,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GACjC;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,EAAE,CAAA;CAAE,GACzD;IAAE,IAAI,EAAE,cAAc,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACvC;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACrD;IAAE,IAAI,EAAE,aAAa,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,GACrD;IACE,IAAI,EAAE,YAAY,CAAC;IACnB,GAAG,EAAE,MAAM,CAAC;IACZ,EAAE,EAAE,OAAO,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,OAAO,CAAC;CACjB,GACD;IAAE,IAAI,EAAE,eAAe,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,QAAQ,CAAA;CAAE,GACxD;IAAE,IAAI,EAAE,kBAAkB,CAAC;IAAC,WAAW,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAChE;IAAE,IAAI,EAAE,iBAAiB,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAC1C;IAAE,IAAI,EAAE,gBAAgB,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACzC;IAAE,IAAI,EAAE,eAAe,CAAC;IAAC,KAAK,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,QAAQ,CAAA;CAAE,CAAC;AAEjE,MAAM,WAAW,UAAU;IACzB,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,eAAe,EAAE,MAAM,CAAC;IACxB,OAAO,EAAE,MAAM,CAAC;IAChB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB;AAED,MAAM,WAAW,WAAW;IAC1B,QAAQ,EAAE,MAAM,CAAC;IACjB,IAAI,EAAE,IAAI,CAAC;IACX,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,UAAU,EAAE,CAAC;IACrB,KAAK,EAAE;QACL,OAAO,EAAE,MAAM,CAAC;QAChB,OAAO,EAAE,MAAM,CAAC;QAChB,IAAI,EAAE,MAAM,CAAC;QACb,MAAM,EAAE,MAAM,CAAC;QACf,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;CACH;AAID,wBAAsB,QAAQ,CAC5B,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,WAAW,EACnB,MAAM,CAAC,EAAE,WAAW,GACnB,OAAO,CAAC,WAAW,CAAC,CA8JtB"}
|