@mdream/crawl 2.0.0-beta.1 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +194 -88
- package/dist/_chunks/crawl.mjs +272 -36
- package/dist/cli.mjs +328 -358
- package/dist/index.d.mts +64 -59
- package/package.json +15 -5
package/README.md
CHANGED
|
@@ -2,16 +2,21 @@
|
|
|
2
2
|
|
|
3
3
|
Multi-page website crawler that generates [llms.txt](https://llmstxt.org/) files. Follows internal links and converts HTML to Markdown using [mdream](../mdream).
|
|
4
4
|
|
|
5
|
+
Upgrading from v1? Read [Migrating from v1](#migrating-from-v1).
|
|
6
|
+
|
|
5
7
|
## Setup
|
|
6
8
|
|
|
7
9
|
```bash
|
|
8
10
|
npm install @mdream/crawl
|
|
9
11
|
```
|
|
10
12
|
|
|
13
|
+
After a global install (`npm install -g @mdream/crawl`), run the `mdream-crawl` command. With `npx`, use the package name: `npx @mdream/crawl`.
|
|
14
|
+
|
|
11
15
|
For JavaScript-heavy sites that require browser rendering, install the optional Playwright dependencies:
|
|
12
16
|
|
|
13
17
|
```bash
|
|
14
18
|
npm install crawlee playwright
|
|
19
|
+
npx playwright install chromium
|
|
15
20
|
```
|
|
16
21
|
|
|
17
22
|
## CLI Usage
|
|
@@ -34,26 +39,35 @@ npx @mdream/crawl -u https://docs.example.com
|
|
|
34
39
|
|
|
35
40
|
### CLI Options
|
|
36
41
|
|
|
42
|
+
Most flags set the [option](#crawloptions) with the same name in camelCase. For example, `--max-pages` sets `maxPages`.
|
|
43
|
+
|
|
37
44
|
| Flag | Alias | Description | Default |
|
|
38
45
|
|------|-------|-------------|---------|
|
|
39
|
-
| `--url <url>` | `-u` |
|
|
46
|
+
| `--url <url>` | `-u` | URL to crawl. Accepts glob patterns. Repeatable. You can also pass URLs without a flag | Required |
|
|
40
47
|
| `--output <dir>` | `-o` | Output directory | `output` |
|
|
41
|
-
| `--depth <number>` | `-d` |
|
|
42
|
-
| `--single-page` | |
|
|
48
|
+
| `--depth <number>` | `-d` | Link depth from 0 to 10. See [Depth](#depth) | `3` |
|
|
49
|
+
| `--single-page` | | Same as `--depth 0`. It takes precedence over `--depth` | |
|
|
43
50
|
| `--driver <type>` | | Crawler driver: `http` or `playwright` | `http` |
|
|
44
|
-
| `--artifacts <list>` | | Comma-separated
|
|
45
|
-
| `--origin <url>` | | Origin
|
|
46
|
-
| `--site-name <name>` | |
|
|
47
|
-
| `--description <
|
|
48
|
-
| `--max-pages <number>` | | Maximum pages to
|
|
49
|
-
| `--crawl-delay <seconds>` | | Delay between requests in seconds |
|
|
50
|
-
| `--exclude <pattern>` | |
|
|
51
|
+
| `--artifacts <list>` | | Comma-separated list of `llms.txt`, `llms-full.txt`, `markdown` | all three |
|
|
52
|
+
| `--origin <url>` | | Origin for relative links in page Markdown | the origin of each page |
|
|
53
|
+
| `--site-name <name>` | | Site name in llms.txt | the home page title |
|
|
54
|
+
| `--description <text>` | | Site description in llms.txt | the home page description |
|
|
55
|
+
| `--max-pages <number>` | | Maximum number of pages to fetch | no limit |
|
|
56
|
+
| `--crawl-delay <seconds>` | | Delay between requests in seconds | the `robots.txt` `Crawl-delay`, or none |
|
|
57
|
+
| `--exclude <pattern>` | | Skip URLs that match a glob pattern. Repeatable | none |
|
|
51
58
|
| `--skip-sitemap` | | Skip `sitemap.xml` and `robots.txt` discovery | `false` |
|
|
52
|
-
| `--sitemap <url>` | | Use
|
|
53
|
-
| `--allow-subdomains` | | Crawl
|
|
54
|
-
| `--
|
|
55
|
-
| `--
|
|
56
|
-
| `--
|
|
59
|
+
| `--sitemap <url>` | | Use this sitemap instead of auto-discovery. Repeatable for multi-part sitemaps | auto-discovered |
|
|
60
|
+
| `--allow-subdomains` | | Crawl other subdomains of the same root domain | `false` |
|
|
61
|
+
| `--keep-boilerplate` | | Keep repeated navigation and footers in page Markdown. Sets `stripBoilerplate: false` | `false` |
|
|
62
|
+
| `--boilerplate-threshold <n>` | | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 | `0.5` |
|
|
63
|
+
| `--verbose` | `-v` | Log details for each failed URL | `false` |
|
|
64
|
+
| `--quiet` | `-q`, `--silent` | Drop all logs. Use it to keep stdout clean for JSON-RPC or MCP | `false` |
|
|
65
|
+
| `--help` | `-h` | Show the help message | |
|
|
66
|
+
| `--version` | | Show the version number | |
|
|
67
|
+
|
|
68
|
+
A flag can also take its value as `--flag=value`. If a flag is unknown, the CLI exits with code 1. A long flag needs two dashes: `-url` is an error.
|
|
69
|
+
|
|
70
|
+
Flags override values from the [config file](#config-file). Config values override the defaults.
|
|
57
71
|
|
|
58
72
|
### CLI Examples
|
|
59
73
|
|
|
@@ -85,7 +99,8 @@ npx @mdream/crawl -u example.com --site-name "My Company" --description "Company
|
|
|
85
99
|
|
|
86
100
|
## Glob Patterns
|
|
87
101
|
|
|
88
|
-
|
|
102
|
+
Use glob patterns to select URLs.
|
|
103
|
+
If you provide a pattern, the crawler uses sitemap discovery to find matching URLs.
|
|
89
104
|
|
|
90
105
|
```bash
|
|
91
106
|
# Crawl only the /docs/ section
|
|
@@ -108,37 +123,49 @@ import { crawlAndGenerate } from '@mdream/crawl'
|
|
|
108
123
|
|
|
109
124
|
const results = await crawlAndGenerate({
|
|
110
125
|
urls: ['https://docs.example.com'],
|
|
111
|
-
|
|
126
|
+
output: './output',
|
|
112
127
|
})
|
|
113
128
|
```
|
|
114
129
|
|
|
115
130
|
### `CrawlOptions`
|
|
116
131
|
|
|
132
|
+
`crawlAndGenerate`, the [config file](#config-file), and the [CLI](#cli-options) use the same options and the same defaults. Only `urls` is required.
|
|
133
|
+
|
|
117
134
|
| Option | Type | Default | Description |
|
|
118
135
|
|--------|------|---------|-------------|
|
|
119
|
-
| `urls` | `string[]` | Required | Starting URLs
|
|
120
|
-
| `
|
|
121
|
-
| `
|
|
122
|
-
| `
|
|
123
|
-
| `
|
|
124
|
-
| `
|
|
125
|
-
| `
|
|
126
|
-
| `generateLlmsFullTxt` | `boolean` | `false` | Generate an `llms-full.txt` file with full page content |
|
|
127
|
-
| `generateIndividualMd` | `boolean` | `true` | Write individual `.md` files for each page |
|
|
128
|
-
| `origin` | `string` | auto-detected | Origin URL for resolving relative paths in HTML |
|
|
129
|
-
| `siteNameOverride` | `string` | auto-extracted | Override the site name in the generated `llms.txt` |
|
|
130
|
-
| `descriptionOverride` | `string` | auto-extracted | Override the site description in the generated `llms.txt` |
|
|
131
|
-
| `globPatterns` | `ParsedUrlPattern[]` | `[]` | Pre-parsed URL glob patterns (advanced usage) |
|
|
132
|
-
| `exclude` | `string[]` | `[]` | Glob patterns for URLs to exclude |
|
|
133
|
-
| `crawlDelay` | `number` | from `robots.txt` | Delay between requests in seconds |
|
|
136
|
+
| `urls` | `string[]` | Required | Starting URLs. Glob patterns limit the crawl |
|
|
137
|
+
| `output` | `string` | `'output'` | Output directory, relative to the working directory |
|
|
138
|
+
| `depth` | `number` | `3` | Link depth from 0 to 10. See [Depth](#depth) |
|
|
139
|
+
| `maxPages` | `number` | no limit | Maximum number of pages to fetch |
|
|
140
|
+
| `artifacts` | `('llms.txt' \| 'llms-full.txt' \| 'markdown')[]` | all three | Files to write. `[]` writes no files, for example when only [hooks](#hooks) consume the pages |
|
|
141
|
+
| `driver` | `'http' \| 'playwright'` | `'http'` | Crawler driver |
|
|
142
|
+
| `sitemap` | `string \| string[]` | auto-discovered | Sitemap URLs to use instead of auto-discovery. All parts are loaded and merged. Ignored when `skipSitemap` is set |
|
|
134
143
|
| `skipSitemap` | `boolean` | `false` | Skip `sitemap.xml` and `robots.txt` discovery |
|
|
135
|
-
| `
|
|
136
|
-
| `
|
|
137
|
-
| `
|
|
138
|
-
| `
|
|
139
|
-
| `
|
|
140
|
-
| `
|
|
141
|
-
| `
|
|
144
|
+
| `siteName` | `string` | the home page title | Site name in `llms.txt` |
|
|
145
|
+
| `description` | `string` | the home page description | Site description in `llms.txt` |
|
|
146
|
+
| `origin` | `string` | the origin of each page | Origin for relative links in page Markdown. The page path stays the same |
|
|
147
|
+
| `exclude` | `string[]` | `[]` | Glob patterns for URLs to skip |
|
|
148
|
+
| `crawlDelay` | `number` | the `robots.txt` `Crawl-delay` | Delay between requests in seconds |
|
|
149
|
+
| `allowSubdomains` | `boolean` | `false` | Crawl other subdomains of the same root domain, for example `docs.example.com` and `blog.example.com`. Output paths start with the hostname, so files do not collide |
|
|
150
|
+
| `stripBoilerplate` | `boolean` | `true` | Remove repeated navigation and footers from page Markdown and `llms-full.txt`. Needs at least 3 pages. Does not run at depth 0 |
|
|
151
|
+
| `boilerplateThreshold` | `number` | `0.5` | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 |
|
|
152
|
+
| `useChrome` | `boolean` | `false` | Use the system Chrome with the Playwright driver. The CLI turns it on when it finds Chrome, unless you set it to `false` |
|
|
153
|
+
| `verbose` | `boolean` | `false` | Log details for each failed URL |
|
|
154
|
+
| `silent` | `boolean` | `false` | Drop all diagnostic and progress logs. A `logger` takes precedence |
|
|
155
|
+
| `logger` | `CrawlLogger` | `@clack/prompts` on stdout | Sink for diagnostic and progress logs, for example one that writes to stderr |
|
|
156
|
+
| `hooks` | `Partial<CrawlHooks>` | | Functions for each stage of the crawl. See [Hooks](#hooks) |
|
|
157
|
+
| `globPatterns` | `ParsedUrlPattern[]` | parsed from `urls` | Parsed URL patterns that replace the patterns from `urls`. Advanced |
|
|
158
|
+
|
|
159
|
+
`crawlAndGenerate` checks the options before it sends a request. An unknown key or an invalid value throws a `TypeError`. A removed v1 key, such as `maxDepth`, gives an error that names its replacement.
|
|
160
|
+
|
|
161
|
+
### Depth
|
|
162
|
+
|
|
163
|
+
`depth` sets how far the crawler follows links from the starting URLs.
|
|
164
|
+
|
|
165
|
+
- `0`: The crawler processes only the given URLs. It does not read `robots.txt` or the sitemap, and it does not follow links.
|
|
166
|
+
- `1` or more: Unless `skipSitemap` is set, the crawler reads `robots.txt` and the sitemap first. If the sitemap gives URLs, the crawler fetches those URLs and does not follow links. If not, the crawler follows links up to `depth` hops from the starting URLs.
|
|
167
|
+
|
|
168
|
+
The maximum is 10. Use `maxPages` to limit the total number of pages.
|
|
142
169
|
|
|
143
170
|
### `CrawlResult`
|
|
144
171
|
|
|
@@ -147,7 +174,7 @@ interface CrawlResult {
|
|
|
147
174
|
url: string
|
|
148
175
|
title: string
|
|
149
176
|
content: string
|
|
150
|
-
filePath?: string // Set when
|
|
177
|
+
filePath?: string // Set when artifacts include 'markdown'
|
|
151
178
|
timestamp: number // Unix timestamp of processing time
|
|
152
179
|
success: boolean
|
|
153
180
|
error?: string // Set when success is false
|
|
@@ -166,7 +193,7 @@ interface PageMetadata {
|
|
|
166
193
|
|
|
167
194
|
### `PageData`
|
|
168
195
|
|
|
169
|
-
The shape passed to the `
|
|
196
|
+
The shape passed to the `crawl:page` hook:
|
|
170
197
|
|
|
171
198
|
```typescript
|
|
172
199
|
interface PageData {
|
|
@@ -202,7 +229,7 @@ await crawlAndGenerate(options, (progress) => {
|
|
|
202
229
|
|
|
203
230
|
### Examples
|
|
204
231
|
|
|
205
|
-
#### Custom page processing with `
|
|
232
|
+
#### Custom page processing with `crawl:page`
|
|
206
233
|
|
|
207
234
|
```typescript
|
|
208
235
|
import { crawlAndGenerate } from '@mdream/crawl'
|
|
@@ -211,15 +238,15 @@ const pages = []
|
|
|
211
238
|
|
|
212
239
|
await crawlAndGenerate({
|
|
213
240
|
urls: ['https://docs.example.com'],
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
}
|
|
241
|
+
artifacts: [],
|
|
242
|
+
hooks: {
|
|
243
|
+
'crawl:page': (page) => {
|
|
244
|
+
pages.push({
|
|
245
|
+
url: page.url,
|
|
246
|
+
title: page.title,
|
|
247
|
+
description: page.metadata.description,
|
|
248
|
+
})
|
|
249
|
+
},
|
|
223
250
|
},
|
|
224
251
|
})
|
|
225
252
|
|
|
@@ -233,10 +260,9 @@ import { crawlAndGenerate } from '@mdream/crawl'
|
|
|
233
260
|
|
|
234
261
|
await crawlAndGenerate({
|
|
235
262
|
urls: ['https://example.com/docs/**'],
|
|
236
|
-
|
|
263
|
+
output: './docs-output',
|
|
237
264
|
exclude: ['/docs/deprecated/*', '/docs/internal/*'],
|
|
238
|
-
|
|
239
|
-
maxDepth: 2,
|
|
265
|
+
depth: 2,
|
|
240
266
|
})
|
|
241
267
|
```
|
|
242
268
|
|
|
@@ -245,36 +271,36 @@ await crawlAndGenerate({
|
|
|
245
271
|
```typescript
|
|
246
272
|
await crawlAndGenerate({
|
|
247
273
|
urls: ['https://example.com'],
|
|
248
|
-
outputDir: './output',
|
|
249
274
|
allowSubdomains: true, // Will also crawl docs.example.com, blog.example.com, etc.
|
|
250
|
-
|
|
251
|
-
maxDepth: 2,
|
|
275
|
+
depth: 2,
|
|
252
276
|
})
|
|
253
277
|
```
|
|
254
278
|
|
|
255
279
|
#### Single-page mode
|
|
256
280
|
|
|
257
|
-
Set `
|
|
281
|
+
Set `depth: 0` to process only the given URLs, with no discovery and no link following:
|
|
258
282
|
|
|
259
283
|
```typescript
|
|
260
284
|
await crawlAndGenerate({
|
|
261
285
|
urls: ['https://example.com/pricing', 'https://example.com/about'],
|
|
262
|
-
|
|
263
|
-
maxDepth: 0,
|
|
286
|
+
depth: 0,
|
|
264
287
|
})
|
|
265
288
|
```
|
|
266
289
|
|
|
267
290
|
## Config File
|
|
268
291
|
|
|
269
|
-
|
|
292
|
+
To set options and register hooks, create `mdream.config.ts` where you run the CLI.
|
|
293
|
+
The file can also use `.js` or `.mjs`.
|
|
294
|
+
The CLI loads it with [c12](https://github.com/unjs/c12) in direct and interactive modes.
|
|
270
295
|
|
|
271
296
|
```typescript
|
|
272
297
|
import { defineConfig } from '@mdream/crawl'
|
|
273
298
|
|
|
274
299
|
export default defineConfig({
|
|
275
300
|
exclude: ['*/admin/*', '*/internal/*'],
|
|
276
|
-
|
|
277
|
-
|
|
301
|
+
depth: 2,
|
|
302
|
+
maxPages: 500,
|
|
303
|
+
artifacts: ['llms.txt', 'markdown'],
|
|
278
304
|
hooks: {
|
|
279
305
|
'crawl:page': (page) => {
|
|
280
306
|
// Strip branding from all page titles
|
|
@@ -284,27 +310,23 @@ export default defineConfig({
|
|
|
284
310
|
})
|
|
285
311
|
```
|
|
286
312
|
|
|
287
|
-
|
|
313
|
+
The config file accepts every [`CrawlOptions`](#crawloptions) key. Each key is optional.
|
|
314
|
+
If you set `urls`, you can run the CLI without a URL argument.
|
|
315
|
+
Interactive mode uses the config values as its initial answers.
|
|
288
316
|
|
|
289
|
-
|
|
317
|
+
The CLI resolves each option in this order:
|
|
290
318
|
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
| `maxPages` | `number` | Maximum pages to crawl |
|
|
297
|
-
| `crawlDelay` | `number` | Delay between requests (seconds) |
|
|
298
|
-
| `skipSitemap` | `boolean` | Skip sitemap discovery |
|
|
299
|
-
| `sitemap` | `string \| string[]` | Explicit sitemap URL(s) to use instead of auto-discovery |
|
|
300
|
-
| `allowSubdomains` | `boolean` | Crawl across subdomains |
|
|
301
|
-
| `verbose` | `boolean` | Enable verbose logging |
|
|
302
|
-
| `artifacts` | `string[]` | Output formats: `llms.txt`, `llms-full.txt`, `markdown` |
|
|
303
|
-
| `hooks` | `object` | Hook functions (see below) |
|
|
319
|
+
1. A CLI flag, or an answer in interactive mode
|
|
320
|
+
2. The config file
|
|
321
|
+
3. The default
|
|
322
|
+
|
|
323
|
+
`exclude` is the exception: the CLI adds its patterns to the config patterns. An unknown key in the config file stops the CLI with an error.
|
|
304
324
|
|
|
305
325
|
## Hooks
|
|
306
326
|
|
|
307
|
-
|
|
327
|
+
Five hooks let you change data during the crawl.
|
|
328
|
+
Each hook receives a mutable object.
|
|
329
|
+
Change its properties to transform the output.
|
|
308
330
|
|
|
309
331
|
### `crawl:url`
|
|
310
332
|
|
|
@@ -324,7 +346,9 @@ defineConfig({
|
|
|
324
346
|
|
|
325
347
|
### `crawl:html`
|
|
326
348
|
|
|
327
|
-
|
|
349
|
+
Runs after fetching, before HTML-to-Markdown conversion.
|
|
350
|
+
Change `ctx.html` to remove elements or add content before conversion.
|
|
351
|
+
The hook does not run if the fetched content is already Markdown.
|
|
328
352
|
|
|
329
353
|
```typescript
|
|
330
354
|
defineConfig({
|
|
@@ -339,7 +363,7 @@ defineConfig({
|
|
|
339
363
|
|
|
340
364
|
### `crawl:page`
|
|
341
365
|
|
|
342
|
-
Called after HTML-to-Markdown conversion, before storage. Mutate `page.title` or other fields. This replaces the `onPage`
|
|
366
|
+
Called after HTML-to-Markdown conversion, before storage. Mutate `page.title` or other fields. This hook replaces the v1 `onPage` option.
|
|
343
367
|
|
|
344
368
|
```typescript
|
|
345
369
|
defineConfig({
|
|
@@ -394,7 +418,7 @@ import { crawlAndGenerate } from '@mdream/crawl'
|
|
|
394
418
|
|
|
395
419
|
await crawlAndGenerate({
|
|
396
420
|
urls: ['https://example.com'],
|
|
397
|
-
|
|
421
|
+
output: './output',
|
|
398
422
|
hooks: {
|
|
399
423
|
'crawl:page': (page) => {
|
|
400
424
|
page.title = page.title.replace(/ \| Brand$/, '')
|
|
@@ -428,7 +452,7 @@ npx @mdream/crawl -u example.com --driver playwright
|
|
|
428
452
|
```typescript
|
|
429
453
|
await crawlAndGenerate({
|
|
430
454
|
urls: ['https://spa-app.example.com'],
|
|
431
|
-
|
|
455
|
+
output: './output',
|
|
432
456
|
driver: 'playwright',
|
|
433
457
|
})
|
|
434
458
|
```
|
|
@@ -454,7 +478,7 @@ Disable with `--skip-sitemap` or `skipSitemap: true`.
|
|
|
454
478
|
|
|
455
479
|
### Overriding the sitemap location
|
|
456
480
|
|
|
457
|
-
|
|
481
|
+
If a site uses a custom sitemap URL or several sitemap files, pass their locations explicitly:
|
|
458
482
|
|
|
459
483
|
```bash
|
|
460
484
|
# Single non-standard location
|
|
@@ -481,13 +505,15 @@ export default defineConfig({
|
|
|
481
505
|
|
|
482
506
|
An explicit sitemap replaces auto-discovery entirely (`robots.txt` is still read for `Crawl-delay`).
|
|
483
507
|
|
|
484
|
-
##
|
|
508
|
+
## Artifacts
|
|
509
|
+
|
|
510
|
+
Choose the files with the `artifacts` option or the `--artifacts` flag. By default, the crawler writes all three.
|
|
485
511
|
|
|
486
|
-
###
|
|
512
|
+
### `markdown`
|
|
487
513
|
|
|
488
514
|
One `.md` file per crawled page, written to the output directory preserving the URL path structure. For example, `https://example.com/docs/getting-started` becomes `output/docs/getting-started.md`.
|
|
489
515
|
|
|
490
|
-
### llms.txt
|
|
516
|
+
### `llms.txt`
|
|
491
517
|
|
|
492
518
|
A site overview file following the [llms.txt specification](https://llmstxt.org/), listing all crawled pages with titles and links to their markdown files.
|
|
493
519
|
|
|
@@ -500,6 +526,86 @@ A site overview file following the [llms.txt specification](https://llmstxt.org/
|
|
|
500
526
|
- [About Us](about.md): https://example.com/about
|
|
501
527
|
```
|
|
502
528
|
|
|
503
|
-
### llms-full.txt
|
|
529
|
+
### `llms-full.txt`
|
|
504
530
|
|
|
505
531
|
Same structure as `llms.txt` but includes the full markdown content of every page inline.
|
|
532
|
+
|
|
533
|
+
## Migrating from v1
|
|
534
|
+
|
|
535
|
+
These docs cover v2.
|
|
536
|
+
For an earlier release, read the README at its release tag.
|
|
537
|
+
|
|
538
|
+
In v2, `crawlAndGenerate`, the config file, and the CLI accept the same options and defaults.
|
|
539
|
+
The library uses the names of the CLI flags.
|
|
540
|
+
|
|
541
|
+
### Renamed options
|
|
542
|
+
|
|
543
|
+
| v1 | v2 |
|
|
544
|
+
|----|----|
|
|
545
|
+
| `outputDir` | `output`. It is optional now, and the default is `'output'` |
|
|
546
|
+
| `maxRequestsPerCrawl` | `maxPages` |
|
|
547
|
+
| `maxDepth` and `followLinks` | `depth`. See [Depth](#depth) |
|
|
548
|
+
| `sitemapUrls` | `sitemap`. It also accepts one string |
|
|
549
|
+
| `generateLlmsTxt`, `generateLlmsFullTxt`, `generateIndividualMd` | `artifacts: ['llms.txt', 'llms-full.txt', 'markdown']` |
|
|
550
|
+
| `siteNameOverride` | `siteName` |
|
|
551
|
+
| `descriptionOverride` | `description` |
|
|
552
|
+
| `onPage: fn` | `hooks: { 'crawl:page': fn }` |
|
|
553
|
+
| `chunkSize` | Remove it. It had no effect |
|
|
554
|
+
| `maxDepth` in `mdream.config` | `depth` |
|
|
555
|
+
| `MdreamCrawlConfig` type | `Partial<CrawlOptions>` |
|
|
556
|
+
| `crawl` command after a global install | `mdream-crawl` |
|
|
557
|
+
|
|
558
|
+
```diff
|
|
559
|
+
await crawlAndGenerate({
|
|
560
|
+
urls: ['https://example.com'],
|
|
561
|
+
- outputDir: './output',
|
|
562
|
+
- maxRequestsPerCrawl: 100,
|
|
563
|
+
- followLinks: true,
|
|
564
|
+
- maxDepth: 2,
|
|
565
|
+
- generateLlmsFullTxt: true,
|
|
566
|
+
- onPage: page => console.log(page.url),
|
|
567
|
+
+ output: './output',
|
|
568
|
+
+ maxPages: 100,
|
|
569
|
+
+ depth: 2,
|
|
570
|
+
+ artifacts: ['llms.txt', 'llms-full.txt', 'markdown'],
|
|
571
|
+
+ hooks: { 'crawl:page': page => console.log(page.url) },
|
|
572
|
+
})
|
|
573
|
+
```
|
|
574
|
+
|
|
575
|
+
A v1 key now throws a `TypeError` that names its replacement. A key with the value `undefined` is ignored.
|
|
576
|
+
The library also checks values the same way as the CLI. For example, `depth` must be an integer from 0 to 10.
|
|
577
|
+
|
|
578
|
+
### New library defaults
|
|
579
|
+
|
|
580
|
+
The library defaults are now the CLI defaults:
|
|
581
|
+
|
|
582
|
+
- `depth` is `3`, so the crawler follows links when the sitemap gives no URLs. By default, the v1 library did not follow links.
|
|
583
|
+
- `artifacts` includes `llms-full.txt`. By default, the v1 library did not write it. To keep the v1 output, set `artifacts: ['llms.txt', 'markdown']`.
|
|
584
|
+
- `output` is `'output'`. In v1, `outputDir` was required.
|
|
585
|
+
|
|
586
|
+
v1 with `followLinks: false` used the sitemap and did not follow links. v2 has no exact match. Use `depth: 0` to process only the given URLs, or `depth: 1` to follow links one hop when there is no sitemap.
|
|
587
|
+
|
|
588
|
+
For your first v2 crawl, process only the supplied URLs and write to a separate directory:
|
|
589
|
+
|
|
590
|
+
```ts
|
|
591
|
+
import { crawlAndGenerate } from '@mdream/crawl'
|
|
592
|
+
|
|
593
|
+
await crawlAndGenerate({
|
|
594
|
+
urls: ['https://example.com'],
|
|
595
|
+
output: './output-v2',
|
|
596
|
+
depth: 0,
|
|
597
|
+
maxPages: 100,
|
|
598
|
+
artifacts: ['llms.txt', 'markdown'],
|
|
599
|
+
})
|
|
600
|
+
```
|
|
601
|
+
|
|
602
|
+
Review the generated files before replacing the previous artifacts.
|
|
603
|
+
To follow links, increase `depth`.
|
|
604
|
+
This example skips sitemap discovery.
|
|
605
|
+
|
|
606
|
+
### CLI changes
|
|
607
|
+
|
|
608
|
+
- Config values for `maxPages`, `artifacts`, `driver`, and `depth` now apply. In v1, the CLI ignored them.
|
|
609
|
+
- Interactive mode loads the config file and its hooks.
|
|
610
|
+
- An unknown flag or a single-dash long flag, such as `-url`, stops the CLI with exit code 1.
|
|
611
|
+
- `--crawl-delay` accepts decimal values, such as `0.5`.
|