@mdream/crawl 1.7.2 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,16 +2,21 @@
2
2
 
3
3
  Multi-page website crawler that generates [llms.txt](https://llmstxt.org/) files. Follows internal links and converts HTML to Markdown using [mdream](../mdream).
4
4
 
5
+ Upgrading from v1? Read [Migrating from v1](#migrating-from-v1).
6
+
5
7
  ## Setup
6
8
 
7
9
  ```bash
8
10
  npm install @mdream/crawl
9
11
  ```
10
12
 
13
+ After a global install (`npm install -g @mdream/crawl`), run the `mdream-crawl` command. With `npx`, use the package name: `npx @mdream/crawl`.
14
+
11
15
  For JavaScript-heavy sites that require browser rendering, install the optional Playwright dependencies:
12
16
 
13
17
  ```bash
14
18
  npm install crawlee playwright
19
+ npx playwright install chromium
15
20
  ```
16
21
 
17
22
  ## CLI Usage
@@ -34,26 +39,35 @@ npx @mdream/crawl -u https://docs.example.com
34
39
 
35
40
  ### CLI Options
36
41
 
42
+ Most flags set the [option](#crawloptions) with the same name in camelCase. For example, `--max-pages` sets `maxPages`.
43
+
37
44
  | Flag | Alias | Description | Default |
38
45
  |------|-------|-------------|---------|
39
- | `--url <url>` | `-u` | Website URL to crawl (supports glob patterns) | Required |
46
+ | `--url <url>` | `-u` | URL to crawl. Accepts glob patterns. Repeatable. You can also pass URLs without a flag | Required |
40
47
  | `--output <dir>` | `-o` | Output directory | `output` |
41
- | `--depth <number>` | `-d` | Crawl depth (0 for single page, max 10) | `3` |
42
- | `--single-page` | | Only process the given URL(s), no crawling. Alias for `--depth 0` | |
48
+ | `--depth <number>` | `-d` | Link depth from 0 to 10. See [Depth](#depth) | `3` |
49
+ | `--single-page` | | Same as `--depth 0`. It takes precedence over `--depth` | |
43
50
  | `--driver <type>` | | Crawler driver: `http` or `playwright` | `http` |
44
- | `--artifacts <list>` | | Comma-separated output formats: `llms.txt`, `llms-full.txt`, `markdown` | all three |
45
- | `--origin <url>` | | Origin URL for resolving relative paths (overrides auto-detection) | auto-detected |
46
- | `--site-name <name>` | | Override the auto-extracted site name used in llms.txt | auto-extracted |
47
- | `--description <desc>` | | Override the auto-extracted site description used in llms.txt | auto-extracted |
48
- | `--max-pages <number>` | | Maximum pages to crawl | unlimited |
49
- | `--crawl-delay <seconds>` | | Delay between requests in seconds | from `robots.txt` or none |
50
- | `--exclude <pattern>` | | Exclude URLs matching glob patterns (repeatable) | none |
51
+ | `--artifacts <list>` | | Comma-separated list of `llms.txt`, `llms-full.txt`, `markdown` | all three |
52
+ | `--origin <url>` | | Origin for relative links in page Markdown | the origin of each page |
53
+ | `--site-name <name>` | | Site name in llms.txt | the home page title |
54
+ | `--description <text>` | | Site description in llms.txt | the home page description |
55
+ | `--max-pages <number>` | | Maximum number of pages to fetch | no limit |
56
+ | `--crawl-delay <seconds>` | | Delay between requests in seconds | the `robots.txt` `Crawl-delay`, or none |
57
+ | `--exclude <pattern>` | | Skip URLs that match a glob pattern. Repeatable | none |
51
58
  | `--skip-sitemap` | | Skip `sitemap.xml` and `robots.txt` discovery | `false` |
52
- | `--sitemap <url>` | | Use an explicit sitemap URL instead of auto-discovery (repeatable for multi-part sitemaps) | auto-discovered |
53
- | `--allow-subdomains` | | Crawl across subdomains of the same root domain | `false` |
54
- | `--verbose` | `-v` | Enable verbose logging | `false` |
55
- | `--help` | `-h` | Show help message | |
56
- | `--version` | | Show version number | |
59
+ | `--sitemap <url>` | | Use this sitemap instead of auto-discovery. Repeatable for multi-part sitemaps | auto-discovered |
60
+ | `--allow-subdomains` | | Crawl other subdomains of the same root domain | `false` |
61
+ | `--keep-boilerplate` | | Keep repeated navigation and footers in page Markdown. Sets `stripBoilerplate: false` | `false` |
62
+ | `--boilerplate-threshold <n>` | | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 | `0.5` |
63
+ | `--verbose` | `-v` | Log details for each failed URL | `false` |
64
+ | `--quiet` | `-q`, `--silent` | Drop all logs. Use it to keep stdout clean for JSON-RPC or MCP | `false` |
65
+ | `--help` | `-h` | Show the help message | |
66
+ | `--version` | | Show the version number | |
67
+
68
+ A flag can also take its value as `--flag=value`. If a flag is unknown, the CLI exits with code 1. A long flag needs two dashes: `-url` is an error.
69
+
70
+ Flags override values from the [config file](#config-file). Config values override the defaults.
57
71
 
58
72
  ### CLI Examples
59
73
 
@@ -85,7 +99,8 @@ npx @mdream/crawl -u example.com --site-name "My Company" --description "Company
85
99
 
86
100
  ## Glob Patterns
87
101
 
88
- URLs support glob patterns for targeted crawling. When a glob pattern is provided, the crawler uses sitemap discovery to find all matching URLs.
102
+ Use glob patterns to select URLs.
103
+ If you provide a pattern, the crawler uses sitemap discovery to find matching URLs.
89
104
 
90
105
  ```bash
91
106
  # Crawl only the /docs/ section
@@ -108,37 +123,49 @@ import { crawlAndGenerate } from '@mdream/crawl'
108
123
 
109
124
  const results = await crawlAndGenerate({
110
125
  urls: ['https://docs.example.com'],
111
- outputDir: './output',
126
+ output: './output',
112
127
  })
113
128
  ```
114
129
 
115
130
  ### `CrawlOptions`
116
131
 
132
+ `crawlAndGenerate`, the [config file](#config-file), and the [CLI](#cli-options) use the same options and the same defaults. Only `urls` is required.
133
+
117
134
  | Option | Type | Default | Description |
118
135
  |--------|------|---------|-------------|
119
- | `urls` | `string[]` | Required | Starting URLs for crawling |
120
- | `outputDir` | `string` | Required | Directory to write output files |
121
- | `driver` | `'http' \| 'playwright'` | `'http'` | Crawler driver to use |
122
- | `maxRequestsPerCrawl` | `number` | `Number.MAX_SAFE_INTEGER` | Maximum total pages to crawl |
123
- | `followLinks` | `boolean` | `false` | Whether to follow internal links discovered on pages |
124
- | `maxDepth` | `number` | `1` | Maximum link-following depth. `0` enables single-page mode |
125
- | `generateLlmsTxt` | `boolean` | `true` | Generate an `llms.txt` file |
126
- | `generateLlmsFullTxt` | `boolean` | `false` | Generate an `llms-full.txt` file with full page content |
127
- | `generateIndividualMd` | `boolean` | `true` | Write individual `.md` files for each page |
128
- | `origin` | `string` | auto-detected | Origin URL for resolving relative paths in HTML |
129
- | `siteNameOverride` | `string` | auto-extracted | Override the site name in the generated `llms.txt` |
130
- | `descriptionOverride` | `string` | auto-extracted | Override the site description in the generated `llms.txt` |
131
- | `globPatterns` | `ParsedUrlPattern[]` | `[]` | Pre-parsed URL glob patterns (advanced usage) |
132
- | `exclude` | `string[]` | `[]` | Glob patterns for URLs to exclude |
133
- | `crawlDelay` | `number` | from `robots.txt` | Delay between requests in seconds |
136
+ | `urls` | `string[]` | Required | Starting URLs. Glob patterns limit the crawl |
137
+ | `output` | `string` | `'output'` | Output directory, relative to the working directory |
138
+ | `depth` | `number` | `3` | Link depth from 0 to 10. See [Depth](#depth) |
139
+ | `maxPages` | `number` | no limit | Maximum number of pages to fetch |
140
+ | `artifacts` | `('llms.txt' \| 'llms-full.txt' \| 'markdown')[]` | all three | Files to write. `[]` writes no files, for example when only [hooks](#hooks) consume the pages |
141
+ | `driver` | `'http' \| 'playwright'` | `'http'` | Crawler driver |
142
+ | `sitemap` | `string \| string[]` | auto-discovered | Sitemap URLs to use instead of auto-discovery. All parts are loaded and merged. Ignored when `skipSitemap` is set |
134
143
  | `skipSitemap` | `boolean` | `false` | Skip `sitemap.xml` and `robots.txt` discovery |
135
- | `sitemapUrls` | `string[]` | auto-discovered | Explicit sitemap URL(s) to use instead of auto-discovery. Supports non-standard locations and multiple parts (all loaded and merged). Ignored when `skipSitemap` is set |
136
- | `allowSubdomains` | `boolean` | `false` | Crawl across subdomains of the same root domain (e.g. `docs.example.com` + `blog.example.com`). Output files are namespaced by hostname to avoid collisions |
137
- | `useChrome` | `boolean` | `false` | Use system Chrome instead of Playwright's bundled browser (Playwright driver only) |
138
- | `chunkSize` | `number` | | Chunk size passed to mdream for markdown conversion |
139
- | `verbose` | `boolean` | `false` | Enable verbose error logging |
140
- | `hooks` | `Partial<CrawlHooks>` | | Hook functions for the crawl pipeline (see [Hooks](#hooks)) |
141
- | `onPage` | `(page: PageData) => Promise<void> \| void` | | **Deprecated.** Use `hooks['crawl:page']` instead. Still works for backwards compatibility |
144
+ | `siteName` | `string` | the home page title | Site name in `llms.txt` |
145
+ | `description` | `string` | the home page description | Site description in `llms.txt` |
146
+ | `origin` | `string` | the origin of each page | Origin for relative links in page Markdown. The page path stays the same |
147
+ | `exclude` | `string[]` | `[]` | Glob patterns for URLs to skip |
148
+ | `crawlDelay` | `number` | the `robots.txt` `Crawl-delay` | Delay between requests in seconds |
149
+ | `allowSubdomains` | `boolean` | `false` | Crawl other subdomains of the same root domain, for example `docs.example.com` and `blog.example.com`. Output paths start with the hostname, so files do not collide |
150
+ | `stripBoilerplate` | `boolean` | `true` | Remove repeated navigation and footers from page Markdown and `llms-full.txt`. Needs at least 3 pages. Does not run at depth 0 |
151
+ | `boilerplateThreshold` | `number` | `0.5` | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 |
152
+ | `useChrome` | `boolean` | `false` | Use the system Chrome with the Playwright driver. The CLI turns it on when it finds Chrome, unless you set it to `false` |
153
+ | `verbose` | `boolean` | `false` | Log details for each failed URL |
154
+ | `silent` | `boolean` | `false` | Drop all diagnostic and progress logs. A `logger` takes precedence |
155
+ | `logger` | `CrawlLogger` | `@clack/prompts` on stdout | Sink for diagnostic and progress logs, for example one that writes to stderr |
156
+ | `hooks` | `Partial<CrawlHooks>` | | Functions for each stage of the crawl. See [Hooks](#hooks) |
157
+ | `globPatterns` | `ParsedUrlPattern[]` | parsed from `urls` | Parsed URL patterns that replace the patterns from `urls`. Advanced |
158
+
159
+ `crawlAndGenerate` checks the options before it sends a request. An unknown key or an invalid value throws a `TypeError`. A removed v1 key, such as `maxDepth`, gives an error that names its replacement.
160
+
161
+ ### Depth
162
+
163
+ `depth` sets how far the crawler follows links from the starting URLs.
164
+
165
+ - `0`: The crawler processes only the given URLs. It does not read `robots.txt` or the sitemap, and it does not follow links.
166
+ - `1` or more: Unless `skipSitemap` is set, the crawler reads `robots.txt` and the sitemap first. If the sitemap gives URLs, the crawler fetches those URLs and does not follow links. If not, the crawler follows links up to `depth` hops from the starting URLs.
167
+
168
+ The maximum is 10. Use `maxPages` to limit the total number of pages.
142
169
 
143
170
  ### `CrawlResult`
144
171
 
@@ -147,7 +174,7 @@ interface CrawlResult {
147
174
  url: string
148
175
  title: string
149
176
  content: string
150
- filePath?: string // Set when generateIndividualMd is true
177
+ filePath?: string // Set when artifacts include 'markdown'
151
178
  timestamp: number // Unix timestamp of processing time
152
179
  success: boolean
153
180
  error?: string // Set when success is false
@@ -166,7 +193,7 @@ interface PageMetadata {
166
193
 
167
194
  ### `PageData`
168
195
 
169
- The shape passed to the `onPage` callback:
196
+ The shape passed to the `crawl:page` hook:
170
197
 
171
198
  ```typescript
172
199
  interface PageData {
@@ -202,7 +229,7 @@ await crawlAndGenerate(options, (progress) => {
202
229
 
203
230
  ### Examples
204
231
 
205
- #### Custom page processing with `onPage`
232
+ #### Custom page processing with `crawl:page`
206
233
 
207
234
  ```typescript
208
235
  import { crawlAndGenerate } from '@mdream/crawl'
@@ -211,15 +238,15 @@ const pages = []
211
238
 
212
239
  await crawlAndGenerate({
213
240
  urls: ['https://docs.example.com'],
214
- outputDir: './output',
215
- generateIndividualMd: false,
216
- generateLlmsTxt: false,
217
- onPage: (page) => {
218
- pages.push({
219
- url: page.url,
220
- title: page.title,
221
- description: page.metadata.description,
222
- })
241
+ artifacts: [],
242
+ hooks: {
243
+ 'crawl:page': (page) => {
244
+ pages.push({
245
+ url: page.url,
246
+ title: page.title,
247
+ description: page.metadata.description,
248
+ })
249
+ },
223
250
  },
224
251
  })
225
252
 
@@ -233,10 +260,9 @@ import { crawlAndGenerate } from '@mdream/crawl'
233
260
 
234
261
  await crawlAndGenerate({
235
262
  urls: ['https://example.com/docs/**'],
236
- outputDir: './docs-output',
263
+ output: './docs-output',
237
264
  exclude: ['/docs/deprecated/*', '/docs/internal/*'],
238
- followLinks: true,
239
- maxDepth: 2,
265
+ depth: 2,
240
266
  })
241
267
  ```
242
268
 
@@ -245,36 +271,36 @@ await crawlAndGenerate({
245
271
  ```typescript
246
272
  await crawlAndGenerate({
247
273
  urls: ['https://example.com'],
248
- outputDir: './output',
249
274
  allowSubdomains: true, // Will also crawl docs.example.com, blog.example.com, etc.
250
- followLinks: true,
251
- maxDepth: 2,
275
+ depth: 2,
252
276
  })
253
277
  ```
254
278
 
255
279
  #### Single-page mode
256
280
 
257
- Set `maxDepth: 0` to process only the provided URLs without crawling or link following:
281
+ Set `depth: 0` to process only the given URLs, with no discovery and no link following:
258
282
 
259
283
  ```typescript
260
284
  await crawlAndGenerate({
261
285
  urls: ['https://example.com/pricing', 'https://example.com/about'],
262
- outputDir: './output',
263
- maxDepth: 0,
286
+ depth: 0,
264
287
  })
265
288
  ```
266
289
 
267
290
  ## Config File
268
291
 
269
- Create a `mdream.config.ts` (or `.js`, `.mjs`) in your project root to set defaults and register hooks. Loaded via [c12](https://github.com/unjs/c12).
292
+ To set options and register hooks, create `mdream.config.ts` where you run the CLI.
293
+ The file can also use `.js` or `.mjs`.
294
+ The CLI loads it with [c12](https://github.com/unjs/c12) in direct and interactive modes.
270
295
 
271
296
  ```typescript
272
297
  import { defineConfig } from '@mdream/crawl'
273
298
 
274
299
  export default defineConfig({
275
300
  exclude: ['*/admin/*', '*/internal/*'],
276
- driver: 'http',
277
- maxDepth: 3,
301
+ depth: 2,
302
+ maxPages: 500,
303
+ artifacts: ['llms.txt', 'markdown'],
278
304
  hooks: {
279
305
  'crawl:page': (page) => {
280
306
  // Strip branding from all page titles
@@ -284,27 +310,23 @@ export default defineConfig({
284
310
  })
285
311
  ```
286
312
 
287
- CLI arguments override config file values. Array options like `exclude` are concatenated (config + CLI).
313
+ The config file accepts every [`CrawlOptions`](#crawloptions) key. Each key is optional.
314
+ If you set `urls`, you can run the CLI without a URL argument.
315
+ Interactive mode uses the config values as its initial answers.
288
316
 
289
- ### Config Options
317
+ The CLI resolves each option in this order:
290
318
 
291
- | Option | Type | Description |
292
- |--------|------|-------------|
293
- | `exclude` | `string[]` | Glob patterns for URLs to exclude |
294
- | `driver` | `'http' \| 'playwright'` | Crawler driver |
295
- | `maxDepth` | `number` | Maximum crawl depth |
296
- | `maxPages` | `number` | Maximum pages to crawl |
297
- | `crawlDelay` | `number` | Delay between requests (seconds) |
298
- | `skipSitemap` | `boolean` | Skip sitemap discovery |
299
- | `sitemap` | `string \| string[]` | Explicit sitemap URL(s) to use instead of auto-discovery |
300
- | `allowSubdomains` | `boolean` | Crawl across subdomains |
301
- | `verbose` | `boolean` | Enable verbose logging |
302
- | `artifacts` | `string[]` | Output formats: `llms.txt`, `llms-full.txt`, `markdown` |
303
- | `hooks` | `object` | Hook functions (see below) |
319
+ 1. A CLI flag, or an answer in interactive mode
320
+ 2. The config file
321
+ 3. The default
322
+
323
+ `exclude` is the exception: the CLI adds its patterns to the config patterns. An unknown key in the config file stops the CLI with an error.
304
324
 
305
325
  ## Hooks
306
326
 
307
- Six hooks let you intercept and transform data at each stage of the crawl pipeline. Hooks receive mutable objects. Mutate in-place to transform output.
327
+ Five hooks let you change data during the crawl.
328
+ Each hook receives a mutable object.
329
+ Change its properties to transform the output.
308
330
 
309
331
  ### `crawl:url`
310
332
 
@@ -324,7 +346,9 @@ defineConfig({
324
346
 
325
347
  ### `crawl:html`
326
348
 
327
- Called after fetching, before HTML-to-Markdown conversion. Mutate `ctx.html` to transform the raw HTML (strip elements, inject content) before mdream processes it. Not called when the fetched content is already markdown.
349
+ Runs after fetching, before HTML-to-Markdown conversion.
350
+ Change `ctx.html` to remove elements or add content before conversion.
351
+ The hook does not run if the fetched content is already Markdown.
328
352
 
329
353
  ```typescript
330
354
  defineConfig({
@@ -339,7 +363,7 @@ defineConfig({
339
363
 
340
364
  ### `crawl:page`
341
365
 
342
- Called after HTML-to-Markdown conversion, before storage. Mutate `page.title` or other fields. This replaces the `onPage` callback (which still works for backwards compatibility).
366
+ Called after HTML-to-Markdown conversion, before storage. Mutate `page.title` or other fields. This hook replaces the v1 `onPage` option.
343
367
 
344
368
  ```typescript
345
369
  defineConfig({
@@ -394,7 +418,7 @@ import { crawlAndGenerate } from '@mdream/crawl'
394
418
 
395
419
  await crawlAndGenerate({
396
420
  urls: ['https://example.com'],
397
- outputDir: './output',
421
+ output: './output',
398
422
  hooks: {
399
423
  'crawl:page': (page) => {
400
424
  page.title = page.title.replace(/ \| Brand$/, '')
@@ -428,7 +452,7 @@ npx @mdream/crawl -u example.com --driver playwright
428
452
  ```typescript
429
453
  await crawlAndGenerate({
430
454
  urls: ['https://spa-app.example.com'],
431
- outputDir: './output',
455
+ output: './output',
432
456
  driver: 'playwright',
433
457
  })
434
458
  ```
@@ -454,7 +478,7 @@ Disable with `--skip-sitemap` or `skipSitemap: true`.
454
478
 
455
479
  ### Overriding the sitemap location
456
480
 
457
- When a site keeps its sitemap at a non-standard URL, or splits it across several links, pin the location(s) explicitly instead of relying on discovery:
481
+ If a site uses a custom sitemap URL or several sitemap files, pass their locations explicitly:
458
482
 
459
483
  ```bash
460
484
  # Single non-standard location
@@ -481,13 +505,15 @@ export default defineConfig({
481
505
 
482
506
  An explicit sitemap replaces auto-discovery entirely (`robots.txt` is still read for `Crawl-delay`).
483
507
 
484
- ## Output Formats
508
+ ## Artifacts
509
+
510
+ Choose the files with the `artifacts` option or the `--artifacts` flag. By default, the crawler writes all three.
485
511
 
486
- ### Individual Markdown Files
512
+ ### `markdown`
487
513
 
488
514
  One `.md` file per crawled page, written to the output directory preserving the URL path structure. For example, `https://example.com/docs/getting-started` becomes `output/docs/getting-started.md`.
489
515
 
490
- ### llms.txt
516
+ ### `llms.txt`
491
517
 
492
518
  A site overview file following the [llms.txt specification](https://llmstxt.org/), listing all crawled pages with titles and links to their markdown files.
493
519
 
@@ -500,6 +526,86 @@ A site overview file following the [llms.txt specification](https://llmstxt.org/
500
526
  - [About Us](about.md): https://example.com/about
501
527
  ```
502
528
 
503
- ### llms-full.txt
529
+ ### `llms-full.txt`
504
530
 
505
531
  Same structure as `llms.txt` but includes the full markdown content of every page inline.
532
+
533
+ ## Migrating from v1
534
+
535
+ These docs cover v2.
536
+ For an earlier release, read the README at its release tag.
537
+
538
+ In v2, `crawlAndGenerate`, the config file, and the CLI accept the same options and defaults.
539
+ The library uses the names of the CLI flags.
540
+
541
+ ### Renamed options
542
+
543
+ | v1 | v2 |
544
+ |----|----|
545
+ | `outputDir` | `output`. It is optional now, and the default is `'output'` |
546
+ | `maxRequestsPerCrawl` | `maxPages` |
547
+ | `maxDepth` and `followLinks` | `depth`. See [Depth](#depth) |
548
+ | `sitemapUrls` | `sitemap`. It also accepts one string |
549
+ | `generateLlmsTxt`, `generateLlmsFullTxt`, `generateIndividualMd` | `artifacts: ['llms.txt', 'llms-full.txt', 'markdown']` |
550
+ | `siteNameOverride` | `siteName` |
551
+ | `descriptionOverride` | `description` |
552
+ | `onPage: fn` | `hooks: { 'crawl:page': fn }` |
553
+ | `chunkSize` | Remove it. It had no effect |
554
+ | `maxDepth` in `mdream.config` | `depth` |
555
+ | `MdreamCrawlConfig` type | `Partial<CrawlOptions>` |
556
+ | `crawl` command after a global install | `mdream-crawl` |
557
+
558
+ ```diff
559
+ await crawlAndGenerate({
560
+ urls: ['https://example.com'],
561
+ - outputDir: './output',
562
+ - maxRequestsPerCrawl: 100,
563
+ - followLinks: true,
564
+ - maxDepth: 2,
565
+ - generateLlmsFullTxt: true,
566
+ - onPage: page => console.log(page.url),
567
+ + output: './output',
568
+ + maxPages: 100,
569
+ + depth: 2,
570
+ + artifacts: ['llms.txt', 'llms-full.txt', 'markdown'],
571
+ + hooks: { 'crawl:page': page => console.log(page.url) },
572
+ })
573
+ ```
574
+
575
+ A v1 key now throws a `TypeError` that names its replacement. A key with the value `undefined` is ignored.
576
+ The library also checks values the same way as the CLI. For example, `depth` must be an integer from 0 to 10.
577
+
578
+ ### New library defaults
579
+
580
+ The library defaults are now the CLI defaults:
581
+
582
+ - `depth` is `3`, so the crawler follows links when the sitemap gives no URLs. By default, the v1 library did not follow links.
583
+ - `artifacts` includes `llms-full.txt`. By default, the v1 library did not write it. To keep the v1 output, set `artifacts: ['llms.txt', 'markdown']`.
584
+ - `output` is `'output'`. In v1, `outputDir` was required.
585
+
586
+ v1 with `followLinks: false` used the sitemap and did not follow links. v2 has no exact match. Use `depth: 0` to process only the given URLs, or `depth: 1` to follow links one hop when there is no sitemap.
587
+
588
+ For your first v2 crawl, process only the supplied URLs and write to a separate directory:
589
+
590
+ ```ts
591
+ import { crawlAndGenerate } from '@mdream/crawl'
592
+
593
+ await crawlAndGenerate({
594
+ urls: ['https://example.com'],
595
+ output: './output-v2',
596
+ depth: 0,
597
+ maxPages: 100,
598
+ artifacts: ['llms.txt', 'markdown'],
599
+ })
600
+ ```
601
+
602
+ Review the generated files before replacing the previous artifacts.
603
+ To follow links, increase `depth`.
604
+ This example skips sitemap discovery.
605
+
606
+ ### CLI changes
607
+
608
+ - Config values for `maxPages`, `artifacts`, `driver`, and `depth` now apply. In v1, the CLI ignored them.
609
+ - Interactive mode loads the config file and its hooks.
610
+ - An unknown flag or a single-dash long flag, such as `-url`, stops the CLI with exit code 1.
611
+ - `--crawl-delay` accepts decimal values, such as `0.5`.