@lenne.tech/cli 1.12.0 → 1.14.0

package/docs/commands.md

@@ -79,13 +79,16 @@ lt server create [name] [options]
 |--------|-------------|
 | `--description <text>` | Project description |
 | `--author <name>` | Author name |
+| `--api-mode <Rest\|GraphQL\|Both>` | API mode (ignored with `--next`) |
+| `--framework-mode <npm\|vendor>` | Framework consumption mode (ignored with `--next`) |
 | `--branch <branch>` / `-b` | Branch of nest-server-starter to use as template |
 | `--copy <path>` / `-c` | Copy from local template directory instead of cloning |
 | `--link <path>` | Symlink to local template directory (fastest, changes affect original) |
 | `--git` | Initialize git repository |
+| `--next` | **Experimental:** clone [`nest-base`](https://github.com/lenneTech/nest-base) (Bun + Prisma 7 + Postgres + Better-Auth) instead of `nest-server-starter`. Skips API-mode / vendor-mode / install / lt.config.json processing. |
 | `--noConfirm` | Skip confirmation prompts |
 
-**CLAUDE.md Patching:** If the project contains a `CLAUDE.md`, the generic API mode description is replaced with the selected mode. In single-mode projects (`Rest` or `GraphQL`), the "API Mode System" documentation section is condensed to a brief note.
+**CLAUDE.md Patching:** If the project contains a `CLAUDE.md`, the generic API mode description is replaced with the selected mode. In single-mode projects (`Rest` or `GraphQL`), the "API Mode System" documentation section is condensed to a brief note. Skipped when `--next` is used.
 
 **Configuration:** `commands.server.create.*`, `defaults.author`, `defaults.noConfirm`
 
@@ -532,6 +535,7 @@ lt fullstack init [options]
 | `--framework-mode <mode>` | Backend framework consumption mode: `npm` (classic) or `vendor` (pilot, core copied to `src/core/`) |
 | `--framework-upstream-branch <ref>` | Upstream `nest-server` branch/tag to vendor from (only with `--framework-mode vendor`) |
 | `--frontend-framework-mode <mode>` | Frontend framework consumption mode: `npm` or `vendor` (nuxt-extensions copied to `app/core/`) |
+| `--next` | **Experimental:** clone [`nest-base`](https://github.com/lenneTech/nest-base) (Bun + Prisma 7 + Postgres + Better-Auth) for the API instead of `nest-server-starter`. Forces `--api-mode Rest` and `--framework-mode npm`, skips workspace install (run `pnpm install` for app and `bun install` for api manually). |
 | `--dry-run` | Print the resolved plan without making any changes |
 | `--git` | Push initial commit to remote repository (git is always initialized) |
 | `--git-link <url>` | Git remote repository URL (required when `--git` is true) |
@@ -1409,6 +1413,58 @@ lt tools regex [pattern] [text]
 
 ---
 
+### `lt tools crawl`
+
+Crawls a website and stores each page as a Markdown file (with YAML frontmatter containing `source_url`, `download_date`, `first_downloaded`, `description`, language, word count, etc.) so it can be consumed as a Claude Code knowledge base. Optionally follows same-origin links up to a configurable depth, seeds the queue from `<origin>/sitemap.xml`, and downloads referenced images into a shared `images/` folder (deduplicated by content hash). Re-running the command against the same output directory updates existing pages while preserving their original `first_downloaded` timestamp.
+
+**Alias:** `cr`
+
+**Usage:**
+```bash
+lt tools crawl <url> [options]
+```
+
+**Options:**
+- `--out <dir>` — Output directory (default: current directory). Single-page crawls write the `.md` directly here; multi-page crawls generate `<out>/README.md` plus `<out>/pages/` and `<out>/images/`.
+- `--depth <n|all>` — Link depth (default `0`). `0` = only the start page, `1` = start page + direct same-origin links, `2` = and their links, ... Use `--depth all` (or `--depth -1`, or the shortcut flag `--all`) to follow every same-origin link transitively; the crawl then stops when `--max-pages` is reached.
+- `--all` — Shortcut for `--depth all`.
+- `--render` / `--no-render` — Render each page through a headless browser before extraction (default **on**). Required for SPAs (Vue/Nuxt/React/Angular) whose content is client-rendered. Uses `playwright-core` with system Chrome / Edge first, then Playwright's bundled Chromium. Use `--no-render` for a plain HTTP fetch when you know the site is static (faster, no browser needed).
+- `--install-browser` — If `--render` finds no browser, auto-install Playwright's Chromium (one-time ~170 MB download).
+- `--prune` / `--no-prune` — After a multi-page crawl, remove any `.md` or image files inside `<out>/pages` and `<out>/images` that were not written by the current run (default **on**). Keeps the knowledge base aligned with the live site on update runs. Empty subdirectories are cleaned up too. Ignored in single-page mode. Use `--no-prune` to preserve old files.
+- `--no-images` — Disable image downloads.
+- `--no-sitemap` — Skip discovery via `<origin>/sitemap.xml`.
+- `--concurrency <n>` — Parallel HTTP requests (default `4`).
+- `--max-pages <n>` — Safety cap on total pages (default `200`).
+- `--selector <css>` — CSS selector scoping the main content (e.g. `article`, `main`).
+- `--timeout <ms>` — HTTP request timeout in ms (default `20000`).
+- `--noConfirm` — Skip confirmation prompts.
+
+**Examples:**
+```bash
+# Single page into the current directory
+lt tools crawl https://example.com/article --noConfirm
+
+# Crawl start page + direct links into ./knowledge
+lt tools crawl https://example.com --out ./knowledge --depth 1 --noConfirm
+
+# Full mini-site with sitemap seeding and images
+lt tools crawl https://example.com --out ./kb --depth 2 --max-pages 100 --noConfirm
+
+# Crawl every reachable same-origin page (safety cap via --max-pages)
+lt tools crawl https://example.com --out ./kb --depth all --max-pages 500 --noConfirm
+
+# Same, using the --all shortcut
+lt tools crawl https://example.com --out ./kb --all --max-pages 500 --noConfirm
+
+# Full SPA-aware crawl (render + prune are on by default)
+lt tools crawl https://lenne.tech --all --noConfirm
+
+# Opt-out: plain HTTP fetch for a known-static site, keep orphans
+lt tools crawl https://example.com --all --no-render --no-prune --noConfirm
+```
+
+---
+
 ## Configuration Priority
 
 All configurable commands follow this priority order (highest to lowest):

package/docs/lt.config.md

@@ -923,6 +923,43 @@ Reinitializes npm packages.
 }
 ```
 
+#### `lt tools crawl`
+
+Crawls a website into Markdown files (knowledge base builder).
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `commands.tools.crawl.out` | `string` | `.` | Output directory |
+| `commands.tools.crawl.depth` | `number \| "all"` | `0` | Link depth (0 = only start page, 1 = direct links, ..., `"all"` or `-1` = follow every same-origin link, bounded by `maxPages`) |
+| `commands.tools.crawl.includeImages` | `boolean` | `true` | Download images and inline with local paths |
+| `commands.tools.crawl.includeSitemap` | `boolean` | `true` | Also seed queue from `<origin>/sitemap.xml` |
+| `commands.tools.crawl.concurrency` | `number` | `4` | Parallel HTTP requests |
+| `commands.tools.crawl.maxPages` | `number` | `200` | Safety cap on total pages |
+| `commands.tools.crawl.selector` | `string` | – | CSS selector for main content |
+| `commands.tools.crawl.timeout` | `number` | `20000` | HTTP request timeout in ms |
+| `commands.tools.crawl.renderJs` | `boolean` | `true` | Render pages through a headless browser (for SPAs). Uses playwright-core. Set to `false` for plain HTTP. |
+| `commands.tools.crawl.prune` | `boolean` | `true` | Remove orphaned `.md` / image files after a multi-page crawl (update-in-place). Set to `false` to preserve old files. |
+| `commands.tools.crawl.noConfirm` | `boolean` | `false` | Skip confirmation prompts |
+
+**Example:**
+```json
+{
+  "commands": {
+    "tools": {
+      "crawl": {
+        "out": "./knowledge",
+        "depth": 2,
+        "includeImages": true,
+        "includeSitemap": true,
+        "concurrency": 4,
+        "maxPages": 200,
+        "noConfirm": true
+      }
+    }
+  }
+}
+```
+
 ---
 
 ### Metadata

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lenne.tech/cli",
-  "version": "1.12.0",
+  "version": "1.14.0",
   "description": "lenne.Tech CLI: lt",
   "keywords": [
     "lenne.Tech",
@@ -61,14 +61,19 @@
     "@lenne.tech/cli-plugin-helper": "0.0.14",
     "axios": "1.15.0",
     "bcrypt": "6.0.0",
+    "defuddle": "0.17.0",
     "glob": "13.0.6",
     "gluegun": "5.2.2",
     "js-sha256": "0.11.1",
     "js-yaml": "4.1.1",
+    "jsdom": "29.0.2",
     "lodash": "4.18.1",
     "open": "11.0.0",
+    "playwright-core": "1.59.1",
     "ts-morph": "28.0.0",
     "ts-node": "10.9.2",
+    "turndown": "7.2.4",
+    "turndown-plugin-gfm": "1.0.2",
     "typescript": "6.0.3"
   },
   "devDependencies": {
@@ -77,8 +82,10 @@
     "@types/ejs": "3.1.5",
     "@types/jest": "30.0.0",
     "@types/js-yaml": "4.0.9",
+    "@types/jsdom": "28.0.1",
     "@types/lodash": "4.17.24",
     "@types/node": "25.6.0",
+    "@types/turndown": "5.0.6",
     "@typescript-eslint/eslint-plugin": "8.58.2",
     "@typescript-eslint/parser": "8.58.2",
     "ejs": "5.0.2",