@gresmcp/mcp 1.1.0 → 1.2.0

package/README.md

@@ -41,19 +41,27 @@ Environment variables:
 # 0. Verify your setup (Node, Postgres, pgvector, schema, Ollama, embedding model)
 gresmcp check --model nomic-embed-text
 
-# 1. Create a knowledge source (probes Ollama to validate the model and detect its dimension)
+# 1. Create a knowledge source named `docs` (probes Ollama to validate the model and detect its dimension)
 gresmcp ks create docs --model nomic-embed-text --description "Project documentation"
 
-# 2. Feed it
+# 2. Feed it the `docs` knowledge source
 gresmcp feed docs --path ./docs                 # a folder (text, markdown, code, html)
 gresmcp feed docs --path ./README.md            # a single file (text, markdown, code, html)
 gresmcp feed docs --text "Note: the API key lives in vault" --title "API keys" --tags secrets
 gresmcp feed docs --url https://example.com/docs/ --max-pages 200 --depth 3   # scrape a website
+gresmcp feed docs --url https://example.com/ --sitemap sitemap-only            # ingest exactly the sitemap.xml / robots.txt URLs
+gresmcp feed docs --url https://example.com/ --sitemap 'nav.docs a'            # only follow links inside <nav class="docs">
+gresmcp feed docs --url https://example.com/ --url-filter '**/docs/**' --url-filter '**/*.md'   # only follow matching links
+gresmcp feed docs --url https://example.com/blog/ --html-filter 'article.post -> .entry-content'   # only ingest matching HTML regions
 
 # 3. Serve it to your AI tool over MCP
 mcp docs          # stdio transport, scoped to the 'docs' knowledge source
 mcp --all         # every knowledge source in the database
 mcp docs,wiki --transport http --port 8080   # HTTP streaming on http://localhost:8080/mcp
+
+# 4. Find help about all the commands
+gresmcp --help
+mcp --help
 ```
 
 The database schema is created automatically on first run (`CREATE EXTENSION vector`, `ks` table, one entry table per knowledge source).
@@ -89,8 +97,11 @@ Feeds data into a knowledge source. Exactly one input of `--path`, `--url`, `--t
 | `--path <path>` | File or folder. Folders are walked recursively (skips `node_modules`, `.git`, `dist`, ...). Supported: text, markdown, code files, HTML (converted to Markdown via turndown). Binaries and empty files are skipped with a warning. |
 | `--url <url>` | Scrape a website (http/https). Follows same-host links from the seed, converts HTML to Markdown (main-content extraction via Readability) and ingests linked `text/markdown` / `text/plain` resources. Each page becomes a source keyed by its URL, so re-feeding is idempotent. Only 2XX responses are ingested; failures are reported as skipped. |
 | `--crawler <name>` | Crawler backend for `--url`: `auto` (default), `crawlee` (static HTTP), `playwright` or `puppeteer` (full browser rendering for SPAs; must be installed globally, e.g. `npm i -g playwright && npx playwright install chromium`). With `auto`, pages that look like empty JS shells are retried with Playwright when available. |
+| `--sitemap <mode>` | Link discovery for `--url`: `auto` (default) uses the site's `sitemap.xml` (then `robots.txt` sitemaps, incl. nested index files) when available and crawls exactly those URLs, falling back to HTML link crawling when no sitemap exists; `sitemap-only` does the same but ingests nothing (with a warning) when no sitemap is found; `html-only` always follows links; any other value is treated as a **CSS selector** — links are only followed inside matching elements on each page (e.g. `nav.docs a`), with a warning if the selector matches no links anywhere. Sitemap mode respects `--max-pages` and `--url-filter` (`--depth` is not applicable — sitemap URLs have no hierarchy). |
 | `--max-pages <n>` | Max pages to fetch when crawling (default 999) |
 | `--depth <n>` | Max link depth from the seed URL (default 5; 0 = seed page only) |
+| `--url-filter <glob>` | Only follow `--url` links whose URL matches the glob. Repeatable — a link is followed if it matches **any** glob. Matched against the full URL, case-insensitive; `*` doesn't cross `/`, `**` does. The seed URL itself is always fetched and ingested; non-matching links are never fetched and don't count towards `--max-pages`. Example: `--url-filter '**/docs/**'` |
+| `--html-filter <selector>` | Limit HTML content extraction to a CSS selector — any selector supported by the DOM works (e.g. `'#article > .content'`, `'article.post'`, `'#article .content:first-child'`). Use ` -> ` to chain several selectors and scope step by step: `'#article -> .content'` takes the first `#article`, then all `.content` inside it. Applies to `--path` HTML files and `--url` pages. Intermediate steps take the first match; the final step keeps **all** matching elements (joined in document order). When set, the automatic Readability main-content extraction is bypassed for pages. Noise elements (`script`, `nav`, `aside`, `footer`, ...) are removed before matching, so child pseudo-classes like `:first-child` see the cleaned DOM. Files/pages where the selector matches nothing are skipped (`html filter '<selector>' matched nothing`). |
 | `--text <text>` / `--stdin` | Manual entry text (chunked the same way as documents) |
 | `--title <title>` | Title for manual entries |
 | `--source-name <name>` | Source name for manual entries (default `manual`) |
@@ -105,7 +116,7 @@ Feeding is idempotent: a chunk whose `(source, chunk_index, content hash)` alrea
 
 While feeding, live progress is shown on stderr: a bar for the currently processed file (chunks processed, chars) plus a total line with files processed/total, cumulative chars and the number of rejected/ignored files — rejected files are reported but excluded from the progress percentage. In a non-interactive terminal (or when piped), plain per-file lines are printed instead. After a successful non-dry-run feed, a `this run` summary prints the same statistics as `gresmcp ks stats` (chunks, distinct sources, chars, avg/min/max per chunk), scoped to the chunks written during that run.
 
-For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
+For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. With `--url-filter`, only same-host links matching at least one glob are followed (the seed itself is exempt). Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
 
 ### `gresmcp init`
 
@@ -230,4 +241,5 @@ gresmcp is built on top of these great projects and tools:
 | [Ollama](https://ollama.com) | The easiest way to automate your work using open models, while keeping your data safe |
 | [Nano](https://nano.org) | The fast, feeless and eco-friendly digital currency |
 | [NanoGPT](https://nano-gpt.com) | Subscription-free, pay-as-you-go access to AI models |
+| [Crawlee](https://crawlee.dev/) | Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living. |
 | {{ YOU }} | This could be you — issues and PRs welcome at [gitea.com/fairking/gresmcp](https://gitea.com/fairking/gresmcp) |