@gresmcp/mcp 1.0.0 → 1.2.0

package/README.md

@@ -15,7 +15,7 @@ Knowledge is stored in PostgreSQL using `pgvector` (semantic similarity) and `ts
 - PostgreSQL with the [`pgvector`](https://github.com/pgvector/pgvector) extension installed (default `gres` database and `gres` user `gres` password used in the examples bellow)
 - An [Ollama](https://ollama.com) server with an embedding model pulled (e.g. `ollama pull nomic-embed-text` or `ollama pull bge-m3`)
 
-Run `gresmcp check` to verify all of these at once (it also creates the `vector` extension for you if it is missing).
+Run `gresmcp check --model <model>` to verify all of these at once (it also creates the `vector` extension for you if it is missing).
 
 ## Install
 
@@ -41,18 +41,27 @@ Environment variables:
 # 0. Verify your setup (Node, Postgres, pgvector, schema, Ollama, embedding model)
 gresmcp check --model nomic-embed-text
 
-# 1. Create a knowledge source (probes Ollama to validate the model and detect its dimension)
+# 1. Create a knowledge source named `docs` (probes Ollama to validate the model and detect its dimension)
 gresmcp ks create docs --model nomic-embed-text --description "Project documentation"
 
-# 2. Feed it
+# 2. Feed it the `docs` knowledge source
 gresmcp feed docs --path ./docs                 # a folder (text, markdown, code, html)
-gresmcp feed docs --path ./README.md            # a single file
+gresmcp feed docs --path ./README.md            # a single file (text, markdown, code, html)
 gresmcp feed docs --text "Note: the API key lives in vault" --title "API keys" --tags secrets
+gresmcp feed docs --url https://example.com/docs/ --max-pages 200 --depth 3   # scrape a website
+gresmcp feed docs --url https://example.com/ --sitemap sitemap-only            # ingest exactly the sitemap.xml / robots.txt URLs
+gresmcp feed docs --url https://example.com/ --sitemap 'nav.docs a'            # only follow links inside <nav class="docs">
+gresmcp feed docs --url https://example.com/ --url-filter '**/docs/**' --url-filter '**/*.md'   # only follow matching links
+gresmcp feed docs --url https://example.com/blog/ --html-filter 'article.post -> .entry-content'   # only ingest matching HTML regions
 
 # 3. Serve it to your AI tool over MCP
 mcp docs          # stdio transport, scoped to the 'docs' knowledge source
 mcp --all         # every knowledge source in the database
 mcp docs,wiki --transport http --port 8080   # HTTP streaming on http://localhost:8080/mcp
+
+# 4. Find help about all the commands
+gresmcp --help
+mcp --help
 ```
 
 The database schema is created automatically on first run (`CREATE EXTENSION vector`, `ks` table, one entry table per knowledge source).
@@ -71,23 +80,33 @@ Edits a knowledge source. The Ollama server URL, name and description can change
 
 Lists knowledge sources with model, dimension, URL and entry counts.
 
+### `gresmcp ks stats <name> [--json]`
+
+Shows the statistics of a knowledge source: entry chunk count, distinct sources, total content volume with average/min/max chunk length, when it was last fed, plus the ks metadata (model, dimension, Ollama URL).
+
 ### `gresmcp ks delete <name> [--yes]`
 
 Deletes a knowledge source and drops its entry table (asks for confirmation unless `--yes`).
 
-### `gresmcp feed <ks> (--path <path> | --text <text> | --stdin) [options]`
+### `gresmcp feed <ks> (--path <path> | --url <url> | --text <text> | --stdin) [options]`
 
-Feeds data into a knowledge source. Exactly one input of `--path`, `--text` or `--stdin` must be given.
+Feeds data into a knowledge source. Exactly one input of `--path`, `--url`, `--text` or `--stdin` must be given.
 
 | Option | Description |
 | --- | --- |
 | `--path <path>` | File or folder. Folders are walked recursively (skips `node_modules`, `.git`, `dist`, ...). Supported: text, markdown, code files, HTML (converted to Markdown via turndown). Binaries and empty files are skipped with a warning. |
+| `--url <url>` | Scrape a website (http/https). Follows same-host links from the seed, converts HTML to Markdown (main-content extraction via Readability) and ingests linked `text/markdown` / `text/plain` resources. Each page becomes a source keyed by its URL, so re-feeding is idempotent. Only 2XX responses are ingested; failures are reported as skipped. |
+| `--crawler <name>` | Crawler backend for `--url`: `auto` (default), `crawlee` (static HTTP), `playwright` or `puppeteer` (full browser rendering for SPAs; must be installed globally, e.g. `npm i -g playwright && npx playwright install chromium`). With `auto`, pages that look like empty JS shells are retried with Playwright when available. |
+| `--sitemap <mode>` | Link discovery for `--url`: `auto` (default) uses the site's `sitemap.xml` (then `robots.txt` sitemaps, incl. nested index files) when available and crawls exactly those URLs, falling back to HTML link crawling when no sitemap exists; `sitemap-only` does the same but ingests nothing (with a warning) when no sitemap is found; `html-only` always follows links; any other value is treated as a **CSS selector** — links are only followed inside matching elements on each page (e.g. `nav.docs a`), with a warning if the selector matches no links anywhere. Sitemap mode respects `--max-pages` and `--url-filter` (`--depth` is not applicable — sitemap URLs have no hierarchy). |
+| `--max-pages <n>` | Max pages to fetch when crawling (default 999) |
+| `--depth <n>` | Max link depth from the seed URL (default 5; 0 = seed page only) |
+| `--url-filter <glob>` | Only follow `--url` links whose URL matches the glob. Repeatable — a link is followed if it matches **any** glob. Matched against the full URL, case-insensitive; `*` doesn't cross `/`, `**` does. The seed URL itself is always fetched and ingested; non-matching links are never fetched and don't count towards `--max-pages`. Example: `--url-filter '**/docs/**'` |
+| `--html-filter <selector>` | Limit HTML content extraction to a CSS selector — any selector supported by the DOM works (e.g. `'#article > .content'`, `'article.post'`, `'#article .content:first-child'`). Use ` -> ` to chain several selectors and scope step by step: `'#article -> .content'` takes the first `#article`, then all `.content` inside it. Applies to `--path` HTML files and `--url` pages. Intermediate steps take the first match; the final step keeps **all** matching elements (joined in document order). When set, the automatic Readability main-content extraction is bypassed for pages. Noise elements (`script`, `nav`, `aside`, `footer`, ...) are removed before matching, so child pseudo-classes like `:first-child` see the cleaned DOM. Files/pages where the selector matches nothing are skipped (`html filter '<selector>' matched nothing`). |
 | `--text <text>` / `--stdin` | Manual entry text (chunked the same way as documents) |
 | `--title <title>` | Title for manual entries |
 | `--source-name <name>` | Source name for manual entries (default `manual`) |
-| `--tags a,b` | Tags stored in metadata, filterable by MCP `search` |
+| `--tags a,b` | Tags stored in metadata, filterable by MCP `search`; for `--url`, merged with each page's HTML `keywords`/`og:article:tag` |
 | `--metadata k=v ...` | Repeatable custom metadata entries |
-| `--url <url>` | Ollama URL override for this run (the model stays the ks's) |
 | `--replace` | Replace all chunks of the same source instead of skipping unchanged ones |
 | `--dry-run` | Parse and chunk only; no embeddings, no writes |
 | `--chunk-size <n>` | Max chunk length in characters (default 1200) |
@@ -95,11 +114,15 @@ Feeds data into a knowledge source. Exactly one input of `--path`, `--text` or `
 
 Feeding is idempotent: a chunk whose `(source, chunk_index, content hash)` already exists is skipped, so re-running a feed only adds new content. Use `--replace` to force a clean re-ingest of a source.
 
+While feeding, live progress is shown on stderr: a bar for the currently processed file (chunks processed, chars) plus a total line with files processed/total, cumulative chars and the number of rejected/ignored files — rejected files are reported but excluded from the progress percentage. In a non-interactive terminal (or when piped), plain per-file lines are printed instead. After a successful non-dry-run feed, a `this run` summary prints the same statistics as `gresmcp ks stats` (chunks, distinct sources, chars, avg/min/max per chunk), scoped to the chunks written during that run.
+
+For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. With `--url-filter`, only same-host links matching at least one glob are followed (the seed itself is exempt). Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
+
 ### `gresmcp init`
 
 Explicitly initializes/repairs the schema (runs automatically for every command anyway).
 
-### `gresmcp check [--model <model>] [--url <url>] [--probe] [--json]`
+### `gresmcp check --model <model> [--url <url>] [--probe] [--json]`
 
 Verifies that your environment meets all requirements:
 
@@ -110,7 +133,7 @@ Verifies that your environment meets all requirements:
 | pgvector | Extension installed (created automatically if missing); warns below 0.5.0, which is required for HNSW indexes |
 | schema | `ks` table and every knowledge source's entry table exist (`gresmcp init` repairs them) |
 | Ollama | Server reachable at the configured URL |
-| models | The embedding model of every existing knowledge source (plus `--model`, if given) is pulled at the right URL; `--probe` also embeds a test string to verify the dimension |
+| models | The embedding model given via `--model` plus the embedding model of every existing knowledge source is pulled at the right URL; `--probe` also embeds a test string to verify the dimension |
 
 ```sh
 gresmcp check --model nomic-embed-text
@@ -145,6 +168,8 @@ Hybrid mode fuses pgvector cosine ranking and `ts_rank_cd` full-text ranking wit
 
 ## Using with AI tools
 
+Please always mention in AGENTS.md which resources are included in the `gresmcp` knowledge base, so the agent aware what to look for and where.
+
 ### opencode
 
 ```json
@@ -216,4 +241,5 @@ gresmcp is built on top of these great projects and tools:
 | [Ollama](https://ollama.com) | The easiest way to automate your work using open models, while keeping your data safe |
 | [Nano](https://nano.org) | The fast, feeless and eco-friendly digital currency |
 | [NanoGPT](https://nano-gpt.com) | Subscription-free, pay-as-you-go access to AI models |
+| [Crawlee](https://crawlee.dev/) | Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living. |
 | {{ YOU }} | This could be you — issues and PRs welcome at [gitea.com/fairking/gresmcp](https://gitea.com/fairking/gresmcp) |