@gresmcp/mcp 1.0.0 → 1.1.0

package/README.md

@@ -15,7 +15,7 @@ Knowledge is stored in PostgreSQL using `pgvector` (semantic similarity) and `ts
 - PostgreSQL with the [`pgvector`](https://github.com/pgvector/pgvector) extension installed (default `gres` database and `gres` user `gres` password used in the examples bellow)
 - An [Ollama](https://ollama.com) server with an embedding model pulled (e.g. `ollama pull nomic-embed-text` or `ollama pull bge-m3`)
 
-Run `gresmcp check` to verify all of these at once (it also creates the `vector` extension for you if it is missing).
+Run `gresmcp check --model <model>` to verify all of these at once (it also creates the `vector` extension for you if it is missing).
 
 ## Install
 
@@ -46,8 +46,9 @@ gresmcp ks create docs --model nomic-embed-text --description "Project documenta
 
 # 2. Feed it
 gresmcp feed docs --path ./docs                 # a folder (text, markdown, code, html)
-gresmcp feed docs --path ./README.md            # a single file
+gresmcp feed docs --path ./README.md            # a single file (text, markdown, code, html)
 gresmcp feed docs --text "Note: the API key lives in vault" --title "API keys" --tags secrets
+gresmcp feed docs --url https://example.com/docs/ --max-pages 200 --depth 3   # scrape a website
 
 # 3. Serve it to your AI tool over MCP
 mcp docs          # stdio transport, scoped to the 'docs' knowledge source
@@ -71,23 +72,30 @@ Edits a knowledge source. The Ollama server URL, name and description can change
 
 Lists knowledge sources with model, dimension, URL and entry counts.
 
+### `gresmcp ks stats <name> [--json]`
+
+Shows the statistics of a knowledge source: entry chunk count, distinct sources, total content volume with average/min/max chunk length, when it was last fed, plus the ks metadata (model, dimension, Ollama URL).
+
 ### `gresmcp ks delete <name> [--yes]`
 
 Deletes a knowledge source and drops its entry table (asks for confirmation unless `--yes`).
 
-### `gresmcp feed <ks> (--path <path> | --text <text> | --stdin) [options]`
+### `gresmcp feed <ks> (--path <path> | --url <url> | --text <text> | --stdin) [options]`
 
-Feeds data into a knowledge source. Exactly one input of `--path`, `--text` or `--stdin` must be given.
+Feeds data into a knowledge source. Exactly one input of `--path`, `--url`, `--text` or `--stdin` must be given.
 
 | Option | Description |
 | --- | --- |
 | `--path <path>` | File or folder. Folders are walked recursively (skips `node_modules`, `.git`, `dist`, ...). Supported: text, markdown, code files, HTML (converted to Markdown via turndown). Binaries and empty files are skipped with a warning. |
+| `--url <url>` | Scrape a website (http/https). Follows same-host links from the seed, converts HTML to Markdown (main-content extraction via Readability) and ingests linked `text/markdown` / `text/plain` resources. Each page becomes a source keyed by its URL, so re-feeding is idempotent. Only 2XX responses are ingested; failures are reported as skipped. |
+| `--crawler <name>` | Crawler backend for `--url`: `auto` (default), `crawlee` (static HTTP), `playwright` or `puppeteer` (full browser rendering for SPAs; must be installed globally, e.g. `npm i -g playwright && npx playwright install chromium`). With `auto`, pages that look like empty JS shells are retried with Playwright when available. |
+| `--max-pages <n>` | Max pages to fetch when crawling (default 999) |
+| `--depth <n>` | Max link depth from the seed URL (default 5; 0 = seed page only) |
 | `--text <text>` / `--stdin` | Manual entry text (chunked the same way as documents) |
 | `--title <title>` | Title for manual entries |
 | `--source-name <name>` | Source name for manual entries (default `manual`) |
-| `--tags a,b` | Tags stored in metadata, filterable by MCP `search` |
+| `--tags a,b` | Tags stored in metadata, filterable by MCP `search`; for `--url`, merged with each page's HTML `keywords`/`og:article:tag` |
 | `--metadata k=v ...` | Repeatable custom metadata entries |
-| `--url <url>` | Ollama URL override for this run (the model stays the ks's) |
 | `--replace` | Replace all chunks of the same source instead of skipping unchanged ones |
 | `--dry-run` | Parse and chunk only; no embeddings, no writes |
 | `--chunk-size <n>` | Max chunk length in characters (default 1200) |
@@ -95,11 +103,15 @@ Feeds data into a knowledge source. Exactly one input of `--path`, `--text` or `
 
 Feeding is idempotent: a chunk whose `(source, chunk_index, content hash)` already exists is skipped, so re-running a feed only adds new content. Use `--replace` to force a clean re-ingest of a source.
 
+While feeding, live progress is shown on stderr: a bar for the currently processed file (chunks processed, chars) plus a total line with files processed/total, cumulative chars and the number of rejected/ignored files — rejected files are reported but excluded from the progress percentage. In a non-interactive terminal (or when piped), plain per-file lines are printed instead. After a successful non-dry-run feed, a `this run` summary prints the same statistics as `gresmcp ks stats` (chunks, distinct sources, chars, avg/min/max per chunk), scoped to the chunks written during that run.
+
+For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
+
 ### `gresmcp init`
 
 Explicitly initializes/repairs the schema (runs automatically for every command anyway).
 
-### `gresmcp check [--model <model>] [--url <url>] [--probe] [--json]`
+### `gresmcp check --model <model> [--url <url>] [--probe] [--json]`
 
 Verifies that your environment meets all requirements:
 
@@ -110,7 +122,7 @@ Verifies that your environment meets all requirements:
 | pgvector | Extension installed (created automatically if missing); warns below 0.5.0, which is required for HNSW indexes |
 | schema | `ks` table and every knowledge source's entry table exist (`gresmcp init` repairs them) |
 | Ollama | Server reachable at the configured URL |
-| models | The embedding model of every existing knowledge source (plus `--model`, if given) is pulled at the right URL; `--probe` also embeds a test string to verify the dimension |
+| models | The embedding model given via `--model` plus the embedding model of every existing knowledge source is pulled at the right URL; `--probe` also embeds a test string to verify the dimension |
 
 ```sh
 gresmcp check --model nomic-embed-text
@@ -145,6 +157,8 @@ Hybrid mode fuses pgvector cosine ranking and `ts_rank_cd` full-text ranking wit
 
 ## Using with AI tools
 
+Please always mention in AGENTS.md which resources are included in the `gresmcp` knowledge base, so the agent aware what to look for and where.
+
 ### opencode
 
 ```json