@gresmcp/mcp 1.2.0 → 1.3.0

package/README.md

@@ -1,14 +1,37 @@
 # gresmcp (`@gresmcp/mcp`)
 
-From raw docs to refined answers — Gresy fires your documents into solid Postgres knowledge, entirely local, your MCP.
+> From raw docs to refined answers — Gresy fires your documents into solid Postgres knowledge, entirely local, your MCP.
 
-A Postgres-backed **knowledge source (ks) MCP server** for AI tools, with a management CLI.
+A Postgres-backed **knowledge source (ks) MCP server** for AI tools, with a management CLI. 
 
-Knowledge is stored in PostgreSQL using `pgvector` (semantic similarity) and `tsvector` (full-text search). Queries are answered with **hybrid search** that fuses both signals via Reciprocal Rank Fusion (RRF). Documents are embedded with an [Ollama](https://ollama.com) embedding model; the model is locked per knowledge source so vector dimensions always stay consistent.
+Use it to create, manage and share a knowledge bases (documentation, notes, specs) across all your projects and AI tools.
 
 - **`gresmcp`** — CLI: create/edit/delete knowledge sources, feed files/folders/url or manual text
 - **`mcp`** — FastMCP server: read-only search/query tools scoped to the knowledge sources you pass on the command line
 
+## Features
+
+- **Hybrid search** — pgvector semantic similarity + `tsvector` full-text search fused via Reciprocal Rank Fusion (RRF); `vector`-only and `keyword`-only modes also available, with results comparable across knowledge sources
+- **Fully local & private** — documents stay in your PostgreSQL database and are embedded by your own [Ollama](https://ollama.com) server; no cloud, no third parties.
+- **Many input types** — feed files, folders, whole websites or manual text/stdin notes
+- **Document conversion built in** 
+  - HTML (Readability main-content extraction), 
+  - PDF (heading/list/table detection, page metadata), 
+  - Word `.docx` (via [mammoth](https://github.com/mwilliamson/mammoth.js)) and `.doc` (via [word-extractor](https://github.com/morungos/node-word-extractor)), 
+  - EPUB ebooks (via [epub2](https://www.npmjs.com/package/epub2), chapters converted to Markdown), 
+  - Markdown, code and plain text — all chunked as clean Markdown
+- **Encrypted file support** — password-protected PDFs and Word documents via `--file-password`
+- **Website crawling** — same-host crawling with `sitemap.xml`/`robots.txt` discovery, URL glob filters, CSS-scoped content extraction (`--html-filter`) and pluggable crawler backends (`crawlee`, `playwright`, `puppeteer`) incl. JS-shell retry
+- **Idempotent feeding** — unchanged chunks are skipped on re-feed; `--replace` forces a clean re-ingest; `--dry-run` previews without writing
+- **Live progress** — per-file progress bar plus end-of-run summary statistics while feeding
+- **Immutable embeddings** — the embedding model and its dimension are locked per knowledge source, so vectors always stay consistent
+- **Multiple knowledge sources** — one table per ks; different embedding models/dimensions coexist and the MCP server can search several ks at once
+- **MCP server** — read-only [FastMCP](https://gofastmcp.com) server with `search`, `get_entry`, `list_entries`, `list_sources` and `list_ks` tools, over stdio or HTTP streaming, hard-scoped to the ks names you pass
+- **Metadata & tags** — attach tags/custom metadata at feed time and filter MCP search results by them
+- **Configurable chunking** — chunk size and overlap tunable per feed
+- **Environment check** — `gresmcp check` verifies Node, Postgres, pgvector, schema, Ollama and embedding models in one shot (script-friendly exit code, `--json` output)
+- **Zero-config schema** — the database schema is created automatically on first run (`gresmcp init` repairs it)
+
 ## Requirements
 
 - Node.js >= 20
@@ -17,10 +40,22 @@ Knowledge is stored in PostgreSQL using `pgvector` (semantic similarity) and `ts
 
 Run `gresmcp check --model <model>` to verify all of these at once (it also creates the `vector` extension for you if it is missing).
 
-## Install
+## Quick start
 
 ```sh
-npm install -g @gresmcp/mcp
+# Installation
+npm install -g @gresmcp/mcp 
+
+# Create a knowledge base
+gresmcp ks create docs --model nomic-embed-text --description "Project documentation" 
+
+# Feed the knowledge base with the documents
+gresmcp feed docs --path ./docs
+
+# Serve it to your AI over stdio
+mcp docs
+# or over http
+mcp docs --transport http --port 8080
 ```
 
 or use it directly with `npx -y @gresmcp/mcp ...` (MCP server) / `npx -y -p @gresmcp/mcp gresmcp ...` (CLI).
@@ -45,9 +80,10 @@ gresmcp check --model nomic-embed-text
 gresmcp ks create docs --model nomic-embed-text --description "Project documentation"
 
 # 2. Feed it the `docs` knowledge source
-gresmcp feed docs --path ./docs                 # a folder (text, markdown, code, html)
-gresmcp feed docs --path ./README.md            # a single file (text, markdown, code, html)
+gresmcp feed docs --path ./docs                 # a folder (text, markdown, code, html, pdf, doc)
+gresmcp feed docs --path ./README.md            # a single file (text, markdown, code, html, pdf, doc)
 gresmcp feed docs --text "Note: the API key lives in vault" --title "API keys" --tags secrets
+gresmcp feed docs --url https://example.com/my-document.docx   # a single remote file (text, markdown, code, html, pdf, doc)
 gresmcp feed docs --url https://example.com/docs/ --max-pages 200 --depth 3   # scrape a website
 gresmcp feed docs --url https://example.com/ --sitemap sitemap-only            # ingest exactly the sitemap.xml / robots.txt URLs
 gresmcp feed docs --url https://example.com/ --sitemap 'nav.docs a'            # only follow links inside <nav class="docs">
@@ -94,14 +130,15 @@ Feeds data into a knowledge source. Exactly one input of `--path`, `--url`, `--t
 
 | Option | Description |
 | --- | --- |
-| `--path <path>` | File or folder. Folders are walked recursively (skips `node_modules`, `.git`, `dist`, ...). Supported: text, markdown, code files, HTML (converted to Markdown via turndown). Binaries and empty files are skipped with a warning. |
-| `--url <url>` | Scrape a website (http/https). Follows same-host links from the seed, converts HTML to Markdown (main-content extraction via Readability) and ingests linked `text/markdown` / `text/plain` resources. Each page becomes a source keyed by its URL, so re-feeding is idempotent. Only 2XX responses are ingested; failures are reported as skipped. |
+| `--path <path>` | File or folder. Folders are walked recursively (skips `node_modules`, `.git`, `dist`, ...). Supported: text, markdown, code files, HTML (converted to Markdown via turndown), PDF (converted to Markdown with heading/list/table detection via [pdf2md](https://github.com/opengovsg/pdf2md); page count and image-only page count are stored in metadata), Word documents (`.docx` converted to Markdown via [mammoth](https://github.com/mwilliamson/mammoth.js) with headings/tables/lists, `.doc` plain-text extraction), EPUB ebooks (`.epub` parsed via [epub2](https://www.npmjs.com/package/epub2); spine chapters are converted to Markdown with TOC titles as headings, chapter count/author/language stored in metadata). Binaries and empty files are skipped with a warning. |
+| `--url <url>` | Scrape a website (http/https). Follows same-host links from the seed, converts HTML to Markdown (main-content extraction via Readability) and ingests linked `text/markdown` / `text/plain` resources as well as `application/pdf`, `application/msword`, `application/vnd.openxmlformats-officedocument.wordprocessingml.document` and `application/epub+zip` responses. Each page becomes a source keyed by its URL, so re-feeding is idempotent. Only 2XX responses are ingested; failures are reported as skipped. |
 | `--crawler <name>` | Crawler backend for `--url`: `auto` (default), `crawlee` (static HTTP), `playwright` or `puppeteer` (full browser rendering for SPAs; must be installed globally, e.g. `npm i -g playwright && npx playwright install chromium`). With `auto`, pages that look like empty JS shells are retried with Playwright when available. |
 | `--sitemap <mode>` | Link discovery for `--url`: `auto` (default) uses the site's `sitemap.xml` (then `robots.txt` sitemaps, incl. nested index files) when available and crawls exactly those URLs, falling back to HTML link crawling when no sitemap exists; `sitemap-only` does the same but ingests nothing (with a warning) when no sitemap is found; `html-only` always follows links; any other value is treated as a **CSS selector** — links are only followed inside matching elements on each page (e.g. `nav.docs a`), with a warning if the selector matches no links anywhere. Sitemap mode respects `--max-pages` and `--url-filter` (`--depth` is not applicable — sitemap URLs have no hierarchy). |
 | `--max-pages <n>` | Max pages to fetch when crawling (default 999) |
 | `--depth <n>` | Max link depth from the seed URL (default 5; 0 = seed page only) |
 | `--url-filter <glob>` | Only follow `--url` links whose URL matches the glob. Repeatable — a link is followed if it matches **any** glob. Matched against the full URL, case-insensitive; `*` doesn't cross `/`, `**` does. The seed URL itself is always fetched and ingested; non-matching links are never fetched and don't count towards `--max-pages`. Example: `--url-filter '**/docs/**'` |
 | `--html-filter <selector>` | Limit HTML content extraction to a CSS selector — any selector supported by the DOM works (e.g. `'#article > .content'`, `'article.post'`, `'#article .content:first-child'`). Use ` -> ` to chain several selectors and scope step by step: `'#article -> .content'` takes the first `#article`, then all `.content` inside it. Applies to `--path` HTML files and `--url` pages. Intermediate steps take the first match; the final step keeps **all** matching elements (joined in document order). When set, the automatic Readability main-content extraction is bypassed for pages. Noise elements (`script`, `nav`, `aside`, `footer`, ...) are removed before matching, so child pseudo-classes like `:first-child` see the cleaned DOM. Files/pages where the selector matches nothing are skipped (`html filter '<selector>' matched nothing`). |
+| `--file-password <password>` | Password for encrypted files — PDFs (user-password protected) and Word documents (encrypted `.docx`/`.doc`). Applies to `--path` and `--url` ingestion. Owner-restricted PDFs without a user password (readable but with copy/print flags set) are read without it. |
 | `--text <text>` / `--stdin` | Manual entry text (chunked the same way as documents) |
 | `--title <title>` | Title for manual entries |
 | `--source-name <name>` | Source name for manual entries (default `manual`) |
@@ -116,7 +153,7 @@ Feeding is idempotent: a chunk whose `(source, chunk_index, content hash)` alrea
 
 While feeding, live progress is shown on stderr: a bar for the currently processed file (chunks processed, chars) plus a total line with files processed/total, cumulative chars and the number of rejected/ignored files — rejected files are reported but excluded from the progress percentage. In a non-interactive terminal (or when piped), plain per-file lines are printed instead. After a successful non-dry-run feed, a `this run` summary prints the same statistics as `gresmcp ks stats` (chunks, distinct sources, chars, avg/min/max per chunk), scoped to the chunks written during that run.
 
-For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. With `--url-filter`, only same-host links matching at least one glob are followed (the seed itself is exempt). Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
+For `--url`, page titles come from `<title>`/`og:title`/`<h1>` and each page's metadata records `source_url`, `crawled_at`, the meta `description` (if present) and tags. Linked PDFs are ingested too: a PDF page is parsed to Markdown (`pdf_pages` / `pdf_image_only_pages` metadata recorded), while image-only (scanned) PDFs with no text layer are skipped with a clear reason. Linked Word documents (`.docx`/`.doc`) are ingested as well — `doc_format` metadata records the detected format. Linked EPUBs (`.epub`, served as `application/epub+zip`) are parsed like local files (`epub_chapters` / `epub_author` / `epub_language` metadata recorded). With `--crawler playwright|puppeteer`, PDFs, Word documents and EPUBs are detected by their URL extension and downloaded directly (browsers would otherwise render PDFs in the internal viewer or download DOC files); use `auto`/`crawlee` for content-type-accurate detection. Servers mislabeling downloads as `application/octet-stream` are handled by sniffing the docx/doc magic bytes (or, for `.epub` URLs, by handing the payload to the EPUB parser). Crawling stays on the seed's hostname; non-HTTP links and obvious binary URLs are skipped. With `--url-filter`, only same-host links matching at least one glob are followed (the seed itself is exempt). Requests are made without consulting robots.txt, and CWD is never polluted (crawlee state goes to a temp directory).
 
 ### `gresmcp init`
 
@@ -226,9 +263,9 @@ npm test             # vitest (DB integration tests skip gracefully when Postgre
 
 Release checklist: `npm version <x.y.z> && npm publish --access public` (the scoped package name `@gresmcp/mcp` must be owned by your npm org).
 
-## Contributors
+## Contributors - THANK YOU
 
-gresmcp is built on top of these great projects and tools:
+`gresmcp` is built on top of these great projects and tools:
 
 | Contributor | Description |
 | --- | --- |
@@ -240,6 +277,11 @@ gresmcp is built on top of these great projects and tools:
 | [OpenChamber](https://openchamber.dev) | Agentic development environment for AI coding across desktop, browser, phone, and VS Code |
 | [Ollama](https://ollama.com) | The easiest way to automate your work using open models, while keeping your data safe |
 | [Nano](https://nano.org) | The fast, feeless and eco-friendly digital currency |
-| [NanoGPT](https://nano-gpt.com) | Subscription-free, pay-as-you-go access to AI models |
+| [NanoGPT](https://nano-gpt.com) | Subscription-free, pay-as-you-go access to huge selecion of AI models with strict retention policy, privacy and server location |
 | [Crawlee](https://crawlee.dev/) | Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living. |
+| [pdf2md](https://github.com/opengovsg/pdf2md) | Amazing JavaScript npm library to parse PDF files and convert them into Markdown |
+| [Mammoth](https://github.com/mwilliamson/mammoth.js) | Convert .docx documents, such as those created by Microsoft Word, Google Docs and LibreOffice, and convert them to HTML. |
+| [word-extractor](https://github.com/morungos/node-word-extractor) | Read data from Word .doc files (97-2003) in pure Node.js — no Office installation required |
+| [office-crypto](https://www.npmjs.com/package/office-crypto) | Decrypt password-protected MS Office files (DOCX/DOC), a TypeScript port of msoffcrypto-tool |
+| [epub2](https://www.npmjs.com/package/epub2) | Parse EPUB ebook files in Node.js — metadata, spine order and chapter extraction |
 | {{ YOU }} | This could be you — issues and PRs welcome at [gitea.com/fairking/gresmcp](https://gitea.com/fairking/gresmcp) |