@jcode.labs/mimir 0.4.2 → 0.4.4

package/README.md

@@ -1,552 +1,63 @@
-# Mimir
+# Mimir Core Package
 
-[![CI](https://github.com/jcode-works/jcode-mimir/actions/workflows/ci.yml/badge.svg)](https://github.com/jcode-works/jcode-mimir/actions/workflows/ci.yml)
-[![CodeQL](https://github.com/jcode-works/jcode-mimir/actions/workflows/codeql.yml/badge.svg)](https://github.com/jcode-works/jcode-mimir/actions/workflows/codeql.yml)
-[![npm](https://img.shields.io/npm/v/@jcode.labs/mimir)](https://www.npmjs.com/package/@jcode.labs/mimir)
-[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/jcode-works/jcode-mimir/blob/main/LICENSE)
+`@jcode.labs/mimir` is the core package for Mimir, an open-source sovereign local RAG toolkit for
+confidential datasets and AI agents.
 
-Open-source, sovereign local RAG for confidential datasets and AI agents.
+**Full documentation:** https://github.com/jcode-works/jcode-mimir#readme
 
-Mimir provides a TypeScript CLI, library, MCP server, and portable agent skills that can be
-installed in any Node.js repository. It indexes local files from the target repository, stores
-vectors locally with LanceDB, and can use either built-in local-hash retrieval or optional
-Transformers.js semantic embeddings. Mimir core returns cited retrieval context; answer synthesis
-belongs to the AI agent, LLM, or local model runtime you choose around it.
+This npm README is intentionally short because package READMEs are displayed separately on npm. The
+GitHub root README is the canonical product documentation.
 
-The intended use case is simple: put confidential company, institutional, legal, operational, or
-research documents in a private local folder, index them locally, then let any compatible AI agent or
-LLM workflow retrieve grounded context for summaries, briefs, audits, and decision support without
-shipping the dataset to a hosted RAG service.
+## What It Does
 
-Created by Jean-Baptiste Thery and published under the JCode Labs npm scope.
+Mimir lets a Node.js repository keep a local knowledge base next to its private documents. It indexes
+supported local files, stores the generated retrieval index in the target repository, and exposes the
+same evidence through:
 
-Built by Jean-Baptiste Thery, freelance full-stack/AI tooling engineer at JCode Labs.
+- the `kb` CLI;
+- a TypeScript library API;
+- a local MCP stdio server for compatible AI agents;
+- portable agent skills copied with `kb install-skill`.
 
-## Open Source
+Mimir does not send documents to a hosted RAG service and does not generate final LLM answers in
+core. It returns cited retrieval context so the agent or model you trust can write from local
+evidence.
 
-Mimir is a public open-source project under the MIT License. It is designed to be
-inspectable, forkable, and usable without a JCode Labs account.
+## Use It For
 
-Contributions are welcome through pull requests. Start with
-[`CONTRIBUTING.md`](https://github.com/jcode-works/jcode-mimir/blob/main/CONTRIBUTING.md).
-Security reports should stay private and follow the policy in
-[`SECURITY.md`](https://github.com/jcode-works/jcode-mimir/blob/main/SECURITY.md).
+- private project, legal, operational, research, or institutional dossiers;
+- codebase and architecture context retrieval;
+- local-first agent workflows with bounded MCP access;
+- cited summaries, audits, briefs, and decision support.
 
-## Sponsors
-
-Mimir stays MIT open source. Sponsorship helps fund maintenance, issue triage,
-documentation, and practical agent-workflow improvements.
-
-Sponsor the project through [GitHub Sponsors](https://github.com/sponsors/jb-thery).
-
-Suggested GitHub Sponsors tiers:
-
-- EUR 5/month: support the project.
-- EUR 15/month: active sponsor.
-- EUR 49/month: priority on issues and questions.
-- EUR 199/month: company sponsor and light advisory support.
-
-## Status
-
-Early public package. APIs may evolve before `1.0.0`.
-
-## Documentation
-
-- [Getting started](https://github.com/jcode-works/jcode-mimir/blob/main/docs/getting-started.md):
-  install Mimir and complete the first local search.
-- [CLI reference](https://github.com/jcode-works/jcode-mimir/blob/main/docs/cli-reference.md):
-  every `kb` and `mimir-tts` command with practical usage notes.
-- [Troubleshooting](https://github.com/jcode-works/jcode-mimir/blob/main/docs/troubleshooting.md):
-  common install, indexing, retrieval, audio, and release issues.
-- [Security hardening](https://github.com/jcode-works/jcode-mimir/blob/main/SECURITY-HARDENING.md):
-  offline operation, threat model, secure deletion limits, and release verification.
-
-## What Mimir Is For
-
-- Build a local RAG knowledge base inside any repository.
-- Analyze confidential datasets while keeping raw files and generated indexes local.
-- Give Claude, Codex, Cursor, internal assistants, or other MCP-compatible tools the same private
-  retrieval layer.
-- Retrieve grounded local evidence through CLI, library calls, MCP tools, or the bundled agent
-  skills so your chosen AI agent can produce cited summaries.
-- Optionally create listenable MP3 or WAV summaries with `kb audio`, `@jcode.labs/mimir-tts`, and
-  the bundled `mimir-audio-summary` skill.
-
-Mimir is not a hosted SaaS, not a remote vector database, and not a certified high-assurance system.
-For regulated or state-grade environments, pair it with encrypted disks, controlled machines, release
-verification, and an external security review.
-
-## Use Cases
-
-Mimir is useful whenever the source material should stay local but an AI agent still needs grounded
-context.
-
-| Use case | Example questions |
-| --- | --- |
-| Understand a code repository | "Where is authentication implemented?", "What depends on this module?", "Summarize the payment flow." |
-| Understand architecture | "What services exist?", "What are the data boundaries?", "Which components are risky to change?" |
-| Analyze specifications | "What does the technical spec require?", "Which requirements are still unclear?", "Generate an implementation checklist." |
-| Work through a request for proposal or tender | "What are the mandatory constraints?", "Which documents prove compliance?", "What risks should be clarified?" |
-| Study courses and training material | "Summarize chapter three.", "Create revision questions.", "Compare these two concepts." |
-| Analyze a book or long report | "Extract the main thesis.", "Find recurring arguments.", "Create a chapter-by-chapter brief." |
-| Build an internal knowledge base | "What is the policy for incident review?", "Who owns this process?", "Which source says that?" |
-| Prepare meetings or decisions | "Give me a one-page briefing.", "What is missing before deciding?", "List action items and evidence." |
-| Ask questions over offline documents | "Which files mention local-only operation?", "What evidence supports this claim?" |
-| Generate audio briefings | "Create a listenable high-quality or offline summary of the current dossier." |
-
-## Requirements
-
-- Node.js 20+
-- pnpm, npm, yarn or bun
-- No model runtime is required for the default `embeddingProvider: "local-hash"` mode.
-- Optional semantic embeddings use Transformers.js with local model files under `.mimir/models` by
-  default.
-- Generated answers are intentionally outside Mimir core. Use Claude, Codex, OpenAI, a local model
-  MCP server, or another trusted model runtime to synthesize from Mimir's cited context.
-- Optional audio summaries use the separate `@jcode.labs/mimir-tts` workspace package. For the
-  highest quality, install the external `edge-tts` CLI and render Edge MP3 output with
-  `fr-FR-DeniseNeural`. For confidential or air-gapped content, use the Transformers.js WAV path
-  with `--engine transformers --offline`; it does not require Python, ffmpeg, Piper, XTTS, or a
-  local server.
-
-## Install From npm
-
-The package is public. Users do not need a JCode Labs account or npm token to install it.
-
-With pnpm:
+## Install
 
 ```bash
 pnpm add -D @jcode.labs/mimir
 ```
 
-With npm:
-
-```bash
-npm install --save-dev @jcode.labs/mimir
-```
-
-Maintainer tokens are only needed to publish new versions.
-
-## Install From Source Checkout
-
-```bash
-git clone git@github.com:jcode-works/jcode-mimir.git
-cd jcode-mimir
-pnpm install
-pnpm build
-```
-
-For local development:
-
-```bash
-pnpm add -D file:../jcode-mimir/packages/mimir
-```
-
-Before creating an npm tarball later, run:
-
-```bash
-pnpm build
-pnpm --dir packages/mimir pack
-```
-
-## Use In Any Repository
-
-Initialize the local project config:
+## Quick Start
 
 ```bash
 pnpm exec kb init
 pnpm exec kb doctor
-```
-
-Add private documents under `private/`, then run:
-
-```bash
-pnpm exec kb ingest
-pnpm exec kb doctor
-pnpm exec kb search "vendor invoice status"
-pnpm exec kb ask "What do the documents prove?"
-pnpm exec kb audit
-pnpm exec kb security-audit
-pnpm exec kb status
-```
-
-With npm, use `npx` after installing the package:
-
-```bash
-npx kb init
-npx kb doctor
-npx kb ingest
-npx kb doctor
-npx kb search "vendor invoice status"
-npx kb ask "What do the documents prove?"
-npx kb audit
-npx kb security-audit
-npx kb status
-```
-
-## Choose A Retrieval Mode
-
-Mimir has two embedding modes.
-
-### Default Local Hash Retrieval
-
-Use this when you want a fully local, no-model smoke test or a dependency-light setup. Retrieval is
-lexical/hash-based, not semantic.
-
-`.kb/config.json`:
-
-```json
-{
-  "embeddingProvider": "local-hash"
-}
-```
-
-Commands:
-
-```bash
 pnpm exec kb ingest
-pnpm exec kb search "offline retrieval approval"
-pnpm exec kb ask "What evidence supports offline operation?"
-```
-
-`kb ask` always returns cited retrieved passages instead of a generated synthesis. You can pass those
-passages to any LLM or agent you trust.
-
-### Optional Semantic Embeddings With Transformers.js
-
-Use this when you want better semantic retrieval while keeping Mimir core free of an LLM server.
-
-`.kb/config.json`:
-
-```json
-{
-  "embeddingProvider": "transformers",
-  "embeddingModel": "mixedbread-ai/mxbai-embed-xsmall-v1",
-  "embeddingModelPath": ".mimir/models",
-  "transformersAllowRemoteModels": false
-}
-```
-
-Commands:
-
-```bash
-pnpm exec kb ingest
-pnpm exec kb ask "Which passages support offline operation?"
-```
-
-Keep `transformersAllowRemoteModels` false for confidential or air-gapped work and preload model
-files into `embeddingModelPath`. Set it to true only when you explicitly allow Transformers.js to
-download model files from Hugging Face.
-
-## Dependency Footprint
-
-Mimir can run retrieval without a model runtime. Some runtime dependencies remain because they own
-core features:
-
-| Dependency | Why it remains |
-| --- | --- |
-| @huggingface/transformers | optional local semantic embeddings |
-| LanceDB | local vector storage and nearest-neighbor retrieval |
-| MCP SDK | MCP server for compatible agents |
-| fast-glob | safe source-file discovery |
-| unpdf, html-to-text, yaml, fflate | document parsing for PDF, HTML, YAML, Office/OpenDocument ZIP files |
-| commander, zod, picocolors | CLI, config validation, readable terminal output |
-
-Removing more dependencies is possible only by dropping features or replacing them with smaller
-internal implementations. The current low-friction path is dependency-light at runtime for users who
-choose `local-hash`, while preserving richer parsing, MCP support, and optional semantic embeddings.
-
-## Example Test Workspace
-
-This repository includes a synthetic example under
-[`examples/sovereign-rag-demo`](./examples/sovereign-rag-demo). It can be used to test ingestion,
-retrieval, `security-audit`, and custom text extensions without using private documents.
-
-From a local checkout:
-
-```bash
-pnpm build
-cd examples/sovereign-rag-demo
-node ../../dist/cli.js security-audit
-node ../../dist/cli.js ingest
-node ../../dist/cli.js search "offline retrieval approval"
-node ../../dist/cli.js audit
-```
-
-The example uses the default local-hash retrieval mode, so it can run without downloading an
-embedding or chat model.
-
-## Typical Workflows
-
-### Understand A Codebase
-
-```bash
-pnpm exec kb init
-printf "src\nREADME.md\ndocs\n" >> .kb/sources.txt
-pnpm exec kb ingest
-pnpm exec kb search "authentication flow"
-pnpm exec kb ask "Explain the architecture and cite the relevant files."
-```
-
-### Analyze Specifications Or A Course
-
-```bash
-pnpm exec kb ingest
-pnpm exec kb ask "Summarize the requirements and list open questions."
-pnpm exec kb ask "Create revision questions from the indexed course material."
-```
-
-### Work Offline
-
-```bash
-pnpm exec kb security-audit --strict
-pnpm exec kb ingest
-pnpm exec kb search "incident review policy"
-pnpm exec kb ask "What does the local evidence prove?"
-```
-
-Use `embeddingProvider: "local-hash"` for a no-model offline workflow. Use
-`embeddingProvider: "transformers"` with preloaded model files for semantic offline retrieval.
-Generated answers should come from a trusted external agent or model runtime.
-
-### Generate An Audio Briefing
-
-Mimir includes a plug-and-play text-to-speech path for listenable summaries. For the same quality
-path as the global Voice Forge skill, install `edge-tts` and render MP3:
-
-```bash
-pnpm exec kb audio --doctor
-pipx install edge-tts
-pnpm exec kb audio /tmp/MIMIR-SUMMARY-project.txt \
-  --engine edge \
-  --out .mimir/audio/project-summary.mp3
-```
-
-The Edge path uses the online Microsoft Edge TTS service through the `edge-tts` CLI. Use it only
-when sending the narration text to that service is acceptable.
-
-By default, `kb audio` uses the Transformers.js WAV path. For confidential or air-gapped work,
-preload Transformers.js-compatible model files and render WAV offline:
-
-```bash
-pnpm exec kb audio /tmp/MIMIR-SUMMARY-project.txt \
-  --engine transformers \
-  --offline \
-  --model-path .mimir/models/tts \
-  --out .mimir/audio/project-summary.wav
-```
-
-The standalone package can also be installed directly:
-
-```bash
-pnpm add -D @jcode.labs/mimir-tts
-pnpm exec mimir-tts render /tmp/MIMIR-SUMMARY-project.txt \
-  --engine edge \
-  --out .mimir/audio/project-summary.mp3
-```
-
-## Agent Skills And MCP
-
-Mimir ships with portable agent skills and a standard MCP server.
-
-Install the agent kit into a repository:
-
-```bash
+pnpm exec kb search "your question"
+pnpm exec kb ask "your question"
 pnpm exec kb install-skill
 ```
 
-This creates:
-
-```plain text
-.mimir/skills/mimir/SKILL.md
-.mimir/skills/mimir-audio-summary/SKILL.md
-.mimir/mcp.json
-.mimir/README.md
-```
-
-Agents that support skill folders can load `.mimir/skills/mimir/` for deep local RAG usage.
-Load `.mimir/skills/mimir-audio-summary/` only when an optional spoken summary is needed.
-Other agents can read the generated `.mimir/README.md` and use the MCP config snippet.
-
-Start the MCP server from the repository root:
-
-```bash
-pnpm exec kb serve-mcp
-```
-
-MCP tools exposed:
-
-- `mimir_status`
-- `mimir_search`
-- `mimir_ask`
-- `mimir_audit`
-- `mimir_security_audit`
-
-This MCP layer is the recommended way to let any compatible LLM or agent query the same local
-knowledge base. The LLM does not need to know about LanceDB or the raw file layout; it asks Mimir for
-ranked passages or cited context and uses the returned citations.
-
-Print the bundled skill path from the installed package:
-
-```bash
-pnpm exec kb skill-path
-```
-
-## Data Boundary
-
-The package code lives in `node_modules` or in this repository. Project data stays in the
-repository where you run the CLI:
-
-```plain text
-your-project/
-  private/          # raw documents to ingest
-  .kb/config.json   # local config
-  .kb/sources.txt   # optional extra source paths
-  .kb/storage/      # generated LanceDB index
-  .kb/access.log    # metadata-only access log
-```
-
-The package never ships project documents. `kb init` adds gitignore entries for `.kb/`
-and `private/**`, and `kb install-skill` keeps `.mimir/` ignored as generated local agent
-state.
-
-## Confidentiality Defaults
-
-Mimir is designed for private repositories and sensitive local evidence.
-
-- Zero telemetry: no analytics or document content is sent to JCode Labs.
-- No LLM generation in core: Mimir returns cited context for the agent/runtime you choose.
-- Local-hash by default: no model runtime is required for the default retrieval path.
-- Transformers.js remote model loading is disabled by default.
-- Redaction before indexing: common secrets and identifiers are redacted before chunks are
-  embedded and stored.
-- Metadata-only access logs: query hashes and action metadata are logged, not raw queries.
-- MCP is read-focused and bounded by `mcpMaxTopK`.
-- Generated local state is ignored by Git.
-
-Run:
-
-```bash
-pnpm exec kb security-audit --strict
-```
-
-Remove the generated vector index:
-
-```bash
-pnpm exec kb destroy-index --yes
-```
-
-For air-gapped operation, release verification, secure deletion limits, and threat model details,
-read
-[`SECURITY-HARDENING.md`](https://github.com/jcode-works/jcode-mimir/blob/main/SECURITY-HARDENING.md).
-
-## Supported Files
-
-Mimir supports common text, document, data, config, log, and source-code files out of the box:
-
-- Markdown: `.md`, `.mdx`
-- Text: `.txt`, `.text`
-- JSON: `.json`
-- YAML: `.yaml`, `.yml`
-- CSV/TSV: `.csv`, `.tsv`
-- HTML: `.html`, `.htm`
-- PDF: `.pdf`
-- Office/OpenDocument: `.docx`, `.pptx`, `.xlsx`, `.odt`, `.ods`, `.odp`
-- Rich text: `.rtf`
-- Line data and logs: `.jsonl`, `.ndjson`, `.log`
-- XML feeds and documents: `.xml`, `.rss`, `.atom`
-- Config and data files: `.toml`, `.ini`, `.conf`, `.cfg`, `.properties`, `.sql`
-- Source code: `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, `.rs`, `.java`, `.rb`, `.php`,
-  `.cs`, `.c`, `.cpp`, `.h`, `.css`
-
-Custom UTF-8 text extensions can be enabled without changing code:
-
-```json
-{
-  "includeExtensions": [".transcript", ".evidence"]
-}
-```
-
-Or through:
-
-```bash
-KB_INCLUDE_EXTENSIONS=".transcript,.evidence" pnpm exec kb ingest
-```
-
-Images, scans, audio/video files, old proprietary Office binaries such as `.doc`, and other formats
-that are not listed should be OCRed, transcribed, converted, or exported to text/PDF/HTML first.
-Mimir intentionally avoids pretending that every binary format can be indexed safely without
-extraction logic.
-
-## Config
-
-`.kb/config.json`:
-
-```json
-{
-  "rawDir": "private",
-  "storageDir": ".kb/storage",
-  "sourcesFile": ".kb/sources.txt",
-  "accessLogPath": ".kb/access.log",
-  "embeddingModelPath": ".mimir/models",
-  "tableName": "chunks",
-  "embeddingProvider": "local-hash",
-  "embeddingModel": "mixedbread-ai/mxbai-embed-xsmall-v1",
-  "transformersAllowRemoteModels": false,
-  "redaction": {
-    "enabled": true,
-    "builtIn": true,
-    "patterns": []
-  },
-  "accessLog": true,
-  "mcpMaxTopK": 10,
-  "topK": 5,
-  "chunkSize": 1200,
-  "chunkOverlap": 150,
-  "includeExtensions": []
-}
-```
-
-Environment overrides:
-
-- `KB_RAW_DIR`
-- `KB_STORAGE_DIR`
-- `KB_SOURCES_FILE`
-- `KB_ACCESS_LOG_PATH`
-- `KB_EMBEDDING_PROVIDER`
-- `KB_EMBEDDING_MODEL`
-- `KB_EMBEDDING_MODEL_PATH`
-- `KB_TRANSFORMERS_ALLOW_REMOTE_MODELS`
-- `KB_REDACTION_ENABLED`
-- `KB_REDACTION_BUILT_IN`
-- `KB_ACCESS_LOG`
-- `KB_MCP_MAX_TOP_K`
-- `KB_TOP_K`
-- `KB_CHUNK_SIZE`
-- `KB_CHUNK_OVERLAP`
-- `KB_INCLUDE_EXTENSIONS`
-
-## Library API
-
-```ts
-import { ingest, search, ask } from "@jcode.labs/mimir"
-
-await ingest({ rebuild: true })
-const results = await search("vendor invoice status")
-const answer = await ask("What documents support the project timeline?")
-```
+By default, Mimir indexes documents from `private/`, stores generated state under `.kb/`, and keeps
+agent integration files under `.mimir/`. `kb init` adds the matching Git ignore entries for local
+generated and private data.
 
-## Privacy
+## Entry Points
 
-- Mimir core does not generate answers or call a chat model.
-- `local-hash` can run ingestion, search, and cited retrieval without a model runtime.
-- Transformers.js remote model loading is disabled by default.
-- Built-in redaction runs before indexing by default.
-- Access logs store query hashes, not raw queries.
-- The vector index is stored locally.
-- Raw private documents should stay in the target repository's ignored `private/` folder.
-- Do not put secrets or scans inside this package repository.
+- CLI: `kb`
+- Library import: `@jcode.labs/mimir`
+- MCP server: `pnpm exec kb serve-mcp`
+- Bundled skills: `pnpm exec kb install-skill`
 
 ## License
 
-MIT © Jean-Baptiste Thery.
+MIT (c) Jean-Baptiste Thery.

package/dist/version.d.ts

@@ -1,2 +1,2 @@
-export declare const VERSION = "0.4.2";
+export declare const VERSION = "0.4.4";
 //# sourceMappingURL=version.d.ts.map

package/dist/version.js

@@ -1,2 +1,2 @@
-export const VERSION = "0.4.2";
+export const VERSION = "0.4.4";
 //# sourceMappingURL=version.js.map

package/examples/sovereign-rag-demo/README.md

@@ -25,7 +25,7 @@ pnpm build
 Then run the CLI from this folder:
 
 ```bash
-cd examples/sovereign-rag-demo
+cd packages/mimir/examples/sovereign-rag-demo
 node ../../dist/cli.js security-audit
 node ../../dist/cli.js ingest
 node ../../dist/cli.js search "offline retrieval approval"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jcode.labs/mimir",
-  "version": "0.4.2",
+  "version": "0.4.4",
   "description": "Mimir: open-source sovereign local RAG for confidential datasets and AI agents.",
   "type": "module",
   "license": "MIT",
@@ -28,7 +28,7 @@
     "url": "git+https://github.com/jcode-works/jcode-mimir.git",
     "directory": "packages/mimir"
   },
-  "homepage": "https://github.com/jcode-works/jcode-mimir/tree/main/packages/mimir#readme",
+  "homepage": "https://github.com/jcode-works/jcode-mimir#readme",
   "bugs": {
     "url": "https://github.com/jcode-works/jcode-mimir/issues"
   },
@@ -65,7 +65,7 @@
     "unpdf": "^1.4.0",
     "yaml": "^2.8.1",
     "zod": "^4.1.13",
-    "@jcode.labs/mimir-tts": "0.4.2"
+    "@jcode.labs/mimir-tts": "0.4.4"
   },
   "devDependencies": {
     "@types/html-to-text": "^9.0.4",