@jcode.labs/mimir 0.2.1 → 0.4.0

package/skills/mimir/SKILL.md

@@ -5,7 +5,9 @@ description: Use this skill whenever a repository uses or should use Mimir, loca
 
 # Mimir
 
-Mimir is a local-first knowledge base for project documents. It indexes files from the current repository, stores vectors locally, and exposes both a CLI and an MCP server.
+Mimir is a sovereign local RAG knowledge base for confidential project documents and datasets. It
+indexes files from the current repository, stores vectors locally, and exposes both a CLI and an MCP
+server.
 
 Use this skill to help an AI agent work with a Mimir-enabled repository without leaking private documents or relying on stale memory.
 
@@ -20,12 +22,15 @@ private/          # raw documents to ingest
 .kb/config.json   # local Mimir config
 .kb/sources.txt   # optional extra source paths
 .kb/storage/      # generated local index
+.kb/access.log    # metadata-only access log
 ```
 
 ## Data Safety
 
 - Do not commit raw documents, secrets, tax IDs, scans, bank documents, tokens, or generated vector stores.
-- Keep `private/**` and `.kb/storage/**` ignored by Git.
+- Keep `private/**`, `.kb/`, and `.mimir/` ignored by Git.
+- Treat `kb search`, `kb ask`, and MCP results as sensitive because they can contain private
+  source passages even when redaction is enabled.
 - Prefer summaries and citations over dumping long private passages into the chat.
 - If the user asks for a high-stakes answer, identify which facts came from Mimir and which still require professional or official verification.
 
@@ -35,6 +40,7 @@ From the repository root:
 
 ```bash
 pnpm exec kb status
+pnpm exec kb security-audit
 ```
 
 If Mimir is not installed:
@@ -51,6 +57,34 @@ npm install --save-dev @jcode.labs/mimir
 npx kb init
 ```
 
+## Provider Modes
+
+Default retrieval mode:
+
+```json
+{
+  "embeddingProvider": "local-hash"
+}
+```
+
+This supports ingestion, search, MCP retrieval, and `kb ask` with cited passages without a model
+server. It is lexical/hash retrieval, not model-semantic search. Do not present it as equivalent to
+semantic embeddings.
+
+Optional semantic embedding mode:
+
+```json
+{
+  "embeddingProvider": "transformers",
+  "embeddingModel": "mixedbread-ai/mxbai-embed-xsmall-v1",
+  "embeddingModelPath": ".mimir/models",
+  "transformersAllowRemoteModels": false
+}
+```
+
+This uses Transformers.js for embeddings only. Keep `transformersAllowRemoteModels` false for
+air-gapped or confidential work and preload model files under `embeddingModelPath`.
+
 ## Ingestion Workflow
 
 After documents are added or changed:
@@ -58,10 +92,12 @@ After documents are added or changed:
 ```bash
 pnpm exec kb ingest
 pnpm exec kb audit
+pnpm exec kb security-audit
 pnpm exec kb status
 ```
 
-The audit must show no missing or stale supported files before relying on the index.
+The audit must show no missing or stale supported files before relying on the index. The security
+audit should not show warnings before relying on Mimir for sensitive work.
 
 ## Query Workflow
 
@@ -71,13 +107,32 @@ Use search when you need exact source passages:
 pnpm exec kb search "your query"
 ```
 
-Use ask when you need a synthesized answer with citations:
+Use ask when you need cited context for the current agent or an external LLM:
 
 ```bash
 pnpm exec kb ask "your question"
 ```
 
-Ground answers in returned sources. If search results are weak, say that the current index does not prove the point and ask for the missing document.
+Ground answers in returned sources. If search results are weak, say that the current index does not
+prove the point and ask for the missing document. `kb ask` returns cited passages rather than LLM
+synthesis. Use those passages as context for the current agent, or tell the user that generative
+synthesis needs a trusted external LLM or model runtime.
+
+## Deep Research Workflow
+
+For broad summaries, audits, planning, or institutional dossiers, do not rely on one query. Build a
+small retrieval plan first:
+
+- check `kb audit` and `kb security-audit`;
+- query the main topic;
+- query names, dates, amounts, obligations, risks, decisions, and missing evidence separately;
+- compare the strongest passages across files;
+- ask a synthesis question only after search has found enough grounded context;
+- cite source paths and chunk numbers in the answer when useful;
+- explicitly say when the index does not prove a claim.
+
+For sensitive work, prefer the smallest useful `topK`; raise it only when the first results are too
+thin. Do not dump large raw passages into the chat unless the user explicitly asks for extracts.
 
 ## MCP Usage
 
@@ -99,11 +154,27 @@ Available MCP tools:
 
 - `mimir_status`: show config and chunk count.
 - `mimir_search`: retrieve source passages.
-- `mimir_ask`: synthesize an answer with local citations.
+- `mimir_ask`: return cited retrieval context.
 - `mimir_audit`: compare source files with the current index.
+- `mimir_security_audit`: inspect local privacy, provider, redaction, MCP, and gitignore posture.
 
 Prefer MCP tools over shell commands when the agent runtime provides them. Use shell commands when MCP is unavailable.
 
+MCP is read-focused and intentionally does not expose index deletion. Use `pnpm exec kb
+destroy-index --yes` from the shell when the user explicitly wants to remove the generated index.
+
+## Optional Audio Summaries
+
+If the user asks for a listenable or TTS summary, load the optional
+`.mimir/skills/mimir-audio-summary/` skill installed by `pnpm exec kb install-skill`.
+
+That skill should:
+
+- gather evidence through Mimir first;
+- write narration text only to a temp file outside the repository;
+- render generated audio under `.mimir/audio/` by default;
+- prefer offline TTS engines for confidential content.
+
 ## Installing This Skill Into A Repository
 
 Run:

package/skills/mimir-audio-summary/SKILL.md

@@ -0,0 +1,134 @@
+---
+name: mimir-audio-summary
+description: >-
+  Create an optional spoken audio summary from a Mimir local knowledge base. Use when the user asks
+  for an audio, TTS, spoken brief, briefing, narration, or listenable summary based on private
+  repository documents indexed by Mimir. The skill is confidentiality-first: gather facts through
+  Mimir, write only a temporary narration text file outside the repository, and render the final
+  audio under ignored local Mimir state unless the user explicitly chooses another output path.
+---
+
+# Mimir Audio Summary
+
+Use this skill to turn a confidential local Mimir knowledge base into an optional audio summary.
+The knowledge base stays local; the final audio is a generated artifact and must not be committed.
+
+## Confidentiality Rules
+
+- Treat the source documents, retrieved passages, generated narration, and final audio as sensitive.
+- Do not use online TTS for confidential content unless the user explicitly allows it.
+- Prefer `pnpm exec kb audio` or `pnpm exec mimir-tts render` for plug-and-play local WAV output.
+- Use `--offline` when model files are already present and remote model loading is not allowed.
+- Write the narration text to a temp file outside the repository, such as `/tmp/MIMIR-SUMMARY-topic.txt`.
+- Render audio under `.mimir/audio/` by default. This directory is ignored by Git when Mimir is installed.
+- Never stage or commit generated audio, temporary text, WAV, AIFF, or intermediate files.
+
+## 1. Verify The Knowledge Base
+
+From the repository root, run:
+
+```bash
+pnpm exec kb status
+pnpm exec kb audit
+pnpm exec kb security-audit
+```
+
+If the audit reports missing or stale files, run:
+
+```bash
+pnpm exec kb ingest
+pnpm exec kb audit
+```
+
+Do not create an audio summary from stale or incomplete evidence unless the user explicitly accepts
+that limitation.
+
+## 2. Search Deeply Before Writing
+
+Use Mimir search or MCP tools to gather evidence before drafting the narration.
+
+For a broad summary, run multiple searches:
+
+```bash
+pnpm exec kb search "<main topic>" --top-k 8
+pnpm exec kb search "<people, dates, money, obligations, risks, or decisions>" --top-k 8
+pnpm exec kb ask "<specific synthesis question>" --top-k 8
+```
+
+When MCP is available, prefer `mimir_search`, `mimir_ask`, `mimir_audit`, and
+`mimir_security_audit` over shell commands.
+
+Keep citations in your working notes, but do not read long raw passages aloud. The audio should be a
+clear synthesis, not a dump of source text.
+
+## 3. Write For Listening
+
+Write one flowing narration in the user's working language. Do not use markdown, headings, bullets,
+tables, SSML, XML tags, or stage directions in the spoken text.
+
+Good audio structure:
+
+1. Start with the purpose of the summary and the two-to-four ideas to retain.
+2. Explain the current evidence in plain language.
+3. Separate proven facts from uncertainty.
+4. Highlight decisions, risks, deadlines, and missing documents.
+5. End with a concise recap and two or three self-check questions.
+
+Use short sentences and natural punctuation. Spell acronyms and symbols in a way a TTS engine can
+pronounce.
+
+## 4. Render The Audio
+
+Create the output directory and write the narration to a temp file outside the repo:
+
+```bash
+mkdir -p .mimir/audio
+```
+
+Then render with the default Mimir TTS path:
+
+```bash
+pnpm exec kb audio /tmp/MIMIR-SUMMARY-<subject-kebab>.txt \
+  --out .mimir/audio/MIMIR-SUMMARY-<subject-kebab>.wav
+```
+
+For air-gapped operation, preload the model files under `.mimir/models/tts` and run:
+
+```bash
+pnpm exec kb audio /tmp/MIMIR-SUMMARY-<subject-kebab>.txt \
+  --out .mimir/audio/MIMIR-SUMMARY-<subject-kebab>.wav \
+  --offline
+```
+
+The default renderer uses `@jcode.labs/mimir-tts` and Transformers.js. It does not require Python,
+ffmpeg, Piper, XTTS, or a local TTS server. The first non-offline render can download public model
+files into `.mimir/models/tts`, but the narration text is processed locally.
+
+Use the legacy voice-forge helper only when the user explicitly wants MP3 output or an engine not
+covered by Mimir TTS:
+
+```bash
+OUT_MP3="<repo-root>/.mimir/audio/MIMIR-SUMMARY-<subject-kebab>.mp3" \
+  TTS_ENGINE=auto \
+  bash <this-skill-dir>/forge-voice.sh /tmp/MIMIR-SUMMARY-<subject-kebab>.txt
+```
+
+Engine selection:
+
+- `auto`: confidentiality-first order: XTTS, macOS `say`, Piper, then online Edge only when
+  `MIMIR_ALLOW_ONLINE_TTS=1`.
+- `xtts`: local Coqui XTTS-v2.
+- `say`: local macOS speech engine, converted to MP3 with `ffmpeg`.
+- `piper`: local neural TTS, converted to MP3 with `ffmpeg`.
+- `edge`: online Edge TTS. Use only with explicit user approval for non-sensitive text.
+
+Voice can be selected with `TTS_VOICE`. Speed for Edge can be selected with `TTS_RATE`.
+
+## 5. Report The Result
+
+After rendering, report:
+
+- the audio path;
+- which renderer/model was used or requested;
+- whether remote model downloads or online TTS were allowed;
+- any evidence limitation, such as stale index, missing documents, or weak search results.

package/skills/mimir-audio-summary/forge-voice.sh

@@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+# Render a Mimir audio-summary text file to MP3. The text is a throwaway intermediate written
+# outside the repository by the skill. The final audio should normally be written under .mimir/audio.
+#
+# Engine via TTS_ENGINE (auto|xtts|say|piper|edge); default "auto" is confidentiality-first:
+#   xtts  - local Coqui XTTS-v2 when installed.
+#   say   - macOS built-in, offline, clean but robotic.
+#   piper - local neural TTS.
+#   edge  - online Edge TTS only when TTS_ENGINE=edge or MIMIR_ALLOW_ONLINE_TTS=1.
+#
+# Usage:  forge-voice.sh <text-file> [voice]
+# Env:    OUT_MP3                 explicit mp3 output path
+#         TTS_ENGINE              force an engine (default: auto)
+#         TTS_VOICE               voice/speaker (engine-specific); the [voice] arg overrides it
+#         TTS_RATE                edge-tts speed delta, default +0%
+#         TTS_SEGMENT=1           edge: render sentence-by-sentence
+#         XTTS_SPEAKER            xtts preset speaker (default Ana Florence)
+#         PIPER_MODEL             piper onnx model path
+#         MIMIR_ALLOW_ONLINE_TTS=1 allow edge in auto mode
+#         KEEP_TEXT=1             keep the source text file after a successful render
+set -euo pipefail
+
+TXT="${1:?usage: forge-voice.sh <text-file> [voice]}"
+[ -f "$TXT" ] || { echo "error: file not found: $TXT" >&2; exit 1; }
+VOICE="${2:-${TTS_VOICE:-}}"
+ENGINE="${TTS_ENGINE:-auto}"
+OUT_FINAL="${OUT_MP3:-${TXT%.txt}.mp3}"
+OUTBASE="${OUT_FINAL%.mp3}"
+
+cleanup() {
+  [ "${KEEP_TEXT:-0}" = "1" ] || rm -f "$TXT"
+  rm -f "${OUTBASE}.wav" "${OUTBASE}.aiff"
+}
+trap cleanup EXIT
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+XTTS_PY="${XTTS_PY:-$HOME/.local/share/voice-forge/xtts/bin/python}"
+XTTS_HELPER="$SCRIPT_DIR/xtts-voice.py"
+
+xtts_ready() { [ -x "$XTTS_PY" ] && [ -f "$XTTS_HELPER" ]; }
+
+finish() {
+  echo "$1"
+  exit 0
+}
+
+to_mp3() {
+  local src="$1" out="${OUTBASE}.mp3"
+  if ! command -v ffmpeg >/dev/null 2>&1; then
+    rm -f "$src"
+    echo "error: ffmpeg is required to convert local TTS output to mp3" >&2
+    exit 1
+  fi
+  if ffmpeg -y -loglevel error -i "$src" -ac 1 -c:a libmp3lame -q:a 4 "$out"; then
+    rm -f "$src"
+    finish "$out"
+  fi
+  rm -f "$src" "$out"
+  echo "error: failed to convert local TTS output to mp3" >&2
+  exit 1
+}
+
+if [ "$ENGINE" = "xtts" ] || { [ "$ENGINE" = "auto" ] && xtts_ready; }; then
+  if ! xtts_ready; then
+    echo "error: TTS_ENGINE=xtts but venv/helper missing ($XTTS_PY)" >&2
+    exit 1
+  fi
+  WAV="${OUTBASE}.wav"
+  if [ -n "$VOICE" ]; then
+    COQUI_TOS_AGREED=1 "$XTTS_PY" "$XTTS_HELPER" "$TXT" "$WAV" "$VOICE" >&2
+  else
+    COQUI_TOS_AGREED=1 "$XTTS_PY" "$XTTS_HELPER" "$TXT" "$WAV" >&2
+  fi
+  to_mp3 "$WAV"
+fi
+
+if [ "$ENGINE" = "say" ] || [ "$ENGINE" = "auto" ]; then
+  if command -v say >/dev/null 2>&1; then
+    AIFF="${OUTBASE}.aiff"
+    say -v "${VOICE:-Jacques}" -f "$TXT" -o "$AIFF" 2>/dev/null || say -f "$TXT" -o "$AIFF"
+    to_mp3 "$AIFF"
+  fi
+  if [ "$ENGINE" = "say" ]; then
+    echo "error: TTS_ENGINE=say but 'say' not available" >&2
+    exit 1
+  fi
+fi
+
+if [ "$ENGINE" = "piper" ] || [ "$ENGINE" = "auto" ]; then
+  if command -v piper >/dev/null 2>&1; then
+    WAV="${OUTBASE}.wav"
+    piper -m "${PIPER_MODEL:-fr_FR-siwis-medium.onnx}" -f "$WAV" < "$TXT"
+    to_mp3 "$WAV"
+  fi
+  if [ "$ENGINE" = "piper" ]; then
+    echo "error: TTS_ENGINE=piper but piper not installed (pip install piper-tts)" >&2
+    exit 1
+  fi
+fi
+
+if [ "$ENGINE" = "edge" ] || { [ "$ENGINE" = "auto" ] && [ "${MIMIR_ALLOW_ONLINE_TTS:-0}" = "1" ]; }; then
+  if command -v edge-tts >/dev/null 2>&1; then
+    OUT="${OUTBASE}.mp3"
+    SPLIT="$SCRIPT_DIR/split-lines.py"
+    voice="${VOICE:-fr-FR-DeniseNeural}"
+    rate="${TTS_RATE:-+0%}"
+    if [ "${TTS_SEGMENT:-0}" = "1" ] && [ -f "$SPLIT" ] \
+       && command -v ffmpeg >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1; then
+      TMP="$(mktemp -d)"
+      ffmpeg -y -loglevel error -f lavfi -i anullsrc=r=24000:cl=mono -t 0.28 \
+        -c:a libmp3lame -q:a 4 "$TMP/sil.mp3"
+      i=0
+      : > "$TMP/list.txt"
+      while IFS= read -r line; do
+        [ -z "$line" ] && continue
+        i=$((i + 1))
+        if edge-tts --text "$line" --voice "$voice" --rate="$rate" \
+          --write-media "$TMP/seg_$i.mp3" >/dev/null 2>&1; then
+          printf "file '%s'\n" "$TMP/seg_$i.mp3" >> "$TMP/list.txt"
+          printf "file '%s'\n" "$TMP/sil.mp3" >> "$TMP/list.txt"
+        else
+          echo "warn: edge-tts failed on a sentence, skipping it" >&2
+        fi
+      done < <(python3 "$SPLIT" "$TXT")
+      ffmpeg -y -loglevel error -f concat -safe 0 -i "$TMP/list.txt" \
+        -ac 1 -c:a libmp3lame -q:a 4 "$OUT"
+      rm -rf "$TMP"
+      finish "$OUT"
+    fi
+    edge-tts --file "$TXT" --voice "$voice" --rate="$rate" --write-media "$OUT" >/dev/null
+    finish "$OUT"
+  fi
+  if [ "$ENGINE" = "edge" ]; then
+    echo "error: TTS_ENGINE=edge but edge-tts not installed (pipx install edge-tts)" >&2
+    exit 1
+  fi
+fi
+
+cat >&2 <<'EOF'
+error: no offline TTS engine available. Install one:
+  XTTS-v2 (local):
+    uv venv --python 3.11 ~/.local/share/voice-forge/xtts
+    uv pip install --python ~/.local/share/voice-forge/xtts/bin/python \
+       coqui-tts 'transformers>=4.57,<5' 'torch==2.8.*' 'torchaudio==2.8.*'
+  piper (local): pip install piper-tts
+  ffmpeg: required for local engine MP3 conversion
+
+Online fallback:
+  pipx install edge-tts
+  MIMIR_ALLOW_ONLINE_TTS=1 TTS_ENGINE=auto ...
+  or explicitly set TTS_ENGINE=edge for non-sensitive text.
+EOF
+exit 127

package/skills/mimir-audio-summary/split-lines.py

@@ -0,0 +1,13 @@
+"""Print one sentence per line from a text file.
+
+Used to render long narrations sentence-by-sentence when an engine benefits from segmentation.
+"""
+import re
+import sys
+
+raw = open(sys.argv[1], encoding="utf-8").read()
+text = re.sub(r"\s+", " ", raw).strip()
+for part in re.split(r"(?<=[.!?…])\s+", text):
+    part = part.strip()
+    if part:
+        print(part)

package/skills/mimir-audio-summary/xtts-voice.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+"""Render a text file to WAV with Coqui XTTS-v2.
+
+Run with the dedicated venv interpreter:
+  ~/.local/share/voice-forge/xtts/bin/python xtts-voice.py <text.txt> <out.wav> [speaker]
+
+Env: XTTS_SPEAKER (preset name), XTTS_LANG (default "fr").
+"""
+import os
+import sys
+
+os.environ.setdefault("COQUI_TOS_AGREED", "1")
+
+
+def main() -> None:
+    if len(sys.argv) < 3:
+        sys.exit("usage: xtts-voice.py <text.txt> <out.wav> [speaker]")
+    text_file, out_wav = sys.argv[1], sys.argv[2]
+    speaker = sys.argv[3] if len(sys.argv) > 3 else os.environ.get("XTTS_SPEAKER", "Ana Florence")
+    language = os.environ.get("XTTS_LANG", "fr")
+
+    with open(text_file, encoding="utf-8") as handle:
+        text = handle.read().strip()
+    if not text:
+        sys.exit("error: empty text file")
+
+    from TTS.api import TTS
+
+    tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
+    speakers = list(getattr(tts, "speakers", None) or [])
+    if speakers and speaker not in speakers:
+        sys.stderr.write(f"speaker '{speaker}' not found; falling back to '{speakers[0]}'\n")
+        speaker = speakers[0]
+
+    tts.tts_to_file(
+        text=text,
+        speaker=speaker,
+        language=language,
+        file_path=out_wav,
+        split_sentences=True,
+    )
+    print(out_wav)
+
+
+if __name__ == "__main__":
+    main()