gufomind 1.0.0__tar.gz

gufomind-1.0.0/CHANGELOG.md

@@ -0,0 +1,60 @@
+# Changelog
+
+All notable changes to GufoMind are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.0.0] — 2026-10-07
+
+First public release.
+
+### Added
+
+- **Import** — import AI chat exports (DeepSeek) and personal writing
+  (Markdown + YAML frontmatter) into a local LanceDB knowledge base.
+- **Auto-categorization** — an LLM tags every chunk with 36 default topics
+  (fully customizable).
+- **RAG** — `gufo ask` answers questions grounded in your own data, with
+  source references.
+- **Dialogue memory** — `gufo chat` remembers context within a session.
+- **Web UI** — `gufo web` (FastAPI + HTMX) with a chat, a sessions sidebar,
+  a source panel and a settings page.
+- **Style transfer** — `gufo write "topic" -t poem` generates new text in your
+  own voice, grounded in your personal writing samples.
+- **Settings page** — edit runtime settings (retrieval, LLM, chunking,
+  license) from the web UI without touching `.env`. Layered resolution:
+  `env > user settings > .env > defaults`.
+- **Multi-provider LLM support** — Ollama (local, free), DeepSeek, OpenAI,
+  Anthropic, OpenRouter.
+- **Streaming** — token-by-token output in the CLI (`--stream`) and the web UI
+  (SSE).
+- **Per-file chunking control** — a `chunk: none` frontmatter key keeps
+  short-form writing (poems, notes) as a single chunk.
+- **Genre filter** — a repeatable `--type` in `gufo write` narrows the style
+  samples by genre (`poem` / `song` / `post` / `article` / `note`).
+- **CLI aliases** — the `gufo` and `gufomind` console commands are equivalent.
+- **Licensing** — Ed25519 offline verification with a periodic remote check.
+- **Web security baseline** — TrustedHost + Origin-check middleware
+  (localhost-only by default).
+
+### Changed
+
+- **Project renamed to GufoMind.** Python import path `gufo_rag` →
+  `gufomind`; environment prefix `CHATMIND_` → `GUFO_`; distribution
+  `gufo-rag` → `gufomind`.
+- **`.env` migration required** if you used the old prefix:
+  `sed -i '' 's/CHATMIND_/GUFO_/g' .env`
+- Minimum Python is **3.10** (3.12 recommended for the ML stack).
+
+### Fixed
+
+- `writing_kind` is no longer dropped by the splitter (before Phase 7.6 every
+  chunk carried a NULL genre).
+- Re-ingesting after a chunking change is now idempotent: `replace_document`
+  replaces a document's chunks by `document_id` instead of leaving orphan rows.
+
+[Unreleased]: https://github.com/chalpanov1/gufomind/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/chalpanov1/gufomind/releases/tag/v1.0.0

gufomind-1.0.0/LICENSE

@@ -0,0 +1,9 @@
+Copyright (c) 2026 Alexander Chalpanov. All rights reserved.
+
+This software and associated documentation files (the "Software")
+are proprietary and confidential. Unauthorized copying,
+modification, distribution, or use of the Software, in whole or
+in part, is strictly prohibited without the express written
+permission of the copyright holder.
+
+The Software is provided "AS IS", without warranty of any kind.

gufomind-1.0.0/MANIFEST.in

@@ -0,0 +1,12 @@
+# Include documentation files that setuptools does not pick up by default.
+include CHANGELOG.md
+
+# Exclude build artifacts from sdist.
+prune src/gufomind.egg-info
+
+# Exclude local data and secrets (defense in depth; gitignore already covers).
+prune data
+prune dist
+prune build
+exclude .env
+exclude contextforai.md

gufomind-1.0.0/PKG-INFO

@@ -0,0 +1,512 @@
+Metadata-Version: 2.4
+Name: gufomind
+Version: 1.0.0
+Summary: Personal RAG over your AI chat history — import, categorize, search, and query your conversations in your own voice.
+Author-email: Chalpanov <chalpanovrich1@yandex.ru>
+License-Expression: LicenseRef-Proprietary
+Project-URL: Homepage, https://github.com/chalpanov1/gufomind
+Project-URL: Repository, https://github.com/chalpanov1/gufomind
+Project-URL: Issues, https://github.com/chalpanov1/gufomind/issues
+Project-URL: Changelog, https://github.com/chalpanov1/gufomind/blob/main/CHANGELOG.md
+Keywords: rag,llm,personal-ai,chat-history,style-transfer,local-first,knowledge-base,lancedb,personal-knowledge-management
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pydantic-settings>=2.0
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: lancedb<0.41,>=0.39
+Requires-Dist: sentence-transformers>=2.7
+Requires-Dist: langchain-text-splitters>=0.2
+Requires-Dist: tiktoken>=0.7
+Requires-Dist: ollama>=0.3
+Requires-Dist: httpx>=0.27
+Requires-Dist: pandas>=2.0
+Requires-Dist: pyarrow>=15.0
+Requires-Dist: tqdm>=4.66
+Requires-Dist: platformdirs>=4.0
+Requires-Dist: python-frontmatter>=1.1
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+Requires-Dist: mypy>=1.11; extra == "dev"
+Provides-Extra: web
+Requires-Dist: fastapi>=0.111; extra == "web"
+Requires-Dist: uvicorn[standard]>=0.30; extra == "web"
+Requires-Dist: jinja2>=3.1; extra == "web"
+Requires-Dist: markdown-it-py>=3.0; extra == "web"
+Requires-Dist: python-multipart>=0.0.9; extra == "web"
+Provides-Extra: licensing
+Requires-Dist: pynacl>=1.5; extra == "licensing"
+Dynamic: license-file
+
+# 🦉 GufoMind
+
+> **Personal RAG over your AI chat history** — import, categorize, search and
+> query all of your conversations in one place, and generate new content in your
+> own voice.
+
+[![CI](https://github.com/chalpanov1/gufomind/actions/workflows/ci.yml/badge.svg)](https://github.com/chalpanov1/gufomind/actions/workflows/ci.yml)
+[![Version: v1.0.0](https://img.shields.io/badge/version-v1.0.0-blue.svg)](CHANGELOG.md)
+[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-red.svg)](LICENSE)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![Status: stable](https://img.shields.io/badge/status-stable-brightgreen.svg)](CHANGELOG.md)
+
+GufoMind (Gufo — Italian for *owl*) is a **local personal-RAG system**: it turns
+the scattered history of your AI conversations, your personal writing and your
+own chat sessions into **one local knowledge core** — and then lets you talk to
+it, search it, and write with it. Website: <https://gufomind.ru>.
+
+## Naming
+
+- **Product name:** GufoMind
+- **Website:** https://gufomind.ru
+- **CLI command:** `gufo`
+- **Python package / PyPI:** `gufomind`
+- **Env var prefix:** `GUFO_`.
+
+The product was previously called *Gufo-RAG*. The rename happened before
+the first public PyPI release, so no migration is required for end users.
+
+---
+
+## ✨ Why Gufo?
+
+- Your best ideas are buried in dozens of AI chats across DeepSeek, ChatGPT,
+  Claude and Gemini — and they are not searchable.
+- Your personal writing (posts, notes, poems, lyrics) lives somewhere else
+  entirely.
+- Asking the same question twice is a waste of time and tokens.
+
+Gufo fixes this: everything goes into **one vector database on your machine**,
+and every new conversation makes that base smarter.
+
+```
+Conversation → save → chunk → embed → vector store → search → new conversation
+     ▲                                                                │
+     └────────────── the loop keeps enriching the base ───────────────┘
+```
+
+## 🎯 Features
+
+| | Feature | Status |
+|---|---|---|
+| 1 | Import chats from DeepSeek / ChatGPT / Claude / Gemini + personal texts | 🚧 Phase 1 |
+| 2 | Automatic categorization (topics tagged by an LLM) | ✅ v0.2.0 |
+| 3 | RAG pipeline — `gufo ask` answers grounded in your archive | ✅ v0.3.0 |
+| 4 | Hybrid search — vectors + BM25 + re-ranker | ⬜ Phase 3.5 |
+| 5 | Dialogue memory — the assistant remembers the context | ✅ v0.4.0 |
+| 6 | Multi-LLM + streaming — Ollama, OpenAI, DeepSeek, Anthropic, OpenRouter | ✅ v0.5.0 |
+| 7 | Web interface with a chat client and auto-saved sessions | ✅ v0.6.0 |
+| 8 | Style transfer — write in the author's own voice | ✅ v0.7.0 |
+| 9 | Write grounded in a chosen genre — repeatable `--type` · per-file `chunk: none` | ✅ v0.7.6 |
+| 10 | Settings page in the web UI — edit runtime settings without touching `.env` | ✅ v0.7.7 |
+
+Legend: ✅ done · 🚧 in progress · ⬜ planned
+
+**Web interface (v0.6)** — `gufo web`:
+
+- ✅ Streamed answers (SSE) rendered as markdown
+- ✅ Multi-turn sessions with memory, listed in a sidebar
+- ✅ Sources panel under each answer
+- ✅ "Enrich the base" — turn a conversation into searchable chunks
+- ✅ Dark/light theme, responsive layout
+
+**Current status** (v0.6.0):
+
+- **Phase 0 (Foundation)** — ✅ complete.
+- **Phase 1 (Multi-Importer)** — 🚧 in progress. `gufo ingest` works end-to-end
+  for DeepSeek exports (verified on a real 64 MB, 236-conversation export →
+  23,966 chunks). The ChatGPT / Claude / Gemini and personal-text importers are
+  still to come.
+- **Phase 2 (Auto-categorization)** — ✅ complete. `gufo categorize` tags every
+  chunk through the DeepSeek API (verified on the full base: 23,966 chunks →
+  23,618 with topics, 348 unclassified, 0 failures, 41m 40s).
+- **Phase 3 (Vector RAG)** — ✅ complete. `gufo ask` answers questions grounded in
+  your archive, with sources (verified live against the real base). Hybrid search
+  (BM25 + re-ranker) follows in Phase 3.5.
+- **Phase 4 (Dialogue Memory)** — ✅ complete. `gufo chat` holds a multi-turn
+  dialogue, persists each session to JSONL and resumes it with `--session <id>`.
+- **Phase 5 (Multi-LLM + Streaming)** — ✅ complete. Five providers behind one
+  `.env` switch (`ollama`, `deepseek`, `openai`, `anthropic`, `openrouter`) and
+  streaming answers in the CLI (`gufo ask --stream`, `gufo chat --stream`).
+- **Phase 6 (Web Interface + Chat)** — ✅ complete. `gufo web` serves a local
+  FastAPI + HTMX chat at `http://127.0.0.1:8000`: streamed answers over SSE, a
+  sessions sidebar, a sources panel, and an "Enrich the base" action that turns a
+  conversation into searchable chunks.
+- **Phase 7 (Style Transfer)** — ✅ complete (v0.7.0). `gufo write` drafts new
+  texts in your own voice, grounded in personal writing samples.
+- **Phase 7.5 (Style Transfer polish)** — ✅ complete (no version bump).
+  Post-audit fixes: a non-blocking license check and prompt hardening.
+- **Phase 7.6 (Writing kind filter + chunking)** — ✅ complete (v0.7.6).
+  Optional, repeatable `--type`; per-file `chunk: none`; idempotent re-ingest.
+- **Phase 7.7 (Settings page + security)** — ✅ complete (v0.7.7). A **⚙️ Settings**
+  page in the web UI edits runtime settings live (retrieval, LLM, chunking) on
+  top of a layered loader (`env > user settings > .env > defaults`), with secrets
+  kept in the OS user-data directory; the local server gains a TrustedHost
+  allowlist and a cross-origin (CSRF) check.
+
+See [ROADMAP.md](ROADMAP.md).
+
+## 📦 Installation
+
+> Gufo is not published to PyPI yet. Until then, install it from source.
+
+### From source (development)
+
+```bash
+git clone https://github.com/chalpanov1/gufomind.git
+cd gufomind
+
+python3.12 -m venv .venv
+source .venv/bin/activate
+
+pip install -e ".[dev]"
+```
+
+> **Optional:** put `HF_TOKEN=...` in `.env` to authenticate Hugging Face model
+> downloads — it is faster and removes the "unauthenticated requests" warning.
+> Create a read-scoped token at
+> [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
+
+### Verify the installation
+
+```bash
+gufo --version   # prints the installed version
+gufo stats       # shows the resolved configuration
+gufo --help      # lists all commands
+```
+
+### Once published (planned)
+
+```bash
+pipx install gufomind    # recommended: isolated CLI install
+# or
+pip install gufomind
+```
+
+## ⚙️ Configuration
+
+Copy the example file and adjust it:
+
+```bash
+cp .env.example .env
+```
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `GUFO_DATA_DIR` | `./data` | root directory for local data |
+| `GUFO_DB_PATH` | `./data/vector_db` | LanceDB database directory |
+| `GUFO_TABLE_NAME` | `knowledge_base` | LanceDB table name |
+| `GUFO_SESSIONS_DIR` | `./data/sessions` | directory of `gufo chat` sessions (one JSONL file per session) |
+| `GUFO_EMBEDDING_MODEL` | `paraphrase-multilingual-MiniLM-L12-v2` | sentence-transformers model |
+| `HF_TOKEN` | — | Hugging Face token (faster downloads, gated models) |
+| `GUFO_LLM_PROVIDER` | **required** | `ollama` (local, slow) \| `deepseek` (cloud) \| `openai` \| `anthropic` \| `openrouter` |
+| `GUFO_LLM_MODEL` | per-provider default | optional model override; unset uses the provider default (`gufo stats` shows it) |
+| `GUFO_LLM_TIMEOUT` | `60` | seconds to wait for one LLM request |
+| `OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL |
+| `GUFO_CHUNK_SIZE` | `512` | target chunk size in tokens |
+| `GUFO_CHUNK_OVERLAP` | `50` | overlap between chunks |
+| `GUFO_TOP_K` | `20` | chunks passed to the LLM |
+| `GUFO_FETCH_K` | `30` | candidates fetched before re-ranking |
+| `GUFO_TEMPERATURE` | `0.3` | generation temperature |
+| `GUFO_HISTORY_WINDOW` | `20` | recent turns of a session passed to the LLM as history |
+| `GUFO_RETRIEVAL_HISTORY_TURNS` | `2` | prior user turns folded into the search query |
+| `GUFO_INCLUDE_THINKING` | `false` | keep the model's chain-of-thought blocks on import |
+| `DEEPSEEK_API_KEY` | — | API key for the `deepseek` provider / `gufo categorize` |
+| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | DeepSeek API root |
+| `OPENAI_API_KEY` | — | API key for the `openai` provider |
+| `OPENAI_BASE_URL` | `https://api.openai.com/v1` | OpenAI API root |
+| `ANTHROPIC_API_KEY` | — | API key for the `anthropic` provider |
+| `ANTHROPIC_BASE_URL` | `https://api.anthropic.com/v1` | Anthropic API root |
+| `OPENROUTER_API_KEY` | — | API key for the `openrouter` provider |
+| `OPENROUTER_BASE_URL` | `https://openrouter.ai/api/v1` | OpenRouter API root |
+| `OPENROUTER_REFERER` | — | optional `HTTP-Referer` attribution header |
+| `OPENROUTER_TITLE` | — | optional `X-Title` attribution header |
+| `GUFO_CATEGORIZATION_MODEL` | `deepseek-chat` | model used to tag chunks |
+| `GUFO_CATEGORIZATION_BATCH_SIZE` | `20` | chunks per LLM request |
+| `GUFO_CATEGORIZATION_MAX_TOPICS` | `5` | maximum topics per chunk |
+| `GUFO_CATEGORIZATION_SEED_TOPICS` | built-in list of 36 | comma-separated topic seeds |
+| `GUFO_LICENSE_SERVER_URL` | `https://license.gufomind.ru` | license server base URL |
+| `GUFO_LICENSE_CHECK_INTERVAL_DAYS` | `30` | days a successful remote check is trusted |
+| `GUFO_LICENSE_GRACE_PERIOD_DAYS` | `90` | offline grace period since the last successful check |
+| `GUFO_STYLE_K` | `5` | style samples retrieved for `gufo write` (1–20) |
+| `GUFO_LOG_LEVEL` | `WARNING` | log level: `DEBUG` / `INFO` / `WARNING` / `ERROR` |
+| `GUFO_IGNORE_USER_SETTINGS` | — | set to `1` to ignore settings saved from the web UI (emergency kill-switch) |
+
+> Values saved from the web UI's **⚙️ Settings** page live in your OS user-data
+> directory, not in `.env` — see [Settings page](#settings-page). Precedence is
+> **real env vars > user settings > `.env` > defaults**.
+
+> Environment variables use the `GUFO_` prefix. The previous prefix
+> (`CHATMIND_`) was renamed in Phase 8 with no backward-compatible alias.
+
+## 💸 Categorization cost
+
+Running `gufo categorize` uses an LLM to tag every chunk. The cost depends
+entirely on which model you use — from **$0 (local Ollama)** to **$80+ (OpenAI
+GPT-4o)** for a 24,000-chunk base.
+
+| Provider / Model | Cost for 24k chunks | Time | Notes |
+|---|---|---|---|
+| **Ollama (local)** | **$0** | ~hours on M1 | Free, private, slow |
+| **DeepSeek (off-peak)** | **~$1.66** | ~40 min | Cheapest API. Reference run. |
+| DeepSeek (peak hours) | ~$3–4 | ~40 min | 2× off-peak pricing |
+| OpenAI gpt-4o-mini | ~$5–8 | ~15 min | Faster, more expensive |
+| Anthropic Claude Haiku | ~$10–15 | ~20 min | Mid-range |
+| OpenAI gpt-4o | ~$60–80 | ~15 min | Fastest, most expensive |
+
+**Incremental runs** (new chunks only) cost proportionally less: 100 new chunks
+≈ $0.007 with DeepSeek off-peak.
+
+**Why categorize?**
+
+- **Without it:** `gufo ask` searches by semantic similarity only.
+- **With it:** you can filter by exact topics — `gufo ask "..." --topic databases`
+  returns only database-related chunks.
+
+> These numbers are indicative and follow the provider's pricing; local Ollama is
+> free but needs a machine that can run the model. A `--force` run asks for
+> confirmation first — pass `--yes` to skip the prompt in scripts and CI.
+
+## 🚀 Usage
+
+```bash
+# Show the resolved configuration and the state of the base
+gufo stats
+
+# Import an export into the knowledge base (Phase 1)
+gufo ingest ./exports/deepseek_conversations.json
+
+# Tag every chunk with topics through the DeepSeek API (Phase 2)
+gufo categorize            # incremental: only untagged chunks
+gufo categorize --force    # re-tag everything
+
+# Ask a question (Phase 3)
+gufo ask "Что я писал про векторные базы данных?"
+
+# Stream the answer as it is generated (Phase 5)
+gufo ask "Что я писал про векторные базы данных?" --stream
+
+# Chat with a memory of the dialogue (Phase 4)
+gufo chat                       # start a new session
+gufo chat --session <id>        # resume a saved session
+gufo chat --stream              # stream each answer as it arrives
+#   inside the REPL: /exit · /new · /clear · /sessions
+
+# Chat in the browser, with a sessions sidebar and sources (Phase 6)
+gufo web                        # serves http://127.0.0.1:8000
+gufo web --open                 # ...and open the browser
+gufo web --port 9000            # pick another port
+
+# Write in your own voice (Phase 7). The first --type is what to write;
+# any further --type values pick the genres to draw samples from.
+gufo write "утро"               # a post (default), grounded in everything you wrote
+gufo write "тишина" -t poem     # a poem, grounded only in poems
+gufo write "тишина" -t poem -t song   # a poem, inspired by poems and songs
+```
+
+### Settings page
+
+Open the **⚙️ Settings** page from the sidebar to change runtime settings
+without editing `.env` or restarting:
+
+- **Retrieval** (`top_k`, `history_window`, …) — applied instantly.
+- **LLM** (provider, model, temperature, API keys) — the client is rebuilt on
+  save; in-flight streams continue on the old client.
+- **Chunking** (`chunk_size`, `chunk_overlap`) — applied to the next ingest /
+  enrich run.
+- **License** intervals — applied on the next restart.
+- **Read-only** fields (`embedding_model`, `db_path`, …) are shown but cannot be
+  edited from the UI. To change them, edit `settings.json` and restart the
+  process.
+
+Your settings are stored in your OS's user-data directory
+(`~/Library/Application Support/Gufo/` on macOS, `~/.local/share/gufo/` on
+Linux, `%APPDATA%\Gufo\` on Windows):
+
+- `settings.json` — non-secret values.
+- `secrets.json` — API keys, written with `0600` (owner-only) permissions.
+
+**Precedence:** real environment variables > your `settings.json` > `.env` >
+defaults. If a field shows `source: env`, editing it in the UI will have no
+effect until the environment variable is unset.
+
+### Network exposure
+
+The web server binds to `127.0.0.1` (loopback) by default and is protected by
+two middlewares:
+
+- **TrustedHostMiddleware** — only `localhost`, `127.0.0.1` and `[::1]` are
+  accepted; anything else returns `400`.
+- **Origin check** — mutating requests (`POST`/`PUT`/`DELETE`) must carry a
+  same-origin `Origin` (or `Referer`) header; cross-origin requests get `403`.
+  Non-browser clients (curl, scripts) that send neither header are allowed.
+
+Running `gufo web --host 0.0.0.0` (or any non-loopback address) triggers a
+warning and requires explicit confirmation, then opens the allowlist to `["*"]`.
+**GufoMind has no authentication** — never expose it to an untrusted network.
+
+⚠️ Bypassing the CLI (`uvicorn gufomind.web.run:app --host 0.0.0.0`) keeps the
+loopback allowlist and will reject LAN clients. Always use `gufo web`.
+
+### Personal writing (Markdown + YAML frontmatter)
+
+Put your own writing (poems, posts, articles, songs, notes) in a folder as
+`.md` files. Each file starts with YAML frontmatter:
+
+```yaml
+---
+type: poem          # poem | post | article | song | note
+title: Morning      # optional; falls back to the file name
+topics: [life]      # optional
+chunk: none         # optional; see below
+date: 2026-10-04    # optional
+---
+
+Your text here.
+```
+
+**`chunk: none`** disables chunk splitting for that file. Use it for short-form
+writing (poems, haiku, notes) so a single work stays one chunk and its voice
+sample is not truncated mid-stanza.
+
+**Re-importing is idempotent**: chunks are keyed by a stable document id, and a
+re-import replaces the previous chunks of the same file — no duplicates even if
+you change `chunk`, `type` or `title` between imports.
+
+#### Migrating from pre-v0.7.6
+
+Personal chunks imported before v0.7.6 have `document_id = NULL`. A re-import
+**with the same chunking structure** works fine — `merge_insert` matches by
+chunk `id` and backfills `document_id`. However, a re-import **after a
+chunking-structure change** (e.g. adding `chunk: none` to a file) will produce
+duplicate chunks: the old NULL rows are not matched by `replace_document` and
+stay behind.
+
+To migrate cleanly, run **once**:
+
+    gufo ingest --reset <path-to-personal-texts>
+
+This drops the table and re-imports everything with `document_id` populated. A
+one-shot backfill command is planned for a future release (see ROADMAP backlog).
+
+## 🔎 Retrieval tuning
+
+By default `gufo ask` and `gufo chat` retrieve the **20** most relevant chunks
+and pass them to the LLM. That balances coverage (enough chunks to answer) and
+precision — LLM attention degrades beyond roughly 25 chunks ("lost in the
+middle"), so a larger `top_k` can dilute focus on a focused question.
+
+Adjust it per call with `--top-k N`:
+
+- **Lower (8–10)** — faster and cheaper; good for a focused question about one
+  specific thing, but it may miss relevant chunks for questions that span a
+  whole project.
+- **Higher (30–50)** — more coverage for broad, cross-cutting questions; may hurt
+  precision on focused ones because of attention dilution.
+
+Rules of thumb:
+
+- Default **20** is tuned for a ~24k-chunk base. On a small base (<5k chunks),
+  **10–15** is usually enough.
+- The cost is small: 20 chunks × ~512 tokens ≈ 10k tokens of context — pennies on
+  a cloud provider and comfortably inside a 32k local Ollama model.
+- Prefer raising `--top-k` over lowering it when answers feel incomplete: better
+  to hand the model a few extra candidates than to miss the one that mattered.
+
+## 🧱 Data model
+
+Every chunk carries three independent axes of metadata, so results can be
+filtered in almost any combination:
+
+| Axis | Values |
+|---|---|
+| **source** | `deepseek`, `chatgpt`, `claude`, `gemini`, `personal`, `session`, `manual` |
+| **content_type** | `chat`, `writing`, `note`, `article`, `code` |
+| **topics** | free-form tags, e.g. `models`, `databases`, `business` |
+
+```python
+# All my personal writing about business — used to generate a new post
+search(
+    query="предпринимательство, рост, стратегия",
+    filters={"source": "personal", "content_type": "writing", "topics": ["business"]},
+)
+```
+
+## 🗂️ Project structure
+
+```
+gufomind/
+├── .clinerules/          # coding rules for the AI agent
+├── memory-bank/          # long-term project memory
+├── src/gufomind/
+│   ├── cli.py            # typer CLI
+│   ├── config.py         # pydantic-settings (single source of truth)
+│   ├── core/models.py    # Message, Chat, Chunk, SearchResult
+│   ├── importers/        # one importer per source
+│   ├── chunking/         # token-aware splitter
+│   ├── embeddings/       # sentence-transformers encoder
+│   ├── storage/          # LanceDB store
+│   ├── categorization/   # LLM topic tagging (port adapter + use case)
+│   ├── llm/              # LLM clients (Ollama, OpenAI, DeepSeek, ...)
+│   └── rag/              # retrieval + generation pipeline
+├── tests/
+└── data/                 # local data (git-ignored)
+```
+
+## 🗺️ Roadmap
+
+See [ROADMAP.md](ROADMAP.md) for the full plan. Short version:
+
+| Phase | Name |
+|---|---|
+| 0 | Foundation ✅ v0.1.0 |
+| 1 | Multi-Importer 🚧 |
+| 2 | Auto-categorization ✅ v0.2.0 |
+| 3 | Vector RAG ✅ v0.3.0 |
+| 3.5 | Hybrid Search ⬜ |
+| 4 | Dialogue Memory ✅ v0.4.0 |
+| 5 | Multi-LLM ✅ v0.5.0 |
+| 6 | Web Interface + Chat ✅ v0.6.0 |
+| 7 | Style Transfer ✅ v0.7.0 |
+| 7.5 | Style Transfer polish ✅ |
+| 7.6 | Writing kind filter + chunking ✅ v0.7.6 |
+| 7.7 | Settings page + security ✅ v0.7.7 |
+| 8 | Release ⬜ |
+
+## 🛠️ Development
+
+The project follows the coding standards in
+[`.clinerules/01-coding-standards.md`](.clinerules/01-coding-standards.md) and the
+workflow in [`.clinerules/04-workflow.md`](.clinerules/04-workflow.md).
+
+```bash
+ruff check .      # lint
+ruff format .     # format
+pytest            # tests
+```
+
+Commits follow [Conventional Commits](https://www.conventionalcommits.org/).
+
+## 📄 License
+
+Copyright © 2026 Alexander Chalpanov. All rights reserved.
+
+This is **proprietary, closed-source software** — not open source. See
+[LICENSE](LICENSE) for the full terms. Unauthorized copying, modification,
+distribution or use, in whole or in part, is strictly prohibited.