docsmind 0.2.0__tar.gz → 0.4.0__tar.gz

{docsmind-0.2.0 → docsmind-0.4.0}/CHANGELOG.md

@@ -2,6 +2,53 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.4.0] - 2026-10-31
+
+### Added
+
+- The Mind: a chat agent (orchestrator, planner, tool-using executor, parallel subagents)
+  with a deterministic harness (context builder with prompt-cache breakpoint, summarizer,
+  memory writer). Answers carry verified `[S#]` citations.
+- Provider-neutral LLM layer with OpenAI and Anthropic streaming adapters.
+- Tools: `search`, `grep`, `read_pages`, `get_outline`, `list_documents`, `delegate`;
+  subagents: `researcher`, `change_analyzer`.
+- Migration `0004`: `conversations`, `messages`, `agent_memory` tables (run `docsmind db upgrade`).
+- HTTP: `POST /api/chat` (SSE), `GET/DELETE /api/conversations`, `GET /api/agent`.
+- UI: streaming chat in Workspace with live agent activity, markdown answers, citation chips
+  and source cards that open the PDF at the cited page; Activity lists saved conversations;
+  Agents reads the real registry. New frontend deps: `react-markdown`, `remark-gfm`.
+- `Docsmind.chat_result()`; `chat`/`achat` accept `conversation_id`; `docsmind chat` REPL keeps
+  the conversation.
+- Config: `agent_*` settings, `openai_base_url`, `openai_reasoning_effort`.
+
+## [0.3.0] - 2026-10-03
+
+### Added
+
+- Bundled frontend UI in the Python package: release UI assets are copied into
+  `src/docsmind/server/ui` and served by `docsmind serve` at `/`.
+- `scripts/bundle_frontend.py` helper to copy `../frontend/dist` into the package
+  before publishing.
+- `docsmind delete <version-id> [-y]` command to remove one ingested version and
+  its derived sections/chunks/images/tables.
+- HTTP `DELETE /api/documents/{version_id}` endpoint.
+
+### Changed
+
+- `docsmind serve` now serves the bundled SPA (when present), in addition to
+  `/api/*` and `/blobs/*`.
+- Ingestion now checks `content_hash` before parsing and raises a friendly
+  "already ingested" error instead of surfacing a raw Postgres unique-constraint
+  failure.
+
+### Fixed
+
+- Image extraction de-duplicates repeated embedded images (e.g. page logo reused
+  across many pages) using perceptual hash, preventing inflated image counts.
+- API duplicate-ingest responses now return HTTP 409 with structured details.
+
+
+
 ## [0.2.0] - 2026-09-27
 
 First functional release: documents can now be ingested from PDF into
@@ -70,9 +117,7 @@ implemented" below).
 - Embeddings and hybrid vector/keyword search. Chunks are stored without
   embeddings, and `hybrid_search` raises `NotImplementedError`.
 - Image captioning with a vision model.
-- The chat agent behind `docsmind chat` and the change-analysis subagent
-  behind `docsmind diff`. Both commands exist but have no implementation
-  behind them yet.
+- The chat agent behind `docsmind chat`.
 - DOCX and plain-text parsers, S3/Supabase blob storage, and the server/UI.
 - Multi-column reading order: PDFs with genuine multi-column layouts may have
   their text interleaved in the saved full text.

{docsmind-0.2.0 → docsmind-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docsmind
-Version: 0.2.0
+Version: 0.4.0
 Summary: Chat with your documents. Version-aware retrieval, change analysis, and an editable table workspace, backed by Postgres/pgvector.
 Project-URL: Homepage, https://github.com/yauheniya-ai/docsmind
 Project-URL: Repository, https://github.com/yauheniya-ai/docsmind
@@ -29,11 +29,13 @@ Requires-Dist: pydantic>=2.6
 Requires-Dist: pymupdf>=1.24
 Requires-Dist: rich>=13.7
 Requires-Dist: sqlalchemy[asyncio]>=2.0
+Requires-Dist: tiktoken>=0.7
 Requires-Dist: typer>=0.12
 Provides-Extra: all
 Requires-Dist: anthropic>=0.34; extra == 'all'
 Requires-Dist: fastapi>=0.111; extra == 'all'
 Requires-Dist: openai>=1.30; extra == 'all'
+Requires-Dist: python-multipart>=0.0.9; extra == 'all'
 Requires-Dist: uvicorn[standard]>=0.30; extra == 'all'
 Provides-Extra: anthropic
 Requires-Dist: anthropic>=0.34; extra == 'anthropic'
@@ -49,6 +51,7 @@ Provides-Extra: openai
 Requires-Dist: openai>=1.30; extra == 'openai'
 Provides-Extra: server
 Requires-Dist: fastapi>=0.111; extra == 'server'
+Requires-Dist: python-multipart>=0.0.9; extra == 'server'
 Requires-Dist: uvicorn[standard]>=0.30; extra == 'server'
 Description-Content-Type: text/markdown
 
@@ -58,8 +61,6 @@ Description-Content-Type: text/markdown
 [![Python versions](https://img.shields.io/pypi/pyversions/docsmind.svg)](https://pypi.org/project/docsmind/)
 [![Downloads](https://static.pepy.tech/badge/docsmind)](https://pepy.tech/project/docsmind)
 [![Downloads / month](https://static.pepy.tech/badge/docsmind/month)](https://pepy.tech/project/docsmind)
-[![Tests](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml/badge.svg)](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml)
-[![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/yauheniya-ai/<GIST_ID>/raw/docsmind-coverage.json)](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml)
 [![License: MIT](https://img.shields.io/pypi/l/docsmind.svg)](https://github.com/yauheniya-ai/docsmind/blob/main/LICENSE)
 [![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)
@@ -67,9 +68,8 @@ Description-Content-Type: text/markdown
 Chat with your documents. Track what changed between versions, and why it matters.
 
 `docsmind` ingests PDFs (and soon DOCX/text) into a version-aware Postgres/pgvector
-store, and gives you an agent, **the Mind**, to chat with them. A built-in subagent,
-**diffra**, will do change analysis between document versions: what changed, where, and
-its compliance implications, with citations back to exact pages.
+store, and gives you an agent, **the Mind**, to analyze them. Built-in subagents
+will perform specialized tasks.
 
 > **Alpha.** Ingestion and inspection work today. Embeddings, chat and change analysis
 > are designed and scaffolded but not implemented yet; see [What works today](#what-works-today).
@@ -82,11 +82,13 @@ its compliance implications, with citations back to exact pages.
 | ✅ Postgres + pgvector storage | Local Docker, or any Postgres with pgvector (Neon, Supabase) |
 | ✅ Inspection CLI | `versions`, `fulltext`, `outline`, `chunks` read back exactly what was saved |
 | ✅ `docsmind doctor` | Checks database, blob storage and API keys, and tells you what to fix |
-| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
+| ✅ Table extraction | Extract and store tables skipping repeating ones (e.g. headers or footers) |
+| ✅ server + UI | FastAPI backend (`docsmind serve`) + React UI (`frontend/`): add documents, browse indexed content, image and table galleries |
+| ✅ Chat agent (the Mind) | Routes each question, plans complex ones, searches/reads your documents with tools, delegates to subagents, and answers with verified `[S#]` citations. Streams to the UI over SSE; conversations are saved |
 | 🚧 Image captioning | Images are extracted and stored; vision captioning is next |
-| 🚧 Chat agent (the Mind) | `docsmind chat` exists as a command, without an agent behind it yet |
+| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
 | 🚧 Change analysis (diffra) | `docsmind diff` exists as a command, without an implementation yet |
-| 🚧 DOCX/text parsers, S3/Supabase blob storage, server + UI | Planned |
+| 🚧 DOCX/text parsers, S3/Supabase blob storage | Planned |
 
 ## Install
 
@@ -96,7 +98,7 @@ pip install docsmind
 pip install "docsmind[all]"
 ```
 
-Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, planned UI),
+Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, serves the bundled UI),
 `docsmind[mlflow]` (tracing/eval export). Requires Python 3.10+.
 
 ## Quickstart
@@ -127,8 +129,9 @@ docsmind doctor
 ```
 
 Configuration is environment-driven (`DOCSMIND_*`, or a `.env` file). Copy
-`.env.example` to `.env` to see every option. Model API keys aren't needed yet;
-they'll be required once embeddings and chat land.
+`.env.example` to `.env` to see every option. Chat needs an LLM key
+(`DOCSMIND_LLM_PROVIDER` plus `DOCSMIND_ANTHROPIC_API_KEY` or `DOCSMIND_OPENAI_API_KEY`);
+ingestion and browsing don't.
 
 ### 2. Ingest a PDF
 
@@ -158,13 +161,81 @@ docsmind chunks <version-id> --page 12  # the chunk(s) saved for one page
 
 You can also inspect the tables directly with `docsmind db psql`.
 
-### 4. Coming next
+### 4. Browse it in the UI
+
+```bash
+pip install "docsmind[server]"
+docsmind serve              # FastAPI backend on http://127.0.0.1:8000
+```
+
+Open `http://127.0.0.1:8000`.
+
+The React UI is bundled into the Python package and served directly by `docsmind serve`
+from `docsmind/src/docsmind/server/ui`.
+
+Clicking **+ Add Document** in the UI runs the same `docsmind ingest` pipeline on the
+uploaded PDF. Selecting a document shows its saved full text/outline (**Content**), and
+galleries of everything extracted from it (**Images**, **Tables**).
+
+For contributors refreshing bundled UI assets before a release:
+
+```bash
+cd ../frontend
+npm install
+npm run build
+cd ../docsmind
+python scripts/bundle_frontend.py
+```
+
+### 5. Ask the Mind
+
+In the UI, open **Workspace** and ask a question (needs `docsmind[openai]` or `docsmind[anthropic]`
+plus an API key). From the terminal or Python:
+
+```bash
+docsmind chat "What are our data retention obligations?"   # one-shot
+docsmind chat                                              # REPL, keeps the conversation
+```
 
 ```python
-print(dm.chat("What are our data retention obligations?"))   # planned
-change_set = dm.compare(v1.id, v2.id)                        # planned (diffra)
+result = dm.chat_result("What are our data retention obligations?")
+print(result.answer, result.citations)
+change_set = dm.compare(v1.id, v2.id)   # planned (diffra)
+```
+
+## How the Mind works
+
+```
+Question -> Orchestrator --simple--> Executor <-> Tools / Subagents -> Answer
+                 |                       ^
+                 +--complex--> Planner --+        Harness feeds every role:
+                                                  context builder, prompt cache,
+                                                  summarizer, memory
 ```
 
+- **Orchestrator**: one cheap call that routes the question (`simple` or `complex`) and rewrites
+  it as a standalone question using the conversation so far.
+- **Planner** (complex only): a short list of steps the executor follows.
+- **Executor**: a bounded tool loop (`search`, `grep`, `read_pages`, `get_outline`,
+  `list_documents`, `delegate`). The last step forces an answer; tool errors go back to the
+  model, not the user; old tool results are elided when the context grows.
+- **Subagents** (`researcher`, `change_analyzer`): fresh context, a subset of the tools, run in
+  parallel, cannot delegate further.
+- **Harness**: deterministic code, not another agent. It orders the prompt stable-first so the
+  prefix caches, summarizes long histories, and saves durable notes to memory after each answer.
+- **Citations**: tools register passages as `S1, S2, ...`; the model cites `[S#]`; invented
+  handles are dropped and the rest renumbered by first appearance. Document text is treated as
+  untrusted data, never as instructions.
+- **Search** is Postgres full-text (ranked) plus exact-text grep until embeddings land.
+
+Tuning (all `DOCSMIND_*`): `AGENT_FAST_MODEL` (router/summarizer/memory model, defaults to
+`LLM_MODEL`), `AGENT_MAX_STEPS`, `AGENT_SUBAGENT_MAX_STEPS`, `AGENT_MEMORY_ENABLED`,
+`OPENAI_REASONING_EFFORT`, `OPENAI_BASE_URL`. See `.env.example`.
+
+HTTP: `POST /api/chat` streams Server-Sent Events (`route`, `plan`, `tool_start`, `tool_end`,
+`subagent_start`, `subagent_end`, `token`, `note`, `answer`, `error`, `done`);
+`GET /api/conversations[/{id}]`, `DELETE /api/conversations/{id}`, `GET /api/agent`.
+
 ## How ingestion works
 
 Ingestion is deliberately simple and predictable. There's no layout guessing, because
@@ -180,6 +251,9 @@ heuristics that work on one PDF break on the next.
   reconstruct structure.
 - **Images** are extracted with their page and position, skipping tiny icons, and saved
   under a readable path: `images/<collection>/<title>/<version>-<id>/`.
+  Visually repeated images (e.g. logo on every page) are de-duplicated by perceptual hash.
+- **Tables** are detected with PyMuPDF and rendered as PNG snippets, saved under
+  `tables/<collection>/<title>/<version>-<id>/`.
 
 ## CLI reference
 
@@ -190,9 +264,12 @@ heuristics that work on one PDF break on the next.
 | `docsmind fulltext <id> [--full]` | Print the saved full text |
 | `docsmind outline <id>` | Print the saved outline |
 | `docsmind chunks <id> [--page N] [--full]` | Print saved chunks |
+| `docsmind delete <id> [-y]` | Delete one version and its derived chunks/images/tables |
 | `docsmind db up` / `upgrade` / `psql` | Start local Postgres, apply migrations, open psql |
 | `docsmind doctor` | Check your setup |
-| `docsmind chat`, `docsmind diff` | Planned |
+| `docsmind serve [--host] [--port] [--reload]` | Run the FastAPI backend for the `frontend/` UI |
+| `docsmind chat [question] [-c <collection>]` | Ask the Mind once, or start a REPL that keeps the conversation |
+| `docsmind diff` | Planned |
 
 ## Architecture
 
@@ -204,7 +281,7 @@ src/docsmind/
 │
 ├── domain/                # pure data + interfaces, no I/O
 │   ├── documents.py       # Document, DocumentVersion, Section, Chunk, Image, ParsedDocument
-│   ├── changes.py         # Change, ChangeSet, ImpactAssessment (diffra)
+│   ├── changes.py         # Change, ChangeSet, ImpactAssessment
 │   ├── citations.py       # Citation
 │   ├── tables.py          # AgentTable, TableRevision
 │   └── ports.py           # LLM, Embedder, VisionCaptioner, VectorStore, BlobStore, Parser
@@ -221,13 +298,24 @@ src/docsmind/
 │   ├── llm/ embeddings/ vision/ tracing/   # planned
 │
 ├── cli/                   # Typer app: ingest, versions, fulltext, outline, chunks, db, doctor
-├── retrieval/             # hybrid vector + keyword search (planned)
-├── agent/                 # the Mind + diffra subagent (planned)
-└── server/                # FastAPI + UI (planned)
+├── retrieval/             # keyword (full-text) search helpers; vector search planned
+├── agent/                 # the Mind: orchestrator, planner, tool loop, tools, subagents, harness
+└── server/                # FastAPI app + routes backing frontend/ (documents, chat SSE, conversations)
 
-alembic/   tests/   examples/   docs/   ui/
+alembic/   tests/   examples/   docs/
 ```
 
+The UI source lives one level up as a sibling project: `frontend/` (React + TypeScript +
+TailwindCSS, see the repo-root `CLAUDE.md`). Release builds are bundled into
+`src/docsmind/server/ui` and served by `server/app.py` at `/`. The UI talks to `server/`
+over `/api/*`, and loads extracted images/tables straight from disk via `/blobs/*`.
+
+## Artwork attribution
+
+- Workspace 3D graph artwork inspiration: Angela Galliat,
+  "Neural nervous system of a brain" (CodePen):
+  https://codepen.io/agalliat/pen/vYGXJxQ
+
 ### Design notes
 
 - **Ports and adapters, once.** `domain/ports.py` defines every external dependency as a
@@ -242,7 +330,9 @@ alembic/   tests/   examples/   docs/   ui/
   Outline paths (e.g. `03.13.11`) give version comparison something stable to align on
   where a document has a table of contents.
 - **Tables are shared state, not a UI feature.** `AgentTable` is a domain object that
-  both the agent and the user write to (append-only revisions).
+  both the agent and the user write to (append-only revisions). This is distinct from
+  the `tables` storage table (extracted PDF table *regions*, rendered as images
+  alongside `images` — browsable read-only in the UI's **Tables** gallery).
 
 ## Development
 

{docsmind-0.2.0 → docsmind-0.4.0}/README.md

@@ -4,8 +4,6 @@
 [![Python versions](https://img.shields.io/pypi/pyversions/docsmind.svg)](https://pypi.org/project/docsmind/)
 [![Downloads](https://static.pepy.tech/badge/docsmind)](https://pepy.tech/project/docsmind)
 [![Downloads / month](https://static.pepy.tech/badge/docsmind/month)](https://pepy.tech/project/docsmind)
-[![Tests](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml/badge.svg)](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml)
-[![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/yauheniya-ai/<GIST_ID>/raw/docsmind-coverage.json)](https://github.com/yauheniya-ai/docsmind/actions/workflows/test.yml)
 [![License: MIT](https://img.shields.io/pypi/l/docsmind.svg)](https://github.com/yauheniya-ai/docsmind/blob/main/LICENSE)
 [![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)
@@ -13,9 +11,8 @@
 Chat with your documents. Track what changed between versions, and why it matters.
 
 `docsmind` ingests PDFs (and soon DOCX/text) into a version-aware Postgres/pgvector
-store, and gives you an agent, **the Mind**, to chat with them. A built-in subagent,
-**diffra**, will do change analysis between document versions: what changed, where, and
-its compliance implications, with citations back to exact pages.
+store, and gives you an agent, **the Mind**, to analyze them. Built-in subagents
+will perform specialized tasks.
 
 > **Alpha.** Ingestion and inspection work today. Embeddings, chat and change analysis
 > are designed and scaffolded but not implemented yet; see [What works today](#what-works-today).
@@ -28,11 +25,13 @@ its compliance implications, with citations back to exact pages.
 | ✅ Postgres + pgvector storage | Local Docker, or any Postgres with pgvector (Neon, Supabase) |
 | ✅ Inspection CLI | `versions`, `fulltext`, `outline`, `chunks` read back exactly what was saved |
 | ✅ `docsmind doctor` | Checks database, blob storage and API keys, and tells you what to fix |
-| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
+| ✅ Table extraction | Extract and store tables skipping repeating ones (e.g. headers or footers) |
+| ✅ server + UI | FastAPI backend (`docsmind serve`) + React UI (`frontend/`): add documents, browse indexed content, image and table galleries |
+| ✅ Chat agent (the Mind) | Routes each question, plans complex ones, searches/reads your documents with tools, delegates to subagents, and answers with verified `[S#]` citations. Streams to the UI over SSE; conversations are saved |
 | 🚧 Image captioning | Images are extracted and stored; vision captioning is next |
-| 🚧 Chat agent (the Mind) | `docsmind chat` exists as a command, without an agent behind it yet |
+| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
 | 🚧 Change analysis (diffra) | `docsmind diff` exists as a command, without an implementation yet |
-| 🚧 DOCX/text parsers, S3/Supabase blob storage, server + UI | Planned |
+| 🚧 DOCX/text parsers, S3/Supabase blob storage | Planned |
 
 ## Install
 
@@ -42,7 +41,7 @@ pip install docsmind
 pip install "docsmind[all]"
 ```
 
-Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, planned UI),
+Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, serves the bundled UI),
 `docsmind[mlflow]` (tracing/eval export). Requires Python 3.10+.
 
 ## Quickstart
@@ -73,8 +72,9 @@ docsmind doctor
 ```
 
 Configuration is environment-driven (`DOCSMIND_*`, or a `.env` file). Copy
-`.env.example` to `.env` to see every option. Model API keys aren't needed yet;
-they'll be required once embeddings and chat land.
+`.env.example` to `.env` to see every option. Chat needs an LLM key
+(`DOCSMIND_LLM_PROVIDER` plus `DOCSMIND_ANTHROPIC_API_KEY` or `DOCSMIND_OPENAI_API_KEY`);
+ingestion and browsing don't.
 
 ### 2. Ingest a PDF
 
@@ -104,13 +104,81 @@ docsmind chunks <version-id> --page 12  # the chunk(s) saved for one page
 
 You can also inspect the tables directly with `docsmind db psql`.
 
-### 4. Coming next
+### 4. Browse it in the UI
+
+```bash
+pip install "docsmind[server]"
+docsmind serve              # FastAPI backend on http://127.0.0.1:8000
+```
+
+Open `http://127.0.0.1:8000`.
+
+The React UI is bundled into the Python package and served directly by `docsmind serve`
+from `docsmind/src/docsmind/server/ui`.
+
+Clicking **+ Add Document** in the UI runs the same `docsmind ingest` pipeline on the
+uploaded PDF. Selecting a document shows its saved full text/outline (**Content**), and
+galleries of everything extracted from it (**Images**, **Tables**).
+
+For contributors refreshing bundled UI assets before a release:
+
+```bash
+cd ../frontend
+npm install
+npm run build
+cd ../docsmind
+python scripts/bundle_frontend.py
+```
+
+### 5. Ask the Mind
+
+In the UI, open **Workspace** and ask a question (needs `docsmind[openai]` or `docsmind[anthropic]`
+plus an API key). From the terminal or Python:
+
+```bash
+docsmind chat "What are our data retention obligations?"   # one-shot
+docsmind chat                                              # REPL, keeps the conversation
+```
 
 ```python
-print(dm.chat("What are our data retention obligations?"))   # planned
-change_set = dm.compare(v1.id, v2.id)                        # planned (diffra)
+result = dm.chat_result("What are our data retention obligations?")
+print(result.answer, result.citations)
+change_set = dm.compare(v1.id, v2.id)   # planned (diffra)
+```
+
+## How the Mind works
+
+```
+Question -> Orchestrator --simple--> Executor <-> Tools / Subagents -> Answer
+                 |                       ^
+                 +--complex--> Planner --+        Harness feeds every role:
+                                                  context builder, prompt cache,
+                                                  summarizer, memory
 ```
 
+- **Orchestrator**: one cheap call that routes the question (`simple` or `complex`) and rewrites
+  it as a standalone question using the conversation so far.
+- **Planner** (complex only): a short list of steps the executor follows.
+- **Executor**: a bounded tool loop (`search`, `grep`, `read_pages`, `get_outline`,
+  `list_documents`, `delegate`). The last step forces an answer; tool errors go back to the
+  model, not the user; old tool results are elided when the context grows.
+- **Subagents** (`researcher`, `change_analyzer`): fresh context, a subset of the tools, run in
+  parallel, cannot delegate further.
+- **Harness**: deterministic code, not another agent. It orders the prompt stable-first so the
+  prefix caches, summarizes long histories, and saves durable notes to memory after each answer.
+- **Citations**: tools register passages as `S1, S2, ...`; the model cites `[S#]`; invented
+  handles are dropped and the rest renumbered by first appearance. Document text is treated as
+  untrusted data, never as instructions.
+- **Search** is Postgres full-text (ranked) plus exact-text grep until embeddings land.
+
+Tuning (all `DOCSMIND_*`): `AGENT_FAST_MODEL` (router/summarizer/memory model, defaults to
+`LLM_MODEL`), `AGENT_MAX_STEPS`, `AGENT_SUBAGENT_MAX_STEPS`, `AGENT_MEMORY_ENABLED`,
+`OPENAI_REASONING_EFFORT`, `OPENAI_BASE_URL`. See `.env.example`.
+
+HTTP: `POST /api/chat` streams Server-Sent Events (`route`, `plan`, `tool_start`, `tool_end`,
+`subagent_start`, `subagent_end`, `token`, `note`, `answer`, `error`, `done`);
+`GET /api/conversations[/{id}]`, `DELETE /api/conversations/{id}`, `GET /api/agent`.
+
 ## How ingestion works
 
 Ingestion is deliberately simple and predictable. There's no layout guessing, because
@@ -126,6 +194,9 @@ heuristics that work on one PDF break on the next.
   reconstruct structure.
 - **Images** are extracted with their page and position, skipping tiny icons, and saved
   under a readable path: `images/<collection>/<title>/<version>-<id>/`.
+  Visually repeated images (e.g. logo on every page) are de-duplicated by perceptual hash.
+- **Tables** are detected with PyMuPDF and rendered as PNG snippets, saved under
+  `tables/<collection>/<title>/<version>-<id>/`.
 
 ## CLI reference
 
@@ -136,9 +207,12 @@ heuristics that work on one PDF break on the next.
 | `docsmind fulltext <id> [--full]` | Print the saved full text |
 | `docsmind outline <id>` | Print the saved outline |
 | `docsmind chunks <id> [--page N] [--full]` | Print saved chunks |
+| `docsmind delete <id> [-y]` | Delete one version and its derived chunks/images/tables |
 | `docsmind db up` / `upgrade` / `psql` | Start local Postgres, apply migrations, open psql |
 | `docsmind doctor` | Check your setup |
-| `docsmind chat`, `docsmind diff` | Planned |
+| `docsmind serve [--host] [--port] [--reload]` | Run the FastAPI backend for the `frontend/` UI |
+| `docsmind chat [question] [-c <collection>]` | Ask the Mind once, or start a REPL that keeps the conversation |
+| `docsmind diff` | Planned |
 
 ## Architecture
 
@@ -150,7 +224,7 @@ src/docsmind/
 │
 ├── domain/                # pure data + interfaces, no I/O
 │   ├── documents.py       # Document, DocumentVersion, Section, Chunk, Image, ParsedDocument
-│   ├── changes.py         # Change, ChangeSet, ImpactAssessment (diffra)
+│   ├── changes.py         # Change, ChangeSet, ImpactAssessment
 │   ├── citations.py       # Citation
 │   ├── tables.py          # AgentTable, TableRevision
 │   └── ports.py           # LLM, Embedder, VisionCaptioner, VectorStore, BlobStore, Parser
@@ -167,13 +241,24 @@ src/docsmind/
 │   ├── llm/ embeddings/ vision/ tracing/   # planned
 │
 ├── cli/                   # Typer app: ingest, versions, fulltext, outline, chunks, db, doctor
-├── retrieval/             # hybrid vector + keyword search (planned)
-├── agent/                 # the Mind + diffra subagent (planned)
-└── server/                # FastAPI + UI (planned)
+├── retrieval/             # keyword (full-text) search helpers; vector search planned
+├── agent/                 # the Mind: orchestrator, planner, tool loop, tools, subagents, harness
+└── server/                # FastAPI app + routes backing frontend/ (documents, chat SSE, conversations)
 
-alembic/   tests/   examples/   docs/   ui/
+alembic/   tests/   examples/   docs/
 ```
 
+The UI source lives one level up as a sibling project: `frontend/` (React + TypeScript +
+TailwindCSS, see the repo-root `CLAUDE.md`). Release builds are bundled into
+`src/docsmind/server/ui` and served by `server/app.py` at `/`. The UI talks to `server/`
+over `/api/*`, and loads extracted images/tables straight from disk via `/blobs/*`.
+
+## Artwork attribution
+
+- Workspace 3D graph artwork inspiration: Angela Galliat,
+  "Neural nervous system of a brain" (CodePen):
+  https://codepen.io/agalliat/pen/vYGXJxQ
+
 ### Design notes
 
 - **Ports and adapters, once.** `domain/ports.py` defines every external dependency as a
@@ -188,7 +273,9 @@ alembic/   tests/   examples/   docs/   ui/
   Outline paths (e.g. `03.13.11`) give version comparison something stable to align on
   where a document has a table of contents.
 - **Tables are shared state, not a UI feature.** `AgentTable` is a domain object that
-  both the agent and the user write to (append-only revisions).
+  both the agent and the user write to (append-only revisions). This is distinct from
+  the `tables` storage table (extracted PDF table *regions*, rendered as images
+  alongside `images` — browsable read-only in the UI's **Tables** gallery).
 
 ## Development
 

{docsmind-0.2.0 → docsmind-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docsmind"
-version = "0.2.0"
+version = "0.4.0"
 description = "Chat with your documents. Version-aware retrieval, change analysis, and an editable table workspace, backed by Postgres/pgvector."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -38,12 +38,13 @@ dependencies = [
     "pillow>=10.0",
     "imagehash>=4.3",
     "rich>=13.7",
+    "tiktoken>=0.7",
 ]
 
 [project.optional-dependencies]
 openai = ["openai>=1.30"]
 anthropic = ["anthropic>=0.34"]
-server = ["fastapi>=0.111", "uvicorn[standard]>=0.30"]
+server = ["fastapi>=0.111", "uvicorn[standard]>=0.30", "python-multipart>=0.0.9"]
 mlflow = ["mlflow>=2.14"]
 dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "testcontainers[postgres]>=4.4", "ruff>=0.5", "mypy>=1.10"]
 all = ["docsmind[openai,anthropic,server]"]
@@ -61,6 +62,15 @@ Issues = "https://github.com/yauheniya-ai/docsmind/issues"
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/docsmind"]
+include = ["src/docsmind/server/ui/**"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/docsmind/**",
+  "README.md",
+  "CHANGELOG.md",
+  "scripts/bundle_frontend.py",
+]
 
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
@@ -79,4 +89,4 @@ target-version = "py310"
 # image is corrupt" — and a single malformed image in an otherwise-fine PDF
 # shouldn't abort the whole ingest, so we deliberately catch broadly here and
 # skip the one image rather than propagate.
-"src/docsmind/ingestion/parsers/pdf.py" = ["BLE001"]
+"src/docsmind/ingestion/parsers/pdf.py" = ["BLE001"]

docsmind-0.4.0/src/docsmind/adapters/llm/__init__.py

@@ -0,0 +1,31 @@
+"""LLM adapter registry. SDKs are imported lazily inside each adapter, so a
+bare `pip install docsmind` never needs openai/anthropic installed."""
+
+from __future__ import annotations
+
+from docsmind.config import DocsmindConfig
+from docsmind.domain.errors import LLMConfigurationError
+from docsmind.domain.ports import LLM
+
+
+def create_llm(config: DocsmindConfig, *, model: str | None = None) -> LLM:
+    """Build the configured LLM. `model` overrides `config.llm_model` (used
+    for the fast model that routes, summarizes and writes memory)."""
+    provider = config.llm_provider.lower()
+    model = model or config.llm_model
+    if provider == "openai":
+        from docsmind.adapters.llm.openai_llm import OpenAILLM
+
+        return OpenAILLM(
+            api_key=config.openai_api_key,
+            model=model,
+            base_url=config.openai_base_url,
+            reasoning_effort=config.openai_reasoning_effort,
+        )
+    if provider == "anthropic":
+        from docsmind.adapters.llm.anthropic_llm import AnthropicLLM
+
+        return AnthropicLLM(api_key=config.anthropic_api_key, model=model)
+    raise LLMConfigurationError(
+        f"Unknown DOCSMIND_LLM_PROVIDER '{config.llm_provider}'. Use 'anthropic' or 'openai'."
+    )