docsmind 0.3.0__tar.gz → 0.6.0__tar.gz

{docsmind-0.3.0 → docsmind-0.6.0}/CHANGELOG.md

@@ -2,6 +2,82 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.6.0] - 2026-11-10
+
+### Added
+
+- Workspace: clicking a source card or citation chip opens the cited PDF page in a side panel
+  next to the chat.
+- Images/Tables galleries: full-view lightbox (blurred backdrop, arrow-key navigation) with
+  the full image caption beneath.
+- Chunk preview shows the embedding vector on one truncated line, expandable to the full
+  vector (`GET /api/documents/{version_id}/chunks/{chunk_id}/embedding`; chunk list now
+  includes `embedding_model`, `embedding_dims`, `embedding_preview`).
+- Agents: clicking a node opens a details panel next to the graph, like the source panel in the
+  Workspace. For an agent (the Mind, subagents, router and planner) it shows the description,
+  model, step budget, tools, the full system prompt (formatted or raw, with copy), the context
+  blocks the harness adds and a description of the user prompt. For a tool it shows the
+  description the model reads, which agents can use it and its arguments (type, required,
+  default, constraints) with the JSON Schema. Tool chips and "Available to" chips navigate
+  between panels; Escape or clicking the empty graph closes it.
+
+### Changed
+
+- `GET /api/agent`: every agent and stage now includes `system_prompt`, `context` (blocks with
+  `label`, `text`, `cached`) and `user_prompt`; every tool now includes `instructions` (the
+  description sent to the model) and `parameters` (its JSON Schema), in addition to `name` and
+  `description`.
+
+### Fixed
+
+- Image captions are now shown in the gallery and refresh when a captioning job finishes.
+- Captioning prompt no longer marks logos and banners as "decorative".
+
+## [0.5.0] - 2026-11-02
+
+### Changed
+
+- Agents view is now an interactive graph: the Mind in the centre, its tools around it, and the
+  subagents with their own tools beyond. Hover (or tap) a node for its description. Names are
+  shown as "Change Analyzer", not `change_analyzer`; the router and planner appear as dashed
+  stages of the Mind (one model call, no tools), not as subagents.
+- `GET /api/agent`: `tools` are now `{name, description}` objects (plain-language summaries),
+  and a new `stages` list describes the router and planner.
+
+### Added
+
+- On-demand enrichment for both pending roadmap items:
+  - Embeddings: estimate by token count, confirmation step, resumable run (`docsmind embed`,
+    `POST /api/enrichment/embeddings/{estimate,run}`).
+  - Image captioning: estimate by image count/resolution + output-token range, confirmation step,
+    resumable run (`docsmind caption`, `POST /api/enrichment/captions/{estimate,run}`).
+- Enrichment status + background jobs API:
+  `GET /api/enrichment/status`, `GET /api/enrichment/jobs/{job_id}`,
+  `POST /api/enrichment/jobs/{job_id}/cancel`.
+- Frontend Document details now shows "not embedded yet" / "not captioned yet" alerts with
+  estimate-and-proceed dialogs and job progress/cancel controls.
+- Hybrid retrieval path: keyword search fused with vector similarity when embeddings are present,
+  with automatic fallback to keyword-only when embeddings or API access are unavailable.
+
+## [0.4.0] - 2026-10-31
+
+### Added
+
+- The Mind: a chat agent (orchestrator, planner, tool-using executor, parallel subagents)
+  with a deterministic harness (context builder with prompt-cache breakpoint, summarizer,
+  memory writer). Answers carry verified `[S#]` citations.
+- Provider-neutral LLM layer with OpenAI and Anthropic streaming adapters.
+- Tools: `search`, `grep`, `read_pages`, `get_outline`, `list_documents`, `delegate`;
+  subagents: `researcher`, `change_analyzer`.
+- Migration `0004`: `conversations`, `messages`, `agent_memory` tables (run `docsmind db upgrade`).
+- HTTP: `POST /api/chat` (SSE), `GET/DELETE /api/conversations`, `GET /api/agent`.
+- UI: streaming chat in Workspace with live agent activity, markdown answers, citation chips
+  and source cards that open the PDF at the cited page; Activity lists saved conversations;
+  Agents reads the real registry. New frontend deps: `react-markdown`, `remark-gfm`.
+- `Docsmind.chat_result()`; `chat`/`achat` accept `conversation_id`; `docsmind chat` REPL keeps
+  the conversation.
+- Config: `agent_*` settings, `openai_base_url`, `openai_reasoning_effort`.
+
 ## [0.3.0] - 2026-10-03
 
 ### Added

{docsmind-0.3.0 → docsmind-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docsmind
-Version: 0.3.0
+Version: 0.6.0
 Summary: Chat with your documents. Version-aware retrieval, change analysis, and an editable table workspace, backed by Postgres/pgvector.
 Project-URL: Homepage, https://github.com/yauheniya-ai/docsmind
 Project-URL: Repository, https://github.com/yauheniya-ai/docsmind
@@ -45,6 +45,8 @@ Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: ruff>=0.5; extra == 'dev'
 Requires-Dist: testcontainers[postgres]>=4.4; extra == 'dev'
+Provides-Extra: langfuse
+Requires-Dist: langfuse>=2.0; extra == 'langfuse'
 Provides-Extra: mlflow
 Requires-Dist: mlflow>=2.14; extra == 'mlflow'
 Provides-Extra: openai
@@ -68,9 +70,8 @@ Description-Content-Type: text/markdown
 Chat with your documents. Track what changed between versions, and why it matters.
 
 `docsmind` ingests PDFs (and soon DOCX/text) into a version-aware Postgres/pgvector
-store, and gives you an agent, **the Mind**, to chat with them. A built-in subagent,
-**diffra**, will do change analysis between document versions: what changed, where, and
-its compliance implications, with citations back to exact pages.
+store, and gives you an agent, **the Mind**, to analyze them. Built-in subagents
+will perform specialized tasks.
 
 > **Alpha.** Ingestion and inspection work today. Embeddings, chat and change analysis
 > are designed and scaffolded but not implemented yet; see [What works today](#what-works-today).
@@ -85,9 +86,9 @@ its compliance implications, with citations back to exact pages.
 | ✅ `docsmind doctor` | Checks database, blob storage and API keys, and tells you what to fix |
 | ✅ Table extraction | Extract and store tables skipping repeating ones (e.g. headers or footers) |
 | ✅ server + UI | FastAPI backend (`docsmind serve`) + React UI (`frontend/`): add documents, browse indexed content, image and table galleries |
-| 🚧 Chat agent (the Mind) | `docsmind chat` exists as a command, without an agent behind it yet |
-| 🚧 Image captioning | Images are extracted and stored; vision captioning is next |
-| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
+| ✅ Chat agent (the Mind) | Routes each question, plans complex ones, searches/reads your documents with tools, delegates to subagents, and answers with verified `[S#]` citations. Streams to the UI over SSE; conversations are saved |
+| ✅ Image captioning (on demand) | The UI/CLI shows what's uncaptioned, estimates cost from image count/resolution, asks for confirmation, then captions only what's missing |
+| ✅ Embeddings + hybrid search (on demand) | The UI/CLI shows what's unembedded, estimates cost from token counts, asks for confirmation, then embeds only what's missing; search uses keyword+semantic fusion when available |
 | 🚧 Change analysis (diffra) | `docsmind diff` exists as a command, without an implementation yet |
 | 🚧 DOCX/text parsers, S3/Supabase blob storage | Planned |
 
@@ -100,7 +101,7 @@ pip install "docsmind[all]"
 ```
 
 Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, serves the bundled UI),
-`docsmind[mlflow]` (tracing/eval export). Requires Python 3.10+.
+`docsmind[mlflow]` / `docsmind[langfuse]` (tracing/eval export, planned). Requires Python 3.10+.
 
 ## Quickstart
 
@@ -130,8 +131,9 @@ docsmind doctor
 ```
 
 Configuration is environment-driven (`DOCSMIND_*`, or a `.env` file). Copy
-`.env.example` to `.env` to see every option. Model API keys aren't needed yet;
-they'll be required once embeddings and chat land.
+`.env.example` to `.env` to see every option. Chat needs an LLM key
+(`DOCSMIND_LLM_PROVIDER` plus `DOCSMIND_ANTHROPIC_API_KEY` or `DOCSMIND_OPENAI_API_KEY`);
+ingestion and browsing don't.
 
 ### 2. Ingest a PDF
 
@@ -187,13 +189,78 @@ cd ../docsmind
 python scripts/bundle_frontend.py
 ```
 
-### 5. Coming next
+### 5. Ask the Mind
+
+In the UI, open **Workspace** and ask a question (needs `docsmind[openai]` or `docsmind[anthropic]`
+plus an API key). From the terminal or Python:
+
+```bash
+docsmind chat "What are our data retention obligations?"   # one-shot
+docsmind chat                                              # REPL, keeps the conversation
+```
 
 ```python
-print(dm.chat("What are our data retention obligations?"))   # planned
-change_set = dm.compare(v1.id, v2.id)                        # planned (diffra)
+result = dm.chat_result("What are our data retention obligations?")
+print(result.answer, result.citations)
+change_set = dm.compare(v1.id, v2.id)   # planned (diffra)
+```
+
+## How the Mind works
+
+```
+Question -> Orchestrator --simple--> Executor <-> Tools / Subagents -> Answer
+                 |                       ^
+                 +--complex--> Planner --+        Harness feeds every role:
+                                                  context builder, prompt cache,
+                                                  summarizer, memory
+```
+
+- **Orchestrator**: one cheap call that routes the question (`simple` or `complex`) and rewrites
+  it as a standalone question using the conversation so far.
+- **Planner** (complex only): a short list of steps the executor follows.
+- **Executor**: a bounded tool loop (`search`, `grep`, `read_pages`, `get_outline`,
+  `list_documents`, `delegate`). The last step forces an answer; tool errors go back to the
+  model, not the user; old tool results are elided when the context grows.
+- **Subagents** (`researcher`, `change_analyzer`): fresh context, a subset of the tools, run in
+  parallel, cannot delegate further.
+- **Harness**: deterministic code, not another agent. It orders the prompt stable-first so the
+  prefix caches, summarizes long histories, and saves durable notes to memory after each answer.
+- **Citations**: tools register passages as `S1, S2, ...`; the model cites `[S#]`; invented
+  handles are dropped and the rest renumbered by first appearance. Document text is treated as
+  untrusted data, never as instructions.
+- **Search** is Postgres full-text (ranked) plus exact-text grep; when embeddings exist, semantic vector matches are fused in via reciprocal rank fusion.
+
+### On-demand enrichment
+
+Embeddings and image captions are **not** created during ingest. They are opt-in because they call paid APIs.
+
+- In the **Document details** page, warning cards show what's missing ("not embedded yet" / "not captioned yet").
+- Clicking **Estimate & run** opens a confirmation dialog with model, token/image counts, and estimated USD cost.
+- Running is idempotent and resumable: it only processes missing rows; cancel/retry is safe.
+
+From CLI:
+
+```bash
+docsmind embed [--version-id <id>] [-y]
+docsmind caption [--version-id <id>] [-y]
 ```
 
+HTTP:
+
+- `GET /api/enrichment/status`
+- `POST /api/enrichment/{embeddings|captions}/estimate`
+- `POST /api/enrichment/{embeddings|captions}/run`
+- `GET /api/enrichment/jobs/{job_id}`
+- `POST /api/enrichment/jobs/{job_id}/cancel`
+
+Tuning (all `DOCSMIND_*`): `AGENT_FAST_MODEL` (router/summarizer/memory model, defaults to
+`LLM_MODEL`), `AGENT_MAX_STEPS`, `AGENT_SUBAGENT_MAX_STEPS`, `AGENT_MEMORY_ENABLED`,
+`OPENAI_REASONING_EFFORT`, `OPENAI_BASE_URL`. See `.env.example`.
+
+HTTP: `POST /api/chat` streams Server-Sent Events (`route`, `plan`, `tool_start`, `tool_end`,
+`subagent_start`, `subagent_end`, `token`, `note`, `answer`, `error`, `done`);
+`GET /api/conversations[/{id}]`, `DELETE /api/conversations/{id}`, `GET /api/agent`.
+
 ## How ingestion works
 
 Ingestion is deliberately simple and predictable. There's no layout guessing, because
@@ -226,7 +293,10 @@ heuristics that work on one PDF break on the next.
 | `docsmind db up` / `upgrade` / `psql` | Start local Postgres, apply migrations, open psql |
 | `docsmind doctor` | Check your setup |
 | `docsmind serve [--host] [--port] [--reload]` | Run the FastAPI backend for the `frontend/` UI |
-| `docsmind chat`, `docsmind diff` | Planned |
+| `docsmind chat [question] [-c <collection>]` | Ask the Mind once, or start a REPL that keeps the conversation |
+| `docsmind embed [--version-id <id>] [-y]` | Estimate cost and create missing embeddings on demand |
+| `docsmind caption [--version-id <id>] [-y]` | Estimate cost and create missing image captions on demand |
+| `docsmind diff` | Planned |
 
 ## Architecture
 
@@ -238,7 +308,7 @@ src/docsmind/
 │
 ├── domain/                # pure data + interfaces, no I/O
 │   ├── documents.py       # Document, DocumentVersion, Section, Chunk, Image, ParsedDocument
-│   ├── changes.py         # Change, ChangeSet, ImpactAssessment (diffra)
+│   ├── changes.py         # Change, ChangeSet, ImpactAssessment
 │   ├── citations.py       # Citation
 │   ├── tables.py          # AgentTable, TableRevision
 │   └── ports.py           # LLM, Embedder, VisionCaptioner, VectorStore, BlobStore, Parser
@@ -255,9 +325,9 @@ src/docsmind/
 │   ├── llm/ embeddings/ vision/ tracing/   # planned
 │
 ├── cli/                   # Typer app: ingest, versions, fulltext, outline, chunks, db, doctor
-├── retrieval/             # hybrid vector + keyword search (planned)
-├── agent/                 # the Mind + diffra subagent (planned)
-└── server/                # FastAPI app + routes backing frontend/ (documents: ingest, read back content/images/tables)
+├── retrieval/             # keyword (full-text) search helpers; vector search planned
+├── agent/                 # the Mind: orchestrator, planner, tool loop, tools, subagents, harness
+└── server/                # FastAPI app + routes backing frontend/ (documents, chat SSE, conversations)
 
 alembic/   tests/   examples/   docs/
 ```
@@ -276,7 +346,7 @@ over `/api/*`, and loads extracted images/tables straight from disk via `/blobs/
 ### Design notes
 
 - **Ports and adapters, once.** `domain/ports.py` defines every external dependency as a
-  `Protocol`. The core never imports `openai`, `anthropic` or `mlflow` directly. Only
+  `Protocol`. The core never imports `openai`, `anthropic`, `mlflow` or `langfuse` directly. Only
   adapters do, and lazily, so a bare `pip install docsmind` doesn't pull in SDKs you're
   not using.
 - **Text and images share one retrieval path.** An image becomes a chunk

{docsmind-0.3.0 → docsmind-0.6.0}/README.md

@@ -11,9 +11,8 @@
 Chat with your documents. Track what changed between versions, and why it matters.
 
 `docsmind` ingests PDFs (and soon DOCX/text) into a version-aware Postgres/pgvector
-store, and gives you an agent, **the Mind**, to chat with them. A built-in subagent,
-**diffra**, will do change analysis between document versions: what changed, where, and
-its compliance implications, with citations back to exact pages.
+store, and gives you an agent, **the Mind**, to analyze them. Built-in subagents
+will perform specialized tasks.
 
 > **Alpha.** Ingestion and inspection work today. Embeddings, chat and change analysis
 > are designed and scaffolded but not implemented yet; see [What works today](#what-works-today).
@@ -28,9 +27,9 @@ its compliance implications, with citations back to exact pages.
 | ✅ `docsmind doctor` | Checks database, blob storage and API keys, and tells you what to fix |
 | ✅ Table extraction | Extract and store tables skipping repeating ones (e.g. headers or footers) |
 | ✅ server + UI | FastAPI backend (`docsmind serve`) + React UI (`frontend/`): add documents, browse indexed content, image and table galleries |
-| 🚧 Chat agent (the Mind) | `docsmind chat` exists as a command, without an agent behind it yet |
-| 🚧 Image captioning | Images are extracted and stored; vision captioning is next |
-| 🚧 Embeddings + hybrid search | Chunks are stored without embeddings for now |
+| ✅ Chat agent (the Mind) | Routes each question, plans complex ones, searches/reads your documents with tools, delegates to subagents, and answers with verified `[S#]` citations. Streams to the UI over SSE; conversations are saved |
+| ✅ Image captioning (on demand) | The UI/CLI shows what's uncaptioned, estimates cost from image count/resolution, asks for confirmation, then captions only what's missing |
+| ✅ Embeddings + hybrid search (on demand) | The UI/CLI shows what's unembedded, estimates cost from token counts, asks for confirmation, then embeds only what's missing; search uses keyword+semantic fusion when available |
 | 🚧 Change analysis (diffra) | `docsmind diff` exists as a command, without an implementation yet |
 | 🚧 DOCX/text parsers, S3/Supabase blob storage | Planned |
 
@@ -43,7 +42,7 @@ pip install "docsmind[all]"
 ```
 
 Extras: `docsmind[openai]`, `docsmind[anthropic]`, `docsmind[server]` (FastAPI, serves the bundled UI),
-`docsmind[mlflow]` (tracing/eval export). Requires Python 3.10+.
+`docsmind[mlflow]` / `docsmind[langfuse]` (tracing/eval export, planned). Requires Python 3.10+.
 
 ## Quickstart
 
@@ -73,8 +72,9 @@ docsmind doctor
 ```
 
 Configuration is environment-driven (`DOCSMIND_*`, or a `.env` file). Copy
-`.env.example` to `.env` to see every option. Model API keys aren't needed yet;
-they'll be required once embeddings and chat land.
+`.env.example` to `.env` to see every option. Chat needs an LLM key
+(`DOCSMIND_LLM_PROVIDER` plus `DOCSMIND_ANTHROPIC_API_KEY` or `DOCSMIND_OPENAI_API_KEY`);
+ingestion and browsing don't.
 
 ### 2. Ingest a PDF
 
@@ -130,13 +130,78 @@ cd ../docsmind
 python scripts/bundle_frontend.py
 ```
 
-### 5. Coming next
+### 5. Ask the Mind
+
+In the UI, open **Workspace** and ask a question (needs `docsmind[openai]` or `docsmind[anthropic]`
+plus an API key). From the terminal or Python:
+
+```bash
+docsmind chat "What are our data retention obligations?"   # one-shot
+docsmind chat                                              # REPL, keeps the conversation
+```
 
 ```python
-print(dm.chat("What are our data retention obligations?"))   # planned
-change_set = dm.compare(v1.id, v2.id)                        # planned (diffra)
+result = dm.chat_result("What are our data retention obligations?")
+print(result.answer, result.citations)
+change_set = dm.compare(v1.id, v2.id)   # planned (diffra)
+```
+
+## How the Mind works
+
+```
+Question -> Orchestrator --simple--> Executor <-> Tools / Subagents -> Answer
+                 |                       ^
+                 +--complex--> Planner --+        Harness feeds every role:
+                                                  context builder, prompt cache,
+                                                  summarizer, memory
+```
+
+- **Orchestrator**: one cheap call that routes the question (`simple` or `complex`) and rewrites
+  it as a standalone question using the conversation so far.
+- **Planner** (complex only): a short list of steps the executor follows.
+- **Executor**: a bounded tool loop (`search`, `grep`, `read_pages`, `get_outline`,
+  `list_documents`, `delegate`). The last step forces an answer; tool errors go back to the
+  model, not the user; old tool results are elided when the context grows.
+- **Subagents** (`researcher`, `change_analyzer`): fresh context, a subset of the tools, run in
+  parallel, cannot delegate further.
+- **Harness**: deterministic code, not another agent. It orders the prompt stable-first so the
+  prefix caches, summarizes long histories, and saves durable notes to memory after each answer.
+- **Citations**: tools register passages as `S1, S2, ...`; the model cites `[S#]`; invented
+  handles are dropped and the rest renumbered by first appearance. Document text is treated as
+  untrusted data, never as instructions.
+- **Search** is Postgres full-text (ranked) plus exact-text grep; when embeddings exist, semantic vector matches are fused in via reciprocal rank fusion.
+
+### On-demand enrichment
+
+Embeddings and image captions are **not** created during ingest. They are opt-in because they call paid APIs.
+
+- In the **Document details** page, warning cards show what's missing ("not embedded yet" / "not captioned yet").
+- Clicking **Estimate & run** opens a confirmation dialog with model, token/image counts, and estimated USD cost.
+- Running is idempotent and resumable: it only processes missing rows; cancel/retry is safe.
+
+From CLI:
+
+```bash
+docsmind embed [--version-id <id>] [-y]
+docsmind caption [--version-id <id>] [-y]
 ```
 
+HTTP:
+
+- `GET /api/enrichment/status`
+- `POST /api/enrichment/{embeddings|captions}/estimate`
+- `POST /api/enrichment/{embeddings|captions}/run`
+- `GET /api/enrichment/jobs/{job_id}`
+- `POST /api/enrichment/jobs/{job_id}/cancel`
+
+Tuning (all `DOCSMIND_*`): `AGENT_FAST_MODEL` (router/summarizer/memory model, defaults to
+`LLM_MODEL`), `AGENT_MAX_STEPS`, `AGENT_SUBAGENT_MAX_STEPS`, `AGENT_MEMORY_ENABLED`,
+`OPENAI_REASONING_EFFORT`, `OPENAI_BASE_URL`. See `.env.example`.
+
+HTTP: `POST /api/chat` streams Server-Sent Events (`route`, `plan`, `tool_start`, `tool_end`,
+`subagent_start`, `subagent_end`, `token`, `note`, `answer`, `error`, `done`);
+`GET /api/conversations[/{id}]`, `DELETE /api/conversations/{id}`, `GET /api/agent`.
+
 ## How ingestion works
 
 Ingestion is deliberately simple and predictable. There's no layout guessing, because
@@ -169,7 +234,10 @@ heuristics that work on one PDF break on the next.
 | `docsmind db up` / `upgrade` / `psql` | Start local Postgres, apply migrations, open psql |
 | `docsmind doctor` | Check your setup |
 | `docsmind serve [--host] [--port] [--reload]` | Run the FastAPI backend for the `frontend/` UI |
-| `docsmind chat`, `docsmind diff` | Planned |
+| `docsmind chat [question] [-c <collection>]` | Ask the Mind once, or start a REPL that keeps the conversation |
+| `docsmind embed [--version-id <id>] [-y]` | Estimate cost and create missing embeddings on demand |
+| `docsmind caption [--version-id <id>] [-y]` | Estimate cost and create missing image captions on demand |
+| `docsmind diff` | Planned |
 
 ## Architecture
 
@@ -181,7 +249,7 @@ src/docsmind/
 │
 ├── domain/                # pure data + interfaces, no I/O
 │   ├── documents.py       # Document, DocumentVersion, Section, Chunk, Image, ParsedDocument
-│   ├── changes.py         # Change, ChangeSet, ImpactAssessment (diffra)
+│   ├── changes.py         # Change, ChangeSet, ImpactAssessment
 │   ├── citations.py       # Citation
 │   ├── tables.py          # AgentTable, TableRevision
 │   └── ports.py           # LLM, Embedder, VisionCaptioner, VectorStore, BlobStore, Parser
@@ -198,9 +266,9 @@ src/docsmind/
 │   ├── llm/ embeddings/ vision/ tracing/   # planned
 │
 ├── cli/                   # Typer app: ingest, versions, fulltext, outline, chunks, db, doctor
-├── retrieval/             # hybrid vector + keyword search (planned)
-├── agent/                 # the Mind + diffra subagent (planned)
-└── server/                # FastAPI app + routes backing frontend/ (documents: ingest, read back content/images/tables)
+├── retrieval/             # keyword (full-text) search helpers; vector search planned
+├── agent/                 # the Mind: orchestrator, planner, tool loop, tools, subagents, harness
+└── server/                # FastAPI app + routes backing frontend/ (documents, chat SSE, conversations)
 
 alembic/   tests/   examples/   docs/
 ```
@@ -219,7 +287,7 @@ over `/api/*`, and loads extracted images/tables straight from disk via `/blobs/
 ### Design notes
 
 - **Ports and adapters, once.** `domain/ports.py` defines every external dependency as a
-  `Protocol`. The core never imports `openai`, `anthropic` or `mlflow` directly. Only
+  `Protocol`. The core never imports `openai`, `anthropic`, `mlflow` or `langfuse` directly. Only
   adapters do, and lazily, so a bare `pip install docsmind` doesn't pull in SDKs you're
   not using.
 - **Text and images share one retrieval path.** An image becomes a chunk

{docsmind-0.3.0 → docsmind-0.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docsmind"
-version = "0.3.0"
+version = "0.6.0"
 description = "Chat with your documents. Version-aware retrieval, change analysis, and an editable table workspace, backed by Postgres/pgvector."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -46,6 +46,7 @@ openai = ["openai>=1.30"]
 anthropic = ["anthropic>=0.34"]
 server = ["fastapi>=0.111", "uvicorn[standard]>=0.30", "python-multipart>=0.0.9"]
 mlflow = ["mlflow>=2.14"]
+langfuse = ["langfuse>=2.0"]
 dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "testcontainers[postgres]>=4.4", "ruff>=0.5", "mypy>=1.10"]
 all = ["docsmind[openai,anthropic,server]"]
 

docsmind-0.6.0/src/docsmind/adapters/embeddings/__init__.py

@@ -0,0 +1,22 @@
+"""Embedding providers. Today: OpenAI (and OpenAI-compatible gateways)."""
+
+from __future__ import annotations
+
+from docsmind.config import DocsmindConfig
+from docsmind.domain.errors import EnrichmentConfigurationError
+
+
+def create_embedder(config: DocsmindConfig):
+    if config.embedding_provider == "openai":
+        from docsmind.adapters.embeddings.openai_embeddings import OpenAIEmbedder
+
+        return OpenAIEmbedder(
+            api_key=config.openai_api_key,
+            model=config.embedding_model,
+            dimensions=config.embedding_dimensions,
+            base_url=config.openai_base_url,
+            batch_size=config.embedding_batch_size,
+        )
+    raise EnrichmentConfigurationError(
+        f"Unknown embedding_provider {config.embedding_provider!r}. Supported: openai."
+    )

docsmind-0.6.0/src/docsmind/adapters/embeddings/openai_embeddings.py

@@ -0,0 +1,79 @@
+"""OpenAI (and OpenAI-compatible) embeddings adapter. Satisfies `domain.ports.Embedder`."""
+
+from __future__ import annotations
+
+from docsmind.domain.errors import EnrichmentConfigurationError
+
+# The API rejects inputs over 8191 tokens; page chunks are far smaller, but a
+# dense page of tiny text can get there. Truncate rather than fail the batch.
+MAX_INPUT_TOKENS = 8000
+_MAX_REQUEST_TOKENS = 250_000  # API limit is 300k tokens per request
+
+
+class OpenAIEmbedder:
+    def __init__(
+        self,
+        *,
+        api_key: str | None,
+        model: str,
+        dimensions: int,
+        base_url: str | None = None,
+        batch_size: int = 64,
+        timeout: float = 120.0,
+    ) -> None:
+        if not api_key:
+            raise EnrichmentConfigurationError(
+                "OpenAI API key is not set. Add DOCSMIND_OPENAI_API_KEY to your .env to create embeddings."
+            )
+        try:
+            from openai import AsyncOpenAI
+        except ImportError as exc:  # pragma: no cover - import guard
+            raise EnrichmentConfigurationError(
+                'The openai package is not installed. Run `pip install "docsmind[openai]"`.'
+            ) from exc
+        self.model_name = model
+        self.dimensions = dimensions
+        self._batch_size = max(1, batch_size)
+        self._client = AsyncOpenAI(api_key=api_key, base_url=base_url, max_retries=4, timeout=timeout)
+
+    async def embed(self, texts: list[str]) -> list[list[float]]:
+        vectors: list[list[float]] = []
+        for batch in self._batches([_truncate(t) for t in texts]):
+            kwargs: dict = {"model": self.model_name, "input": batch}
+            # Only the v3 models accept (and need) a `dimensions` argument.
+            if "text-embedding-3" in self.model_name:
+                kwargs["dimensions"] = self.dimensions
+            response = await self._client.embeddings.create(**kwargs)
+            ordered = sorted(response.data, key=lambda d: d.index)
+            vectors.extend([list(d.embedding) for d in ordered])
+        if len(vectors) != len(texts):
+            raise RuntimeError(f"Embeddings API returned {len(vectors)} vectors for {len(texts)} inputs.")
+        return vectors
+
+    def _batches(self, texts: list[str]):
+        batch: list[str] = []
+        tokens = 0
+        for text in texts:
+            estimate = max(1, len(text) // 3)  # conservative: ~3 chars/token
+            if batch and (len(batch) >= self._batch_size or tokens + estimate > _MAX_REQUEST_TOKENS):
+                yield batch
+                batch, tokens = [], 0
+            batch.append(text)
+            tokens += estimate
+        if batch:
+            yield batch
+
+
+def _truncate(text: str) -> str:
+    # The API also rejects empty strings; a single space embeds harmlessly.
+    text = text if text.strip() else " "
+    try:
+        import tiktoken
+
+        encoding = tiktoken.get_encoding("cl100k_base")
+        tokens = encoding.encode(text, disallowed_special=())
+        if len(tokens) > MAX_INPUT_TOKENS:
+            return encoding.decode(tokens[:MAX_INPUT_TOKENS])
+        return text
+    except Exception:  # noqa: BLE001 - fall back to a character cap
+        return text[: MAX_INPUT_TOKENS * 3]

docsmind-0.6.0/src/docsmind/adapters/llm/__init__.py

@@ -0,0 +1,31 @@
+"""LLM adapter registry. SDKs are imported lazily inside each adapter, so a
+bare `pip install docsmind` never needs openai/anthropic installed."""
+
+from __future__ import annotations
+
+from docsmind.config import DocsmindConfig
+from docsmind.domain.errors import LLMConfigurationError
+from docsmind.domain.ports import LLM
+
+
+def create_llm(config: DocsmindConfig, *, model: str | None = None) -> LLM:
+    """Build the configured LLM. `model` overrides `config.llm_model` (used
+    for the fast model that routes, summarizes and writes memory)."""
+    provider = config.llm_provider.lower()
+    model = model or config.llm_model
+    if provider == "openai":
+        from docsmind.adapters.llm.openai_llm import OpenAILLM
+
+        return OpenAILLM(
+            api_key=config.openai_api_key,
+            model=model,
+            base_url=config.openai_base_url,
+            reasoning_effort=config.openai_reasoning_effort,
+        )
+    if provider == "anthropic":
+        from docsmind.adapters.llm.anthropic_llm import AnthropicLLM
+
+        return AnthropicLLM(api_key=config.anthropic_api_key, model=model)
+    raise LLMConfigurationError(
+        f"Unknown DOCSMIND_LLM_PROVIDER '{config.llm_provider}'. Use 'anthropic' or 'openai'."
+    )