doclex-cli 0.1.1__tar.gz → 0.2.1__tar.gz

doclex_cli-0.2.1/CHANGELOG.md

@@ -0,0 +1,75 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/).
+
+## [0.2.1] - 2026-11-08
+
+### Fixed
+
+- Search scanned the whole segments table on every query to map index ids back to segments, so its cost grew with the corpus: on 25,000 segments that lookup took 94 ms of a 110 ms search. The lookup is now indexed and search takes 10-13 ms. Existing indexes pick this up when next opened; no reindex is needed.
+
+## [0.2.0] - 2026-09-14
+
+Search returns **documents, not fragments**, cited by page. Upgrading requires a full reindex (`doclex index --project <name> --force`); Doclex warns when it finds an index built by an older version.
+
+### Added
+
+- Results are grouped by document: a file matching in many places is listed once, with its best passages beneath it. Candidate passages are over-fetched so grouping still fills the result page.
+- `doclex search --passages N` — how many excerpts to show per document.
+- `doclex search --json` — machine-readable results with match offsets, for other frontends.
+- File names and their containing folders are scored in their own BM25 index, so a document whose title states its subject outranks one that merely quotes the words.
+- Document scores are scaled by how much of the query their best passage covers, each term weighted by its rarity in the corpus, so matching a rare term outweighs repeating a common one.
+- `doclex status --all --filter no-text` — finds scanned PDFs, which index cleanly but hold no searchable text and can never be found.
+- Index format versioning: `status` and `search` report an index built by an older Doclex instead of silently searching stale segments.
+
+### Changed
+
+- PDFs are segmented by page, taken from the extractor's per-page text. Citations read "Page 12" instead of a chunk index that named nothing. The old form-feed split never ran, since the extractor emits none.
+- Structural units are kept whole. The 1024-character limit is now a 8192-character safety valve for pathological input rather than the retrieval strategy; the reference corpus went from 3054 segments to 725.
+- Snippets carry match offsets instead of pre-rendered markup, letting each frontend highlight its own way.
+- Search no longer loads every segment in the corpus to map index ids back to segments.
+- Search startup dropped from 2.1s to 190ms: the file-name index is persisted rather than rebuilt per process, and extraction is imported lazily.
+
+### Fixed
+
+- Query terms matched as substrings when building snippets, so "me" landed inside "conforme" and steered results to the wrong passage. Terms now match as words.
+- Overlapping matches ("ande" and "de" inside "ANDE") spliced broken markup and crashed the renderer.
+- Document text containing square brackets was parsed as console markup.
+
+## [0.1.1] - 2026-08-02
+
+### Fixed
+
+- Search result snippets now anchor on the window with the highest density of query-term matches instead of the first occurrence of the first term, so the snippet favors where the query actually clusters rather than an early, isolated mention.
+
+## [0.1.0] - 2026-08-02
+
+First release — CLI MVP (Fase 1): local, embeddings-free document search over PDF, DOCX, XLSX, CSV, Markdown and TXT files.
+
+### Added
+
+- Local document search CLI (`doclex`) — BM25 lexical search, no embeddings, no vector DB, no cloud.
+- Ingestion pipeline: folder scanning with hash-based change detection, Kreuzberg-based text extraction, structural segmentation per file type (pages, sheets, sections, blocks), BM25X indexing, SQLite metadata store, and a local extraction cache.
+- Real-time incremental updates via Watchdog file monitoring, plus full reconciliation on startup.
+- `doclex add`/`projects`/`remove` — named project registry mapping folder paths to project names, rejecting overlapping/nested folders.
+- `doclex index --project <name> [--file ...] [--force]` — incremental or forced full reindexing, with per-file targeting.
+- `doclex search <query> --project <name> [--top] [--full]` — top-k BM25 search with citations (file, location, score, snippet).
+- `doclex status --project <name> [--file ...] [--all] [--filter]` — corpus/index status, file listing, and per-file inspection.
+- OCR support (`--ocr`) for scanned PDFs, and layout-aware segmentation via Kreuzberg.
+- Recursive character-based splitter keeping segments within a 1024-character limit while preserving semantic boundaries.
+
+### Changed
+
+- CLI output migrated to Rich (progress bars, tables) instead of plain text.
+- Segmentation cache replaced with a unified page-index-style tree; split sub-segments now nest under their extracted heading parent.
+- `list`/`reindex` merged into `index`/`status` flags; cache directory renamed from `.cache` to `.doclex`.
+- Layout-aware PDF segmentation made the default (removed the `--precise` flag).
+
+### Fixed
+
+- Ghost BM25 index entries left behind after a forced reindex (index and SQLite are now rebuilt fully in sync).
+- `status`/`search` no longer create the index directory before a project has actually been indexed.
+- Search result snippets now tokenize the query the same way BM25X tokenizes documents (splitting on punctuation, not just whitespace) and snap to word boundaries instead of cutting words mid-way.

{doclex_cli-0.1.1 → doclex_cli-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: doclex-cli
-Version: 0.1.1
+Version: 0.2.1
 Summary: Local document search — BM25 lexical search over PDFs, DOCX, XLSX, CSV, Markdown and TXT files. No embeddings, no vector DB, no cloud.
 Project-URL: Repository, https://github.com/EnPaiva93/doclex-cli
 Author-email: EnPaiva_93 <enpaiva93@gmail.com>
@@ -64,13 +64,21 @@ uv run doclex search "quarterly budget" --project mydocs
 ```
 
 ```
-[1] budget_2024.pdf — page 3  (score: 4.812)
-    "...the quarterly budget was approved with a 12% increase in..."
-
-[2] notes.md — Planning  (score: 2.107)
-    "...need to revisit the budget assumptions before..."
+[1] budget_2024.pdf  4.812
+    finance/2024
+    Page 3
+    ...the quarterly budget was approved with a 12% increase in...
+    Page 11
+    ...budget variance is reviewed at the end of each quarter...
+
+[2] notes.md  2.107
+    planning
+    Planning
+    ...need to revisit the budget assumptions before...
 ```
 
+Results are **documents, not fragments**: a file that matches in twenty places is listed once, with its best passages beneath it. Use `--passages` to show more excerpts per document, and `--json` to get the same results with match offsets for your own highlighting.
+
 ## Walkthrough
 
 A typical day-to-day flow, from an empty project to keeping it in sync:
@@ -91,11 +99,15 @@ Processing: 42 new, 0 modified, 0 deleted
 doclex search "termination clause" --project contracts
 ```
 ```
-[1] MSA_2023_AcmeCorp.pdf — Section 9.2  (score: 6.104)
-    "...either party may terminate this Agreement upon 30 days written notice..."
-
-[2] vendor_agreement_v3.docx — Termination  (score: 5.887)
-    "...termination for convenience requires 60 days advance notice to..."
+[1] MSA_2023_AcmeCorp.pdf  6.104
+    vendors/acme
+    Section 9.2
+    ...either party may terminate this Agreement upon 30 days written notice...
+
+[2] vendor_agreement_v3.docx  5.887
+    vendors
+    Termination
+    ...termination for convenience requires 60 days advance notice to...
 ```
 ```bash
 # 4. check what's indexed
@@ -137,6 +149,21 @@ doclex index --project hr-policies
 doclex search "vacation days" --project hr-policies --top 5
 ```
 
+## Search latency
+
+`uv run` re-resolves the environment on every invocation, which costs about two
+seconds before Doclex runs at all. For searching, install the command once and
+call it directly:
+
+```bash
+uv tool install .
+doclex search "quarterly budget" --project mydocs
+```
+
+A search then costs roughly a second, most of it Python importing its own
+dependencies. Repeated searches inside one process — a future server or MCP
+frontend — cost about 70 ms.
+
 ## Projects
 
 Every command operates on a **registered project** — a name bound to a folder — instead of a raw `--folder` flag. This keeps folder resolution unambiguous and lets Doclex reject overlapping folders (a subfolder registered as its own project would silently duplicate indexing and caching):
@@ -156,7 +183,7 @@ doclex remove mydocs            # deregisters and purges its cache/index
 | `doclex list` | List registered projects |
 | `doclex remove <name>` | Remove a project and purge its cache/index |
 | `doclex index --project <name>` | Scan and index (incremental by default) |
-| `doclex search <query> --project <name>` | Search and print results with citations |
+| `doclex search <query> --project <name>` | Search and print matching documents with citations |
 | `doclex status --project <name>` | Show corpus/index status |
 
 Run `doclex <command> --help` for all flags and examples.
@@ -173,8 +200,17 @@ doclex index --project mydocs --file report.pdf --file notes.docx
 # more results, full segment text instead of a snippet
 doclex search "risk assessment" --project mydocs --top 20 --full
 
+# show up to three matching passages per document
+doclex search "risk assessment" --project mydocs --passages 3
+
+# machine-readable output, with match offsets for highlighting
+doclex search "risk assessment" --project mydocs --json
+
 # list every indexed file and its status
 doclex status --project mydocs --all --filter error
+
+# find scanned PDFs: they index cleanly but hold no searchable text
+doclex status --project mydocs --all --filter no-text
 ```
 
 ## Architecture

{doclex_cli-0.1.1 → doclex_cli-0.2.1}/README.md

@@ -44,13 +44,21 @@ uv run doclex search "quarterly budget" --project mydocs
 ```
 
 ```
-[1] budget_2024.pdf — page 3  (score: 4.812)
-    "...the quarterly budget was approved with a 12% increase in..."
+[1] budget_2024.pdf  4.812
+    finance/2024
+    Page 3
+    ...the quarterly budget was approved with a 12% increase in...
+    Page 11
+    ...budget variance is reviewed at the end of each quarter...
 
-[2] notes.md — Planning  (score: 2.107)
-    "...need to revisit the budget assumptions before..."
+[2] notes.md  2.107
+    planning
+    Planning
+    ...need to revisit the budget assumptions before...
 ```
 
+Results are **documents, not fragments**: a file that matches in twenty places is listed once, with its best passages beneath it. Use `--passages` to show more excerpts per document, and `--json` to get the same results with match offsets for your own highlighting.
+
 ## Walkthrough
 
 A typical day-to-day flow, from an empty project to keeping it in sync:
@@ -71,11 +79,15 @@ Processing: 42 new, 0 modified, 0 deleted
 doclex search "termination clause" --project contracts
 ```
 ```
-[1] MSA_2023_AcmeCorp.pdf — Section 9.2  (score: 6.104)
-    "...either party may terminate this Agreement upon 30 days written notice..."
+[1] MSA_2023_AcmeCorp.pdf  6.104
+    vendors/acme
+    Section 9.2
+    ...either party may terminate this Agreement upon 30 days written notice...
 
-[2] vendor_agreement_v3.docx — Termination  (score: 5.887)
-    "...termination for convenience requires 60 days advance notice to..."
+[2] vendor_agreement_v3.docx  5.887
+    vendors
+    Termination
+    ...termination for convenience requires 60 days advance notice to...
 ```
 ```bash
 # 4. check what's indexed
@@ -117,6 +129,21 @@ doclex index --project hr-policies
 doclex search "vacation days" --project hr-policies --top 5
 ```
 
+## Search latency
+
+`uv run` re-resolves the environment on every invocation, which costs about two
+seconds before Doclex runs at all. For searching, install the command once and
+call it directly:
+
+```bash
+uv tool install .
+doclex search "quarterly budget" --project mydocs
+```
+
+A search then costs roughly a second, most of it Python importing its own
+dependencies. Repeated searches inside one process — a future server or MCP
+frontend — cost about 70 ms.
+
 ## Projects
 
 Every command operates on a **registered project** — a name bound to a folder — instead of a raw `--folder` flag. This keeps folder resolution unambiguous and lets Doclex reject overlapping folders (a subfolder registered as its own project would silently duplicate indexing and caching):
@@ -136,7 +163,7 @@ doclex remove mydocs            # deregisters and purges its cache/index
 | `doclex list` | List registered projects |
 | `doclex remove <name>` | Remove a project and purge its cache/index |
 | `doclex index --project <name>` | Scan and index (incremental by default) |
-| `doclex search <query> --project <name>` | Search and print results with citations |
+| `doclex search <query> --project <name>` | Search and print matching documents with citations |
 | `doclex status --project <name>` | Show corpus/index status |
 
 Run `doclex <command> --help` for all flags and examples.
@@ -153,8 +180,17 @@ doclex index --project mydocs --file report.pdf --file notes.docx
 # more results, full segment text instead of a snippet
 doclex search "risk assessment" --project mydocs --top 20 --full
 
+# show up to three matching passages per document
+doclex search "risk assessment" --project mydocs --passages 3
+
+# machine-readable output, with match offsets for highlighting
+doclex search "risk assessment" --project mydocs --json
+
 # list every indexed file and its status
 doclex status --project mydocs --all --filter error
+
+# find scanned PDFs: they index cleanly but hold no searchable text
+doclex status --project mydocs --all --filter no-text
 ```
 
 ## Architecture

{doclex_cli-0.1.1 → doclex_cli-0.2.1}/pyproject.toml

@@ -1,47 +1,47 @@
-[build-system]
-requires = ["hatchling"]
-build-backend = "hatchling.build"
-
-[project]
-name = "doclex-cli"
-version = "0.1.1"
-description = "Local document search — BM25 lexical search over PDFs, DOCX, XLSX, CSV, Markdown and TXT files. No embeddings, no vector DB, no cloud."
-readme = "README.md"
-license = "MIT"
-authors = [
-    { name = "EnPaiva_93", email = "enpaiva93@gmail.com" },
-]
-requires-python = ">=3.12"
-dependencies = [
-    "kreuzberg==4.8.2",
-    "bm25x",
-    "watchdog",
-    "aiosqlite",
-    "typer",
-]
-
-[project.urls]
-Repository = "https://github.com/EnPaiva93/doclex-cli"
-
-[project.scripts]
-doclex = "doclex.cli:app"
-
-[project.optional-dependencies]
-dev = [
-    "pytest",
-    "pytest-asyncio",
-    "ranx",
-]
-
-[tool.pytest.ini_options]
-asyncio_mode = "auto"
-testpaths = ["tests"]
-
-[tool.hatch.build.targets.wheel]
-packages = ["src/doclex"]
-
-[dependency-groups]
-dev = [
-    "ranx>=0.3.21",
-    "tqdm>=4.0",
-]
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "doclex-cli"
+version = "0.2.1"
+description = "Local document search — BM25 lexical search over PDFs, DOCX, XLSX, CSV, Markdown and TXT files. No embeddings, no vector DB, no cloud."
+readme = "README.md"
+license = "MIT"
+authors = [
+    { name = "EnPaiva_93", email = "enpaiva93@gmail.com" },
+]
+requires-python = ">=3.12"
+dependencies = [
+    "kreuzberg==4.8.2",
+    "bm25x",
+    "watchdog",
+    "aiosqlite",
+    "typer",
+]
+
+[project.urls]
+Repository = "https://github.com/EnPaiva93/doclex-cli"
+
+[project.scripts]
+doclex = "doclex.cli:app"
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "pytest-asyncio",
+    "ranx",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/doclex"]
+
+[dependency-groups]
+dev = [
+    "ranx>=0.3.21",
+    "tqdm>=4.0",
+]