doc3gpp 0.3.0__tar.gz → 0.3.2__tar.gz

{doc3gpp-0.3.0 → doc3gpp-0.3.2}/AGENTS.md

@@ -357,7 +357,7 @@ Workflows in one line (full prose in `docs/architecture.md`):
    rebuild flag. Hybrid weighting comes from
    `Settings.semantic_search.fts5_weight`.
  - `doc3gpp spec doc parse --spec <id> [--release R] [--version V] [--force]` → `SpecDocService.parse_many` → resolve the numeric-newest `SpecVersion` with a non-empty `.zip` URL via `resolve_spec_doc_version` (after optional release filtering); explicit `--version` is strict and fails clearly when its ZIP link is missing. Fetch the ZIP with internal `fetch_spec_doc_zip` when absent (or use the zip cache at `{spec_doc.cache_dir}/zips/<spec_id>/<version>.zip`), and record the download. `--force` re-downloads before re-parsing, including pairs that already carry `parsed_at`.
- - `doc3gpp spec doc parse --spec <id> [--release R] [--version V] [--force]` → per spec: immutable-skip when `(spec_id, version)` already carries `parsed_at` (unless `--force`) → `parse` = fetch-if-missing + `list_spec_docx` + `convert_document_to_blocks` + `order_spec_files` + `extract_spec_toc` + `chunk_blocks` → `upsert_toc` + `replace_chunks` + per-file markdown cache (`{spec_doc.cache_dir}/markdown/<spec_id>/<version>/`) + `record_parsed` + best-effort auto-index (`upsert_for_version`) / auto-embed (`index_for_version`). Oversized zips → skip bucket; unknown spec/version or empty (no-docx) zips → failure bucket; one spec never aborts the batch (`SpecDocBatchResult(successes/skipped/failures)`). Prints `ok/skipped/failed` buckets only (`--format/--output/--compact` accepted but ignored). Chunk/search metadata uses newline-delimited combined `sections` and `tables`; TOC entries retain `section_no` / `title`.
+ - `doc3gpp spec doc parse --spec <id> [--release R] [--version V] [--force]` → per spec: immutable-skip when `(spec_id, version)` already carries `parsed_at` (unless `--force`) → `parse` = fetch-if-missing + `list_spec_docx` + `convert_document_to_blocks` + `order_spec_files` + `extract_spec_toc` + `chunk_blocks` → `upsert_toc` + `replace_chunks` + per-file markdown cache (`{spec_doc.cache_dir}/markdown/<spec_id>/<version>/`) + `record_parsed` + best-effort auto-index (`upsert_for_version`) / auto-embed (`index_for_version`). Oversized zips → skip bucket; unknown spec/version or empty (no-docx) zips → failure bucket; one spec never aborts the batch (`SpecDocBatchResult(successes/skipped/failures)`). Prints `ok/skipped/failed` buckets only (`--format/--output/--compact` accepted but ignored). Background `parse_spec_docs` jobs log and store per-spec errors in `result.failure_details`; all-failed batches have job status `failed`, while partial successes remain `succeeded`. Chunk/search metadata uses newline-delimited combined `sections` and `tables`; TOC entries retain `section_no` / `title`.
  - `doc3gpp spec doc toc show --spec <id> --version <v> [--fields] [--format/--output/--compact]` → `SpecDocService.get_toc` (stored `(spec_id, version)` rows; miss → `BadParameter`). Default fields `section_no, title, level, source_file` (`[output.fields] spec_doc_toc`).
   - `doc3gpp spec doc search [--text TEXT] [--semantic TEXT] [filters]` →
     `SpecDocSearchFacade.search` with the same four mode-selection rules and

{doc3gpp-0.3.0 → doc3gpp-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: doc3gpp
-Version: 0.3.0
+Version: 0.3.2
 Summary: Provide 3GPP TDoc information with full text search capability — a Python CLI/library, a local web UI, and an MCP server for AI clients.
 Project-URL: Homepage, https://github.com/jerrywang121/doc3gpp
 Project-URL: Repository, https://github.com/jerrywang121/doc3gpp
@@ -377,6 +377,11 @@ doc3gpp spec doc index --rebuild
 doc3gpp spec doc schema --format json
 ```
 
+Background `parse_spec_docs` jobs retain per-spec failure reasons in the job
+result (`failure_details`) and log. A batch where every spec fails has status
+`failed`; a partial success retains status `succeeded` with the failed specs
+listed in the result.
+
 Spec-document rows live in a separate sibling database, normally
 `<main-stem>_specdata.db`, and are cached below the dedicated
 `~/.cache/doc3gpp/specs` root by default. The TDoc extraction cache remains

{doc3gpp-0.3.0 → doc3gpp-0.3.2}/README.md

@@ -294,6 +294,11 @@ doc3gpp spec doc index --rebuild
 doc3gpp spec doc schema --format json
 ```
 
+Background `parse_spec_docs` jobs retain per-spec failure reasons in the job
+result (`failure_details`) and log. A batch where every spec fails has status
+`failed`; a partial success retains status `succeeded` with the failed specs
+listed in the result.
+
 Spec-document rows live in a separate sibling database, normally
 `<main-stem>_specdata.db`, and are cached below the dedicated
 `~/.cache/doc3gpp/specs` root by default. The TDoc extraction cache remains

doc3gpp-0.3.2/docs/TODO.md

@@ -0,0 +1,42 @@
+# TODO
+
+Open improvement backlog from the repository review at commit `c0af0ee`.
+The review was read-only; the findings below have not been implemented.
+
+## Critical
+
+- [ ] **Add HTTP and MCP authentication and authorization.** References: `src/doc3gpp/web/app.py:154-185`, `:224-235`; `src/doc3gpp/web/routes/jobs.py:157-341`, `:454-583`; `src/doc3gpp/cli_server.py:207-265`. Protect corpus reads, job inspection, sync, parse, indexing, cache purge, and cancellation. Treat host/origin checks as transport controls, not identity. Refuse non-loopback binding unless an explicit authentication mechanism is configured. Add tests for unauthenticated and authorized read/mutating requests.
+- [ ] **Prevent stored XSS in cached TDoc content.** References: `src/doc3gpp/web/routes/tdocs.py:86`, `:439-455`; `src/doc3gpp/web/templates/tdoc_content.html:6`. Disable raw HTML in Markdown rendering or sanitize rendered HTML with a strict allowlist before using `safe`. Add a regression test with `<script>` and event-handler payloads.
+
+## Important
+
+- [ ] **Make queued job cancellation durable.** References: `src/doc3gpp/web/state.py:88-115`; `src/doc3gpp/web/routes/jobs.py:558-583`. Persist cancellation state or atomically transition queued jobs to `CANCELLED`, and ensure the worker claims only still-queued rows. Add restart and worker-unavailable tests.
+- [ ] **Make job ownership and orphan recovery safe for multiple processes.** References: `src/doc3gpp/web/workers/job_worker.py:195-223`; `src/doc3gpp/cli_server.py:214-255`; `src/doc3gpp/storage/repositories/jobs_sql.py:162-225`. Add worker ownership, leases, and heartbeats with stale-only recovery, or enforce a process singleton. Require `status = 'running'` in terminal status transitions so an old worker cannot overwrite newer state.
+- [ ] **Fix double compilation of hybrid FTS5 queries.** References: `src/doc3gpp/services/tdoc_search_facade.py:59-74`; `src/doc3gpp/services/semantic_search_service.py:172-183`; `src/doc3gpp/services/search_service.py:125-133`. Establish one explicit raw-query versus compiled-expression boundary. Add hybrid regression tests for TDoc IDs such as `R5-123456` and dotted spec IDs.
+- [ ] **Make the MCP TDoc content tool use the authoritative cache reader.** References: `src/doc3gpp/web/mcp_server.py:443-469`; `src/doc3gpp/web/routes/tdocs.py:416-437`; `src/doc3gpp/services/tdoc_cr_service.py:153-176`. Read current ZIP Markdown sidecars, legacy gzip entries, and plain UTF-8 through the shared helper, using the persisted `tdoc_extracts.cache_file` key. Add a test for a ZIP Markdown sidecar and direct URL parsing.
+- [ ] **Make document parse persistence atomic or recoverable.** References: `src/doc3gpp/services/spec_doc_service.py:277-306`; `src/doc3gpp/storage/repositories/spec_doc_sql.py:114-238`; `src/doc3gpp/services/tdoc_cr_service.py:675-710`. Use a per-document database transaction or unit of work, stage and atomically replace filesystem artifacts, and update search indexes only after durable relational and cache state is committed. Add failure-injection tests for partial TOC/chunk/cache writes and TDoc extraction sidecars.
+- [ ] **Enforce the documented settings source allowlist for dotenv files.** References: `src/doc3gpp/settings/schema.py:58-70`, `:73-97`, `:924-963`. Apply the same filtering to dotenv and file-secret sources, or explicitly document the broader precedence policy. Add tests proving that TOML-only server/MCP settings cannot be injected through `.env` if that restriction remains intended.
+- [ ] **Make vector schema initialization dimension-aware.** References: `src/doc3gpp/storage/db/migrate.py:298-350`, `:409-458`; `src/doc3gpp/services/factory.py:697-715`. Configure or discover embedding dimension before creating vector tables, record the dimension with the index, and support model/dimension changes during rebuild. Add clean-database and rebuild tests using a non-384-dimensional embedder.
+- [ ] **Remove stale auxiliary TDoc rows after confirmed upstream listings.** References: `src/doc3gpp/services/tdoc_file_service.py:29-68`; `src/doc3gpp/storage/repositories/tdoc_file_sql.py:150-163`; `src/doc3gpp/repository/protocols.py:590-598`. Replace rows for affected TDoc IDs in one transaction after a complete successful listing. Do not delete rows after partial or failed network results.
+- [ ] **Verify and resolve the `spec sync --tsg --force` contract.** References: `src/doc3gpp/cli.py:4639-4771`; `src/doc3gpp/services/spec_service.py:84-176`, `:299-307`. The CLI advertises `--force`, but the TSG sweep appears not to pass it to the per-spec skip logic. Confirm intended behavior, then either propagate `force` and test a recently synced spec or correct the help and documentation.
+
+## Minor
+
+- [ ] **Centralize cache-key validation and enforce root containment on Windows.** References: `src/doc3gpp/storage/cache.py:13-15`; `src/doc3gpp/models/tdoc_cr.py:266-285`; `src/doc3gpp/web/routes/tdocs.py:111-125`, `:416-431`. Reject both slash styles and unsafe traversal, validate persisted cache names, and verify every resolved path remains under the configured cache root.
+- [ ] **Close the shared remote embedder during application shutdown.** References: `src/doc3gpp/web/app.py:49-58`, `:217-222`; `src/doc3gpp/services/embedding/remote_embedder.py:96-97`. Store the embedder in `WebState` or register an explicit shutdown callback that calls `close()`.
+- [ ] **Avoid writing embedding bearer tokens into project TOML by default.** References: `src/doc3gpp/cli.py:5914-5977`; `src/doc3gpp/settings/config_writer.py:77-81`. Prefer environment or secret-file configuration, warn when writing sensitive settings, and use restrictive permissions for user configuration files.
+- [ ] **Make `config set` writes atomic.** Reference: `src/doc3gpp/settings/config_writer.py:77-81`. Use a temporary file, flush/fsync, and `os.replace`, matching the safer `config init` implementation at `src/doc3gpp/cli.py:5841-5853`.
+
+## Test And CI Follow-Up
+
+- [ ] **Cap xdist workers on Windows and investigate fixture/cache isolation.** `python -m pytest -n 4 -q` passed with `2563 passed, 1 skipped`. `python -m pytest -n auto` used 16 workers and was not reliable: separate runs produced repeated `no such table: meetings` failures in `tests/unit/test_tdoc_cli_fields.py` and a Windows `WinError 1920` multiprocessing failure in `tests/unit/test_cr_parser.py`. The affected CLI test passed serially and under `-n 2`, and the complete `test_tdoc_cli_fields.py` file passed under xdist. Use a platform-aware worker cap such as `-n 4` in scripts/CI, then audit global engine/settings caches and fixture isolation for the missing-table symptom.
+- [ ] **Install or configure Ruff in the development environment.** `ruff check .` could not be run during review because `ruff` was not installed or available on `PATH`.
+
+## Suggested Order
+
+1. Keep the server loopback-only until authentication, authorization, and XSS protections are in place.
+2. Make job cancellation and worker ownership durable before supporting multiple server processes.
+3. Fix the FTS query boundary and MCP cache reader, with regression tests.
+4. Make document persistence and cache replacement transactional or recoverable.
+5. Make vector schema initialization dimension-aware.
+6. Harden cache-key validation and atomic configuration writes.