documentation-engine 0.4.0__tar.gz → 0.6.0__tar.gz

{documentation_engine-0.4.0 → documentation_engine-0.6.0}/.github/workflows/ci.yml

@@ -47,7 +47,7 @@ jobs:
           version="$(uv run python -c 'import docsystem; print(docsystem.__version__)')"
           test -f "$dist_dir/documentation_engine-${version}-py3-none-any.whl"
           test -f "$dist_dir/documentation_engine-${version}.tar.gz"
-          uvx --from 'twine>=6,<7' twine check --strict "$dist_dir"/*
+          uvx --from 'twine>=7,<8' twine check --strict "$dist_dir"/*
 
       # Dogfooding: walk the documented adoption sequence end-to-end against
       # the public example profile, on a scratch copy so the fixture stays
@@ -83,7 +83,8 @@ jobs:
           $env:TMP = $env:RUNNER_TEMP
           $env:TEMP = $env:RUNNER_TEMP
           uv run pytest tests/test_cli.py tests/test_cli_platform.py `
-            tests/test_context_cli.py tests/test_mcp_adapter.py tests/test_readiness.py
+            tests/test_context_cli.py tests/test_mcp_adapter.py tests/test_readiness.py `
+            tests/test_provider_artifact.py
 
       - name: Installed CLI UTF-8 PowerShell smoke
         shell: pwsh

{documentation_engine-0.4.0 → documentation_engine-0.6.0}/AGENTS.md

@@ -76,10 +76,13 @@ Expected git mode for `scripts/installed_cli_smoke.sh` is `100755`.
   enabled worker profile, and write both the complete task contract and a
   `WORKER_TASK_INTENT` JSON file below `.orchestrator/prompts/`.
 - Set `intent.verification` before dispatch and pass the intent with
-  `worker run --intent-file`. The declared verification level is authoritative:
-  generic or copied task prose must not broaden it. If scope changes enough to
-  require another level, stop and dispatch corrected intent rather than
-  silently changing the gate.
+  `worker run --intent-file`. The declared verification level is the required
+  baseline: generic or copied task prose must not broaden it. A concrete risk
+  discovered during the task may raise the level when the broader check remains
+  within the task's permissions and authorizations; record the reason and the
+  actual level in handoff evidence. Otherwise stop with a verification
+  escalation request. A user-requested scope change still requires corrected
+  intent before continuing.
 - Dispatch once with a stable task id. Do not spend model turns polling worker
   state and do not dispatch a duplicate merely because a wait timed out.
 - For a bounded wait that fits the active Codex turn, prefer one direct

documentation_engine-0.6.0/ARCHITECTURE_REVIEW_2026-09-12.md

@@ -0,0 +1,260 @@
+# Documentation Engine architecture review
+
+Date: 2026-09-12
+Reviewed package version: 0.5.0
+Source baseline: `3f5f9cc42e1ad4da4b9d6b9fd7387e9f3dcc01f4`
+Disposition: retain the architecture; strengthen implementation boundaries and persistence contracts before expanding its responsibilities.
+
+## Executive assessment
+
+The architecture is well suited to the project's published purpose: maintaining structured Markdown knowledge that remains accessible to humans and AI clients as a project grows. Stable addresses, explicit graph semantics, inspectable context selection, disposable projections and provider-neutral interfaces form a coherent foundation. The implementation contains substantial evidence that these principles guide behavior, rather than existing only in documentation.
+
+The main weakness is the concentration of application behavior in the CLI. The project has useful domain modules, but the boundary between application services and command presentation is incomplete. Additional concerns are inconsistent write-recovery guarantees, an ambiguity between physical byte identity and normalized text identity, whole-catalog work for small reads, and coupling between otherwise optional capabilities through configuration loading.
+
+This is a strong local documentation engine with growing architectural debt. It is not yet evidence of a scalable concurrent knowledge service. That distinction does not make its current design unsuccessful: the documented trust model, completeness rules and single-writer assumptions deliberately favor correctness and transparent failure. A rewrite, mandatory database or mandatory server is not justified by this review.
+
+The highest-value direction is to preserve the source-of-truth and retrieval model while extracting reusable application services and making durability, identity and execution limits explicit.
+
+## Scope and method
+
+The assessment uses the published goals in [README](README.md), [Architecture](docs/architecture.md), [AI client integration](docs/client-integration.md), [MCP adapter](docs/mcp-adapter.md), and the related public contracts. Exploratory product ideas are not treated as approved requirements or implementation defects. Private planning documents and adopter corpora were not inspected for this review.
+
+The review covered source-module dependencies, representative end-to-end read/context and write paths, configuration loading, projection verification, federation, provider artifacts, representative regression tests, CI and packaging configuration. It is a single-reviewer architectural assessment, not a complete security audit or a line-by-line proof of every operation.
+
+Two isolated diagnostics exercised concrete questions about newline identity and optional configuration. They used a synthetic project in a temporary directory under `/tmp`, which was removed after execution. No project source, configuration, tests or private planning files were modified. This report is the only new repository deliverable. A pre-existing working-tree change to `AGENTS.md` was left untouched.
+
+### Structural observations
+
+An AST-based inventory of `src/docsystem/*.py` and `tests/test_*.py` found:
+
+| Observation | Result | Interpretation |
+| --- | ---: | --- |
+| Source modules | 32 | A compact package with several distinct capabilities |
+| Source lines, including comments and blanks | 25,020 | Scale indicator, not a quality score |
+| Lines in `cli.py` | 9,540, or 38.1% of source | Application and presentation responsibilities are concentrated |
+| Functions in `cli.py`, including nested definitions | 186 | The CLI is more than argument dispatch |
+| Largest CLI functions | Parser: 922 lines; maintenance: 570; context: 363; main: 350 | Parser size alone does not explain the concentration |
+| Test modules / test function definitions | 35 / 578 | Definitions are not collected test-case counts or coverage percentages |
+| Test modules importing `docsystem.cli` functions | 26 | Much behavior is verified through the CLI-facing seam |
+| Explicit intra-package import cycles found by AST inspection | 0 | The module graph has useful dependency discipline |
+
+Dynamic imports and runtime call dependencies are outside the import-cycle observation. Test quantity alone does not establish correctness.
+
+## Fit against the published goals
+
+| Goal | Assessment | Evidence and qualification |
+| --- | --- | --- |
+| Markdown remains the source of truth | Strong | Catalogs derive from Markdown; stale/corrupt projections visibly fall back. Physical byte identity needs clarification under newline normalization. |
+| Complete access with task-sized reads | Strong delivery design | Explicit sections, omissions, outline, known-revision and compact modes preserve inspectability. Small responses do not currently imply small server-side work. |
+| Stable identity and explainable relationships | Strong | Document IDs, canonical section anchors, revision pins and distinct authored/observed/generated edges avoid treating every hyperlink as authority. |
+| Provider neutrality | Strong | Base dependencies are limited to PyYAML; MCP is optional; the core does not require a model provider or orchestration runtime. |
+| Deterministic, verifiable derived state | Strong within stated assumptions | Generation manifests, configuration fingerprints, shard verification and direct/projected parity are explicit mechanisms. |
+| Safe adoption and preservation of authored files | Partially satisfied | Scalar-preserving migration and managed-write guards are valuable. Migration rollback does not establish crash-atomic multi-file persistence. |
+| Optional advanced capabilities | Partial isolation | Capabilities are separated conceptually and often by module, but one configuration loader validates them all and the CLI imports/composes them together. |
+| Sustainable growth in corpus and integrations | Promising, insufficiently demonstrated | Sharding and portable artifacts help. Common reads still materialize the catalog, and the published benchmark measures context bytes rather than latency or memory. |
+
+## Architecture worth preserving
+
+**One semantic implementation behind multiple clients.** The MCP adapter invokes the same CLI that ordinary consumers use, avoiding a second implementation of graph validation and projection fallback. The optional SDK and small base dependency set preserve easy local installation. See [package configuration](pyproject.toml) and [MCP invocation](src/docsystem/mcp_server.py#L30).
+
+**Explicit evidence and omissions.** The graph distinguishes authored metadata, observed links and generated containment. Compact context merges overlapping ranges while retaining the addresses and reasons that requested them. Reverse traversal requires sufficient graph validity to support a complete answer. These choices directly support trustworthy agent context. See [graph model](src/docsystem/graph.py#L1), [compact delivery](src/docsystem/cli.py#L2992), and [context regression tests](tests/test_context_cli.py#L336).
+
+**Verified projections with a usable fallback.** A generation is bound to configuration and derived records, and reads reject unverifiable state. Both serving paths reduce to a shared document view. The tests inspect parity, stale configurations, shard tampering and manifest tampering. See [projection loader](src/docsystem/projection.py#L667) and [parity regression](tests/test_vertical.py#L630). Those tests were inspected, not rerun here.
+
+**Useful domain separation already exists.** Metadata, sections, graph traversal, admission, execution-result validation and lifecycle evaluation have dedicated modules. The journal has no imports from other `docsystem` modules. The absence of explicit import cycles means extraction can be incremental; the project is not an inseparable dependency knot.
+
+**A deliberate external evidence boundary.** Provider artifacts are complete, body-free and independently verifiable. Consumers do not need to understand internal projection shards. Unavailable generations are distinguished from entity absence. See [provider artifacts](docs/provider-artifacts.md) and [artifact regressions](tests/test_provider_artifact.py#L105).
+
+**Verification addresses real operating boundaries.** CI includes Linux tests and lint, lock/build checks, installed CLI consumption, a public adoption walkthrough, and focused Windows UTF-8 tests. See [CI](.github/workflows/ci.yml) and [installed-consumer smoke](scripts/installed_cli_smoke.sh). This review did not inspect a live CI run or claim that these configured checks currently pass.
+
+## Findings, ordered by architectural importance
+
+Priority denotes recommended attention, not a vulnerability severity. High means resolve the boundary before substantial adjacent expansion; medium means address it during the next relevant capability change. Each recommendation remains a proposal.
+
+### AR-01 — Application behavior lives inside the CLI
+
+**Priority: high. Confidence: high. Type: maintainability and integration boundary.**
+
+The CLI owns internal document views, direct/projected adaptation, context selection and packet construction, execution-handoff assembly, maintenance policy checks and transaction coordination. These are application responsibilities as well as command behavior. For example, [`_build_execution_handoff`](src/docsystem/cli.py#L5674) binds admission, mandate snapshots and graph-derived read/review scope, while [`maintenance`](src/docsystem/cli.py#L7227) combines authorization checks and the write workflow. Neither responsibility is intrinsically tied to terminal output.
+
+The module's size is supporting evidence, not the finding by itself. [`_load_views`](src/docsystem/cli.py#L2630) and [`_compact_content_delivery`](src/docsystem/cli.py#L2992) demonstrate that central retrieval abstractions are private CLI implementation details. Many tests appropriately exercise this surface, but another in-process adapter cannot reuse the same behavior through a documented application-service boundary.
+
+**Consequence:** new transports or capabilities must invoke a subprocess, depend on private CLI functions, or duplicate composition logic. Changes to presentation, orchestration and invariants become harder to review independently.
+
+**Recommendation:** incrementally extract application services for context retrieval, evidence assembly and managed maintenance. Return typed results and diagnostics; leave argument validation specific to CLI syntax, exit-code mapping and rendering in the adapter. Keep the CLI/JSON contract stable during extraction. Internal typed services need not immediately become a public Python API.
+
+**Acceptance evidence for future work:** unchanged CLI and MCP contract fixtures, direct/projected output parity, service-level tests without captured stdout, and an import rule preventing application/domain modules from depending on command presentation.
+
+### AR-02 — Migration and managed maintenance have different recovery models
+
+**Priority: high. Confidence: high for the failure boundary. Type: authored-data preservation.**
+
+[`apply_migration_plan`](src/docsystem/migration.py#L255) checks original contents, prepares sibling temporary files, and replaces destinations one at a time. Its rollback keeps original bytes in process memory and runs only when a replacement raises `OSError`. The [rollback test](tests/test_migration.py#L147) exercises a recoverable exception while that process remains alive.
+
+If the process terminates after replacing the first file and before replacing the next, the exception handler cannot run. The result can contain both migrated and unmigrated files, and this path provides no durable migration record from which to reconstruct the transaction. Rollback itself can also fail while writing original bytes. No crash experiment was performed; the boundary follows from the control flow and volatile storage of originals.
+
+The [architecture guide](docs/architecture.md) describes migration with an unqualified claim that a failure never leaves a partially migrated multi-file change. That exceeds what the implementation establishes. Managed maintenance separately uses an [exclusive journal operation](src/docsystem/journal.py#L706) and [persisted before/after data](src/docsystem/journal.py#L777). That is a stronger recovery foundation, although it should not be assumed to prove every power-loss guarantee either.
+
+**Consequence:** adopters cannot infer a uniform preservation/recovery contract across source-writing commands. This matters directly to the promise to preserve existing documentation during migration.
+
+**Recommendation:** define the distinction between exception rollback, process-crash recovery, concurrent-reader visibility and power-loss durability. Either explicitly constrain migration to its actual failure model or design a durable migration transaction using shared recovery primitives. Preserve scalar formatting and idempotence. Avoid claiming atomic visibility across multiple ordinary file replacements.
+
+**Acceptance evidence for future work:** interruption after each durable transaction phase, partial replacement, rollback failure, concurrent source modification and recovery replay; each should have an explicit supported outcome. No fault-injection or source-write fix is included in this review.
+
+### AR-03 — Source hashes identify normalized text rather than physical source bytes
+
+**Priority: medium. Confidence: high; reproduced. Type: identity and traceability contract.**
+
+The catalog reads Markdown with [`Path.read_text`](src/docsystem/catalog.py#L226), which normalizes newlines. The projection hashes that string through [`_sha`](src/docsystem/projection.py#L73), and the [verification loader](src/docsystem/projection.py#L667) repeats the same text read. Section hashes additionally join split lines with LF.
+
+An isolated diagnostic built a projection from a valid CRLF document, rewrote only its newline encoding to LF, and verified the old projection. The raw file SHA-256 changed, but the source hash and generation remained unchanged, and the loader returned `projection current`.
+
+This does not show lost prose, a broken dependency graph or arbitrary tampering acceptance. CRLF and LF represent the same normalized text. The issue is that the documented byte-for-byte freshness language is stronger than the implemented identity semantics, and callers may need physical byte identity for evidence or interoperability.
+
+**Recommendation:** explicitly distinguish raw-source byte hashes, normalized document-text hashes and delivered-fragment hashes. Decide which identity each contract needs. If physical source identity is intended, hash bytes before decoding. If normalized identity is intended, name and version that contract rather than presenting it as exact file-byte identity. A change to generation semantics requires compatibility and projection-version consideration.
+
+**Acceptance evidence for future work:** CRLF/LF, terminal newline, Unicode and direct/projected fixtures; verify both identity semantics and unchanged human-readable content.
+
+### AR-04 — Task-sized output still performs whole-catalog work
+
+**Priority: medium. Confidence: high for the work performed; no latency claim. Type: scalability.**
+
+[`read_document`](src/docsystem/cli.py#L2246) calls `_load_views` before selecting one document or section. On the direct path, [`build_catalog`](src/docsystem/catalog.py#L226) reads and parses every included Markdown file. On the projected path, [`load_verified_projection`](src/docsystem/projection.py#L667) reads all included source texts, checks their hashes, loads all document shards and the reverse shards for dependency targets, then builds all document views.
+
+For total source volume B, a small read therefore still processes source volume proportional to B and retains whole-catalog content. A sequence of separate reads repeats that work. The specialized projected reference path has narrower shard loading; the finding should not be generalized to every command.
+
+The integrity rationale is valid: the current contract detects source drift without relying on filesystem timestamps. The [context-efficiency measurement](docs/context-efficiency.md) correctly measures returned bytes over three predefined tasks, not runtime, memory or engineering productivity.
+
+**Consequence:** efficient model context does not establish efficient service execution. Latency and memory may become limiting on large corpora, slower filesystems or many simultaneous consumers. Their actual thresholds are unmeasured here.
+
+**Recommendation:** establish reproducible public synthetic performance fixtures first. Then consider query-scoped document materialization, reuse of a verified snapshot across related requests, and a documented freshness protocol. Preserve complete-graph checks where the answer requires them. A timestamp shortcut alone would weaken current guarantees and is not recommended as an automatic fix.
+
+**Acceptance evidence for future work:** cold and repeated reads, single-section versus broad context, varying corpus sizes and graph density, source/configuration drift, peak memory, and bounded concurrency. Report response bytes separately from execution cost.
+
+### AR-05 — Optional capabilities share one configuration failure boundary
+
+**Priority: medium. Confidence: high; reproduced. Type: capability isolation.**
+
+[`load_config`](src/docsystem/config.py#L1315) eagerly validates maintenance, context views, profiles, traceability, workstreams, intake and admission, even for an ordinary document read. [`_admission_criteria`](src/docsystem/config.py#L1034) rejects unknown keys. A diagnostic confirmed that adding an invalid key to the otherwise unused admission table prevents configuration loading for a simple synthetic documentation project.
+
+Rejecting malformed policy is sensible. The architectural concern is that operational optionality does not imply independent failure or evolution boundaries: an error in an advanced capability can disable basic retrieval. This is a tradeoff, not evidence that policy validation should silently be skipped.
+
+**Recommendation:** separate core configuration types from capability-specific types and make validation dependencies explicit. Retain a full-profile validation command. Decide deliberately whether an ordinary read requires the entire profile to be valid or only its declared dependencies; never ignore malformed safety-relevant configuration implicitly. Document the choice before changing behavior.
+
+**Acceptance evidence for future work:** focused configuration tests proving the chosen behavior for unrelated invalid capability sections, required retrieval policy, mutations and older consumers. Configuration changes must include tests under the repository rules.
+
+### AR-06 — The MCP subprocess boundary has no explicit execution limits
+
+**Priority: medium. Confidence: high for the implementation. Type: adapter operability.**
+
+[`_invoke`](src/docsystem/mcp_server.py#L30) uses synchronous `subprocess.run(..., capture_output=True)` without a timeout. It buffers the result and does not expose a child-process cancellation or deadline contract. Each request starts a new CLI process and repeats its loading work.
+
+This boundary is valuable for semantic consistency and simple installation. The concern appears when reads stall or take a long time, or when results become large: the adapter itself has no declared way to bound that operation or propagate cancellation. The surrounding host may impose its own limits; this review did not test host cancellation behavior.
+
+**Recommendation:** define transport execution policy, structured timeout/cancellation outcomes and bounded resource handling. Consider a reusable executor around the existing CLI before introducing another semantic implementation. If a requested operation exceeds a supported delivery bound, return an explicit failure or a versioned paging mechanism; do not silently truncate source context.
+
+**Acceptance evidence for future work:** stalled child process, cancellation, oversized output, nonzero exit with valid failure payload, Unicode diagnostics and repeated invocation. Preserve the current distinction between legitimate domain-negative results and transport failure.
+
+## Deliberate boundaries that should not be misclassified as bugs
+
+1. **Federation requires a complete, available source set.** [`build_federated_catalog`](src/docsystem/federation.py#L196) checks every registered source and rejects incomplete results. This supports complete cross-source graph answers. Partial or authorization-scoped queries would need a different explicit contract, not filtering after a full query.
+2. **Visibility metadata is not authentication.** The [workspace contract](docs/workspace-sources.md) assigns untrusted-caller enforcement to filesystem or host boundaries. The existing local MCP interface should not be exposed as a multi-user service merely because its commands are read-only.
+3. **Retained provider generations are not an indefinite archive.** [Provider retention](docs/provider-snapshots.md#pagination-and-deterministic-bytes) governs comparison availability; portable comparison of previously exported snapshot artifacts is explicitly deferred. Consumers needing long reconciliation windows must account for that limitation.
+4. **Projection concurrency is limited by design.** The [architecture guide](docs/architecture.md#scalable-projection) states a single-writer assumption and explains retention races. It would be inappropriate to claim concurrent service readiness from atomic replacement of the current pointer alone.
+5. **Structural evidence does not prove semantic correctness.** Revision pins, graph completeness, admission and lifecycle records can validate authored claims and lineage. They cannot determine that an architectural recommendation is true or that an agent understood the delivered text. Human/project acceptance remains meaningful.
+
+## Recommended direction
+
+Keep the current package usable as a local tool. Strengthen its internal boundaries in small steps, preserving public CLI/JSON behavior.
+
+```mermaid
+flowchart TD
+    Clients[Human and AI clients] --> MCP[MCP adapter]
+    Clients --> CLI[CLI adapter]
+    MCP -->|Current subprocess contract| CLI
+    CLI --> Services[Explicit application services]
+    Services --> Retrieval[Catalog, sections and graph]
+    Services --> Evidence[Optional evidence and policy capabilities]
+    Services --> Writes[Explicit source-write transactions]
+    Retrieval --> Markdown[Authored Markdown]
+    Retrieval --> Projection[Verified disposable projection]
+    Writes --> Markdown
+    Writes --> Journal[Recovery journal]
+```
+
+This is a proposed internal decomposition, not a diagram of already completed extraction. It does not require a new service process, a database, a public Python SDK or replacement of the existing MCP transport.
+
+| Order | Proposed work | Why this order |
+| --- | --- | --- |
+| 1 | Decide and document migration recovery and hash-identity guarantees | These affect preservation and the meaning of evidence; avoid building new assumptions on ambiguous contracts. |
+| 2 | Extract context application services with unchanged output fixtures | Moves the most central reusable behavior out of presentation without a broad rewrite. |
+| 3 | Extract governed evidence and maintenance coordination | Gives optional capabilities explicit dependencies and consistent transaction entry points. |
+| 4 | Define configuration validation boundaries and enforce module dependency rules | Reduces future coupling while keeping policy failures explicit. |
+| 5 | Add reproducible performance evidence and adapter execution limits | Makes optimizations and deployment decisions evidence-based. |
+
+Do not turn every optional capability into a separate package immediately. The current base dependency footprint is already small. Package separation should follow a real dependency, compatibility or runtime boundary. Preserve independent project ownership and explicit acceptance if future knowledge applications consume this engine.
+
+## Verification record and limits
+
+| Activity | Status | What it establishes |
+| --- | --- | --- |
+| Public-contract and representative source review | Completed | Evidence for the assessments above; not exhaustive behavioral validation |
+| AST module, size and import inventory | Completed | Reported structural counts and absence of explicit import cycles in the inspected source |
+| Synthetic newline-identity diagnostic | Completed; confirmed AR-03 | Raw bytes change while the existing normalized-text projection remains accepted |
+| Synthetic unused-admission configuration diagnostic | Completed; confirmed AR-05 | One optional-section validation failure blocks shared configuration loading |
+| Existing tests and CI definitions | Inspected only | Relevant regression intent exists; no assertion that those checks passed in this review |
+| Markdown links and cited line bounds in this report | Passed | Referenced repository files and line positions exist |
+| `git diff --check` | Passed | No whitespace errors in the tracked working-tree diff |
+| New-report whitespace and structure check | Passed | Covers the report while it is still untracked |
+| Full pytest and Ruff gates | **Not run** | No implementation, configuration, packaging or test changes; the report does not claim suite success |
+| Lock/build/distribution/installed-consumer checks | **Not run** | Packaging inputs were not changed |
+| Performance benchmarks, crash/power-loss injection and host cancellation tests | **Not run** | Performance thresholds and those failure behaviors remain explicitly unverified |
+
+The verification scope was structural review plus two focused isolated diagnostics prompted by concrete contract questions. The diagnostics did not require modifying repository files or running a test suite. Proposed acceptance checks above describe future work, not checks completed for this report.
+
+### Diagnostic reproduction outline
+
+Run from an installed contributor environment. All generated source and projection state below belongs to a disposable synthetic project, not to an adopter's documentation.
+
+```python
+from pathlib import Path
+from tempfile import TemporaryDirectory
+import hashlib
+
+from docsystem.catalog import build_catalog, validate_catalog
+from docsystem.config import DEFAULT_CONFIG, load_config
+from docsystem.projection import (
+    build_projection,
+    load_verified_projection,
+    write_projection,
+)
+
+with TemporaryDirectory(dir="/tmp") as directory:
+    root = Path(directory)
+    config_path = root / ".docsystem.toml"
+    config_path.write_text(DEFAULT_CONFIG, encoding="utf-8")
+    document = root / "plan/architecture/README.md"
+    document.parent.mkdir(parents=True)
+    lf = b"---\nid: DOC-001\nrevision: 1\n---\n# Example\n\n## Purpose\nText.\n"
+    crlf = lf.replace(b"\n", b"\r\n")
+    document.write_bytes(crlf)
+    config = load_config(root)
+    catalog = build_catalog(config)
+    assert not [x for x in validate_catalog(catalog, config) if x.severity != "warning"]
+    before = build_projection(catalog, config)
+    write_projection(config, before)
+    document.write_bytes(lf)
+    after = build_projection(build_catalog(config), config)
+    loaded, reason = load_verified_projection(config)
+    print(hashlib.sha256(crlf).digest() != hashlib.sha256(lf).digest())  # True
+    print(before["generation"] == after["generation"])  # True
+    print(loaded is not None, reason)  # True, projection current
+    config_path.write_text(
+        DEFAULT_CONFIG.replace("[admission]\n", "[admission]\nunknown_option = true\n"),
+        encoding="utf-8",
+    )
+    try:
+        load_config(root)
+    except ValueError as error:
+        print(error)  # admission has unknown key(s): unknown_option
+```

documentation_engine-0.6.0/ARCHITECTURE_REVIEW_2026-09-12_SECOND_PASS.md

@@ -0,0 +1,222 @@
+# Documentation Engine architecture review — second pass
+
+Date: 2026-09-12
+Package version: 0.5.0
+Reviewed source baseline: `3f5f9cc42e1ad4da4b9d6b9fd7387e9f3dcc01f4`
+Related assessment: [first architectural review](ARCHITECTURE_REVIEW_2026-09-12.md)
+
+## Assessment
+
+The second pass supports retaining the existing architecture, but changes the immediate emphasis: **consistency between capabilities should take priority over broad code reorganization**. The Markdown source model, explicit graph relationships, verifiable projections and provider-neutral adapters remain appropriate for the published goals. Their most important weakness is that different consumers reconstruct parts of the same semantic or validation model independently.
+
+This review confirmed one profile-validation defect, one ambiguity in command-level validation guarantees, one inconsistent transport-decoding boundary, and repeated full-inventory work during complete provider export. These are more specific than the first report's concerns about module size. They show where architectural boundaries already affect observable behavior.
+
+The findings do not justify a rewrite, a database in the document core, or a mandatory background server. They support shared semantic accessors, explicitly named validation scopes, consistent artifact decoding and preparation of immutable export queries once per operation.
+
+## Scope and independence
+
+This is a second pass by the same reviewer, with new hypotheses and fresh synthetic diagnostics. It is not an independent-agent endorsement of the first report. The source commit is unchanged. Existing changes to `AGENTS.md` and the first report were preserved.
+
+The review concentrated on composition: local versus source-qualified relations, readiness versus the normal validation gate, artifact serialization versus canonical verification, and paged versus complete exports. Relevant implementation and public contracts were inspected. Private planning documents, adopter data and external services were not used.
+
+No implementation, configuration or test files were changed. All runtime diagnostics used disposable synthetic projects under `/tmp`. This report is the only additional repository deliverable. No full test suite or performance benchmark was run.
+
+Priority below describes recommended engineering attention, not security severity. A confirmed behavior can be a deliberate contract choice rather than a defect; those cases are identified explicitly.
+
+## SP-01 — Source-qualified relations disappear from document-profile validation
+
+**Priority: high. Confidence: high. Classification: confirmed behavioral defect.**
+
+The metadata parser stores source-qualified references separately in `DocumentMetadata.federated_references`. However, the profile helpers [`_metadata_fields`](src/docsystem/profiles.py#L59) and [`_document_relations`](src/docsystem/profiles.py#L73) inspect only ordinary references and legacy references. Consequently, profile constraints depend on how a relation target is addressed, even when the authored relation is the same.
+
+The [profile contract](docs/document-profiles.md) states that an explicit empty `allowed_relations` list allows no values and that a required semantic field must contain a valid relation value. It does not exempt qualified references from those rules.
+
+### Observed behavior
+
+A synthetic document of type `spec` was checked against a profile with `allowed_relations = []`:
+
+| Authored metadata | Result of `profile-check --json` |
+| --- | --- |
+| `depends_on: [DOC-001]` | Exit 1; `relation-not-allowed` |
+| `depends_on: ["peer::DOC-001"]` | Exit 0; `valid: true`; no violations |
+| Qualified value above, with `required_metadata = ["depends_on"]` also configured | Exit 1; `missing-metadata` for the authored `depends_on` field |
+
+The diagnostic did not connect to a real second project. It established that a syntactically accepted source-qualified relation is absent from the profile's relation inventory. Resolution or availability of its remote target is a separate question; the relation should not silently disappear from local policy evaluation.
+
+### Architectural cause and impact
+
+[`_references`](src/docsystem/metadata.py#L166) has a richer relation model than profile evaluation consumes. A capability added to the common metadata representation has not been incorporated into every downstream semantic helper.
+
+This is directly relevant to the goals of project-authored policy and multi-source documentation. A project can prohibit a relation but receive a successful profile check after only changing the target's address form. Conversely, a required field can be reported absent when it is authored.
+
+**Recommendation:** expose one authoritative relation inventory that preserves relation name, target qualification, resolution state and pin information. Profile presence and allowlist checks should consume that inventory without requiring disclosure of the target body. Keep unresolved-target diagnostics separate from field absence and relation prohibition.
+
+**Future acceptance evidence:** a matrix covering local, qualified, pinned, legacy, empty and malformed relation values against required-field and allowlist rules, including CLI/MCP and ordinary validation entry points. The existing [profile tests](tests/test_profiles.py) exercise local relations and workspace selection, but those are not equivalent to qualified relation values inside a document.
+
+## SP-02 — Readiness and the normal quality gate validate different scopes without naming the distinction in the readiness result
+
+**Priority: medium. Confidence: high. Classification: reproduced contract ambiguity, not proof that all reads or index writes should be blocked.**
+
+[`validate`](src/docsystem/cli.py#L1514) combines catalog validation, graph diagnostics, document profiles, delivery policy and program-plan checks. In contrast, [`evaluate_readiness`](src/docsystem/readiness.py#L68) evaluates a narrower set of catalog/metadata/navigation conditions. [`index_projection`](src/docsystem/cli.py#L4314) also uses the narrower catalog validator.
+
+### Observed behavior
+
+For a structurally sound synthetic catalog containing a local `depends_on` relation forbidden by its configured profile:
+
+| Command | Observed outcome |
+| --- | --- |
+| `readiness --json` | Exit 0, `ready: true` |
+| `validate` | Exit 1, `profile spec: relation-not-allowed` |
+| `index --write` | Exit 0, projection created |
+
+The source did not change between those calls. The same authored profile violation therefore coexists with successful readiness and a current projection.
+
+### Interpretation
+
+Adoption readiness, catalog indexability and policy compliance can legitimately differ. Indexing useful documents despite a policy violation may be desirable. The problem is that the machine-facing `ready` flag and suggested next action do not disclose which configured validation dimensions were omitted. A downstream agent may treat readiness or projection freshness as a broader quality endorsement than it is.
+
+The same distinction applies to cycle diagnostics. The [agent contract](docs/agent-contract.md) explicitly makes those corpus-wide checks available through `doctor`/`validate` rather than blocking every individual reference query. That is a deliberate boundary, not a reason to reject all cyclic-document reads.
+
+**Recommendation:** define named validation scopes and expose their coverage. For example, a readiness result could distinguish structural readiness from profile/graph/delivery checks that passed, failed or were not evaluated. Decide deliberately whether a recommendation to index should carry outstanding policy diagnostics. Do not silently widen every query to the full quality gate.
+
+**Future acceptance evidence:** valid structure with invalid profile, invalid program plan, graph-cycle diagnostics and absent optional policy; verify both command outcomes and explicit coverage of the checks that justify each outcome.
+
+## SP-03 — Artifact verification accepts duplicate JSON members before checking the digest
+
+**Priority: medium. Confidence: high. Classification: reproduced interoperability and decoding-policy gap.**
+
+[`load_and_verify_artifact`](src/docsystem/provider_artifact.py#L745) calls ordinary `json.loads` and then verifies the resulting object. Repeated JSON object members are resolved by the parser before [`verify_artifact`](src/docsystem/provider_artifact.py#L703) evaluates the schema and canonical content digest.
+
+### Observed behavior
+
+A valid snapshot artifact was exported. A second `schema_version` member was inserted before the genuine member:
+
+```json
+{
+  "schema_version": 999,
+  "schema_version": 1,
+  "artifact_kind": "provider-snapshot-artifact"
+}
+```
+
+The snippet illustrates the duplicate members; the actual diagnostic retained all other required fields from the complete artifact. The existing digest was not changed. `load_and_verify_artifact` still returned `valid: true`, because the parsed object retained the last `schema_version` value and therefore had the same canonical digest as before.
+
+This is not a SHA-256 collision or signature bypass. The [artifact contract](docs/provider-artifacts.md) correctly describes a digest of canonical content and states that authentication and provenance are separate responsibilities. Whitespace differences also legitimately do not change that digest. The narrower concern is acceptance of a transport representation whose duplicate members can be interpreted differently or rejected by another consumer.
+
+There is already a stricter precedent in the package: [`load_shared_finish_record`](src/docsystem/shared_finish.py#L59) uses [`_reject_duplicate_keys`](src/docsystem/shared_finish.py#L44). YAML metadata also rejects repeated keys. Several other JSON loaders, including [execution packets](src/docsystem/execution.py#L65), use ordinary decoding; this review did not reproduce every loader.
+
+**Recommendation:** define a shared unambiguous JSON input policy for integrity-bearing artifacts and requests. Reject duplicate members recursively before canonicalization. Keep this separate from the documented allowance for unknown optional fields in future-compatible artifacts: unique optional fields and duplicate keys are different compatibility questions.
+
+**Future acceptance evidence:** root and nested duplicates, duplicate digest or scope members, malformed input, and unique unknown optional fields. Confirm that strict decoding preserves the intended additive compatibility policy and deterministic error classes.
+
+## SP-04 — Complete export rebuilds the complete entity inventory for every page
+
+**Priority: medium. Confidence: high for repeated work; no measured latency threshold. Classification: confirmed scalability inefficiency.**
+
+[`snapshot_artifact`](src/docsystem/provider_artifact.py#L150) assembles pages through [`_collect_pages`](src/docsystem/provider_artifact.py#L95). Each callback invokes [`snapshot_response`](src/docsystem/provider.py#L384), which rebuilds `observations(snapshot)` for the whole immutable generation before slicing a page.
+
+The analogous [comparison response](src/docsystem/provider.py#L413) reconstructs both inventories and their comparison each time [complete comparison export](src/docsystem/provider_artifact.py#L200) asks for another page.
+
+### Observed behavior
+
+A synthetic snapshot with 1,104 observations required three pages at the maximum page size of 500. Instrumentation around the existing `observations` function recorded three full inventory constructions during one `snapshot_artifact` call.
+
+| Measurement | Result |
+| --- | ---: |
+| Complete artifact observations | 1,104 |
+| Maximum requested page size | 500 |
+| Full inventory constructions during one export | 3 |
+
+For N entities across P pages, inventory work is repeated P times. With a fixed page-size ceiling, that component can approach quadratic growth in N. This is a work-count observation, not a runtime benchmark or a claim that memory consumption is quadratic. Generation loading itself was performed once in this diagnostic.
+
+The current [artifact regression](tests/test_provider_artifact.py#L105) covers complete multi-page output and deterministic verification. That is valuable correctness evidence, but does not constrain how often the same immutable query is prepared.
+
+**Recommendation:** prepare and validate the immutable inventory or comparison once per complete export, then expose a page view over that prepared query. Reuse public contract semantics without repeatedly rebuilding the entire result merely to keep an internal page-shaped call boundary. Preserve ordering, cursor binding, byte limits and complete-or-fail assembly.
+
+**Future acceptance evidence:** operation counts over increasing entity totals, complete/paged result equality, byte-limited pages, empty results and multi-page comparisons. Performance benchmarks can then determine whether additional streaming or memory changes are necessary.
+
+## Reassessment of the first report
+
+| First-pass finding | Second-pass disposition |
+| --- | --- |
+| AR-01: application logic in the CLI | Supported, with a sharper remedy. Extract ownership of shared semantics and validation before performing a mechanical file split. SP-01 and SP-02 illustrate why. |
+| AR-02: migration recovery model | Retained as a static failure-model concern. No new crash or rollback-failure experiment was run; the second pass does not upgrade it to a reproduced crash incident. |
+| AR-03: normalized-text hashes | Retained with the existing qualification: physical file identity and normalized text identity differ. This is not evidence of lost prose or an insecure hash algorithm. |
+| AR-04: whole-catalog work for small reads | Supported. SP-04 identifies a separate repeated-work issue in complete provider export. Neither review establishes actual latency limits. |
+| AR-05: shared configuration failure boundary | Treat as a design tradeoff, not an instruction to ignore invalid optional policy. Address only through an explicit validation-dependency contract. |
+| AR-06: MCP execution limits | Supported as an adapter-operability concern. The absence of a timeout does not establish observed host failure and does not justify making a daemon mandatory. |
+
+The first review's positive observations remain valid: a small base dependency set, no explicit intra-package import cycles in the inspected AST graph, useful domain modules, explicit context omissions and strong direct/projected parity intent in tests. These are foundations to preserve, not reasons to overlook cross-capability inconsistencies.
+
+## Recommended order of attention
+
+1. Correct qualified-relation handling in profile checks and add semantic-form parity coverage.
+2. Define a consistent strict decoder for integrity-bearing JSON; retain additive field compatibility where the contract allows it.
+3. Name and expose the validation scope behind readiness, quality gates and projection status.
+4. Prepare complete-export inventories once; measure work counts before broad optimization.
+5. Use these concrete boundaries to guide the application-service extraction proposed in the first review.
+
+Migration durability and hash-identity decisions from the first report remain parallel prerequisites for expanding write automation or external traceability guarantees. No implementation or repository reorganization is authorized by this report.
+
+## Verification record
+
+| Activity | Outcome |
+| --- | --- |
+| Source baseline and existing working-tree inspection | Completed; source commit unchanged |
+| Local versus qualified relation diagnostic | Completed; SP-01 reproduced |
+| Qualified required-field diagnostic | Completed; SP-01 reproduced |
+| Readiness / validate / index comparison | Completed; SP-02 reproduced on unchanged synthetic source |
+| Duplicate-member artifact diagnostic | Completed; SP-03 reproduced with the original canonical digest |
+| Complete-export inventory-call instrumentation | Completed; SP-04 reproduced, 3 full inventories for 1,104 observations |
+| Relevant existing tests and public contracts | Inspected; not claimed to have passed |
+| Report links, cited line bounds and Markdown structure | Passed |
+| `git diff --check` and new-report whitespace validation | Passed |
+| Preservation hashes of `AGENTS.md` and the first report | Unchanged |
+| Full pytest / Ruff gate | **Not run** — no implementation or configuration changes |
+| Build, lock and installed-consumer checks | **Not run** — packaging inputs unchanged |
+| Timing, memory, cancellation and crash experiments | **Not run** |
+
+The diagnostic examples contain only synthetic content. Their temporary files and projections were removed after execution. Model-worker orchestration was not dispatched because the existing project binding still targeted a different host task; its binding and delivery state were left unchanged. This is a scope limitation of reviewer independence, not a defect in Documentation Engine.
+
+## Reproduction recipes
+
+These recipes describe the isolated diagnostics, rather than proposing changes to repository fixtures. Use an existing contributor environment and disposable directories under `/tmp`.
+
+### Profile and validation scope
+
+Create a minimal valid catalog with `DOC-001` as its index linking to `DOC-002`. Map a root area to `.`. Give `DOC-002` the type `spec` and configure:
+
+```toml
+[profiles.spec]
+document_types = ["spec"]
+history_mode = "living"
+allowed_relations = []
+```
+
+Run `profile-check --json` first with `depends_on: [DOC-001]`, then with `depends_on: ["peer::DOC-001"]`. Add `required_metadata = ["depends_on"]` to the same profile and repeat the qualified case. Finally, restore the local relation and the profile without the required-field rule, then compare `readiness --json`, `validate` and `index --write` on the same temporary project. Expected observed results are recorded in SP-01 and SP-02.
+
+### Complete export and duplicate members
+
+Create a provider-enabled synthetic catalog containing an index and a target with 1,100 H2 sections. Build and retain one generation, then load it once with `load_pinned_projection`. Instrumentation used the existing implementation without changing a source file:
+
+```python
+from unittest.mock import patch
+import docsystem.provider as provider
+from docsystem.provider_artifact import (
+    encode_artifact,
+    load_and_verify_artifact,
+    snapshot_artifact,
+)
+
+# snapshot is the pinned synthetic generation; temporary_path is under /tmp.
+with patch.object(provider, "observations", wraps=provider.observations) as counter:
+    artifact = snapshot_artifact(snapshot)
+    print(len(artifact["observations"]), counter.call_count)  # 1104, 3
+
+encoded = encode_artifact(artifact)
+ambiguous = encoded.replace("{", '{\n  "schema_version": 999,', 1)
+temporary_path.write_text(ambiguous, encoding="utf-8")
+print(load_and_verify_artifact(temporary_path)["valid"])  # True
+```
+
+The code is a contributor-side diagnostic of implementation behavior, not a recommendation for external consumers to import internal package modules. External integrations should continue to use the documented CLI/artifact contracts.