tracecite 0.0.2__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (76) hide show
  1. tracecite/__init__.py +35 -0
  2. tracecite/__main__.py +3 -0
  3. tracecite/cli.py +656 -0
  4. tracecite/docs/__init__.py +56 -0
  5. tracecite/docs/build.py +381 -0
  6. tracecite/docs/config.py +114 -0
  7. tracecite/docs/contract.py +17 -0
  8. tracecite/docs/modes.py +240 -0
  9. tracecite/docs/stage.py +407 -0
  10. tracecite/docs/vectorize.py +457 -0
  11. tracecite/evidence/__init__.py +3 -0
  12. tracecite/evidence/chunking.py +214 -0
  13. tracecite/evidence/commands.py +552 -0
  14. tracecite/evidence/config.py +49 -0
  15. tracecite/evidence/embedding.py +59 -0
  16. tracecite/evidence/manifest.py +197 -0
  17. tracecite/evidence/page_extraction.py +200 -0
  18. tracecite/evidence/page_output.py +80 -0
  19. tracecite/evidence/page_selection.py +116 -0
  20. tracecite/evidence/parsers/__init__.py +1 -0
  21. tracecite/evidence/parsers/base.py +65 -0
  22. tracecite/evidence/parsers/markdown.py +142 -0
  23. tracecite/evidence/parsers/pdf.py +315 -0
  24. tracecite/evidence/parsers/workbook.py +334 -0
  25. tracecite/evidence/paths.py +94 -0
  26. tracecite/evidence/schema.py +570 -0
  27. tracecite/evidence/source_links.py +185 -0
  28. tracecite/evidence/sync.py +1434 -0
  29. tracecite/evidence/vector_backend.py +130 -0
  30. tracecite/evidence/verify.py +368 -0
  31. tracecite/skills/routing-documentation-source-links/SKILL.md +167 -0
  32. tracecite/skills/routing-documentation-source-links/agents/openai.yaml +3 -0
  33. tracecite/skills/routing-documentation-source-links/references/documenter-integration.md +71 -0
  34. tracecite/skills/routing-documentation-source-links/references/registry-contract.md +86 -0
  35. tracecite/skills/routing-documentation-source-links/references/staging-and-rewriting.md +118 -0
  36. tracecite/skills/routing-documentation-source-links/references/validation-and-failure-modes.md +67 -0
  37. tracecite/skills/using-tracecite/SKILL.md +97 -0
  38. tracecite/skills/using-tracecite/agents/openai.yaml +3 -0
  39. tracecite/skills/using-tracecite/references/architecture.md +68 -0
  40. tracecite/skills/using-tracecite/references/pdf-and-assets.md +67 -0
  41. tracecite/skills/using-tracecite/references/retrieval.md +45 -0
  42. tracecite/skills/using-tracecite/references/schema-and-migrations.md +52 -0
  43. tracecite/skills/using-tracecite/references/synchronisation.md +98 -0
  44. tracecite/skills/using-tracecite/references/workspace-runtime.md +44 -0
  45. tracecite/skills/writing-evidence-backed-reports/SKILL.md +91 -0
  46. tracecite/skills/writing-evidence-backed-reports/agents/openai.yaml +3 -0
  47. tracecite/skills/writing-evidence-backed-reports/fixtures/README.md +13 -0
  48. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/invalid-bad-path.md +5 -0
  49. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/invalid-grammar.md +8 -0
  50. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/invalid-missing-definition.md +3 -0
  51. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/invalid-page-not-indexed.md +5 -0
  52. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/invalid-unindexed-path.md +5 -0
  53. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/valid-grammar.md +12 -0
  54. tracecite/skills/writing-evidence-backed-reports/fixtures/reports/valid-report.md +27 -0
  55. tracecite/skills/writing-evidence-backed-reports/fixtures/source-links.toml +17 -0
  56. tracecite/skills/writing-evidence-backed-reports/fixtures/validate_fixtures.py +295 -0
  57. tracecite/skills/writing-evidence-backed-reports/references/images-and-source-links.md +34 -0
  58. tracecite/skills/writing-evidence-backed-reports/references/quotation-and-verification.md +43 -0
  59. tracecite/skills/writing-evidence-backed-reports/references/reference-style-citations.md +48 -0
  60. tracecite/skills/writing-evidence-backed-reports/references/report-structure.md +58 -0
  61. tracecite/tables/__init__.py +41 -0
  62. tracecite/tables/_text.py +142 -0
  63. tracecite/tables/document.py +582 -0
  64. tracecite/tables/html.py +299 -0
  65. tracecite/tables/models.py +119 -0
  66. tracecite/tables/normalise.py +786 -0
  67. tracecite/tables/pandoc.py +503 -0
  68. tracecite/tables/publish.py +238 -0
  69. tracecite/tables/render.py +128 -0
  70. tracecite/tables/site.py +457 -0
  71. tracecite-0.0.2.dist-info/METADATA +320 -0
  72. tracecite-0.0.2.dist-info/RECORD +76 -0
  73. tracecite-0.0.2.dist-info/WHEEL +5 -0
  74. tracecite-0.0.2.dist-info/entry_points.txt +2 -0
  75. tracecite-0.0.2.dist-info/licenses/LICENSE +21 -0
  76. tracecite-0.0.2.dist-info/top_level.txt +1 -0
@@ -0,0 +1,71 @@
1
+ # Documenter.jl Integration
2
+
3
+ ## General boundary
4
+
5
+ Add routing after all maintained/generated Markdown sources exist and before `Documenter.makedocs` reads them.
6
+
7
+ ```text
8
+ source generation -> target-specific staging -> makedocs -> generated site
9
+ ```
10
+
11
+ Do not place publication routing inside analysis notebooks or Literate source generation.
12
+
13
+ ## Documenter lifecycle template
14
+
15
+ In a typical Documenter workflow:
16
+
17
+ ```sh
18
+ julia --project=docs -e 'using Pkg; Pkg.instantiate()'
19
+ julia --project=docs docs/render_literate.jl
20
+ julia --project=docs docs/make.jl
21
+ open docs/build/index.html
22
+ ```
23
+
24
+ Preserve these responsibilities:
25
+
26
+ - source generation, if any, creates maintained Markdown before routing begins;
27
+ - `make.jl` or the equivalent build driver reads a source tree, computes navigation, and runs Documenter;
28
+ - composition scripts should remain thin wrappers around generation and build steps;
29
+ - existing page registries, navigation functions, page kinds, statuses, data-layer labels, and ordering fields remain the host repository's navigation authority.
30
+
31
+ ## Template target selection
32
+
33
+ The following target selection and staging pattern is a generic template to re-verify against the host repository before `makedocs`.
34
+
35
+ ```julia
36
+ const LINK_TARGET = lowercase(strip(get(ENV, "DOCS_LINK_TARGET", "local")))
37
+ LINK_TARGET in ("local", "public") ||
38
+ error("DOCS_LINK_TARGET must be local or public")
39
+ ```
40
+
41
+ Use a sibling staging tree such as:
42
+
43
+ ```text
44
+ docs/src/ maintained
45
+ docs/.documenter-source/ generated staging
46
+ docs/build/ generated HTML
47
+ ```
48
+
49
+ Point `makedocs(source = STAGED_SRC, ...)` at the staging tree while retaining the existing page registry and navigation computation. Reinspect the actual target branch before producing a patch; constants and file structure may have changed.
50
+
51
+ ## Build commands
52
+
53
+ Local remains the default:
54
+
55
+ ```sh
56
+ julia --project=docs docs/make.jl
57
+ ```
58
+
59
+ Explicit local:
60
+
61
+ ```sh
62
+ DOCS_LINK_TARGET=local julia --project=docs docs/make.jl
63
+ ```
64
+
65
+ Public:
66
+
67
+ ```sh
68
+ DOCS_LINK_TARGET=public julia --project=docs docs/make.jl
69
+ ```
70
+
71
+ A GitHub Pages job should normally build from committed `docs/src/` rather than rerunning data-dependent Literate analyses. Verify action versions from current official documentation in the deployment task; do not hard-code stale versions in this skill.
@@ -0,0 +1,86 @@
1
+ # Source-Link Registry Contract
2
+
3
+ ## Purpose
4
+
5
+ `docs/source-links.toml` routes one stable local `.pdf` or `.md` source path to:
6
+
7
+ - the repository-local authoritative copy used for verification;
8
+ - the public representation used by generated public documentation.
9
+
10
+ It is location metadata, not an evidence graph or bibliography database. It does not replace BibTeX, BibLaTeX, CSL, Zotero, or DOI-resolution tooling, and it must not accumulate their responsibilities.
11
+
12
+ ## Schema
13
+
14
+ ```toml
15
+ schema_version = 3
16
+
17
+ [[source]]
18
+ name = "aemo-isp-2026"
19
+ local_path = "sources/aemo/2026-integrated-system-plan-isp.pdf"
20
+ public_url = "https://www.aemo.com.au/path/report.pdf?rev=...&sc_lang=en"
21
+
22
+ [source.metadata]
23
+ bibtex_id = "aemo2026isp"
24
+ publisher = "Australian Energy Market Operator"
25
+ type = "report"
26
+ ```
27
+
28
+ The same contract routes a maintained local Markdown page to a public HTML destination:
29
+
30
+ ```toml
31
+ [[source]]
32
+ name = "tracecite-searchable-evidence"
33
+ local_path = "docs/guide/searchable-evidence.md"
34
+ public_url = "https://example.org/guide/searchable-evidence/"
35
+
36
+ [source.metadata]
37
+ type = "documentation"
38
+ ```
39
+
40
+ Required fields:
41
+
42
+ | Field | Rule |
43
+ |---|---|
44
+ | `schema_version` | Integer `3`. |
45
+ | `name` | Non-empty, unique, stable human/machine identifier. No bibliography-specific key syntax is imposed. |
46
+ | `local_path` | Normalised repository-relative path ending in `.pdf` or `.md`; no query or fragment; unique per registry (one `local_path` maps to one entry). |
47
+ | `public_url` | Absolute `https` URL; preserve required query parameters; no fragment, credentials, or user info. |
48
+
49
+ Optional field:
50
+
51
+ | Field | Rule |
52
+ |---|---|
53
+ | `metadata` | An optional `[source.metadata]` table. TraceCite preserves arbitrary keys and values opaquely: it does not interpret, normalise, export, or publish them, and assigns no behaviour to keys such as `bibtex_id`, `publisher`, or `type`. Do not treat `metadata` as a second editable copy of TraceCite evidence identity; it stays external to the evidence database. |
54
+
55
+ Reject any other source-level field (for example a misspelled `publc_url`) rather than silently absorbing it into metadata.
56
+
57
+ ## Identity and compatibility
58
+
59
+ - `local_path` must match the maintained report's reference-definition or narrow inline-link destination and the TraceCite database path after normalisation.
60
+ - `name` is the stable identifier for the entry; citation labels in Markdown remain opaque, and the destination path plus `#page=N` fragment (PDF) or optional `#anchor` (Markdown) carry routing authority.
61
+ - The TraceCite verifier must read this `[[source]]` shape when `--source-links` is supplied.
62
+ - Do not retain an old schema-v2 (`title`/`publisher`/`public_origin`) or earlier registry shape as the publication authority; `local_path` is the routing key and schema v3 is the only accepted schema.
63
+
64
+ ## Path rules
65
+
66
+ Reject:
67
+
68
+ - absolute local paths;
69
+ - `..` traversal that escapes the repository root;
70
+ - symlink resolution outside approved source roots;
71
+ - query strings or fragments in `local_path`;
72
+ - a `local_path` extension other than `.pdf` or `.md`;
73
+ - duplicate normalised local paths.
74
+
75
+ ## URL rules
76
+
77
+ Reject:
78
+
79
+ - non-HTTPS publication URLs;
80
+ - credentials or secrets in URLs;
81
+ - `file:`, `javascript:`, or data URLs;
82
+ - a pre-existing fragment in `public_url`.
83
+
84
+ Query strings in `public_url` remain allowed; the router appends the authored PDF page or Markdown anchor fragment after any existing query string.
85
+
86
+ Do not fetch every URL during each documentation build. Perform remote reachability and optional hash comparison in a separate scheduled check so publication does not depend on third-party availability.
@@ -0,0 +1,118 @@
1
+ # Staging and Rewriting
2
+
3
+ ## Atomic staging sequence
4
+
5
+ ```text
6
+ validate registry
7
+ create fresh temporary staging directory
8
+ copy maintained source tree
9
+ rewrite target-specific definitions in the temporary tree
10
+ validate transformed tree
11
+ rename old staging tree aside
12
+ atomically rename temporary tree to final staging path
13
+ remove old staging tree after success
14
+ ```
15
+
16
+ If any step fails, delete the temporary candidate and preserve the previous valid staging tree.
17
+
18
+ ## Allowed transformation surface
19
+
20
+ Rewrite only Markdown link reference definitions or narrow inline links whose
21
+ destination resolves to a mapped local `.pdf` or `.md` source.
22
+
23
+ PDF destinations require a positive `#page=N` fragment:
24
+
25
+ ```markdown
26
+ [label]: ../../../sources/aemo/report.pdf#page=14
27
+ ```
28
+
29
+ ```markdown
30
+ [descriptive text](../../../sources/aemo/report.pdf#page=14)
31
+ ```
32
+
33
+ Mapped Markdown destinations use `relative.md` or `relative.md#anchor`:
34
+
35
+ ```markdown
36
+ [label]: ../guide/searchable-evidence.md#anchor
37
+ ```
38
+
39
+ ```markdown
40
+ [descriptive text](../guide/searchable-evidence.md)
41
+ ```
42
+
43
+ The destination must be whitespace-free, repository-relative, unwrapped, and
44
+ have no title or query string. A PDF candidate must end in a positive
45
+ physical `#page=N` fragment. A Markdown candidate must end in `.md`, with at
46
+ most one `#anchor` fragment carried through verbatim. Multiple recognised
47
+ links on one line are supported.
48
+
49
+ A PDF candidate with an unsafe path, malformed/non-positive page, or missing
50
+ registry entry fails rather than being partially rewritten. A Markdown
51
+ candidate behaves more conservatively: only a destination that both parses
52
+ cleanly (no query string, no angle-bracket wrapping, no remote scheme, exactly
53
+ one optional non-empty anchor, and a `.md` extension) and resolves to exactly
54
+ one registered entry is rewritten; every other `.md`-shaped destination
55
+ (malformed, escaping, absolute, query-bearing, or simply unmapped) is left
56
+ untouched rather than erroring, because most Markdown links are ordinary
57
+ documentation navigation rather than source citations.
58
+
59
+ Do not rewrite:
60
+
61
+ - images, autolinks, remote URLs, links with titles, and unsupported inline syntax;
62
+ - code fences using backticks or tildes;
63
+ - indented code blocks;
64
+ - Documenter raw blocks such as `@raw html`;
65
+ - frontmatter, Markdown table rows, generated logs, non-document output, or
66
+ artifacts;
67
+ - links that do not resolve to a registered `.pdf`/`.md` entry;
68
+ - maintained Markdown files in place.
69
+
70
+ Committed or generated `.md` files inside the staged documentation source tree
71
+ are routable documentation inputs. This does not extend routing to generated
72
+ logs, non-document output, or artifacts. Markdown tables inside those files
73
+ remain byte-identical, including their header, delimiter, and body rows.
74
+
75
+ A conservative implementation may use a line-oriented state machine, but it must preserve line endings and all untouched bytes where practical. Do not use a broad regular expression over the entire document without fence/raw-block awareness.
76
+
77
+ ## Path resolution
78
+
79
+ For each candidate definition or inline link:
80
+
81
+ 1. split the destination into base path, query, and fragment;
82
+ 2. for a PDF candidate, require a positive `#page=N` fragment; for a Markdown candidate, accept an optional non-empty `#anchor`;
83
+ 3. resolve the base path relative to the Markdown file in the staged tree;
84
+ 4. translate that path back to the corresponding repository-relative maintained path;
85
+ 5. find exactly one registry entry whose normalised `local_path` matches (a PDF miss/ambiguity fails; a Markdown miss simply means "not a routing candidate," so leave it untouched);
86
+ 6. treat the Markdown reference label as opaque; do not decode, validate, or infer a source path from it;
87
+ 7. validate authority only from the resolved destination's normalised `local_path` plus the fragment (positive physical `#page=N` for PDF, verbatim `#anchor` for Markdown);
88
+ 8. render the destination for the requested target.
89
+
90
+ Resolve inline destinations relative to the containing Markdown file, not the
91
+ documentation tree root. Do not rewrite inline code, four-space indented code,
92
+ frontmatter, backtick/tilde fences (including Documenter raw fences), raw HTML,
93
+ or escaped link syntax. Preserve all untouched bytes and the maintained source.
94
+
95
+ ## Target rendering
96
+
97
+ ### Local
98
+
99
+ Preserve the local destination or recalculate an equivalent path relative to the staged page when staging depth differs. Do not convert it to an absolute filesystem path. The fragment (`#page=N` or `#anchor`, if any) is carried through unchanged.
100
+
101
+ ### Public
102
+
103
+ ```text
104
+ public_url + original_fragment
105
+ ```
106
+
107
+ When `public_url` contains a query string, append the fragment after it:
108
+
109
+ ```text
110
+ https://host/report.pdf?rev=abc&lang=en#page=14
111
+ https://host/guide/searchable-evidence/?rev=abc#anchor
112
+ ```
113
+
114
+ A bare Markdown destination with no authored anchor appends no fragment at all.
115
+
116
+ ## Idempotence
117
+
118
+ Running the same target build twice from unchanged inputs must produce byte-identical staged Markdown. Public staged output must never be used as the next run's source; always restage from maintained Markdown.
@@ -0,0 +1,67 @@
1
+ # Validation and Failure Modes
2
+
3
+ ## Required fixture coverage
4
+
5
+ Test both targets with fixtures covering:
6
+
7
+ - a valid local PDF definition;
8
+ - a valid local Markdown-to-HTML mapping, with and without an authored anchor;
9
+ - an official URL containing a query string (PDF and Markdown);
10
+ - a positive physical page fragment;
11
+ - repeated use of one reference definition;
12
+ - multiple reports referencing the same source;
13
+ - fenced-code and Documenter raw-block examples that remain byte-identical;
14
+ - malformed, missing, duplicate (by `name` and by `local_path`), and ambiguous mappings;
15
+ - path traversal and absolute-path rejection;
16
+ - public URL scheme and fragment rejection;
17
+ - opaque `metadata` surviving parsing without affecting routing or verification;
18
+ - explicit schema-v2 (or earlier) rejection with a clear schema-v3 message;
19
+ - atomic staging failure that preserves the previous valid tree;
20
+ - idempotent repeated builds;
21
+ - no mutation of maintained Markdown;
22
+ - multiple inline PDF/Markdown links on one line;
23
+ - positive and malformed inline page fragments, query strings, titles, images,
24
+ autolinks, inline code, escaped syntax, and nested file-relative paths;
25
+ - an unmapped local Markdown candidate left untouched rather than erroring;
26
+ - a host-renderer pressure test that checks the selected maintained syntax
27
+ produces an actual HTML anchor.
28
+
29
+ ## Cross-system validation
30
+
31
+ After adopting the final registry:
32
+
33
+ 1. patch the existing TraceCite source-link reader rather than creating a second verifier;
34
+ 2. rerun the complete TraceCite test suite;
35
+ 3. update its skill references that mention the placeholder schema;
36
+ 4. re-audit and repackage `using-tracecite`;
37
+ 5. build reporting fixtures against the final `[[source]]` schema.
38
+
39
+ ## Build exclusions
40
+
41
+ The public deployment artifact must not contain:
42
+
43
+ ```text
44
+ source PDFs
45
+ tracecite.sqlite
46
+ embedding-model caches
47
+ source-capture workspaces
48
+ private evidence
49
+ staging directories
50
+ local absolute paths
51
+ ```
52
+
53
+ Upload only the generated static site and deliberately public assets.
54
+
55
+ ## Diagnostics
56
+
57
+ Errors must identify:
58
+
59
+ - Markdown file and line;
60
+ - reference key or inline link text;
61
+ - parsed local destination and page fragment or anchor;
62
+ - expected source path;
63
+ - registry entries considered;
64
+ - target mode;
65
+ - corrective action.
66
+
67
+ Do not silently leave a local PDF link in public output when routing was required. Conversely, do not turn an unmapped local file into a guessed public URL. An unmapped local Markdown link is deliberately left untouched rather than treated as an error (see `references/staging-and-rewriting.md`).
@@ -0,0 +1,97 @@
1
+ ---
2
+ name: using-tracecite
3
+ description: Use when local PDF, Markdown, or OOXML workbook sources must be synchronised into a TraceCite SQLite evidence database, searched lexically and semantically, inspected through source-specific locators, pruned, or diagnosed. PDF-only operations also support selected pages, assets, derivative PDFs, quotation checks, and report verification. Requires explicit runtime paths or a profile. Do not use for source capture, report prose, or public-link routing.
4
+ ---
5
+
6
+ # Using TraceCite
7
+
8
+ Synchronise an explicit local corpus into a TraceCite database with FTS5 lexical search, sqlite-vec semantic search, retained PDF-page, Markdown-line, and workbook sheet/A1 locators, generated PDF assets, quotation verification, report verification, explicit pruning, and `doctor` diagnostics.
9
+
10
+ ## When to use
11
+
12
+ Use this skill once source files already exist on disk and an agent needs to:
13
+
14
+ - sync configured `.pdf`, `.md`, `.xlsx`, and `.xlsm` sources without auto-pruning unavailable content;
15
+ - search by wording and semantic similarity;
16
+ - retrieve PDF evidence by normalised root-relative path plus selected physical pages;
17
+ - inspect workbook evidence through its source SHA-256, worksheet, bounding range, and exact A1 ranges;
18
+ - verify a quotation or a Markdown report against retained source text;
19
+ - inspect database/assets with `doctor` or run an explicit prune preview/apply.
20
+
21
+ Do not use this skill for discovering or downloading sources, writing report narrative (`writing-evidence-backed-reports` owns that), or substituting local/public documentation links (`routing-documentation-source-links` owns that). This skill owns TraceCite runtime use, path/page evidence identity, synchronisation, search, verification, pruning, and diagnostics.
22
+
23
+ ## Prerequisites
24
+
25
+ - A TraceCite package and a Python environment that installs the `tracecite` console script. Installing this skill alone does not install the executable package.
26
+ - A source root containing one or more supported `.pdf`, `.md`, `.xlsx`, or `.xlsm` files.
27
+ - A combined TraceCite config with explicit inline `[[source]] path = ...`, `[[include]] glob = ...`, and optional `[[exclude]] glob = ...` rules. Separate `--manifest` inputs may augment those inline rules.
28
+ - A database path and model-cache path owned by the caller. Repository-neutral callers may choose any paths.
29
+
30
+ ## Exact invocation
31
+
32
+ Prefer the installed console script:
33
+
34
+ ```bash
35
+ tracecite <command> [--config PATH] [...]
36
+ ```
37
+
38
+ When the package checkout and runtime environment differ, resolve them separately and use the environment that owns the runtime artifacts; do not rely on imports from a disposable checkout.
39
+
40
+ ## Procedure
41
+
42
+ 1. **Confirm runtime and inputs.** Verify `tracecite --help`, source root, manifests/profile, database path, and model-cache path. Do not invent separate public IDs. PDF and Markdown evidence use the normalised root-relative path plus their page or line locator. Workbook evidence additionally uses the indexed source SHA-256, worksheet, and exact A1 ranges.
43
+ 2. **Sync the corpus:**
44
+ ```bash
45
+ tracecite sync --config <config-path>
46
+ tracecite sync path/inside/root.pdf --root <dir> --manifest <manifest.toml> --database <db> --model-cache-dir <cache>
47
+ ```
48
+ Sync is incremental, idempotent, and non-pruning. Missing selected sources and indexed-but-unselected sources remain stored until explicit pruning.
49
+ 3. **Search, retrieve, and verify:**
50
+ ```bash
51
+ tracecite search "later coal retirement" --database <db> --limit 20
52
+ tracecite page path/to/source.pdf 65 --database <db>
53
+ tracecite page path/to/source.pdf 63-66 --database <db>
54
+ tracecite page path/to/source.pdf 63-66 --format json --database <db>
55
+ tracecite extract-pages path/to/source.pdf 63-66 --output-dir <dir> --database <db>
56
+ tracecite verify quote path/to/source.pdf 65 "Exact quotation" --database <db>
57
+ tracecite verify report docs/reports/<report>.md --root <dir> --database <db> [--source-links docs/source-links.toml --source-links-root <dir>]
58
+ tracecite doctor --database <db>
59
+ ```
60
+ 4. **Build evidence from retrieval results.** Treat search rank and fused score as candidate-discovery signals, not proof. Narrow a mixed corpus with more specific wording and by inspecting the returned `source_path`, `source_type`, and `source_sha256`; do not infer that a result is from the intended year or source merely because it ranked first. Markdown results can help locate a claim but are not external-source proof.
61
+
62
+ For workbook results, inspect `locator.sheet` and `locator.exact_ranges` against the indexed workbook. `locator.range` is a bounding rectangle and may include cells that did not contribute to the passage. Use a workbook viewer for cell inspection; `tracecite page`, `extract-pages`, quotation verification, and PDF assets do not apply to workbook sources.
63
+
64
+ For each candidate, retrieve the complete physical page with `tracecite page`. Omission selects physical page 1. Selectors accept positive pages, closed ranges (`63-66`), open ranges (`63-` or `-66`), and comma-separated combinations; overlapping terms are deduplicated in ascending physical-page order. The standalone selector `all` explicitly selects every indexed page and cannot be combined with another term.
65
+
66
+ Inspect neighbouring pages when the claim depends on a table, figure, footnote, or continuation. When layout matters, use `tracecite page ... --format json` to locate the validated indexed page render and figure crops. If a PDF derivative is required by the inspection tool, use `tracecite extract-pages` with the same selector and an existing caller-owned output directory outside the source root. Do not open the complete source PDF for a page-local question; use `all` only when complete-source inspection is intentional.
67
+
68
+ If a quotation is used, run `tracecite verify quote` against the physical page. `exact` means the quotation matches retained text; `normalised` is acceptable only when the quoted wording is exact after the tool's documented normalisation; `not-found` is not verification and must not be silently paraphrased as verified. Resolve conflicts by retaining the competing source/page evidence and stating the limitation.
69
+
70
+ Construct the final link from the returned normalised `source_path` and physical page, for example `<source-path>#page=<N>`. Use an official publisher or agency URL when the documentation's public-link policy provides one; otherwise use the local source link and label its local scope. A physical PDF page may differ from the printed page label.
71
+ 5. **Prune only on explicit request.** Run `tracecite prune ...` first as a preview. A normal prune must contain a positive `[[source]]` or `[[include]]` selector; excludes only narrow it, and a zero-match selection is rejected. Use `tracecite prune --all ...` only when the user explicitly intends to remove every indexed source. Only `tracecite prune --apply ...` may commit removals. Treat exit status `3` as a committed database prune with incomplete filesystem cleanup, not as an uncommitted failure.
72
+ 6. **Run `doctor` after bulk syncs, restores, or suspected corruption.** Treat reported database, vector, asset, or hash issues as blocking until resolved.
73
+
74
+ ### Query syntax and punctuation
75
+
76
+ The query is passed to both FTS5 `MATCH` and vector retrieval. Valid raw FTS5 expressions, including quoted phrases, boolean operators, and prefix expressions such as `retire*`, retain their normal lexical behaviour. Quote the shell argument when it contains spaces or FTS punctuation.
77
+
78
+ Ordinary prose containing punctuation that FTS5 rejects, such as `10%, 50%, and 90%`, does not abort hybrid retrieval. TraceCite reports the lexical fallback on stderr, retries lexical search with a quoted query, and sends the original text to vector retrieval. The fallback is only for the identified FTS5 syntax error; unrelated SQLite errors still propagate. Because diagnostics use stderr, `--json`-compatible search output remains valid JSON on stdout.
79
+
80
+ ## Boundaries
81
+
82
+ - PDF, Markdown, `.xlsx`, and `.xlsm` OOXML workbooks are supported source formats. Legacy binary `.xls` is not supported.
83
+ - Workbook ingestion reads stored package content only. It does not execute VBA, recalculate formulae, refresh external data, or reproduce all Excel display formatting.
84
+ - Do not add old command/package compatibility aliases.
85
+ - Never mutate authoritative source files, and never let verification rewrite maintained Markdown.
86
+ - Keep selected-page derivatives outside the source root. They are disposable caller-owned runtime outputs and must not be synchronised automatically.
87
+ - Do not commit generated databases, embedding-model caches, SQLite sidecars, PDF renders/crops, or other runtime derivatives.
88
+ - Help commands must remain lightweight: they must not import or touch parsers, PDF/OCR, embeddings, sqlite-vec, manifests, databases, projects, caches, or network resources.
89
+
90
+ ## References
91
+
92
+ - `references/architecture.md` — module layout, layering, and out-of-scope technology list.
93
+ - `references/schema-and-migrations.md` — current schema and path/page evidence contract.
94
+ - `references/synchronisation.md` — invalidation, retention, rename, transaction, and pruning behaviour.
95
+ - `references/retrieval.md` — hybrid FTS5/vector search design and result fields.
96
+ - `references/pdf-and-assets.md` — PDF page/chunk extraction, quotation verification, and asset semantics.
97
+ - `references/workspace-runtime.md` — Git worktree-safe path resolution and runtime setup.
@@ -0,0 +1,3 @@
1
+ interface:
2
+ display_name: "Using TraceCite"
3
+ short_description: "Synchronise local PDFs, Markdown, and OOXML workbooks into a searchable TraceCite SQLite database."
@@ -0,0 +1,68 @@
1
+ # Architecture
2
+
3
+ ## Module layout
4
+
5
+ ```text
6
+ tracecite/
7
+ pyproject.toml, uv.lock # locked dependency truth
8
+ src/tracecite/
9
+ __init__.py
10
+ cli.py # command parser and router for every subcommand
11
+ evidence/
12
+ commands.py # sync, search, page, extract-pages, verify, prune, doctor
13
+ parsers/
14
+ base.py # ParsedPage / ParsedChunkUnit / units_from_layout
15
+ pdf.py # PyMuPDF-backed extraction, page render, figure crops
16
+ markdown.py
17
+ workbook.py # OOXML worksheet and A1-range extraction
18
+ chunking.py # greedy char-budget grouping, normalisation, hashing
19
+ schema.py # versioned schema + path-keyed sources + connect/ensure_schema
20
+ sync.py # the whole incremental synchronisation lifecycle
21
+ vector_backend.py # SqliteVecBackend: all vec0 SQL
22
+ page_selection.py # physical PDF page selector grammar
23
+ page_output.py # validated PDF page/crop result payloads
24
+ page_extraction.py # selected-page derivative PDFs and manifests
25
+ verify.py # verify quote / verify report
26
+ tests/ # pytest coverage for package and CLI behaviour
27
+ fixtures/ # stable test fixtures
28
+ ```
29
+
30
+ Install the package and use the `tracecite` console script. There is no supported old-name package or command shim.
31
+
32
+ ## Layering
33
+
34
+ ```text
35
+ parsers/{pdf,markdown,workbook}.py -- format-specific extraction only, no SQL
36
+ |
37
+ chunking.py -- format-agnostic grouping + identity hashing, no SQL
38
+ |
39
+ sync.py -- the only place that opens a write transaction
40
+ |
41
+ schema.py + vector_backend.py -- the only places that know SQL/vec0 syntax
42
+ ```
43
+
44
+ - `commands.py` and `sync.py` reach vectors only through `SqliteVecBackend` (`upsert`, `delete`,
45
+ `search`, `integrity_check`), never through vec0 virtual-table SQL directly.
46
+ - `pages` rows (see `references/schema-and-migrations.md`) are the retained parser output. A
47
+ chunker- or normalisation-only change rebuilds chunk-input units straight from `pages.layout_json`
48
+ (`pdf.units_from_page_layout` for PDFs, `base.units_from_layout` for Markdown and workbooks)
49
+ without reopening the source file — see `references/synchronisation.md`.
50
+
51
+ ## Explicit-input contract
52
+
53
+ The CLI never infers a project checkout. Commands that touch a corpus take `--root`, one or more `--manifest` values, `--database`, and `--model-cache-dir` explicitly, or receive those values from an explicitly named profile. Read-only commands such as `search`, `page`, `verify quote`, `verify report`, and `doctor` use the database plus the path/page arguments they need. A host bootstrap may supply defaults, but the package remains repository-neutral.
54
+
55
+ ## Out of scope by design (do not reintroduce)
56
+
57
+ - An approximate-nearest-neighbour index. Retrieval does an exact k-NN scan through
58
+ `SqliteVecBackend`; see `references/retrieval.md` for why and how bounded filters still work
59
+ without one.
60
+ - A second document store (JSONL, a `document.json` per file, LanceDB, DuckDB as the live index).
61
+ `pages`/`chunks` in `tracecite.sqlite` are the only retained parsed representation.
62
+ - Deterministic hash vectors as a stand-in for real semantic embeddings.
63
+ - OCR, code/notebook/YAML/JSON parser adapters, and a Rust extension/sidecar — all explicitly
64
+ deferred until a representative fixture proves the pure-Python/PyMuPDF/sqlite path is
65
+ insufficient. `parsers/base.py`'s `Parser` protocol is the seam for adding an adapter later.
66
+ - A ChatGPT `skill-creator`/`init_skill.py`/`package_skill.py` packaging step — this repository uses
67
+ its own `writing-a-skill` and `auditing-a-skill` checks plus a plain deterministic zip instead
68
+ (see the repository's operator guide).
@@ -0,0 +1,67 @@
1
+ # PDF Parsing, Assets, and Verification
2
+
3
+ ## PDF extraction
4
+
5
+ `parsers/pdf.py` uses PyMuPDF (`pymupdf`) exclusively, behind the `Parser` protocol in
6
+ `parsers/base.py`. Per physical page: `page.get_text("blocks")` is filtered to text blocks and
7
+ sorted into reading order (`round(y0, 1)`, then `round(x0, 1)`); each block becomes one
8
+ `ParsedChunkUnit` with a positional `logical_key` (`page{NNNN}-block{NNN}`), plus a lightweight
9
+ heading heuristic (`_looks_like_heading`: short, no trailing sentence punctuation) that seeds
10
+ `heading_path` for subsequent body blocks until the next heading-like block. The retained page
11
+ `layout_json` stores every block's text/bbox/heading_path/offsets, so a later chunker- or
12
+ normalisation-only change reconstructs the same units via `pdf.units_from_page_layout()` without
13
+ reopening the PDF.
14
+
15
+ `chunking.group_units()` merges adjacent units into one chunk while they share the same
16
+ `heading_path` and `physical_page` and stay under `--max-chunk-chars` (default 1200) — so a chunk
17
+ never silently straddles a heading or page boundary. PDF chunks are page-local: a chunk never spans two pages.
18
+
19
+ ## Markdown extraction
20
+
21
+ `parsers/markdown.py` parses `#`-`######` heading hierarchy and blank-line-delimited paragraphs.
22
+ Oversized paragraphs are further split on whitespace boundaries (`DEFAULT_MAX_UNIT_CHARS = 2400`)
23
+ so `chunking.group_units()` never has to split a single unit. `logical_key` is
24
+ `<heading-path>#<heading-occurrence>:p<paragraph-index>` (with a `:sub{n}` suffix for a split
25
+ oversized paragraph), robust to line-number shifts elsewhere in the document. A Markdown source
26
+ gets one synthetic `pages` row (`physical_page = 1`) holding the full document text and a
27
+ `layout_json` unit list, mirroring the PDF retained-extraction model.
28
+
29
+ ## Physical-page retrieval
30
+
31
+ `tracecite page <source-path> [selector]` reads `pages.text` directly from SQLite—no search ranking or PDF reopening is involved. Omission selects physical page 1. A selector accepts a positive page, a closed range (`5-9`), an open range (`5-` or `-9`), or comma-separated combinations. Terms are expanded against indexed pages, deduplicated, and returned in ascending physical-page order. The standalone selector `all` selects every indexed page and cannot be combined with another term. TraceCite validates the complete selection before writing stdout, so a missing implied page cannot produce partial output.
32
+
33
+ Plain-text output for one selected page retains the original page command contract. Multiple pages have explicit physical-page delimiters. `--format json` always returns an ordered array; each element contains retained text, printed-label and extraction metadata, a `pdf_link`, one validated whole-page render or `null`, and validated figure-crop entries. Asset entries expose their database-relative identifier, validated local path, SHA-256, dimensions, labels, nearby text, and bounding box. Missing, escaped, or hash-mismatched assets fail before JSON is printed.
34
+
35
+ `tracecite extract-pages <source-path> [selector] --output-dir <dir>` uses the same selector grammar and page-1 omission default. Unlike `page`, it verifies and opens the indexed source PDF, writes only the normalized selected pages, and creates a provenance manifest. The existing output directory must be a real directory outside the source root. Source-hash mismatch, PDF/index page-count mismatch, overlap, symlinks, and existing outputs are rejected. The PDF and manifest are staged together and rolled back together after promotion failure; generated files remain caller-owned disposable runtime artifacts.
36
+
37
+ ## Quotation verification
38
+
39
+ `tracecite verify quote <source-path> <physical-page> <quote>` checks, in order: (1) exact substring
40
+ match against `pages.text`; (2) a whitespace-normalised match (`chunking.normalise_text`: collapse
41
+ all whitespace, no case-folding) against the normalised page text; (3) otherwise `"not-found"`.
42
+ An empty quotation is `"structural-error"`. Only `exact` and `normalised` return process status
43
+ `0`; all other verification states return non-zero. This is intentionally independent of chunk
44
+ boundaries — a quotation may straddle two chunks.
45
+
46
+ ## Report verification
47
+
48
+ `tracecite verify report <report.md> --root <dir> --database <db>` parses Markdown reference definitions whose destinations contain a local path plus `#page=<PAGE>` and inline citations (`[text][<label>]`). Labels are opaque. For every citation used in the body it checks: a matching definition exists; the definition's relative path resolves (relative to the report file's own directory) to an indexed source path; and the cited page is indexed. Blockquote lines (`> "..."`) are matched
49
+ only when the immediately associated prose paragraph resolves to exactly one viable citation label.
50
+ Ambiguous, missing, or empty quote associations are structural errors. Each quote result emits the
51
+ bound `citation_key` and resolved `source_path`, or `null` when no unique binding exists. Report
52
+ verification never writes to the report file — read-only, always.
53
+
54
+ When `--source-links docs/source-links.toml` is supplied, report verification additionally confirms every cited source path has exactly one schema-v3 registry entry keyed by `local_path`; it never edits the registry or the report. The `source-links.toml` schema is owned by `routing-documentation-source-links` and uses a `[[source]]` array of tables with `name`, `local_path`, `public_url`, and an opaque optional `metadata` table. Do not restore an obsolete placeholder registry shape.
55
+
56
+ ## Assets
57
+
58
+ For every PDF page that gets (re)parsed, `sync._persist_assets()` writes one whole-page PNG render
59
+ (`pdf.render_page()`, via `page.get_pixmap()`) under `imgs/generations/<generation>/<source_pk>/page-{NNNN}.png`, plus one crop
60
+ per embedded raster image found on that page (`pdf.render_figure_crops()`, via
61
+ `page.get_images()`/`extract_image()`/`get_image_rects()`) to
62
+ `imgs/generations/<generation>/<source_pk>/page-{NNN}-image-{NN}.png`. New sources render into a same-generation staging directory and are atomically renamed to the assigned internal `source_pk` before asset rows are inserted. Both renders and crops are recorded in `assets` with their sha256,
63
+ dimensions, and (for crops) bounding box. **Limitation:** only embedded raster images are cropped;
64
+ a figure composed purely of vector graphics has no dedicated crop yet — use the whole-page render
65
+ for those until a fixture demonstrates the need for vector-composited crops. Assets live under `imgs/` next to the database (never inside SQLite); active database rows and `doctor` validate asset paths and hashes by internal key, not public source identity.
66
+
67
+ Page JSON and PDF search output resolve these existing asset rows only. Retrieval never regenerates a page render or crop and never treats an unchecked database path as a filesystem path.
@@ -0,0 +1,45 @@
1
+ # Retrieval
2
+
3
+ ## Hybrid search
4
+
5
+ ```text
6
+ FTS5 BM25 top --fts-limit (default 50)
7
+ +
8
+ SqliteVecBackend exact k-NN top --vector-limit (default 50)
9
+ |
10
+ v
11
+ reciprocal-rank fusion (k = 60), reduced to --limit (default 10)
12
+ ```
13
+
14
+ `commands.hybrid_search()` computes both candidate lists, fuses them by `score(chunk) = sum(1 / (60 + rank))` over whichever list(s) contain that chunk, then returns the top `--limit` chunks (configurable 10-20 is the practical default; both `--fts-limit`
15
+ and `--vector-limit` are independently configurable). Each result reports `rank`, `source_path`, `source_type`, `source_sha256`, `heading_path`, `passage` (`body`), `physical_page`, `page_offsets`, `page_range`, `line_range`, `locator`, `content_type`, `provenance` (`["lexical"]`, `["vector"]`, or both), and `fused_score`. PDF results for a physical page also include a ready-to-use `pdf_link` of the form `<source-path>#page=<N>`, a validated `page_render`, and zero or more validated `figure_crops`. Results without a PDF physical page use `null` for `page_render` and an empty list for `figure_crops`. Internal `chunk_id`, `source_pk`, page IDs, and asset IDs are not part of the result contract. Fused scores are relative ranking signals, not calibrated probabilities; never present them as absolute semantic confidence.
16
+
17
+ ## Query recovery
18
+
19
+ The query is passed unchanged to vector embedding. Valid raw FTS5 `MATCH` expressions—quoted phrases, operators, and prefix expressions—retain their lexical behavior. If FTS5 reports its lexical syntax error, TraceCite emits a diagnostic on stderr and retries only the lexical half with the query quoted as a phrase. This permits ordinary punctuation-heavy text such as `10%, 50%, and 90%` to continue through hybrid retrieval without contaminating JSON stdout. Other SQLite errors propagate unchanged; the fallback is not a general query sanitizer.
20
+
21
+ Search results are for discovery. For PDFs, inspect `source_path`, retrieve the full physical page or neighbouring page selection with `tracecite page`, and use JSON page output to locate indexed page renders and figure crops when layout matters. Create a selected-page derivative with `tracecite extract-pages` only when the inspection tool requires PDF input. Use `tracecite verify quote` before treating wording as an exact or normalised quotation. A physical page number is the PDF page identity and may differ from the printed label.
22
+
23
+ For workbooks, inspect `source_sha256`, `locator.sheet`, and `locator.exact_ranges` in the authoritative workbook. `locator.range` is a bounding rectangle rather than a claim that every enclosed cell contributed to the result. Workbook search does not create a portable browser deep link or independently verify a displayed Excel rendering. Rank, fused score, Markdown hits, workbook hits, and a failed verification do not establish external-source proof.
24
+
25
+ Vector search is scoped to the **active** embedding model only: `hybrid_search()` restricts
26
+ `allowed_embedding_ids` to `embeddings` rows whose `model_id` matches the current `kb_config`, so a
27
+ retained-but-inactive older model's vectors never leak into ranking.
28
+
29
+ ## No ANN index
30
+
31
+ There is no approximate-nearest-neighbour index. `SqliteVecBackend.search()` runs exact `vec0` k-NN queries, applying allowed-ID filters before ranking. Each shard returns at most `top_k` candidates, bounded by sqlite-vec's per-query limit, and Python selects the global nearest results. Filtering after truncation would silently miss eligible vectors. Add quantisation or an ANN structure only after a measured retrieval-quality/latency need on a real corpus, not speculatively.
32
+
33
+ ## Metadata filters
34
+
35
+ `SqliteVecBackend.search(conn, query_vector, top_k, allowed_embedding_ids=...)` is the seam for
36
+ bounded filtering (e.g. restrict to a source, page range, or content type) — resolve the allowed
37
+ `embedding_id` set with an ordinary SQL join over `chunks`/`chunk_embeddings`/`embeddings`, then pass
38
+ it through. Do not add vec0-specific partition/auxiliary-column SQL outside `vector_backend.py`.
39
+
40
+ ## Exact terms
41
+
42
+ FTS5's `unicode61` tokenizer preserves exact identifiers, numbers, and acronyms in the lexical
43
+ half of retrieval; do not rely on the vector half alone for exact-wording queries (`tracecite verify quote`
44
+ and `tracecite verify report` intentionally bypass ranked search entirely and check the retained `pages.text`
45
+ directly — see `references/pdf-and-assets.md`).