lovspor 0.2.0__tar.gz

lovspor-0.2.0/.gitignore

@@ -0,0 +1,43 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+
+# Virtual environment
+.venv/
+venv/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.coverage.*
+htmlcov/
+.mutmut-cache
+.mutmut-cache/
+mutmut-stats.json
+html/
+.tox/
+.uv-cache/
+.hypothesis/
+
+# Environment
+.env
+.envrc
+
+# Local data — never commit (see CLAUDE.md and docs/legal-and-sources.md)
+data/cache/
+data/raw/
+data/state/
+*.tar.bz2
+*.tar.gz
+
+# Editor / OS
+.DS_Store
+.idea/
+.vscode/
+*.swp

lovspor-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bartosz Kobylinski
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lovspor-0.2.0/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.4
+Name: lovspor
+Version: 0.2.0
+Summary: Norwegian law change tracker — produces the lovverk corpus from Lovdata public data
+Project-URL: Homepage, https://github.com/bartoszkobylinski/lovspor
+Project-URL: Corpus, https://github.com/bartoszkobylinski/lovverk
+Project-URL: Issues, https://github.com/bartoszkobylinski/lovspor/issues
+Author-email: Bartosz Kobylinski <bartosz.kobylinski@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Bartosz Kobylinski
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: legal-corpus,lovdata,nlod,norwegian-law,rag
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Text Processing :: Markup
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: lxml>=5.3.0
+Requires-Dist: mcp>=1.0.0
+Requires-Dist: numpy>=1.26.0
+Requires-Dist: pydantic>=2.9.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: tiktoken>=0.7.0
+Requires-Dist: typer>=0.12.5
+Description-Content-Type: text/markdown
+
+# lovspor
+
+Norwegian law change tracker. Engine that produces the [`lovverk`](https://github.com/bartoszkobylinski/lovverk) corpus from Lovdata's public-data API and serves it to AI assistants over MCP (Model Context Protocol).
+
+## Status
+
+**Production.** A scheduled GitHub Actions workflow runs daily at 04:00 UTC, pulls the latest tarballs from Lovdata, classifies each document as new / updated / renamed / removed, renders the changes to Markdown, and pushes the diff to `lovverk` as conventional-commit history. The corpus mirrors close to **6 000 acts** (Norwegian *lover* and central *forskrifter*), each with a structured per-act change history under `<dataset>/history/<slug>.json`. The exact live count is always available from the MCP `corpus_status` tool.
+
+Sprint 9 (MERGED 2026-05-06) added per-section embeddings to the corpus and a four-layer anti-hallucination story to the MCP surface: `semantic_search` (cosine over embeddings), `verify_quote` (verbatim-citation guard), validated `cross_references` on `get_section`, and `validate_citation` as the off-ramp for ambiguous citations.
+
+See [`docs/decisions.md`](docs/decisions.md) for the full architecture and design rationale.
+
+## MCP server
+
+`lovspor` ships a stdio MCP server that exposes the `lovverk` corpus to AI assistants — Claude Desktop, Claude Code, or any MCP client. Once configured, you can ask *"what changed in Skatteloven this year?"* or *"are there forskrifter about jernbane?"* and the assistant answers from the live corpus instead of stale training data.
+
+**Setup — three steps:**
+
+1. **Install [`uv`](https://docs.astral.sh/uv/)** (it provides `uvx`, which runs `lovspor` on demand — no manual install).
+
+2. **Fetch the corpus.** One command shallow-clones the legal text to the default cache (`~/.cache/lovverk`):
+
+   ```bash
+   uvx --from "git+https://github.com/bartoszkobylinski/lovspor.git" lovspor fetch-corpus
+   ```
+
+   Re-run it any time to update — it reports `cloned`, `updated`, or `unchanged`.
+
+3. **Register the server.** `lovspor mcp` finds that cache automatically. With Claude Code:
+
+   ```bash
+   claude mcp add lovverk -- \
+     uvx --from "git+https://github.com/bartoszkobylinski/lovspor.git" lovspor mcp
+   ```
+
+   Or add it to your client config directly — Claude Desktop's `claude_desktop_config.json`, or `~/.claude.json` for Claude Code:
+
+   ```jsonc
+   {
+     "mcpServers": {
+       "lovverk": {
+         "command": "uvx",
+         "args": [
+           "--from", "git+https://github.com/bartoszkobylinski/lovspor.git",
+           "lovspor", "mcp"
+         ]
+       }
+     }
+   }
+   ```
+
+Restart the client and `lovverk` appears in its MCP list. Fifteen of the sixteen tools work immediately — no key, and no network access beyond your local corpus clone.
+
+**Optional — enable `semantic_search`.** The one search-by-meaning tool needs an OpenAI API key: it embeds *your query* at call time (the corpus vectors ship pre-computed, so you never re-embed the corpus yourself). Bring your own key via the server's `env`:
+
+```jsonc
+{
+  "mcpServers": {
+    "lovverk": {
+      "command": "uvx",
+      "args": [
+        "--from", "git+https://github.com/bartoszkobylinski/lovspor.git",
+        "lovspor", "mcp"
+      ],
+      "env": { "OPENAI_API_KEY": "sk-...your-own-key..." }
+    }
+  }
+}
+```
+
+It's your key in your own local config file — keep that file private and never commit it. Without a key, `semantic_search` is simply disabled; the other fifteen tools are unaffected.
+
+Keep the corpus fresh by re-running `lovspor fetch-corpus` (the engine re-syncs daily at 04:00 UTC); the `corpus_status` tool tells the assistant when your clone has drifted.
+
+> **Coming soon:** once `lovspor` is published to PyPI, the `--from git+https://…` prefix drops — both commands collapse to plain `uvx lovspor fetch-corpus` and `uvx lovspor mcp …`. Tracked in [`docs/roadmap.md`](docs/roadmap.md).
+
+See [`docs/mcp.md`](docs/mcp.md) for the full setup guide, all sixteen tools documented with examples (`get_law`, `get_law_at`, `list_law_versions`, `diff_law_versions`, `get_section`, `list_sections`, `get_law_history`, `list_recent_changes`, `search_laws`, `search_body`, `semantic_search`, `validate_citation`, `verify_quote`, `get_eu_basis`, `search_eu_implementations`, `corpus_status`), troubleshooting, and limitations. The binary embedding format that powers `semantic_search` is documented in [`docs/embeddings.md`](docs/embeddings.md).
+
+Persona-driven offline evals for the MCP tool surface live in [`evals/`](evals/). Run them with `uv run lovspor-eval`.
+
+## Sources
+
+- `https://api.lovdata.no/v1/publicData/get/gjeldende-lover.tar.bz2` — current Norwegian laws
+- `https://api.lovdata.no/v1/publicData/get/gjeldende-sentrale-forskrifter.tar.bz2` — current central regulations
+
+Data is licensed under [Norsk lisens for offentlige data (NLOD) 2.0](https://data.norge.no/nlod/no/2.0/).
+
+## License
+
+The engine code in this repository is licensed under MIT. See [LICENSE](LICENSE).
+
+The legal text produced by this engine is published in the [`lovverk`](https://github.com/bartoszkobylinski/lovverk) repository under NLOD 2.0, with attribution to Lovdata.
+
+## Related work
+
+- [`cloveras/lovdata2`](https://github.com/cloveras/lovdata2) — JSON tooling and MCP server for the same Lovdata public data. `lovspor` is complementary, focused on Markdown rendering, Git-based change tracking, and an MCP server scoped to the `lovverk` corpus shape.

lovspor-0.2.0/README.md

@@ -0,0 +1,96 @@
+# lovspor
+
+Norwegian law change tracker. Engine that produces the [`lovverk`](https://github.com/bartoszkobylinski/lovverk) corpus from Lovdata's public-data API and serves it to AI assistants over MCP (Model Context Protocol).
+
+## Status
+
+**Production.** A scheduled GitHub Actions workflow runs daily at 04:00 UTC, pulls the latest tarballs from Lovdata, classifies each document as new / updated / renamed / removed, renders the changes to Markdown, and pushes the diff to `lovverk` as conventional-commit history. The corpus mirrors close to **6 000 acts** (Norwegian *lover* and central *forskrifter*), each with a structured per-act change history under `<dataset>/history/<slug>.json`. The exact live count is always available from the MCP `corpus_status` tool.
+
+Sprint 9 (MERGED 2026-05-06) added per-section embeddings to the corpus and a four-layer anti-hallucination story to the MCP surface: `semantic_search` (cosine over embeddings), `verify_quote` (verbatim-citation guard), validated `cross_references` on `get_section`, and `validate_citation` as the off-ramp for ambiguous citations.
+
+See [`docs/decisions.md`](docs/decisions.md) for the full architecture and design rationale.
+
+## MCP server
+
+`lovspor` ships a stdio MCP server that exposes the `lovverk` corpus to AI assistants — Claude Desktop, Claude Code, or any MCP client. Once configured, you can ask *"what changed in Skatteloven this year?"* or *"are there forskrifter about jernbane?"* and the assistant answers from the live corpus instead of stale training data.
+
+**Setup — three steps:**
+
+1. **Install [`uv`](https://docs.astral.sh/uv/)** (it provides `uvx`, which runs `lovspor` on demand — no manual install).
+
+2. **Fetch the corpus.** One command shallow-clones the legal text to the default cache (`~/.cache/lovverk`):
+
+   ```bash
+   uvx --from "git+https://github.com/bartoszkobylinski/lovspor.git" lovspor fetch-corpus
+   ```
+
+   Re-run it any time to update — it reports `cloned`, `updated`, or `unchanged`.
+
+3. **Register the server.** `lovspor mcp` finds that cache automatically. With Claude Code:
+
+   ```bash
+   claude mcp add lovverk -- \
+     uvx --from "git+https://github.com/bartoszkobylinski/lovspor.git" lovspor mcp
+   ```
+
+   Or add it to your client config directly — Claude Desktop's `claude_desktop_config.json`, or `~/.claude.json` for Claude Code:
+
+   ```jsonc
+   {
+     "mcpServers": {
+       "lovverk": {
+         "command": "uvx",
+         "args": [
+           "--from", "git+https://github.com/bartoszkobylinski/lovspor.git",
+           "lovspor", "mcp"
+         ]
+       }
+     }
+   }
+   ```
+
+Restart the client and `lovverk` appears in its MCP list. Fifteen of the sixteen tools work immediately — no key, and no network access beyond your local corpus clone.
+
+**Optional — enable `semantic_search`.** The one search-by-meaning tool needs an OpenAI API key: it embeds *your query* at call time (the corpus vectors ship pre-computed, so you never re-embed the corpus yourself). Bring your own key via the server's `env`:
+
+```jsonc
+{
+  "mcpServers": {
+    "lovverk": {
+      "command": "uvx",
+      "args": [
+        "--from", "git+https://github.com/bartoszkobylinski/lovspor.git",
+        "lovspor", "mcp"
+      ],
+      "env": { "OPENAI_API_KEY": "sk-...your-own-key..." }
+    }
+  }
+}
+```
+
+It's your key in your own local config file — keep that file private and never commit it. Without a key, `semantic_search` is simply disabled; the other fifteen tools are unaffected.
+
+Keep the corpus fresh by re-running `lovspor fetch-corpus` (the engine re-syncs daily at 04:00 UTC); the `corpus_status` tool tells the assistant when your clone has drifted.
+
+> **Coming soon:** once `lovspor` is published to PyPI, the `--from git+https://…` prefix drops — both commands collapse to plain `uvx lovspor fetch-corpus` and `uvx lovspor mcp …`. Tracked in [`docs/roadmap.md`](docs/roadmap.md).
+
+See [`docs/mcp.md`](docs/mcp.md) for the full setup guide, all sixteen tools documented with examples (`get_law`, `get_law_at`, `list_law_versions`, `diff_law_versions`, `get_section`, `list_sections`, `get_law_history`, `list_recent_changes`, `search_laws`, `search_body`, `semantic_search`, `validate_citation`, `verify_quote`, `get_eu_basis`, `search_eu_implementations`, `corpus_status`), troubleshooting, and limitations. The binary embedding format that powers `semantic_search` is documented in [`docs/embeddings.md`](docs/embeddings.md).
+
+Persona-driven offline evals for the MCP tool surface live in [`evals/`](evals/). Run them with `uv run lovspor-eval`.
+
+## Sources
+
+- `https://api.lovdata.no/v1/publicData/get/gjeldende-lover.tar.bz2` — current Norwegian laws
+- `https://api.lovdata.no/v1/publicData/get/gjeldende-sentrale-forskrifter.tar.bz2` — current central regulations
+
+Data is licensed under [Norsk lisens for offentlige data (NLOD) 2.0](https://data.norge.no/nlod/no/2.0/).
+
+## License
+
+The engine code in this repository is licensed under MIT. See [LICENSE](LICENSE).
+
+The legal text produced by this engine is published in the [`lovverk`](https://github.com/bartoszkobylinski/lovverk) repository under NLOD 2.0, with attribution to Lovdata.
+
+## Related work
+
+- [`cloveras/lovdata2`](https://github.com/cloveras/lovdata2) — JSON tooling and MCP server for the same Lovdata public data. `lovspor` is complementary, focused on Markdown rendering, Git-based change tracking, and an MCP server scoped to the `lovverk` corpus shape.

lovspor-0.2.0/docs/architecture.md

@@ -0,0 +1,190 @@
+# Architecture
+
+`lovspor` is the **engine**. It downloads Lovdata public-data tarballs, extracts
+and normalizes XML, hashes the normalized XML to detect changes, deterministically
+renders Markdown, and commits only the changed laws into the sibling
+[`lovverk`](https://github.com/bartoszkobylinski/lovverk) corpus repo. It also
+ships a read-only **MCP server** that serves that corpus to AI clients.
+
+Legal text never lives in this repo — only the code that produces it. See
+[`decisions.md`](decisions.md) for the *why* behind each choice; this document is
+the *how the code is organized and flows* view.
+
+## High-level flow
+
+```
+Lovdata public API
+        │  GET /v1/publicData/list, /get/{filename}
+        ▼
+download tar.bz2  ──►  data/cache/  (gitignored)
+        │
+        ▼
+extract XML members (in-memory, never extractall)
+        │
+        ▼
+SHA-256 on normalized (C14N) XML          ◄── the change-detection key
+        │
+        ▼
+diff against manifest.json (prior state)
+        │
+        ▼
+render changed docs → Markdown  (+ optional OpenAI embeddings sidecar)
+        │
+        ▼
+write to the lovverk clone (sibling working tree)
+        │
+        ▼
+git commit per changed document  (+ a final manifest/index/history commit)
+```
+
+The rationale-level version of this diagram, including commit modes, lives in
+[`decisions.md` §5](decisions.md).
+
+## Module map
+
+Grouped by subpackage under `src/lovspor/`. Subpackage `__init__.py` files are
+empty package markers except `embeddings/__init__.py`, which is a re-export
+facade.
+
+### Top level — `src/lovspor/`
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `cli.py` | Typer CLI entry point; loads `.env` in the group callback before Typer resolves `envvar=` options. | `app`; commands `info`, `seed`, `sync`, `mcp` |
+| `settings.py` | Runtime config resolved from env / `.env`; frozen Pydantic model; paths resolved absolute. | `Settings.from_env()`, `load_env()` |
+| `errors.py` | Exception hierarchy so callers never catch bare `Exception`. | `LovsporError` + `NetworkError`, `ParseError`, `RenderError`, `ExtractionError`, `ConfigError`, `CorpusStateError`, `MassRemovalError` |
+| `retry.py` | Dependency-free exponential-backoff retry helper. | `retry_with_backoff(...)` |
+| `history.py` | Per-act change history from `git log --follow --numstat`; writes `history/<slug>.{json,md}`. | `extract_history()`, `write_history()`, `render_history_markdown()` |
+| `timetravel.py` | Time-machine: a doc's text as of a past date via `git log --follow` + `git show <sha>:<path>`. | `get_law_at_revision(...)`, `resolve_law_at_revision(...)` |
+| `mcp.py` | Stdio MCP server exposing 16 read-only tools over a local `lovverk` clone. | `serve()`, `build_server()`, `CorpusReader` |
+
+### `sources/` — Lovdata API boundary
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `sources/lovdata.py` | HTTP client for `api.lovdata.no/v1/publicData`: list catalogue, stream-download archives with sha256 + size verify, atomic `.part` rename, retry on 429/5xx. | `LovdataClient`, `LovdataArchive`, `DownloadResult` |
+
+### `extraction/` — tar safety
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `extraction/tarball.py` | Safe in-memory iteration of XML members from a `.tar.bz2`. Never `extractall`/`extract`; validates member names. | `iter_tarball_xml()`, `TarballMember` |
+
+### `parsing/` — normalization + hashing
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `parsing/xml_normalizer.py` | Hardened lxml parser (XXE / billion-laughs safe) + W3C C14N canonicalization + SHA-256 change-detection hash. | `safe_parser()`, `canonicalize_xml()`, `hash_normalized_xml()` |
+
+### `rendering/` — XML → Markdown (deterministic)
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `rendering/markdown_renderer.py` | Deterministic Lovdata-HTML → Markdown body; escapes Markdown specials; raises `RenderError` on dropped text. | `render_markdown()` |
+| `rendering/document.py` | Extract doc metadata from `<header><dl>`; build the full front-matter model incl. EU basis (CELEX). | `build_frontmatter()`, `extract_xml_metadata()`, `LegalDocumentFrontMatter` |
+| `rendering/frontmatter.py` | Deterministic minimal YAML serializer (no PyYAML). | `serialize_frontmatter()` |
+| `rendering/slug.py` | Human-readable filename slug (short_title→title→doc_id), UTF-8 byte cap, deterministic collision resolution. | `derive_slug()`, `resolve_collisions()` |
+
+### `storage/` — manifest ledger
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `storage/manifest.py` | The change-detection ledger; deterministic sorted-JSON read/write; versioned schema. | `read_manifest()`, `write_manifest()`, `Manifest`, `ManifestRecord` |
+
+### `sync/` — orchestration + IO + git
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `sync/orchestrator.py` | **The pipeline.** `run_sync()` composes download→extract→hash→diff→render→embed→commit; handles renames, tombstones, the mass-removal guard, and one-time migrations. | `run_sync()`, `SyncReport` |
+| `sync/change_detector.py` | Pure classifier: upstream hashes vs prior manifest → new/changed/removed/unchanged. | `detect_changes()`, `ChangeSet` |
+| `sync/document_io.py` | Compose renderer + front matter + filesystem IO; dataset↔subdir mapping; write/delete docs; generate `INDEX.md`. | `render_full_document()`, `write_document()`, `generate_index()` |
+| `sync/git_commit.py` | Thin `git` CLI wrappers (list args, never `shell=True`). | `add()`, `commit()`, `has_staged_changes()` |
+
+### `embeddings/` — semantic-search vectors
+
+| Module | Responsibility | Key public API |
+|---|---|---|
+| `embeddings/model.py` | OpenAI `text-embedding-3-large` client (sync httpx), tiktoken-aware truncation, batching + retry, L2-normalized output; token chunker. | `OpenAIEmbedder`, `EmbeddingModel` (Protocol), `split_to_token_chunks()` |
+| `embeddings/sections.py` | Split rendered Markdown into `### § N-M.` embedding units. | `iter_sections()`, `EmbeddingSection` |
+| `embeddings/quantize.py` | float32 ↔ int8 linear quantization with per-batch scale. | `quantize_int8()`, `dequantize_int8()` |
+| `embeddings/store.py` | Binary `<slug>.bin` (LSPE) format read/write; atomic temp-rename. | `write_embeddings()`, `read_embeddings()`, `EmbeddingFile` |
+| `embeddings/search.py` | In-memory int8 top-K cosine index; chunked matmul; deterministic tie-break; dedup by `(slug, section_id)`. | `EmbeddingIndex`, `SearchHit` |
+
+The binary embedding format is documented in [`embeddings.md`](embeddings.md); the
+data models these modules exchange are in [`data-model.md`](data-model.md).
+
+## The sync pipeline
+
+There is **one orchestrator**: `run_sync(settings)` in `sync/orchestrator.py`.
+Both the `seed` and `sync` CLI commands call it — they are the same pipeline; a
+missing manifest just makes the change detector classify everything as new.
+
+In order:
+
+1. **CLI entry** (`cli.py`) — the group callback runs `load_env()` first, then the
+   command builds `Settings.from_env()` and calls `run_sync()`.
+2. **Preconditions** — the corpus must be a git repo; a **dirty worktree aborts**
+   with `CorpusStateError` (crash residue would otherwise be misread as "nothing
+   changed"). The prior `manifest.json` is read here.
+3. **One-time migrations** — Sprint 5 history backfill, Sprint 8 `eu_basis`
+   re-render, and Sprint 9 embeddings backfill (the last only if an OpenAI key is
+   set). Each fires once and emits its own commit(s) before regular work.
+4. **Collect upstream** — download each tracked dataset tarball
+   (`gjeldende-lover`, `gjeldende-sentrale-forskrifter`) into
+   `data/cache/archives/`, iterate XML members, extract metadata, derive the base
+   slug, and **compute `hash_normalized_xml(member)`**. Slug collisions are
+   resolved **per dataset**.
+5. **Change detection** — `detect_changes(upstream_hashes, prior)` returns disjoint
+   sorted `new/changed/removed/unchanged`; renames are then identified among
+   unchanged-content docs whose slug/path moved. **No-op fast path:** if nothing is
+   new/changed/removed/renamed, return early *without* rewriting the manifest or
+   committing — the "nothing to do → zero commits" contract.
+6. **Mass-removal guard** — abort with `MassRemovalError` if any single dataset
+   (≥ 20 current docs) would lose more than `max_removal_ratio` (default 10%).
+7. **Carry forward** — unchanged records are copied verbatim (preserving
+   `last_seen` so the manifest stays byte-identical); tombstones (`status="removed"`)
+   are carried so removals stay permanent.
+8. **Render + embed** — for each new/changed/renamed doc, write the `.md`
+   (front matter + body) and, when an embedder is available, the
+   `<dataset>/embeddings/<slug>.bin` sidecar. A two-phase write-all-then-delete
+   sequence prevents path-cascade corruption when slugs shuffle between docs.
+9. **Commit** — mode-aware (`git_commit_mode`, default `per-document`): one commit
+   per add/update/rename/remove, then history is extracted (`git log --follow` needs
+   those commits to exist), then one final commit bundling manifest + `INDEX.md` +
+   history. The `single` mode and migration paths use one bulk commit plus a history
+   follow-up (history needs the docs commit first).
+10. **Return** a `SyncReport` of new/changed/removed/unchanged counts.
+
+## Key invariants (enforced in code)
+
+| Invariant | Where | How |
+|---|---|---|
+| **Deterministic rendering** (same XML → byte-identical Markdown) | `rendering/markdown_renderer.py`, `rendering/frontmatter.py` | No wall-clock in body; custom deterministic YAML serializer; front-matter key order = model field order. Manifest and `.bin` writes are likewise byte-deterministic. |
+| **Hash on normalized XML** (never on Markdown/HTML) | `parsing/xml_normalizer.py:hash_normalized_xml` | SHA-256 of C14N-canonicalized XML, called only on raw XML bytes. A documented gap: `remove_blank_text=True` makes inter-element whitespace hash-invisible (accepted — [`decisions.md` §14](decisions.md)). |
+| **Safe XML parser** (no XXE, billion-laughs, network, or unbounded tree) | `parsing/xml_normalizer.py:safe_parser` | `resolve_entities=False, no_network=True, huge_tree=False, remove_comments=True`. Unresolved custom entities raise `ParseError`. |
+| **Tar safety / CVE-2007-4559** | `extraction/tarball.py` | Never `extractall`/`extract`; reads members into memory via `extractfile()`; rejects null bytes, absolute names, and `..` components. |
+| **Mass-removal guard** | `sync/orchestrator.py:_guard_mass_removal` | Per-dataset abort if removed/total exceeds `max_removal_ratio`; guards against a truncated/empty upstream tarball wiping the corpus. |
+| **Tombstones** (removals are permanent) | `sync/orchestrator.py` (`_tombstone`, `_carry_tombstones`) | Removed docs keep their record with `status="removed"`, carried forward every sync. |
+| **No shell injection** | `sync/git_commit.py`, `history.py`, `timetravel.py` | Every `subprocess.run` uses list args, never `shell=True`. |
+
+## External boundaries
+
+- **Lovdata public-data API** — reached only from `sources/lovdata.py`
+  (`GET /list`, `GET /get/{filename}`). The only outbound HTTP to Lovdata; no HTML
+  scraping anywhere.
+- **OpenAI API** — reached only from `embeddings/model.py`
+  (`text-embedding-3-large`, 3072-dim). On the engine side it is optional (no key →
+  embeddings skipped, Markdown still produced). On the MCP side it powers only
+  `semantic_search`; see the privacy note in [`mcp.md`](mcp.md).
+- **git / the `lovverk` corpus** — `sync/git_commit.py` (write path), plus
+  `history.py`, `timetravel.py`, and `mcp.py` (read paths) shell out to `git`
+  against the local clone. Nothing in the engine code pushes to GitHub — the
+  scheduled workflow does that.
+
+## MCP server
+
+`lovspor mcp` builds a `FastMCP("lovverk")` server (`mcp.py`) transported over
+**stdio** with 16 read-only tools. A `CorpusReader` reads `manifest.json` plus the
+Markdown / `.bin` files from the local clone, caching in memory and dropping caches
+when `manifest.json`'s mtime changes so a `git pull` under a long-lived server is
+picked up. Full tool reference, setup, and limitations are in [`mcp.md`](mcp.md).