rfclib 0.2.0__tar.gz

rfclib-0.2.0/.gitignore

@@ -0,0 +1,20 @@
+__pycache__/
+*.pyc
+.venv/
+dist/
+build/
+*.egg-info/
+data/
+*.db
+*.db-wal
+*.db-shm
+
+export-packs/
+export-formats/
+android-reader/local.properties
+android-reader/.gradle/
+android-reader/build/
+android-reader/app/build/
+
+# the reader UI lives on the reader-ui branch; ignore local leftovers
+reader-ui/

rfclib-0.2.0/CHANGELOG.md

@@ -0,0 +1,39 @@
+# Changelog
+
+## 0.2.0 (unreleased)
+
+First release intended for PyPI
+(`uvx --from "rfclib[mcp,semantic]" rfclib mcp`).
+
+- **Optional features.** The core package (CLI, library, keyword search)
+  has four dependencies (~9 MB); the MCP server (`rfclib[mcp]`), semantic
+  search (`rfclib[semantic]`) and ZIM export (`rfclib[zim]`) are extras,
+  loaded lazily and reported by `rfclib status`. The reader UI and the
+  Docker images moved to the `reader-ui` branch.
+
+- **Library lifecycle.** Data lives in the per-user platform data directory
+  (or `$RFCLIB_HOME`), never the working directory. `rfclib init` sets up
+  the catalog (RFC index, errata, subject tree) in seconds; each RFC is
+  fetched from rfc-editor.org and indexed the first time it is read;
+  `rfclib init --content current|all` prefetches everything. `rfclib status`
+  reports what the library holds. `RFCLIB_NETWORK=auto|on-demand|offline`
+  controls fetching. The library is portable (relative file paths), safe
+  to share between processes, and loads its embedding model offline.
+- **MCP server: nine tools** (`search`, `read`, `outline`, `grep`, `cite`,
+  `links`, `info`, `browse`, `errata`), compact plain text, one reference
+  grammar (`RFC9000§9.6¶2`) that every tool accepts and every page footer
+  emits, prompts (`/rfclib:explain`, `checklist`, `current`), an
+  `rfc://{ref}` resource template and completions.
+- **Structure.** RFC 8650+ from RFCXML (exact sections and paragraph ids);
+  older RFCs segmented from their own Table of Contents (heading recall
+  59% -> 98%). Section-precise citation links for every era.
+- **Search.** Hybrid BM25 + embeddings + document titles, plus an exact
+  identifier index (`preferred_address`, `SETTINGS_INITIAL_WINDOW_SIZE`)
+  and acronym-aware title matching. Status-aware: obsolete documents are
+  flagged with their current replacement.
+- **Errata** from the RFC Editor bulk API, shown per section.
+
+## 0.1.0
+
+Initial local library: RFC Editor sync, SQLite metadata and full-text
+index, CLI, reader UI, content packs.

rfclib-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Justin Capella
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rfclib-0.2.0/NOTICE

@@ -0,0 +1,24 @@
+rfclib
+Copyright (c) 2026 Justin Capella. Licensed under the MIT License (LICENSE).
+
+RFC content is not part of this software's license
+----------------------------------------------------
+rfclib downloads documents of the RFC Series and their metadata from the
+RFC Editor (https://www.rfc-editor.org) at the user's request and stores
+them locally. Those documents are published by the IETF Trust under the
+IETF Trust Legal Provisions (https://trustee.ietf.org/license-info), not
+under this software's license. In particular, RFCs may be copied and
+distributed in full but not modified outside the IETF Standards Process;
+code components in documents published on or after November 10, 2008 are
+available under the Revised BSD License. Documents published before 1986
+may carry different terms. See the IETF Trust FAQ
+(https://trustee.ietf.org/about/faq).
+
+The test fixtures in tests/fixtures/library are unmodified copies of a few
+RFCs and trimmed copies of RFC Editor metadata, included for offline tests
+under the same terms.
+
+Subject taxonomy and errata are fetched from the RFC Editor's published
+pages and API (rfc-editor.org/subjects, rfc-editor.org/api/v1/errata.json).
+The embedding model (minishlab/potion-base-8M) is downloaded from Hugging
+Face under its own license when semantic search is first built.

rfclib-0.2.0/PKG-INFO

@@ -0,0 +1,511 @@
+Metadata-Version: 2.5
+Name: rfclib
+Version: 0.2.0
+Summary: The RFC series as a local library and MCP server: section-precise search, paging by citation, status, errata and citations
+Project-URL: Homepage, https://github.com/b1tninja/rfclib
+Project-URL: Source, https://github.com/b1tninja/rfclib
+Project-URL: Issues, https://github.com/b1tninja/rfclib/issues
+Project-URL: Changelog, https://github.com/b1tninja/rfclib/blob/master/CHANGELOG.md
+Author-email: Justin Capella <justincapella@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: agents,ietf,llm,mcp,model-context-protocol,rfc,rfc-editor,search,standards
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.27
+Requires-Dist: platformdirs>=4.2
+Requires-Dist: rich>=13
+Requires-Dist: typer>=0.12
+Provides-Extra: all
+Requires-Dist: libzim>=3.7; extra == 'all'
+Requires-Dist: mcp>=1.2; extra == 'all'
+Requires-Dist: model2vec>=0.3; extra == 'all'
+Requires-Dist: sqlite-vec; extra == 'all'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2; extra == 'mcp'
+Provides-Extra: semantic
+Requires-Dist: model2vec>=0.3; extra == 'semantic'
+Requires-Dist: sqlite-vec; extra == 'semantic'
+Provides-Extra: zim
+Requires-Dist: libzim>=3.7; extra == 'zim'
+Description-Content-Type: text/markdown
+
+# rfclib — Local RFC Library + MCP Server
+
+A local, self-maintaining library of **every published RFC** (plus BCP/STD/FYI),
+sourced exclusively from the **official RFC Editor infrastructure**
+([rfc-editor.org](https://www.rfc-editor.org/) — the authoritative publisher of
+the RFC Series for all five streams, operating under the IETF Administration
+LLC; see RFC 9280, RFC 4714, RFC 7841).
+
+Everything lives in **one SQLite file** and is exposed to AI agents over **MCP**,
+and to scripts over a CLI. Fully offline after the initial sync: no API keys,
+no external services; embeddings run locally on CPU.
+
+## Install
+
+```bash
+uvx rfclib init      # catalog: RFC index, errata, subjects (~1 min)
+claude mcp add rfclib -- uvx --from "rfclib[mcp,semantic]" rfclib mcp
+```
+
+The core package is small (CLI, library, keyword search, ~9 MB with its
+dependencies); heavier features are extras:
+
+| Extra | Adds | Size |
+|---|---|---|
+| `rfclib[mcp]` | the MCP server (`rfclib mcp`) | ~29 MB |
+| `rfclib[semantic]` | embedding search fused into hybrid search | ~69 MB |
+| `rfclib[zim]` | Kiwix ZIM export | native wheel |
+| `rfclib[all]` | all of the above | |
+
+`rfclib status` shows which are installed; a missing one says how to add it,
+and search without `semantic` uses keywords, identifiers and titles.
+
+[`uvx`](https://docs.astral.sh/uv/) fetches rfclib into an isolated, cached
+environment (and a suitable Python if needed); `pip install rfclib` works
+too. RFCs are then fetched from rfc-editor.org the first time they are read;
+`uvx rfclib init --content current` prefetches every current RFC so
+full-text section search covers them all (~10 min, ~1.5 GB).
+
+Requirements: Python 3.12+ on Windows, macOS or Linux. Semantic search
+(`rfclib[semantic]`) uses a ~30 MB local embedding model (`minishlab/potion-base-8M`,
+[model2vec](https://github.com/MinishLab/model2vec)) and `sqlite-vec`; on a
+Python whose sqlite3 cannot load extensions, search is keyword-only.
+
+## Sources (first-party only)
+
+| What | Source |
+|---|---|
+| Bulk mirror / periodic re-mirror | `rsync.rfc-editor.org::rfcs` (official rsync modules, see [download docs](https://www.rfc-editor.org/series/rfc-download/)) |
+| Master manifest (all docs + obsoletes/updates/alias graph) | `https://www.rfc-editor.org/rfc-index.xml` |
+| Per-document content | `https://www.rfc-editor.org/{rfc,bcp,std,fyi}/<id>.{txt,html,xml,pdf}` |
+| New-publication feed | `https://www.rfc-editor.org/rfcrss.xml` |
+| Errata (bulk, `rfclib init`; per-doc fallback) | `https://www.rfc-editor.org/api/v1/errata.json`, `https://errata.rfc-editor.org/` |
+
+Every local file is recorded with source URL + SHA-256 for provenance.
+
+## From source
+
+```bash
+git clone https://github.com/b1tninja/rfclib && cd rfclib
+uv sync --group dev              # .venv with rfclib (editable) + pytest
+uv run pytest -q                 # offline tests, sample library
+
+rfclib init                     # catalog: RFC index, errata, subject tree
+rfclib status                   # where the library lives, what it holds
+```
+
+That is enough to use the MCP server: status, obsolete chains, document
+search, browsing and errata cover every RFC, and each RFC is fetched from
+rfc-editor.org, segmented and indexed the first time it is read (about a
+second). To make full-text section search cover everything up front:
+
+```bash
+rfclib init --content current   # + every current RFC, indexes, vectors (~10 min)
+rfclib init --content all       # + obsolete RFCs too
+```
+
+Later updates are incremental (`rfclib sync && rfclib index build && rfclib
+index embed`). The official rsync service still works for a bulk mirror:
+`scripts\mirror.ps1`, then `rfclib scan`.
+
+```powershell
+# Explore
+.venv\Scripts\rfclib.exe hsearch "proving address ownership"  # hybrid — recommended
+.venv\Scripts\rfclib.exe search "congestion control"          # titles/abstracts
+.venv\Scripts\rfclib.exe search "KeyUpdate" --body            # BM25, sections
+.venv\Scripts\rfclib.exe vsearch "proving address ownership"  # semantic only
+.venv\Scripts\rfclib.exe toc rfc9000                          # outline + line ranges
+.venv\Scripts\rfclib.exe get rfc9000 --start-line 6858 --lines 40
+.venv\Scripts\rfclib.exe rels rfc2616
+.venv\Scripts\rfclib.exe resolve bcp14
+.venv\Scripts\rfclib.exe subjects get quic               # curated topic listing
+.venv\Scripts\rfclib.exe status
+```
+
+## CLI reference
+
+| Command | Purpose |
+|---|---|
+| `manifest` | Fetch/parse the official rfc-index.xml into the DB (metadata + relations) |
+| `sync` | Incremental HTTP sync of missing docs (`--include-obsolete`, `--formats txt,html`, `--concurrency`) |
+| `scan` | Register files from an rsync-produced mirror into the DB |
+| `status` | Library + index statistics |
+| `search <q> [--body]` | Titles/abstracts/keywords; `--body` = BM25 over sections (raw FTS5 syntax allowed) |
+| `vsearch <q>` | Semantic (embedding) search over sections — needs `index embed` |
+| `hsearch <q>` | Hybrid BM25 + embeddings + title coverage fused (RRF) — the recommended default |
+| `get <id>` | Print a document (`--start-line N --lines N`, `--fmt txt\|html\|xml`) |
+| `meta <id>` / `rels <id>` / `resolve <id>` | Bibliographic record / relation graph / BCP→RFC |
+| `toc <id>` | Section outline with exact line ranges (feed into `get --start-line`) |
+| `refs [<id>] [--build]` | Citation graph mined from txt: what an RFC cites / cites it, with §-level anchors (`--build` = incremental mine, ~50 s full corpus) |
+| `errata <id> [--refresh]` | Verified-errata overlay from the RFC Editor errata service (on-demand fetch + local cache; `--all` for other statuses) |
+| `subjects sync [--force] [--limit N]` | Mirror the RFC Editor subject taxonomy (~640 pages, one polite crawl) |
+| `subjects list [filter] [--descriptions]` | Show the subject tree (or a filtered slice), parents before children |
+| `subjects get <name> [--limit N]` | Documents inside one subject incl. descendants, newest first |
+| `index build\|embed\|status` | Maintain retrieval indexes (`embed --limit N`, `--model <id>`) |
+| `packs list [--collections-dir D]` | Show collection manifests and resolved doc counts |
+| `packs build <id...> \| --all` | Export `.rfcpack` zips (embedded FTS5 search, `--formats`, `--out`, `--no-search`) |
+| `packs export <id...> \| --all` | Export for third-party readers: EPUB e-books + static HTML mirror + Kiwix ZIM + Dash docset (`--formats epub,html,zim,docset`, `--link-refs none\|internal\|external`) |
+| `packs catalog [--out D]` | Rebuild `catalog.json` (name/size/sha256 per pack) |
+| `packs serve [--dir D] [--port 8100]` | LAN HTTP distribution for the Android reader |
+| `mcp` | Run the MCP server on stdio |
+
+All searches accept `--include-obsolete` (obsolete docs are hidden by design —
+an agent reading RFC 2616 in 2026 is a failure mode).
+
+## Retrieval indexes (full-text + semantic)
+
+Documents are chunked by section (heuristic parser for RFC txt, exact source
+line ranges kept; long sections split into ~2.4k-char parts with per-part
+line ranges). All three search modes live in the same SQLite file: BM25 via
+FTS5 (`search --body`), KNN via sqlite-vec (`vsearch`), and hybrid
+(`hsearch`). Chunk IDs are content-addressed, so re-chunking preserves
+embeddings for every section whose text didn't change (a chunker tweak
+re-embeds only the diff, e.g. 15k of 298k chunks in ~11 s).
+
+`hsearch` fuses up to four lists with diversified, weighted RRF (the fourth,
+an exact-identifier index, joins when the query contains tokens such as
+`preferred_address`, `SETTINGS_INITIAL_WINDOW_SIZE` or `X25519`, weighted by
+how rare they are; see docs/retrieval-design.md): section-level
+BM25 (terms OR'd + proximity-boosted), the embedding KNN list, and a
+document-level list ranked by how much of the query the doc's own **title**
+carries (IDF-weighted term coverage — the spec titled "TCP Congestion
+Control" beats one that merely mentions "NewReno"). Each doc appears once
+per list before fusion, chunks of the same doc corroborate each other, and
+standards-track/BCP docs get a status boost while References/boilerplate
+sections are penalized. Without `index embed` it still works (BM25 + title).
+
+Full design rationale, measured trade-offs, and FTS5/chunker gotchas:
+[docs/retrieval-design.md](docs/retrieval-design.md).
+
+### Accuracy loop (regression-test retrieval changes)
+
+```powershell
+python scripts\eval_retrieval.py     # golden queries -> expected doc, top-5
+```
+
+A set of hand-picked (query → canonical-RFC) pairs run through `hsearch`.
+Any change to chunking, FTS query shape, or fusion should be built,
+re-embedded, and re-evaluated with this script before shipping
+(currently 10/12 in top-5; slowest cold query ~1.8 s — see
+[docs/retrieval-design.md §5](docs/retrieval-design.md) for the two known-hard misses).
+
+## MCP server
+
+Any MCP client launches the server as a subprocess over stdio. With `uvx`
+there is nothing to install first (`--from "rfclib[mcp,semantic]"` brings the
+MCP and semantic-search extras); set `RFCLIB_HOME` in `env` only to use a
+library somewhere other than the default per-user data directory.
+
+**Claude Code**
+
+```bash
+claude mcp add rfclib -- uvx --from "rfclib[mcp,semantic]" rfclib mcp
+```
+
+**Claude Desktop** (`claude_desktop_config.json`) and **Cursor**
+(`~/.cursor/mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "rfclib": {
+      "command": "uvx",
+      "args": ["--from", "rfclib[mcp,semantic]", "rfclib", "mcp"]
+    }
+  }
+}
+```
+
+**VS Code** (`.vscode/mcp.json`):
+
+```json
+{
+  "servers": {
+    "rfclib": {
+      "type": "stdio",
+      "command": "uvx",
+      "args": ["--from", "rfclib[mcp,semantic]", "rfclib", "mcp"]
+    }
+  }
+}
+```
+
+GUI clients don't always see your shell's `PATH`; if `uvx` isn't found,
+use its absolute path (`which uvx` / `where uvx`). From a source checkout,
+point `command` at the venv's `rfclib` executable instead (this repository's
+`.mcp.json` does that for development).
+
+### Tools (9)
+
+One job per tool, all answering in compact plain text; every tool's main
+argument is `ref` (only `search` takes `query`), all addressed by
+**refs**, which are also citations. Every page of `read` ends with
+`next: <ref>` to continue from:
+`RFC9000` (abstract + outline), `RFC9000§9.6` (section incl. subsections),
+`RFC9000§9.6¶2` (paragraph), `RFC9000#table-2` (RFCXML block id),
+`RFC9000:2964-2979` (txt lines); `BCP14`/`STD7` resolve to their RFCs.
+Errors come back as `isError` results with the fix in the message; all
+tools are annotated read-only; `search` and `read` carry
+`_meta["anthropic/alwaysLoad"]` so they stay visible when a client defers
+tools behind tool search.
+
+| Tool | Args | What it returns |
+|---|---|---|
+| `search` | `query`, `scope`, `mode`, `subject`, `status`, `limit`, `detail` | Paragraph-precise hits with snippets. `scope=sections` (default) or `documents` (titles/abstracts); `mode=hybrid` (BM25 + embeddings), `exact` (FTS5 syntax) or `semantic` |
+| `read` | `ref`, `max_tokens`, `view`, `lang`, `format` | **The way to read text.** Pages cut at paragraph boundaries, page furniture stripped, `[§9.6¶2]` markers, footer `next:`/`up:`/`after:` refs, status + errata in the header. `view=normative` (MUST/SHOULD statements), `code` (ABNF/figures, `lang=`), `refs`; `format=xml\|html\|json` for raw slices of other renderings |
+| `outline` | `ref`, `depth`, `contains` | One line per section with its line range (depth 2 by default; RFC 9000 ≈ 6k chars) |
+| `grep` | `ref`, `pattern` | Regex inside one document or section; hits labelled with paragraph refs + lines |
+| `cite` | `ref`, `style` | Checkable citation: label, status/obsolescence, anchor URL, errata status, span hash |
+| `links` | `ref`, `direction` | What a section cites / is cited by, section-precise (`RFC9000§8.1 → RFC9002 §6.2 [normative]`), plus reference-list edges; `related` for subject neighbours |
+| `info` | `ref`, `detail` | Status, obsoletes/updates graph with the **current** replacement (RFC2616 → RFC9110–9112), alias members, authors, subjects, errata summary, abstract; `detail=full` = JSON record |
+| `browse` | `subject`, `parent`, `contains`, `recent`, `offset` | RFC Editor subject tree one level at a time, a subject's documents, or the newest RFCs |
+| `errata` | `ref`, `status`, `section` | Verified (default), open or all errata with the corrected text; `RFC791§3.1` narrows to a section |
+
+**Prompts** (user-invoked slash commands in Claude Code):
+`/rfclib:explain <ref>`, `/rfclib:checklist <ref>` (normative statements →
+implementation checklist), `/rfclib:current <topic>` (which RFCs are
+current). **Resource template** `rfc://{ref}`: `@rfclib:rfc://RFC9112§6.3`
+attaches a section to a prompt. **Completions** for document ids and
+section numbers.
+
+**Agent guidance shipped in the repo** (`.claude/`): the `rfc-research`
+skill (workflow, ref grammar, recipes) and a `PreToolUse` hook
+(`scripts/hooks/guard_mirror.py`) that denies `Read`/`Grep`/`Glob` on
+`data/mirror/**` and points at the MCP call to use instead.
+
+Missing documents are fetched on demand from rfc-editor.org.
+
+### Subject views (RFC-Editor organization)
+
+The RFC Editor curates a ~640-subject tree (`web > http > http2`,
+`transport > quic`, …) and assigns every document to one or more
+subjects; it is published only as HTML at
+[rfc-editor.org/subjects/](https://www.rfc-editor.org/subjects/).
+`rfclib subjects sync` mirrors it (one polite crawl, a few minutes; HTML
+cached under `meta/subjects/` so re-parses are offline) into `subjects` +
+`doc_subjects` tables. On top of that: the `browse` /
+`links(direction='related')` MCP tools, `search(scope='documents',
+subject=...)`, subject tags in `info`, and the same via CLI
+(`rfclib subjects list|get`). This answers *"what RFCs exist about X"*
+and *"what else is like this RFC"* without a keyword query — the
+citation-graph view (`links`) stays available for normative context.
+
+## Configuration
+
+Environment variables (the same in a shell or an MCP client's `env`
+block):
+
+- `RFCLIB_HOME`: the library root. Default: the per-user data directory
+  (`%LOCALAPPDATA%
+fclib` on Windows, `~/.local/share/rfclib` or
+  `$XDG_DATA_HOME/rfclib` on Linux, `~/Library/Application Support/rfclib`
+  on macOS). Never the working directory, because MCP clients start
+  servers from arbitrary directories. Older versions defaulted to `./data`;
+  `rfclib status` points out such a library and how to keep using it.
+- `RFCLIB_NETWORK`: `auto` (default; fetch documents on demand),
+  `on-demand`, or `offline` (never touch the network; answer from what
+  is local, and say so).
+- `RFCLIB_AUTO_INIT=1`: let the MCP server initialize an empty library's
+  catalog in the background on first use (opt-in; off by default).
+
+The library is portable: file paths are stored relative to `mirror/`, so a
+library can be moved, copied to another OS, or mounted into a container.
+Several processes can share it (one MCP server per client session plus
+`rfclib sync`): SQLite runs in WAL mode with a busy timeout, downloads are
+written atomically, and bulk writers take a lock file.
+
+```
+<RFCLIB_HOME>/
+  rfclib.db     one SQLite file: docs, doc_relations, files (sha256 provenance),
+                chunks, chunks_fts + docs_fts (FTS5), chunks_vec (sqlite-vec),
+                embed_state (content-hashed embed coverage),
+                blocks + xrefs (paragraphs and section-level citations),
+                seg_quality (how each document was segmented), errata
+  meta/         rfc-index.xml (official manifest)
+  mirror/       document files in rfc-editor.org layout (rfc/, bcp/, ...)
+  models/       the embedding model (~30 MB), loaded offline
+```
+
+## Query tips
+
+- `hsearch`/`search` treat words as OR + proximity — natural phrasing
+  works ("does this spec allow sending packets early?"); no operators needed.
+- `search --body` and `search(mode='exact')` pass your query **raw to FTS5**:
+  quote phrases (`"KeyUpdate message"`), combine with `AND`/`OR`/`NOT`,
+  prefix with `*`. NEAR takes the comma form: `NEAR(tls handshake, 40)`
+  (`NEAR(tls handshake / 40)` is a syntax error).
+- Keep version tokens in queries (`TLS 1.3`, `802.11`): they are strong
+  discriminators and hybrid search preserves them unstemmed.
+- For a long RFC, `outline` first, then `read('RFC9000§9.6')` and follow
+  the `next:` ref — don't page through 8,000-line documents, and never point
+  an agent at the mirror files (its file reader truncates them).
+- RFCs since 8650 are indexed from their RFCXML source (`rfclib sync
+  --formats xml && rfclib index build`); older RFCs are segmented from
+  their own Table of Contents (`rfclib/index/txtseg.py`, quality per doc
+  in the `seg_quality` table). Either way you get exact sections,
+  paragraph refs, `view=normative`/`code` and section-level `links`.
+- `rfclib errata --sync-all` loads every erratum from the RFC Editor bulk
+  API once, so errata status is known offline for every document.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `vsearch`/`search(mode='semantic')` return nothing | Run `rfclib index embed` (`index status` shows chunk vs embedded counts) |
+| Hybrid results lack conceptual recall | Same — `hsearch` degrades to BM25+title without vectors |
+| Section titles look like ToC junk | Chunker heuristics improved since your index; `rfclib index build && rfclib index embed` (only the diff re-embeds) |
+| `database is locked` | An `index build`/`embed` is running elsewhere; they commit in batches — wait or stop the other run |
+| Expected old/obsolete RFC missing | It's obsoleted and hidden by default: add `--include-obsolete` / `include_obsolete: true` |
+| `sync` too slow / rsync unavailable (Windows) | `sync` is HTTP-based and cross-platform; `scripts\mirror.ps1` shells out to rsync under WSL for the initial bulk pull |
+
+## Content packs & Android reader
+
+RFCs are published rolling (~5–25/month, no official batches), so updates are
+distributed as **content packs**: a `.rfcpack` zip holds one curated
+collection — document files + a `pack.json` manifest (per-file sha256) + an
+offline SQLite **FTS5** index (`search.sqlite`) so the phone searches without
+a backend. Collections are tiny JSON manifests (explicit doc ids and/or
+selectors over series/stream/area/WG/year); built-ins:
+
+| Collection | Contents |
+|---|---|
+| `essentials` | ~52 hand-picked current core protocols (IP/TCP/DNS/TLS 1.3/QUIC/HTTP/DHCP/email + process docs) |
+| `standards-current` | Every RFC that is a member of a current STD or BCP — the closest thing to an official essential list |
+| `fyi-current` | Current FYI notes |
+| `year-2025` / `year-2026` | Everything published that year (example of the year selector) |
+| `all-current-txt` | The whole current archive (~8.8k docs) |
+
+Put your own `<id>.json` in `data/collections/` (or `--collections-dir`) to
+define new ones. `history: true` pulls in the obsoleted lineage too.
+
+```powershell
+.venv\Scripts\rfclib.exe packs list
+.venv\Scripts\rfclib.exe packs build essentials standards-current   # -> export-packs/
+.venv\Scripts\rfclib.exe packs serve --dir export-packs --port 8100
+```
+
+`android-reader/` is a dependency-free Android app (Java, mirrors the
+caionline pack-install design): install a `.rfcpack` via the file picker,
+`adb push`, or **Sync** against `packs serve` on your LAN (downloads are
+sha256-verified against `catalog.json`, installs are verified file-by-file
+against `pack.json` before promotion). Browse packs → documents, full-text
+search across all installed packs, and read in the WebView via a loopback
+server — fully offline after sync.
+
+### Third-party export formats (`packs export`)
+
+For readers outside this repo (e-book apps, offline browsing), the same
+collections export to standard formats — one RFC .txt becomes one clean
+XHTML chapter: page furniture stripped, ASCII diagrams preserved verbatim
+in `<pre>`, section anchors generated with the same heuristics as the
+search index, and `RFC 9000`/`[RFC9000]` references linkified (internal to
+the set by default; `--link-refs external` adds rfc-editor.org links).
+
+```powershell
+.venv\Scripts\rfclib.exe packs export essentials          # -> export-formats/essentials.epub + essentials/
+.venv\Scripts\rfclib.exe packs export standards-current --formats epub
+.venv\Scripts\rfclib.exe packs export standards-current --formats zim   # needs: uv sync --extra zim
+.venv\Scripts\rfclib.exe packs export standards-current --formats docset
+```
+
+* **EPUB** (EPUB3, pure stdlib zip): works in ReadEra/Apple Books/Play
+  Books; the nav document gives per-document chapters nested with their
+  top sections. Reader-side full-text search indexes the chapters.
+* **Dash docset** (`--formats docset`, pure stdlib): `<id>.docset` bundle +
+  `.tgz` — drag into Dash (or Zeal/Xcode). The `docSet.dsidx` lookup index
+  carries one entry per document plus one per top-level section/appendix
+  (`RFC9110: 4.2. HTTP Messaging` → deep-link straight to that anchor).
+* **HTML mirror**: plain folder of pages + `index.html` — greppable,
+  Everything/Windows-Search friendly, opens anywhere (the CHM successor,
+  without CHM's dead tooling; CHM itself was rejected: no maintained
+  Python writer, no Android support).
+* **ZIM (Kiwix)** (`--formats zim`; optional native bindings, `uv sync
+  --extra zim` — libzim wheels carry Xapian): one `.zim` per collection
+  with a **full-text index + title suggestions built at write time**, so
+  Kiwix apps / `kiwix-serve --port 8080 export-formats\standards-current.zim`
+  give offline full-text search over the corpus. Entries are flat and
+  lowercase (`rfc9000`), the main page is the collection index, and a
+  generated 48x48 PNG icon shows in the Kiwix catalogue.
+
+Validation: `python scripts\check_export.py` verifies EPUB/HTML structure
+(mimetype placement, OPF/nav well-formedness, anchors, cross-links,
+furniture stripping), `python scripts\check_zim.py` reopens a built
+ZIM with the libzim reader (entry counts, main-page redirect, metadata,
+illustration, anchors/links, and live Xapian suggestion + full-text
+queries), and `python scripts\check_docset.py` validates the docset
+bundle (plist, dsidx rows + every url/anchor resolving, tgz layout) —
+all end-to-end against the `essentials` collection.
+
+Search engine compatibility: the packs carry an FTS5 index, but **AOSP's
+system SQLite does not enable FTS5** — `external/sqlite/dist/Android.bp`
+on Android 11 and 16 defines FTS3/FTS4 only, and an API 36 emulator image
+has no `fts5` module (`MATCH` fails with `no such module: fts5`; OEM ROMs
+sometimes add it). `RfcSearch` therefore probes FTS5 once per process and
+falls back to portable LIKE-based AND matching with density ranking when
+the module is absent — correct, though slower and unstemmed. Queries pass
+a stop-word filter: SQLite FTS5 has no native stop-word support (no
+built-in tokenizer option exists, verified through SQLite 3.54), so
+filtering happens at query time and needs no pack rebuild; RFC normative
+keywords (MUST/SHOULD/MAY/CAN) are deliberately kept out of the stop list
+because they carry meaning in this corpus. A future option for indexed
+search on every device is an FTS4 mirror table (FTS4 is guaranteed on
+AOSP) or bundling a native SQLite with FTS5.
+
+```powershell
+cd android-reader
+.\gradlew.bat assembleDebug          # APK: app\build\outputs\apk\debug
+.\gradlew.bat test                   # installer validation unit tests
+adb install app\build\outputs\apk\debug\app-debug.apk
+```
+
+## Reader UI and Docker images
+
+The React reader UI (`rfclib ui`, with its stdlib JSON API) and the Docker
+images that serve it live on the [`reader-ui`](https://github.com/b1tninja/rfclib/tree/reader-ui)
+branch, kept out of the package so it needs no npm toolchain. Check that
+branch out to build or run them.
+
+## Roadmap
+
+1. ✅ Bulk download + incremental sync (rsync bootstrap + HTTP syncer)
+2. ✅ MCP presentation (search / get / metadata / aliases / recents)
+3. ✅ Full-text + semantic + hybrid retrieval, golden-set eval harness
+4. ✅ React + Vite reader UI (`rfclib ui`): JSON API + SPA — now on the
+   `reader-ui` branch
+5. ✅ Cross-references: citation graph mined from txt (normative/
+   informative/mention + section-level anchors, `rfclib refs`), exposed
+   in CLI and MCP with section deep links
+6. ✅ Verified-errata overlay from the RFC Editor errata service
+   (bulk API + per-doc fallback, `rfclib errata`, MCP `errata`)
+7. ✅ Content packs (`.rfcpack` + collections + LAN serve) and offline
+   Android reader (`android-reader/`) with FTS5 search
+8. ✅ Third-party export (`packs export`): EPUB3, static HTML mirror, Kiwix
+   ZIM (libzim + Xapian full-text index, optional `zim` extra) and Dash
+   docset (stdlib dsidx with section deep-links) — all four over the
+   shared txt→XHTML render layer
+9. ✅ Multi-stage Docker workflow serving the reader UI — on the
+   `reader-ui` branch
+
+
+## License
+
+rfclib is MIT-licensed (see [LICENSE](LICENSE)). The RFCs it downloads are
+not: they are published by the IETF Trust under the
+[Trust Legal Provisions](https://trustee.ietf.org/license-info). They may be
+copied and distributed in full but not modified; see [NOTICE](NOTICE).