zimlore 0.1.0__tar.gz

zimlore-0.1.0/.gitignore

@@ -0,0 +1,49 @@
+# The corpus is not the point of this repository and must never be in it.
+#
+# ZIM files are other people's content under their own licences (see NOTICE). They live in ZIMLORE_HOME
+# (default ~/.zimlore), never in the checkout. A ZIM is a single file that can be gigabytes, and git history
+# cannot walk it back, so every way of naming one is ignored from the first commit.
+*.zim
+*.zim.*
+*.zimaa
+*.meta4
+*.idx.tmp/
+zim/
+zimlore-home/
+.zimlore/
+
+.venv/
+__pycache__/
+*.pyc
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+.coverage
+.coverage.*
+htmlcov/
+
+# --- Captures and secrets ---
+*.har
+.env
+.env.*
+*.pem
+*.key
+
+# --- Scratch: the workspace pre-commit hook refuses these ---
+probe_*.py
+*.log
+*.out
+*.tmp
+scratch/
+tmp/
+
+# --- Agent worktrees and local tool state ---
+.claude/worktrees/
+.claude/settings.local.json
+.vscode/
+.idea/
+.DS_Store
+Thumbs.db

zimlore-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Justin Capella
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without
+limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
+Software, and to permit persons to whom the Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+This licence covers the code in this repository. It does not cover any ITU-T Recommendation text, which is
+copyright ITU and is not included here - see NOTICE.

zimlore-0.1.0/NOTICE

@@ -0,0 +1,21 @@
+NOTICE
+======
+
+This repository contains no ZIM file and no text from any ZIM file.
+
+A ZIM is an archive of someone else's content, and each one carries that content's own licence: DevDocs ZIMs
+archive upstream project documentation under whatever licence each project chose, Stack Exchange ZIMs are
+user-contributed content under Creative Commons terms with attribution requirements, Wikipedia is another
+case again. Those licences differ from each other and from this software's. zimlore therefore never ships,
+vendors or embeds ZIM content: files live in `ZIMLORE_HOME` (default `~/.zimlore`, subdirectory `zim/`) on
+each user's machine, fetched by the user with `zimlore get` or copied there, and `.gitignore` names every way
+a ZIM can be spelled so that one cannot be staged by accident. `tests/test_license.py` is a tripwire against
+the obvious copy, not a proof against a determined one; its test ZIMs are built from invented text at test
+time, in a temporary directory.
+
+What zimlore says about a document is a citation (library, ZIM edition date, entry path) plus the licence
+metadata the ZIM itself carries in its `License` field. Many ZIMs, including the DevDocs ones measured while
+writing this, carry none, and zimlore reports that as "not stated in the ZIM" rather than guessing: the
+licence of the text you reuse is the upstream project's, and you should read it there.
+
+The code in this repository is licensed under the MIT licence in LICENSE. That licence covers the code only.

zimlore-0.1.0/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.5
+Name: zimlore
+Version: 0.1.0
+Summary: A Kiwix library (ZIM files) served to AI agents over MCP: offline, citable, token-budgeted documentation search and reading. You supply the ZIMs; none are shipped.
+Project-URL: Homepage, https://github.com/b1tninja/zimlore
+Project-URL: Source, https://github.com/b1tninja/zimlore
+Project-URL: Issues, https://github.com/b1tninja/zimlore/issues
+Author-email: Justin Capella <justincapella@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: context7,devdocs,documentation,kiwix,libzim,mcp,model-context-protocol,offline,openzim,zim
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: libzim>=4.1; sys_platform != 'win32' or platform_machine != 'ARM64'
+Provides-Extra: all
+Requires-Dist: mcp>=2.3; extra == 'all'
+Provides-Extra: mcp
+Requires-Dist: mcp>=2.3; extra == 'mcp'
+Description-Content-Type: text/markdown
+
+# zimlore
+
+A [Kiwix](https://kiwix.org) library (ZIM files) served to AI agents over MCP: **offline, citable,
+token-budgeted documentation**. It is the offline counterpart of [Context7](https://github.com/upstash/context7):
+instead of asking a hosted service for snippets, an agent searches and reads ZIM files you hold on disk, and
+every answer carries the ZIM's name, edition date and entry path.
+
+You supply the ZIMs. **None are shipped**, and `.gitignore` names every way one could be staged
+(see [NOTICE](NOTICE)).
+
+```
+pip install "zimlore[mcp]"
+zimlore catalog flask              # search the public Kiwix catalogue
+zimlore get devdocs_en_flask       # download it, verified by SHA-256, into ~/.zimlore/zim
+zimlore search "application factory" -l flask
+zimlore read flask§patterns/appfactories/index
+zimlore mcp                        # the MCP server, on stdio
+```
+
+Managing what is installed (none of these touch the network):
+
+```
+zimlore du                         # disk use per library
+zimlore verify [--adopt]           # check files against their provenance record and libzim's checksum
+zimlore remove flask --yes         # delete (dry run without --yes; --edition DATE or --all when several)
+zimlore prune --keep 2 --yes       # drop old zimlore-installed editions and stale partial downloads
+                                   # (--include-unmanaged also deletes files zimlore did not install)
+```
+
+Putting ZIMs in place (`add` is local; `sync` and `get` use the network and resume an interrupted download):
+
+```
+zimlore add some.zim [--link]      # copy (or link) a local ZIM in after libzim opens it and its checksum holds
+zimlore sync --group web-testing --dry-run   # plan against a manifest or a named group; sizes are approximate
+zimlore sync --manifest .zimlore.toml [--update]   # install what is missing; --update stages newer editions
+```
+
+A manifest (`./.zimlore.toml`, else `$ZIMLORE_HOME/libraries.toml`) names catalogue entries only, never URLs or
+paths, so it can be committed: `libraries = ["devdocs_en_python"]`, `groups = ["infra"]`, `max_age_days = 365`,
+`budget = "2GB"` (the library may not grow past it). `sync` never removes anything. The shipped groups
+(`languages-core`, `web-testing`, `infra`) are an editorial default awaiting the maintainer's decision.
+
+Every download records a sidecar (`<file>.zimlore.json`: source URL, verified SHA-256, time). A file without one
+reports `provenance: unrecorded` in `zim_libraries`, `zim_resolve` and `zim_cite`; it is never presented as official.
+
+Register it with a client (this edits your client's configuration, so it is yours to run):
+
+```
+claude mcp add --scope user zimlore -- <venv>\Scripts\zimlore.exe mcp
+```
+
+## Installing documentation, by hand or by an agent with a shell
+
+Any agent that can run a command can install documentation; nothing needs the MCP server for that. Every
+missing-library error already names the command. The Kiwix catalogue is searchable (`q`, `lang`, `tag`,
+`maxsize`; the OPDS description lists them), so:
+
+```
+zimlore catalog playwright                       # search the live catalogue
+zimlore catalog --tag devdocs --max-mb 1         # every DevDocs ZIM of at most 1 MB (140 on 2026-11-07)
+zimlore available flask                          # the same, offline, from a cached snapshot (catalog --refresh)
+zimlore get devdocs_en_flask                     # download, verified against the .meta4 SHA-256
+zimlore add ~/Downloads/some.zim                 # a ZIM you already have: opened, checksum-checked, recorded
+zimlore sync --group languages-core --dry-run    # plan a whole set first; drop --dry-run to install
+```
+
+Each install writes a provenance record, so `zim_cite` can say where a file came from. Installing never needs
+the MCP server, and the MCP server never installs: planned on-demand installation from a package URL
+(`pkg:pypi/flask@3.1.0` to that version's official docs) is designed in `docs/on-demand-docs.md`, not built.
+
+## Where the files live
+
+`ZIMLORE_HOME` (default `~/.zimlore`), subdirectory `zim/`. Set `ZIMLORE_ZIM_PATH` (a list separated by `;` on
+Windows, `:` elsewhere) to read an existing Kiwix library where it already is. Nothing is ever written inside
+the repository. An optional `$ZIMLORE_HOME/aliases.json` maps your own names: `{"py": "devdocs_en_python"}`.
+
+## The tools
+
+| Tool | Does |
+|---|---|
+| `zim_libraries` | installed libraries with edition date, size, search type, `freshness`; ranks them against a name |
+| `zim_resolve` | one name or alias (`flask`, `python`, `security.stackexchange.com`) to exactly one library; an ambiguous name returns the candidates |
+| `zim_search` | one library or all of them; hits are refs with an excerpt and a citation |
+| `zim_read` | an article, a section (`#anchor`), an API definition, or one block (`¶n`) as markdown or text, paged by a token budget |
+| `zim_outline` | headings and API definitions with anchors, or a page of a library's articles |
+| `zim_links` | links out, internal ones resolved to refs and marked `in_zim` |
+| `zim_cite` | the citation string, edition, freshness and licence note |
+| `zim_available` | **offline** search of the cached catalogue snapshot: what could be installed, the exact `zimlore get <name>` command, and the snapshot's age |
+
+One reference grammar, accepted everywhere: `devdocs_en_flask§blueprints/index#why-blueprints¶3`. The library
+may be any alias (`flask§…`), `::` is the ASCII spelling of `§`, and a bare path works when `library` is given.
+`¶n` counts blocks of *our conversion* of the article, so the entry path and anchor are the durable part.
+
+Conventions inherited from `rfclib` and `itulib`:
+
+- A missing or empty library is an **`error` object that names the fix**, never an empty list. `hits: []` means
+  the search matched nothing, not that the topic is undocumented.
+- Long reads end with `next` (`{"ref", "start"}`); pass it back. A page always makes progress, even when one code
+  block alone exceeds the budget (it is returned whole and flagged `oversize`).
+- Every response states `freshness` (`date`, `age_days`, `stale`). A ZIM is a snapshot, and one with no `Date`
+  metadata reports `stale: null`, never "fresh".
+- HTML becomes markdown with **code blocks (and their language), tables, headings, lists and definition lists
+  kept**; scripts, navigation chrome and inline base64 images are dropped (images appear as `[image]`).
+
+## Offline catalogue
+
+`zimlore catalog --refresh` (the one command here that uses the network, besides `catalog QUERY`, `get` and
+`outdated`) stores the English Kiwix catalogue in `$ZIMLORE_HOME/catalog.json`, about 0.4 MB for 1,309 entries.
+After that `zimlore available flask` and `zimlore catalog --offline [flask]` say what could be installed and the
+exact `zimlore get <name>` to run, with no network and with the snapshot's age. The MCP server reads the same
+file: `zim_available`, and a `zim_resolve`/`zim_search` miss carries `obtainable` (the top five), so an agent can
+tell the user what to install instead of concluding a topic is undocumented. Without a snapshot the answer is an
+error naming `zimlore catalog --refresh`; a damaged snapshot is an error too, never an empty list.
+
+## Against Context7
+
+| | Context7 | zimlore |
+|---|---|---|
+| Where it runs | hosted MCP server `mcp.context7.com`; the MCP server is MIT, the crawler and API are private | your machine; no network after the download |
+| Rate limits | yes; unkeyed requests get low limits, a free key raises them | none |
+| Privacy | queries go to a third party | nothing leaves the machine |
+| Library names | `resolve-library-id` → `/owner/repo[/version]` | `zim_resolve` → a ZIM `Name` plus its edition date |
+| Versions | pin `/owner/repo/vX.Y.Z`; falls back to the nearest indexed | **pinned by ZIM date**; a ZIM does not record the upstream release number, the pages do |
+| Freshness | continuously re-crawled by the service | as fresh as the ZIM you hold (`zimlore outdated` compares with the catalogue) |
+| Coverage | very large and growing; any GitHub repo can be added | what Kiwix publishes: see below |
+| Result shape | curated code and info snippets ranked by the service | whole pages, or one section/definition, as markdown, cut to your budget |
+| Ranking | service-side, semantic, tuned | Xapian keyword search; **no semantic matching** (see Limits) |
+| Citations | library id | ZIM name, edition date, entry path, anchor |
+
+Pick Context7 when you need the newest release of a fast-moving library and are online. Pick zimlore when you
+are offline, behind a policy that forbids sending queries out, want a reproducible snapshot, or want the same
+documents every run. They are not exclusive.
+
+## What Kiwix has for developers
+
+Measured against the live catalogue on 2026-11-07 (`docs/research.md` has the method):
+
+- **243 DevDocs ZIMs**, 0.67 GB in total (median 0.85 MB, largest scikit-learn at 58 MB), dated 2026-05-04 to
+  2026-10-31: Python, Rust, Go, Node, React, Vue, TypeScript, Django, Flask, FastAPI, PostgreSQL, Docker,
+  Kubernetes, Terraform and more. This is the Context7-shaped part, and it is tiny.
+- `docs.python.org_en_all` (3.96 GB, 2026-08-28), `peps.python_en_all`, `devhints.io_en_all`, `archlinux_en_all`,
+  `gentoo_en_all`, `www.mankier.com_en_all` (man pages), `php.net_en_all` (**2024-08-13**, old).
+- **Stack Exchange**: 181 English-filter ZIMs, 191 GB in total; `stackoverflow.com_en_all` alone is 114.9 GB
+  (2026-07-02).
+
+## Limits
+
+- **Keyword search, not semantic.** On 34 questions paraphrased from page titles (`scripts/eval_retrieval.py`,
+  DevDocs Flask and Python), the right page was first for **10/34**, in the top 5 for **23/34**, MRR 0.436.
+  Questions that share words with the page do well; "isolated environment for project packages" does not find
+  "Virtual Environments and Packages". Searching with the library's own vocabulary works much better. With no
+  library given, 1 of 4 questions found its page: results from different ZIMs are interleaved because libzim
+  exposes no comparable score.
+- **No upstream version.** The ZIM date says when the docs were scraped; which release they document is on the
+  pages.
+- **Licences are not in the ZIMs we measured.** Neither DevDocs ZIM had a `License` metadata key; `zim_cite`
+  says "not stated in the ZIM" rather than guessing. DevDocs itself is MPL-2.0, but the text it archives is
+  each upstream project's.
+- **`libzim` is GPL-3.0-or-later** (the binding; the C++ library is GPL-2.0 per its repository). zimlore is MIT
+  and imports it at run time; see `PLAN.md` for what that means for redistribution.
+- Wheels exist for Windows x64, macOS and Linux (glibc and musl) on CPython 3.11+. On Windows ARM64 `pip
+  install zimlore` succeeds without `libzim`, and every tool reports the missing binding and the remedy.
+
+## Development
+
+```
+python -m pip install -e . --group dev      # pip 25.1+
+python -m pytest -q                  # no ZIM needed: tests build a synthetic one with libzim's writer
+python -m ruff check . ; python -m ruff format --check . ; python -m mypy zimlore tests
+python scripts/eval_retrieval.py     # needs devdocs_en_flask and devdocs_en_python installed
+```
+
+[ROADMAP.md](ROADMAP.md) lists every obligation with the test that would fail if it regressed;
+[PLAN.md](PLAN.md) records the design decisions; [docs/research.md](docs/research.md) the sources.

zimlore-0.1.0/PLAN.md

@@ -0,0 +1,139 @@
+# zimlore plan
+
+zimlore serves a Kiwix library (ZIM files) to AI agents over MCP as an offline-first alternative to Context7.
+This file records the decisions and why. Facts behind them are in [docs/research.md](docs/research.md); status
+of each obligation is in [ROADMAP.md](ROADMAP.md).
+
+## 1. Scope
+
+In: find a library by name, search it, read an article/section/definition/block as markdown or text under a token
+budget, follow links, cite with the ZIM's edition date, say how stale a snapshot is, and fetch ZIMs from the
+public catalogue with a verified hash.
+
+Out, on purpose:
+
+- **Shipping or caching ZIM content.** Files live in `ZIMLORE_HOME`, never in the repository.
+- **Semantic retrieval.** libzim gives Xapian keyword search and a title index. An embedding index would be a
+  second corpus to build and keep in step with each ZIM; if it is wanted it is a separate extra, not a promise
+  made here (ROADMAP `zl.semantic_search`, `[ ]`).
+- **Re-hosting upstream documentation as curated snippets.** Context7's value is curation; zimlore returns the
+  page, section or definition as published.
+- **Replacing kiwix-serve.** It is a library for agents, not a web server for people.
+
+## 2. Decisions
+
+**Name: `zimlore`.** Free on PyPI (`names.py`: simple 404, json 404) with *zero* one-keystroke neighbours in the
+931,439-name snapshot, free as a GitHub repository name under `b1tninja` (404), and not in the workspace. Rejected:
+`zimlib` (ten live neighbours incl. `ziplib`, `zilib`), `zimdoc`/`zimdocs` (one keystroke from `simdoc`/`vimdoc`
+and from each other), `docsmith` (taken), `zimdex` (neighbours `zindex`, `imdex`). Sibling pattern is
+`rfclib`/`itulib`; `lib` would have been natural but every `zim*lib` form collides.
+
+**One module owns the native import (`_libzim.py`).** `libzim` is a compiled extension with wheels for Windows
+x64, macOS, Linux (glibc, musl) and no Windows ARM64 wheel. The dependency carries a marker that excludes only
+Windows-on-ARM, so installation succeeds there and every tool answers with the missing-binding remedy instead of
+a compiler error. Everything that does not read a ZIM (CLI `info`, the reference grammar, the converter, the
+catalogue client) works without it. This is the `rfclib[zim]` pattern inverted: rfclib writes ZIMs and makes
+libzim optional; zimlore *is* a ZIM reader, so it is a hard dependency.
+
+**HTML to markdown with the standard library only.** `html.parser`, no BeautifulSoup/lxml/markdownify. The
+measured inputs (DevDocs) are regular, and a hand-written converter is the only way to guarantee the three
+things an agent needs and generic converters lose: code blocks with their language, anchors on headings and API
+definitions (so `#anchor` reads work), and the removal of inline base64 images that would cost thousands of
+tokens each. Cost: Stack Exchange and Wikipedia HTML has not been exercised (ROADMAP `zl.html_other_scrapers`).
+
+**The unit of paging is the block.** A paragraph, heading, list item, table, definition or code fence is never
+split. A page always contains at least one block, so a single oversized block (a 400-line log) is returned whole
+and flagged rather than cut, because a cursor that cannot advance is worse than a long page. `¶n` is the nth
+block; it indexes zimlore's conversion, so it is documented as less durable than the entry path and anchor.
+
+**Sections and definitions are first-class.** DevDocs pages are whole chapters (the `os` page is the whole module).
+`#anchor` reads one `<section id>` or one `<dt id>` definition and its body, which is what makes "just
+`os.getcwd`" cost 40 tokens instead of 20,000.
+
+**Library identity is the ZIM `Name`, version is the ZIM `Date`.** Aliases are derived (DevDocs slug, title
+without "Docs", Stack Exchange host, user `aliases.json`) and *ranked*: exact > all words > prefix > substring >
+fuzzy. A tie is an error listing the candidates, never a coin flip. Two editions of one `Name` resolve to the
+newest and the response lists the others. The response says outright that a ZIM does not record the upstream
+release (Context7's `/org/repo/vX.Y.Z` has no equivalent), because the honest alternative to a version number is
+a date.
+
+**Versions are not editions; a library has a family.** Producers that build documentation per package version
+name each build differently (`golang_github.com_spf13_cobra_v1.8.1`, `pypi_cryptography_50.0.2`), so by the rule
+above each version is its own library, and measured on the Go prototype that made `cobra` resolve to v1.8.1 over
+v1.10.2 (the alias is one character shorter and the ambiguity guard compared only scores). A `Book` therefore has
+a derived `family` and `version`, with no change to the ZIM format or any stored state. The family is the `purl:`
+tag without its version when the ZIM carries one (the producer's own statement), else the `Name` without a
+trailing `_v?<digits>(.<alnum>)*(-<alnum>)?` token; a `_YYYY-MM` date suffix is an edition marker and is dropped
+first, and a bare integer of more than two digits or a four-digit year (`wikipedia_en_100`, `ubuntu_en_all_2024`)
+is a count or a date, not a release. A name with no such token is its own family with no version, which is every
+DevDocs and Stack Exchange ZIM, so nothing changes for them. Resolution scores a *family* (the best of its members,
+so versions of one library never tie each other and alias length cannot pick a version), then picks inside it:
+no version asked means the newest by `library.version_key` (numeric components, `v` ignored, pre-releases below
+releases, a version with no number last); a version asked (`cobra v1.8.1`, `cobra==1.8.1`, `cobra@1.8`, a purl's
+`@version`) means that exact string, else the newest whose components start with it (`3.1` never matches `3.10`),
+reported as `exact` or `prefix`; no match is an error listing the installed versions, and it names no build
+command because none exists. A whole `Name` always selects that very ZIM. A purl is accepted wherever a library
+name is (`zimlore/purl.py` reads nine types and refuses the rest): it matches a ZIM's `purl:` tag on type,
+namespace and name, and falls back to the package name for ZIMs that carry no tag. Deleting stays per `Name`:
+`prune` keeps the newest N *editions* of each Name and never counts another version as one, and `remove` of a
+package with several versions refuses until a version or `--all-versions` is given. The cost accepted: a ZIM with
+a purl tag and one without are different families even for one package.
+
+**Staleness is data.** Every response carries `freshness {date, age_days, stale, stale_after_days}`. `stale` is
+`age > 365` days, a stated convention; `null` when the ZIM has no date. `zimlore outdated` compares with the
+catalogue (network).
+
+**Search = title matches + full-text, with a documented fallback.** Exact-title hits first, then strict
+full-text, then remaining title hits, then partial matches marked `some terms`. Partial matches come from
+per-word searches fused by `ln(pages / pages matching)` weights and a title boost, because libzim's parser
+requires every term (found by running it; first eval run scored hit@1 8/34). Across libraries, results are
+interleaved rank by rank and the response says so, since libzim exposes no comparable score. The 34-case eval
+(MRR 0.436, hit@5 23/34) is the honest summary: lexical, and weak on paraphrase.
+
+**Citations are built, not typed.** `Title [Name], ZIM edition DATE, entry PATH#anchor`, plus the ZIM's `License`
+metadata or the sentence "not stated in the ZIM". No ZIM we measured had a licence key, and inventing one from
+the project name would be an unverifiable claim.
+
+**Network is confined to `catalog.py`, and downloads are verified.** The hash comes from kiwix.org's `.meta4`;
+bytes may come from any mirror (the `.zim` URL redirects to third-party hosts); a mismatch discards the file.
+Only `https` (or `http` to localhost, for tests) is accepted. Nothing in the MCP server touches the network, which
+is the point of the project.
+
+**ZIM content is untrusted data.** A ZIM is third-party content: its article text, titles and metadata (`Title`,
+`Name`, `Source`, and therefore the `cite` strings built from them) must be quoted and cited, never followed as
+instructions. The MCP server instructions say so, and the read path is bounded against hostile files (an entry
+size budget, a converter output ceiling, capped list indentation, bounded redirect chains).
+
+**Licence of the code: MIT, with a flagged tension.** `libzim` (the binding) is GPL-3.0-or-later and zimlore
+imports it. MIT code that imports a GPL library is fine to publish and to use, but a *combined* binary or image
+that redistributes both is GPL-3.0 as a whole. The siblings (`rfclib[zim]`) take the same position. This is a
+decision for the maintainer (section 5), not something this file can settle.
+
+## 3. Verification model
+
+Tests build a tiny synthetic ZIM with libzim's writer (invented text, DevDocs-shaped markup) so the suite needs
+no download; `tests/test_license.py` is the tripwire against ZIM content entering the tree. What the suite cannot
+prove (converter behaviour on real pages, ranking) is measured by `scripts/eval_retrieval.py` and by the live
+MCP run recorded in `docs/research.md`, and ROADMAP rows that depend on them stay `[part]`.
+
+## 4. Portfolio position
+
+`PORTFOLIO.md` §4 defines "reference libraries - not tiered" as the corpora every protocol project *cites from*
+(`rfclib`, `itulib`), justified "by a citation some other project could not make". zimlore is **not** that:
+it is a developer-documentation corpus for agents, not a source for protocol citations. It is therefore not
+sequenced in a wave and does not count against the two-project rule, but it also is not covered by the existing
+sentence. A proposed entry is in [docs/portfolio-entry.md](docs/portfolio-entry.md); `PORTFOLIO.md` has **not**
+been edited, because the workspace root is shared and the classification is the maintainer's call.
+
+## 5. Decisions waiting for the maintainer
+
+1. **Does `PORTFOLIO.md` list zimlore, and as what?** Recommendation: a third "reference library" row with the
+   sentence widened to "or that agents read from", no wave, no tier.
+2. **Licence of zimlore** given GPL-3.0-or-later `libzim`. Recommendation: stay MIT (library use is the norm),
+   state it in NOTICE (done), and revisit if a bundled distribution is ever planned.
+3. **Name `zimlore`**, and whether the GitHub repository is `b1tninja/zimlore`. Nothing has been pushed or created.
+4. **Registering the MCP server** with Claude Code / Cline (`claude mcp add ...`): edits the user's configuration,
+   so not done.
+5. **Publishing to PyPI**, and the release workflow's trusted-publisher setup. Not done.
+6. **Which ZIMs to install by default** (a starter set). The DevDocs set is 0.67 GB in total and is the natural
+   candidate; Stack Overflow is 115 GB and is not.

zimlore-0.1.0/README.md

@@ -0,0 +1,174 @@
+# zimlore
+
+A [Kiwix](https://kiwix.org) library (ZIM files) served to AI agents over MCP: **offline, citable,
+token-budgeted documentation**. It is the offline counterpart of [Context7](https://github.com/upstash/context7):
+instead of asking a hosted service for snippets, an agent searches and reads ZIM files you hold on disk, and
+every answer carries the ZIM's name, edition date and entry path.
+
+You supply the ZIMs. **None are shipped**, and `.gitignore` names every way one could be staged
+(see [NOTICE](NOTICE)).
+
+```
+pip install "zimlore[mcp]"
+zimlore catalog flask              # search the public Kiwix catalogue
+zimlore get devdocs_en_flask       # download it, verified by SHA-256, into ~/.zimlore/zim
+zimlore search "application factory" -l flask
+zimlore read flask§patterns/appfactories/index
+zimlore mcp                        # the MCP server, on stdio
+```
+
+Managing what is installed (none of these touch the network):
+
+```
+zimlore du                         # disk use per library
+zimlore verify [--adopt]           # check files against their provenance record and libzim's checksum
+zimlore remove flask --yes         # delete (dry run without --yes; --edition DATE or --all when several)
+zimlore prune --keep 2 --yes       # drop old zimlore-installed editions and stale partial downloads
+                                   # (--include-unmanaged also deletes files zimlore did not install)
+```
+
+Putting ZIMs in place (`add` is local; `sync` and `get` use the network and resume an interrupted download):
+
+```
+zimlore add some.zim [--link]      # copy (or link) a local ZIM in after libzim opens it and its checksum holds
+zimlore sync --group web-testing --dry-run   # plan against a manifest or a named group; sizes are approximate
+zimlore sync --manifest .zimlore.toml [--update]   # install what is missing; --update stages newer editions
+```
+
+A manifest (`./.zimlore.toml`, else `$ZIMLORE_HOME/libraries.toml`) names catalogue entries only, never URLs or
+paths, so it can be committed: `libraries = ["devdocs_en_python"]`, `groups = ["infra"]`, `max_age_days = 365`,
+`budget = "2GB"` (the library may not grow past it). `sync` never removes anything. The shipped groups
+(`languages-core`, `web-testing`, `infra`) are an editorial default awaiting the maintainer's decision.
+
+Every download records a sidecar (`<file>.zimlore.json`: source URL, verified SHA-256, time). A file without one
+reports `provenance: unrecorded` in `zim_libraries`, `zim_resolve` and `zim_cite`; it is never presented as official.
+
+Register it with a client (this edits your client's configuration, so it is yours to run):
+
+```
+claude mcp add --scope user zimlore -- <venv>\Scripts\zimlore.exe mcp
+```
+
+## Installing documentation, by hand or by an agent with a shell
+
+Any agent that can run a command can install documentation; nothing needs the MCP server for that. Every
+missing-library error already names the command. The Kiwix catalogue is searchable (`q`, `lang`, `tag`,
+`maxsize`; the OPDS description lists them), so:
+
+```
+zimlore catalog playwright                       # search the live catalogue
+zimlore catalog --tag devdocs --max-mb 1         # every DevDocs ZIM of at most 1 MB (140 on 2026-11-07)
+zimlore available flask                          # the same, offline, from a cached snapshot (catalog --refresh)
+zimlore get devdocs_en_flask                     # download, verified against the .meta4 SHA-256
+zimlore add ~/Downloads/some.zim                 # a ZIM you already have: opened, checksum-checked, recorded
+zimlore sync --group languages-core --dry-run    # plan a whole set first; drop --dry-run to install
+```
+
+Each install writes a provenance record, so `zim_cite` can say where a file came from. Installing never needs
+the MCP server, and the MCP server never installs: planned on-demand installation from a package URL
+(`pkg:pypi/flask@3.1.0` to that version's official docs) is designed in `docs/on-demand-docs.md`, not built.
+
+## Where the files live
+
+`ZIMLORE_HOME` (default `~/.zimlore`), subdirectory `zim/`. Set `ZIMLORE_ZIM_PATH` (a list separated by `;` on
+Windows, `:` elsewhere) to read an existing Kiwix library where it already is. Nothing is ever written inside
+the repository. An optional `$ZIMLORE_HOME/aliases.json` maps your own names: `{"py": "devdocs_en_python"}`.
+
+## The tools
+
+| Tool | Does |
+|---|---|
+| `zim_libraries` | installed libraries with edition date, size, search type, `freshness`; ranks them against a name |
+| `zim_resolve` | one name or alias (`flask`, `python`, `security.stackexchange.com`) to exactly one library; an ambiguous name returns the candidates |
+| `zim_search` | one library or all of them; hits are refs with an excerpt and a citation |
+| `zim_read` | an article, a section (`#anchor`), an API definition, or one block (`¶n`) as markdown or text, paged by a token budget |
+| `zim_outline` | headings and API definitions with anchors, or a page of a library's articles |
+| `zim_links` | links out, internal ones resolved to refs and marked `in_zim` |
+| `zim_cite` | the citation string, edition, freshness and licence note |
+| `zim_available` | **offline** search of the cached catalogue snapshot: what could be installed, the exact `zimlore get <name>` command, and the snapshot's age |
+
+One reference grammar, accepted everywhere: `devdocs_en_flask§blueprints/index#why-blueprints¶3`. The library
+may be any alias (`flask§…`), `::` is the ASCII spelling of `§`, and a bare path works when `library` is given.
+`¶n` counts blocks of *our conversion* of the article, so the entry path and anchor are the durable part.
+
+Conventions inherited from `rfclib` and `itulib`:
+
+- A missing or empty library is an **`error` object that names the fix**, never an empty list. `hits: []` means
+  the search matched nothing, not that the topic is undocumented.
+- Long reads end with `next` (`{"ref", "start"}`); pass it back. A page always makes progress, even when one code
+  block alone exceeds the budget (it is returned whole and flagged `oversize`).
+- Every response states `freshness` (`date`, `age_days`, `stale`). A ZIM is a snapshot, and one with no `Date`
+  metadata reports `stale: null`, never "fresh".
+- HTML becomes markdown with **code blocks (and their language), tables, headings, lists and definition lists
+  kept**; scripts, navigation chrome and inline base64 images are dropped (images appear as `[image]`).
+
+## Offline catalogue
+
+`zimlore catalog --refresh` (the one command here that uses the network, besides `catalog QUERY`, `get` and
+`outdated`) stores the English Kiwix catalogue in `$ZIMLORE_HOME/catalog.json`, about 0.4 MB for 1,309 entries.
+After that `zimlore available flask` and `zimlore catalog --offline [flask]` say what could be installed and the
+exact `zimlore get <name>` to run, with no network and with the snapshot's age. The MCP server reads the same
+file: `zim_available`, and a `zim_resolve`/`zim_search` miss carries `obtainable` (the top five), so an agent can
+tell the user what to install instead of concluding a topic is undocumented. Without a snapshot the answer is an
+error naming `zimlore catalog --refresh`; a damaged snapshot is an error too, never an empty list.
+
+## Against Context7
+
+| | Context7 | zimlore |
+|---|---|---|
+| Where it runs | hosted MCP server `mcp.context7.com`; the MCP server is MIT, the crawler and API are private | your machine; no network after the download |
+| Rate limits | yes; unkeyed requests get low limits, a free key raises them | none |
+| Privacy | queries go to a third party | nothing leaves the machine |
+| Library names | `resolve-library-id` → `/owner/repo[/version]` | `zim_resolve` → a ZIM `Name` plus its edition date |
+| Versions | pin `/owner/repo/vX.Y.Z`; falls back to the nearest indexed | **pinned by ZIM date**; a ZIM does not record the upstream release number, the pages do |
+| Freshness | continuously re-crawled by the service | as fresh as the ZIM you hold (`zimlore outdated` compares with the catalogue) |
+| Coverage | very large and growing; any GitHub repo can be added | what Kiwix publishes: see below |
+| Result shape | curated code and info snippets ranked by the service | whole pages, or one section/definition, as markdown, cut to your budget |
+| Ranking | service-side, semantic, tuned | Xapian keyword search; **no semantic matching** (see Limits) |
+| Citations | library id | ZIM name, edition date, entry path, anchor |
+
+Pick Context7 when you need the newest release of a fast-moving library and are online. Pick zimlore when you
+are offline, behind a policy that forbids sending queries out, want a reproducible snapshot, or want the same
+documents every run. They are not exclusive.
+
+## What Kiwix has for developers
+
+Measured against the live catalogue on 2026-11-07 (`docs/research.md` has the method):
+
+- **243 DevDocs ZIMs**, 0.67 GB in total (median 0.85 MB, largest scikit-learn at 58 MB), dated 2026-05-04 to
+  2026-10-31: Python, Rust, Go, Node, React, Vue, TypeScript, Django, Flask, FastAPI, PostgreSQL, Docker,
+  Kubernetes, Terraform and more. This is the Context7-shaped part, and it is tiny.
+- `docs.python.org_en_all` (3.96 GB, 2026-08-28), `peps.python_en_all`, `devhints.io_en_all`, `archlinux_en_all`,
+  `gentoo_en_all`, `www.mankier.com_en_all` (man pages), `php.net_en_all` (**2024-08-13**, old).
+- **Stack Exchange**: 181 English-filter ZIMs, 191 GB in total; `stackoverflow.com_en_all` alone is 114.9 GB
+  (2026-07-02).
+
+## Limits
+
+- **Keyword search, not semantic.** On 34 questions paraphrased from page titles (`scripts/eval_retrieval.py`,
+  DevDocs Flask and Python), the right page was first for **10/34**, in the top 5 for **23/34**, MRR 0.436.
+  Questions that share words with the page do well; "isolated environment for project packages" does not find
+  "Virtual Environments and Packages". Searching with the library's own vocabulary works much better. With no
+  library given, 1 of 4 questions found its page: results from different ZIMs are interleaved because libzim
+  exposes no comparable score.
+- **No upstream version.** The ZIM date says when the docs were scraped; which release they document is on the
+  pages.
+- **Licences are not in the ZIMs we measured.** Neither DevDocs ZIM had a `License` metadata key; `zim_cite`
+  says "not stated in the ZIM" rather than guessing. DevDocs itself is MPL-2.0, but the text it archives is
+  each upstream project's.
+- **`libzim` is GPL-3.0-or-later** (the binding; the C++ library is GPL-2.0 per its repository). zimlore is MIT
+  and imports it at run time; see `PLAN.md` for what that means for redistribution.
+- Wheels exist for Windows x64, macOS and Linux (glibc and musl) on CPython 3.11+. On Windows ARM64 `pip
+  install zimlore` succeeds without `libzim`, and every tool reports the missing binding and the remedy.
+
+## Development
+
+```
+python -m pip install -e . --group dev      # pip 25.1+
+python -m pytest -q                  # no ZIM needed: tests build a synthetic one with libzim's writer
+python -m ruff check . ; python -m ruff format --check . ; python -m mypy zimlore tests
+python scripts/eval_retrieval.py     # needs devdocs_en_flask and devdocs_en_python installed
+```
+
+[ROADMAP.md](ROADMAP.md) lists every obligation with the test that would fail if it regressed;
+[PLAN.md](PLAN.md) records the design decisions; [docs/research.md](docs/research.md) the sources.