@tibia.sh/tibiawiki-data 3.0.0 → 3.0.2

package/README.md

@@ -28,9 +28,11 @@ major are data refreshes of the same schema.
 
 `SCHEMA_VERSION` is a literal in `src/index.ts`, and `pnpm test` asserts that it
 equals both the `package.json` major and the index's `mcp_schema_version` row.
-Change all three together. The literal is deliberate. Derived from `package.json`,
-that assertion would compare a value with itself and could never fail, and it is the
-only thing that stops a release from shipping under the wrong major.
+Change all three together, following
+[Bumping the schema version](#bumping-the-schema-version). The literal is deliberate.
+Derived from `package.json`, that assertion would compare a value with itself and could
+never fail, and it is the only thing that stops a release from shipping under the wrong
+major.
 
 ## Why the index is committed
 
@@ -76,6 +78,7 @@ What each release was built from:
 | Release | Generator `version` | `generate_time` |
 |---|---|---|
 | `3.0.0` | `9.0.0` | `2026-09-12T19:53:53.020856+00:00` |
+| `3.0.1` | `9.0.0` | `2026-09-13T07:02:58.860376+00:00` |
 
 To read them from any index:
 
@@ -99,7 +102,12 @@ pnpm build-index
 
 That builds `dist/`, then `scripts/build.ts` runs the devDependency's
 `tibiawiki-mcp build-index` with `TIBIAWIKI_MCP_DB` set to `DB_PATH`, so the build
-writes exactly the file this package ships and exports.
+writes exactly the file this package ships and exports. From `0.3.1`, the server installs
+the generator from its own requirements file, where every Python dependency is pinned and
+hashed, so every build runs the same packages. Builds still set a 7-day PyPI cooldown,
+matching the one pnpm applies to npm packages. It is a backstop for a server older than
+`0.3.1`, which resolves those dependencies fresh on every build. You can override it with
+`UV_EXCLUDE_NEWER`.
 
 `build-index` needs [`uv`](https://docs.astral.sh/uv/) and network access to
 TibiaWiki. It validates the new index before replacing `index.db`, so a build that
@@ -109,21 +117,63 @@ its journal beside that. `.gitignore` excludes both, and must never exclude
 `index.db`. The `3.0.0` build took just under six minutes, most of it the generator's
 crawl.
 
+`pnpm test` pins the generator too. It fails for an index whose `database_info` `version`
+is anything but `9.0.0`. The major version covers only the server's enrichment tables,
+and no version covers the tables tibiawiki-sql writes.
+
+A generator upgrade does not bump the major. A user who installed the oldest published server
+that depends on `^N` gets every new `N.x` of this package, so before every publish, and in CI,
+the `oldest-consumer` job installs that server package from npm together with the packed
+candidate, and pages every item through its `tibia_find_items`. The gate passes only when no
+page is an error, every page carries the candidate index's `generate_time`, and every item in
+the index comes back exactly once. `pnpm oldest-consumer` runs the same gate, and needs network
+access to npm.
+
+The sweep covers items only, not creatures, NPCs, quests or spells. So the `9.0.0` pin in
+`test/data.test.ts` stays as the explicit decision point for a generator upgrade.
+
+### Drift
+
+`.github/workflows/drift.yml` rebuilds the index every Monday at 06:17 UTC, and when you
+run it by hand from the Actions tab. It digests the committed `index.db` with the
+devDependency's `tibiawiki-mcp index-digest`, runs `pnpm build-index`, digests the
+rebuilt index, and runs `pnpm test` against it. The digest covers what the server reads,
+and leaves out stamps that change on every run, such as `generate_time`. The run's log
+shows both digests.
+
+When the digests match, the run ends green and opens nothing. When they differ, the
+content changed. The run pushes the rebuilt `index.db` to the `drift/index` branch, with
+`version` set to the next patch npm does not have. It opens a pull request carrying both
+digests, or updates the one already open. Merging that pull request publishes the new
+patch.
+
+- The pull request is opened with `GITHUB_TOKEN`, so its CI waits for you. Click
+  "Approve workflows to run" on it, then review the pull request before you merge it.
+- The workflow never merges and never turns on auto-merge, so a bad day on the wiki can at
+  most open a pull request.
+- A red run is a signal. A tripped gate, a failing test, an unreadable registry, or a
+  `version` on `main` that npm does not list yet each end the run red, and nothing is
+  pushed or opened. Find out why before the next run.
+- Each run that finds a change replaces `drift/index`, so an open pull request always
+  carries the newest rebuild.
+- GitHub turns off a schedule after 60 days without activity in a public repository, and
+  that stops the job without a red run. Turn it back on from the Actions tab.
+
 ## The devDependency on the server
 
-`@tibia.sh/tibiawiki-mcp` is a devDependency for two jobs: its `build-index` produces
-the index, and its `serve` validates it, in `pnpm test` here and in `pnpm smoke`
-against an installed copy.
+`@tibia.sh/tibiawiki-mcp` is a devDependency for three jobs: its `build-index` produces
+the index, its `index-digest` tells the drift job whether a rebuild changed it, and its
+`serve` validates it, in `pnpm test` here and in `pnpm smoke` against an installed copy.
 
-**When to bump it.** On a `0.x` version, `^0.1.0` means `>=0.1.0 <0.2.0`. Left alone,
-it pins every rebuild to the 0.1 generator and its gates while the server moves on.
+**When to bump it.** On a `0.x` version, `^0.3.0` means `>=0.3.0 <0.4.0`. Left alone,
+it pins every rebuild to the 0.3 generator and its gates while the server moves on.
 Bump it whenever the server's indexer changes: `build-index`, its enrichment, its
 gates, or the schema. Write the new range by hand. This repository saves exact
 versions, so `pnpm add` records a pin instead.
 
 **The dependency cycle is intentional.** The server depends on this package, and this
 package devDepends on the server. npm and pnpm allow it because this side is
-dev-only and never resolved at runtime. Do not "fix" it. Once the server depends on
+dev-only and never resolved at runtime. Do not "fix" it. Because the server depends on
 this package, `node_modules` here also holds a published copy of this package,
 installed as the server's dependency. The server's default index resolution could
 find that copy instead of `index.db`. So the test always passes `TIBIAWIKI_MCP_DB`
@@ -132,7 +182,7 @@ explicitly, and checks that the answer's `indexGeneratedAt` matches `index.db`.
 ## Development
 
 Requires Node 22.18 or later, because the tests run TypeScript directly, and pnpm
-10.33.0, pinned in `packageManager`.
+12.4.1, pinned in `packageManager`.
 
 ```bash
 pnpm install --frozen-lockfile
@@ -145,8 +195,17 @@ pnpm test
 2. It typechecks `src/`, `test/` and `scripts/`, the JavaScript in `scripts/`
    included. This must come after the build: the test imports this package by its own
    name, so it typechecks against the built declarations, as a consumer does.
-3. It runs `test/data.test.ts`. That test spawns the server from the devDependency
-   against `index.db`, and makes a real query.
+3. It runs every `test/*.test.ts`. `test/data.test.ts` spawns the server from the
+   devDependency against `index.db`, and makes a real query. The other files check the
+   workflows, the smoke check, the oldest-consumer gate's decisions and the test floor, with
+   no network.
+
+The run fails when fewer than `MIN_TESTS` tests pass. `node --test` still exits 0 for a
+file that declares no tests, for a skipped test, and for a `--test-name-pattern` that
+filters tests away, one inherited through `NODE_OPTIONS` included. Without the floor, an
+emptied test file would pass the gate every publish runs. Adding a test needs no change. When
+you remove or skip one on purpose, lower `MIN_TESTS` in `test/min-tests.ts` in the same
+commit.
 
 `prepublishOnly` runs `pnpm test` too, so publishing from the directory always builds
 `dist/` first.
@@ -173,6 +232,117 @@ installed package and it spawns the installed server, so it checks the artefact
 than this checkout. That is why the test file imports only node builtins and packages
 by name.
 
+## Releases
+
+Merging a commit to `main` publishes its `package.json` `version` if npm does not have
+that version yet. On every push to `main`, `.github/workflows/release.yml` runs the
+`oldest-consumer` gate from [How the index is built](#how-the-index-is-built), and its release
+job waits for that gate. The release job then asks npm whether it lists that exact version. If
+it does, the run publishes nothing and ends green. Every merge that leaves `version` alone,
+and passes the gate, ends this way. If it does not, the run installs from the lockfile, runs
+`pnpm test`, and runs `npm publish`.
+
+- The gate runs on every push, whether the run publishes or not, so an index that breaks the
+  oldest published server turns the run red even when nothing is published.
+- The check is for existence, never a comparison with `latest`. A revert leaves
+  `version` below `latest`, and `npm publish` moves `latest` itself.
+- A registry the check cannot read fails the run. It is never taken for a missing
+  version.
+- The run packs the committed `index.db` and never rebuilds it, so the file published is
+  the file reviewed in the pull request.
+- It publishes through npm trusted publishing, so no npm token exists to leak, and npm
+  attaches a provenance attestation for the merged commit. The trusted publisher is
+  registered for the file name `release.yml`, and renaming the file breaks publishing
+  with no warning.
+- Every pull request runs the same `pnpm test` and the same gate, in `.github/workflows/ci.yml`.
+
+Nothing is tagged, so a failed publish leaves nothing stranded. When a release run fails,
+follow [docs/RELEASING.md](docs/RELEASING.md).
+
+### Bumping the schema version
+
+**Not yet exercised.** No schema bump has gone through this procedure.
+
+A bump from N-1 to N cannot pass the automated gates. This package's `N.0.0` runs
+`pnpm test` against its devDependency server, which has to read schema N, so it needs a
+schema-N server on the registry. That server's CI needs this package's `N.0.0` on the
+registry: its `^N` dependency has to install, and its `test/data-package.test.ts` and
+regression sweep read the installed index. So one side is published outside its
+pipeline. It is this package, because no published server installs `N.0.0`. Server
+`0.1.0` does not use this package, and every later server depends on a major below N.
+
+1. On the server's schema-N branch, run `pnpm build`, then `npm pack`. Build this
+   repository's index with that tarball's `build-index`. The published server stamps the
+   index N-1, which this repository's tests reject. Then cross-validate: install each
+   repository's counterpart from the other's local tarball, and run both full test
+   suites. Neither repository builds `dist/` when it packs, so build before every
+   `npm pack`, or the tarball carries a stale `dist/` or none.
+2. The maintainer publishes this package's `N.0.0` by hand, from the validated tarball.
+   `npm publish` runs no lifecycle scripts for a tarball, so step 1 is the only gate it
+   gets. The release carries no provenance and no trusted publisher.
+3. The server's pull request sets `MCP_SCHEMA_VERSION` to N and its dependency range to
+   `^N`. It also adds `trustPolicyExclude` for exactly `@tibia.sh/tibiawiki-data@N.0.0`
+   to the server's `pnpm-workspace.yaml`, with the reason in a comment:
+
+   ```yaml
+   # @tibia.sh/tibiawiki-data N.0.0 was published by hand, so it has no trusted publisher.
+   trustPolicyExclude:
+     - '@tibia.sh/tibiawiki-data@N.0.0'
+   ```
+
+   Its CI passes against the registry, and its release PR publishes it through the
+   server's pipeline.
+4. This repository's pull request commits that exact `index.db`, with `SCHEMA_VERSION` N,
+   `version` `N.0.0`, and the devDependency moved to the new server. The new server
+   depends on `^N`, so the install here resolves the hand-published `N.0.0` too, and the
+   pull request adds the same exclude to this repository's `pnpm-workspace.yaml`. Its CI
+   passes, and merging it publishes nothing, because npm already has `N.0.0`.
+
+Do not merge a data refresh here between steps 2 and 4. `main` is still on N-1 then, and
+npm refuses to publish a version below `N.0.0` without a dist-tag, so its release run
+fails.
+
+Both repositories set `trustPolicy: no-downgrade`, which makes pnpm refuse a version with
+weaker trust evidence than any version published before it. The release workflow publishes
+with a trusted publisher, and a publish by hand has none. So once npm has a release of this
+package from the release workflow, an install that resolves `N.0.0` without the exclude
+fails with `ERR_PNPM_TRUST_DOWNGRADE`. pnpm reads only the first `trustPolicyExclude` entry
+that names a package, so keep one entry for it.
+
+Once npm has `N.0.1` or later from the release workflow, remove both excludes. In each
+repository, remove it in the pull request that moves the lockfile off `N.0.0` with
+`pnpm update @tibia.sh/tibiawiki-data --no-save`. Without `--no-save`, pnpm also raises the
+server's `^N` to the new version, such as `^N.0.1`, and the server's
+`test/data-package.test.ts` rejects that. Without the exclude, a lockfile still on `N.0.0`
+fails the next `pnpm dedupe`. pnpm 12.4.1 fails `update --no-save` with
+`ERR_PNPM_STRICT_MIN_RELEASE_AGE_REQUIRES_SAVE` whenever `minimumReleaseAge` is set
+([pnpm#14835](https://github.com/pnpm/pnpm/issues/14835)). Until `packageManager` names a
+pnpm with the fix, run `pnpm update @tibia.sh/tibiawiki-data` without `--no-save`, restore
+`^N` in the server's `package.json`, then run `pnpm install`. The lockfile moves and the range
+stays.
+
+**Verify the deadlock before relying on this.** On a scratch branch, set the server's
+`MCP_SCHEMA_VERSION` to N: its `test/data-package.test.ts` and regression sweep must
+fail. Here, set `SCHEMA_VERSION` and `version` to N against the schema-(N-1)
+devDependency: the suite must fail before anything is published. If either suite passes,
+it is not checking what this procedure assumes, so stop and find out why. Once npm has a
+release of this package from the release workflow, the server's install of the
+hand-published `N.0.0` fails with `ERR_PNPM_TRUST_DOWNGRADE` without the exclude from
+step 3. Do not try to reproduce that against the real registry, because it takes a real
+publish by hand.
+
+This repository's half was checked on 2026-09-13 for N = 4, on a clone. `pnpm test`
+fails at `the shipped index carries SCHEMA_VERSION`. With the index's
+`mcp_schema_version` row set to 4 as well, it fails at the server test instead, because
+the schema-3 server refuses a schema-4 index. Either way `npm publish` stops in
+`prepublishOnly` and packs nothing.
+
+The trust failure was checked on 2026-09-13 with pnpm 10.33.0, against a local stand-in
+for the registry and never the real one. With `3.0.1` from a trusted publisher and `4.0.0`
+published by hand after it, the server's install of `^4` and this repository's install of
+a server that depends on `^4` both fail with `ERR_PNPM_TRUST_DOWNGRADE`. An exclude for
+exactly `@tibia.sh/tibiawiki-data@4.0.0` lets both through.
+
 ## Licence
 
 `index.db` is adapted from TibiaWiki (https://tibia.fandom.com), whose text is

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tibia.sh/tibiawiki-data",
-  "version": "3.0.0",
+  "version": "3.0.2",
   "description": "The prebuilt TibiaWiki index served by @tibia.sh/tibiawiki-mcp. Its major version is the index schema version.",
   "type": "module",
   "license": "(CC-BY-SA-3.0 AND MIT)",
@@ -12,7 +12,7 @@
   "bugs": {
     "url": "https://github.com/tibia-sh/tibiawiki-data/issues"
   },
-  "packageManager": "pnpm@10.33.0",
+  "packageManager": "pnpm@12.4.1",
   "exports": {
     ".": {
       "types": "./dist/index.d.ts",
@@ -31,14 +31,15 @@
   "scripts": {
     "typecheck": "tsc --noEmit",
     "build": "tsc -p tsconfig.build.json",
-    "test": "pnpm build && pnpm typecheck && node --test test/*.test.ts",
+    "test": "pnpm build && pnpm typecheck && node --test --test-reporter=spec --test-reporter-destination=stdout --test-reporter=./test/min-tests.ts --test-reporter-destination=stderr test/*.test.ts",
     "build-index": "pnpm build && node scripts/build.ts",
     "smoke": "node scripts/smoke.mjs",
+    "oldest-consumer": "pnpm build && node scripts/oldest-consumer.ts",
     "prepublishOnly": "pnpm test"
   },
   "devDependencies": {
     "@modelcontextprotocol/client": "2.0.0",
-    "@tibia.sh/tibiawiki-mcp": "^0.1.0",
+    "@tibia.sh/tibiawiki-mcp": "^0.3.0",
     "@types/node": "24.13.3",
     "typescript": "7.0.2"
   }