@tibia.sh/tibiawiki-data 3.0.3 → 3.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +56 -315
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -1,350 +1,91 @@
1
1
  # tibiawiki-data
2
2
 
3
- The prebuilt TibiaWiki index served by
4
- [`@tibia.sh/tibiawiki-mcp`](https://github.com/tibia-sh/tibiawiki-mcp). It is one
5
- SQLite file, plus a module that gives its path and its schema version.
3
+ A snapshot of [TibiaWiki](https://tibia.fandom.com) as one SQLite file, published to npm as
4
+ `@tibia.sh/tibiawiki-data`. It is the index that
5
+ [`@tibia.sh/tibiawiki-mcp`](https://github.com/tibia-sh/tibiawiki-mcp) serves to MCP clients.
6
6
 
7
- ## Use
8
-
9
- ```js
10
- import { DB_PATH, SCHEMA_VERSION } from '@tibia.sh/tibiawiki-data';
11
- ```
12
-
13
- - `DB_PATH` is the absolute path to `index.db` inside the installed package.
14
- - `SCHEMA_VERSION` is the index's enrichment schema version (its `mcp_schema_version`
15
- row), and always equals this package's major version.
16
-
17
- The file is also exported as the subpath `@tibia.sh/tibiawiki-data/index.db`. The
18
- server finds the packaged index by resolving that subpath, so it must stay in
19
- `exports`. Without it, Node throws `ERR_PACKAGE_PATH_NOT_EXPORTED`.
7
+ You rarely install it yourself. The server depends on it, so installing the server brings the
8
+ index with it. To ask questions over the data without installing anything, add the hosted server
9
+ `https://mcp.tibia.sh/wiki` to your MCP client.
20
10
 
21
- ## The major version is the schema version
11
+ ## What is inside
22
12
 
23
- The first release is `3.0.0`, not `1.0.0`, because the server's `MCP_SCHEMA_VERSION`
24
- is `3`. Do not reset it. A server that reads schema N depends on `^N`, so npm refuses
25
- to install an index the server cannot read. Without that, the server would only find
26
- out at startup, and would then answer every query with an error. Releases within a
27
- major are data refreshes of the same schema.
13
+ `index.db` is about 18 MB and holds the wiki's structured data. Some of it:
28
14
 
29
- `SCHEMA_VERSION` is a literal in `src/index.ts`, and `pnpm test` asserts that it
30
- equals both the `package.json` major and the index's `mcp_schema_version` row.
31
- Change all three together, following
32
- [Bumping the schema version](#bumping-the-schema-version). The literal is deliberate.
33
- Derived from `package.json`, that assertion would compare a value with itself and could
34
- never fail, and it is the only thing that stops a release from shipping under the wrong
35
- major.
36
-
37
- ## Why the index is committed
38
-
39
- `index.db` is committed to this repository in plain git.
40
-
41
- - **A fresh clone is a complete package.** It has an index to test and to pack, so
42
- publishing a release packs the committed file and never needs a crawl.
43
- - **A data refresh is reviewable.** It is a pull request whose diff is the new
44
- `index.db`, so the file reviewed is the file published.
45
-
46
- **Measured cost, for the `3.0.0` index (2026-09-12):**
47
-
48
- | Measurement | Size |
15
+ | Table | Rows |
49
16
  |---|---|
50
- | `index.db` on disk | 18,042,880 bytes |
51
- | One committed build, as a git pack | 5.36 MiB |
52
- | The npm tarball | 5.38 MiB |
53
- | A second real build added to the same repository, after `git gc --aggressive` | +0.48 MiB |
54
-
55
- SQLite does not diff as text, but git's binary deltas are effective on it. The second
56
- build was generated the same day as the first, so a refresh after a week of wiki
57
- edits may delta less well. Budget for a full 5.4 MiB per committed refresh as the
58
- upper bound.
59
-
60
- **Why not Git LFS.** Plain git is self-contained: there is no LFS storage or bandwidth
61
- quota, every checkout gets the index without extra configuration, and a clone is
62
- everything needed to test and pack. Move to LFS only if clone times become a real
63
- complaint.
17
+ | `item` | 9,800 |
18
+ | `creature` | 2,193 |
19
+ | `npc` | 1,245 |
20
+ | `book` | 1,226 |
21
+ | `house` | 1,090 |
22
+ | `achievement` | 571 |
23
+ | `quest` | 370 |
24
+ | `spell` | 211 |
25
+
26
+ The counts are from the `3.0.3` snapshot. The tables come from
27
+ [tibiawiki-sql](https://github.com/Galarzaa90/tibiawiki-sql), which generates the file. The server
28
+ adds a few more for its own queries. Images are not included, only links to them.
64
29
 
65
- ## Identifying a release
66
-
67
- A release is a snapshot of a wiki that keeps changing. It can be identified, but not
68
- reproduced byte for byte. Three values identify it:
69
-
70
- - the package version;
71
- - `version` in the index's `database_info` table, which is the tibiawiki-sql
72
- generator version;
73
- - `generate_time` in the same table, which records when the index was generated.
74
- It is a timestamp, not a wiki revision. The generator records no revision.
75
-
76
- What each release was built from:
77
-
78
- | Release | Generator `version` | `generate_time` |
79
- |---|---|---|
80
- | `3.0.0` | `9.0.0` | `2026-09-12T19:53:53.020856+00:00` |
81
- | `3.0.1` | `9.0.0` | `2026-09-13T07:02:58.860376+00:00` |
82
-
83
- To read them from any index:
30
+ ## Use
84
31
 
85
32
  ```bash
86
- node -e "const { DatabaseSync } = require('node:sqlite'); const db = new DatabaseSync('index.db', { readOnly: true }); console.log(db.prepare(\"select key, value from database_info where key in ('version', 'generate_time')\").all())"
33
+ npm install @tibia.sh/tibiawiki-data
87
34
  ```
88
35
 
89
- The server reports the same two values. Its MCP instructions name both, and every
90
- tool response carries `generate_time` as `indexGeneratedAt`.
91
-
92
- ## How the index is built
36
+ The example needs Node 22.13 or later, for `node:sqlite`.
93
37
 
94
- The index is built by the server's own `build-index` command, run from this
95
- repository's devDependency. The server's gates decide whether a build is good enough.
96
- Those gates cover coverage, parse failures, image resolution and spell shapes, and
97
- they are defined and tested in the server. They are not repeated here.
38
+ ```js
39
+ import { DatabaseSync } from 'node:sqlite';
40
+ import { DB_PATH, SCHEMA_VERSION } from '@tibia.sh/tibiawiki-data';
98
41
 
99
- ```bash
100
- pnpm build-index
42
+ const db = new DatabaseSync(DB_PATH, { readOnly: true });
43
+ console.log(db.prepare('select title, hitpoints from creature where name = ?').get('Dragon Lord'));
101
44
  ```
102
45
 
103
- That builds `dist/`, then `scripts/build.ts` runs the devDependency's
104
- `tibiawiki-mcp build-index` with `TIBIAWIKI_MCP_DB` set to `DB_PATH`, so the build
105
- writes exactly the file this package ships and exports. From `0.3.1`, the server installs
106
- the generator from its own requirements file, where every Python dependency is pinned and
107
- hashed, so every build runs the same packages. Builds still set a 7-day PyPI cooldown,
108
- matching the one pnpm applies to npm packages. It is a backstop for a server older than
109
- `0.3.1`, which resolves those dependencies fresh on every build. You can override it with
110
- `UV_EXCLUDE_NEWER`.
111
-
112
- `build-index` needs [`uv`](https://docs.astral.sh/uv/) and network access to
113
- TibiaWiki. It validates the new index before replacing `index.db`, so a build that
114
- trips a gate exits non-zero and leaves the committed `index.db` as it was. While it
115
- works, it writes `.tibiawiki.db.<pid>.<hex>.tmp` next to the target, and SQLite keeps
116
- its journal beside that. `.gitignore` excludes both, and must never exclude
117
- `index.db`. The `3.0.0` build took just under six minutes, most of it the generator's
118
- crawl.
119
-
120
- `pnpm test` pins the generator too. It fails for an index whose `database_info` `version`
121
- is anything but `9.0.0`. The major version covers only the server's enrichment tables,
122
- and no version covers the tables tibiawiki-sql writes.
123
-
124
- A generator upgrade does not bump the major. If you installed any published server that depends
125
- on `^N`, your next install gets every new `N.x` of this package. So before every publish, and in
126
- CI, the `oldest-consumer` job installs the oldest and the newest published `^N` server from npm,
127
- each together with the packed candidate, and pages every item through each one's
128
- `tibia_find_items`. The oldest has the oldest serving code that still gets a new `N.x`. The newest
129
- is the one a fresh install gets, and newer serving code can refuse an index the oldest serves. When
130
- one server is both, the job sweeps it once. The gate passes only when, for each server, no page
131
- is an error, every page carries the candidate index's `generate_time`, and every item in the
132
- index comes back exactly once. `pnpm oldest-consumer` runs the same gate, and needs network
133
- access to npm.
134
-
135
- The sweep covers items only, not creatures, NPCs, quests or spells. So the `9.0.0` pin in
136
- `test/data.test.ts` stays as the explicit decision point for a generator upgrade.
46
+ 1. `DB_PATH` => the absolute path to `index.db` inside the installed package
47
+ 2. `SCHEMA_VERSION` => the version of the tables the server adds. It always equals this package's major version
137
48
 
138
- ### Drift
49
+ ## Versions
139
50
 
140
- `.github/workflows/drift.yml` rebuilds the index every Monday at 06:17 UTC, and when you
141
- run it by hand from the Actions tab. It digests the committed `index.db` with the
142
- devDependency's `tibiawiki-mcp index-digest`, runs `pnpm build-index`, digests the
143
- rebuilt index, and runs `pnpm test` against it. The digest covers what the server reads,
144
- and leaves out stamps that change on every run, such as `generate_time`. The run's log
145
- shows both digests.
51
+ The major version is the server's schema version, which is why the first release was `3.0.0`.
52
+ A server that reads schema 3 depends on `^3`, so npm never installs an index it cannot read.
53
+ Every release within a major is a newer snapshot. The tables the server adds keep their shape
54
+ within a major. The rest come from tibiawiki-sql, and `version` in `database_info` says which
55
+ release of it wrote them.
146
56
 
147
- When the digests match, the run ends green and opens nothing. When they differ, the
148
- content changed. The run pushes the rebuilt `index.db` to the `drift/index` branch, with
149
- `version` set to the next patch npm does not have. It opens a pull request carrying both
150
- digests, or updates the one already open. Merging that pull request publishes the new
151
- patch.
57
+ A snapshot says when it was taken. `generate_time` in the `database_info` table is the moment
58
+ the wiki was read, and `version` there is the tibiawiki-sql version that read it:
152
59
 
153
- - The pull request is opened with `GITHUB_TOKEN`, so its CI waits for you. Click
154
- "Approve workflows to run" on it, then review the pull request before you merge it.
155
- - The workflow never merges and never turns on auto-merge, so a bad day on the wiki can at
156
- most open a pull request.
157
- - A red run is a signal. A tripped gate, a failing test, an unreadable registry, or a
158
- `version` on `main` that npm does not list yet each end the run red, and nothing is
159
- pushed or opened. Find out why before the next run.
160
- - Each run that finds a change replaces `drift/index`, so an open pull request always
161
- carries the newest rebuild.
162
- - GitHub turns off a schedule after 60 days without activity in a public repository, and
163
- that stops the job without a red run. Turn it back on from the Actions tab.
60
+ ```js
61
+ db.prepare("select key, value from database_info where key in ('version', 'generate_time')").all();
62
+ ```
164
63
 
165
- ## The devDependency on the server
64
+ The [releases page](https://github.com/tibia-sh/tibiawiki-data/releases) says what each snapshot
65
+ holds: those two values, and how the row counts moved since the release before it.
166
66
 
167
- `@tibia.sh/tibiawiki-mcp` is a devDependency for three jobs: its `build-index` produces
168
- the index, its `index-digest` tells the drift job whether a rebuild changed it, and its
169
- `serve` validates it, in `pnpm test` here and in `pnpm smoke` against an installed copy.
67
+ ## How it stays current
170
68
 
171
- **When to bump it.** On a `0.x` version, `^0.3.0` means `>=0.3.0 <0.4.0`. Left alone,
172
- it pins every rebuild to the 0.3 generator and its gates while the server moves on.
173
- Bump it whenever the server's indexer changes: `build-index`, its enrichment, its
174
- gates, or the schema. Write the new range by hand. This repository saves exact
175
- versions, so `pnpm add` records a pin instead.
69
+ A workflow rebuilds the index every Monday. When the wiki's content changed, it opens a pull
70
+ request with the new `index.db`. A maintainer reviews and merges it. The merge publishes the next
71
+ patch version to npm with a provenance attestation. The hosted server picks the new version
72
+ up by itself within minutes.
176
73
 
177
- **The dependency cycle is intentional.** The server depends on this package, and this
178
- package devDepends on the server. npm and pnpm allow it because this side is
179
- dev-only and never resolved at runtime. Do not "fix" it. Because the server depends on
180
- this package, `node_modules` here also holds a published copy of this package,
181
- installed as the server's dependency. The server's default index resolution could
182
- find that copy instead of `index.db`. So the test always passes `TIBIAWIKI_MCP_DB`
183
- explicitly, and checks that the answer's `indexGeneratedAt` matches `index.db`.
74
+ The file you install is the file that was reviewed. It is committed to this repository, and a
75
+ release packs it without rebuilding.
184
76
 
185
- ## Development
77
+ ## Contributing
186
78
 
187
- Requires Node 22.18 or later, because the tests run TypeScript directly, and pnpm
188
- 12.4.1, pinned in `packageManager`.
79
+ Wrong or missing data belongs on TibiaWiki. Fix the wiki page and the next snapshot carries it.
80
+ Problems with the package itself are welcome as issues here.
189
81
 
190
82
  ```bash
191
83
  pnpm install --frozen-lockfile
192
84
  pnpm test
193
85
  ```
194
86
 
195
- `pnpm test` does three things, in this order:
196
-
197
- 1. It builds `dist/`.
198
- 2. It typechecks `src/`, `test/` and `scripts/`, the JavaScript in `scripts/`
199
- included. This must come after the build: the test imports this package by its own
200
- name, so it typechecks against the built declarations, as a consumer does.
201
- 3. It runs every `test/*.test.ts`. `test/data.test.ts` spawns the server from the
202
- devDependency against `index.db`, and makes a real query. The other files check the
203
- workflows, the smoke check, the oldest-consumer gate's decisions and the test floor, with
204
- no network.
205
-
206
- The run fails when fewer than `MIN_TESTS` tests pass. `node --test` still exits 0 for a
207
- file that declares no tests, for a skipped test, and for a `--test-name-pattern` that
208
- filters tests away, one inherited through `NODE_OPTIONS` included. Without the floor, an
209
- emptied test file would pass the gate every publish runs. Adding a test needs no change. When
210
- you remove or skip one on purpose, lower `MIN_TESTS` in `test/min-tests.ts` in the same
211
- commit.
212
-
213
- `prepublishOnly` runs `pnpm test` too, so publishing from the directory always builds
214
- `dist/` first.
215
-
216
- After changing `version` in `package.json`, run `pnpm install` before `pnpm test`.
217
- `verifyDepsBeforeRun` treats a version change as a workspace change and refuses to
218
- run scripts until you do.
219
-
220
- The tarball ships `index.db` and `dist/`, plus the `package.json`, `README.md` and
221
- `LICENSE` that npm always adds. `@tibia.sh/*` packages are exempt from this
222
- repository's seven-day install cooldown. `pnpm-workspace.yaml` says why.
223
-
224
- ### Checking a release, before and after publishing
225
-
226
- ```bash
227
- pnpm smoke ./tibia.sh-tibiawiki-data-3.0.0.tgz # a packed tarball, before publishing
228
- pnpm smoke @tibia.sh/tibiawiki-data@3.0.0 # the published version, after
229
- ```
230
-
231
- `pnpm smoke` installs the package under test into a throwaway directory, together with
232
- the server and the MCP client at the versions `package.json` names, and runs
233
- `test/data.test.ts` there. Inside that directory the test's imports land on the
234
- installed package and it spawns the installed server, so it checks the artefact rather
235
- than this checkout. That is why the test file imports only node builtins and packages
236
- by name.
237
-
238
- ## Releases
239
-
240
- Merging a commit to `main` publishes its `package.json` `version` if npm does not have
241
- that version yet. On every push to `main`, `.github/workflows/release.yml` runs the
242
- `oldest-consumer` gate from [How the index is built](#how-the-index-is-built), and its release
243
- job waits for that gate. The release job then asks npm whether it lists that exact version. If
244
- it does, the run publishes nothing and ends green. Every merge that leaves `version` alone,
245
- and passes the gate, ends this way. If it does not, the run installs from the lockfile, runs
246
- `pnpm test`, and runs `npm publish`.
247
-
248
- - The gate runs on every push, whether the run publishes or not, so an index that breaks the
249
- oldest or the newest published server turns the run red even when nothing is published.
250
- - The check is for existence, never a comparison with `latest`. A revert leaves
251
- `version` below `latest`, and `npm publish` moves `latest` itself.
252
- - A registry the check cannot read fails the run. It is never taken for a missing
253
- version.
254
- - The run packs the committed `index.db` and never rebuilds it, so the file published is
255
- the file reviewed in the pull request.
256
- - It publishes through npm trusted publishing, so no npm token exists to leak, and npm
257
- attaches a provenance attestation for the merged commit. The trusted publisher is
258
- registered for the file name `release.yml`, and renaming the file breaks publishing
259
- with no warning.
260
- - Every pull request runs the same `pnpm test` and the same gate, in `.github/workflows/ci.yml`.
261
-
262
- Nothing is tagged, so a failed publish leaves nothing stranded. When a release run fails,
263
- follow [docs/RELEASING.md](docs/RELEASING.md).
264
-
265
- ### Bumping the schema version
266
-
267
- **Not yet exercised.** No schema bump has gone through this procedure.
268
-
269
- A bump from N-1 to N cannot pass the automated gates. This package's `N.0.0` runs
270
- `pnpm test` against its devDependency server, which has to read schema N, so it needs a
271
- schema-N server on the registry. That server's CI needs this package's `N.0.0` on the
272
- registry: its `^N` dependency has to install, and its `test/data-package.test.ts` and
273
- regression sweep read the installed index. So one side is published outside its
274
- pipeline. It is this package, because no published server installs `N.0.0`. Server
275
- `0.1.0` does not use this package, and every later server depends on a major below N.
276
-
277
- 1. On the server's schema-N branch, run `pnpm build`, then `npm pack`. Build this
278
- repository's index with that tarball's `build-index`. The published server stamps the
279
- index N-1, which this repository's tests reject. Then cross-validate: install each
280
- repository's counterpart from the other's local tarball, and run both full test
281
- suites. Neither repository builds `dist/` when it packs, so build before every
282
- `npm pack`, or the tarball carries a stale `dist/` or none.
283
- 2. The maintainer publishes this package's `N.0.0` by hand, from the validated tarball.
284
- `npm publish` runs no lifecycle scripts for a tarball, so step 1 is the only gate it
285
- gets. The release carries no provenance and no trusted publisher.
286
- 3. The server's pull request sets `MCP_SCHEMA_VERSION` to N and its dependency range to
287
- `^N`. It also adds `trustPolicyExclude` for exactly `@tibia.sh/tibiawiki-data@N.0.0`
288
- to the server's `pnpm-workspace.yaml`, with the reason in a comment:
289
-
290
- ```yaml
291
- # @tibia.sh/tibiawiki-data N.0.0 was published by hand, so it has no trusted publisher.
292
- trustPolicyExclude:
293
- - '@tibia.sh/tibiawiki-data@N.0.0'
294
- ```
295
-
296
- Its CI passes against the registry, and its release PR publishes it through the
297
- server's pipeline.
298
- 4. This repository's pull request commits that exact `index.db`, with `SCHEMA_VERSION` N,
299
- `version` `N.0.0`, and the devDependency moved to the new server. The new server
300
- depends on `^N`, so the install here resolves the hand-published `N.0.0` too, and the
301
- pull request adds the same exclude to this repository's `pnpm-workspace.yaml`. Its CI
302
- passes, and merging it publishes nothing, because npm already has `N.0.0`.
303
-
304
- Do not merge a data refresh here between steps 2 and 4. `main` is still on N-1 then, and
305
- npm refuses to publish a version below `N.0.0` without a dist-tag, so its release run
306
- fails.
307
-
308
- Both repositories set `trustPolicy: no-downgrade`, which makes pnpm refuse a version with
309
- weaker trust evidence than any version published before it. The release workflow publishes
310
- with a trusted publisher, and a publish by hand has none. So once npm has a release of this
311
- package from the release workflow, an install that resolves `N.0.0` without the exclude
312
- fails with `ERR_PNPM_TRUST_DOWNGRADE`. pnpm reads only the first `trustPolicyExclude` entry
313
- that names a package, so keep one entry for it.
314
-
315
- Once npm has `N.0.1` or later from the release workflow, remove both excludes. In each
316
- repository, remove it in the pull request that moves the lockfile off `N.0.0` with
317
- `pnpm update @tibia.sh/tibiawiki-data --no-save`. Without `--no-save`, pnpm also raises the
318
- server's `^N` to the new version, such as `^N.0.1`, and the server's
319
- `test/data-package.test.ts` rejects that. Without the exclude, a lockfile still on `N.0.0`
320
- fails the next `pnpm dedupe`. pnpm 12.4.1 fails `update --no-save` with
321
- `ERR_PNPM_STRICT_MIN_RELEASE_AGE_REQUIRES_SAVE` whenever `minimumReleaseAge` is set
322
- ([pnpm#14835](https://github.com/pnpm/pnpm/issues/14835)). Until `packageManager` names a
323
- pnpm with the fix, run `pnpm update @tibia.sh/tibiawiki-data` without `--no-save`, restore
324
- `^N` in the server's `package.json`, then run `pnpm install`. The lockfile moves and the range
325
- stays.
326
-
327
- **Verify the deadlock before relying on this.** On a scratch branch, set the server's
328
- `MCP_SCHEMA_VERSION` to N: its `test/data-package.test.ts` and regression sweep must
329
- fail. Here, set `SCHEMA_VERSION` and `version` to N against the schema-(N-1)
330
- devDependency: the suite must fail before anything is published. If either suite passes,
331
- it is not checking what this procedure assumes, so stop and find out why. Once npm has a
332
- release of this package from the release workflow, the server's install of the
333
- hand-published `N.0.0` fails with `ERR_PNPM_TRUST_DOWNGRADE` without the exclude from
334
- step 3. Do not try to reproduce that against the real registry, because it takes a real
335
- publish by hand.
336
-
337
- This repository's half was checked on 2026-09-13 for N = 4, on a clone. `pnpm test`
338
- fails at `the shipped index carries SCHEMA_VERSION`. With the index's
339
- `mcp_schema_version` row set to 4 as well, it fails at the server test instead, because
340
- the schema-3 server refuses a schema-4 index. Either way `npm publish` stops in
341
- `prepublishOnly` and packs nothing.
342
-
343
- The trust failure was checked on 2026-09-13 with pnpm 10.33.0, against a local stand-in
344
- for the registry and never the real one. With `3.0.1` from a trusted publisher and `4.0.0`
345
- published by hand after it, the server's install of `^4` and this repository's install of
346
- a server that depends on `^4` both fail with `ERR_PNPM_TRUST_DOWNGRADE`. An exclude for
347
- exactly `@tibia.sh/tibiawiki-data@4.0.0` lets both through.
87
+ [docs/MAINTAINING.md](https://github.com/tibia-sh/tibiawiki-data/blob/main/docs/MAINTAINING.md) covers how the index is built and tested, and
88
+ [docs/RELEASING.md](https://github.com/tibia-sh/tibiawiki-data/blob/main/docs/RELEASING.md) covers publishing.
348
89
 
349
90
  ## Licence
350
91
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@tibia.sh/tibiawiki-data",
3
- "version": "3.0.3",
3
+ "version": "3.0.4",
4
4
  "description": "The prebuilt TibiaWiki index served by @tibia.sh/tibiawiki-mcp. Its major version is the index schema version.",
5
5
  "type": "module",
6
6
  "license": "(CC-BY-SA-3.0 AND MIT)",