wiki-entity 0.7.0 → 1.0.0

package/CHANGELOG.md

@@ -0,0 +1,172 @@
+# Changelog
+
+All notable changes to this project are documented here.
+This project adheres to [Semantic Versioning](https://semver.org/).
+
+## 1.0.0
+
+A full modernization pass: the package is now dependency-free, dual ESM/CJS,
+strictly typed, and covered by a real test suite. Several long-standing data
+bugs are fixed, and the generated Wikidata tables have been rebuilt.
+
+### Breaking changes
+
+- **Node.js >= 20.19 is required.** `axios` is gone; the package uses the
+  platform `fetch`.
+- **Dual ESM/CommonJS build.** The package now has an `exports` map and ships
+  from `dist/`. Deep imports such as
+  `require("wiki-entity/lib/simpleEntity/getEntityCountry")` no longer resolve.
+- **The public API is down to 14 exports.** It had grown to 40, many of which
+  were internals with no standalone use. Removed: the array helpers (`chunk`,
+  `uniq`), the id predicates (`isItemId`, `isPropertyId`, `isEntityId`), the
+  claim renderers (`stringifyTime`, `stringifyCoordinates`), the pieces of
+  `convertToSimpleEntity` (`getEntityData`, `getEntityCountryCode`,
+  `getEntityTypeByExtract`, and the two `getEntityType` variants), the
+  low-level claim simplifiers (`simplifyClaim`, `simplifyClaims`,
+  `simplifyPropertyClaims`, `exploreEntityClaims`), the deprecated
+  `WikipediaApi` builder and its `getExtract`/`getExtracts`/`getRedirects`
+  wrappers (use `queryPages`), the singular `getEntityTypesByName` (use
+  `getEntityTypesByNames`), and the constants `KNOWN_TYPE_PREFIXES`,
+  `SIMPLE_ENTITY_TYPES` and `WIKIDATA_PROPS`.
+- **`redirect` is now `followEntityRedirects: boolean`.** `redirect: "yes"|"no"`
+  sat one letter away from the unrelated `redirects: boolean`, which fetches
+  Wikipedia redirect titles. There is no alias; rename at the call site.
+- **`InvalidParamsError` is gone**; bad arguments now throw a plain `TypeError`.
+- **`WikidataPropsParam` (enum) and `WIKIDATA_PROPS` (const) are gone.** `props`
+  takes plain strings, typed by the `WikidataProp` union.
+- **`PlainObject`, `AnyPlainObject` and `StringPlainObject` are gone** — they
+  were aliases for `Record<string, T>`. The `Raw*` claim types no longer leak
+  either; `simplifyEntity` takes the documented `WikibaseEntityJson` shape.
+- `ApiResult` is renamed `WikipediaPage`.
+- **`SimpleEntity.lang` and `SimpleEntity.wikiDataId` are now required.**
+  `convertToSimpleEntity` always sets both, so typing them optional forced
+  callers into null checks that could never fire.
+- **`SimpleEntity.abbr` is removed.** Nothing in the package ever assigned it.
+- `WikiEntityToEntityOptions` is renamed `ConvertToSimpleEntityOptions`, after
+  the function it belongs to rather than one that no longer exists.
+- **`pageid` now always means the Wikipedia page id.** It used to fall back to
+  Wikidata's own page id whenever the Wikipedia lookup was skipped
+  (`wikiPageId: false`, no sitelink, missing article), which silently produced a
+  wrong `wikiPageId` in `SimpleEntity`. It is now left unset in those cases.
+- **Deprecated statements are dropped** and **preferred statements sort first**,
+  so `claims[p].values[0]` is the value Wikidata recommends. Pass
+  `keepDeprecatedClaims` to `simplifyEntity` to keep them.
+- **`wikibase-property` claim values** are now the property id (`"P31"`) rather
+  than the raw Wikibase object, which used to stringify to `"[object Object]"`.
+- `simplifySitelinks` and friends return `{}` instead of `null` for empty input.
+- `convertToSimpleEntity` omits empty optional fields (`categories`, `data`,
+  `types`) instead of setting them to `[]`/`{}`.
+- `getEntityType`, `getEntityTypeByExtract` and related helpers return
+  `undefined` rather than `null` when nothing matches.
+- `getManyEntities` returns `{}` instead of `null` when nothing was found —
+  the `null` used to crash the caller.
+
+### Migrating
+
+`getExtract`, `getExtracts` and `getRedirects` are replaced by `queryPages`,
+which batches titles and follows paginated responses instead of sending one
+request and truncating:
+
+```ts
+// before                                    // now
+getExtracts({ lang, titles, sentences: 2 }); // queryPages({ lang, titles, extract: 2 })
+getExtract(lang, title, 2); // queryPages({ lang, titles: [title], extract: 2 })
+getRedirects(lang, title); // queryPages({ lang, titles: [title], redirects: true })
+```
+
+`queryPages` returns `{ pages, resolved, requestedTitleOf }`. `pages` is not in
+the order the titles were given, so use `requestedTitleOf` to map a result back
+to the title you asked for. See the README for details.
+
+The `WikipediaApi` builder maps onto the same call:
+
+```ts
+// before
+new WikipediaApi().extract(2).redirects().query("en", { titles: "A|B" });
+// now
+queryPages({ lang: "en", titles: ["A", "B"], extract: 2, redirects: true });
+```
+
+### Fixed
+
+- **Labels went missing for a large share of items.** Wikidata has been moving
+  names that are identical across languages to the `mul` ("multiple languages")
+  code and deleting the per-language duplicates, which left `label` undefined
+  for entities such as Albert Einstein, Douglas Adams and iPhone 5. Labels and
+  aliases now fall back to `mul`, and `mul` is requested automatically alongside
+  any `languages` filter.
+- **Only the first entity got an extract.** Without `exintro`, MediaWiki
+  silently lowers `exlimit` to 1, so `getEntities({ titles: [a, b, c], extract })`
+  returned an extract for `a` only.
+- **`languages` was ignored.** It was serialized as PHP array syntax
+  (`languages[]=en&languages[]=ru`), which MediaWiki rejects with a warning and
+  then returns _every_ language for.
+- **BCE dates lost their sign.** `-0044-03-15` (Julius Caesar's death) rendered
+  as `0044-03-15`.
+- **Coordinates lost integer digits.** Trailing-zero trimming turned latitude
+  `10` into `1`.
+- **`claims: "property"` never worked.** Property entities (`P31`, …) were
+  filtered out as invalid ids, so property labels and descriptions were never
+  populated.
+- **The `types` prefix filter was ignored.** `types: ["dbo", "schema"]` computed
+  the allowed prefixes and then never applied them.
+- **Wikipedia results were mismatched to entities.** MediaWiki normalizes titles
+  and follows redirects, so the returned title often differs from the requested
+  one; results are now mapped back through the normalization and redirect
+  chains. Previously this attached data to the wrong entity, or threw.
+- **Titles that are redirects found no entity.** Single-title lookups now use
+  `normalize=1`, and multi-title lookups resolve anything Wikidata missed
+  through Wikipedia's canonical title.
+- **Romanian extract patterns never matched.** They relied on `\b`, which is
+  ASCII-only, so words ending in a diacritic (`oraș`, `cântăreț`, `comună`)
+  could never match. Patterns are now Unicode-aware; English patterns added.
+- More than 50 Wikipedia titles are now batched instead of being sent in one
+  over-long request, and paginated (`continue`) responses are followed, so
+  `redirects` and `categories` are no longer truncated at the first page.
+- `getEntityData` no longer throws on a structured claim value with no readable
+  rendering; it skips it.
+- DBpedia is queried over HTTPS, with the resource IRI properly escaped.
+
+### Added
+
+- `getSimpleEntities(params, options?)` — `getEntities` plus
+  `convertToSimpleEntity` in one call, taking the language from `params` so it
+  cannot diverge from the language the entities were fetched in.
+- `AbortSignal` support on every public call (`signal` param).
+- Automatic retries with exponential backoff and jitter on 429/5xx/network
+  errors, honouring `Retry-After`.
+- Typed error classes: `WikiEntityError`, `HttpError`, `ApiError`,
+  `InvalidParamsError`.
+- `getEntityTypesByNames()` resolves DBpedia types for many entities in a single
+  `VALUES` query instead of one request per entity.
+- `queryPages()` — the batching, continuation-following Wikipedia query used
+  internally, exported for direct use.
+- `setDbpediaEndpoint()` / `getDbpediaEndpoint()` to use a DBpedia mirror.
+- `rank` on claim values; `descriptions` map on entities; `datatype` on property
+  entities.
+- `chunk`, `isItemId`, `isPropertyId`, `isEntityId` utilities.
+- Unit tests (fully mocked) and an integration suite that runs against the live
+  APIs, plus ESLint, Prettier and strict TypeScript.
+
+### Changed
+
+- **Claim resolution is batched.** `claims: "item"` used to issue one request
+  per entity, serially; all referenced items across all entities are now fetched
+  together, 50 per request.
+- **Regenerated `data/entity-types.json` and `data/countries.json`** from the
+  live Wikidata Query Service. The type tables dated from 2018 and missed how
+  Wikidata now models products, recurring events and administrative entities.
+  The generator expands the class graph breadth-first with batched queries, and
+  verifies known classifications before writing.
+- The type table is delta-encoded, cutting it from 456 KB to ~130 KB and
+  expanding lazily on first use.
+- `isValidWikiId` is deprecated in favour of `isItemId` / `isEntityId`.
+
+## 0.7.0
+
+- Added a default `User-Agent` built from `package.json`, plus `setUserAgent()`
+  and the `WIKI_ENTITY_USER_AGENT` environment variable.
+
+## 0.6.0
+
+- Added `mapRedirects()`.

package/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2016-present, Dumitru Cantea
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.

package/README.md

@@ -1,24 +1,52 @@
 # wiki-entity
 
-Wiki(pedia/data) entity extractor.
+Fetch and normalize entities from **Wikidata**, **Wikipedia** and **DBpedia**.
+
+- Zero runtime dependencies — built on `fetch`, ESM + CommonJS, fully typed.
+- Batches, retries and rate-limit aware by default.
+- Flattens Wikidata's deeply nested claim format into something you can read.
+
+```bash
+npm install wiki-entity
+```
+
+Requires Node.js >= 20.19.
 
 ## Usage
 
+```ts
+import { getEntities, setUserAgent } from "wiki-entity";
+
+setUserAgent("MyApp/1.0 (https://myapp.example; contact@myapp.example)");
+
+// by Wikipedia article title
+const [europe] = await getEntities({ language: "en", titles: ["Europe"] });
+
+// by Wikidata id, with a Wikipedia summary and resolved claim labels
+const [einstein] = await getEntities({
+  language: "en",
+  ids: ["Q937"],
+  extract: 2,
+  claims: "all"
+});
+
+einstein.label; // "Albert Einstein"
+einstein.extract; // "Albert Einstein was a German-born theoretical physicist…"
+einstein.claims.P569.label; // "date of birth"
+einstein.claims.P569.values[0].value_string; // "1879-03-14"
 ```
-import { getEntities } from 'wiki-entity';
 
-// get Europe by title
-getEntities({ language: 'en', titles: ['Europe'] }).then(entities => {});
+CommonJS works too:
 
-// get Europe by id
-getEntities({ language: 'en', ids: ['Q46'] }).then(entities => {});
+```js
+const { getEntities } = require("wiki-entity");
 ```
 
 ## User-Agent
 
-[Wikimedia's User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy) requires every client to send an identifying `User-Agent` header. Generic UAs are rate-limited or blocked (HTTP `429`).
-
-Since `0.7.0`, `wiki-entity` sends a default UA built from `package.json`, but you should override it with your app's name and a contact URL/email:
+[Wikimedia's User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy)
+requires every client to identify itself. Generic agents get throttled or
+blocked with HTTP `429`. Set yours once at startup:
 
 ```ts
 import { setUserAgent } from "wiki-entity";
@@ -26,115 +54,285 @@ import { setUserAgent } from "wiki-entity";
 setUserAgent("MyApp/1.0 (https://myapp.example; contact@myapp.example)");
 ```
 
-Or set it via environment variable:
+or via the environment:
 
-```
+```bash
 WIKI_ENTITY_USER_AGENT="MyApp/1.0 (https://myapp.example; contact@myapp.example)"
 ```
 
 ## API
 
-### getEntities(params): Promise<WikiEntity[]>
+### `getEntities(params): Promise<WikiEntity[]>`
+
+Fetches entities from Wikidata and enriches them from Wikipedia and DBpedia.
+Results follow the order of the requested `ids`/`titles`; entities that do not
+exist are omitted.
+
+| Param                   | Type                  | Default | Description                                                                                    |
+| ----------------------- | --------------------- | ------- | ---------------------------------------------------------------------------------------------- |
+| `ids`                   | `string[]`            | —       | Wikidata ids, max 500. Either this or `titles` is required.                                    |
+| `titles`                | `string[]`            | —       | Wikipedia article titles in `language`, max 500.                                               |
+| `language`              | `string`              | `"en"`  | Language of `titles` and of the resulting `label`/`description`.                               |
+| `languages`             | `string[]`            | —       | Extra languages to populate `labels` with.                                                     |
+| `props`                 | `string[]`            | all     | `info`, `sitelinks`, `aliases`, `labels`, `descriptions`, `claims`, `datatype`.                |
+| `claims`                | `string`              | `none`  | `none`, `item`, `property` or `all` — how deeply to resolve claim labels.                      |
+| `extract`               | `number`              | —       | Sentences of the Wikipedia lead section to fetch.                                              |
+| `types`                 | `boolean \| string[]` | `false` | `true` for DBpedia ontology types, or an array of prefixes to keep (e.g. `["dbo", "schema"]`). |
+| `redirects`             | `boolean`             | `false` | Titles of _Wikipedia articles_ that redirect to this entity.                                   |
+| `categories`            | `boolean`             | `false` | The article's non-hidden categories.                                                           |
+| `wikiPageId`            | `boolean`             | `true`  | Fetch the Wikipedia `pageid`.                                                                  |
+| `httpTimeout`           | `number`              | `15000` | Per-request timeout in milliseconds.                                                           |
+| `signal`                | `AbortSignal`         | —       | Cancels every underlying request.                                                              |
+| `followEntityRedirects` | `boolean`             | `true`  | Resolve _Wikidata items_ that were merged into another item.                                   |
+
+`extract`, `redirects` and `categories` need the entity to have a sitelink for
+`language`, so keep `sitelinks` in `props` when you narrow it.
+
+Note `redirects` and `followEntityRedirects` are unrelated: the first fetches
+the titles of Wikipedia articles pointing at this entity, the second controls
+whether a merged-away Wikidata item resolves to the item that replaced it.
+
+### `mapRedirects(titles, lang): Promise<Record<string, string>>`
+
+Resolves Wikipedia redirect titles to the articles they point at. Only titles
+that really are redirects appear in the result.
 
-Gets entities from wikidata and wikipedia.
+```ts
+await mapRedirects(["Brashov"], "ro"); // { Brashov: "Brașov" }
+```
 
-#### params
+### `getSimpleEntities(params, options?): Promise<SimpleEntity[]>`
 
-Required. `ids` or `titles` require. Params properties:
+`getEntities` followed by `convertToSimpleEntity`, in one call. Takes the same
+params, plus an optional `{ defaultType }`.
 
-- **ids** :*string[]* - wikidata ids; (max 500)
-- **titles** :*string[]* - wikipedia article titles; (max 500)
-- **language** :*string* - language of `titles` param and of the result object info; default: `en`
-- **props** :*string[]* - entity props to get. Can by: `info`|`sitelinks`|`aliases`|`labels`|`descriptions`|`claims`|`datatype`;
-- **claims** :*string* - How to resolve the claims. Can be: `none`, `all`, `item`, `property`. Default: `none`. `all` resolves `item` and `property` types.
-- **extract** :*number* - Sentences in the extract. Default: `0`. Works only if `sitelinks` is present.
-- **types** :*boolean* | *string*[] - `true` to get entity types. Filter types by [prefixes](https://dbpedia.org/sparql?nsdecl). Example: [`dbo`, `schema`] will return only types defined by `dbpedia.org/ontology/` and `schema.org`. Default: `false`.
-- **redirects** : *boolean* - get wikipedia redirects titles. Default: `false`. Works only if `sitelinks` is present.
-- **categories** : *boolean* - get wikipedia article categories. Default: `false`. Works only if `sitelinks` is present.
-- **httpTimeout**: *number* - http requests timeout
-- **languages**: *string[]* - languages for `WikiEntity.labels`
-- **wikiPageId**: *boolean* - get wikiPageId, Default `true`
+```ts
+const entities = await getSimpleEntities({
+  language: "en",
+  ids: ["Q937"],
+  types: true,
+  extract: 2
+});
+
+entities[0].name; // "Albert Einstein"
+entities[0].type; // SimpleEntityType.PERSON
+```
 
-### convertToSimpleEntity(wikiEntity: WikiEntity): SimpleEntity
+Prefer this over calling the two by hand: it takes the language from `params`,
+so it cannot drift. Converting with a different language than you fetched with
+silently mixes them — `name` and `about` come from the fetch, while
+`wikiPageTitle` is read from the sitelinks of whichever language you passed.
 
-Convert a complex WikiEntity object to SimpleEntity object.
+### `convertToSimpleEntity(wikiEntity, lang, options?): SimpleEntity`
 
-## WikiEntity
+Flattens a single `WikiEntity` into the compact shape. Use it when you already
+have entities from `getEntities`.
 
-`WikiEntity` is a simple version of an item returned by Wikidata API. It also may include some extra properties from Wikipadia API.
+### Escape hatches
+
+For what `getEntities` does not cover. Everything else in the package is an
+implementation detail and may change without a major release.
+
+| Export                                      | Purpose                                                         |
+| ------------------------------------------- | --------------------------------------------------------------- |
+| `queryPages(options)`                       | Query Wikipedia articles directly — batched, continuation-safe. |
+| `simplifyEntity(lang, raw, options?)`       | Flatten a raw `wbgetentities` entity you fetched yourself.      |
+| `getEntityTypesByNames(names, options?)`    | DBpedia ontology types for English Wikipedia titles.            |
+| `setDbpediaEndpoint` / `getDbpediaEndpoint` | Point type lookups at a DBpedia mirror.                         |
+| `setUserAgent` / `getUserAgent`             | The `User-Agent` sent with every request.                       |
+
+#### `queryPages(options)`
+
+Talks to Wikipedia only — no Wikidata lookup, no entity ids. Use it when you
+have article titles and want their summaries, redirects or categories.
 
 ```ts
+import { queryPages } from "wiki-entity";
 
-type WikiEntity = {
-    id: string;
-    label?: string;
-    labels?: PlainObject<string>;
-    description?: string;
-    extract?: string;
-    pageid?: number;
-    aliases?: string[];
-    sitelinks?: PlainObject<string>;
-    claims?: PlainObject<WikidataProperty>;
-    types?: string[];
-    redirects?: string[];
-    categories?: string[];
-    redirectsToId?: string;
-    redirectsFromId?: string;
+const { pages } = await queryPages({
+  lang: "en",
+  titles: ["Chișinău", "Italy"],
+  extract: 2,
+  followRedirects: true
+});
+
+for (const page of pages) {
+  console.log(page.pageid, page.title, page.extract);
 }
+```
+
+| Option            | Type          | Description                                      |
+| ----------------- | ------------- | ------------------------------------------------ |
+| `lang`            | `string`      | Wikipedia language code, e.g. `"en"`. Required.  |
+| `titles`          | `string[]`    | Article titles. Required; batched automatically. |
+| `extract`         | `number`      | Sentences of the lead section to fetch.          |
+| `redirects`       | `boolean`     | Titles of articles redirecting to each page.     |
+| `categories`      | `boolean`     | Each article's non-hidden categories.            |
+| `followRedirects` | `boolean`     | Resolve titles that are themselves redirects.    |
+| `httpTimeout`     | `number`      | Per-request timeout in milliseconds.             |
+| `signal`          | `AbortSignal` | Cancels every underlying request.                |
+
+It handles the parts of the MediaWiki API that are easy to get wrong: titles are
+batched (50 per request, or 20 when `extract` is set, since the API refuses more
+intro extracts than that), and paginated `continue` responses are followed, so
+long `redirects` and `categories` lists are not truncated at the first page.
+
+**`pages` is not in the order you asked for**, and MediaWiki normalizes titles
+and follows redirects, so the title that comes back may not be the one you sent.
+Two maps tie the results back to your input:
+
+```ts
+const { pages, resolved, requestedTitleOf } = await queryPages({
+  lang: "en",
+  titles: ["Kishinev"],
+  categories: true,
+  followRedirects: true
+});
+
+pages[0].title; // "Chișinău" — the redirect target
+resolved.get("Kishinev"); // { title: "Chișinău", redirected: true }
+requestedTitleOf.get("Chișinău"); // "Kishinev"
+```
+
+Earlier releases had `getExtract`, `getExtracts` and `getRedirects` for this.
+They were thin wrappers that only ever handled one request's worth of titles:
+
+```ts
+// before                                    // now
+getExtracts({ lang, titles, sentences: 2 }); // queryPages({ lang, titles, extract: 2 })
+getExtract(lang, title, 2); // queryPages({ lang, titles: [title], extract: 2 })
+getRedirects(lang, title); // queryPages({ lang, titles: [title], redirects: true })
+```
+
+`getExtract` returned a single object or `null`; the equivalent is
+`(await queryPages({ lang, titles: [title], extract: 2 })).pages[0] ?? null`.
+
+### Errors
+
+All errors extend `WikiEntityError`:
+
+- `HttpError` — non-2xx status, network failure or timeout. `status`, `url` and
+  `retryable` describe it; 429 and 5xx are retried automatically.
+- `ApiError` — the request succeeded but MediaWiki reported an error (`code`).
+
+Bad arguments throw a plain `TypeError`.
+
+DBpedia types are best-effort enrichment: if the endpoint is unreachable,
+`types` is simply left unset instead of failing the call.
+
+## WikiEntity
+
+`WikiEntity` is a flattened Wikidata item, plus the extras pulled from
+Wikipedia and DBpedia.
+
+```ts
+type WikiEntity = {
+  id: string;
+  label?: string;
+  labels?: Record<string, string>;
+  description?: string;
+  descriptions?: Record<string, string>;
+  aliases?: string[];
+  sitelinks?: Record<string, string>;
+  claims?: Record<string, WikidataProperty>;
+  /** Wikipedia page id (not Wikidata's). */
+  pageid?: number;
+  extract?: string;
+  types?: string[];
+  redirects?: string[];
+  categories?: string[];
+  redirectsToId?: string;
+  redirectsFromId?: string;
+};
 
 type WikidataProperty = {
-    id: string;
-    label?: string;
-    description?: string;
-    values: WikidataPropertyValue[];
-}
+  id: string;
+  label?: string;
+  description?: string;
+  values: WikidataPropertyValue[];
+};
 
 type WikidataPropertyValue = {
-    datatype: string;
-    value: any;
-    pageid?: number;
-    value_string?: string;
-    label?: string;
-    description?: string;
-    qualifiers?: WikidataEntityClaims | null;
-}
+  datatype: string;
+  /** Scalar for simple datatypes, the raw object for time/quantity/coordinates. */
+  value: string | number | object;
+  /** Readable rendering of a structured `value`. */
+  value_string?: string;
+  rank?: "preferred" | "normal" | "deprecated";
+  label?: string;
+  description?: string;
+  qualifiers?: Record<string, WikidataProperty> | null;
+};
+```
 
-type PlainObject<T> = {
-    [index: string]: T
-}
+Notes on the claim values:
 
-```
+- Statements Wikidata marks **deprecated** are dropped, and **preferred** ones
+  are sorted first, so `values[0]` is the value Wikidata recommends.
+- Dates render at their stated precision and keep the sign of BCE years:
+  `-0044-03-15`.
+- Coordinates render as `"47.0105,28.8638"`.
+- Labels fall back to Wikidata's `mul` ("multiple languages") code, which is
+  where many items now keep a name that is spelled the same everywhere.
 
 ## SimpleEntity
 
 ```ts
-
-export enum SimpleEntityType {
-    EVENT = 'E',
-    ORG = 'O',
-    PERSON = 'H',
-    PLACE = 'P',
-    PRODUCT = 'R',
-    WORK = 'W',
+enum SimpleEntityType {
+  EVENT = "E",
+  ORG = "O",
+  PERSON = "H",
+  PLACE = "P",
+  PRODUCT = "R",
+  WORK = "W"
 }
 
-export type SimpleEntityData = { [prop: string]: string[] }
-
-export type SimpleEntity = {
-    lang?: string
-    wikiDataId?: string
-    name?: string
-    abbr?: string
-    description?: string
-    about?: string
-    wikiPageId?: number
-    wikiPageTitle?: string
-    type?: SimpleEntityType
-    types?: string[]
-    countryCodes?: string[]
-    data?: SimpleEntityData
-    categories?: string[]
-    redirectsToId?: string;
-}
+type SimpleEntity = {
+  /** Always set. */
+  lang: string;
+  /** Always set. */
+  wikiDataId: string;
+  name?: string;
+  description?: string;
+  about?: string;
+  wikiPageId?: number;
+  wikiPageTitle?: string;
+  type?: SimpleEntityType;
+  types?: string[];
+  countryCodes?: string[];
+  data?: Record<string, string[]>;
+  categories?: string[];
+  redirectsToId?: string;
+  redirectsFromId?: string;
+};
+```
+
+The entity `type` is resolved from the DBpedia ontology types first, then from
+the `P31` claims against the generated Wikidata class tables, and finally from
+the opening words of the extract.
+
+## Generated data
 
+`data/countries.json` and `data/entity-types.json` are derived from Wikidata
+and checked in. Regenerate them against the live query service with:
+
+```bash
+npm run data:countries
+npm run data:entity-types
 ```
+
+`data:entity-types` verifies a set of known classifications before writing, so
+a bad expansion fails loudly instead of silently degrading type detection.
+
+## Development
+
+```bash
+npm install
+npm test              # unit tests, fully mocked
+npm run test:integration   # hits the live Wikimedia and DBpedia APIs
+npm run check         # format, lint, typecheck, test, build
+```
+
+## License
+
+ISC