@mailwoman/resolver-wof-sqlite 9.0.0 → 9.2.0

package/README.md

@@ -15,11 +15,11 @@ Requires Node 22+ for built-in `node:sqlite`.
 ## Quick start
 
 ```ts
-import { WofSqlitePlaceLookup } from "@mailwoman/resolver-wof-sqlite"
+import { WOFSQLitePlaceLookup } from "@mailwoman/resolver-wof-sqlite"
 
-const lookup = new WofSqlitePlaceLookup({
+const lookup = new WOFSQLitePlaceLookup({
 	databasePath: "/path/to/whosonfirst-data-admin-us-latest.db",
-	buildFts: true, // build the FTS5 index on first open (one-time cost)
+	buildFTS: true, // build the FTS5 index on first open (one-time cost)
 })
 
 const candidates = await lookup.findPlace({
@@ -35,6 +35,25 @@ for (const c of candidates) {
 lookup.close()
 ```
 
+## A shard that cannot answer says so on construction
+
+Shards are `ATTACH`ed by a schema name **derived from the filename**, and queries route to them by matching that name against the requested placetype — `postalcode_us` serves `postalcode`. Two ways that used to fail quietly, and both now throw when you build the lookup:
+
+- The name does not route. `postcode-ca-overture.db` derives `postcode_ca_overture`, and the router tests `startsWith("postalcode_")` — **"postcode" is not "postalcode"**. It held 843,739 Canadian codes and answered every query with zero hits, which is indistinguishable from "this country has no places".
+- The shard carries `spr` but no `place_search`. It routes, then dies mid-`SELECT`.
+
+The predicate is the **table, not the filename**: a shard carrying `spr` is claiming to be a place shard, and every lookup path here reaches the FTS index. A relation-table shard like `postcode-locality-<cc>.db` carries no `spr`, never makes that claim, and is exempt — which is what keeps the documented default shard list working.
+
+```
+WOFSQLitePlaceLookup: …/postcode-ca-overture.db carries "spr" but no "place_search" table, so it
+cannot serve a lookup. Build it with the FTS index, or leave it out — it is usable as a BUILD input
+either way. Its schema name "postcode_ca_overture" also matches no routed placetype (postalcode,
+locality, region, county, country, venue), so it would never have been queried even with the table
+— check the filename's spelling.
+```
+
+The second sentence appears only when the name routes nowhere. It is the half that turns zero hits into a diagnosis.
+
 ## Multi-shard (admin + postcode in one connection)
 
 Pass an array of paths to open multiple WOF shards on a single connection — each is opened as a
@@ -44,7 +63,7 @@ separate SQLite schema via `ATTACH DATABASE`. Schema names auto-derive from file
 `postalcode_us` shard automatically, everything else hits main.
 
 ```ts
-const lookup = new WofSqlitePlaceLookup({
+const lookup = new WOFSQLitePlaceLookup({
 	databasePath: ["/data/wof/whosonfirst-data-admin-us-latest.db", "/data/wof/whosonfirst-data-postalcode-us-latest.db"],
 })
 
@@ -55,7 +74,7 @@ await lookup.findPlace({ text: "62701", placetype: "postalcode" }) // → postco
 Override schema names or routing explicitly when needed:
 
 ```ts
-new WofSqlitePlaceLookup({
+new WOFSQLitePlaceLookup({
 	databasePath: ["/data/wof/admin.db", { path: "/data/oddly-named.db", schemaName: "pc", placetypes: ["postalcode"] }],
 })
 ```
@@ -83,7 +102,7 @@ bunzip2 whosonfirst-data-admin-us-latest.db.bz2
 
 Upstream WOF SQLite distributions ship a `places` table but **not** an FTS5 index. The resolver needs FTS5 to do fast prefix + token-bag matching. Two options:
 
-1. **`buildFts: true` on construction** — builds the index lazily on first open. Cost is one-time but expensive (~minutes on the full US admin shard). Use for prototyping.
+1. **`buildFTS: true` on construction** — builds the index lazily on first open. Cost is one-time but expensive (~minutes on the full US admin shard). Use for prototyping.
 2. **Pre-build the index with `mailwoman gazetteer build fts`** — ship the DB with the index included so first-open is fast. Recommended for production.
 
 ### `mailwoman gazetteer build fts`
@@ -143,7 +162,7 @@ What survives in the slim DB:
 
 What gets dropped: the `geojson` table, which is build-time only — `lookup.ts` never reads it at query time, and it accounts for ~95% of the on-disk size. The `place_population` aux table consumes `wof:population` from geojson before we drop it.
 
-`WofSqlitePlaceLookup` opens the slim DB without any code change. Out-of-set queries (a locality not in the top-K) correctly return zero hits.
+`WOFSQLitePlaceLookup` opens the slim DB without any code change. Out-of-set queries (a locality not in the top-K) correctly return zero hits.
 
 You can also build the index programmatically via the package's `./fts` subpath:
 
@@ -209,7 +228,7 @@ R*Tree because it computes haversine distance against the centroid columns direc
 All weights are configurable via the second ctor argument:
 
 ```ts
-new WofSqlitePlaceLookup({ databasePath }, { countryMatchBoost: 0.5 })
+new WOFSQLitePlaceLookup({ databasePath }, { countryMatchBoost: 0.5 })
 ```
 
 Defaults are in `lookup.ts::DEFAULT_WEIGHTS`.
@@ -246,7 +265,7 @@ Coverage includes: placetype filtering, country filtering, the empty-result case
 
 ## Concurrency model
 
-This package opens a single `node:sqlite` connection per `WofSqlitePlaceLookup` instance. SQLite is single-writer / many-reader; the Kysely wrapper around the connection serializes all queries through a mutex. For high-concurrency HTTP servers, instantiate one resolver per request handler or per pool slot — sharing a single instance across concurrent requests is fine (queries queue) but won't parallelize across cores.
+This package opens a single `node:sqlite` connection per `WOFSQLitePlaceLookup` instance. SQLite is single-writer / many-reader; the Kysely wrapper around the connection serializes all queries through a mutex. For high-concurrency HTTP servers, instantiate one resolver per request handler or per pool slot — sharing a single instance across concurrent requests is fine (queries queue) but won't parallelize across cores.
 
 ## License
 

package/address-point-interpolation.ts

@@ -36,11 +36,11 @@
 import { DatabaseSync } from "node:sqlite"
 
 import type { InterpolationLookup } from "@mailwoman/resolver"
+import { haversineKm } from "@mailwoman/spatial"
 
-import { haversineKm } from "./geo.ts"
 import type { InterpolatedHit, InterpolationQuery, StreetInterpolator } from "./interpolation.ts"
-import { hasTable } from "./sqlite-utils.ts"
-import { canonicalizeRouteKey, normalizeStreetForKey } from "./street-normalize.ts"
+import { hasTable, prepareAll, type PreparedAll } from "./sqlite-utils.ts"
+import { canonicalizeRouteKey, type RouteKey, streetKeyVariants } from "./street-normalize.ts"
 
 /**
  * Extrapolation cap for a single-sided bracket: at most one pair-span beyond the nearest known point (`t = 2`). Past
@@ -71,7 +71,7 @@ export class AddressPointInterpolator implements InterpolationLookup {
 	readonly #db: DatabaseSync
 	readonly #ownsDB: boolean
 	readonly #fallback: StreetInterpolator | undefined
-	readonly #byPostcode: ReturnType<DatabaseSync["prepare"]> | undefined
+	readonly #byPostcode: PreparedAll<[postcode: string, street: RouteKey, number: number], PointRow> | undefined
 
 	constructor(opts: { dbPath?: string; database?: DatabaseSync; fallback?: StreetInterpolator }) {
 		if (opts.database) {
@@ -91,7 +91,8 @@ export class AddressPointInterpolator implements InterpolationLookup {
 		if (hasTable(this.#db, "address_point")) {
 			// Strictly-numeric neighbor numbers on the route-folded street key within the ZIP. The
 			// queried number itself is excluded HERE (see module doc: non-circular by construction).
-			this.#byPostcode = this.#db.prepare(
+			this.#byPostcode = prepareAll(
+				this.#db,
 				`SELECT CAST(number AS INTEGER) AS n, lat, lon, source, release
 				 FROM address_point
 				 WHERE postcode = ? AND street_key = ?
@@ -102,16 +103,25 @@ export class AddressPointInterpolator implements InterpolationLookup {
 	}
 
 	find(query: InterpolationQuery): InterpolatedHit | null {
-		const streetKey = canonicalizeRouteKey(normalizeStreetForKey(query.street))
 		const numberRaw = query.number.trim()
 
-		if (!streetKey || !/^\d+$/.test(numberRaw)) return null
+		if (!/^\d+$/.test(numberRaw)) return null
 		const n = Number(numberRaw)
 
 		// No own table (empty shard) or no postcode → defer to the segment fallback rather than query.
 		if (!this.#byPostcode || !query.postcode) return this.#fallback?.find(query) ?? null
 
-		const rows = this.#byPostcode.all(query.postcode.trim(), streetKey, n) as unknown as PointRow[]
+		// Key-variant ladder (see `streetKeyVariants`) — same probe order as the exact-point reader.
+		let rows: PointRow[] = []
+
+		for (const variant of streetKeyVariants(query.street)) {
+			const streetKey = canonicalizeRouteKey(variant)
+
+			rows = this.#byPostcode(query.postcode.trim(), streetKey, n)
+
+			if (rows.length) break
+		}
+
 		const hit = rows.length >= 2 ? interpolateFromNeighbors(rows, n) : null
 
 		return hit ?? this.#fallback?.find(query) ?? null

package/address-point-schema.ts

@@ -18,6 +18,8 @@
 
 import type { Kysely } from "kysely"
 
+import type { NameKey, RouteKey, StreetKey } from "./street-normalize.ts"
+
 /**
  * One rooftop address point. `(street_norm, number)` within a `postcode` (preferred) or `locality_norm` scope is the
  * lookup; `street_key` is the #483 route-fold key for interpolation. Coordinates are non-null (the builder drops
@@ -27,11 +29,12 @@ export interface AddressPointTable {
 	/**
 	 * Shared {@link normalizeStreetForKey} of the street — the build/query-consistent probe key.
 	 */
-	street_norm: string
+	street_norm: StreetKey
 	/**
-	 * `canonicalizeRouteKey(street_norm)` — the route-fold key (#483 Method 2).
+	 * `canonicalizeRouteKey(street_norm)` — the route-fold key (#483 Method 2). Its own brand, so it cannot be
+	 * interchanged with the plain `street_norm` above.
 	 */
-	street_key: string
+	street_key: RouteKey
 	/**
 	 * House number, normalized lower-case (kept TEXT — "123-A", "12 1/2" must survive).
 	 */
@@ -41,7 +44,7 @@ export interface AddressPointTable {
 	/**
 	 * Shared {@link normalizeLocalityForKey} of the locality — the fallback scope.
 	 */
-	locality_norm: string | null
+	locality_norm: NameKey | null
 	/**
 	 * The street as it appeared in the source (kept for display / debugging).
 	 */
@@ -65,6 +68,15 @@ export interface AddressPointDatabase {
 	address_point: AddressPointTable
 }
 
+/**
+ * The slice of a Kysely handle the `address_point` DDL touches — the parameter type its builders take.
+ *
+ * Kysely is invariant in its schema parameter (the incompatibility is in `transaction()`), so a shard that EXTENDS
+ * `AddressPointTable` — OSM adds `h3_cell` — cannot pass its own handle to a `Kysely<AddressPointDatabase>` parameter.
+ * Naming only `schema` lets it, and the DDL below needs nothing else.
+ */
+export type AddressPointSchemaHandle = Pick<Kysely<AddressPointDatabase>, "schema">
+
 /**
  * The `address_point` columns in INSERT order. The builder's positional prepared statement derives its placeholder list
  * from this, so the positional order can't drift from the DDL / the reader.
@@ -86,7 +98,7 @@ export const ADDRESS_POINT_COLUMNS = [
 /**
  * Create the `address_point` table — called before the streaming bulk load.
  */
-export async function createAddressPointTable(db: Kysely<AddressPointDatabase>): Promise<void> {
+export async function createAddressPointTable(db: AddressPointSchemaHandle): Promise<void> {
 	await db.schema
 		.createTable("address_point")
 		.addColumn("street_norm", "text", (c) => c.notNull())
@@ -107,7 +119,7 @@ export async function createAddressPointTable(db: Kysely<AddressPointDatabase>):
 /**
  * Create the three probe indexes the reader relies on (postcode-scope, locality-scope, route-key).
  */
-export async function createAddressPointIndexes(db: Kysely<AddressPointDatabase>): Promise<void> {
+export async function createAddressPointIndexes(db: AddressPointSchemaHandle): Promise<void> {
 	await db.schema
 		.createIndex("idx_ap_postcode")
 		.on("address_point")

package/address-point.ts

@@ -20,10 +20,13 @@ import { DatabaseSync } from "node:sqlite"
 import type { AddressPointHit, AddressPointLookup } from "@mailwoman/resolver"
 
 import type { AddressPointTable } from "./address-point-schema.ts"
-import { hasTable } from "./sqlite-utils.ts"
+import { hasTable, prepareGet, type PreparedGet } from "./sqlite-utils.ts"
 import {
 	normalizeLocalityForKey,
-	normalizeStreetForKeyLocale,
+	type NameKey,
+	type StreetKey,
+	streetKeyVariants,
+	streetLocaleForSurface,
 	stripArrondissement,
 	type StreetLocale,
 } from "./street-normalize.ts"
@@ -32,20 +35,25 @@ import {
  * The columns this lookup projects — a typed slice of the SHARED {@link AddressPointTable}, so a column rename in
  * `build-address-point-shard.ts` (the writer) is a compile error here (the reader).
  */
-type AddressPointRow = Pick<AddressPointTable, "lat" | "lon" | "source" | "release">
+type AddressPointRow = Pick<AddressPointTable, "lat" | "lon" | "source" | "release" | "locality_norm" | "postcode">
 
 /**
  * The 4 columns the reader SELECTs, in the schema's order — referenced by the prepared SELECTs so the projected
  * `AddressPointRow` stays in lockstep with the shared schema.
  */
-const SELECT_COLS = "lat, lon, source, release"
+const SELECT_COLS = "lat, lon, source, release, locality_norm, postcode"
 
 export class AddressPointSqliteLookup implements AddressPointLookup {
 	readonly #db: DatabaseSync
 	readonly #locale: StreetLocale
-	readonly #byPostcode: ReturnType<DatabaseSync["prepare"]> | undefined
-	readonly #byLocality: ReturnType<DatabaseSync["prepare"]> | undefined
-	readonly #byBbox: ReturnType<DatabaseSync["prepare"]> | undefined
+	readonly #byPostcode: PreparedGet<[postcode: string, street: StreetKey, number: string], AddressPointRow> | undefined
+	readonly #byLocality: PreparedGet<[locality: NameKey, street: StreetKey, number: string], AddressPointRow> | undefined
+	readonly #byBbox:
+		| PreparedGet<
+				[street: StreetKey, number: string, minLat: number, maxLat: number, minLon: number, maxLon: number],
+				AddressPointRow
+		  >
+		| undefined
 
 	/**
 	 * @param dbPath Shard path.
@@ -59,17 +67,20 @@ export class AddressPointSqliteLookup implements AddressPointLookup {
 		// Degrade gracefully on an empty/tableless shard (interrupted build, stray 0-byte file): with no
 		// `address_point` table this lookup is a no-op miss, not a crash that loses the whole state (#568).
 		if (hasTable(this.#db, "address_point")) {
-			this.#byPostcode = this.#db.prepare(
+			this.#byPostcode = prepareGet(
+				this.#db,
 				`SELECT ${SELECT_COLS} FROM address_point
 				 WHERE postcode = ? AND street_norm = ? AND number = ? LIMIT 1`
 			)
 
-			this.#byLocality = this.#db.prepare(
+			this.#byLocality = prepareGet(
+				this.#db,
 				`SELECT ${SELECT_COLS} FROM address_point
 				 WHERE locality_norm = ? AND street_norm = ? AND number = ? LIMIT 1`
 			)
 
-			this.#byBbox = this.#db.prepare(
+			this.#byBbox = prepareGet(
+				this.#db,
 				`SELECT ${SELECT_COLS} FROM address_point
 				 WHERE street_norm = ? AND number = ? AND lat BETWEEN ? AND ? AND lon BETWEEN ? AND ? LIMIT 1`
 			)
@@ -84,15 +95,99 @@ export class AddressPointSqliteLookup implements AddressPointLookup {
 		bbox?: { minLat: number; maxLat: number; minLon: number; maxLon: number }
 	}): AddressPointHit | null {
 		if (!this.#byPostcode || !this.#byLocality || !this.#byBbox) return null
-		const streetNorm = normalizeStreetForKeyLocale(query.street, this.#locale)
 		const number = query.number.trim().toLowerCase()
 
-		if (!streetNorm || !number) return null
+		if (!number) return null
 
+		// Key-variant ladder (see `streetKeyVariants`): the literal key first, then the doubled-type
+		// collapse and the saint↔st register swap — each variant runs the FULL number ladder below, and
+		// the first variant to answer wins, so an attested literal key is never second-guessed.
+		let row: AddressPointRow | undefined
+
+		for (const streetNorm of streetKeyVariants(query.street, streetLocaleForSurface(query.street, this.#locale))) {
+			row = this.#findForKey(streetNorm, number, query)
+
+			if (row) break
+		}
+
+		if (!row) return null
+
+		return {
+			lat: row.lat,
+			lon: row.lon,
+			source: row.source,
+			release: row.release,
+			...(row.locality_norm ? { localityNorm: row.locality_norm } : {}),
+			...(row.postcode ? { postcode: row.postcode } : {}),
+		}
+	}
+
+	/**
+	 * The full number ladder for ONE normalized street key: exact, then the range low-end, then letter-suffix spacing
+	 * with the base-number fall — see each rung's note in place.
+	 */
+	#findForKey(
+		streetNorm: StreetKey,
+		number: string,
+		query: {
+			street: string
+			number: string
+			postcode?: string
+			locality?: string
+			bbox?: { minLat: number; maxLat: number; minLon: number; maxLon: number }
+		}
+	) {
+		let row = this.#probe(streetNorm, number, query)
+
+		// Range-surface fallback: every register this reader serves stores ONE number per point
+		// (G-NAF `NUMBER_FIRST`, BAN, OA, OSM `addr:housenumber`), but the attested surface is often
+		// a range — "385-387 Esplanade" keys `385`. Null-only: an exact range key that matched above
+		// (some OSM points DO carry "385-387" verbatim) is never second-guessed.
+		if (!row) {
+			const low = /^(\d+[a-z]?)-\d+[a-z]?$/.exec(number)?.[1]
+
+			if (low) {
+				row = this.#probe(streetNorm, low, query)
+			}
+		}
+
+		// Letter-suffix spacing fallback: the registers disagree on the joint — BAN stores "3 a"
+		// (space-separated), G-NAF and most OA sources store "3a" — and the parsed surface can arrive
+		// either way. On a miss, retry the OTHER spacing; on a double miss, the BASE number (the
+		// register attests no 3A but does attest 3 — the adjacent-parcel approximation, priced the
+		// same as the range fallback's low end). Null-only throughout, and only for the
+		// digits+single-letter shape (never touches "12 1/2" or unit-bearing forms).
+		if (!row) {
+			const joined = /^(\d+)\s+([a-z])$/.exec(number)
+			const spaced = /^(\d+)([a-z])$/.exec(number)
+
+			if (joined) {
+				row = this.#probe(streetNorm, `${joined[1]}${joined[2]}`, query) ?? this.#probe(streetNorm, joined[1]!, query)
+			} else if (spaced) {
+				row = this.#probe(streetNorm, `${spaced[1]} ${spaced[2]}`, query) ?? this.#probe(streetNorm, spaced[1]!, query)
+			}
+		}
+
+		return row
+	}
+
+	/**
+	 * The scope ladder for one (street, number) key: postcode, then locality, then the bbox fall-through — each rung only
+	 * when the prior missed.
+	 */
+	#probe(
+		streetNorm: StreetKey,
+		number: string,
+		query: {
+			postcode?: string
+			locality?: string
+			bbox?: { minLat: number; maxLat: number; minLon: number; maxLon: number }
+		}
+	): AddressPointRow | undefined {
 		let row: AddressPointRow | undefined
 
 		if (query.postcode) {
-			row = this.#byPostcode.get(query.postcode.trim(), streetNorm, number) as AddressPointRow | undefined
+			row = this.#byPostcode!(query.postcode.trim(), streetNorm, number)
 		}
 
 		if (!row && query.locality) {
@@ -104,19 +199,17 @@ export class AddressPointSqliteLookup implements AddressPointLookup {
 					? stripArrondissement(normalizeLocalityForKey(query.locality))
 					: normalizeLocalityForKey(query.locality)
 
-			row = this.#byLocality.get(localityKey, streetNorm, number) as AddressPointRow | undefined
+			row = this.#byLocality!(localityKey, streetNorm, number)
 		}
 
 		// Bbox fall-through (#247): the point carries no postcode/locality of its own, but its coordinate falls
 		// inside the resolved locality's box. Only reached when the scoped probes missed AND a bbox was supplied.
 		if (!row && query.bbox) {
 			const b = query.bbox
-			row = this.#byBbox.get(streetNorm, number, b.minLat, b.maxLat, b.minLon, b.maxLon) as AddressPointRow | undefined
+			row = this.#byBbox!(streetNorm, number, b.minLat, b.maxLat, b.minLon, b.maxLon)
 		}
 
-		if (!row) return null
-
-		return { lat: row.lat, lon: row.lon, source: row.source, release: row.release }
+		return row
 	}
 
 	close(): void {

package/ancestry.ts

@@ -7,7 +7,7 @@
  *   joined with `spr` for canonical names + centroids, ordered NEAREST-FIRST (deepest placetype
  *   first, country last).
  *
- *   Factored out of `WOFSqlitePlaceLookup.ancestors()` (#404) so the reverse geocoder (`reverse.ts`,
+ *   Factored out of `WOFSQLitePlaceLookup.ancestors()` (#404) so the reverse geocoder (`reverse.ts`,
  *   #484) reuses the SAME walk instead of growing a second one. The placetype-specificity ordering
  *   lives here as `PLACETYPE_DEPTH` — a single TS map instead of the previous SQL CASE, and
  *   extended below `localadmin` (locality/borough/neighbourhood/microhood now rank correctly
@@ -17,6 +17,8 @@
 
 import type { DatabaseSync } from "node:sqlite"
 
+import { allRows } from "./sqlite-utils.ts"
+
 /**
  * WOF placetype → containment depth, coarsest = 1. Higher = finer. Placetypes we never resolve (continent, empire, …)
  * map to 0 and sort last. NOT the same table as the FST's `PLACETYPE_ORDER` (fst-serialize.ts) — that one is a
@@ -57,17 +59,18 @@ export interface AncestorPlaceRow {
 
 /**
  * The ancestor lineage of `id` — self excluded, nearest-first. Returns `[]` when the place has no recorded ancestry.
- * NOT memoized here; `WOFSqlitePlaceLookup` keeps its own per-id cache.
+ * NOT memoized here; `WOFSQLitePlaceLookup` keeps its own per-id cache.
  */
 export function ancestorLineage(db: DatabaseSync, id: number, schemaName = "main"): AncestorPlaceRow[] {
-	const rows = db
-		.prepare(
+	const rows = allRows<AncestorPlaceRow>(
+		db.prepare(
 			`SELECT a.ancestor_id AS id, a.ancestor_placetype AS placetype, s.name AS name,
 				s.country AS country, s.latitude AS lat, s.longitude AS lon
 			FROM ${schemaName}.ancestors a JOIN ${schemaName}.spr s ON s.id = a.ancestor_id
 			WHERE a.id = ? AND a.ancestor_id != a.id`
-		)
-		.all(id) as unknown as AncestorPlaceRow[]
+		),
+		id
+	)
 
 	rows.sort((a, b) => placetypeDepth(b.placetype) - placetypeDepth(a.placetype))