@mailwoman/corpus 10.2.2 → 10.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -17
- package/data/PROVENANCE.md +273 -273
- package/data/address-source-register.json +517 -23
- package/data/license-decisions.json +120 -0
- package/data/licenses/PROVENANCE.md +5 -5
- package/data/licenses/cdla-permissive-2.0.md +6 -6
- package/data/licenses/geonames-publication.md +5 -5
- package/data/licenses/licence-ouverte-2.0.md +3 -3
- package/data/licenses/national-address-database.md +12 -12
- package/data/licenses/nlod-2.0.md +3 -3
- package/data/licenses/sirene-publication.md +13 -6
- package/data/licenses/whosonfirst-licenses.md +4 -4
- package/data/source-resolutions.json +11 -0
- package/lib/adapters/cordis/adapter.ts +8 -8
- package/lib/adapters/gleif/adapter.ts +7 -8
- package/lib/adapters/po-box/adapter.ts +1 -1
- package/lib/adapters/source-id.ts +1 -1
- package/lib/adapters.ts +2 -2
- package/lib/au/adapters/gnaf/adapter.ts +1 -1
- package/lib/be/adapters/brussels/adapter.ts +2 -2
- package/lib/be/adapters/vlaanderen/adapter.ts +1 -1
- package/lib/build.ts +1 -1
- package/lib/cz/adapters/cuzk/adapter.ts +1 -1
- package/lib/cz/tools/fetch/cuzk.ts +4 -4
- package/lib/de/recipes/locale.ts +1 -1
- package/lib/dk/adapters/inspire/adapter.ts +1 -1
- package/lib/ee/tools/fetch/ads.ts +2 -2
- package/lib/es/adapters/bizkaia/adapter.ts +1 -1
- package/lib/es/adapters/navarra/adapter.ts +2 -2
- package/lib/es/cadastre.ts +2 -2
- package/lib/es/tools/fetch/bizkaia.ts +7 -7
- package/lib/es/tools/fetch/catastro.ts +6 -6
- package/lib/es/tools/fetch/navarra.ts +4 -4
- package/lib/fingerprints.ts +3 -3
- package/lib/fr/adapters/finess/adapter.ts +6 -5
- package/lib/fr/adapters/finess/voie-type.ts +1 -1
- package/lib/fr/tools/fetch/annuaire-education.ts +3 -3
- package/lib/gb/constituent-country.ts +52 -0
- package/lib/inspire/address.ts +3 -3
- package/lib/international/recipes/locale.ts +41 -7
- package/lib/it/adapters/anac/adapter.ts +4 -3
- package/lib/jp/adapters/wof-admin/adapter.ts +2 -2
- package/lib/kr/adapters/juso/label-rows.ts +1 -1
- package/lib/kr/adapters/localdata/key-index.ts +2 -1
- package/lib/lu/adapters/bd-adresses/adapter.ts +1 -1
- package/lib/nl/adapters/kadaster/adapter.ts +2 -2
- package/lib/nl/tools/fetch/kadaster.ts +1 -1
- package/lib/parquet/shuffle.ts +2 -2
- package/lib/parquet/span-sql.ts +8 -6
- package/lib/parquet/streams.ts +1 -1
- package/lib/parquet/writers.ts +2 -2
- package/lib/pl/tools/fetch/emuia.ts +2 -2
- package/lib/recipes/scaffold.ts +2 -2
- package/lib/recipes/sources.ts +3 -3
- package/lib/recipes/sub/venue/render.ts +1 -1
- package/lib/recipes/sub/venue/sources.ts +4 -3
- package/lib/recipes/sub/venue.ts +4 -3
- package/lib/recipes/venue.ts +11 -11
- package/lib/ro/adapters/retea-scolara/adapter.ts +2 -2
- package/lib/runner.ts +3 -3
- package/lib/sk/tools/fetch/inspire.ts +3 -3
- package/lib/source-register/effective-manifest.ts +2 -2
- package/lib/source-register/types.ts +2 -2
- package/lib/subvenue/harvest.ts +6 -5
- package/lib/subvenue/promotions.ts +3 -3
- package/lib/subvenue/table.ts +6 -6
- package/lib/surfaces/locale.ts +1 -1
- package/lib/synthesizers/anchor-absorption.ts +1 -1
- package/lib/synthesizers/boundary-stress.ts +3 -2
- package/lib/synthesizers/no-street.ts +1 -1
- package/lib/synthesizers/po-box.ts +1 -1
- package/lib/synthesizers/tail.ts +2 -2
- package/lib/test-kit/corpus-recipe.ts +1 -1
- package/lib/tools/align-canonical.ts +3 -2
- package/lib/tools/audit.ts +1 -1
- package/lib/tools/corpus-stats.ts +1 -1
- package/lib/tools/fetch/download/network.ts +1 -1
- package/lib/tools/fetch/geonames/dump.ts +1 -1
- package/lib/tools/fetch/gleif.ts +1 -1
- package/lib/tools/fetch/header.ts +1 -1
- package/lib/tools/fetch/inspire-addresses.ts +1 -1
- package/lib/tools/fetch/vlaanderen-ad.ts +1 -1
- package/lib/tools/fetch/wfs-harvest.ts +6 -6
- package/lib/tools/fetch/zip-archive.ts +2 -2
- package/lib/tools/golden/expand.ts +1 -1
- package/lib/tools/golden/promote.ts +3 -3
- package/lib/tools/golden/relabel-street/directory.ts +2 -1
- package/lib/tools/lint/recipe-output/vocab.ts +2 -2
- package/lib/tools/overlay/manifest.ts +62 -10
- package/lib/tools/overlay/split-slice.ts +2 -10
- package/lib/tools/postcode-triples.ts +2 -2
- package/lib/tools/sub/venue/head-nouns.ts +4 -3
- package/lib/types.ts +3 -3
- package/lib/us/adapters/state/tx-notaries/adapter.ts +1 -1
- package/lib/us/adapters/usgov/irs-bmf/adapter.ts +2 -2
- package/lib/us/adapters/usgov/nppes/adapter.ts +1 -1
- package/lib/us/synthesizers/intersection.ts +4 -4
- package/lib/us/synthesizers/street.ts +3 -3
- package/lib/us/tools/fetch/nad.ts +2 -2
- package/lib/utils/split.ts +59 -6
- package/lib/utils/wof-json.ts +1 -1
- package/out/adapters/cordis/adapter.d.ts +7 -7
- package/out/adapters/cordis/adapter.js +8 -8
- package/out/adapters/cordis/adapter.js.map +1 -1
- package/out/adapters/gleif/adapter.d.ts +7 -8
- package/out/adapters/gleif/adapter.d.ts.map +1 -1
- package/out/adapters/gleif/adapter.js +7 -8
- package/out/adapters/gleif/adapter.js.map +1 -1
- package/out/adapters/po-box/adapter.d.ts +1 -1
- package/out/adapters/source-id.d.ts +1 -1
- package/out/adapters/source-id.js +1 -1
- package/out/adapters.d.ts +2 -2
- package/out/adapters.js +2 -2
- package/out/au/adapters/gnaf/adapter.js +1 -1
- package/out/be/adapters/brussels/adapter.d.ts +1 -1
- package/out/be/adapters/brussels/adapter.js +2 -2
- package/out/be/adapters/vlaanderen/adapter.d.ts +1 -1
- package/out/build.js +1 -1
- package/out/build.js.map +1 -1
- package/out/cz/adapters/cuzk/adapter.d.ts +1 -1
- package/out/cz/adapters/cuzk/adapter.js +1 -1
- package/out/cz/tools/fetch/cuzk.d.ts +2 -2
- package/out/cz/tools/fetch/cuzk.js +2 -2
- package/out/de/recipes/locale.d.ts +1 -1
- package/out/de/recipes/locale.js +1 -1
- package/out/dk/adapters/inspire/adapter.js +1 -1
- package/out/ee/tools/fetch/ads.d.ts +2 -2
- package/out/ee/tools/fetch/ads.js +1 -1
- package/out/es/adapters/bizkaia/adapter.js +1 -1
- package/out/es/adapters/navarra/adapter.d.ts +2 -2
- package/out/es/adapters/navarra/adapter.js +2 -2
- package/out/es/cadastre.d.ts +2 -2
- package/out/es/cadastre.js +2 -2
- package/out/es/tools/fetch/bizkaia.d.ts +5 -5
- package/out/es/tools/fetch/bizkaia.js +5 -5
- package/out/es/tools/fetch/catastro.d.ts +4 -4
- package/out/es/tools/fetch/catastro.js +2 -2
- package/out/es/tools/fetch/navarra.d.ts +2 -2
- package/out/es/tools/fetch/navarra.js +2 -2
- package/out/fingerprints.d.ts +3 -3
- package/out/fingerprints.js +3 -3
- package/out/fr/adapters/finess/adapter.d.ts +6 -5
- package/out/fr/adapters/finess/adapter.d.ts.map +1 -1
- package/out/fr/adapters/finess/adapter.js +6 -5
- package/out/fr/adapters/finess/adapter.js.map +1 -1
- package/out/fr/adapters/finess/voie-type.d.ts +1 -1
- package/out/fr/adapters/finess/voie-type.js +1 -1
- package/out/fr/tools/fetch/annuaire-education.d.ts +3 -3
- package/out/fr/tools/fetch/annuaire-education.js +3 -3
- package/out/gb/constituent-country.d.ts +36 -0
- package/out/gb/constituent-country.d.ts.map +1 -0
- package/out/gb/constituent-country.js +30 -0
- package/out/gb/constituent-country.js.map +1 -0
- package/out/inspire/address.d.ts +3 -3
- package/out/inspire/address.js +3 -3
- package/out/international/recipes/locale.d.ts +6 -0
- package/out/international/recipes/locale.d.ts.map +1 -1
- package/out/international/recipes/locale.js +29 -7
- package/out/international/recipes/locale.js.map +1 -1
- package/out/it/adapters/anac/adapter.d.ts +1 -1
- package/out/it/adapters/anac/adapter.d.ts.map +1 -1
- package/out/it/adapters/anac/adapter.js +4 -3
- package/out/it/adapters/anac/adapter.js.map +1 -1
- package/out/jp/adapters/wof-admin/adapter.d.ts +2 -2
- package/out/jp/adapters/wof-admin/adapter.js +2 -2
- package/out/kr/adapters/juso/label-rows.d.ts +1 -1
- package/out/kr/adapters/localdata/key-index.d.ts +2 -1
- package/out/kr/adapters/localdata/key-index.d.ts.map +1 -1
- package/out/kr/adapters/localdata/key-index.js +2 -1
- package/out/kr/adapters/localdata/key-index.js.map +1 -1
- package/out/lu/adapters/bd-adresses/adapter.d.ts +1 -1
- package/out/lu/adapters/bd-adresses/adapter.js +1 -1
- package/out/nl/adapters/kadaster/adapter.js +2 -2
- package/out/nl/tools/fetch/kadaster.d.ts +1 -1
- package/out/nl/tools/fetch/kadaster.js +1 -1
- package/out/parquet/shuffle.d.ts +2 -2
- package/out/parquet/shuffle.js +2 -2
- package/out/parquet/span-sql.d.ts +8 -6
- package/out/parquet/span-sql.d.ts.map +1 -1
- package/out/parquet/span-sql.js +8 -6
- package/out/parquet/span-sql.js.map +1 -1
- package/out/parquet/streams.d.ts +1 -1
- package/out/parquet/writers.d.ts +2 -2
- package/out/parquet/writers.js +1 -1
- package/out/pl/tools/fetch/emuia.d.ts +2 -2
- package/out/recipes/scaffold.d.ts +2 -2
- package/out/recipes/sources.d.ts +2 -2
- package/out/recipes/sources.js +3 -3
- package/out/recipes/sources.js.map +1 -1
- package/out/recipes/sub/venue/render.d.ts +1 -1
- package/out/recipes/sub/venue/render.js +1 -1
- package/out/recipes/sub/venue/sources.d.ts +4 -3
- package/out/recipes/sub/venue/sources.d.ts.map +1 -1
- package/out/recipes/sub/venue/sources.js +4 -3
- package/out/recipes/sub/venue/sources.js.map +1 -1
- package/out/recipes/sub/venue.d.ts +3 -2
- package/out/recipes/sub/venue.d.ts.map +1 -1
- package/out/recipes/sub/venue.js +2 -2
- package/out/recipes/sub/venue.js.map +1 -1
- package/out/recipes/venue.d.ts +7 -6
- package/out/recipes/venue.d.ts.map +1 -1
- package/out/recipes/venue.js +11 -10
- package/out/recipes/venue.js.map +1 -1
- package/out/ro/adapters/retea-scolara/adapter.d.ts +2 -2
- package/out/ro/adapters/retea-scolara/adapter.js +2 -2
- package/out/runner.d.ts +3 -3
- package/out/runner.js +2 -2
- package/out/sk/tools/fetch/inspire.d.ts +3 -3
- package/out/sk/tools/fetch/inspire.js +1 -1
- package/out/source-register/effective-manifest.d.ts +2 -2
- package/out/source-register/types.d.ts +2 -2
- package/out/subvenue/harvest.d.ts +4 -3
- package/out/subvenue/harvest.d.ts.map +1 -1
- package/out/subvenue/harvest.js +6 -5
- package/out/subvenue/harvest.js.map +1 -1
- package/out/subvenue/promotions.d.ts +2 -2
- package/out/subvenue/promotions.js +3 -3
- package/out/subvenue/promotions.js.map +1 -1
- package/out/subvenue/table.d.ts +6 -6
- package/out/subvenue/table.js +1 -1
- package/out/surfaces/locale.d.ts +1 -1
- package/out/surfaces/locale.js +1 -1
- package/out/synthesizers/boundary-stress.d.ts.map +1 -1
- package/out/synthesizers/boundary-stress.js +2 -1
- package/out/synthesizers/boundary-stress.js.map +1 -1
- package/out/synthesizers/tail.d.ts +2 -2
- package/out/synthesizers/tail.js +2 -2
- package/out/test-kit/corpus-recipe.d.ts +1 -1
- package/out/test-kit/corpus-recipe.js +1 -1
- package/out/tools/align-canonical.d.ts.map +1 -1
- package/out/tools/align-canonical.js +3 -2
- package/out/tools/align-canonical.js.map +1 -1
- package/out/tools/audit.js +1 -1
- package/out/tools/corpus-stats.d.ts +1 -1
- package/out/tools/fetch/download/network.d.ts +1 -1
- package/out/tools/fetch/download/network.js +1 -1
- package/out/tools/fetch/geonames/dump.d.ts +1 -1
- package/out/tools/fetch/geonames/dump.js +1 -1
- package/out/tools/fetch/gleif.d.ts +1 -1
- package/out/tools/fetch/gleif.js +1 -1
- package/out/tools/fetch/header.d.ts +1 -1
- package/out/tools/fetch/header.js +1 -1
- package/out/tools/fetch/inspire-addresses.d.ts +1 -1
- package/out/tools/fetch/vlaanderen-ad.d.ts +1 -1
- package/out/tools/fetch/wfs-harvest.d.ts +3 -3
- package/out/tools/fetch/wfs-harvest.js +4 -4
- package/out/tools/fetch/wfs-harvest.js.map +1 -1
- package/out/tools/fetch/zip-archive.d.ts +2 -2
- package/out/tools/fetch/zip-archive.js +2 -2
- package/out/tools/golden/expand.js +1 -1
- package/out/tools/golden/expand.js.map +1 -1
- package/out/tools/golden/promote.d.ts +1 -1
- package/out/tools/golden/promote.js +3 -3
- package/out/tools/golden/promote.js.map +1 -1
- package/out/tools/golden/relabel-street/directory.d.ts +2 -1
- package/out/tools/golden/relabel-street/directory.d.ts.map +1 -1
- package/out/tools/golden/relabel-street/directory.js.map +1 -1
- package/out/tools/lint/recipe-output/vocab.js +2 -2
- package/out/tools/overlay/manifest.d.ts +15 -2
- package/out/tools/overlay/manifest.d.ts.map +1 -1
- package/out/tools/overlay/manifest.js +41 -10
- package/out/tools/overlay/manifest.js.map +1 -1
- package/out/tools/overlay/split-slice.d.ts.map +1 -1
- package/out/tools/overlay/split-slice.js +2 -9
- package/out/tools/overlay/split-slice.js.map +1 -1
- package/out/tools/postcode-triples.d.ts +1 -1
- package/out/tools/postcode-triples.js +2 -2
- package/out/tools/postcode-triples.js.map +1 -1
- package/out/tools/sub/venue/head-nouns.d.ts +1 -1
- package/out/tools/sub/venue/head-nouns.d.ts.map +1 -1
- package/out/tools/sub/venue/head-nouns.js +4 -3
- package/out/tools/sub/venue/head-nouns.js.map +1 -1
- package/out/types.d.ts +3 -3
- package/out/types.js +1 -1
- package/out/us/adapters/state/tx-notaries/adapter.d.ts +1 -1
- package/out/us/adapters/state/tx-notaries/adapter.js +1 -1
- package/out/us/adapters/usgov/irs-bmf/adapter.d.ts +1 -1
- package/out/us/adapters/usgov/irs-bmf/adapter.js +2 -2
- package/out/us/synthesizers/intersection.js +4 -4
- package/out/us/synthesizers/intersection.js.map +1 -1
- package/out/us/synthesizers/street.js +3 -3
- package/out/us/synthesizers/street.js.map +1 -1
- package/out/us/tools/fetch/nad.d.ts +2 -2
- package/out/us/tools/fetch/nad.js +2 -2
- package/out/utils/split.d.ts +21 -0
- package/out/utils/split.d.ts.map +1 -1
- package/out/utils/split.js +49 -5
- package/out/utils/split.js.map +1 -1
- package/out/utils/wof-json.d.ts +1 -1
- package/out/utils/wof-json.js +1 -1
- package/package.json +14 -14
package/data/PROVENANCE.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
## `address-source-register.json` — the address-source register (#2323)
|
|
4
4
|
|
|
5
5
|
Which jurisdictions exist, what research has resolved for each, and what is known about every
|
|
6
|
-
source's terms.
|
|
6
|
+
source's terms. The file is generated, and a hand edit to it is refused.
|
|
7
7
|
|
|
8
8
|
### Regenerate
|
|
9
9
|
|
|
@@ -22,16 +22,16 @@ yarn format
|
|
|
22
22
|
That is the command that produced the committed copy. The two CSVs are research working documents and
|
|
23
23
|
live in the `mailwoman-internal` repository under `research/`, so regenerating needs a checkout of it.
|
|
24
24
|
That repository is checked out at `internal/`, which `.git/info/exclude` keeps out of this one, and
|
|
25
|
-
the paths above read from there. They were under this repository's gitignored `.notes/` until 2026-09-20
|
|
26
|
-
|
|
27
|
-
is the committed record here
|
|
25
|
+
the paths above read from there. They were under this repository's gitignored `.notes/` until 2026-09-20. No
|
|
26
|
+
version control tracked them there. A reviewer could not determine which copy produced the register. The
|
|
27
|
+
register is the committed record here. It stores every field a reader needs to check a row: the
|
|
28
28
|
publisher, the retrieved URL, and the research pass.
|
|
29
29
|
|
|
30
30
|
Two further inputs are committed beside this file, and both exist because the write below replaces the
|
|
31
31
|
register whole: a value recorded in the register itself is erased by the next run with no error and a
|
|
32
32
|
normal-looking count.
|
|
33
33
|
|
|
34
|
-
`--decisions` defaults to `license-decisions.json` and holds every
|
|
34
|
+
`--decisions` defaults to `license-decisions.json` and holds every license decision a reviewer made by
|
|
35
35
|
reading a publisher's terms. The build derives each decision as `unchecked` from the research pass's
|
|
36
36
|
access labels and applies the recorded ones over that, so a decision survives a rebuild (#2351).
|
|
37
37
|
|
|
@@ -41,8 +41,8 @@ first three as the placeholders `varies`, `country-specific` and the empty strin
|
|
|
41
41
|
non-rail rows, and has no column for a personal-data review, so a review records its findings here
|
|
42
42
|
rather than by rewriting the record of what the research pass found on 2026-09-18 (#2323).
|
|
43
43
|
|
|
44
|
-
The `oxfmt` pass is required
|
|
45
|
-
reproduce. The generator is deterministic, so the artifact is reproducible from the same inputs.
|
|
44
|
+
The `oxfmt` pass is required. Committed JSON is oxfmt-clean, and `JSON.stringify` cannot
|
|
45
|
+
reproduce that formatting. The generator is deterministic, so the artifact is reproducible from the same inputs.
|
|
46
46
|
|
|
47
47
|
### What no check can establish about this file (#2352)
|
|
48
48
|
|
|
@@ -52,10 +52,10 @@ of that one, so the register's correctness rests on whoever last ran the command
|
|
|
52
52
|
`auditAddressSourceRegister` does not change that: it checks the file's structure, and a publisher's
|
|
53
53
|
name rewritten to something that names no institution is structurally sound.
|
|
54
54
|
|
|
55
|
-
The inputs
|
|
55
|
+
The inputs have been versioned since 2026-09-20. A reviewer can diff the CSV that
|
|
56
56
|
produced the register against the one a later build reads, and a research pass that moves a
|
|
57
|
-
publisher's name leaves a commit
|
|
58
|
-
inputs still
|
|
57
|
+
publisher's name leaves a commit that records the move. A check that the committed register matches
|
|
58
|
+
its inputs still requires a run of the command with both repositories checked out.
|
|
59
59
|
|
|
60
60
|
A committed digest would not close it either. A sweep that edits the artifact and the digest together
|
|
61
61
|
passes, and one that edits only the artifact fails without saying which side is right.
|
|
@@ -71,74 +71,74 @@ concluding the publisher renamed itself.
|
|
|
71
71
|
| file | rows | what is read |
|
|
72
72
|
| ---------------------------------------------- | ----: | ---------------------------------------------------------------------- |
|
|
73
73
|
| `global-address-jurisdiction-inventory-v3.csv` | 250 | all 250 rows — every ISO 3166-1 alpha-2 code plus the operational `XK` |
|
|
74
|
-
| `global-functional-authority-corpora-v2.csv` | 2,426 | the 426 rows naming a national source
|
|
74
|
+
| `global-functional-authority-corpora-v2.csv` | 2,426 | the 426 rows naming a national source. 2,000 are dropped |
|
|
75
75
|
|
|
76
|
-
The 2,000 dropped rows
|
|
77
|
-
pass repeated once per jurisdiction
|
|
78
|
-
|
|
76
|
+
The 2,000 dropped rows have `origin = global_research_rail`. They are eight global discovery lookups the
|
|
77
|
+
pass repeated once per jurisdiction. With the `iso2` and `jurisdiction` columns ignored, they collapse
|
|
78
|
+
to eight distinct row bodies. They name a lookup to perform rather than a source, so they are the
|
|
79
79
|
README's procedure and not rows here.
|
|
80
80
|
|
|
81
|
-
### Why three columns the CSV
|
|
81
|
+
### Why three columns of the CSV reach no register field
|
|
82
82
|
|
|
83
83
|
`global-functional-authority-corpora-v2.csv` has an `address_role`, a `coverage` and an `upstream`
|
|
84
84
|
column, and the register declares all three unresolved. That is not the build dropping usable data.
|
|
85
85
|
Over the 426 rows it keeps, `address_role` reads `varies` on all 426 and `coverage` reads
|
|
86
86
|
`country-specific` on all 426. `upstream` is empty on 407 of them: the nineteen rows the 2026-10-01
|
|
87
87
|
rights review added name their own upstream, as `adresse.data.gouv.fr`'s department directory and
|
|
88
|
-
Geonorge's area codelist. The first two are the research
|
|
89
|
-
pass
|
|
88
|
+
Geonorge's area codelist. The first two values are the research
|
|
89
|
+
pass's record that it did not determine the field per source.
|
|
90
90
|
|
|
91
|
-
The build reads the columns and refuses to
|
|
91
|
+
The build reads the columns and refuses to copy a placeholder, because
|
|
92
92
|
`ingestEligibilityProblems` reports "no address role is resolved" and "no coverage has been
|
|
93
93
|
measured" as two of its five conditions. The others are the unreviewed license decision, the absent
|
|
94
|
-
personal-data review, and a status that records the source as uninspected.
|
|
95
|
-
into `addressRole` would stop both from firing
|
|
96
|
-
placeholder set is
|
|
94
|
+
personal-data review, and a status that records the source as uninspected. A `varies` written
|
|
95
|
+
into `addressRole` would stop both from firing and would resolve no field. A value outside the declared
|
|
96
|
+
placeholder set is copied through, and an `address_role` that is neither a placeholder nor an
|
|
97
97
|
`AddressRole` fails the build rather than being dropped.
|
|
98
98
|
|
|
99
99
|
### How an edit to the committed file is detected
|
|
100
100
|
|
|
101
|
-
The register
|
|
101
|
+
The register stores `contentDigest`, a sha256 the build writes over every other field in it.
|
|
102
102
|
`readAddressSourceRegister` recomputes it on every read and refuses a file that no longer matches,
|
|
103
103
|
naming the regenerate command above.
|
|
104
104
|
|
|
105
105
|
It exists because no check can regenerate this artifact and compare. The two research CSVs are
|
|
106
|
-
working documents under `.notes/`, which is not committed, so a CI job has
|
|
106
|
+
working documents under `.notes/`, which is not committed, so a CI job has no input to rebuild from.
|
|
107
107
|
Without a recorded digest, a hand edit to a generated file is invisible: a prose sweep rewrote
|
|
108
108
|
`Contracts Finder / Find a Tender` to `Interfaces Finder / Find a Tender` and two other publisher
|
|
109
109
|
names, and the structural audit passed, because it checks shape rather than whether a name is the
|
|
110
110
|
publisher's (#2352).
|
|
111
111
|
|
|
112
|
-
The digest covers the register as parsed
|
|
113
|
-
build writes it. It detects an edit rather than attributing one
|
|
114
|
-
produces a new digest
|
|
112
|
+
The digest covers the register as parsed rather than as bytes, because `oxfmt` reformats the file
|
|
113
|
+
after the build writes it. It detects an edit rather than attributing one. A build rerun after a change
|
|
114
|
+
produces a new digest through the intended path. A hand edit to the committed copy produces none.
|
|
115
115
|
|
|
116
|
-
Of the four artifacts in this directory, the register is the one
|
|
116
|
+
Of the four artifacts in this directory, the register is the one with a digest. The sub-venue
|
|
117
117
|
lexicon, the reviewed VE postcode tuples and the license decisions do not. The license decisions are
|
|
118
|
-
hand-written by design
|
|
118
|
+
hand-written by design. That file is an input to this build rather than an output of one.
|
|
119
119
|
|
|
120
|
-
### What the register
|
|
120
|
+
### What the register stores
|
|
121
121
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
inside the first
|
|
122
|
+
The register holds two tables rather than one nested list. The jurisdiction table is a complete
|
|
123
|
+
250-row enumeration. The source table is zero or more rows per jurisdiction, 426 over 240 of them. If
|
|
124
|
+
the second were nested inside the first, a jurisdiction no researcher examined would be structurally identical to one whose
|
|
125
125
|
list happens to be empty, and no reader could tell which they were looking at.
|
|
126
126
|
|
|
127
127
|
| research state | jurisdictions | meaning |
|
|
128
128
|
| -------------- | ------------: | ------------------------------------------------------------- |
|
|
129
129
|
| `seeded` | 240 | at least one national source resolved |
|
|
130
130
|
| `exception` | 7 | somebody looked and recorded that no ordinary registry exists |
|
|
131
|
-
| `unexamined` | 3 |
|
|
131
|
+
| `unexamined` | 3 | no source resolved and no source ruled out |
|
|
132
132
|
|
|
133
|
-
`exception` and `unexamined` are separate values on purpose
|
|
134
|
-
an unexamined jurisdiction is not
|
|
135
|
-
as "
|
|
133
|
+
`exception` and `unexamined` are separate values on purpose. A researched absence is a finding and
|
|
134
|
+
an unexamined jurisdiction is not. If the two were merged, a consumer would read "we could not look"
|
|
135
|
+
as "no registry exists there". Every non-`seeded` row has a `stateReason`, and the audit refuses
|
|
136
136
|
one that does not.
|
|
137
137
|
|
|
138
138
|
| source status | rows | meaning |
|
|
139
139
|
| ----------------------- | ---: | ----------------------------------------------------------------------- |
|
|
140
|
-
| `verified-authority` | 221 | register confirmed
|
|
141
|
-
| `retained-original` | 142 |
|
|
140
|
+
| `verified-authority` | 221 | register confirmed. Address fields, bulk access and terms not inspected |
|
|
141
|
+
| `retained-original` | 142 | copied from the earlier memo without recheck. No publisher or URL |
|
|
142
142
|
| `verified-corpus` | 29 | a bulk corpus confirmed reachable |
|
|
143
143
|
| `verified-corpus-stale` | 1 | reachable, and the copy examined has stopped being updated |
|
|
144
144
|
|
|
@@ -148,12 +148,12 @@ repository already has locale tiers 1 through 5 in `scope.config.json` and the
|
|
|
148
148
|
|
|
149
149
|
### Three things the research pass did not resolve, on any row
|
|
150
150
|
|
|
151
|
-
|
|
151
|
+
This section lists them so a reader does not have to discover them by querying:
|
|
152
152
|
|
|
153
153
|
- **`addressRole` — 0 of 426.** Every row reads `varies`. The specification's section 2 asks for a
|
|
154
154
|
role per source and the pass recorded none, so the research CSV cannot say whether any of these
|
|
155
155
|
sources holds premises, registered offices or mailing addresses.
|
|
156
|
-
- **`upstreamLineage` — 19 of 426.** The 2026-09-18 pass recorded none, so two databases
|
|
156
|
+
- **`upstreamLineage` — 19 of 426.** The 2026-09-18 pass recorded none, so two databases built from one
|
|
157
157
|
upstream submission could not be collapsed into the single observation they are. The 2026-10-01
|
|
158
158
|
rights review fills the column as it goes, and the nineteen rows it added name the directory or
|
|
159
159
|
codelist the source was found through.
|
|
@@ -161,7 +161,7 @@ Say this plainly rather than making a reader discover it by querying:
|
|
|
161
161
|
measurement. A national portal is not evidence of national coverage.
|
|
162
162
|
|
|
163
163
|
A later review resolves these per source in `source-resolutions.json`, and the register's `unresolved`
|
|
164
|
-
array
|
|
164
|
+
array lists any field no row has yet. The audit checks that claim in both directions: a field
|
|
165
165
|
listed there must be absent from every row, and a field not listed must be present on at least one.
|
|
166
166
|
The array is empty today, because a review has resolved each of the four on at least one source.
|
|
167
167
|
|
|
@@ -175,50 +175,50 @@ adapter emits a source's rows, `readSourceEligibility` keys each declaring sourc
|
|
|
175
175
|
`country`. The pair is the key because the register scopes a source to one publisher in one
|
|
176
176
|
jurisdiction while one adapter can serve several: `ban` emits eleven jurisdictions from one schema,
|
|
177
177
|
each under its own license decision. Eleven sources declare an adapter today, all of them BAN's, so a
|
|
178
|
-
release-eligible build admits those eleven and refuses every other row with the jurisdiction
|
|
179
|
-
A source that declares no adapter stays out of that map
|
|
180
|
-
since no adapter emits
|
|
178
|
+
release-eligible build admits those eleven and refuses every other row with an error that identifies the jurisdiction.
|
|
179
|
+
A source that declares no adapter stays out of that map. That exclusion is correct rather than
|
|
180
|
+
restrictive, since no adapter emits the source and no row can arrive under it.
|
|
181
181
|
|
|
182
182
|
### Licenses: 54 read, 372 unchecked
|
|
183
183
|
|
|
184
184
|
All 426 rows point at a decision record of their own. 47 are `elected`, 7 are `refused`, and the
|
|
185
185
|
other 372 read `unchecked`, which is a finding rather than a placeholder. The 253 web-researched rows
|
|
186
|
-
read `CHECK NATIONAL / DATASET TERMS` verbatim. The other 173
|
|
187
|
-
memo
|
|
188
|
-
six compound labels over the remaining 24, as `Free/varied` on 7 and `Free/gated` on 6
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
`unchecked` is a first-class value distinct from a
|
|
193
|
-
|
|
186
|
+
read `CHECK NATIONAL / DATASET TERMS` verbatim. The other 173 have an access label from the original
|
|
187
|
+
memo. The labels are `Free` on 130 rows, `Free-reg` on 15, `Gated` on 2, `Licensed` on 1 and `Varied`
|
|
188
|
+
on 1, with six compound labels over the remaining 24, such as `Free/varied` on 7 and `Free/gated` on 6.
|
|
189
|
+
Such a label states what the download costs and grants no rights. A build that treated one as
|
|
190
|
+
permissive would admit a source on a sentence about price.
|
|
191
|
+
|
|
192
|
+
`unchecked` is a first-class value distinct from a license reviewed and found permissive. The
|
|
193
|
+
`elected` value records the reviewed case: the terms, the retrieved copy, the version, and why that grant rather than
|
|
194
194
|
another. `electedLicenseLabel()` is what the mechanical prefix filter in `utils/license.ts` reads,
|
|
195
195
|
and it answers `undefined` for anything not elected.
|
|
196
196
|
|
|
197
197
|
## `licenses/` — the retrieved texts a decision would be read from
|
|
198
198
|
|
|
199
199
|
Seven grants as retrieved, each with the URL and the retrieval date at its head, plus the failures a
|
|
200
|
-
fetch returned.
|
|
201
|
-
retrieval
|
|
200
|
+
fetch returned. Each file is written by hand from a fetch rather than by a build, and is never
|
|
201
|
+
edited after retrieval. A correction is a new retrieval.
|
|
202
202
|
|
|
203
|
-
The directory
|
|
204
|
-
retrieved, and the two readings a retrieval moved.
|
|
205
|
-
the directory is accounted for rather than overlooked.
|
|
203
|
+
The directory has its own `PROVENANCE.md`. That file lists the directory's contents, the texts that
|
|
204
|
+
could not be retrieved, and the two readings a retrieval moved. This section points to it so a reader
|
|
205
|
+
of this file knows the directory is accounted for rather than overlooked.
|
|
206
206
|
|
|
207
|
-
`ElectedLicense.retrievedCopy` in the source register
|
|
208
|
-
recorded in `license-decisions.json` points at the text it was made from.
|
|
207
|
+
`ElectedLicense.retrievedCopy` in the source register holds the path of a file there. Through that
|
|
208
|
+
path, a decision recorded in `license-decisions.json` points at the text it was made from.
|
|
209
209
|
|
|
210
|
-
## `license-decisions.json` —
|
|
210
|
+
## `license-decisions.json` — license decisions a reviewer made by reading terms (#2351)
|
|
211
211
|
|
|
212
212
|
Hand-written, and the only file in this directory that is. `address-source-register.json` is
|
|
213
213
|
generated and rewritten whole on every build, so a decision recorded there is erased by the next run
|
|
214
214
|
with no error and a normal-looking count. This file is an input to that build instead.
|
|
215
215
|
|
|
216
216
|
Each key is a `licenseID` the register already declares. Each value is an `ElectedLicense` or a
|
|
217
|
-
`RefusedLicense` minus its `licenseID
|
|
218
|
-
two decisions. `buildSourceRegister` applies these over the `unchecked` defaults it derives from the
|
|
219
|
-
research pass's access labels, and refuses a decision naming a
|
|
217
|
+
`RefusedLicense` minus its `licenseID`. The build fills the id from the key, so one license cannot
|
|
218
|
+
hold two decisions. `buildSourceRegister` applies these over the `unchecked` defaults it derives from the
|
|
219
|
+
research pass's access labels, and refuses a decision naming a license no source points at.
|
|
220
220
|
|
|
221
|
-
An election records four things
|
|
221
|
+
An election records the four things the corpus acceptance rules ask for: the elected terms, the
|
|
222
222
|
copy they were read from, that copy's version where the publisher gives one, and why this grant and
|
|
223
223
|
not another. `auditAddressSourceRegister` refuses an incomplete record and the build refuses a
|
|
224
224
|
register that fails its audit, so an incomplete entry never reaches the committed artifact.
|
|
@@ -230,10 +230,10 @@ SIRENE in each of nine overseas territories. Kadaster dedicates the Dutch addres
|
|
|
230
230
|
domain under CC0 1.0, Klimadatastyrelsen licenses the Danish one under CC BY 4.0, and ČÚZK states
|
|
231
231
|
`žádné podmínky neplatí` on all 6,259 rights elements of the Czech feed. Maa- ja Ruumiamet dedicates
|
|
232
232
|
the Estonian theme to the public domain under CC0 1.0, stated in the ISO 19139 dataset record its own
|
|
233
|
-
WFS capabilities links rather than in the capabilities document
|
|
234
|
-
read `puudub` and describe the service. The Dirección General del
|
|
233
|
+
WFS capabilities links rather than in the capabilities document. That document's `Fees` and
|
|
234
|
+
`AccessConstraints` read `puudub` and describe the service. The Dirección General del
|
|
235
235
|
Catastro grants its own INSPIRE
|
|
236
|
-
access-and-use
|
|
236
|
+
access-and-use license over the cadastral addresses of 52 Spanish territorial offices, and the three
|
|
237
237
|
foral councils that publish through the same national feed grant their own terms over the rest of
|
|
238
238
|
Spain's cadastral addresses: Navarra under `CC-BY-4.0`, Gipuzkoa under `CC-BY-SA-4.0`, and Bizkaia
|
|
239
239
|
under a statement naming no instrument. Each is retained under
|
|
@@ -246,43 +246,43 @@ upphovsrätt. Avtal för användning krävs, avgifter för användning tas ut`.
|
|
|
246
246
|
`unchecked`.
|
|
247
247
|
|
|
248
248
|
**A refusal is recorded as a decision rather than left as an absence, and the research pass's access
|
|
249
|
-
label can contradict it.** Sweden's row
|
|
250
|
-
|
|
249
|
+
label can contradict it.** Sweden's row has the label `Free`. That label describes access to the feed.
|
|
250
|
+
The feed itself requires an agreement and charges fees for use. The label is what prompted the
|
|
251
251
|
reading, and the reading is what the register records.
|
|
252
252
|
|
|
253
253
|
A decision may name a body declared once under `sharedReadings` rather than restating it. One publisher
|
|
254
|
-
can hold a
|
|
255
|
-
jurisdiction, so one reading of INSEE's terms produces nine decisions. Each
|
|
256
|
-
own, so no election spans two jurisdictions
|
|
254
|
+
can hold a license over several jurisdictions, and a decision is scoped to one publisher in one
|
|
255
|
+
jurisdiction, so one reading of INSEE's terms produces nine decisions. Each license id still has its
|
|
256
|
+
own decision, so no election spans two jurisdictions. The shared body removes the risk of nine hand-maintained
|
|
257
257
|
copies drifting. The build refuses a `sameAs` naming a reading the file does not declare.
|
|
258
258
|
|
|
259
|
-
|
|
260
|
-
address role per column, measured coverage, and a personal-data review that does not read
|
|
261
|
-
|
|
262
|
-
have exactly one blocking condition left, and fifteen of the sixteen
|
|
263
|
-
reading
|
|
264
|
-
a question about one. The sixteenth is Uruguay, blocked on its address role instead.
|
|
259
|
+
An election of terms is necessary and not sufficient. `ingestEligibilityProblems` also requires a
|
|
260
|
+
resolved address role per column, measured coverage, and a personal-data review that does not read
|
|
261
|
+
`present`. A license decision touches none of these. They live in `source-resolutions.json`. **Sixteen
|
|
262
|
+
sources have exactly one blocking condition left, and for fifteen of the sixteen that condition is the
|
|
263
|
+
personal-data reading.** The personal-data question is therefore one decision that blocks fifteen
|
|
264
|
+
sources rather than a question about one. The sixteenth is Uruguay, blocked on its address role instead.
|
|
265
265
|
|
|
266
266
|
### Why there are 426 of them rather than 12
|
|
267
267
|
|
|
268
|
-
A decision is scoped to one publisher in one jurisdiction, so the register
|
|
269
|
-
|
|
268
|
+
A decision is scoped to one publisher in one jurisdiction, so the register holds one per source.
|
|
269
|
+
A decision keyed only by the research pass's access label was as wide as the label:
|
|
270
270
|
`CHECK NATIONAL / DATASET TERMS` covers 253 of the 426 sources across 228 publishers, and `Free`
|
|
271
|
-
covers 130 across 22.
|
|
271
|
+
covers 130 across 22. One election recorded against either would have granted every source under it on a
|
|
272
272
|
single reading of one publisher's terms.
|
|
273
273
|
|
|
274
|
-
Spain was the worked case of the same defect inside one row. `es-mixed-regulatory-1`
|
|
275
|
-
`Catastro, INE` and
|
|
276
|
-
institutions. The same national feed
|
|
277
|
-
same pass gave three rows of their own. The 2026-09-30 rights review split the row per publisher
|
|
278
|
-
row's scope onto its own publisher.
|
|
279
|
-
that
|
|
274
|
+
Spain was the worked case of the same defect inside one row. `es-mixed-regulatory-1` had the
|
|
275
|
+
name `Catastro, INE` and no publisher, so its decision scoped to the source id and covered two
|
|
276
|
+
institutions. The same national feed publishes three foral cadastres under their own terms, and the
|
|
277
|
+
same pass gave them three rows of their own. The 2026-09-30 rights review split the row per publisher.
|
|
278
|
+
Each row's scope then moved onto its own publisher. The scoping works only when a row identifies one
|
|
279
|
+
publisher, and a row that lists two defeats it.
|
|
280
280
|
|
|
281
281
|
The jurisdiction is part of the scope because a publisher name is not unique across states. The pass
|
|
282
282
|
wrote `Ministry of Justice` for Belarus, Lebanon and Timor-Leste, `Ministry of Commerce and Industry`
|
|
283
283
|
for five countries, and `Commercial-registration authority` as a description rather than a name.
|
|
284
284
|
|
|
285
|
-
The cost falls on a publisher that
|
|
285
|
+
The cost falls on a publisher that serves several jurisdictions. INSEE covers mainland
|
|
286
286
|
France and nine overseas territories, so it holds ten decisions, and a reviewer who reads its terms
|
|
287
287
|
once records that conclusion ten times. That is the direction to err in: an over-wide election is the
|
|
288
288
|
failure review cannot undo.
|
|
@@ -292,53 +292,53 @@ failure review cannot undo.
|
|
|
292
292
|
Hand-written, and an input to the register build for the same reason as `license-decisions.json`: the
|
|
293
293
|
build rewrites the register whole.
|
|
294
294
|
|
|
295
|
-
Each key is a `sourceID` the register declares, and each value
|
|
295
|
+
Each key is a `sourceID` the register declares, and each value holds any of `addressRoles`,
|
|
296
296
|
`coverage`, `upstreamLineage` and `personalDataReview`. `buildSourceRegister` applies these over the
|
|
297
297
|
rows it derives from the research CSV and refuses a resolution naming a source the register does not
|
|
298
|
-
|
|
298
|
+
declare, because a resolution that resolves no row is a typo or a source that has been removed.
|
|
299
299
|
|
|
300
300
|
The fields live here rather than in the CSV because the CSV is the record of a research pass. All 390
|
|
301
301
|
of its non-rail rows write `varies` for `address_role`, `country-specific` for `coverage` and an empty
|
|
302
|
-
`upstream`, and no pass has a personal-data column at all.
|
|
302
|
+
`upstream`, and no pass has a personal-data column at all. An edit to those placeholders would change
|
|
303
303
|
what a pass is recorded as having found. A row's other columns are a different matter: a later pass
|
|
304
304
|
rewrites one when reading the publisher settles something the earlier pass recorded wrongly, and the
|
|
305
305
|
row's `origin` column then names that later pass so the change is legible.
|
|
306
306
|
|
|
307
307
|
`addressRoles` is keyed by the address column's path in the published record rather than holding one
|
|
308
|
-
role for the source, because a publication can
|
|
308
|
+
role for the source, because a publication can hold two roles on one record. AusTender's OCDS release
|
|
309
309
|
gives a `counterparty` address for the supplier and a `facility` address for the procuring entity on
|
|
310
310
|
every contracting process. Taiwan's GCIS register gives a registered company address and a tax-office
|
|
311
311
|
business address in separate columns. `ingestEligibilityProblems` requires at least one column to have
|
|
312
|
-
a role, and the audit refuses an empty map
|
|
312
|
+
a role, and the audit refuses an empty map. An empty map would read as resolved while it states no role.
|
|
313
313
|
|
|
314
|
-
**Fifty-four sources of 426
|
|
315
|
-
Five of those fifty-four
|
|
316
|
-
reachable data
|
|
314
|
+
**Fifty-four sources of 426 have at least one resolved field, and forty-five have all four.**
|
|
315
|
+
Five of those fifty-four have only `upstreamLineage`. Those five are located publishers with no
|
|
316
|
+
reachable data: Aruba, Curaçao, Sint Maarten, the Caribbean Netherlands and Hong Kong. AusTender
|
|
317
317
|
and five OCDS procurement releases are recorded from whole-year measurements of the published data.
|
|
318
318
|
The nine SIRENE territories are recorded from INSEE's own `dessin de fichier`. Seven of the fifteen
|
|
319
319
|
address publications are recorded from an element census over a whole GML file, the largest being the
|
|
320
320
|
Netherlands' 29,677,448,685 bytes and 10,066,060 features and the newest Wallonia's 5,163,917,131
|
|
321
321
|
bytes and 1,772,312 features. Denmark's is recorded from the column schema of its GeoPackage instead,
|
|
322
322
|
because it publishes the theme as a database, and Finland's and France's from the headers of their
|
|
323
|
-
CSVs
|
|
323
|
+
CSVs. Every row of those CSVs has exactly the fields its header declares, 24 for Finland and 23 for
|
|
324
324
|
France. The remaining four are recorded from every
|
|
325
325
|
element of every feature type their WFS publishes, because none of them publishes a file: five types
|
|
326
326
|
and 101 properties for Estonia, five and 44 for Slovakia, four and 32 for Flanders, and one type and
|
|
327
327
|
19 elements for Poland. Each is retained under `internal/strategy/rights-receipts/`.
|
|
328
328
|
|
|
329
329
|
**A whole-file census answers what a sample cannot.** Wallonia's 56 element names were counted over
|
|
330
|
-
all 5,163,917,131 bytes
|
|
330
|
+
all 5,163,917,131 bytes. That count shows that `ad:locatorName` appears 0 times in the publication
|
|
331
331
|
rather than 0 times in the rows that were read. The count is self-checking: `ad:Address`,
|
|
332
332
|
`ad:AddressLocator`, `ad:GeographicPosition`, `ad:locator` and `ad:position` each appear 1,772,312
|
|
333
333
|
times, as a GML file holds one of each per address, and those 1,772,312 addresses plus the file's
|
|
334
334
|
55,766 thoroughfare names, 1,963 address-area names, 598 postal descriptors and 264 administrative-unit
|
|
335
335
|
names equal its 1,830,903 `gml:featureMember` elements exactly. A first pass reported those five
|
|
336
|
-
counts as 1,772,315, 1,772,316 and 1,772,314
|
|
337
|
-
|
|
336
|
+
counts as 1,772,315, 1,772,316 and 1,772,314. Counts slightly above the true value are the symptom
|
|
337
|
+
of a chunked reader that re-scans its own overlap.
|
|
338
338
|
|
|
339
339
|
**Seventeen sources are ingest-eligible and fifteen have one blocking condition each.** Fourteen of the
|
|
340
340
|
fifteen read `personalDataReview: present`. The fifteenth is Uruguay, refused on its address role: no
|
|
341
|
-
party object in 127,885
|
|
341
|
+
party object in 127,885 has an address field of any kind, so that publication is reachable in bulk
|
|
342
342
|
and is not an address source.
|
|
343
343
|
|
|
344
344
|
Sixteen of the seventeen eligible sources are national or regional address publications of European
|
|
@@ -351,32 +351,32 @@ Walloon Region's Service public de Wallonie under CC BY 4.0, Finland's Suomen ym
|
|
|
351
351
|
CC BY 4.0, France's IGN under Licence Ouverte 2.0, Luxembourg's Administration du cadastre et de
|
|
352
352
|
la topographie under CC0 1.0, and Norway's Kartverket under CC BY 4.0.
|
|
353
353
|
|
|
354
|
-
**An elected
|
|
354
|
+
**An elected license does not make a source eligible, and Switzerland is the worked case.** swisstopo's
|
|
355
355
|
own terms grant use, distribution, enrichment, processing and commercial use against a mandatory
|
|
356
|
-
source reference, so its
|
|
357
|
-
`ingestEligibilityProblems` refuses on that
|
|
358
|
-
|
|
356
|
+
source reference, so its license is elected. Its `personalDataReview` reads `present`, and
|
|
357
|
+
`ingestEligibilityProblems` refuses on that condition by itself, so the 3,303,210-row directory stays
|
|
358
|
+
out. The license and the eligibility are separate answers and the register keeps them separate.
|
|
359
359
|
|
|
360
360
|
**The largest of them is France's Base Adresse Nationale at 26,117,350 rows**, measured by parsing
|
|
361
361
|
the whole file, against the `plus de 25 millions d'adresses` its publisher states. Poland's 8,625,921
|
|
362
362
|
follows, then the Flemish Region's 4,563,062, Finland's 3,862,373, the Walloon Region's 1,772,312 and
|
|
363
363
|
Slovakia's 1,704,196. The INSPIRE Addresses model has no element for a party, so a party reaches a
|
|
364
|
-
|
|
364
|
+
harmonized publication only through a free-text name element. INSPIRE Annex I makes Addresses a
|
|
365
365
|
mandatory theme for every EU member state, so the remaining states are the place to look for more.
|
|
366
366
|
|
|
367
367
|
**Belgium takes three rows and Finland's is not an INSPIRE service at all.** Belgium's regions publish
|
|
368
368
|
separately, so Flanders, Wallonia and Brussels each need their own row, of which the first two are
|
|
369
369
|
recorded. Finland's eligible publication is a pair of bulk files on the publisher's own host rather
|
|
370
|
-
than a WFS or an ATOM feed
|
|
370
|
+
than a WFS or an ATOM feed. For that reason `addressRoles` keys it by two CSV columns, `address_fin` and
|
|
371
371
|
`address_swe`, each holding a complete address in its own language.
|
|
372
372
|
|
|
373
373
|
**A service may answer the theme's name with the national schema.** Poland's download service
|
|
374
374
|
publishes one feature type, `ms:AD.Address`, whose element names are the Polish register's own
|
|
375
|
-
columns: `miejscowosc`, `ulica`, `numer`, `kod` and `adres
|
|
375
|
+
columns: `miejscowosc`, `ulica`, `numer`, `kod` and `adres`. The last holds the whole address as
|
|
376
376
|
one string. No `ad:ThoroughfareName`, `ad:PostalDescriptor` or `ad:AdminUnitName` type exists there to
|
|
377
377
|
join to, so that build needs no component resolution and the personal-data review has to ask what the
|
|
378
|
-
columns are rather than whether `AD:LocatorName` is populated.
|
|
379
|
-
of the
|
|
378
|
+
columns are rather than whether `AD:LocatorName` is populated. A reader that took the theme's name as a promise
|
|
379
|
+
of the harmonized model would have found no matching elements.
|
|
380
380
|
|
|
381
381
|
**A regional row covers a region.** The Flemish theme's 4,563,062 addresses are the Flemish Region,
|
|
382
382
|
and Brussels and Wallonia publish their own services, so a Belgian build needs three rows where a
|
|
@@ -385,12 +385,12 @@ cadastres.
|
|
|
385
385
|
|
|
386
386
|
**Switzerland is why that element is opened on every source rather than assumed empty.** Its
|
|
387
387
|
`BDG_NAME` is populated on 20,989 of 3,303,210 rows, 0.64%, and every populated value was read: of
|
|
388
|
-
1,158 matching a two-
|
|
388
|
+
1,158 matching a two-capitalized-word shape, 684 contain a structure word and 474 do not, and **at least
|
|
389
389
|
twelve of those 474 are a surname and a given name on a row whose category reads `residential`** —
|
|
390
390
|
`Chaignat Aimée` in Jura, `Rutz Nelly` and `Frigerio Cesare Vittorio` in Ticino, `NB EFH Lauber Jörg`
|
|
391
|
-
in Valais among them. That is a natural person at their home
|
|
392
|
-
|
|
393
|
-
farm or building name
|
|
391
|
+
in Valais among them. That is a natural person at their home. On 2026-09-30 this repository
|
|
392
|
+
decided it would not include such a row. Every other address publication's name element has read as a place,
|
|
393
|
+
farm or building name. This one does not, and a reader that had assumed the element empty would have
|
|
394
394
|
admitted it.
|
|
395
395
|
|
|
396
396
|
`AD:LocatorName` is that free-text element in the INSPIRE model, typed `GeographicalName`, and each
|
|
@@ -401,49 +401,50 @@ than assumed empty. The Dutch, Czech, Danish, Slovak and Flemish themes leave it
|
|
|
401
401
|
measured at 0 of 800 addresses over 40 windows across its 4,563,062 and Slovakia at 0 of 800 over 40
|
|
402
402
|
windows across its 1,704,196. Estonia fills it on 1,200 of 1,200 addresses sampled over 60 windows
|
|
403
403
|
spread across its 729,973, every one typed `siteName` or `buildingName` in the INSPIRE codelist, with
|
|
404
|
-
0 of the 1,200
|
|
405
|
-
|
|
404
|
+
0 of the 1,200 in the given-name-and-family-name shape. The values are Estonian farm names. A rural
|
|
405
|
+
Estonian address designates its addressable unit by a farm name.
|
|
406
406
|
|
|
407
407
|
**A reader that finds the element on one service can miss it on another.** Slovakia nests the locator
|
|
408
408
|
and omits the `name` key entirely, so its absence is structural as well as sampled. Estonia flattens
|
|
409
409
|
the path to `locator_addresslocator_name_locatorname_name_spelling_text`. One traversal run against
|
|
410
|
-
both would report one of them wrongly
|
|
411
|
-
read, and
|
|
410
|
+
both would report one of them wrongly. For that reason each service's measurement states the shape
|
|
411
|
+
it read, and Slovakia's traversal was checked against Estonia before its 0 was recorded.
|
|
412
412
|
|
|
413
|
-
Those publishers state seven different grants,
|
|
414
|
-
|
|
413
|
+
Those publishers state seven different grants, so a mandatory EU theme does not come with uniform
|
|
414
|
+
terms. Kadaster and Maa- ja Ruumiamet dedicate to the public
|
|
415
415
|
domain, Denmark, Navarra, Wallonia and Finland's SYKE license under CC BY 4.0, Gipuzkoa under
|
|
416
416
|
CC BY-SA 4.0, ČÚZK, the Slovak Ministerstvo vnútra and Poland's GUGiK state the INSPIRE controlled
|
|
417
417
|
value for an absence of conditions, each in its own language, IGN grants Licence Ouverte 2.0 over the
|
|
418
418
|
Base Adresse Nationale, Catastro and Bizkaia grant their own
|
|
419
|
-
access-and-use terms, and Flanders grants its own model
|
|
420
|
-
attribution the sole condition
|
|
421
|
-
property in works created from the data. The Directive mandates the data rather than its
|
|
419
|
+
access-and-use terms, and Flanders grants its own model license for no-charge re-use. Article 4 of that
|
|
420
|
+
license makes attribution the sole condition. Article 5 leaves the licensee holding the intellectual
|
|
421
|
+
property in works created from the data. The Directive mandates the data rather than its license.
|
|
422
422
|
|
|
423
|
-
**A publisher may state a
|
|
423
|
+
**A publisher may state a license that covers part of what it publishes.** SYKE's record reads
|
|
424
424
|
`Avoimen tiedon osalta Creative Commons Nimeä 4.0 Kansainvälinen … Muiden tietotuotteiden osalta
|
|
425
425
|
käyttöehdot on määritelty tietoluvassa`: CC BY 4.0 for the open-data part, and a data permit for other
|
|
426
|
-
products.
|
|
427
|
-
terms page
|
|
428
|
-
|
|
426
|
+
products. A reading of the first clause without the second would grant what the second withholds.
|
|
427
|
+
The publisher's own terms page therefore decides which files are which, and the two files this
|
|
428
|
+
register records have the name `open_address`.
|
|
429
|
+
Finland's other address publication states the permit requirement in its own text: the population register's
|
|
429
430
|
building and dwelling data reads `Tietojen käyttöön saaminen edellyttää Digi- ja väestötietoviraston
|
|
430
431
|
myöntämää käyttöoikeuslupaa`.
|
|
431
432
|
|
|
432
|
-
A member state's WFS states its own access terms in fields that are not the data's
|
|
433
|
+
A member state's WFS states its own access terms in fields that are not the data's license. `Fees`
|
|
433
434
|
and `AccessConstraints` in a capabilities document describe what calling the service costs, and the
|
|
434
435
|
grant over the data sits in the ISO 19139 dataset record the document links through
|
|
435
436
|
`inspire_dls:SpatialDataSetIdentifier`. Estonia's capabilities read `puudub` in both fields while its
|
|
436
437
|
dataset record grants CC0 1.0, and Slovakia's read `NONE` while its register record states that no
|
|
437
|
-
conditions apply and
|
|
438
|
-
Flanders states the price of the service in `Fees` and sends the reader to
|
|
439
|
-
the data's terms
|
|
440
|
-
`No constraints` while its dataset record states
|
|
441
|
-
record is linked per feature type rather than through the
|
|
442
|
-
`inspire_dls:SpatialDataSetIdentifier` states the unreplaced placeholder `mycode`.
|
|
438
|
+
conditions apply and identifies CC0. A reader that takes the service fields as the license
|
|
439
|
+
understates both grants. Flanders states the price of the service in `Fees` and sends the reader to
|
|
440
|
+
each dataset's metadata for the data's terms. That metadata identifies its model license. Poland's
|
|
441
|
+
read `No fees` and `No constraints` while its dataset record states
|
|
442
|
+
`Brak warunków dostępu i użytkowania`. That record is linked per feature type rather than through the
|
|
443
|
+
extended capabilities. The capabilities' `inspire_dls:SpatialDataSetIdentifier` states the unreplaced placeholder `mycode`.
|
|
443
444
|
|
|
444
445
|
**A service's own count can be wrong in more than one direction at once.** Poland states three
|
|
445
|
-
numbers for one type and none is its count: a bare `resultType=hits` answers 1000,
|
|
446
|
-
MapServer
|
|
446
|
+
numbers for one type and none is its count: a bare `resultType=hits` answers 1000, the
|
|
447
|
+
MapServer default feature cap. The same request at `startIndex=1` answers 1, and a feature page
|
|
447
448
|
answers `unknown`. `countWFSFeaturesByPaging` in `@mailwoman/core/api` measures such a type by asking
|
|
448
449
|
which indices hold a feature, doubling then bisecting, and a page straddling the last index confirms
|
|
449
450
|
the boundary. Poland's type holds 8,625,921 features, measured in 51 requests. That function refuses
|
|
@@ -451,42 +452,42 @@ a service which answers two different `startIndex` values with the same leading
|
|
|
451
452
|
paging an ignored parameter would measure the page cap instead.
|
|
452
453
|
|
|
453
454
|
**`Fees` and `AccessConstraints` can disagree in the same document.** Austria's BEV Addresses service
|
|
454
|
-
states `Fees` `no conditions apply` and `AccessConstraints` `restricted`, and its path
|
|
455
|
-
customer token.
|
|
456
|
-
|
|
455
|
+
states `Fees` `no conditions apply` and `AccessConstraints` `restricted`, and its path contains a
|
|
456
|
+
customer token. A reader of the fee field without the access field records that service as free when
|
|
457
|
+
its own access field states a restriction, so both are read before a service is called reachable.
|
|
457
458
|
|
|
458
459
|
**An HTTP 422 is a validation message naming the fix, where a 404 is an absence.** The export URL in
|
|
459
460
|
Slovakia's own capabilities document answers 422 with
|
|
460
|
-
`{"loc":["path","collection_record_id"],"msg":"value is not a valid uuid"}
|
|
461
|
-
the path where the capabilities put it in the query
|
|
462
|
-
`export_format` values,
|
|
461
|
+
`{"loc":["path","collection_record_id"],"msg":"value is not a valid uuid"}`. That message places the
|
|
462
|
+
id in the path where the capabilities put it in the query. A second 422 enumerates the permitted
|
|
463
|
+
`export_format` values, and they exclude the `gmd` the capabilities asks for. Those two bodies
|
|
463
464
|
turned a record recorded as unreachable into
|
|
464
465
|
`https://rpi.gov.sk/api/collection_record/86dea70e-a55b-4241-bd63-4024b3f76b72`. Its own
|
|
465
|
-
`downloadable_urls` holds the service's capabilities URL
|
|
466
|
-
|
|
466
|
+
`downloadable_urls` holds the service's capabilities URL. That URL ties the record to the service,
|
|
467
|
+
rather than a title match.
|
|
467
468
|
|
|
468
|
-
**A dataset record may
|
|
469
|
+
**A dataset record may state more than one instrument, in separate constraint blocks that answer
|
|
469
470
|
different questions.** The Flemish Adressen record states `Geen beperkingen op de publieke toegang` in
|
|
470
|
-
its access block, and in its re-use block names four things: the model
|
|
471
|
+
its access block, and in its re-use block names four things: the model license for no-charge re-use, two
|
|
471
472
|
instruments governing public-task use by GDI-Vlaanderen participants and by bodies that are not
|
|
472
|
-
participants, and the attribution string the licensor requires, `Bron: Digitaal Vlaanderen`.
|
|
473
|
-
those blocks loses which instrument governs which re-user.
|
|
473
|
+
participants, and the attribution string the licensor requires, `Bron: Digitaal Vlaanderen`. A reader
|
|
474
|
+
that flattens those blocks loses which instrument governs which re-user.
|
|
474
475
|
|
|
475
476
|
**An identifier that will not serve its own text may serve it under a file extension.** The Flemish
|
|
476
|
-
|
|
477
|
+
license identifier answers `text/html`, `text/turtle`, `application/ld+json` and `application/rdf+xml`
|
|
477
478
|
with one 541,080-byte application rendering to its title. The same path with `.ttl` appended returns
|
|
478
|
-
the RDF
|
|
479
|
-
|
|
479
|
+
the RDF. The RDF states `cc:requires cc:Attribution`, and the record's `rdfs:seeAlso` gives the page
|
|
480
|
+
that publishes the nine articles.
|
|
480
481
|
|
|
481
482
|
Denmark also shows that the theme is not always GML: it serves a GeoPackage, so its `addressRoles`
|
|
482
483
|
is keyed by a table name rather than an element name.
|
|
483
484
|
|
|
484
485
|
The eighth is Italy's ANAC procurement release, and it is eligible for a different reason.
|
|
485
486
|
`internal/strategy/personal-data-policy-2026-09-30.md` records the operator's decision that a corpus
|
|
486
|
-
|
|
487
|
+
contains the addresses of organizations and no address of an identifiable natural person. Italy's
|
|
487
488
|
release names natural persons among its suppliers and records an address for none of them: 0 of
|
|
488
|
-
11,075 supplier parties
|
|
489
|
-
or `payer`, both Italian public entities. Its `addressRoles`
|
|
489
|
+
11,075 supplier parties have an `address` object, and every address in the release sits on `buyer`
|
|
490
|
+
or `payer`, both Italian public entities. Its `addressRoles` lists only those two fields. That
|
|
490
491
|
reading is `assessed` rather than `absent`, because the publication does name people.
|
|
491
492
|
|
|
492
493
|
The other `verified-corpus` sources are business registers and procurement portals where the address
|
|
@@ -494,19 +495,19 @@ belongs to a party, and some of those parties are natural persons trading on the
|
|
|
494
495
|
policy document records why a name-shape filter, an organization identifier and a diffusion-status
|
|
495
496
|
filter each fail to separate the two kinds of party.
|
|
496
497
|
|
|
497
|
-
Every measured source places the role differently
|
|
498
|
-
than one role per source.
|
|
498
|
+
Every measured source places the role differently. For that reason the field is a map keyed by a path
|
|
499
|
+
rather than one role per source.
|
|
499
500
|
|
|
500
|
-
- AusTender
|
|
501
|
-
- Companies House
|
|
502
|
-
- Catastro's INSPIRE AD theme
|
|
501
|
+
- AusTender has two roles in two column families on one record.
|
|
502
|
+
- Companies House has one role in one family.
|
|
503
|
+
- Catastro's INSPIRE AD theme has one role on the record itself rather than in a column. The
|
|
503
504
|
feature is the address, so the map is keyed by the feature's element name, `AD:Address`.
|
|
504
|
-
- SIRENE
|
|
505
|
-
unit's seat, so the same columns are a `registered-office` address on one row and a `facility` address
|
|
505
|
+
- SIRENE records the role in a row-level flag. `etablissementSiege` states whether the establishment is
|
|
506
|
+
its legal unit's seat, so the same columns are a `registered-office` address on one row and a `facility` address
|
|
506
507
|
on the next.
|
|
507
508
|
- An OCDS release keys addresses by party role, and the roles differ per publisher. Germany uses fifteen
|
|
508
|
-
where AusTender uses two. Italy's `supplier` role
|
|
509
|
-
`buyer`'s and a `payer`'s
|
|
509
|
+
where AusTender uses two. Italy's `supplier` role has no address at all, so its addresses are a
|
|
510
|
+
`buyer`'s and a `payer`'s. That is AusTender's shape inverted. Chile records a region and neither a
|
|
510
511
|
postcode nor a locality on any of 544,754 party objects.
|
|
511
512
|
|
|
512
513
|
An OCDS role takes a role here only where OCDS's own codelist defines it. Germany's eight extension roles
|
|
@@ -515,15 +516,15 @@ take none rather than a guessed one.
|
|
|
515
516
|
## `training-manifests/<version>.json` — what reached one base corpus (#2375)
|
|
516
517
|
|
|
517
518
|
The frozen training manifest `buildCorpus` writes beside the corpus it produced, copied here so a release
|
|
518
|
-
path can read it. Each file
|
|
519
|
-
the register decision in force at build time
|
|
519
|
+
path can read it. Each file lists the sources that contributed, their row counts, their license label, and
|
|
520
|
+
the register decision in force at build time. It also stores a `contentDigest` over its own contents. The
|
|
520
521
|
corpus `MANIFEST.json` records the same digest as `training_manifest_digest`, so a committed copy that
|
|
521
|
-
drifted from the build it claims is detectable
|
|
522
|
+
drifted from the build it claims is detectable. `auditTrainingManifest` recomputes the digest and compares
|
|
522
523
|
the source rows against `totalRows`.
|
|
523
524
|
|
|
524
525
|
`mwops release rights-audit` looks a manifest up by the `training.corpus_version` a package's
|
|
525
|
-
`model-card.json` names. Observed 2026-09-28: of the twelve `neural-weights-*` cards, ten
|
|
526
|
-
`training.corpus_version` at all and the two that do
|
|
526
|
+
`model-card.json` names. Observed 2026-09-28: of the twelve `neural-weights-*` cards, ten have no
|
|
527
|
+
`training.corpus_version` at all and the two that do hold prose rather than a version, so no card yet
|
|
527
528
|
resolves to a file here. The record exists for the corpus the next base model trains on.
|
|
528
529
|
|
|
529
530
|
`v0.7.0-de-holdout.json` is byte-identical to
|
|
@@ -546,11 +547,11 @@ establishes is which path each adapter was pointed at. A corpus `MANIFEST.json`
|
|
|
546
547
|
`assembleOverlayManifest` writes it next to the `MANIFEST.json` it produced, and the copy here is the one a
|
|
547
548
|
later assembly on the same base reads. The `v0.6.0-register-surface` plan (66 files in three lists) was
|
|
548
549
|
produced by three scripts in a session scratch directory with the old data root written into them, and was
|
|
549
|
-
|
|
550
|
+
passed into the next assembly as a hand-written list.
|
|
550
551
|
|
|
551
|
-
No plan exists for `v0.7.0-de-holdout`. Its overlays were assembled before `assembleOverlayManifest`
|
|
552
|
+
No plan exists for `v0.7.0-de-holdout`. Its corpus overlays were assembled before `assembleOverlayManifest`
|
|
552
553
|
wrote one, and its `MANIFEST.json` records the `split`, `source` and `path` of each of the 71 labeled
|
|
553
|
-
|
|
554
|
+
overlay files while recording the `route` of none. A plan reconstructed from the manifest would therefore assert
|
|
554
555
|
a route it cannot read, and that reconstruction is the mechanism `OverlayPlan`'s docstring names: a
|
|
555
556
|
hand-placed split reached `v0.6.0-register-surface` through the caller route and put 770 of DE's
|
|
556
557
|
validation `source_id`s in train, and the filename was the only record that showed it (#2359). The next
|
|
@@ -570,15 +571,15 @@ weight, the sampler, `augment_exclude_sources` and every overlay merged after th
|
|
|
570
571
|
written. `effective-manifest.run.ts` derives it from the frozen manifest and an `audit_epoch_mixture`
|
|
571
572
|
output, and `mwops release rights-audit` reads `<corpus_version>.effective.json` beside the frozen file.
|
|
572
573
|
|
|
573
|
-
Each row
|
|
574
|
+
Each row records the corpus row count, the config weight, the rows drawn, the rows emitted after
|
|
574
575
|
augmentation, and the reason an excluded source was excluded. `emittedButUnrecorded` holds the sources the
|
|
575
576
|
epoch emitted that the frozen manifest does not name, with their row counts.
|
|
576
577
|
|
|
577
578
|
`v0.7.0-de-holdout.effective.json` reads the corpus through
|
|
578
579
|
`corpus-python/src/mailwoman_train/configs/v7.0.0-de-holdout-60k.yaml` at seed 43 over 1,000,000 draws. Of
|
|
579
|
-
the eleven sources the frozen manifest names, nine drew rows and two
|
|
580
|
+
the eleven sources the frozen manifest names, nine drew rows and two have weight 0.0. Thirty-nine further
|
|
580
581
|
sources were emitted under no frozen-manifest entry, 886,620 of the 1,000,000 rows, because the overlays
|
|
581
|
-
|
|
582
|
+
that contain them were merged after that manifest was written.
|
|
582
583
|
|
|
583
584
|
## The `surface` column on a corpus built up to `v0.7.0-de-holdout`
|
|
584
585
|
|
|
@@ -602,9 +603,9 @@ From the next build, an adapter that assembles a line declares `rendered`, a rec
|
|
|
602
603
|
and `attested` is reserved for a source that publishes the written line itself. No adapter in the tree
|
|
603
604
|
emits `attested` today, so a corpus reporting zero attested rows is a correct census rather than a defect.
|
|
604
605
|
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
and `wof-admin` and
|
|
606
|
+
A measurement on `v0.7.0-de-holdout`, taken when this defect was found, shows that GB holds 813,781
|
|
607
|
+
rows with a street, all of which read `composed`, from `rendered-gb`. Its 13,000,821 rows reading `attested` come from
|
|
608
|
+
`wof-postalcode` and `wof-admin` and have no street.
|
|
608
609
|
|
|
609
610
|
## `reviewed-ve-postcode-tuples.json` — reviewed Venezuelan postcode placement (#1821)
|
|
610
611
|
|
|
@@ -622,15 +623,15 @@ postcode-to-place join unchanged.
|
|
|
622
623
|
## `sub-venue-lexicon.json` — the sub-venue designator lexicon (#35, waves 1–2)
|
|
623
624
|
|
|
624
625
|
The vocabulary a corpus recipe (and eventually the span proposer) reads to recognize `Terminal 5`,
|
|
625
|
-
`North Terminal`, `Concourse B`, `第1ターミナル` as venue-interior structure.
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
decode weight would take a bias scale near 11 nats where the stronger designator+identifier evidence
|
|
626
|
+
`North Terminal`, `Concourse B`, `第1ターミナル` as venue-interior structure. The file is
|
|
627
|
+
generated and is never edited by hand. `docs/records/engineering/sub-venue-corpus-task.mdx` explains
|
|
628
|
+
why it exists. In short, the `unit` tag was never taught the modifier+designator shape, so closing the
|
|
629
|
+
class by decode weight would take a bias scale near 11 nats where the stronger designator+identifier evidence
|
|
629
630
|
needed 6.
|
|
630
631
|
|
|
631
632
|
The one hand-authored input is `corpus/src/tools/sub-venue-promotions.ts`, the curation ledger. The
|
|
632
633
|
builder cannot decide that Spanish `terminal` is safe and British `hall` is not, because that is a
|
|
633
|
-
|
|
634
|
+
judgment about a language's confounds. The builder applies those decisions.
|
|
634
635
|
|
|
635
636
|
### Regenerate
|
|
636
637
|
|
|
@@ -651,12 +652,12 @@ npx oxfmt corpus/data/sub-venue-lexicon.json
|
|
|
651
652
|
```
|
|
652
653
|
|
|
653
654
|
`$E` is `$MAILWOMAN_DATA_ROOT/sub-venue/extracts`. `--extracts` takes `REGION=path` pairs because the
|
|
654
|
-
region is the axis every curation decision is taken on, and no extract filename
|
|
655
|
-
`ile-de-france` is FR, `great-britain` is GB. A bare path lands region `""`, whose surfaces can never
|
|
655
|
+
region is the axis every curation decision is taken on, and no extract filename states it reliably.
|
|
656
|
+
`ile-de-france` is FR, and `great-britain` is GB. A bare path lands region `""`, whose surfaces can never
|
|
656
657
|
be promoted.
|
|
657
658
|
|
|
658
|
-
The oxfmt pass is required
|
|
659
|
-
reproduce. The generator is byte-deterministic and oxfmt is too, so the artifact is reproducible from
|
|
659
|
+
The oxfmt pass is required. Committed JSON is oxfmt-clean, and raw `JSON.stringify` cannot
|
|
660
|
+
reproduce that formatting. The generator is byte-deterministic and oxfmt is too, so the artifact is reproducible from
|
|
660
661
|
the same fetch outputs. `sub-venue-lexicon.test.ts` pins determinism directly.
|
|
661
662
|
|
|
662
663
|
The extract JSONLs and the `.osm.pbf` files are build inputs under
|
|
@@ -675,32 +676,32 @@ The extract JSONLs and the `.osm.pbf` files are build inputs under
|
|
|
675
676
|
| Overture Places | `poi.db` spatial layer, vintage `2026-05-20.0` | CDLA-Permissive-2.0 | 9,219 |
|
|
676
677
|
|
|
677
678
|
**The ODbL question, unchanged from wave 1 and still open for the recipe output.** The committed artifact
|
|
678
|
-
contains no OSM geometry and no OSM row. What survives the OSM
|
|
679
|
-
|
|
680
|
-
`examples` are gate
|
|
681
|
-
a substantial extraction from a database, so this table is not treated as a
|
|
682
|
-
reading matches `osm/README.md`'s
|
|
683
|
-
rather than
|
|
684
|
-
|
|
679
|
+
contains no OSM geometry and no OSM row. What survives the OSM read is surface counts, such as the count of 1,215
|
|
680
|
+
Japanese feature names that contain the token `ターミナル`. The `identifierShapes` distribution also
|
|
681
|
+
survives. Its `examples` are `aeroway=gate` `ref` strings (`B32`, `1A`, `16-18`). Facts and short
|
|
682
|
+
factual strings are not a substantial extraction from a database, so this table is not treated as a
|
|
683
|
+
Derived Database. That reading matches `osm/README.md`'s position that the ODbL obligation attaches to
|
|
684
|
+
the built recipe output rather than to code. **Whether a corpus recipe output built from OSM rows is a
|
|
685
|
+
derived work is still an open question.** That question blocks step 4 rather than this table.
|
|
685
686
|
|
|
686
|
-
### What
|
|
687
|
+
### What each source is for, and what each cannot do
|
|
687
688
|
|
|
688
|
-
Overture and OSM fail differently,
|
|
689
|
+
Overture and OSM fail differently, so the build reads both.
|
|
689
690
|
|
|
690
691
|
- **Overture (`poi.db`)** is curated venue-interior naming. `concourse` appears 35 times in its
|
|
691
692
|
`airport_terminal` category against 4 in the whole Great Britain OSM extract, and 3 of those 4 are a
|
|
692
693
|
street called CONCOURSE WAY. **But poi.db is four countries** — US 11,521,612 / CA 794,418 /
|
|
693
|
-
FR 721,352 / MX 644,316, and
|
|
694
|
-
and es-MX and
|
|
694
|
+
FR 721,352 / MX 644,316, and no other country (measured 2026-08-05). It can attest en-US, en-CA, fr-FR
|
|
695
|
+
and es-MX and no other locale, so a zero count in it is evidence of absence in four countries rather than in
|
|
695
696
|
the world.
|
|
696
|
-
- **OSM** reaches any region with a Geofabrik extract, and
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
The Overture category set is measured rather than guessed. A
|
|
702
|
-
category,
|
|
703
|
-
predicts. `gas_station` leads the entire table with 12,996 hits, every one of them `station` inside
|
|
697
|
+
- **OSM** reaches any region with a Geofabrik extract, and it has the `name:<lang>` family. Every
|
|
698
|
+
non-Latin surface in this table comes from that family. OSM has no curated notion of "interior":
|
|
699
|
+
3,204 of Great Britain's 3,273 `hall` hits sit on a `public_transport=platform`, because a British
|
|
700
|
+
bus stop takes its name from the village hall it stands outside.
|
|
701
|
+
|
|
702
|
+
The Overture category set is measured rather than guessed. A scan of all 13,681,698 rows counted, per
|
|
703
|
+
category, the rows with a name that contain a designator token. The ranking does not follow what a
|
|
704
|
+
category name predicts. `gas_station` leads the entire table with 12,996 hits, every one of them `station` inside
|
|
704
705
|
"Holiday Station" or "Chevron Station Seward", and `shoe_store` contributes 708 hits of `wing` because
|
|
705
706
|
Red Wing sells boots. Four categories survived reading the distribution — `airport_terminal`,
|
|
706
707
|
`campus_building`, `pier`, `airport_lounge` — and `overture-subvenue.ts` lists the rejects with the
|
|
@@ -708,20 +709,20 @@ number that rejected them.
|
|
|
708
709
|
|
|
709
710
|
### The wave-1 defect this build fixes
|
|
710
711
|
|
|
711
|
-
Wave 1 attributed every harvested phrase to `row.designatorID`, the rule that matched the
|
|
712
|
-
|
|
713
|
-
**108 of its 133 OSM-derived surfaces
|
|
712
|
+
Wave 1 attributed every harvested phrase to `row.designatorID`, the rule that matched the feature.
|
|
713
|
+
A bus stop tagged `public_transport=platform` has a name such as "Village Hall" or "West Kensington".
|
|
714
|
+
**108 of its 133 OSM-derived surfaces therefore identified a different record than the one they pointed at**: the
|
|
714
715
|
shipped artifact claimed `west → platform`, `hall → platform`, `biggin → platform`, `salon →
|
|
715
716
|
platform`. Attribution now runs through a phrase → record index, and the row's own designator is kept
|
|
716
|
-
as `context
|
|
717
|
+
as `context`. A confound board needs that axis, since a `hall` on a platform is a bus stop
|
|
717
718
|
and a `hall` on a terminal is a hall.
|
|
718
719
|
|
|
719
720
|
### Head nouns: what the curation pass needed before it could start
|
|
720
721
|
|
|
721
|
-
Wikidata gives the
|
|
722
|
-
Spanish label is `terminal aeroportuaria
|
|
723
|
-
shipped 1,014 uncurated surfaces and could promote none of them
|
|
724
|
-
|
|
722
|
+
Wikidata gives the encyclopedic name of a concept rather than the designator as addressed. Q849706's
|
|
723
|
+
Spanish label is `terminal aeroportuaria`. The form on an envelope is `Terminal`. For that reason wave 1
|
|
724
|
+
shipped 1,014 uncurated surfaces and could promote none of them, because the table held no promotable
|
|
725
|
+
surface. Two derivations run before the harvest:
|
|
725
726
|
|
|
726
727
|
- **Latin script — the cognate test.** A token whose ASCII fold shares five leading characters with
|
|
727
728
|
the designator's own id. `terminal aeroportuaria` → `terminal`, `letištní terminál` → `terminál`,
|
|
@@ -729,99 +730,98 @@ table. Two derivations run before the harvest:
|
|
|
729
730
|
the record and, because Dutch `universiteit` is one, derived `universitario`, `universitaire` and
|
|
730
731
|
twenty more as head nouns of `campus`. Those are the modifier half, and they would have taught the
|
|
731
732
|
harvest to read "Ciudad Universitaria" as sub-venue structure.
|
|
732
|
-
- **Han / Kana / Hangul — the shared-substring test.** Every substring of length ≥ 2
|
|
733
|
+
- **Han / Kana / Hangul — the shared-substring test.** Every substring of length ≥ 2 that occurs in two or
|
|
733
734
|
more surfaces of the same record and language, maximal-only, capped at six. `ターミナル` is in none
|
|
734
|
-
of the five Japanese labels on its own
|
|
735
|
-
addresses
|
|
736
|
-
|
|
735
|
+
of the five Japanese labels on its own, because every label is a compound. It is the form Japanese
|
|
736
|
+
addresses use. No other step in the pipeline can produce it, and the Japan harvest attests it
|
|
737
|
+
1,215 times.
|
|
737
738
|
|
|
738
739
|
Run over every non-Latin phrase instead, the second derivation produced 90 fragments of Cyrillic,
|
|
739
740
|
Greek, Arabic, Thai, Burmese and Tamil words — `сгра`, `κτίρ`, `ิ่งก่อสร้า` — because those languages
|
|
740
741
|
have one surface per concept and the only shared substrings are pieces of one word. None could ever
|
|
741
|
-
be counted
|
|
742
|
+
be counted, because no source in reach attests a Thai or Burmese surface. The derivation is scoped to the four
|
|
742
743
|
scripts where it works.
|
|
743
744
|
|
|
744
745
|
The harvest also gained a script-aware match. `第1ターミナル` has no word boundaries, so the
|
|
745
|
-
token-boundary rule that
|
|
746
|
-
Han and Kana the match is a substring test, and the Germanic-compound objection does not transfer.
|
|
746
|
+
token-boundary rule that keeps Latin script from matching `Briggate` finds no match at all in Japanese.
|
|
747
|
+
For Han and Kana the match is a substring test, and the Germanic-compound objection does not transfer.
|
|
747
748
|
|
|
748
749
|
### The curation ledger
|
|
749
750
|
|
|
750
|
-
`curated: false` is still the default and
|
|
751
|
-
|
|
752
|
-
|
|
751
|
+
`curated: false` is still the default, and no surface is promoted automatically. A surface becomes
|
|
752
|
+
curated only when it matches a decision in `sub-venue-promotions.ts`. Each decision identifies a
|
|
753
|
+
designator, a phrase and a locale. The locale is part of the decision because the same token is a designator in one language and a disaster in another.
|
|
753
754
|
|
|
754
755
|
Every decision below is backed by a census in that locale's own data. `real` counts occurrences in
|
|
755
|
-
|
|
756
|
-
number is not a board, and "3,273 hits"
|
|
757
|
-
|
|
758
|
-
| designator | phrase | locale | real | confound | what the confound is
|
|
759
|
-
| ---------- | ------------ | ------ | ----: | -------: |
|
|
760
|
-
| gate
|
|
761
|
-
| gate
|
|
762
|
-
| hall | `hall` | en-GB | 0 | 3,273 | 3,204 bus stops
|
|
763
|
-
| hall | `hall` | en-US | 2,095 | 27,081 | City Hall, Kingdom Hall, event halls, dormitory halls
|
|
764
|
-
| hall | `hall` | fr-FR | 35 | 5 | three English `Town Hall` strings on `name:en` tags
|
|
765
|
-
| hall | `halle` | de-DE | 32 | 168 | the CITY Halle (Saale) / Halle (Westf), plus village halls
|
|
766
|
-
| pier | `pier` | en-GB | 120 | 44 | Pier Road / Street / Avenue / Terrace — street names
|
|
767
|
-
| pier | `pier` | en-US | 278 | 2,330 | Pier 1 Imports and franchises, seafood restaurants
|
|
768
|
-
| terminal | `terminal` | ca-ES | 15 | 0 |
|
|
769
|
-
| terminal | `terminal` | es-ES | 190 | 0 |
|
|
770
|
-
| terminal | `terminal` | fr-FR | 169 | 0 |
|
|
771
|
-
| terminal | `ターミナル` | ja-JP | 1,213 | 2 | two `ターミナル前` bus stops, arguably real
|
|
772
|
-
| wing | `wing` | en-GB | 23 | 6 | the Buckinghamshire village of Wing — Wing Close/Road
|
|
773
|
-
| wing | `wing` | en-US | 4 | 3,354 | Red Wing boots (676), chicken-wing restaurants (759)
|
|
774
|
-
| wing | `wing` | fr-FR | 0 | 26 | Wing Chun and Wing Tsun martial-arts clubs
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
0-of-3,273 in Great Britain and 35-of-40 in France
|
|
778
|
-
4-of-3,358 in the United States
|
|
779
|
-
|
|
780
|
-
**The test that separates a promote from a reject is whether
|
|
756
|
+
venue-interior naming. `confound` counts the rest, and the note states what the rest is. A bare
|
|
757
|
+
number is not a board, and "3,273 hits" did not tell a reader that 3,204 of them were bus stops.
|
|
758
|
+
|
|
759
|
+
| designator | phrase | locale | real | confound | what the confound is |
|
|
760
|
+
| ---------- | ------------ | ------ | ----: | -------: | -------------------------------------------------------------------- |
|
|
761
|
+
| `gate` | `flugsteig` | de-DE | 19 | 0 | none. An aviation-only term with no collision in German |
|
|
762
|
+
| `gate` | `porte` | fr-FR | 19 | 927 | the Portes of Paris and their Métro stations |
|
|
763
|
+
| hall | `hall` | en-GB | 0 | 3,273 | 3,204 bus stops that share a village hall's name, and Hall Lane/Road |
|
|
764
|
+
| hall | `hall` | en-US | 2,095 | 27,081 | City Hall, Kingdom Hall, event halls, dormitory halls |
|
|
765
|
+
| hall | `hall` | fr-FR | 35 | 5 | three English `Town Hall` strings on `name:en` tags |
|
|
766
|
+
| hall | `halle` | de-DE | 32 | 168 | the CITY Halle (Saale) / Halle (Westf), plus village halls |
|
|
767
|
+
| pier | `pier` | en-GB | 120 | 44 | Pier Road / Street / Avenue / Terrace — street names |
|
|
768
|
+
| pier | `pier` | en-US | 278 | 2,330 | Pier 1 Imports and franchises, seafood restaurants |
|
|
769
|
+
| terminal | `terminal` | ca-ES | 15 | 0 | none |
|
|
770
|
+
| terminal | `terminal` | es-ES | 190 | 0 | none |
|
|
771
|
+
| terminal | `terminal` | fr-FR | 169 | 0 | none |
|
|
772
|
+
| terminal | `ターミナル` | ja-JP | 1,213 | 2 | two `ターミナル前` bus stops, arguably real |
|
|
773
|
+
| wing | `wing` | en-GB | 23 | 6 | the Buckinghamshire village of Wing — Wing Close/Road |
|
|
774
|
+
| wing | `wing` | en-US | 4 | 3,354 | Red Wing boots (676), chicken-wing restaurants (759) |
|
|
775
|
+
| wing | `wing` | fr-FR | 0 | 26 | Wing Chun and Wing Tsun martial-arts clubs |
|
|
776
|
+
|
|
777
|
+
The ledger holds nine promotions and six rejections. Three pairs show why the decision is per-locale.
|
|
778
|
+
`hall` is 0-of-3,273 in Great Britain and 35-of-40 in France. `wing` is 23-of-29 in Great Britain and
|
|
779
|
+
4-of-3,358 in the United States. `pier` is promotable in Great Britain and not in the United States.
|
|
780
|
+
|
|
781
|
+
**The test that separates a promote from a reject is whether shape can isolate the confound**, and it
|
|
781
782
|
is checked by enumeration rather than asserted. `halle` keeps a 168-hit confound and is promoted
|
|
782
783
|
because dumping all 32 `<halle> <identifier>` hits returns numbered factory, trade-fair and airport
|
|
783
784
|
halls — VW Halle 42, Audi GVZ Halle G, Messe West Halle 8 — and not one instance of the city of
|
|
784
785
|
240,000 people with the same name. `porte` has a smaller confound ratio in that bucket and is
|
|
785
786
|
rejected, because 17 of its 36 shape hits are Porte Saint-Martin, Porte Saint-Denis and Porte
|
|
786
|
-
Notre-Dame.
|
|
787
|
+
Notre-Dame. The same test rejected `pier` for en-US, because Pier 1 Imports has the designator+identifier shape.
|
|
787
788
|
|
|
788
789
|
**A caveat on `wing`, the designator the corpus task's board rests on.** 24 of its 29 Great Britain
|
|
789
|
-
hits sit on a `public_transport=platform`, because British bus stops
|
|
790
|
+
hits sit on a `public_transport=platform`, because British bus stops take their names from the hospital wing
|
|
790
791
|
they serve. The extractor maps no building wings at all: `aile` in France is **0 hits**, `ala` in
|
|
791
792
|
Spain **0**, `flügel` in Germany **0**. The en-GB evidence is a property of British stop naming rather
|
|
792
793
|
than of a source that has wings in it, and the localized wing surfaces the corpus task asked for are
|
|
793
794
|
out of reach until the extractor gains an `indoor=*` rule or the lexicon gains a hand-seeded surface
|
|
794
795
|
list.
|
|
795
796
|
|
|
796
|
-
**A rejection of a
|
|
797
|
-
English vocabulary with no locale
|
|
797
|
+
**A rejection of a shipped designator is advisory.** `neural/venue-structure.ts` holds a flat
|
|
798
|
+
English vocabulary with no per-locale filter, and `wing`, `terminal` and `concourse` are in it. This table
|
|
798
799
|
cannot un-ship them: the `wing` / en-US rejection tells a recipe author which locale to leave out of a
|
|
799
|
-
generated line, and it does
|
|
800
|
-
vocabulary
|
|
801
|
-
want that does not exist yet.
|
|
800
|
+
generated line, and it does not stop the span proposer from firing on "Red Wing". A per-locale filter on the shipped
|
|
801
|
+
vocabulary belongs to step 4. It is the largest missing piece the recipe will need.
|
|
802
802
|
|
|
803
|
-
The
|
|
803
|
+
The ledger does apply to every surface the lexicon adds. `pier` is promoted for en-GB and rejected for
|
|
804
804
|
en-US, and because it is not in the shipped list, the rejection has teeth: the region-free English
|
|
805
805
|
`pier` surface stays `curated: false` and only the 164 Great Britain attestations are marked usable.
|
|
806
806
|
|
|
807
807
|
### Identifier shapes are per-region now
|
|
808
808
|
|
|
809
809
|
`Gate A12` is a rendering rather than a string anyone wrote down: all 658 Great Britain `aeroway=gate`
|
|
810
|
-
features but 13 are unnamed and
|
|
811
|
-
than a phrase list
|
|
812
|
-
|
|
813
|
-
|
|
814
|
-
| region |
|
|
815
|
-
| ------ |
|
|
816
|
-
| GB |
|
|
817
|
-
| JP |
|
|
818
|
-
| FR |
|
|
819
|
-
| DE |
|
|
820
|
-
| ES |
|
|
821
|
-
|
|
822
|
-
Britain and Japan
|
|
823
|
-
is the outlier
|
|
824
|
-
|
|
810
|
+
features but 13 are unnamed and have only a `ref`. The table therefore stores a distribution rather
|
|
811
|
+
than a phrase list. The distribution differs by country far more than the shared vocabulary suggests,
|
|
812
|
+
so a recipe that generates `Gate <ref>` for a French address has to sample France's:
|
|
813
|
+
|
|
814
|
+
| region | `ref`s | most common shape | second | third |
|
|
815
|
+
| ------ | -----: | ---------------------- | ---------------: | --------------- |
|
|
816
|
+
| GB | 655 | digit 463 (71%) | letter-digit 19% | digit-letter 8% |
|
|
817
|
+
| JP | 450 | digit 402 (89%) | digit-letter 5% | other 3% |
|
|
818
|
+
| FR | 628 | letter-digit 385 (61%) | digit 29% | letter 5% |
|
|
819
|
+
| DE | 642 | letter-digit 387 (60%) | digit 18% | range 11% |
|
|
820
|
+
| ES | 493 | letter-digit 189 (38%) | range 35% | digit 15% |
|
|
821
|
+
|
|
822
|
+
Britain and Japan use bare numbers for `aeroway=gate` refs. France and Germany use a letter and then
|
|
823
|
+
a number (`A37`, `B05`). Spain is the outlier: a third of its refs are a range (`B18-B20`, `D42-D43`).
|
|
824
|
+
No other country uses ranges at that rate. A generator that samples Great Britain's 71%-bare-digit shape into a Spanish line
|
|
825
825
|
produces a plausible string that is wrong about Spain.
|
|
826
826
|
|
|
827
827
|
### The seed duplicates `neural/venue-structure.ts`
|
|
@@ -829,5 +829,5 @@ produces a plausible string that is wrong about Spain.
|
|
|
829
829
|
`@mailwoman/corpus` does not depend on `@mailwoman/neural`, so the shipped designator and modifier
|
|
830
830
|
lists are re-declared in `sub-venue-lexicon.ts`. That is a drift surface, and
|
|
831
831
|
`sub-venue-lexicon.test.ts` pins both lists literally so a change in either place fails a test rather
|
|
832
|
-
than passing silently. The
|
|
832
|
+
than passing silently. The planned change is still the reverse direction: have `neural/venue-structure.ts`
|
|
833
833
|
read a committed lexicon extract and delete both copies.
|