@mailwoman/corpus 4.10.0 → 4.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +79 -0
- package/out/src/adapters/ban/adapter.d.ts +6 -2
- package/out/src/adapters/ban/adapter.d.ts.map +1 -1
- package/out/src/adapters/ban/adapter.js +8 -4
- package/out/src/adapters/ban/adapter.js.map +1 -1
- package/out/src/adapters/index.d.ts.map +1 -1
- package/out/src/adapters/index.js +2 -0
- package/out/src/adapters/index.js.map +1 -1
- package/out/src/adapters/openaddresses/adapter.d.ts +5 -2
- package/out/src/adapters/openaddresses/adapter.d.ts.map +1 -1
- package/out/src/adapters/openaddresses/adapter.js +2 -7
- package/out/src/adapters/openaddresses/adapter.js.map +1 -1
- package/out/src/adapters/overture/adapter.d.ts +37 -0
- package/out/src/adapters/overture/adapter.d.ts.map +1 -0
- package/out/src/adapters/overture/adapter.js +119 -0
- package/out/src/adapters/overture/adapter.js.map +1 -0
- package/out/src/adapters/synth-po-box/adapter.d.ts +8 -0
- package/out/src/adapters/synth-po-box/adapter.d.ts.map +1 -1
- package/out/src/adapters/synth-po-box/adapter.js +37 -2
- package/out/src/adapters/synth-po-box/adapter.js.map +1 -1
- package/out/src/adapters/tiger/adapter.js +1 -1
- package/out/src/adapters/tiger/adapter.js.map +1 -1
- package/out/src/adapters/usgov-hrsa-fqhc/adapter.d.ts +1 -1
- package/out/src/adapters/usgov-hrsa-fqhc/adapter.js +1 -1
- package/out/src/align.js +1 -1
- package/out/src/align.js.map +1 -1
- package/out/src/build.d.ts +14 -0
- package/out/src/build.d.ts.map +1 -1
- package/out/src/build.js +25 -0
- package/out/src/build.js.map +1 -1
- package/out/src/license.d.ts +37 -0
- package/out/src/license.d.ts.map +1 -0
- package/out/src/license.js +46 -0
- package/out/src/license.js.map +1 -0
- package/out/src/synthesize-boundary-stress.d.ts +77 -0
- package/out/src/synthesize-boundary-stress.d.ts.map +1 -0
- package/out/src/synthesize-boundary-stress.js +449 -0
- package/out/src/synthesize-boundary-stress.js.map +1 -0
- package/out/src/synthesize-no-street.d.ts +1 -1
- package/out/src/synthesize-no-street.js +2 -2
- package/out/src/synthesize-no-street.js.map +1 -1
- package/out/src/synthesize-po-box.d.ts +10 -1
- package/out/src/synthesize-po-box.d.ts.map +1 -1
- package/out/src/synthesize-po-box.js +63 -3
- package/out/src/synthesize-po-box.js.map +1 -1
- package/out/src/synthesize.d.ts +15 -4
- package/out/src/synthesize.d.ts.map +1 -1
- package/out/src/synthesize.js +124 -4
- package/out/src/synthesize.js.map +1 -1
- package/out/src/wof-json.d.ts +1 -1
- package/out/src/wof-json.js +1 -1
- package/package.json +4 -4
package/README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# @mailwoman/corpus
|
|
2
|
+
|
|
3
|
+
**BIO-labeled training-corpus pipeline** for the Mailwoman address parser.
|
|
4
|
+
|
|
5
|
+
Generates sequence-labeling training data from reference sources
|
|
6
|
+
(OpenAddresses, libpostal dictionaries, synthetic shards) and assembles them into
|
|
7
|
+
the TSV format consumed by the Modal training pipeline. This package produces
|
|
8
|
+
the data that trains `@mailwoman/neural-weights-*`.
|
|
9
|
+
|
|
10
|
+
```ts
|
|
11
|
+
// The corpus pipeline is primarily build-time CLI tooling.
|
|
12
|
+
// Key entry points:
|
|
13
|
+
import { expandGolden } from "@mailwoman/corpus" // Expand reference addresses
|
|
14
|
+
import { synthesizeShard } from "@mailwoman/corpus" // Generate synthetic training shards
|
|
15
|
+
import { alignRow } from "@mailwoman/corpus" // Align raw address → BIO tokens
|
|
16
|
+
import { validateCorpus } from "@mailwoman/corpus" // Validate corpus integrity
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## What it produces
|
|
20
|
+
|
|
21
|
+
The corpus pipeline assembles training data from multiple sources:
|
|
22
|
+
|
|
23
|
+
| Source | Description |
|
|
24
|
+
| -------------------- | ------------------------------------------------------------------------ |
|
|
25
|
+
| **OpenAddresses** | Real government address point data (US, FR, DE, …) |
|
|
26
|
+
| **NAD** | National Address Database (US-specific) |
|
|
27
|
+
| **libpostal** | Multilingual street/place name dictionaries |
|
|
28
|
+
| **Synthetic shards** | Generated address variations (boundary stress, order variants, all-caps) |
|
|
29
|
+
| **Overture Maps** | Address theme ingestion (alpha) |
|
|
30
|
+
|
|
31
|
+
Output format: TSV rows with `raw<TAB>BIO_labels` consumed by the Python
|
|
32
|
+
training pipeline (`corpus-python/`).
|
|
33
|
+
|
|
34
|
+
## Key modules
|
|
35
|
+
|
|
36
|
+
| Module | Purpose |
|
|
37
|
+
| ----------------------- | ------------------------------------------------------------------ |
|
|
38
|
+
| **`expand-golden.ts`** | Expand reference addresses into training rows with alignment |
|
|
39
|
+
| **`align.ts`** | Tokenize raw address → BIO label sequence |
|
|
40
|
+
| **`validate.ts`** | Validate corpus integrity, label coverage, shard balance |
|
|
41
|
+
| **`synthesize-*.ts`** | Synthetic shard generators (boundary stress, order variants, etc.) |
|
|
42
|
+
| **`ingest/`** | Overture Maps + NAD ingestion |
|
|
43
|
+
| **`shard-registry.ts`** | Shard metadata and composition |
|
|
44
|
+
| **`stats.ts`** | Per-shard and per-tag statistics |
|
|
45
|
+
|
|
46
|
+
## Build-time tooling
|
|
47
|
+
|
|
48
|
+
The corpus is assembled via scripts in `scripts/`:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
# Validate the corpus
|
|
52
|
+
node scripts/validate-corpus.mjs
|
|
53
|
+
|
|
54
|
+
# Rebuild shards
|
|
55
|
+
node scripts/build-boundary-stress-shard.mjs
|
|
56
|
+
|
|
57
|
+
# Corpus statistics
|
|
58
|
+
node scripts/corpus-stats.mjs
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Design
|
|
62
|
+
|
|
63
|
+
- **BIO (Begin/Inside/Outside) labeling** over SentencePiece tokens.
|
|
64
|
+
- **Character-offset aligned** — labels track the raw string, not the
|
|
65
|
+
normalized form, so the model learns real input distributions.
|
|
66
|
+
- **Source-homogeneous shards** — each shard comes from one source, ordered by
|
|
67
|
+
type, so eval splits are honest (no bleed between train and held-out).
|
|
68
|
+
|
|
69
|
+
## Related
|
|
70
|
+
|
|
71
|
+
- [`@mailwoman/neural`](../neural) — the runtime that loads and runs the trained model
|
|
72
|
+
- [`@mailwoman/neural-weights-en-us`](../neural-weights-en-us) — the trained model itself
|
|
73
|
+
- [Corpus Construction concepts](https://mailwoman.sister.software/articles/concepts/corpus-construction/)
|
|
74
|
+
- [Training Pipeline concepts](https://mailwoman.sister.software/articles/concepts/training-pipeline/)
|
|
75
|
+
- [CONTRIBUTING_MODEL_WORK](https://mailwoman.sister.software/articles/plan/CONTRIBUTING_MODEL_WORK/)
|
|
76
|
+
|
|
77
|
+
## License
|
|
78
|
+
|
|
79
|
+
[AGPL-3.0-only](https://www.gnu.org/licenses/agpl-3.0.html)
|
|
@@ -18,8 +18,12 @@
|
|
|
18
18
|
* left for the wof-postalcode + wof-admin cross-reference at corpus build time (a future pass;
|
|
19
19
|
* for Phase 1 the row's region is simply absent).
|
|
20
20
|
*
|
|
21
|
-
* License:
|
|
22
|
-
*
|
|
21
|
+
* License: the official BAN (adresse.data.gouv.fr) is DUAL-licensed — Licence Ouverte 2.0 (Etalab,
|
|
22
|
+
* attribution-only) OR ODbL (share-alike). We ELECT Licence Ouverte 2.0 (issue #26 Tier B:
|
|
23
|
+
* allowed for training with attribution; the ODbL option's share-alike obligation would defeat
|
|
24
|
+
* the proprietary-weights goal). Stamped onto every row as `Licence Ouverte 2.0` — NOT the older
|
|
25
|
+
* conservative `ODbL-1.0` label, which wrongly read as Tier-C-denied in the corpus license audit.
|
|
26
|
+
* The model card MUST carry the BAN attribution (Tier B obligation).
|
|
23
27
|
*
|
|
24
28
|
* The adapter is streaming-aware: it uses `csv-parse` in streaming mode so a 25M-row dump never
|
|
25
29
|
* sits in memory. Honors `opts.limit` for fixture / smoke runs, `opts.signal` for cancellation,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/ban/adapter.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/ban/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AAMH,OAAO,KAAK,EAAgC,aAAa,EAAE,MAAM,gBAAgB,CAAA;AAGjF,eAAO,MAAM,cAAc,QAAQ,CAAA;AA4CnC,wBAAgB,gBAAgB,IAAI,aAAa,CAwEhD;AAED,eAAO,MAAM,UAAU,eAAqB,CAAA"}
|
|
@@ -18,8 +18,12 @@
|
|
|
18
18
|
* left for the wof-postalcode + wof-admin cross-reference at corpus build time (a future pass;
|
|
19
19
|
* for Phase 1 the row's region is simply absent).
|
|
20
20
|
*
|
|
21
|
-
* License:
|
|
22
|
-
*
|
|
21
|
+
* License: the official BAN (adresse.data.gouv.fr) is DUAL-licensed — Licence Ouverte 2.0 (Etalab,
|
|
22
|
+
* attribution-only) OR ODbL (share-alike). We ELECT Licence Ouverte 2.0 (issue #26 Tier B:
|
|
23
|
+
* allowed for training with attribution; the ODbL option's share-alike obligation would defeat
|
|
24
|
+
* the proprietary-weights goal). Stamped onto every row as `Licence Ouverte 2.0` — NOT the older
|
|
25
|
+
* conservative `ODbL-1.0` label, which wrongly read as Tier-C-denied in the corpus license audit.
|
|
26
|
+
* The model card MUST carry the BAN attribution (Tier B obligation).
|
|
23
27
|
*
|
|
24
28
|
* The adapter is streaming-aware: it uses `csv-parse` in streaming mode so a 25M-row dump never
|
|
25
29
|
* sits in memory. Honors `opts.limit` for fixture / smoke runs, `opts.signal` for cancellation,
|
|
@@ -64,7 +68,7 @@ function composeRaw(house, street, postcode, locality) {
|
|
|
64
68
|
export function createBanAdapter() {
|
|
65
69
|
return {
|
|
66
70
|
id: BAN_ADAPTER_ID,
|
|
67
|
-
defaultLicense: "
|
|
71
|
+
defaultLicense: "Licence Ouverte 2.0",
|
|
68
72
|
description: "Base Adresse Nationale (FR): house-number-level street addresses (~25M rows).",
|
|
69
73
|
async *rows(opts) {
|
|
70
74
|
if (opts.country && opts.country !== "FR") {
|
|
@@ -122,7 +126,7 @@ export function createBanAdapter() {
|
|
|
122
126
|
source: BAN_ADAPTER_ID,
|
|
123
127
|
source_id: sourceId,
|
|
124
128
|
corpus_version: "",
|
|
125
|
-
license: "
|
|
129
|
+
license: "Licence Ouverte 2.0",
|
|
126
130
|
};
|
|
127
131
|
emitted++;
|
|
128
132
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/ban/adapter.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/ban/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AAEH,OAAO,EAAE,KAAK,IAAI,QAAQ,EAAE,MAAM,WAAW,CAAA;AAC7C,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AAErD,OAAO,EAAE,iBAAiB,EAAE,MAAM,uBAAuB,CAAA;AAEzD,MAAM,CAAC,MAAM,cAAc,GAAG,KAAK,CAAA;AAenC;;;GAGG;AACH,SAAS,kBAAkB,CAAC,MAAc,EAAE,GAAW;IACtD,MAAM,CAAC,GAAG,MAAM,CAAC,IAAI,EAAE,CAAA;IACvB,MAAM,CAAC,GAAG,GAAG,CAAC,IAAI,EAAE,CAAA;IACpB,IAAI,CAAC,CAAC;QAAE,OAAO,EAAE,CAAA;IACjB,OAAO,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAA;AAC3B,CAAC;AAED;;;;;;;;GAQG;AACH,SAAS,UAAU,CAAC,KAAa,EAAE,MAAc,EAAE,QAAgB,EAAE,QAAgB;IACpF,MAAM,KAAK,GAAa,EAAE,CAAA;IAC1B,MAAM,UAAU,GAAG,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAA;IACnE,IAAI,UAAU;QAAE,KAAK,CAAC,IAAI,CAAC,UAAU,CAAC,CAAA;IACtC,MAAM,QAAQ,GAAG,CAAC,QAAQ,EAAE,QAAQ,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAA;IACtE,IAAI,QAAQ;QAAE,KAAK,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAA;IAClC,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAA;AACpD,CAAC;AAED,MAAM,UAAU,gBAAgB;IAC/B,OAAO;QACN,EAAE,EAAE,cAAc;QAClB,cAAc,EAAE,qBAAqB;QACrC,WAAW,EAAE,+EAA+E;QAE5F,KAAK,CAAC,CAAC,IAAI,CAAC,IAAoB;YAC/B,IAAI,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI,EAAE,CAAC;gBAC3C,MAAM,IAAI,KAAK,CAAC,+CAA+C,IAAI,CAAC,OAAO,EAAE,CAAC,CAAA;YAC/E,CAAC;YAED,MAAM,MAAM,GAAG,gBAAgB,CAAC,IAAI,CAAC,SAAS,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,CAAC,CAAA;YACrE,MAAM,MAAM,GAAG,MAAM,CAAC,IAAI,CACzB,QAAQ,CAAC;gBACR,SAAS,EAAE,GAAG;gBACd,OAAO,EAAE,IAAI;gBACb,gBAAgB,EAAE,IAAI;gBACtB,YAAY,EAAE,IAAI;gBAClB,kBAAkB,EAAE,IAAI;aACxB,CAAC,CACF,CAAA;YAED,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,CAAC;gBACJ,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI,MAA+B,EAAE,CAAC;oBAC5D,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;wBAAE,MAAK;oBAC/B,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,IAAI,CAAC,KAAK;wBAAE,MAAK;oBAE5D,MAAM,KAAK,GAAG,kBAAkB,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,EAAE,MAAM,CAAC,GAAG,IAAI,EAAE,CAAC,CAAA;oBACvE,MAAM,MAAM,GAAG,CAAC,MAAM,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;oBAC7C,MAAM,QAAQ,GAAG,CAAC,MAAM,CAAC,WAAW,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;oBAClD,MAAM,QAAQ,GAAG,CAAC,MAAM,CAAC,WAAW,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;oBAElD,IAAI,CAAC,MAAM,IAAI,CAAC,QAAQ;wBAAE,SAAQ;oBAClC,IAAI,CAAC,KAAK,IAAI,CAAC,QAAQ;wBAAE,SAAQ;oBAEjC,MAAM,UAAU,GAAG,iBAAiB,CAAC,MAAM,CAAC,CAAA;oBAE5C,MAAM,UAAU,GAA+B,EAAE,CAAA;oBACjD,IAAI,KAAK;wBAAE,UAAU,CAAC,YAAY,GAAG,KAAK,CAAA;oBAC1C,IAAI,UAAU,CAAC,MAAM;wBAAE,UAAU,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAA;oBACnE,IAAI,UAAU,CAAC,MAAM;wBAAE,UAAU,CAAC,MAAM,GAAG,UAAU,CAAC,MAAM,CAAA;oBAC5D,IAAI,QAAQ;wBAAE,UAAU,CAAC,QAAQ,GAAG,QAAQ,CAAA;oBAC5C,IAAI,QAAQ;wBAAE,UAAU,CAAC,QAAQ,GAAG,QAAQ,CAAA;oBAE5C,MAAM,GAAG,GAAG,UAAU,CAAC,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAA;oBACzD,IAAI,CAAC,GAAG;wBAAE,SAAQ;oBAElB,MAAM,OAAO,GAAG,mBAAmB,CAAC,UAAU,EAAE,GAAG,CAAC,CAAA;oBACpD,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,KAAK,CAAC;wBAAE,SAAQ;oBAE/C,MAAM,QAAQ,GAAG,MAAM,CAAC,EAAE,EAAE,IAAI,EAAE;wBACjC,CAAC,CAAC,GAAG,cAAc,IAAI,MAAM,CAAC,EAAE,CAAC,IAAI,EAAE,EAAE;wBACzC,CAAC,CAAC,cAAc,CAAC,cAAc,EAAE,OAAO,CAAC,CAAA;oBAE1C,MAAM;wBACL,GAAG;wBACH,UAAU,EAAE,OAAO;wBACnB,OAAO,EAAE,IAAI;wBACb,MAAM,EAAE,OAAO;wBACf,MAAM,EAAE,cAAc;wBACtB,SAAS,EAAE,QAAQ;wBACnB,cAAc,EAAE,EAAE;wBAClB,OAAO,EAAE,qBAAqB;qBAC9B,CAAA;oBACD,OAAO,EAAE,CAAA;gBACV,CAAC;YACF,CAAC;oBAAS,CAAC;gBACV,MAAM,CAAC,OAAO,EAAE,CAAA;YACjB,CAAC;QACF,CAAC;KACD,CAAA;AACF,CAAC;AAED,MAAM,CAAC,MAAM,UAAU,GAAG,gBAAgB,EAAE,CAAA"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/adapters/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAGH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAA;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/adapters/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAGH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAA;AAoBhD;;;;;;;;;;GAUG;AACH,eAAO,MAAM,gBAAgB,EAAE,SAAS,aAAa,EAmBpD,CAAA;AAQD,OAAO,EAAE,cAAc,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAA;AACjG,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,qBAAqB,GACrB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACtG,OAAO,EACN,wBAAwB,EACxB,6BAA6B,EAC7B,oBAAoB,GACpB,MAAM,4BAA4B,CAAA;AACnC,OAAO,EACN,2BAA2B,EAC3B,gCAAgC,EAChC,qBAAqB,GACrB,MAAM,+BAA+B,CAAA;AACtC,OAAO,EACN,+BAA+B,EAC/B,oCAAoC,EACpC,yBAAyB,GACzB,MAAM,mCAAmC,CAAA;AAC1C,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AAC1F,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,oBAAoB,GACpB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EACN,yBAAyB,EACzB,8BAA8B,EAC9B,mBAAmB,GACnB,MAAM,6BAA6B,CAAA;AACpC,OAAO,EAAE,wBAAwB,EAAE,6BAA6B,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA;AACxH,OAAO,EAAE,oBAAoB,EAAE,yBAAyB,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AACzG,OAAO,EAAE,sBAAsB,EAAE,2BAA2B,EAAE,iBAAiB,EAAE,MAAM,0BAA0B,CAAA;AACjH,OAAO,EACN,uBAAuB,EACvB,4BAA4B,EAC5B,kCAAkC,GAClC,MAAM,6CAA6C,CAAA;AACpD,OAAO,EAAE,oBAAoB,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAA;AACnF,OAAO,EAAE,yBAAyB,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAA"}
|
|
@@ -24,6 +24,7 @@ import { fccBdcAdapter } from "./fcc-bdc/adapter.js";
|
|
|
24
24
|
import { geonamesPostalAdapter } from "./geonames-postal/adapter.js";
|
|
25
25
|
import { geonamesAdapter } from "./geonames/adapter.js";
|
|
26
26
|
import { openaddressesAdapter } from "./openaddresses/adapter.js";
|
|
27
|
+
import { overtureAdapter } from "./overture/adapter.js";
|
|
27
28
|
import { stateHiSchoolsAdapter } from "./state-hi-schools/adapter.js";
|
|
28
29
|
import { stateIaContractorsAdapter } from "./state-ia-contractors/adapter.js";
|
|
29
30
|
import { stateNyNotariesAdapter } from "./state-ny-notaries/adapter.js";
|
|
@@ -55,6 +56,7 @@ export const BUILTIN_ADAPTERS = [
|
|
|
55
56
|
banAdapter,
|
|
56
57
|
tigerAdapter,
|
|
57
58
|
openaddressesAdapter,
|
|
59
|
+
overtureAdapter,
|
|
58
60
|
fccBdcAdapter,
|
|
59
61
|
usgovHrsaFqhcAdapter,
|
|
60
62
|
usgovNppesAdapter,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/adapters/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,sBAAsB,EAAE,MAAM,eAAe,CAAA;AAEtD,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAA;AACpD,OAAO,EAAE,qBAAqB,EAAE,MAAM,8BAA8B,CAAA;AACpE,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EAAE,oBAAoB,EAAE,MAAM,4BAA4B,CAAA;AACjE,OAAO,EAAE,qBAAqB,EAAE,MAAM,+BAA+B,CAAA;AACrE,OAAO,EAAE,yBAAyB,EAAE,MAAM,mCAAmC,CAAA;AAC7E,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAA;AACvE,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAA;AACvE,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAA;AACnE,OAAO,EAAE,mBAAmB,EAAE,MAAM,6BAA6B,CAAA;AACjE,OAAO,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA;AAC/D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AACxD,OAAO,EAAE,iBAAiB,EAAE,MAAM,0BAA0B,CAAA;AAC5D,OAAO,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAA;AAC7D,OAAO,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAA;AAEvE;;;;;;;;;;GAUG;AACH,MAAM,CAAC,MAAM,gBAAgB,GAA6B;IACzD,eAAe;IACf,oBAAoB;IACpB,eAAe;IACf,qBAAqB;IACrB,UAAU;IACV,YAAY;IACZ,oBAAoB;IACpB,aAAa;IACb,oBAAoB;IACpB,iBAAiB;IACjB,eAAe;IACf,mBAAmB;IACnB,kBAAkB;IAClB,yBAAyB;IACzB,sBAAsB;IACtB,sBAAsB;IACtB,qBAAqB;CACrB,CAAA;AAED,KAAK,MAAM,OAAO,IAAI,gBAAgB,EAAE,CAAC;IACxC,IAAI,CAAC,sBAAsB,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,EAAE,CAAC;QAC7C,sBAAsB,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAA;IACzC,CAAC;AACF,CAAC;AAED,OAAO,EAAE,cAAc,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAA;AACjG,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,qBAAqB,GACrB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACtG,OAAO,EACN,wBAAwB,EACxB,6BAA6B,EAC7B,oBAAoB,GACpB,MAAM,4BAA4B,CAAA;AACnC,OAAO,EACN,2BAA2B,EAC3B,gCAAgC,EAChC,qBAAqB,GACrB,MAAM,+BAA+B,CAAA;AACtC,OAAO,EACN,+BAA+B,EAC/B,oCAAoC,EACpC,yBAAyB,GACzB,MAAM,mCAAmC,CAAA;AAC1C,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AAC1F,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,oBAAoB,GACpB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EACN,yBAAyB,EACzB,8BAA8B,EAC9B,mBAAmB,GACnB,MAAM,6BAA6B,CAAA;AACpC,OAAO,EAAE,wBAAwB,EAAE,6BAA6B,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA;AACxH,OAAO,EAAE,oBAAoB,EAAE,yBAAyB,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AACzG,OAAO,EAAE,sBAAsB,EAAE,2BAA2B,EAAE,iBAAiB,EAAE,MAAM,0BAA0B,CAAA;AACjH,OAAO,EACN,uBAAuB,EACvB,4BAA4B,EAC5B,kCAAkC,GAClC,MAAM,6CAA6C,CAAA;AACpD,OAAO,EAAE,oBAAoB,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAA;AACnF,OAAO,EAAE,yBAAyB,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAA"}
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/adapters/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,sBAAsB,EAAE,MAAM,eAAe,CAAA;AAEtD,OAAO,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7C,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAA;AACpD,OAAO,EAAE,qBAAqB,EAAE,MAAM,8BAA8B,CAAA;AACpE,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EAAE,oBAAoB,EAAE,MAAM,4BAA4B,CAAA;AACjE,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EAAE,qBAAqB,EAAE,MAAM,+BAA+B,CAAA;AACrE,OAAO,EAAE,yBAAyB,EAAE,MAAM,mCAAmC,CAAA;AAC7E,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAA;AACvE,OAAO,EAAE,sBAAsB,EAAE,MAAM,gCAAgC,CAAA;AACvE,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,EAAE,oBAAoB,EAAE,MAAM,8BAA8B,CAAA;AACnE,OAAO,EAAE,mBAAmB,EAAE,MAAM,6BAA6B,CAAA;AACjE,OAAO,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA;AAC/D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AACxD,OAAO,EAAE,iBAAiB,EAAE,MAAM,0BAA0B,CAAA;AAC5D,OAAO,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAA;AAC7D,OAAO,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAA;AAEvE;;;;;;;;;;GAUG;AACH,MAAM,CAAC,MAAM,gBAAgB,GAA6B;IACzD,eAAe;IACf,oBAAoB;IACpB,eAAe;IACf,qBAAqB;IACrB,UAAU;IACV,YAAY;IACZ,oBAAoB;IACpB,eAAe;IACf,aAAa;IACb,oBAAoB;IACpB,iBAAiB;IACjB,eAAe;IACf,mBAAmB;IACnB,kBAAkB;IAClB,yBAAyB;IACzB,sBAAsB;IACtB,sBAAsB;IACtB,qBAAqB;CACrB,CAAA;AAED,KAAK,MAAM,OAAO,IAAI,gBAAgB,EAAE,CAAC;IACxC,IAAI,CAAC,sBAAsB,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,EAAE,CAAC;QAC7C,sBAAsB,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAA;IACzC,CAAC;AACF,CAAC;AAED,OAAO,EAAE,cAAc,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,EAAE,kBAAkB,EAAE,uBAAuB,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAA;AACjG,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,qBAAqB,GACrB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EAAE,mBAAmB,EAAE,wBAAwB,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACtG,OAAO,EACN,wBAAwB,EACxB,6BAA6B,EAC7B,oBAAoB,GACpB,MAAM,4BAA4B,CAAA;AACnC,OAAO,EACN,2BAA2B,EAC3B,gCAAgC,EAChC,qBAAqB,GACrB,MAAM,+BAA+B,CAAA;AACtC,OAAO,EACN,+BAA+B,EAC/B,oCAAoC,EACpC,yBAAyB,GACzB,MAAM,mCAAmC,CAAA;AAC1C,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EACN,4BAA4B,EAC5B,iCAAiC,EACjC,sBAAsB,GACtB,MAAM,gCAAgC,CAAA;AACvC,OAAO,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAA;AAC1F,OAAO,EACN,0BAA0B,EAC1B,+BAA+B,EAC/B,oBAAoB,GACpB,MAAM,8BAA8B,CAAA;AACrC,OAAO,EACN,yBAAyB,EACzB,8BAA8B,EAC9B,mBAAmB,GACnB,MAAM,6BAA6B,CAAA;AACpC,OAAO,EAAE,wBAAwB,EAAE,6BAA6B,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA;AACxH,OAAO,EAAE,oBAAoB,EAAE,yBAAyB,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAA;AACzG,OAAO,EAAE,sBAAsB,EAAE,2BAA2B,EAAE,iBAAiB,EAAE,MAAM,0BAA0B,CAAA;AACjH,OAAO,EACN,uBAAuB,EACvB,4BAA4B,EAC5B,kCAAkC,GAClC,MAAM,6CAA6C,CAAA;AACpD,OAAO,EAAE,oBAAoB,EAAE,eAAe,EAAE,MAAM,6BAA6B,CAAA;AACnF,OAAO,EAAE,yBAAyB,EAAE,oBAAoB,EAAE,MAAM,kCAAkC,CAAA"}
|
|
@@ -46,8 +46,11 @@ export interface OpenaddressesAdapterOptions {
|
|
|
46
46
|
*/
|
|
47
47
|
defaultLicense?: string;
|
|
48
48
|
/**
|
|
49
|
-
*
|
|
50
|
-
*
|
|
49
|
+
* Per-adapter share-alike drop. Default **true** (include) as of 2026-06-19: exclusion is a
|
|
50
|
+
* deliberate BUILD-level act (`buildCorpus({ excludeLicenses })` / `--exclude-share-alike`), NOT
|
|
51
|
+
* a silent adapter default (#26 — "purposely exclude, don't opt in to include"). Set false only
|
|
52
|
+
* for an explicit adapter-scoped drop; the build-level `--exclude-share-alike` is the normal
|
|
53
|
+
* path.
|
|
51
54
|
*/
|
|
52
55
|
allowShareAlike?: boolean;
|
|
53
56
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/openaddresses/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;
|
|
1
|
+
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/openaddresses/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AAOH,OAAO,KAAK,EAAgC,aAAa,EAAE,MAAM,gBAAgB,CAAA;AAEjF,eAAO,MAAM,wBAAwB,kBAAkB,CAAA;AACvD,eAAO,MAAM,6BAA6B,cAAc,CAAA;AA8CxD,MAAM,WAAW,2BAA2B;IAC3C;;;;OAIG;IACH,cAAc,CAAC,EAAE,MAAM,CAAA;IAEvB;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,OAAO,CAAA;CACzB;AAED;;;GAGG;AACH,wBAAgB,0BAA0B,CAAC,IAAI,GAAE,2BAAgC,GAAG,aAAa,CAwFhG;AAED,eAAO,MAAM,oBAAoB,eAA+B,CAAA"}
|
|
@@ -39,6 +39,7 @@ import { createReadStream } from "node:fs";
|
|
|
39
39
|
import { createInterface } from "node:readline";
|
|
40
40
|
import { stableSourceId } from "../../adapter.js";
|
|
41
41
|
import { formatAddress, reconcileComponents } from "../../format.js";
|
|
42
|
+
import { SHARE_ALIKE_PATTERN } from "../../license.js";
|
|
42
43
|
export const OPENADDRESSES_ADAPTER_ID = "openaddresses";
|
|
43
44
|
export const OPENADDRESSES_DEFAULT_LICENSE = "CC-BY-4.0";
|
|
44
45
|
/** Return a lowercase-keyed view of a Feature's properties so case variants both work. */
|
|
@@ -73,19 +74,13 @@ function parseFeatureLine(line) {
|
|
|
73
74
|
return null;
|
|
74
75
|
return normalizeProperties(obj.properties);
|
|
75
76
|
}
|
|
76
|
-
/**
|
|
77
|
-
* Licenses that require share-alike or create a copyleft obligation on derived works. Rows carrying
|
|
78
|
-
* any of these licenses are dropped by default because they would contaminate proprietary-weights
|
|
79
|
-
* training. Per the licensing strategy (#26).
|
|
80
|
-
*/
|
|
81
|
-
const SHARE_ALIKE_PATTERN = /^ODbL|^Open Database License|^CC-BY-SA|^CC-SA/i;
|
|
82
77
|
/**
|
|
83
78
|
* Build an OpenAddresses adapter. The optional `defaultLicense` lets callers stamp a non-default
|
|
84
79
|
* fallback for dumps known to carry a single license throughout (e.g. a PDDL-only state slice).
|
|
85
80
|
*/
|
|
86
81
|
export function createOpenaddressesAdapter(opts = {}) {
|
|
87
82
|
const defaultLicense = opts.defaultLicense ?? OPENADDRESSES_DEFAULT_LICENSE;
|
|
88
|
-
const allowShareAlike = opts.allowShareAlike ??
|
|
83
|
+
const allowShareAlike = opts.allowShareAlike ?? true;
|
|
89
84
|
return {
|
|
90
85
|
id: OPENADDRESSES_ADAPTER_ID,
|
|
91
86
|
defaultLicense,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/openaddresses/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AAEH,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,EAAE,aAAa,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;
|
|
1
|
+
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/openaddresses/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAoCG;AAEH,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,EAAE,aAAa,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AACpE,OAAO,EAAE,mBAAmB,EAAE,MAAM,kBAAkB,CAAA;AAGtD,MAAM,CAAC,MAAM,wBAAwB,GAAG,eAAe,CAAA;AACvD,MAAM,CAAC,MAAM,6BAA6B,GAAG,WAAW,CAAA;AAmBxD,0FAA0F;AAC1F,SAAS,mBAAmB,CAAC,GAAY;IACxC,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAA;IAC9C,MAAM,GAAG,GAA2B,EAAE,CAAA;IACtC,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAA8B,CAAC,EAAE,CAAC;QACrE,IAAI,OAAO,CAAC,KAAK,QAAQ;YAAE,GAAG,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,GAAG,CAAC,CAAA;aAC9C,IAAI,OAAO,CAAC,KAAK,QAAQ;YAAE,GAAG,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAA;IACjE,CAAC;IACD,OAAO,GAAmB,CAAA;AAC3B,CAAC;AAED,+FAA+F;AAC/F,SAAS,gBAAgB,CAAC,IAAY;IACrC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAA;IAC3B,IAAI,CAAC,OAAO,IAAI,OAAO,CAAC,UAAU,CAAC,GAAG,CAAC;QAAE,OAAO,IAAI,CAAA;IACpD,IAAI,MAAe,CAAA;IACnB,IAAI,CAAC;QACJ,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAA;IAC7B,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAA;IACZ,CAAC;IACD,IAAI,CAAC,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ;QAAE,OAAO,IAAI,CAAA;IACtD,MAAM,GAAG,GAAG,MAAiD,CAAA;IAC7D,IAAI,GAAG,CAAC,IAAI,KAAK,SAAS;QAAE,OAAO,IAAI,CAAA;IACvC,OAAO,mBAAmB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAA;AAC3C,CAAC;AAoBD;;;GAGG;AACH,MAAM,UAAU,0BAA0B,CAAC,OAAoC,EAAE;IAChF,MAAM,cAAc,GAAG,IAAI,CAAC,cAAc,IAAI,6BAA6B,CAAA;IAC3E,MAAM,eAAe,GAAG,IAAI,CAAC,eAAe,IAAI,IAAI,CAAA;IAEpD,OAAO;QACN,EAAE,EAAE,wBAAwB;QAC5B,cAAc;QACd,WAAW,EAAE,6EAA6E;QAE1F,KAAK,CAAC,CAAC,IAAI,CAAC,WAA2B;YACtC,IAAI,CAAC,WAAW,CAAC,OAAO,EAAE,CAAC;gBAC1B,MAAM,IAAI,KAAK,CACd,0HAA0H,CAC1H,CAAA;YACF,CAAC;YACD,MAAM,OAAO,GAAG,WAAW,CAAC,OAAO,CAAA;YAEnC,MAAM,MAAM,GAAG,gBAAgB,CAAC,WAAW,CAAC,SAAS,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,CAAC,CAAA;YAC5E,MAAM,KAAK,GAAG,eAAe,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAA;YAErE,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,iBAAiB,GAAG,CAAC,CAAA;YACzB,IAAI,CAAC;gBACJ,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;oBAChC,IAAI,WAAW,CAAC,MAAM,EAAE,OAAO;wBAAE,MAAK;oBACtC,IAAI,WAAW,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,WAAW,CAAC,KAAK;wBAAE,MAAK;oBAE1E,MAAM,KAAK,GAAG,gBAAgB,CAAC,IAAI,CAAC,CAAA;oBACpC,IAAI,CAAC,KAAK;wBAAE,SAAQ;oBAEpB,MAAM,WAAW,GAAG,KAAK,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBAC9C,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACzC,MAAM,IAAI,GAAG,KAAK,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACrC,MAAM,IAAI,GAAG,KAAK,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACrC,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACzC,MAAM,QAAQ,GAAG,KAAK,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBAE7C,iFAAiF;oBACjF,wDAAwD;oBACxD,IAAI,CAAC,MAAM;wBAAE,SAAQ;oBACrB,IAAI,CAAC,IAAI,IAAI,CAAC,QAAQ;wBAAE,SAAQ;oBAEhC,MAAM,OAAO,GAAG,CAAC,KAAK,CAAC,OAAO,EAAE,IAAI,EAAE,IAAI,cAAc,CAAC,CAAC,IAAI,EAAE,CAAA;oBAEhE,IAAI,CAAC,eAAe,IAAI,mBAAmB,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;wBAC3D,iBAAiB,EAAE,CAAA;wBACnB,SAAQ;oBACT,CAAC;oBAED,MAAM,UAAU,GAA+B,EAAE,CAAA;oBACjD,IAAI,WAAW;wBAAE,UAAU,CAAC,YAAY,GAAG,WAAW,CAAA;oBACtD,IAAI,MAAM;wBAAE,UAAU,CAAC,MAAM,GAAG,MAAM,CAAA;oBACtC,IAAI,IAAI;wBAAE,UAAU,CAAC,IAAI,GAAG,IAAI,CAAA;oBAChC,IAAI,IAAI;wBAAE,UAAU,CAAC,QAAQ,GAAG,IAAI,CAAA;oBACpC,IAAI,MAAM;wBAAE,UAAU,CAAC,MAAM,GAAG,MAAM,CAAA;oBACtC,IAAI,QAAQ;wBAAE,UAAU,CAAC,QAAQ,GAAG,QAAQ,CAAA;oBAE5C,MAAM,GAAG,GAAG,aAAa,CAAC,UAAU,EAAE,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;oBACnE,IAAI,CAAC,GAAG;wBAAE,SAAQ;oBAElB,MAAM,OAAO,GAAG,mBAAmB,CAAC,UAAU,EAAE,GAAG,CAAC,CAAA;oBACpD,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,KAAK,CAAC;wBAAE,SAAQ;oBAE/C,MAAM,YAAY,GAAG,KAAK,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,KAAK,CAAC,EAAE,EAAE,IAAI,EAAE,CAAA;oBAC3D,MAAM,QAAQ,GAAG,YAAY;wBAC5B,CAAC,CAAC,GAAG,wBAAwB,IAAI,YAAY,EAAE;wBAC/C,CAAC,CAAC,cAAc,CAAC,wBAAwB,EAAE,OAAO,CAAC,CAAA;oBAEpD,MAAM;wBACL,GAAG;wBACH,UAAU,EAAE,OAAO;wBACnB,OAAO;wBACP,MAAM,EAAE,wBAAwB;wBAChC,SAAS,EAAE,QAAQ;wBACnB,cAAc,EAAE,EAAE;wBAClB,OAAO;qBACP,CAAA;oBACD,OAAO,EAAE,CAAA;gBACV,CAAC;YACF,CAAC;oBAAS,CAAC;gBACV,KAAK,CAAC,KAAK,EAAE,CAAA;gBACb,MAAM,CAAC,OAAO,EAAE,CAAA;gBAChB,IAAI,iBAAiB,GAAG,CAAC,EAAE,CAAC;oBAC3B,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,oBAAoB,iBAAiB,8BAA8B,OAAO,SAAS,CAAC,CAAA;gBAC1G,CAAC;YACF,CAAC;QACF,CAAC;KACD,CAAA;AACF,CAAC;AAED,MAAM,CAAC,MAAM,oBAAoB,GAAG,0BAA0B,EAAE,CAAA"}
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @copyright Sister Software
|
|
3
|
+
* @license AGPL-3.0
|
|
4
|
+
* @author Teffen Ellis, et al.
|
|
5
|
+
*
|
|
6
|
+
* `overture`: Overture Maps Addresses adapter (epic #470 — the gated corpus adapter, realized
|
|
7
|
+
* 2026-06-20). Overture's global Addresses theme is the single-schema, well-normalized address
|
|
8
|
+
* dataset that fixes OpenAddresses' per-country patchiness (OA dropped Spain; OA-DE omits the
|
|
9
|
+
* Bundesland) — it even re-hosts the OA Spain data the standalone OA bucket no longer serves.
|
|
10
|
+
*
|
|
11
|
+
* This adapter consumes a per-country LINE-DELIMITED JSON dump of the corpus-relevant fields (`{
|
|
12
|
+
* street, number, unit, postcode, locality }`), produced by `scripts/ingest-overture-addresses.ts
|
|
13
|
+
* --corpus-jsonl` (which does the DuckDB / S3 heavy lifting and flattens `address_levels` → the
|
|
14
|
+
* municipality locality). The split keeps `@mailwoman/corpus` — a RUNTIME dep of the `mailwoman`
|
|
15
|
+
* CLI — free of the heavy native `@duckdb/node-api`; the adapter just streams JSONL line-by-line,
|
|
16
|
+
* exactly like `openaddresses`.
|
|
17
|
+
*
|
|
18
|
+
* The `street` surface carries the locale's street keyword verbatim (`"CALLE JULAN"`, `"VIA
|
|
19
|
+
* ROMA"`). We map it to `street` whole and let the downstream affix-relabel split `street_prefix`
|
|
20
|
+
* — the same path every other source rides. This shard exists because the model was
|
|
21
|
+
* en-us/fr-trained and never saw non-en/fr street formats (the 2026-06-19 EU parse-blocker
|
|
22
|
+
* measured loc-correct ES 21% / IT 59% / NL 64% vs FR/US ~98%).
|
|
23
|
+
*
|
|
24
|
+
* `--country` is REQUIRED (the JSONL is per-country and the rows omit a country field), matching
|
|
25
|
+
* `openaddresses`. License is Overture's CDLA-Permissive-2.0 (attribution; not share-alike).
|
|
26
|
+
*
|
|
27
|
+
* | Field | ComponentTag | | --------- | ---------------------------------------------- | |
|
|
28
|
+
* `street` | `street` (keyword incl.; affix-relabel splits prefix) | | `number` | `house_number`
|
|
29
|
+
* (skipped when "S-N"/"S/N" = sin número) | | `unit` | `unit` (if non-empty) | | `postcode`|
|
|
30
|
+
* `postcode` | | `locality`| `locality` (Overture address_levels municipality, or postal_city) |
|
|
31
|
+
*/
|
|
32
|
+
import type { CorpusAdapter } from "../../types.js";
|
|
33
|
+
export declare const OVERTURE_ADAPTER_ID = "overture";
|
|
34
|
+
export declare const OVERTURE_DEFAULT_LICENSE = "CDLA-Permissive-2.0";
|
|
35
|
+
export declare function createOvertureAdapter(): CorpusAdapter;
|
|
36
|
+
export declare const overtureAdapter: CorpusAdapter;
|
|
37
|
+
//# sourceMappingURL=adapter.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/overture/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AAMH,OAAO,KAAK,EAAgC,aAAa,EAAE,MAAM,gBAAgB,CAAA;AAEjF,eAAO,MAAM,mBAAmB,aAAa,CAAA;AAC7C,eAAO,MAAM,wBAAwB,wBAAwB,CAAA;AAsB7D,wBAAgB,qBAAqB,IAAI,aAAa,CAmErD;AAED,eAAO,MAAM,eAAe,eAA0B,CAAA"}
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @copyright Sister Software
|
|
3
|
+
* @license AGPL-3.0
|
|
4
|
+
* @author Teffen Ellis, et al.
|
|
5
|
+
*
|
|
6
|
+
* `overture`: Overture Maps Addresses adapter (epic #470 — the gated corpus adapter, realized
|
|
7
|
+
* 2026-06-20). Overture's global Addresses theme is the single-schema, well-normalized address
|
|
8
|
+
* dataset that fixes OpenAddresses' per-country patchiness (OA dropped Spain; OA-DE omits the
|
|
9
|
+
* Bundesland) — it even re-hosts the OA Spain data the standalone OA bucket no longer serves.
|
|
10
|
+
*
|
|
11
|
+
* This adapter consumes a per-country LINE-DELIMITED JSON dump of the corpus-relevant fields (`{
|
|
12
|
+
* street, number, unit, postcode, locality }`), produced by `scripts/ingest-overture-addresses.ts
|
|
13
|
+
* --corpus-jsonl` (which does the DuckDB / S3 heavy lifting and flattens `address_levels` → the
|
|
14
|
+
* municipality locality). The split keeps `@mailwoman/corpus` — a RUNTIME dep of the `mailwoman`
|
|
15
|
+
* CLI — free of the heavy native `@duckdb/node-api`; the adapter just streams JSONL line-by-line,
|
|
16
|
+
* exactly like `openaddresses`.
|
|
17
|
+
*
|
|
18
|
+
* The `street` surface carries the locale's street keyword verbatim (`"CALLE JULAN"`, `"VIA
|
|
19
|
+
* ROMA"`). We map it to `street` whole and let the downstream affix-relabel split `street_prefix`
|
|
20
|
+
* — the same path every other source rides. This shard exists because the model was
|
|
21
|
+
* en-us/fr-trained and never saw non-en/fr street formats (the 2026-06-19 EU parse-blocker
|
|
22
|
+
* measured loc-correct ES 21% / IT 59% / NL 64% vs FR/US ~98%).
|
|
23
|
+
*
|
|
24
|
+
* `--country` is REQUIRED (the JSONL is per-country and the rows omit a country field), matching
|
|
25
|
+
* `openaddresses`. License is Overture's CDLA-Permissive-2.0 (attribution; not share-alike).
|
|
26
|
+
*
|
|
27
|
+
* | Field | ComponentTag | | --------- | ---------------------------------------------- | |
|
|
28
|
+
* `street` | `street` (keyword incl.; affix-relabel splits prefix) | | `number` | `house_number`
|
|
29
|
+
* (skipped when "S-N"/"S/N" = sin número) | | `unit` | `unit` (if non-empty) | | `postcode`|
|
|
30
|
+
* `postcode` | | `locality`| `locality` (Overture address_levels municipality, or postal_city) |
|
|
31
|
+
*/
|
|
32
|
+
import { createReadStream } from "node:fs";
|
|
33
|
+
import { createInterface } from "node:readline";
|
|
34
|
+
import { stableSourceId } from "../../adapter.js";
|
|
35
|
+
import { formatAddress, reconcileComponents } from "../../format.js";
|
|
36
|
+
export const OVERTURE_ADAPTER_ID = "overture";
|
|
37
|
+
export const OVERTURE_DEFAULT_LICENSE = "CDLA-Permissive-2.0";
|
|
38
|
+
function parseLine(line) {
|
|
39
|
+
const t = line.trim();
|
|
40
|
+
if (!t || t.startsWith("#"))
|
|
41
|
+
return null;
|
|
42
|
+
try {
|
|
43
|
+
const o = JSON.parse(t);
|
|
44
|
+
return o && typeof o === "object" ? o : null;
|
|
45
|
+
}
|
|
46
|
+
catch {
|
|
47
|
+
return null;
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
export function createOvertureAdapter() {
|
|
51
|
+
return {
|
|
52
|
+
id: OVERTURE_ADAPTER_ID,
|
|
53
|
+
defaultLicense: OVERTURE_DEFAULT_LICENSE,
|
|
54
|
+
description: "Overture Maps Addresses (global): per-country JSONL of street/number/postcode/locality.",
|
|
55
|
+
async *rows(opts) {
|
|
56
|
+
if (!opts.country) {
|
|
57
|
+
throw new Error("overture adapter: --country is required (the Overture JSONL is per-country and rows omit a country field)");
|
|
58
|
+
}
|
|
59
|
+
const country = opts.country;
|
|
60
|
+
const stream = createReadStream(opts.inputPath, { encoding: "utf8" });
|
|
61
|
+
const lines = createInterface({ input: stream, crlfDelay: Infinity });
|
|
62
|
+
let emitted = 0;
|
|
63
|
+
try {
|
|
64
|
+
for await (const line of lines) {
|
|
65
|
+
if (opts.signal?.aborted)
|
|
66
|
+
break;
|
|
67
|
+
if (opts.limit !== undefined && emitted >= opts.limit)
|
|
68
|
+
break;
|
|
69
|
+
const r = parseLine(line);
|
|
70
|
+
if (!r)
|
|
71
|
+
continue;
|
|
72
|
+
const street = r.street?.trim() ?? "";
|
|
73
|
+
const number = r.number?.trim() ?? "";
|
|
74
|
+
const unit = r.unit?.trim() ?? "";
|
|
75
|
+
const postcode = r.postcode?.trim() ?? "";
|
|
76
|
+
const locality = r.locality?.trim() ?? "";
|
|
77
|
+
// Only useful with a street + (postcode OR locality); point-only rows quarantine anyway.
|
|
78
|
+
if (!street)
|
|
79
|
+
continue;
|
|
80
|
+
if (!postcode && !locality)
|
|
81
|
+
continue;
|
|
82
|
+
const components = {};
|
|
83
|
+
// Overture "S-N" / "S/N" = sin número; only keep a real numeric house number.
|
|
84
|
+
if (/^\d/.test(number))
|
|
85
|
+
components.house_number = number;
|
|
86
|
+
components.street = street;
|
|
87
|
+
if (unit)
|
|
88
|
+
components.unit = unit;
|
|
89
|
+
if (postcode)
|
|
90
|
+
components.postcode = postcode;
|
|
91
|
+
if (locality)
|
|
92
|
+
components.locality = locality;
|
|
93
|
+
const raw = formatAddress(components, country, { separator: ", " });
|
|
94
|
+
if (!raw)
|
|
95
|
+
continue;
|
|
96
|
+
const aligned = reconcileComponents(components, raw);
|
|
97
|
+
if (Object.keys(aligned).length === 0)
|
|
98
|
+
continue;
|
|
99
|
+
yield {
|
|
100
|
+
raw,
|
|
101
|
+
components: aligned,
|
|
102
|
+
country,
|
|
103
|
+
source: OVERTURE_ADAPTER_ID,
|
|
104
|
+
source_id: stableSourceId(OVERTURE_ADAPTER_ID, aligned),
|
|
105
|
+
corpus_version: "",
|
|
106
|
+
license: OVERTURE_DEFAULT_LICENSE,
|
|
107
|
+
};
|
|
108
|
+
emitted++;
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
finally {
|
|
112
|
+
lines.close();
|
|
113
|
+
stream.destroy();
|
|
114
|
+
}
|
|
115
|
+
},
|
|
116
|
+
};
|
|
117
|
+
}
|
|
118
|
+
export const overtureAdapter = createOvertureAdapter();
|
|
119
|
+
//# sourceMappingURL=adapter.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/overture/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AAEH,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,EAAE,aAAa,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AAGpE,MAAM,CAAC,MAAM,mBAAmB,GAAG,UAAU,CAAA;AAC7C,MAAM,CAAC,MAAM,wBAAwB,GAAG,qBAAqB,CAAA;AAW7D,SAAS,SAAS,CAAC,IAAY;IAC9B,MAAM,CAAC,GAAG,IAAI,CAAC,IAAI,EAAE,CAAA;IACrB,IAAI,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,CAAC,GAAG,CAAC;QAAE,OAAO,IAAI,CAAA;IACxC,IAAI,CAAC;QACJ,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;QACvB,OAAO,CAAC,IAAI,OAAO,CAAC,KAAK,QAAQ,CAAC,CAAC,CAAE,CAAuB,CAAC,CAAC,CAAC,IAAI,CAAA;IACpE,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAA;IACZ,CAAC;AACF,CAAC;AAED,MAAM,UAAU,qBAAqB;IACpC,OAAO;QACN,EAAE,EAAE,mBAAmB;QACvB,cAAc,EAAE,wBAAwB;QACxC,WAAW,EAAE,yFAAyF;QAEtG,KAAK,CAAC,CAAC,IAAI,CAAC,IAAoB;YAC/B,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;gBACnB,MAAM,IAAI,KAAK,CACd,2GAA2G,CAC3G,CAAA;YACF,CAAC;YACD,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAA;YAE5B,MAAM,MAAM,GAAG,gBAAgB,CAAC,IAAI,CAAC,SAAS,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,CAAC,CAAA;YACrE,MAAM,KAAK,GAAG,eAAe,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAA;YAErE,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,CAAC;gBACJ,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;oBAChC,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;wBAAE,MAAK;oBAC/B,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,IAAI,CAAC,KAAK;wBAAE,MAAK;oBAE5D,MAAM,CAAC,GAAG,SAAS,CAAC,IAAI,CAAC,CAAA;oBACzB,IAAI,CAAC,CAAC;wBAAE,SAAQ;oBAEhB,MAAM,MAAM,GAAG,CAAC,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACrC,MAAM,MAAM,GAAG,CAAC,CAAC,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACrC,MAAM,IAAI,GAAG,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACjC,MAAM,QAAQ,GAAG,CAAC,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBACzC,MAAM,QAAQ,GAAG,CAAC,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;oBAEzC,yFAAyF;oBACzF,IAAI,CAAC,MAAM;wBAAE,SAAQ;oBACrB,IAAI,CAAC,QAAQ,IAAI,CAAC,QAAQ;wBAAE,SAAQ;oBAEpC,MAAM,UAAU,GAA+B,EAAE,CAAA;oBACjD,8EAA8E;oBAC9E,IAAI,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC;wBAAE,UAAU,CAAC,YAAY,GAAG,MAAM,CAAA;oBACxD,UAAU,CAAC,MAAM,GAAG,MAAM,CAAA;oBAC1B,IAAI,IAAI;wBAAE,UAAU,CAAC,IAAI,GAAG,IAAI,CAAA;oBAChC,IAAI,QAAQ;wBAAE,UAAU,CAAC,QAAQ,GAAG,QAAQ,CAAA;oBAC5C,IAAI,QAAQ;wBAAE,UAAU,CAAC,QAAQ,GAAG,QAAQ,CAAA;oBAE5C,MAAM,GAAG,GAAG,aAAa,CAAC,UAAU,EAAE,OAAO,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;oBACnE,IAAI,CAAC,GAAG;wBAAE,SAAQ;oBAElB,MAAM,OAAO,GAAG,mBAAmB,CAAC,UAAU,EAAE,GAAG,CAAC,CAAA;oBACpD,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,KAAK,CAAC;wBAAE,SAAQ;oBAE/C,MAAM;wBACL,GAAG;wBACH,UAAU,EAAE,OAAO;wBACnB,OAAO;wBACP,MAAM,EAAE,mBAAmB;wBAC3B,SAAS,EAAE,cAAc,CAAC,mBAAmB,EAAE,OAAO,CAAC;wBACvD,cAAc,EAAE,EAAE;wBAClB,OAAO,EAAE,wBAAwB;qBACjC,CAAA;oBACD,OAAO,EAAE,CAAA;gBACV,CAAC;YACF,CAAC;oBAAS,CAAC;gBACV,KAAK,CAAC,KAAK,EAAE,CAAA;gBACb,MAAM,CAAC,OAAO,EAAE,CAAA;YACjB,CAAC;QACF,CAAC;KACD,CAAA;AACF,CAAC;AAED,MAAM,CAAC,MAAM,eAAe,GAAG,qBAAqB,EAAE,CAAA"}
|
|
@@ -42,6 +42,14 @@ export interface SynthPoBoxAdapterOptions {
|
|
|
42
42
|
* Deterministic seed for reproducible synthesis. Default Date.now().
|
|
43
43
|
*/
|
|
44
44
|
seed?: number;
|
|
45
|
+
/**
|
|
46
|
+
* Probability (0..1), evaluated per input tuple, of ALSO emitting one US military/diplomatic
|
|
47
|
+
* PO-box row (`PSC/CMR/Unit <id> Box <box>, APO/FPO/DPO AA/AE/AP <zip>`, #517). These rows are
|
|
48
|
+
* self-contained — they draw no field from the input tuple, so military volume scales with the
|
|
49
|
+
* input stream size. Default 0 (off) — the adapter's contract is "one row per input"; the corpus
|
|
50
|
+
* build recipe opts in to seed the rare-but-real military class without changing the default.
|
|
51
|
+
*/
|
|
52
|
+
militaryRatio?: number;
|
|
45
53
|
}
|
|
46
54
|
export declare function createSynthPoBoxAdapter(opts?: SynthPoBoxAdapterOptions): CorpusAdapter;
|
|
47
55
|
export declare const synthPoBoxAdapter: CorpusAdapter;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/synth-po-box/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAKH,OAAO,
|
|
1
|
+
{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../../src/adapters/synth-po-box/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAKH,OAAO,EAKN,KAAK,cAAc,EACnB,MAAM,4BAA4B,CAAA;AACnC,OAAO,KAAK,EAAgC,aAAa,EAAE,MAAM,gBAAgB,CAAA;AAEjF,eAAO,MAAM,uBAAuB,iBAAiB,CAAA;AACrD,eAAO,MAAM,oBAAoB,qEAAgE,CAAA;AAEjG,MAAM,WAAW,aAAc,SAAQ,cAAc;IACpD,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,WAAW,CAAC,EAAE,MAAM,CAAA;CACpB;AAED,MAAM,WAAW,wBAAwB;IACxC;;;OAGG;IACH,gBAAgB,CAAC,EAAE,MAAM,CAAA;IACzB;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB;;OAEG;IACH,IAAI,CAAC,EAAE,MAAM,CAAA;IACb;;;;;;OAMG;IACH,aAAa,CAAC,EAAE,MAAM,CAAA;CACtB;AAUD,wBAAgB,uBAAuB,CAAC,IAAI,GAAE,wBAA6B,GAAG,aAAa,CA6G1F;AAED,eAAO,MAAM,iBAAiB,eAA4B,CAAA"}
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
import { createReadStream } from "node:fs";
|
|
23
23
|
import { createInterface } from "node:readline";
|
|
24
24
|
import { stableSourceId } from "../../adapter.js";
|
|
25
|
-
import { synthesizePoBoxRow } from "../../synthesize-po-box.js";
|
|
25
|
+
import { countryToLocale, REGION_OPTIONAL_LOCALES, synthesizeMilitaryPoBoxRow, synthesizePoBoxRow, } from "../../synthesize-po-box.js";
|
|
26
26
|
export const SYNTH_PO_BOX_ADAPTER_ID = "synth-po-box";
|
|
27
27
|
export const SYNTH_PO_BOX_LICENSE = "Synthetic — derived from CC-BY / public-domain input tuples";
|
|
28
28
|
function makeRandom(seed) {
|
|
@@ -35,6 +35,7 @@ function makeRandom(seed) {
|
|
|
35
35
|
export function createSynthPoBoxAdapter(opts = {}) {
|
|
36
36
|
const variantsPerInput = opts.variantsPerInput ?? 1;
|
|
37
37
|
const pmbRatio = opts.pmbRatio ?? 0.15;
|
|
38
|
+
const militaryRatio = opts.militaryRatio ?? 0;
|
|
38
39
|
return {
|
|
39
40
|
id: SYNTH_PO_BOX_ADAPTER_ID,
|
|
40
41
|
defaultLicense: SYNTH_PO_BOX_LICENSE,
|
|
@@ -45,6 +46,7 @@ export function createSynthPoBoxAdapter(opts = {}) {
|
|
|
45
46
|
const rl = createInterface({ input: stream, crlfDelay: Infinity });
|
|
46
47
|
let emitted = 0;
|
|
47
48
|
let skipped = 0;
|
|
49
|
+
let militarySeq = 0;
|
|
48
50
|
for await (const line of rl) {
|
|
49
51
|
if (options.signal?.aborted)
|
|
50
52
|
break;
|
|
@@ -61,7 +63,11 @@ export function createSynthPoBoxAdapter(opts = {}) {
|
|
|
61
63
|
skipped++;
|
|
62
64
|
continue;
|
|
63
65
|
}
|
|
64
|
-
|
|
66
|
+
// Region is required EXCEPT for region-less locales (NZ: `Private Bag 12, Auckland 1010`
|
|
67
|
+
// has no region token, #517). synthesizePoBoxRow handles region absence; the guard just
|
|
68
|
+
// must not discard those tuples as "missing region".
|
|
69
|
+
const regionOptional = input.country ? REGION_OPTIONAL_LOCALES.has(countryToLocale(input.country)) : false;
|
|
70
|
+
if (!input.locality || !input.postcode || !input.country || (!input.region && !regionOptional)) {
|
|
65
71
|
skipped++;
|
|
66
72
|
continue;
|
|
67
73
|
}
|
|
@@ -93,6 +99,35 @@ export function createSynthPoBoxAdapter(opts = {}) {
|
|
|
93
99
|
if (options.limit !== undefined && emitted >= options.limit)
|
|
94
100
|
break;
|
|
95
101
|
}
|
|
102
|
+
// US military/diplomatic PO-box rows (#517): self-contained — draw nothing from the input
|
|
103
|
+
// tuple — emitted per input line with probability `militaryRatio` (off by default, so the
|
|
104
|
+
// default random stream and output are byte-identical). Military volume scales with the
|
|
105
|
+
// stream rather than the US-tuple count. US-only: suppressed under a non-US country filter
|
|
106
|
+
// and counted against `limit` like any other row.
|
|
107
|
+
const militaryAllowed = !options.country || options.country === "US";
|
|
108
|
+
if (militaryRatio > 0 &&
|
|
109
|
+
militaryAllowed &&
|
|
110
|
+
(options.limit === undefined || emitted < options.limit) &&
|
|
111
|
+
random() < militaryRatio) {
|
|
112
|
+
const mil = synthesizeMilitaryPoBoxRow({ random });
|
|
113
|
+
const sourceId = stableSourceId(SYNTH_PO_BOX_ADAPTER_ID, {
|
|
114
|
+
po_box: `${mil.components.po_box}#mil${militarySeq++}`,
|
|
115
|
+
locality: mil.components.locality,
|
|
116
|
+
region: mil.components.region,
|
|
117
|
+
postcode: mil.components.postcode,
|
|
118
|
+
});
|
|
119
|
+
yield {
|
|
120
|
+
raw: mil.raw,
|
|
121
|
+
components: mil.components,
|
|
122
|
+
country: "US",
|
|
123
|
+
locale: mil.locale,
|
|
124
|
+
source: SYNTH_PO_BOX_ADAPTER_ID,
|
|
125
|
+
source_id: sourceId,
|
|
126
|
+
corpus_version: "",
|
|
127
|
+
license: SYNTH_PO_BOX_LICENSE,
|
|
128
|
+
};
|
|
129
|
+
emitted++;
|
|
130
|
+
}
|
|
96
131
|
}
|
|
97
132
|
},
|
|
98
133
|
};
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/synth-po-box/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,
|
|
1
|
+
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/synth-po-box/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/C,OAAO,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAA;AACjD,OAAO,EACN,eAAe,EACf,uBAAuB,EACvB,0BAA0B,EAC1B,kBAAkB,GAElB,MAAM,4BAA4B,CAAA;AAGnC,MAAM,CAAC,MAAM,uBAAuB,GAAG,cAAc,CAAA;AACrD,MAAM,CAAC,MAAM,oBAAoB,GAAG,6DAA6D,CAAA;AAgCjG,SAAS,UAAU,CAAC,IAAY;IAC/B,IAAI,CAAC,GAAG,IAAI,CAAA;IACZ,OAAO,GAAG,EAAE;QACX,CAAC,GAAG,CAAC,CAAC,GAAG,OAAO,GAAG,UAAU,CAAC,GAAG,UAAU,CAAA;QAC3C,OAAO,CAAC,GAAG,UAAU,CAAA;IACtB,CAAC,CAAA;AACF,CAAC;AAED,MAAM,UAAU,uBAAuB,CAAC,OAAiC,EAAE;IAC1E,MAAM,gBAAgB,GAAG,IAAI,CAAC,gBAAgB,IAAI,CAAC,CAAA;IACnD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAA;IACtC,MAAM,aAAa,GAAG,IAAI,CAAC,aAAa,IAAI,CAAC,CAAA;IAE7C,OAAO;QACN,EAAE,EAAE,uBAAuB;QAC3B,cAAc,EAAE,oBAAoB;QACpC,WAAW,EACV,sKAAsK;QAEvK,KAAK,CAAC,CAAC,IAAI,CAAC,OAAuB;YAClC,MAAM,MAAM,GAAG,UAAU,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,GAAG,EAAE,CAAC,CAAA;YAElD,MAAM,MAAM,GAAG,gBAAgB,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,CAAC,CAAA;YACxE,MAAM,EAAE,GAAG,eAAe,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,EAAE,CAAC,CAAA;YAElE,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,WAAW,GAAG,CAAC,CAAA;YAEnB,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,EAAE,EAAE,CAAC;gBAC7B,IAAI,OAAO,CAAC,MAAM,EAAE,OAAO;oBAAE,MAAK;gBAClC,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,OAAO,CAAC,KAAK;oBAAE,MAAK;gBAElE,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAA;gBAC3B,IAAI,CAAC,OAAO;oBAAE,SAAQ;gBAEtB,IAAI,KAAoB,CAAA;gBACxB,IAAI,CAAC;oBACJ,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAkB,CAAA;gBAC7C,CAAC;gBAAC,MAAM,CAAC;oBACR,OAAO,EAAE,CAAA;oBACT,SAAQ;gBACT,CAAC;gBAED,yFAAyF;gBACzF,wFAAwF;gBACxF,qDAAqD;gBACrD,MAAM,cAAc,GAAG,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,uBAAuB,CAAC,GAAG,CAAC,eAAe,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAA;gBAC1G,IAAI,CAAC,KAAK,CAAC,QAAQ,IAAI,CAAC,KAAK,CAAC,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,IAAI,CAAC,CAAC,KAAK,CAAC,MAAM,IAAI,CAAC,cAAc,CAAC,EAAE,CAAC;oBAChG,OAAO,EAAE,CAAA;oBACT,SAAQ;gBACT,CAAC;gBAED,IAAI,OAAO,CAAC,OAAO,IAAI,OAAO,CAAC,OAAO,KAAK,KAAK,CAAC,OAAO;oBAAE,SAAQ;gBAElE,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,gBAAgB,EAAE,CAAC,EAAE,EAAE,CAAC;oBAC3C,MAAM,KAAK,GAAG,kBAAkB,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,CAAC,CAAA;oBAC7D,IAAI,CAAC,KAAK;wBAAE,SAAQ;oBAEpB,6EAA6E;oBAC7E,iDAAiD;oBACjD,MAAM,QAAQ,GAAG,cAAc,CAAC,uBAAuB,EAAE;wBACxD,QAAQ,EAAE,GAAG,KAAK,CAAC,QAAQ,IAAI,CAAC,EAAE;wBAClC,MAAM,EAAE,KAAK,CAAC,MAAM;wBACpB,QAAQ,EAAE,KAAK,CAAC,QAAQ;wBACxB,OAAO,EAAE,KAAK,CAAC,OAAO;qBACtB,CAAC,CAAA;oBAEF,MAAM;wBACL,GAAG,EAAE,KAAK,CAAC,GAAG;wBACd,UAAU,EAAE,KAAK,CAAC,UAAU;wBAC5B,OAAO,EAAE,KAAK,CAAC,OAAO;wBACtB,MAAM,EAAE,KAAK,CAAC,MAAM;wBACpB,MAAM,EAAE,uBAAuB;wBAC/B,SAAS,EAAE,QAAQ;wBACnB,cAAc,EAAE,EAAE;wBAClB,OAAO,EAAE,oBAAoB;qBAC7B,CAAA;oBACD,OAAO,EAAE,CAAA;oBAET,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,OAAO,CAAC,KAAK;wBAAE,MAAK;gBACnE,CAAC;gBAED,0FAA0F;gBAC1F,0FAA0F;gBAC1F,wFAAwF;gBACxF,2FAA2F;gBAC3F,kDAAkD;gBAClD,MAAM,eAAe,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,OAAO,CAAC,OAAO,KAAK,IAAI,CAAA;gBACpE,IACC,aAAa,GAAG,CAAC;oBACjB,eAAe;oBACf,CAAC,OAAO,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC;oBACxD,MAAM,EAAE,GAAG,aAAa,EACvB,CAAC;oBACF,MAAM,GAAG,GAAG,0BAA0B,CAAC,EAAE,MAAM,EAAE,CAAC,CAAA;oBAClD,MAAM,QAAQ,GAAG,cAAc,CAAC,uBAAuB,EAAE;wBACxD,MAAM,EAAE,GAAG,GAAG,CAAC,UAAU,CAAC,MAAM,OAAO,WAAW,EAAE,EAAE;wBACtD,QAAQ,EAAE,GAAG,CAAC,UAAU,CAAC,QAAS;wBAClC,MAAM,EAAE,GAAG,CAAC,UAAU,CAAC,MAAO;wBAC9B,QAAQ,EAAE,GAAG,CAAC,UAAU,CAAC,QAAS;qBAClC,CAAC,CAAA;oBACF,MAAM;wBACL,GAAG,EAAE,GAAG,CAAC,GAAG;wBACZ,UAAU,EAAE,GAAG,CAAC,UAAU;wBAC1B,OAAO,EAAE,IAAI;wBACb,MAAM,EAAE,GAAG,CAAC,MAAM;wBAClB,MAAM,EAAE,uBAAuB;wBAC/B,SAAS,EAAE,QAAQ;wBACnB,cAAc,EAAE,EAAE;wBAClB,OAAO,EAAE,oBAAoB;qBAC7B,CAAA;oBACD,OAAO,EAAE,CAAA;gBACV,CAAC;YACF,CAAC;QACF,CAAC;KACD,CAAA;AACF,CAAC;AAED,MAAM,CAAC,MAAM,iBAAiB,GAAG,uBAAuB,EAAE,CAAA"}
|
|
@@ -111,7 +111,7 @@ export function createTigerAdapter() {
|
|
|
111
111
|
return {
|
|
112
112
|
id: TIGER_ADAPTER_ID,
|
|
113
113
|
defaultLicense: TIGER_DEFAULT_LICENSE,
|
|
114
|
-
description: "US Census TIGER/Line streets + places consumer (public-domain); SQLite DB
|
|
114
|
+
description: "US Census TIGER/Line streets + places consumer (public-domain); SQLite DB built via `mailwoman tiger fetch`.",
|
|
115
115
|
async *rows(opts) {
|
|
116
116
|
if (opts.country && opts.country !== "US") {
|
|
117
117
|
throw new Error(`tiger adapter: only US supported, got country=${opts.country}`);
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/tiger/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAEH,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,8BAA8B,CAAA;AAC9D,OAAO,EAAE,aAAa,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AAEpE,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AAEvD,MAAM,CAAC,MAAM,gBAAgB,GAAG,OAAO,CAAA;AACvC,MAAM,CAAC,MAAM,qBAAqB,GAAG,eAAe,CAAA;AAEpD;;;GAGG;AACH,MAAM,kBAAkB,GAAG,0BAA0B,CAAA;AAiBrD;;;;;;GAMG;AACH,QAAQ,CAAC,CAAC,cAAc,CAAC,GAAmB;IAI3C,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAA;IACpC,IAAI,CAAC,QAAQ;QAAE,OAAM;IACrB,MAAM,KAAK,GAAG,eAAe,CAAC,GAAG,CAAC,OAAO,CAAC,CAAA;IAC1C,IAAI,CAAC,KAAK;QAAE,OAAM;IAElB,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;IACnC,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;IAEnC,MAAM,UAAU,GAAG,eAAe,CAAC,QAAQ,CAAC,CAAA;IAE5C,MAAM,cAAc,GAA+B;QAClD,MAAM,EAAE,KAAK,CAAC,YAAY;QAC1B,MAAM,EAAE,UAAU,CAAC,MAAM;KACzB,CAAA;IACD,IAAI,UAAU,CAAC,MAAM;QAAE,cAAc,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAA;IACvE,IAAI,UAAU,CAAC,MAAM;QAAE,cAAc,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAA;IAEvE,IAAI,CAAC,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC;QACpB,MAAM,EAAE,UAAU,EAAE,cAAc,EAAE,UAAU,EAAE,QAAQ,EAAE,CAAA;QAC1D,OAAM;IACP,CAAC;IACD,IAAI,IAAI,IAAI,IAAI,IAAI,IAAI,KAAK,IAAI,EAAE,CAAC;QACnC,MAAM;YACL,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE;YACjD,UAAU,EAAE,OAAO,IAAI,EAAE;SACzB,CAAA;QACD,OAAM;IACP,CAAC;IACD,IAAI,IAAI;QAAE,MAAM,EAAE,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,UAAU,EAAE,QAAQ,IAAI,EAAE,EAAE,CAAA;IACjG,IAAI,IAAI,IAAI,IAAI,KAAK,IAAI;QAAE,MAAM,EAAE,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,UAAU,EAAE,QAAQ,IAAI,EAAE,EAAE,CAAA;AACnH,CAAC;AAED,sEAAsE;AACtE,QAAQ,CAAC,CAAC,aAAa,CAAC,GAAkB;IAIzC,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAA;IAC5B,IAAI,CAAC,IAAI;QAAE,OAAM;IACjB,MAAM,KAAK,GAAG,eAAe,CAAC,GAAG,CAAC,OAAO,CAAC,CAAA;IAC1C,IAAI,CAAC,KAAK;QAAE,OAAM;IAElB,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;QAC9B,UAAU,EAAE,eAAe;KAC3B,CAAA;IACD,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,CAAC,YAAY,EAAE;QAC1D,UAAU,EAAE,aAAa;KACzB,CAAA;IACD,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,CAAC,YAAY,EAAE,OAAO,EAAE,kBAAkB,EAAE;QACvF,UAAU,EAAE,qBAAqB;KACjC,CAAA;AACF,CAAC;AAED,yFAAyF;AACzF,MAAM,UAAU,kBAAkB;IACjC,OAAO;QACN,EAAE,EAAE,gBAAgB;QACpB,cAAc,EAAE,qBAAqB;QACrC,WAAW,EACV,
|
|
1
|
+
{"version":3,"file":"adapter.js","sourceRoot":"","sources":["../../../../src/adapters/tiger/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAEH,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAA;AAC1C,OAAO,EAAE,eAAe,EAAE,MAAM,8BAA8B,CAAA;AAC9D,OAAO,EAAE,aAAa,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AAEpE,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AAEvD,MAAM,CAAC,MAAM,gBAAgB,GAAG,OAAO,CAAA;AACvC,MAAM,CAAC,MAAM,qBAAqB,GAAG,eAAe,CAAA;AAEpD;;;GAGG;AACH,MAAM,kBAAkB,GAAG,0BAA0B,CAAA;AAiBrD;;;;;;GAMG;AACH,QAAQ,CAAC,CAAC,cAAc,CAAC,GAAmB;IAI3C,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAA;IACpC,IAAI,CAAC,QAAQ;QAAE,OAAM;IACrB,MAAM,KAAK,GAAG,eAAe,CAAC,GAAG,CAAC,OAAO,CAAC,CAAA;IAC1C,IAAI,CAAC,KAAK;QAAE,OAAM;IAElB,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;IACnC,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAA;IAEnC,MAAM,UAAU,GAAG,eAAe,CAAC,QAAQ,CAAC,CAAA;IAE5C,MAAM,cAAc,GAA+B;QAClD,MAAM,EAAE,KAAK,CAAC,YAAY;QAC1B,MAAM,EAAE,UAAU,CAAC,MAAM;KACzB,CAAA;IACD,IAAI,UAAU,CAAC,MAAM;QAAE,cAAc,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAA;IACvE,IAAI,UAAU,CAAC,MAAM;QAAE,cAAc,CAAC,aAAa,GAAG,UAAU,CAAC,MAAM,CAAA;IAEvE,IAAI,CAAC,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC;QACpB,MAAM,EAAE,UAAU,EAAE,cAAc,EAAE,UAAU,EAAE,QAAQ,EAAE,CAAA;QAC1D,OAAM;IACP,CAAC;IACD,IAAI,IAAI,IAAI,IAAI,IAAI,IAAI,KAAK,IAAI,EAAE,CAAC;QACnC,MAAM;YACL,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE;YACjD,UAAU,EAAE,OAAO,IAAI,EAAE;SACzB,CAAA;QACD,OAAM;IACP,CAAC;IACD,IAAI,IAAI;QAAE,MAAM,EAAE,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,UAAU,EAAE,QAAQ,IAAI,EAAE,EAAE,CAAA;IACjG,IAAI,IAAI,IAAI,IAAI,KAAK,IAAI;QAAE,MAAM,EAAE,UAAU,EAAE,EAAE,GAAG,cAAc,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,UAAU,EAAE,QAAQ,IAAI,EAAE,EAAE,CAAA;AACnH,CAAC;AAED,sEAAsE;AACtE,QAAQ,CAAC,CAAC,aAAa,CAAC,GAAkB;IAIzC,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAA;IAC5B,IAAI,CAAC,IAAI;QAAE,OAAM;IACjB,MAAM,KAAK,GAAG,eAAe,CAAC,GAAG,CAAC,OAAO,CAAC,CAAA;IAC1C,IAAI,CAAC,KAAK;QAAE,OAAM;IAElB,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;QAC9B,UAAU,EAAE,eAAe;KAC3B,CAAA;IACD,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,CAAC,YAAY,EAAE;QAC1D,UAAU,EAAE,aAAa;KACzB,CAAA;IACD,MAAM;QACL,UAAU,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,CAAC,YAAY,EAAE,OAAO,EAAE,kBAAkB,EAAE;QACvF,UAAU,EAAE,qBAAqB;KACjC,CAAA;AACF,CAAC;AAED,yFAAyF;AACzF,MAAM,UAAU,kBAAkB;IACjC,OAAO;QACN,EAAE,EAAE,gBAAgB;QACpB,cAAc,EAAE,qBAAqB;QACrC,WAAW,EACV,8GAA8G;QAE/G,KAAK,CAAC,CAAC,IAAI,CAAC,IAAoB;YAC/B,IAAI,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC,OAAO,KAAK,IAAI,EAAE,CAAC;gBAC3C,MAAM,IAAI,KAAK,CAAC,iDAAiD,IAAI,CAAC,OAAO,EAAE,CAAC,CAAA;YACjF,CAAC;YAED,MAAM,EAAE,GAAG,IAAI,YAAY,CAAC,IAAI,CAAC,SAAS,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAA;YAC/D,IAAI,OAAO,GAAG,CAAC,CAAA;YACf,IAAI,CAAC;gBACJ,MAAM,UAAU,GAAG,EAAE,CAAC,OAAO,CAAC,mEAAmE,CAAC,CAAA;gBAClG,MAAM,SAAS,GAAG,EAAE,CAAC,OAAO,CAAC,qDAAqD,CAAC,CAAA;gBAEnF,KAAK,MAAM,GAAG,IAAI,UAAU,CAAC,OAAO,EAAsC,EAAE,CAAC;oBAC5E,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;wBAAE,OAAM;oBAChC,KAAK,MAAM,OAAO,IAAI,cAAc,CAAC,GAAG,CAAC,EAAE,CAAC;wBAC3C,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,IAAI,CAAC,KAAK;4BAAE,OAAM;wBAC7D,MAAM,GAAG,GAAG,aAAa,CAAC,OAAO,CAAC,UAAU,EAAE,IAAI,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;wBACxE,IAAI,CAAC,GAAG;4BAAE,SAAQ;wBAClB,MAAM,OAAO,GAAG,mBAAmB,CAAC,OAAO,CAAC,UAAU,EAAE,GAAG,CAAC,CAAA;wBAC5D,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,KAAK,CAAC;4BAAE,SAAQ;wBAE/C,MAAM;4BACL,GAAG;4BACH,UAAU,EAAE,OAAO;4BACnB,OAAO,EAAE,IAAI;4BACb,MAAM,EAAE,OAAO;4BACf,MAAM,EAAE,gBAAgB;4BACxB,SAAS,EAAE,GAAG,gBAAgB,OAAO,GAAG,CAAC,QAAQ,IAAI,OAAO,CAAC,UAAU,EAAE;4BACzE,cAAc,EAAE,EAAE;4BAClB,OAAO,EAAE,qBAAqB;yBAC9B,CAAA;wBACD,OAAO,EAAE,CAAA;oBACV,CAAC;gBACF,CAAC;gBAED,KAAK,MAAM,GAAG,IAAI,SAAS,CAAC,OAAO,EAAqC,EAAE,CAAC;oBAC1E,IAAI,IAAI,CAAC,MAAM,EAAE,OAAO;wBAAE,OAAM;oBAChC,KAAK,MAAM,OAAO,IAAI,aAAa,CAAC,GAAG,CAAC,EAAE,CAAC;wBAC1C,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,OAAO,IAAI,IAAI,CAAC,KAAK;4BAAE,OAAM;wBAC7D,MAAM,GAAG,GAAG,aAAa,CAAC,OAAO,CAAC,UAAU,EAAE,IAAI,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;wBACxE,IAAI,CAAC,GAAG;4BAAE,SAAQ;wBAClB,MAAM,OAAO,GAAG,mBAAmB,CAAC,OAAO,CAAC,UAAU,EAAE,GAAG,CAAC,CAAA;wBAC5D,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,KAAK,CAAC;4BAAE,SAAQ;wBAE/C,MAAM;4BACL,GAAG;4BACH,UAAU,EAAE,OAAO;4BACnB,OAAO,EAAE,IAAI;4BACb,MAAM,EAAE,OAAO;4BACf,MAAM,EAAE,gBAAgB;4BACxB,SAAS,EAAE,GAAG,gBAAgB,OAAO,GAAG,CAAC,KAAK,IAAI,OAAO,CAAC,UAAU,EAAE;4BACtE,cAAc,EAAE,EAAE;4BAClB,OAAO,EAAE,qBAAqB;yBAC9B,CAAA;wBACD,OAAO,EAAE,CAAA;oBACV,CAAC;gBACF,CAAC;YACF,CAAC;oBAAS,CAAC;gBACV,EAAE,CAAC,KAAK,EAAE,CAAA;YACX,CAAC;QACF,CAAC;KACD,CAAA;AACF,CAAC;AAED,MAAM,CAAC,MAAM,YAAY,GAAG,kBAAkB,EAAE,CAAA"}
|
|
@@ -20,7 +20,7 @@
|
|
|
20
20
|
* the README documents the expected set.
|
|
21
21
|
*
|
|
22
22
|
* Output: one row per CSV record, with `venue` component carrying the site name and the address
|
|
23
|
-
* quad on `(house_number, street, locality, region, postcode)`. Component order is
|
|
23
|
+
* quad on `(house_number, street, locality, region, postcode)`. Component order is critical:
|
|
24
24
|
* `venue` is inserted FIRST so alignment claims its surface span before `locality` searches for
|
|
25
25
|
* its own (the kryptonite case "Buffalo Health Clinic, …, Buffalo, NY" relies on `venue`
|
|
26
26
|
* consuming the first "Buffalo" so locality lands on the second).
|
|
@@ -20,7 +20,7 @@
|
|
|
20
20
|
* the README documents the expected set.
|
|
21
21
|
*
|
|
22
22
|
* Output: one row per CSV record, with `venue` component carrying the site name and the address
|
|
23
|
-
* quad on `(house_number, street, locality, region, postcode)`. Component order is
|
|
23
|
+
* quad on `(house_number, street, locality, region, postcode)`. Component order is critical:
|
|
24
24
|
* `venue` is inserted FIRST so alignment claims its surface span before `locality` searches for
|
|
25
25
|
* its own (the kryptonite case "Buffalo Health Clinic, …, Buffalo, NY" relies on `venue`
|
|
26
26
|
* consuming the first "Buffalo" so locality lands on the second).
|