@mailwoman/corpus 9.4.0 → 10.1.0

package/out/adapters/cordis/adapter.d.ts

@@ -0,0 +1,324 @@
+/**
+ * @copyright Sister Software
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ *
+ * `cordis`: the organization files CORDIS publishes for the participants in the EU's research
+ * framework programmes, read for the address each participating organization registered.
+ *
+ * Input is the publisher's own CSV archives, one per programme, as
+ * `https://cordis.europa.eu/data/cordis-<programme>projects-csv.zip` serves them for FP7
+ * (`fp7`), Horizon 2020 (`h2020`) and Horizon Europe (`HORIZON`).
+ * Each archive holds an `organization.csv`: `;`-separated, every field double-quoted, one row per participation.
+ * An organizationtherefore appears once for every project it took part in.
+ * The adapter streams the member out of the archive through `readZipEntry` and `CSVSpliterator`,
+ * so no extracted copy sits on disk.
+ * `opts.inputPath` is one archive, one extracted `organization.csv`, or a directory of archives.
+ * `fetchCORDIS` writes the directory form.
+ *
+ * Measured on the archives fetched 2026-10-03, the three members hold 420,811 participation rows and
+ * 78,842 distinct published records, and the adapter writes 57,930 rows over 158 countries.
+ *
+ * ## The columns read
+ *
+ * `name`, `shortName`, `street`, `postCode`, `city` and `country` reach a row or decide whether
+ * one is written. `organisationID`, `activityType`, `SME`, `vatNumber` and `nutsCode` were read while
+ * deciding the rules below and reach no row.
+ *
+ * ## Organizations, and no natural person
+ *
+ * A corpus holds the addresses of organizations and no address of an identifiable natural person.
+ * CORDIS publishes no legal-form column.
+ * `activityType` takes the five values `HES`, `PRC`, `REC`, `OTH` and `PUB`, plus an empty value on
+ * older rows, and no value marks a natural person.
+ * A natural person still enters the participant register: a sole trader, or a researcher contracted
+ * in their own name, is registered under their own name, mostly with `activityType` `PRC`.
+ * Measured over the three archives, the names of those registrations take three forms, and
+ * {@linkcode naturalPersonForm} refuses each:
+ *
+ * 1. **Every word is a personal name.** `FISK ADRIAN`, `SCHWARZ KAI-UWE WOLFGANG`, `Kerstin
+ *    Minnich`: two to four words, each a given name or surname in libpostal's `all` dictionaries,
+ *    at least one a given name. 194 records.
+ * 2. **The participant register's `SURNAME Given` casing.** `BUCKENHUSKES Herbert Johannes`: an
+ *    upper-case run followed by title-case words that are all personal names, one of them a given
+ *    name, for a surname the dictionary lacks. 5 records.
+ * 3. **A sole-trader legal form.** German `e.K.`, `e.Kfm.` and `e.Kfr.`, and `Inh.` (owner), in the
+ *    name or the short name, as in `Melina Bucher e.Kfr.`. 3 records.
+ *
+ * Neither name rule fires on a name holding a word that is a country's name, so `ERICSSON FRANCE` and
+ * `IDEMIA Public Security France` are admitted.
+ *
+ * The check is a refusal rule rather than a classifier. A sole trader trading under a business name
+ * that holds no personal name passes it. Some organizations are refused: `JOHN COCKERILL` by the first
+ * rule, and `UAB Nando` and `US Forest Service` among the 5 the casing rule refuses. The count each
+ * rule refuses is in the `dropped` map. The natural person a reader would expect first,
+ * `CORDOVA Trey Christian`, publishes no street, postcode or city at all and is refused as
+ * {@linkcode CORDISRefusal.LocalityAbsent}.
+ *
+ * The publisher's legal notice states the same limit from its side: reuse may require clearing
+ * additional rights "if a specific content depicts identifiable private individuals".
+ *
+ * ## Country codes
+ *
+ * CORDIS writes EU country codes, and those differ from ISO 3166-1 in two places: `EL` is Greece and
+ * `UK` is the United Kingdom. Both are mapped, to `GR` and `GB`. Any other value that is not an ISO
+ * alpha-2 code refuses the row: `XK` (Kosovo, a user-assigned code, 34 records), `ZZ` (1), an empty
+ * value (323), and `DE;HU` (1), two countries in one field.
+ *
+ * A country whose codex `layoutForCountry` is `null` has no rendering to write the row in.
+ * The row is refused with {@linkcode CORDISRefusal.CountryNoLayout} rather than rendered in another
+ * country's order: 353 records over 37 countries, Ghana's 77 the most.
+ *
+ * ## The street line
+ *
+ * `street` holds the whole street line as the participant typed it, usually but not always in the
+ * country's own order: `THE WELLCOME TRUST LIMITED` writes `EUSTON ROAD 215 GIBBS BUILDING`, a
+ * number-last line in a number-first country. The house number is therefore split off only on the
+ * side the country's order puts it, and {@linkcode streetOrderForCountry} reads that side from codex's
+ * `house-number-precedes-street` claim. A country whose observations disagree, or state no side,
+ * has no unambiguous side, and its lines are kept only where they hold no digit.
+ *
+ * **A street line holding a digit the split could not place refuses the row**, as `it-anac` does, and
+ * so does a line whose remainder still holds a digit once the number is split off. `C GRAN DE GRACIA
+ * NUM 1, PLANTA 4, PUERTA 3` would otherwise label `3`, the door, as the house number, and `TRIERSTRAAT
+ * 49 STRATENPLAN BUS 7` would label the box. The cost is the street whose name holds a digit, `PLACE DU
+ * 20 AOUT 7`, which is refused with the rest. These two rules refuse 14,438 and 4,646 records,
+ * the largest share of the 20,911 refused, led by France, Great Britain and Spain.
+ *
+ * A civic-number marker between the street and the number (`NR`, `NO`, `NUM`, `Nº`) is dropped, so the
+ * street is `HAUPTSTRASSE` rather than `HAUPTSTRASSE NR.`. A kilometer point, `CTRA SANT LLORENC DE
+ * MORUNYS KM, 2`, is refused rather than read as a house number.
+ *
+ * ## Placeholders and the postcode
+ *
+ * A value holding no letter and no digit (`-`, `.`), or one of `N/A`, `NA`, `none`, `Not applicable`,
+ * is treated as absent. A postcode with no digit (`EIRE`, `Clare`, `POSTFACH`) and a postcode of
+ * zeroes alone are dropped from the row rather than refusing it, and counted as `component:postcode:*`.
+ * A postcode written with its country prefix, `LV-1063` or `LT-01103`, is kept as published: Latvia
+ * and Lithuania write that prefix in their official form.
+ *
+ * A layout with no slot for a published component leaves it out, and the row is still written:
+ * the United Arab Emirates' layout has no locality line, and Japan's Latin layout none below the
+ * prefecture. Each is counted as `component:<tag>:unplaced`.
+ *
+ * ## Deduplication and identity
+ *
+ * An organizationappears once per project, across three programmes. The adapter reads each distinct
+ * published record once, keyed on the five fields that reach a row, and writes each distinct rendered
+ * address once. A record whose address changed between programmes contributes one row per version.
+ * The row id is content-addressed over the aligned components.
+ *
+ * ## License
+ *
+ * The CORDIS legal notice at `https://cordis.europa.eu/about/legal`, read 2026-10-03, states:
+ * "Unless otherwise noted (e.g. in individual copyright notices), the reuse of the editorial content on
+ * this website owned by the EU is authorized under the Creative Commons Attribution 4.0 International
+ * (CC BY 4.0) licence." The same notice records that "The Commission's reuse policy is implemented by
+ * Commission Decision 2011/833/EU of 12 December 2011 on the reuse of Commission documents", and the
+ * data.europa.eu records for `cordisfp7projects`, `cordish2020projects` and
+ * `cordis-eu-research-projects-under-horizon-europe-2021-2027` label each CSV distribution with that
+ * decision (`COM_REUSE`). CC BY requires credit and a statement that changes were made, so every row
+ * records the license and {@linkcode CORDIS_ATTRIBUTION} is the credit the model card prints.
+ *
+ * The adapter honors `opts.limit`, `opts.signal` and `opts.country`.
+ */
+import { type PathBuilderLike } from "path-ts";
+import { type SplitStreetLine } from "#adapters/street-line";
+import { type CanonicalRow, type CorpusAdapter } from "#types";
+/**
+ * Registry id for this adapter.
+ *
+ * Stamped into every row it emits, so a corpus record can be traced back to the dataset it came from.
+ */
+export declare const CORDIS_ADAPTER_ID = "cordis";
+/**
+ * The license the CORDIS legal notice states for content the EU owns.
+ */
+export declare const CORDIS_LICENSE = "CC-BY-4.0";
+/**
+ * The credit CC BY 4.0 obliges.
+ *
+ * The model card prints it beside the license and a statement that the data was changed.
+ */
+export declare const CORDIS_ATTRIBUTION = "\u00A9 European Union, CORDIS (Publications Office of the European Union)";
+/**
+ * The archive member that holds the participants in every programme's CSV zip.
+ */
+export declare const CORDIS_ORGANIZATION_MEMBER = "organization.csv";
+/**
+ * The columns the adapter indexes by the publisher's own spelling.
+ *
+ * Checked against the first record, so a renamed column raises instead of
+ * reaching every row as an empty string.
+ */
+export declare const CORDIS_REQUIRED_COLUMNS: readonly string[];
+/**
+ * The two places CORDIS's EU country codes differ from ISO 3166-1 alpha-2.
+ */
+export declare const CORDIS_COUNTRY_ALIASES: Readonly<Record<string, string>>;
+/**
+ * Why a published record did not become a row.
+ */
+export declare const CORDISRefusal: {
+    /**
+     * `country` is not an ISO 3166-1 alpha-2 code once `EL` and `UK` are mapped.
+     */
+    readonly CountryNotISO: "country-not-iso";
+    /**
+     * Codex has no layout for the country, so no rendering states where each component goes.
+     */
+    readonly CountryNoLayout: "country-no-layout";
+    /**
+     * Every word of the name is a personal name.
+     */
+    readonly NamePersonalWords: "name-personal-words";
+    /**
+     * The name takes the participant register's `SURNAME Given` casing.
+     */
+    readonly NamePersonalCasing: "name-personal-casing";
+    /**
+     * The name or short name holds a sole-trader legal form.
+     */
+    readonly NameSoleTrader: "name-sole-trader";
+    /**
+     * The record states no city.
+     */
+    readonly LocalityAbsent: "locality-absent";
+    /**
+     * The street line holds a digit and no house number could be split off on the country's side.
+     */
+    readonly StreetNumberUnplaced: "street-number-unplaced";
+    /**
+     * A house number was split off, and the street that remains still holds a digit.
+     */
+    readonly StreetDigitInRemainder: "street-digit-in-remainder";
+    /**
+     * Beside the locality, the record holds neither a street nor a postcode.
+     */
+    readonly ComponentsTooFew: "components-too-few";
+    /**
+     * The components do not render in the country's layout.
+     */
+    readonly Unrenderable: "unrenderable";
+};
+/**
+ * One of the {@link CORDISRefusal} reasons.
+ */
+export type CORDISRefusal = (typeof CORDISRefusal)[keyof typeof CORDISRefusal];
+/**
+ * What one published record yielded.
+ *
+ * `discarded` lists the `component:<tag>:<reason>` keys of values kept out of an admitted row.
+ */
+export type CORDISReading = {
+    readonly admitted: CanonicalRow;
+    readonly discarded: readonly string[];
+} | {
+    readonly refused: CORDISRefusal;
+    readonly country: string | null;
+};
+/**
+ * The columns of one `organization.csv` record the adapter reads.
+ */
+export interface CORDISOrganizationRecord {
+    organisationID?: string;
+    name?: string;
+    shortName?: string;
+    activityType?: string;
+    SME?: string;
+    vatNumber?: string;
+    street?: string;
+    postCode?: string;
+    city?: string;
+    country?: string;
+    nutsCode?: string;
+}
+/**
+ * The given names and surnames the personal-name rules consult, lower-cased.
+ */
+export interface PersonNameLexicon {
+    readonly givenNames: ReadonlySet<string>;
+    readonly surnames: ReadonlySet<string>;
+}
+/**
+ * Load libpostal's language-independent given-name and surname dictionaries.
+ */
+export declare function loadPersonNameLexicon(): Promise<PersonNameLexicon>;
+/**
+ * Which side of the street name a country writes the house number on.
+ */
+export declare const StreetOrder: {
+    readonly NumberFirst: "number-first";
+    readonly NumberLast: "number-last";
+};
+/**
+ * One of the {@link StreetOrder} values.
+ */
+export type StreetOrder = (typeof StreetOrder)[keyof typeof StreetOrder];
+/**
+ * The side a country writes its house number on, or `null` where codex's sources disagree or state none.
+ *
+ * Read from the `house-number-precedes-street` claim, which collects libaddressinput's `fmt` and `lfmt`,
+ * the OpenCage street order, the board's hand-authored layout and a retrieved UPU S42 template.
+ * A side is returned only when at least one source states it and none states the other.
+ *
+ * Hong Kong's Chinese-script `fmt` writes the number after the street
+ * and its Latin `lfmt` before it, so Hong Kong has no side.
+ */
+export declare function streetOrderForCountry(country: string): StreetOrder | null;
+/**
+ * Whether a published value is a placeholder for an absent one, or holds no letter and no digit.
+ */
+export declare function isCORDISPlaceholder(value: string): boolean;
+/**
+ * The personal-name form a participant's name takes, or `null` where it takes none.
+ *
+ * Exported so a test and a measurement can read the rule the adapter applies rather than a copy of it.
+ */
+export declare function naturalPersonForm(name: string, shortName: string, lexicon: PersonNameLexicon): CORDISRefusal | null;
+/**
+ * The street line split on the side the country writes its number, or the refusal the line meets.
+ *
+ * `order` is `null` for a country with no unambiguous side, and such a line
+ * is kept only where it holds no digit.
+ */
+export declare function splitCORDISStreetLine(line: string, order: StreetOrder | null): SplitStreetLine | {
+    refused: CORDISRefusal;
+} | null;
+/**
+ * The ISO 3166-1 alpha-2 code for a CORDIS `country` value, or `null` for one that maps to none.
+ */
+export declare function cordisCountryCode(value: string | null | undefined): string | null;
+/**
+ * Read one published record into a row, or report why it yielded none.
+ *
+ * Exported because the refusal reasons are the measurement this adapter is checked by.
+ */
+export declare function readCORDISRecord(record: CORDISOrganizationRecord, lexicon: PersonNameLexicon): CORDISReading;
+/**
+ * One distinct published record and what it yielded.
+ */
+export interface CORDISRecordReading {
+    readonly record: CORDISOrganizationRecord;
+    readonly reading: CORDISReading;
+}
+/**
+ * Every distinct published record under `inputPath`, read once each.
+ *
+ * A record is distinct on the five fields that reach a row, so an organization that took part in many
+ * projects with one address is read once, and one whose address changed is read once per version.
+ * `onDuplicate` is called once for every participation row a record already read covers.
+ *
+ * The adapter and the measurement both read through this, so the counts a
+ * measurement reports are the adapter's own.
+ */
+export declare function readCORDISRecords(inputPath: PathBuilderLike, lexicon: PersonNameLexicon, options?: {
+    signal?: AbortSignal;
+    onDuplicate?: () => void;
+}): AsyncGenerator<CORDISRecordReading>;
+export declare function createCORDISAdapter(): CorpusAdapter;
+/**
+ * The configured adapter instance registered with the corpus builder.
+ */
+export declare const cordisAdapter: CorpusAdapter;
+//# sourceMappingURL=adapter.d.ts.map

package/out/adapters/cordis/adapter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../lib/adapters/cordis/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2HG;AAQH,OAAO,EAAe,KAAK,eAAe,EAAE,MAAM,SAAS,CAAA;AAM3D,OAAO,EAA4C,KAAK,eAAe,EAAE,MAAM,uBAAuB,CAAA;AAEtG,OAAO,EAGN,KAAK,YAAY,EACjB,KAAK,aAAa,EAGlB,MAAM,QAAQ,CAAA;AAEf;;;;GAIG;AACH,eAAO,MAAM,iBAAiB,WAAW,CAAA;AAEzC;;GAEG;AACH,eAAO,MAAM,cAAc,cAAc,CAAA;AAEzC;;;;GAIG;AACH,eAAO,MAAM,kBAAkB,8EAAyE,CAAA;AAExG;;GAEG;AACH,eAAO,MAAM,0BAA0B,qBAAqB,CAAA;AAE5D;;;;;GAKG;AACH,eAAO,MAAM,uBAAuB,EAAE,SAAS,MAAM,EASpD,CAAA;AAED;;GAEG;AACH,eAAO,MAAM,sBAAsB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAGnE,CAAA;AAID;;GAEG;AACH,eAAO,MAAM,aAAa;IACzB;;OAEG;aACH,aAAa,EAAE,iBAAiB;IAChC;;OAEG;aACH,eAAe,EAAE,mBAAmB;IACpC;;OAEG;aACH,iBAAiB,EAAE,qBAAqB;IACxC;;OAEG;aACH,kBAAkB,EAAE,sBAAsB;IAC1C;;OAEG;aACH,cAAc,EAAE,kBAAkB;IAClC;;OAEG;aACH,cAAc,EAAE,iBAAiB;IACjC;;OAEG;aACH,oBAAoB,EAAE,wBAAwB;IAC9C;;OAEG;aACH,sBAAsB,EAAE,2BAA2B;IACnD;;OAEG;aACH,gBAAgB,EAAE,oBAAoB;IACtC;;OAEG;aACH,YAAY,EAAE,cAAc;CACnB,CAAA;AAEV;;GAEG;AACH,MAAM,MAAM,aAAa,GAAG,CAAC,OAAO,aAAa,CAAC,CAAC,MAAM,OAAO,aAAa,CAAC,CAAA;AAE9E;;;;GAIG;AACH,MAAM,MAAM,aAAa,GACtB;IAAE,QAAQ,CAAC,QAAQ,EAAE,YAAY,CAAC;IAAC,QAAQ,CAAC,SAAS,EAAE,SAAS,MAAM,EAAE,CAAA;CAAE,GAC1E;IAAE,QAAQ,CAAC,OAAO,EAAE,aAAa,CAAC;IAAC,QAAQ,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAA;CAAE,CAAA;AAEvE;;GAEG;AACH,MAAM,WAAW,wBAAwB;IACxC,cAAc,CAAC,EAAE,MAAM,CAAA;IACvB,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAA;CACjB;AAED;;GAEG;AACH,MAAM,WAAW,iBAAiB;IACjC,QAAQ,CAAC,UAAU,EAAE,WAAW,CAAC,MAAM,CAAC,CAAA;IACxC,QAAQ,CAAC,QAAQ,EAAE,WAAW,CAAC,MAAM,CAAC,CAAA;CACtC;AAED;;GAEG;AACH,wBAAsB,qBAAqB,IAAI,OAAO,CAAC,iBAAiB,CAAC,CAOxE;AAED;;GAEG;AACH,eAAO,MAAM,WAAW;aACvB,WAAW,EAAE,cAAc;aAC3B,UAAU,EAAE,aAAa;CAChB,CAAA;AAEV;;GAEG;AACH,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,WAAW,CAAC,CAAC,MAAM,OAAO,WAAW,CAAC,CAAA;AAIxE;;;;;;;;;GASG;AACH,wBAAgB,qBAAqB,CAAC,OAAO,EAAE,MAAM,GAAG,WAAW,GAAG,IAAI,CAqBzE;AAOD;;GAEG;AACH,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAM1D;AA+DD;;;;GAIG;AACH,wBAAgB,iBAAiB,CAAC,IAAI,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,OAAO,EAAE,iBAAiB,GAAG,aAAa,GAAG,IAAI,CAgCnH;AAkBD;;;;;GAKG;AACH,wBAAgB,qBAAqB,CACpC,IAAI,EAAE,MAAM,EACZ,KAAK,EAAE,WAAW,GAAG,IAAI,GACvB,eAAe,GAAG;IAAE,OAAO,EAAE,aAAa,CAAA;CAAE,GAAG,IAAI,CA8BrD;AAED;;GAEG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,GAAG,MAAM,GAAG,IAAI,CAKjF;AAED;;;;GAIG;AACH,wBAAgB,gBAAgB,CAAC,MAAM,EAAE,wBAAwB,EAAE,OAAO,EAAE,iBAAiB,GAAG,aAAa,CAoF5G;AAED;;GAEG;AACH,MAAM,WAAW,mBAAmB;IACnC,QAAQ,CAAC,MAAM,EAAE,wBAAwB,CAAA;IACzC,QAAQ,CAAC,OAAO,EAAE,aAAa,CAAA;CAC/B;AAED;;;;;;;;;GASG;AACH,wBAAuB,iBAAiB,CACvC,SAAS,EAAE,eAAe,EAC1B,OAAO,EAAE,iBAAiB,EAC1B,OAAO,GAAE;IAAE,MAAM,CAAC,EAAE,WAAW,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,IAAI,CAAA;CAAO,GAC9D,cAAc,CAAC,mBAAmB,CAAC,CAoCrC;AA4CD,wBAAgB,mBAAmB,IAAI,aAAa,CA2DnD;AAED;;GAEG;AACH,eAAO,MAAM,aAAa,eAAwB,CAAA"}