@finbheara/archive-client 0.40.0 → 0.43.0

package/README.md

@@ -94,6 +94,8 @@ c.features;   // [{ feature: "dancers", state: "full" | "missing", missing: [...
 | `dancers.records.names` | the `names_printed` a record of `dancers.records` carries; absent, never refused, without it | 215 |
 | `dancers.years` | `dancers.years` | 245 |
 | `results.stream.competitors_printed`, `majors.stream.competitors_printed` | the `competitors_printed` a streamed row carries (the competed count the page printed); absent, never refused, without it | 345 |
+| `results.attach_basis`, `majors.attach_basis` | the `attach_basis` a row of `results.query`, `majors.results`, `results.stream` and `results.entries` carries (how its `dancer_id` was answered, or why not); absent, never refused, without it | 345 |
+| `dancers.records.birth_consistency`, `dancers.years.birth_consistency` | the `birth_consistency` (0-100, floored: the percent of the dancer's, or the dancer-year's, readable rows that fit the birth window) and its denominator `birth_consistency_rows` a record of `dancers.records` and a dancer-year of `dancers.years` carry; absent, never refused, without it | 345 |
 | `dancers.years.printed` | the `printed_regions` a dancer-year of `dancers.years` carries (every region the year printed, where `region` picks one); absent, never refused, without it | 341 |
 | `majors.tabs` | `majors.tabulated` | 215 |
 | `teams` | `teams.results`, `teams.rosters`, `teams.resultRosters` | 260 |
@@ -146,6 +148,48 @@ A file already in place with no receipt (laid out by hand) is adopted where the
 vouches for it, and fetched again where it does not. A published release never changes, so a
 verified file is never fetched twice.
 
+EACH DOWNLOAD WRITES A TEMP FILE OF ITS OWN beside its destination,
+`<root>/<tag>/.<file>.<pid>-<hex>.part`, and renames it into place only once both its hashes
+checked. A failure or a cancel leaves only what was there before, and two processes fetching one
+release into one cache each land a whole, verified file (the last rename wins, the same bytes). A
+temp file whose process is no longer running (a killed fetch's, or an index's
+`<name>.sqlite.building.<pid>`) is removed by the next fetch or build in its directory.
+
+## Downloads: concurrent, streamed, reported
+
+`rel.fetch(...assets)` (and so `openRelease`, `fetch`, `index`, `use` and `update`) downloads up
+to `concurrency` files at once (the option; default `DEFAULT_CONCURRENCY`, 4), and each one is
+hashed, decompressed and hashed again WHILE its bytes arrive, as a container pull downloads and
+extracts at once: a file is verified when its last byte lands. `use` (and `index`) builds each
+index as soon as its own inputs have landed, while the rest still download. The first file that
+fails cancels the others, waits for them to stop, and fails the call with its own error, naming
+the file; `use` then moves no pin. `rel.fetching(assets)` answers at once with each asset's own
+promise (`each(asset)`), the whole (`all`) and `cancel()`.
+
+A `Remote` streams where it has `bytes(repo, tag, name, signal)` (`gh release download -O -`, or
+the HTTPS body); one without it is read through `download`, to a temp file, then decompressed.
+
+The `progress` option is told every file's and every index's state and bytes
+(`ProgressEvent`: `downloading`, `unpacking`, `verifying`, `indexing`, then `done`, `failed` or
+`cancelled`), and `progressRenderer(mode, { line })` prints them in one of three modes:
+
+- `tty`: one line per file and per index, redrawn in place: name, bytes of the total, unpacked
+  bytes, rate and state
+- `plain`: one `start` line when a file (or an index) starts and one `done` (or `failed`,
+  `cancelled`) line when it ends, with its size and time; no control character, nothing else
+- `none`: nothing
+
+The command line picks `tty` where stderr is a terminal and `plain` where it is not (redirected, a
+CI log, an agent's capture). `--progress tty|plain|none`, or `IDQ_PROGRESS`, forces a mode. A
+cache already current fetches nothing and prints nothing:
+
+    start   <tag> results.sqlite.zst (179.0 MB)
+    start   <tag> school-registry.jsonl (272.5 kB)
+    done    <tag> school-registry.jsonl 272.5 kB in 0.9 s
+    done    <tag> results.sqlite.zst 179.0 MB -> 1.42 GB in 9.8 s
+    start   <tag> index schools
+    done    <tag> index schools in 12.3 s
+
 ## The release manager
 
 ```ts
@@ -604,6 +648,8 @@ A release without them streams every row with the six fields ABSENT, never null.
 
 `competitors_printed` (feature `results.stream.competitors_printed`, `majors.stream.competitors_printed` for scope `majors`) is the competed count the page itself printed, null where it printed none. `competitors` stays the best value. A release without it streams the field ABSENT.
 
+`attach_basis` (feature `results.attach_basis`, `majors.attach_basis` for scope `majors` and `rel.majors.results`) says how a row's `dancer_id` was answered, or why it was not, in one integer: `attach_basis & 63` is the feis attach's outcome for the row's address (`0` on a majors row, `1` a platform row it carded no card for, else attached how or the gate that refused it), `attach_basis & ~63` the bits of where the id came from (none where `dancer_id` is null). `unpackAttachBasis(n)` decodes it, `{ outcome: "birth-window", via: ["feis-local"] }`, off the register `ATTACH_OUTCOMES` and `ATTACH_VIA` the package exports, and the release's `DATA-DICTIONARY.md` says what each value means. `rel.results.query` and `rel.majors.results` rows carry it too. A release without it returns the field ABSENT.
+
 THE COMPETITION'S KEY is `competition` (`<event-year>.<comp>` at a major, `<platform>:<host>.<comp>`
 at a feis), which holds across releases. Inside one release `(file, competition_line)` is exact: a
 competition captured twice (two files of one platform) carries one `competition` in both and two
@@ -623,10 +669,14 @@ the place the address's recall row carries.
 ```ts
 await rel.dancers.records();                   // every dancer: display, appearances, birth_lo, birth_hi,
                                                // birth_source, birth_conflict, genders, scope,
-                                               // names_printed (dancers.records.names)
+                                               // names_printed (dancers.records.names),
+                                               // birth_consistency, birth_consistency_rows
+                                               // (dancers.records.birth_consistency)
 await rel.dancers.records({ ids: [1001] });
 await rel.dancers.years({ ids: [1001], year: 2025 });  // region, region_source, nationals, geohash
-                                               // printed_regions (dancers.years.printed)
+                                               // printed_regions (dancers.years.printed),
+                                               // birth_consistency, birth_consistency_rows
+                                               // (dancers.years.birth_consistency)
 await rel.feises.dates();                      // every feis id with its first and last day
 await rel.majors.tabulated({ min_true_round: 3 });     // the majors addresses marked to round 3 or later
 ```
@@ -965,6 +1015,9 @@ is passed to the dancer resolver as asked (`rel.dancers`, above).
 
 ## The command line
 
+Every verb takes `--cache <dir>`, `--repo <owner/name>`, `--json`, `--live` and
+`--progress tty|plain|none` (above).
+
     idq config repo <owner/name>
     idq config keep <N>
     idq list [--remote]