@finbheara/archive-client 0.39.0 → 0.42.0

package/README.md

@@ -93,6 +93,8 @@ c.features;   // [{ feature: "dancers", state: "full" | "missing", missing: [...
 | `dancers.records` | `dancers.records` | 215 |
 | `dancers.records.names` | the `names_printed` a record of `dancers.records` carries; absent, never refused, without it | 215 |
 | `dancers.years` | `dancers.years` | 245 |
+| `results.stream.competitors_printed`, `majors.stream.competitors_printed` | the `competitors_printed` a streamed row carries (the competed count the page printed); absent, never refused, without it | 345 |
+| `results.attach_basis`, `majors.attach_basis` | the `attach_basis` a row of `results.query`, `majors.results`, `results.stream` and `results.entries` carries (how its `dancer_id` was answered, or why not); absent, never refused, without it | 345 |
 | `dancers.years.printed` | the `printed_regions` a dancer-year of `dancers.years` carries (every region the year printed, where `region` picks one); absent, never refused, without it | 341 |
 | `majors.tabs` | `majors.tabulated` | 215 |
 | `teams` | `teams.results`, `teams.rosters`, `teams.resultRosters` | 260 |
@@ -145,6 +147,48 @@ A file already in place with no receipt (laid out by hand) is adopted where the
 vouches for it, and fetched again where it does not. A published release never changes, so a
 verified file is never fetched twice.
 
+EACH DOWNLOAD WRITES A TEMP FILE OF ITS OWN beside its destination,
+`<root>/<tag>/.<file>.<pid>-<hex>.part`, and renames it into place only once both its hashes
+checked. A failure or a cancel leaves only what was there before, and two processes fetching one
+release into one cache each land a whole, verified file (the last rename wins, the same bytes). A
+temp file whose process is no longer running (a killed fetch's, or an index's
+`<name>.sqlite.building.<pid>`) is removed by the next fetch or build in its directory.
+
+## Downloads: concurrent, streamed, reported
+
+`rel.fetch(...assets)` (and so `openRelease`, `fetch`, `index`, `use` and `update`) downloads up
+to `concurrency` files at once (the option; default `DEFAULT_CONCURRENCY`, 4), and each one is
+hashed, decompressed and hashed again WHILE its bytes arrive, as a container pull downloads and
+extracts at once: a file is verified when its last byte lands. `use` (and `index`) builds each
+index as soon as its own inputs have landed, while the rest still download. The first file that
+fails cancels the others, waits for them to stop, and fails the call with its own error, naming
+the file; `use` then moves no pin. `rel.fetching(assets)` answers at once with each asset's own
+promise (`each(asset)`), the whole (`all`) and `cancel()`.
+
+A `Remote` streams where it has `bytes(repo, tag, name, signal)` (`gh release download -O -`, or
+the HTTPS body); one without it is read through `download`, to a temp file, then decompressed.
+
+The `progress` option is told every file's and every index's state and bytes
+(`ProgressEvent`: `downloading`, `unpacking`, `verifying`, `indexing`, then `done`, `failed` or
+`cancelled`), and `progressRenderer(mode, { line })` prints them in one of three modes:
+
+- `tty`: one line per file and per index, redrawn in place: name, bytes of the total, unpacked
+  bytes, rate and state
+- `plain`: one `start` line when a file (or an index) starts and one `done` (or `failed`,
+  `cancelled`) line when it ends, with its size and time; no control character, nothing else
+- `none`: nothing
+
+The command line picks `tty` where stderr is a terminal and `plain` where it is not (redirected, a
+CI log, an agent's capture). `--progress tty|plain|none`, or `IDQ_PROGRESS`, forces a mode. A
+cache already current fetches nothing and prints nothing:
+
+    start   <tag> results.sqlite.zst (179.0 MB)
+    start   <tag> school-registry.jsonl (272.5 kB)
+    done    <tag> school-registry.jsonl 272.5 kB in 0.9 s
+    done    <tag> results.sqlite.zst 179.0 MB -> 1.42 GB in 9.8 s
+    start   <tag> index schools
+    done    <tag> index schools in 12.3 s
+
 ## The release manager
 
 ```ts
@@ -597,10 +641,14 @@ THE COMPETITION FIELDS, where the release carries them (`rel.capability("results
 | `dance` | the competition's one dance as the release reads it (`reel`), null where it reads none |
 | `recalled_only` | true where the page printed the recalled dancers only |
 | `comp_flags` | what the page printed about itself, int-packed, one bit a flag (bit 10: recalled only). The release's `DATA-DICTIONARY.md` carries the register of every bit, and a published bit never moves |
-| `competitors_basis` | what `competitors` stands on: `printed` (the page printed a competed count), `counted` (the page printed the whole field, its cards counted), `entered` or `registered` (only the top was printed: the page's entered or registered count), null where there is no number |
+| `competitors_basis` | what `competitors` stands on: `printed` (the page printed a competed count), `counted` (the page printed the whole field, its cards counted), `entered` or `registered` (only the top was printed: the page's entered or registered count), `estimated` (a majors sheet with no count that does not show its whole field: the recall rule over a recall-only sheet (twice the recalled, plus or minus 1, or the recalled where fewer than 10), else the dancers it printed, a FLOOR; never a count, so leave it out where a real count is needed), null where there is no number |
 
 A release without them streams every row with the six fields ABSENT, never null.
 
+`competitors_printed` (feature `results.stream.competitors_printed`, `majors.stream.competitors_printed` for scope `majors`) is the competed count the page itself printed, null where it printed none. `competitors` stays the best value. A release without it streams the field ABSENT.
+
+`attach_basis` (feature `results.attach_basis`, `majors.attach_basis` for scope `majors` and `rel.majors.results`) says how a row's `dancer_id` was answered, or why it was not, in one integer: `attach_basis & 63` is the feis attach's outcome for the row's address (`0` on a majors row, `1` a platform row it carded no card for, else attached how or the gate that refused it), `attach_basis & ~63` the bits of where the id came from (none where `dancer_id` is null). `unpackAttachBasis(n)` decodes it, `{ outcome: "birth-window", via: ["feis-local"] }`, off the register `ATTACH_OUTCOMES` and `ATTACH_VIA` the package exports, and the release's `DATA-DICTIONARY.md` says what each value means. `rel.results.query` and `rel.majors.results` rows carry it too. A release without it returns the field ABSENT.
+
 THE COMPETITION'S KEY is `competition` (`<event-year>.<comp>` at a major, `<platform>:<host>.<comp>`
 at a feis), which holds across releases. Inside one release `(file, competition_line)` is exact: a
 competition captured twice (two files of one platform) carries one `competition` in both and two
@@ -962,6 +1010,9 @@ is passed to the dancer resolver as asked (`rel.dancers`, above).
 
 ## The command line
 
+Every verb takes `--cache <dir>`, `--repo <owner/name>`, `--json`, `--live` and
+`--progress tty|plain|none` (above).
+
     idq config repo <owner/name>
     idq config keep <N>
     idq list [--remote]