TSVZ 5.1__tar.gz → 5.2__tar.gz

{tsvz-5.1 → tsvz-5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 5.1
+Version: 5.2
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/tsvz-org/TSVZ
 Author: Yufei Pan
@@ -163,6 +163,20 @@ named like an operation then needs a path prefix (`./read`).
   with a warning.
 - **TSVZ extensions:**
   - `--x-header H` and `--x-defaults KEY<d>D1<d>...` decode backslash escapes such as `\t`;
+  - `--x-header H` makes line 1 of a `.tsv`-family file its header for every operation
+    when line 1 is H: the same number of cells, each equal to H's once the whitespace
+    around it is taken off (a trailing empty cell counts: `id\tv\t` is not `id\tv`; a UTF-8
+    byte order mark before line 1, as Excel writes one, is skipped). It
+    is then never a key, row or count, `scrub` and `clear` keep it, and
+    `read --format table` shows it above the rows. When line 1 differs, the file has no
+    header and line 1 is a row, with 3.39's "Header mismatch" warning (`--x-strict`: an
+    error, exit 1, nothing written). A missing or empty file gets H first. Without
+    `--x-header`, line 1 is a row, as in 3.39. (The library's own header check, a
+    `header=` given to `readTabularFile`, `appendTabularFile`, `clearTabularFile`,
+    `scrubTabularFile` or `TSVZed`, is 3.39's, looser one: each cell of H within the
+    cell of line 1 under it, so `idaho\tvalue` passes for `id\tv`.) A rewrite without
+    `--x-header` can put a row equal to H on line 1
+    ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
   - `--x-strict` / `--x-force` turn 3.39's column and header checks on or off.
 
 ## Serving a store
@@ -193,7 +207,13 @@ status stay the same.
   and `fsync`s everything and removes `STORE.serve`.
 - **Bypassing it.** `--x-direct` makes a `tsvz` command use the files. Command
   lines with `--x-header`, `--x-defaults` or `--x-strict`, or another delimiter
-  than the server's, use the files too.
+  than the server's, use the files too. A server started with `--x-header H`
+  answers every request by the `--x-header` rule above, and one started with
+  `--x-strict` too answers as a direct `--x-strict` command does: a line 1 that is
+  not H fails the request (status 1, nothing written), and reads and scrubs drop
+  the rows of another width. It does not start when line 1 is not H then. The
+  table of a routed `read --format table` shows no H line: its first row is set
+  as the table's header, as with no `--x-header`.
 
 ```python
 import TSVZ
@@ -261,7 +281,9 @@ microseconds as a write starts is followed as a rotation instead); while that re
 fails or is refused, the rows are appended to a new file there, and memory keeps all of
 them for the rewrite that gets through. A compressed file another writer left damaged
 is made again too (L5: every read stops at the damage, so rows
-appended after it would be read by no one); the bytes after the damage go. While that
+appended after it would be read by no one): what the read did not get goes, the bytes after
+the damage and also the lines of the damaged member before it that the codec never handed
+over (Python's gzip gives none of a first member cut short, where `zcat` shows them). While that
 rewrite fails, the rows are appended after the damage and memory keeps them for the
 next rewrite that gets through; while it is refused (a row changed in place to text
 the encoding cannot hold), they are appended after it as in 4.9, and the next read
@@ -301,6 +323,106 @@ empty; the keys it sets or deletes then take its values, and the keys it left ou
 memory's. An explicit `reload()` while the file is damaged replaces memory with the
 damaged file, as asked, and warns.
 
+## Upgrading to 5.2 from 5.1
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.2 writes.
+
+- **A scrub of a `.tsv` whose rows were all deleted shrinks it.** `scrubTabularFile`,
+  `tsvz scrub` (also through a `tsvz serve` handler) of a `.tsv`-family file (`.tsv`,
+  `.csv`, `.nsv`, `.psv` and their compressed forms) in which every key has been deleted
+  leaves the file as an empty store: the header line it has, and its `#_defaults_#`
+  lines as a scrub of a file with rows keeps them (an empty one after the header, the
+  last non-empty one after it, both as wide as the header, or as the column count the
+  read took, which caller defaults set when given). A file with no header line keeps its
+  empty first line, or becomes 0 bytes when its first line was a row; a first line that
+  is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an
+  empty first line, which the scrub then writes. Comment lines go, as a scrub of a file
+  with rows already drops them. A scrub that is given
+  a header (and checks it, the default) keeps the file's first line as its header, an
+  empty one included, with a "Header mismatch" warning when it differs; the header it is
+  given is written only with `verifyHeader=False`, or when the file's first line cannot be
+  decoded (plain text named `.gz`); `tsvz scrub --x-header` follows the `--x-header` rule
+  below instead. A scrub of a path that is not there
+  makes no file and returns `{}`, as in 5.1, unless `createIfNotExist=True`. A compressed
+  file whose read decodes no row (plain text named `.gz`, a first member cut short) is
+  made again as an empty store, as a scrub of a damaged file with rows already was in
+  5.1: what the read did not get goes, the bytes after the damage and also the lines of the
+  damaged member before it that the codec never handed over (Python's gzip gives none of
+  a first member cut short, where `zcat` shows them) (see the damaged-stream notes above), and
+  `tsvz clear --x-header H --x-strict` (`clearTabularFile(verifyHeader=True)`) now clears
+  such a file, where 5.1 exited 1 with the codec's error.
+  5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that
+  appended keys, deleted them all and scrubbed, over and over, saw the file grow without
+  limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after
+  every scrub. A file with no byte (or a compressed one with only empty streams) is
+  still left alone, and so is a file whose strict scrub (`strict=True`, `--x-strict`) read
+  dropped a line for its width and found no row: that line is not a deletion, as 5.1 left
+  it. A scrub that finds the file changed since its read still reads
+  again and writes nothing if it keeps changing, as for a file with rows.
+- **`get_file_obj('ab')` on a compressed store writes its trailer before the lock is let
+  go.** A program that calls it on a `.tsv.gz`, `.bz2`, `.xz` or `.zst` store and writes
+  through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included,
+  in the file when `release_file_obj()` unlocks it. 5.1 and earlier wrote them after the
+  unlock, where another writer could append in between and corrupt the stream.
+- **A `TSVZed` no longer lets a file grow with every session of sets and deletes.** It
+  rewrites a `.tsv`-family file at open when the file holds a key's row and the deletion
+  of it, as 3.39 did only for a key with two values, so a program that sets keys and
+  deletes them in each session (default settings, or `rewrite_interval`) keeps the file
+  the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session
+  that opens such a file now rewrites it once. With `rewrite_on_load=False` the open
+  does not, and the next exit or periodic rewrite does it (`rewrite_on_exit=True`,
+  `rewrite_interval`). What such a rewrite does, it now does at a plain open:
+  - it cuts a row wider than the header, and the cut stays
+    ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)):
+    `id\tv\na\t1\t2\nb\t1\nb\t\n` becomes `id\tv\na\t1  \n`, and
+    `id\tv\na\t\t7\nb\t1\nb\t\n` becomes `id\tv\na\t  \n`, the 7 lost;
+  - it writes over the file in place and pads with spaces (`a\t1  ` above): TSVZ readers
+    strip them, tools that read the raw bytes (`awk`, `cut`) see them;
+  - on a file it cannot write (read-only), it prints `Failed to open file … Permission
+    denied` twice and leaves the file as it is, as 5.1 did for a key with two values.
+- **`--x-header H` makes line 1 the header for every `tsvz` operation.** On a
+  `.tsv`-family file whose line 1 is H (an exact match: the same cells, each without the
+  whitespace around it, a trailing empty cell included), line 1 is the header: `keys`,
+  `len`, `read`, `get`, `has`, `pop`, `popitem` and `delete` never list, count, print,
+  pop or delete it, and `scrub` and `clear` keep it. When line 1 differs, the file has no
+  header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and
+  the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases
+  below), and in such a file it never writes a row that is H as line 1: it puts an empty
+  line first (`a\t1\nid\tv\na\t\n` becomes `\nid\tv\n`), so the reads still take that row
+  for a row. With `--x-strict` a line 1 that is not H is an error (exit 1, nothing
+  written) for every operation, through a handler too. A missing or empty file gets H
+  first at its first write, as in 5.1. A `tsvz serve STORE --x-header H` handler applies
+  the rule to every request it answers, routed `tsvz` commands included; with
+  `--x-strict` it is strict for them as above (5.1 ignored `--x-strict` at `serve`): such
+  a handler refuses to start on a file whose line 1 is not H, its routed `.tsv` reads and
+  scrubs drop lines of another column count, and its `.tsvz` reads are strict, as for a
+  direct `--x-strict` command. The library's own header check (`header=` to
+  `readTabularFile`, `appendTabularFile`, `clearTabularFile`, `scrubTabularFile`,
+  `TSVZed`) is unchanged, 3.39's containment check. The CLI's writes check line 1 more
+  strictly than 5.1's did: 5.1's `set`, `append`, `delete` and `clear` with `--x-header`
+  silently took a line 1 that merely held H's cells (`id\tvalue` for `id\tv`) for the
+  header and fitted rows to H's width; now they warn and write the row at its own width,
+  and with `--x-strict` exit 1. A rewrite without `--x-header` (a scrub without it,
+  `scrubTabularFile`, a `TSVZed` rewrite) can still put a row equal to H on line 1 of a
+  header-less file
+  ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
+  In 5.1 only `set`, `append`, `delete` and `clear` used `--x-header`: the reads and
+  `scrub` took line 1 as a row (`tsvz keys`
+  printed `id a b` for two keys, `len` said 3, `popitem first` popped the header), and
+  so did every request to a handler. Without `--x-header` nothing changed: line 1 is a
+  row, as in 5.1 and 3.39. In a `.tsvz` store `pop`, `popitem` and `setdefault` now check
+  `--x-header` as `set` does; nothing else changed there.
+- **Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row,
+  and can lose cells or keys.** A scrub given defaults narrower than the rows, a row wider
+  than the header, and a `.tsv` that begins with a bare `#_defaults_#` line (read without
+  defaults) are examples; the entry lists each, and which is new in 5.2
+  ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)).
+  A strict scrub of a file whose read dropped a line for its width and found no row leaves it,
+  so it grows again over append, delete-all and scrub cycles
+  ([limitations](docs/limitations.md#a-strict-scrub-leaves-a-file-whose-read-dropped-a-line-for-its-width)).
+  Both wait for the column-width design after 5.2.
+
 ## Upgrading to 5.1 from 5.0
 
 The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
@@ -357,12 +479,11 @@ The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family
   file's own header line.)
   `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
   a program asks for.
-- **Known: a scrub of a `.tsv` whose rows were all deleted does not compact it** when
-  the scrub is given the file's header (`scrubTabularFile(path, header=H)`) or the file
-  has no header line: it leaves the file as it is, deletion lines included; reads are
-  right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file reads the header line as a
-  row, so it does compact it.)
-  ([limitations](docs/limitations.md#a-scrub-of-a-tsv-whose-rows-were-all-deleted-does-not-compact-it)).
+- **Known in 5.1, fixed in 5.2: a scrub of a `.tsv` whose rows were all deleted did not
+  compact it** when the scrub was given the file's header (`scrubTabularFile(path, header=H)`)
+  or the file had no header line: it left the file as it was, deletion lines included;
+  reads were right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file read the header
+  line as a row, so it did compact it.)
 - **Known: a scrub run with other defaults than the store's program can change how
   early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
   cells before any defaults line, the reset line the scrub writes before the rows keeps

{tsvz-5.1 → tsvz-5.2}/README.md

@@ -131,6 +131,20 @@ named like an operation then needs a path prefix (`./read`).
   with a warning.
 - **TSVZ extensions:**
   - `--x-header H` and `--x-defaults KEY<d>D1<d>...` decode backslash escapes such as `\t`;
+  - `--x-header H` makes line 1 of a `.tsv`-family file its header for every operation
+    when line 1 is H: the same number of cells, each equal to H's once the whitespace
+    around it is taken off (a trailing empty cell counts: `id\tv\t` is not `id\tv`; a UTF-8
+    byte order mark before line 1, as Excel writes one, is skipped). It
+    is then never a key, row or count, `scrub` and `clear` keep it, and
+    `read --format table` shows it above the rows. When line 1 differs, the file has no
+    header and line 1 is a row, with 3.39's "Header mismatch" warning (`--x-strict`: an
+    error, exit 1, nothing written). A missing or empty file gets H first. Without
+    `--x-header`, line 1 is a row, as in 3.39. (The library's own header check, a
+    `header=` given to `readTabularFile`, `appendTabularFile`, `clearTabularFile`,
+    `scrubTabularFile` or `TSVZed`, is 3.39's, looser one: each cell of H within the
+    cell of line 1 under it, so `idaho\tvalue` passes for `id\tv`.) A rewrite without
+    `--x-header` can put a row equal to H on line 1
+    ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
   - `--x-strict` / `--x-force` turn 3.39's column and header checks on or off.
 
 ## Serving a store
@@ -161,7 +175,13 @@ status stay the same.
   and `fsync`s everything and removes `STORE.serve`.
 - **Bypassing it.** `--x-direct` makes a `tsvz` command use the files. Command
   lines with `--x-header`, `--x-defaults` or `--x-strict`, or another delimiter
-  than the server's, use the files too.
+  than the server's, use the files too. A server started with `--x-header H`
+  answers every request by the `--x-header` rule above, and one started with
+  `--x-strict` too answers as a direct `--x-strict` command does: a line 1 that is
+  not H fails the request (status 1, nothing written), and reads and scrubs drop
+  the rows of another width. It does not start when line 1 is not H then. The
+  table of a routed `read --format table` shows no H line: its first row is set
+  as the table's header, as with no `--x-header`.
 
 ```python
 import TSVZ
@@ -229,7 +249,9 @@ microseconds as a write starts is followed as a rotation instead); while that re
 fails or is refused, the rows are appended to a new file there, and memory keeps all of
 them for the rewrite that gets through. A compressed file another writer left damaged
 is made again too (L5: every read stops at the damage, so rows
-appended after it would be read by no one); the bytes after the damage go. While that
+appended after it would be read by no one): what the read did not get goes, the bytes after
+the damage and also the lines of the damaged member before it that the codec never handed
+over (Python's gzip gives none of a first member cut short, where `zcat` shows them). While that
 rewrite fails, the rows are appended after the damage and memory keeps them for the
 next rewrite that gets through; while it is refused (a row changed in place to text
 the encoding cannot hold), they are appended after it as in 4.9, and the next read
@@ -269,6 +291,106 @@ empty; the keys it sets or deletes then take its values, and the keys it left ou
 memory's. An explicit `reload()` while the file is damaged replaces memory with the
 damaged file, as asked, and warns.
 
+## Upgrading to 5.2 from 5.1
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.2 writes.
+
+- **A scrub of a `.tsv` whose rows were all deleted shrinks it.** `scrubTabularFile`,
+  `tsvz scrub` (also through a `tsvz serve` handler) of a `.tsv`-family file (`.tsv`,
+  `.csv`, `.nsv`, `.psv` and their compressed forms) in which every key has been deleted
+  leaves the file as an empty store: the header line it has, and its `#_defaults_#`
+  lines as a scrub of a file with rows keeps them (an empty one after the header, the
+  last non-empty one after it, both as wide as the header, or as the column count the
+  read took, which caller defaults set when given). A file with no header line keeps its
+  empty first line, or becomes 0 bytes when its first line was a row; a first line that
+  is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an
+  empty first line, which the scrub then writes. Comment lines go, as a scrub of a file
+  with rows already drops them. A scrub that is given
+  a header (and checks it, the default) keeps the file's first line as its header, an
+  empty one included, with a "Header mismatch" warning when it differs; the header it is
+  given is written only with `verifyHeader=False`, or when the file's first line cannot be
+  decoded (plain text named `.gz`); `tsvz scrub --x-header` follows the `--x-header` rule
+  below instead. A scrub of a path that is not there
+  makes no file and returns `{}`, as in 5.1, unless `createIfNotExist=True`. A compressed
+  file whose read decodes no row (plain text named `.gz`, a first member cut short) is
+  made again as an empty store, as a scrub of a damaged file with rows already was in
+  5.1: what the read did not get goes, the bytes after the damage and also the lines of the
+  damaged member before it that the codec never handed over (Python's gzip gives none of
+  a first member cut short, where `zcat` shows them) (see the damaged-stream notes above), and
+  `tsvz clear --x-header H --x-strict` (`clearTabularFile(verifyHeader=True)`) now clears
+  such a file, where 5.1 exited 1 with the codec's error.
+  5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that
+  appended keys, deleted them all and scrubbed, over and over, saw the file grow without
+  limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after
+  every scrub. A file with no byte (or a compressed one with only empty streams) is
+  still left alone, and so is a file whose strict scrub (`strict=True`, `--x-strict`) read
+  dropped a line for its width and found no row: that line is not a deletion, as 5.1 left
+  it. A scrub that finds the file changed since its read still reads
+  again and writes nothing if it keeps changing, as for a file with rows.
+- **`get_file_obj('ab')` on a compressed store writes its trailer before the lock is let
+  go.** A program that calls it on a `.tsv.gz`, `.bz2`, `.xz` or `.zst` store and writes
+  through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included,
+  in the file when `release_file_obj()` unlocks it. 5.1 and earlier wrote them after the
+  unlock, where another writer could append in between and corrupt the stream.
+- **A `TSVZed` no longer lets a file grow with every session of sets and deletes.** It
+  rewrites a `.tsv`-family file at open when the file holds a key's row and the deletion
+  of it, as 3.39 did only for a key with two values, so a program that sets keys and
+  deletes them in each session (default settings, or `rewrite_interval`) keeps the file
+  the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session
+  that opens such a file now rewrites it once. With `rewrite_on_load=False` the open
+  does not, and the next exit or periodic rewrite does it (`rewrite_on_exit=True`,
+  `rewrite_interval`). What such a rewrite does, it now does at a plain open:
+  - it cuts a row wider than the header, and the cut stays
+    ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)):
+    `id\tv\na\t1\t2\nb\t1\nb\t\n` becomes `id\tv\na\t1  \n`, and
+    `id\tv\na\t\t7\nb\t1\nb\t\n` becomes `id\tv\na\t  \n`, the 7 lost;
+  - it writes over the file in place and pads with spaces (`a\t1  ` above): TSVZ readers
+    strip them, tools that read the raw bytes (`awk`, `cut`) see them;
+  - on a file it cannot write (read-only), it prints `Failed to open file … Permission
+    denied` twice and leaves the file as it is, as 5.1 did for a key with two values.
+- **`--x-header H` makes line 1 the header for every `tsvz` operation.** On a
+  `.tsv`-family file whose line 1 is H (an exact match: the same cells, each without the
+  whitespace around it, a trailing empty cell included), line 1 is the header: `keys`,
+  `len`, `read`, `get`, `has`, `pop`, `popitem` and `delete` never list, count, print,
+  pop or delete it, and `scrub` and `clear` keep it. When line 1 differs, the file has no
+  header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and
+  the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases
+  below), and in such a file it never writes a row that is H as line 1: it puts an empty
+  line first (`a\t1\nid\tv\na\t\n` becomes `\nid\tv\n`), so the reads still take that row
+  for a row. With `--x-strict` a line 1 that is not H is an error (exit 1, nothing
+  written) for every operation, through a handler too. A missing or empty file gets H
+  first at its first write, as in 5.1. A `tsvz serve STORE --x-header H` handler applies
+  the rule to every request it answers, routed `tsvz` commands included; with
+  `--x-strict` it is strict for them as above (5.1 ignored `--x-strict` at `serve`): such
+  a handler refuses to start on a file whose line 1 is not H, its routed `.tsv` reads and
+  scrubs drop lines of another column count, and its `.tsvz` reads are strict, as for a
+  direct `--x-strict` command. The library's own header check (`header=` to
+  `readTabularFile`, `appendTabularFile`, `clearTabularFile`, `scrubTabularFile`,
+  `TSVZed`) is unchanged, 3.39's containment check. The CLI's writes check line 1 more
+  strictly than 5.1's did: 5.1's `set`, `append`, `delete` and `clear` with `--x-header`
+  silently took a line 1 that merely held H's cells (`id\tvalue` for `id\tv`) for the
+  header and fitted rows to H's width; now they warn and write the row at its own width,
+  and with `--x-strict` exit 1. A rewrite without `--x-header` (a scrub without it,
+  `scrubTabularFile`, a `TSVZed` rewrite) can still put a row equal to H on line 1 of a
+  header-less file
+  ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
+  In 5.1 only `set`, `append`, `delete` and `clear` used `--x-header`: the reads and
+  `scrub` took line 1 as a row (`tsvz keys`
+  printed `id a b` for two keys, `len` said 3, `popitem first` popped the header), and
+  so did every request to a handler. Without `--x-header` nothing changed: line 1 is a
+  row, as in 5.1 and 3.39. In a `.tsvz` store `pop`, `popitem` and `setdefault` now check
+  `--x-header` as `set` does; nothing else changed there.
+- **Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row,
+  and can lose cells or keys.** A scrub given defaults narrower than the rows, a row wider
+  than the header, and a `.tsv` that begins with a bare `#_defaults_#` line (read without
+  defaults) are examples; the entry lists each, and which is new in 5.2
+  ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)).
+  A strict scrub of a file whose read dropped a line for its width and found no row leaves it,
+  so it grows again over append, delete-all and scrub cycles
+  ([limitations](docs/limitations.md#a-strict-scrub-leaves-a-file-whose-read-dropped-a-line-for-its-width)).
+  Both wait for the column-width design after 5.2.
+
 ## Upgrading to 5.1 from 5.0
 
 The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
@@ -325,12 +447,11 @@ The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family
   file's own header line.)
   `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
   a program asks for.
-- **Known: a scrub of a `.tsv` whose rows were all deleted does not compact it** when
-  the scrub is given the file's header (`scrubTabularFile(path, header=H)`) or the file
-  has no header line: it leaves the file as it is, deletion lines included; reads are
-  right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file reads the header line as a
-  row, so it does compact it.)
-  ([limitations](docs/limitations.md#a-scrub-of-a-tsv-whose-rows-were-all-deleted-does-not-compact-it)).
+- **Known in 5.1, fixed in 5.2: a scrub of a `.tsv` whose rows were all deleted did not
+  compact it** when the scrub was given the file's header (`scrubTabularFile(path, header=H)`)
+  or the file had no header line: it left the file as it was, deletion lines included;
+  reads were right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file read the header
+  line as a row, so it did compact it.)
 - **Known: a scrub run with other defaults than the store's program can change how
   early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
   cells before any defaults line, the reset line the scrub writes before the rows keeps

{tsvz-5.1 → tsvz-5.2}/TSVZ.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 5.1
+Version: 5.2
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/tsvz-org/TSVZ
 Author: Yufei Pan
@@ -163,6 +163,20 @@ named like an operation then needs a path prefix (`./read`).
   with a warning.
 - **TSVZ extensions:**
   - `--x-header H` and `--x-defaults KEY<d>D1<d>...` decode backslash escapes such as `\t`;
+  - `--x-header H` makes line 1 of a `.tsv`-family file its header for every operation
+    when line 1 is H: the same number of cells, each equal to H's once the whitespace
+    around it is taken off (a trailing empty cell counts: `id\tv\t` is not `id\tv`; a UTF-8
+    byte order mark before line 1, as Excel writes one, is skipped). It
+    is then never a key, row or count, `scrub` and `clear` keep it, and
+    `read --format table` shows it above the rows. When line 1 differs, the file has no
+    header and line 1 is a row, with 3.39's "Header mismatch" warning (`--x-strict`: an
+    error, exit 1, nothing written). A missing or empty file gets H first. Without
+    `--x-header`, line 1 is a row, as in 3.39. (The library's own header check, a
+    `header=` given to `readTabularFile`, `appendTabularFile`, `clearTabularFile`,
+    `scrubTabularFile` or `TSVZed`, is 3.39's, looser one: each cell of H within the
+    cell of line 1 under it, so `idaho\tvalue` passes for `id\tv`.) A rewrite without
+    `--x-header` can put a row equal to H on line 1
+    ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
   - `--x-strict` / `--x-force` turn 3.39's column and header checks on or off.
 
 ## Serving a store
@@ -193,7 +207,13 @@ status stay the same.
   and `fsync`s everything and removes `STORE.serve`.
 - **Bypassing it.** `--x-direct` makes a `tsvz` command use the files. Command
   lines with `--x-header`, `--x-defaults` or `--x-strict`, or another delimiter
-  than the server's, use the files too.
+  than the server's, use the files too. A server started with `--x-header H`
+  answers every request by the `--x-header` rule above, and one started with
+  `--x-strict` too answers as a direct `--x-strict` command does: a line 1 that is
+  not H fails the request (status 1, nothing written), and reads and scrubs drop
+  the rows of another width. It does not start when line 1 is not H then. The
+  table of a routed `read --format table` shows no H line: its first row is set
+  as the table's header, as with no `--x-header`.
 
 ```python
 import TSVZ
@@ -261,7 +281,9 @@ microseconds as a write starts is followed as a rotation instead); while that re
 fails or is refused, the rows are appended to a new file there, and memory keeps all of
 them for the rewrite that gets through. A compressed file another writer left damaged
 is made again too (L5: every read stops at the damage, so rows
-appended after it would be read by no one); the bytes after the damage go. While that
+appended after it would be read by no one): what the read did not get goes, the bytes after
+the damage and also the lines of the damaged member before it that the codec never handed
+over (Python's gzip gives none of a first member cut short, where `zcat` shows them). While that
 rewrite fails, the rows are appended after the damage and memory keeps them for the
 next rewrite that gets through; while it is refused (a row changed in place to text
 the encoding cannot hold), they are appended after it as in 4.9, and the next read
@@ -301,6 +323,106 @@ empty; the keys it sets or deletes then take its values, and the keys it left ou
 memory's. An explicit `reload()` while the file is damaged replaces memory with the
 damaged file, as asked, and warns.
 
+## Upgrading to 5.2 from 5.1
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.2 writes.
+
+- **A scrub of a `.tsv` whose rows were all deleted shrinks it.** `scrubTabularFile`,
+  `tsvz scrub` (also through a `tsvz serve` handler) of a `.tsv`-family file (`.tsv`,
+  `.csv`, `.nsv`, `.psv` and their compressed forms) in which every key has been deleted
+  leaves the file as an empty store: the header line it has, and its `#_defaults_#`
+  lines as a scrub of a file with rows keeps them (an empty one after the header, the
+  last non-empty one after it, both as wide as the header, or as the column count the
+  read took, which caller defaults set when given). A file with no header line keeps its
+  empty first line, or becomes 0 bytes when its first line was a row; a first line that
+  is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an
+  empty first line, which the scrub then writes. Comment lines go, as a scrub of a file
+  with rows already drops them. A scrub that is given
+  a header (and checks it, the default) keeps the file's first line as its header, an
+  empty one included, with a "Header mismatch" warning when it differs; the header it is
+  given is written only with `verifyHeader=False`, or when the file's first line cannot be
+  decoded (plain text named `.gz`); `tsvz scrub --x-header` follows the `--x-header` rule
+  below instead. A scrub of a path that is not there
+  makes no file and returns `{}`, as in 5.1, unless `createIfNotExist=True`. A compressed
+  file whose read decodes no row (plain text named `.gz`, a first member cut short) is
+  made again as an empty store, as a scrub of a damaged file with rows already was in
+  5.1: what the read did not get goes, the bytes after the damage and also the lines of the
+  damaged member before it that the codec never handed over (Python's gzip gives none of
+  a first member cut short, where `zcat` shows them) (see the damaged-stream notes above), and
+  `tsvz clear --x-header H --x-strict` (`clearTabularFile(verifyHeader=True)`) now clears
+  such a file, where 5.1 exited 1 with the codec's error.
+  5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that
+  appended keys, deleted them all and scrubbed, over and over, saw the file grow without
+  limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after
+  every scrub. A file with no byte (or a compressed one with only empty streams) is
+  still left alone, and so is a file whose strict scrub (`strict=True`, `--x-strict`) read
+  dropped a line for its width and found no row: that line is not a deletion, as 5.1 left
+  it. A scrub that finds the file changed since its read still reads
+  again and writes nothing if it keeps changing, as for a file with rows.
+- **`get_file_obj('ab')` on a compressed store writes its trailer before the lock is let
+  go.** A program that calls it on a `.tsv.gz`, `.bz2`, `.xz` or `.zst` store and writes
+  through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included,
+  in the file when `release_file_obj()` unlocks it. 5.1 and earlier wrote them after the
+  unlock, where another writer could append in between and corrupt the stream.
+- **A `TSVZed` no longer lets a file grow with every session of sets and deletes.** It
+  rewrites a `.tsv`-family file at open when the file holds a key's row and the deletion
+  of it, as 3.39 did only for a key with two values, so a program that sets keys and
+  deletes them in each session (default settings, or `rewrite_interval`) keeps the file
+  the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session
+  that opens such a file now rewrites it once. With `rewrite_on_load=False` the open
+  does not, and the next exit or periodic rewrite does it (`rewrite_on_exit=True`,
+  `rewrite_interval`). What such a rewrite does, it now does at a plain open:
+  - it cuts a row wider than the header, and the cut stays
+    ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)):
+    `id\tv\na\t1\t2\nb\t1\nb\t\n` becomes `id\tv\na\t1  \n`, and
+    `id\tv\na\t\t7\nb\t1\nb\t\n` becomes `id\tv\na\t  \n`, the 7 lost;
+  - it writes over the file in place and pads with spaces (`a\t1  ` above): TSVZ readers
+    strip them, tools that read the raw bytes (`awk`, `cut`) see them;
+  - on a file it cannot write (read-only), it prints `Failed to open file … Permission
+    denied` twice and leaves the file as it is, as 5.1 did for a key with two values.
+- **`--x-header H` makes line 1 the header for every `tsvz` operation.** On a
+  `.tsv`-family file whose line 1 is H (an exact match: the same cells, each without the
+  whitespace around it, a trailing empty cell included), line 1 is the header: `keys`,
+  `len`, `read`, `get`, `has`, `pop`, `popitem` and `delete` never list, count, print,
+  pop or delete it, and `scrub` and `clear` keep it. When line 1 differs, the file has no
+  header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and
+  the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases
+  below), and in such a file it never writes a row that is H as line 1: it puts an empty
+  line first (`a\t1\nid\tv\na\t\n` becomes `\nid\tv\n`), so the reads still take that row
+  for a row. With `--x-strict` a line 1 that is not H is an error (exit 1, nothing
+  written) for every operation, through a handler too. A missing or empty file gets H
+  first at its first write, as in 5.1. A `tsvz serve STORE --x-header H` handler applies
+  the rule to every request it answers, routed `tsvz` commands included; with
+  `--x-strict` it is strict for them as above (5.1 ignored `--x-strict` at `serve`): such
+  a handler refuses to start on a file whose line 1 is not H, its routed `.tsv` reads and
+  scrubs drop lines of another column count, and its `.tsvz` reads are strict, as for a
+  direct `--x-strict` command. The library's own header check (`header=` to
+  `readTabularFile`, `appendTabularFile`, `clearTabularFile`, `scrubTabularFile`,
+  `TSVZed`) is unchanged, 3.39's containment check. The CLI's writes check line 1 more
+  strictly than 5.1's did: 5.1's `set`, `append`, `delete` and `clear` with `--x-header`
+  silently took a line 1 that merely held H's cells (`id\tvalue` for `id\tv`) for the
+  header and fitted rows to H's width; now they warn and write the row at its own width,
+  and with `--x-strict` exit 1. A rewrite without `--x-header` (a scrub without it,
+  `scrubTabularFile`, a `TSVZed` rewrite) can still put a row equal to H on line 1 of a
+  header-less file
+  ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
+  In 5.1 only `set`, `append`, `delete` and `clear` used `--x-header`: the reads and
+  `scrub` took line 1 as a row (`tsvz keys`
+  printed `id a b` for two keys, `len` said 3, `popitem first` popped the header), and
+  so did every request to a handler. Without `--x-header` nothing changed: line 1 is a
+  row, as in 5.1 and 3.39. In a `.tsvz` store `pop`, `popitem` and `setdefault` now check
+  `--x-header` as `set` does; nothing else changed there.
+- **Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row,
+  and can lose cells or keys.** A scrub given defaults narrower than the rows, a row wider
+  than the header, and a `.tsv` that begins with a bare `#_defaults_#` line (read without
+  defaults) are examples; the entry lists each, and which is new in 5.2
+  ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)).
+  A strict scrub of a file whose read dropped a line for its width and found no row leaves it,
+  so it grows again over append, delete-all and scrub cycles
+  ([limitations](docs/limitations.md#a-strict-scrub-leaves-a-file-whose-read-dropped-a-line-for-its-width)).
+  Both wait for the column-width design after 5.2.
+
 ## Upgrading to 5.1 from 5.0
 
 The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
@@ -357,12 +479,11 @@ The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family
   file's own header line.)
   `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
   a program asks for.
-- **Known: a scrub of a `.tsv` whose rows were all deleted does not compact it** when
-  the scrub is given the file's header (`scrubTabularFile(path, header=H)`) or the file
-  has no header line: it leaves the file as it is, deletion lines included; reads are
-  right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file reads the header line as a
-  row, so it does compact it.)
-  ([limitations](docs/limitations.md#a-scrub-of-a-tsv-whose-rows-were-all-deleted-does-not-compact-it)).
+- **Known in 5.1, fixed in 5.2: a scrub of a `.tsv` whose rows were all deleted did not
+  compact it** when the scrub was given the file's header (`scrubTabularFile(path, header=H)`)
+  or the file had no header line: it left the file as it was, deletion lines included;
+  reads were right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file read the header
+  line as a row, so it did compact it.)
 - **Known: a scrub run with other defaults than the store's program can change how
   early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
   cells before any defaults line, the reset line the scrub writes before the rows keeps

{tsvz-5.1 → tsvz-5.2}/TSVZ.py

@@ -42,10 +42,10 @@ if os.name == 'nt':
 elif os.name == 'posix':
 	import fcntl
 
-version = '5.1'
+version = '5.2'
 __version__ = version
 author = 'pan@zopyr.us'
-COMMIT_DATE = '2026-11-01'
+COMMIT_DATE = '2026-11-03'
 
 DEFAULT_DELIMITER = '\t'
 DEFAULTS_INDICATOR_KEY = '#_defaults_#'
@@ -1345,6 +1345,8 @@ def _processLine(line,taskDic,correctColumnNum,strict = True,delimiter = ...,def
 			correctColumnNum = len(lineCache)
 		# if verbose:
 		# 	__teePrintOrNot(f"detected correctColumnNum: {len(lineCache)}",teeLogger=teeLogger)
+	if isinstance(defaults, _DefaultsList) and defaults.columns is None:
+		defaults.columns = correctColumnNum
 	if len(lineCache) == 1 or not any(lineCache[1:]):
 		if correctColumnNum == 1: 
 			taskDic[lineCache[0]] = lineCache if not storeOffset else offset
@@ -1364,6 +1366,8 @@ def _processLine(line,taskDic,correctColumnNum,strict = True,delimiter = ...,def
 		return correctColumnNum , []
 	elif len(lineCache) != correctColumnNum:
 		if strict and not any(defaults[1:]):
+			if isinstance(defaults, _DefaultsList):
+				defaults.dropped = True
 			if reporter is not None:
 				reporter.note('strict-drop', None, 'dropped a line whose column count is not {} (strict mode)'.format(correctColumnNum))
 			return correctColumnNum , []
@@ -1605,6 +1609,21 @@ def _headerMatches(header, line, delimiter):
 	return _headerCellsMatch(header, _formatHeader(line, delimiter=get_delimiter(delimiter)))
 
 
+def _isHeaderLine(header, line, delimiter):
+	"""Whether ``line`` is the header ``header`` to the CLI's ``--x-header`` rule: the same number of cells, each equal
+	once the whitespace around it (and the line's end, and a UTF-8 byte order mark before it, as Excel writes one) is
+	taken off. Stricter than 3.39's ``_lineContainHeader``."""
+	cells = line.rstrip('\r\n').lstrip('\ufeff').split(delimiter)
+	return len(cells) == len(header) and all(a.strip() == b.strip() for a, b in zip(cells, header))
+
+
+def _headerMismatch(header, line, teeLogger, strict, delimiter):
+	"""Print 3.39's "Header mismatch" (as ``_lineContainHeader`` prints it) for ``line``; raise as it does when strict."""
+	__teePrintOrNot(f"Header mismatch: \n{_formatHeader(line, delimiter=delimiter)} \n!= \n{header}",teeLogger=teeLogger)
+	if strict:
+		raise ValueError("Data format error! Header mismatch")
+
+
 def _lineContainHeader(header,line,verbose = False,teeLogger = None,strict = False,delimiter = ...):
 	"""
 	Verify if a line contains the header.
@@ -2063,7 +2082,10 @@ def _legacyKeptHeader(fileName, header, verifyHeader, verbose, teeLogger, encodi
 	``verifyHeader`` (the header given for a file that holds nothing), else ``header``; [] when it has no cell: the file
 	is then left with no header line."""
 	if any(header) and verifyHeader:
-		line = _legacyFirstLine(fileName, encoding, teeLogger)
+		try:
+			line = _legacyFirstLine(fileName, encoding, teeLogger)
+		except Exception:  # a first line the codec cannot decode (gzip: BadGzipFile, zlib.error; bz2, xz, zstd: their own errors): none to keep
+			line = ''  # (a scrub or clear makes the file again; an append, which calls _legacyFirstLine itself, raises)
 		if line:  # a file that holds nothing ('') is an empty store with any header, as the readers take it: the one given is kept
 			if warn and not _lineContainHeader(header,line,verbose = verbose,teeLogger = teeLogger,strict = strict,delimiter = delimiter):
 				__teePrintOrNot(f'Warning: Header mismatch in {fileName}. Keeping original header in file...','warning',teeLogger)
@@ -2287,6 +2309,8 @@ class _DefaultsList(list):
 	replaces its cells in place), not the caller's defaults it began with."""
 	found = False
 	reset = False  # an empty defaults line was read (it stays one whatever follows it)
+	columns = None  # the column count the read took (_processLine), None before it took one
+	dropped = False  # a strict read dropped a line for its width (_processLine): a read with no row may not be all deletions
 	line = None  # the cells of that line as the file has them (``defaults`` holds them filled and fitted to the columns)
 
 	def __setitem__(self, index, value):
@@ -2295,7 +2319,8 @@ class _DefaultsList(list):
 
 
 def _legacyScrub(fileName,teeLogger = None,header = '',createIfNotExist = False,verifyHeader = True,verbose = False,
-				 taskDic = None,encoding = 'utf8',strict = False,delimiter = ...,defaults = ...,correctColumnNum = -1,errors = 'strict'):
+				 taskDic = None,encoding = 'utf8',strict = False,delimiter = ...,defaults = ...,correctColumnNum = -1,errors = 'strict',
+				 notFirst = None):
 	"""Compact a legacy file in place; return ``(taskDic, done)``: the rows read, and whether the file was written (or
 	had nothing to write). ``done`` is False when nothing was written because the file kept changing (the shape of
 	``_specScrub``): it is rewritten only if it is what the read saw (spec §19.8), so a row another process appends
@@ -2305,7 +2330,15 @@ def _legacyScrub(fileName,teeLogger = None,header = '',createIfNotExist = False,
 	non-empty one after the rows (those before it keep their empty cells, those after it were filled when read), so
 	that every row reads as it did and the header, if any, stays on the first line. An empty one that was read is
 	written before the rows (_legacyScrub, ``reset``), as wide as they are. The defaults the caller passed are not
-	written. With no header the file begins with an empty line only if the first line of the file was not a row."""
+	written. With no header the file begins with an empty line only if the first line of the file was not a row.
+
+	A file whose read finds no live row (all deleted) is written the same way, with no row: the header line, the defaults
+	lines, and the empty line for a header-less file that began with one. A reset is as wide as the header then, or, with
+	no header, as the lines the read took its column count from. A file that holds nothing (no byte, or only empty streams) is left as it is, and so is one whose
+	strict read dropped a line for its width: that line is not a deletion, and writing the file empty would lose it.
+
+	``notFirst``, a header list, for a file written with no header: when its first row would be that header
+	(``_isHeaderLine``), the file begins with an empty line instead, so a reader given that header still reads the row."""
 	given = list(taskDic.items()) if taskDic else []
 	givenDefaults = list(defaults) if defaults not in (..., None) else []
 	for tries in range(_SPEC_REWRITE_ATTEMPTS):
@@ -2319,8 +2352,8 @@ def _legacyScrub(fileName,teeLogger = None,header = '',createIfNotExist = False,
 		file =  readTabularFile(fileName,teeLogger = teeLogger,header = header,createIfNotExist = createIfNotExist,
 								verifyHeader = verifyHeader,verbose = verbose,taskDic = taskDic,
 								encoding = encoding,strict = strict,delimiter = delimiter,defaults=readDefaults,correctColumnNum = correctColumnNum,errors = errors)
-		if not file:
-			return file, True
+		if not file and (readDefaults.dropped or not os.path.exists(fileName) or _fileHoldsNothing(fileName)):
+			return file, True  # no row, and no file (not made by a scrub), no byte (or only empty streams) to compact, or a line the strict read dropped (not a deletion; it stays): as it is
 		dlm = get_delimiter(delimiter,file_name=fileName)
 		formatted = _formatHeader(header,delimiter=dlm)
 		if not _verifyFileExistence(fileName,createIfNotExist = True,teeLogger = teeLogger,header = formatted,encoding = encoding,strict = False,delimiter=dlm,errors = errors):
@@ -2331,6 +2364,17 @@ def _legacyScrub(fileName,teeLogger = None,header = '',createIfNotExist = False,
 		payload, count, columns = _legacyFormatRows(fileName, file, teeLogger, formatted, verifyHeader, verbose, encoding,
 													strict, dlm, errors, assume=_sanitize(kept,delimiter=dlm) if kept else [])
 		blank = not kept and _legacyBeginsBlank(fileName, encoding, teeLogger, dlm)
+		if not kept and not blank and notFirst and count and \
+				_isHeaderLine(notFirst, payload.split(b'\n', 1)[0].decode(encoding or 'utf8', errors='replace'), dlm):
+			blank = True
+		if not count:
+			# No row to take the width from: a reader takes it from the header line, else from the first line it reads,
+			# which a defaults line written here would be, so it is as wide as the file's lines were.
+			columns = len(kept) or readDefaults.columns or 0
+		last = readDefaults.line if readDefaults.found else None
+		last = last if last and len(last) > 1 else None
+		if not count and not kept and ((readDefaults.reset and columns > 1) or last):
+			blank = True  # a defaults line is never the first line of a file with no header: a reader given one would take it for it
 		if readDefaults.reset and columns > 1:
 			# An empty defaults line was read: no defaults reach the rows that follow it, whoever reads them (the rows
 			# were read with it, and any the caller of that read gave are not written). Written before the rows, so
@@ -2343,8 +2387,9 @@ def _legacyScrub(fileName,teeLogger = None,header = '',createIfNotExist = False,
 			else:
 				first, newline, rest = payload.partition(b'\n')
 				payload = first + newline + reset + rest
-		last = readDefaults.line if readDefaults.found else None
-		if last and len(last) > 1:
+		if last:
+			if not count and columns > 1:
+				last = (list(last) + [''] * columns)[:columns]  # (the same defaults to a reader, fitted to the width of the file as it fits a line)
 			payload += _encodeRow(_sanitize(last,delimiter=dlm),dlm,encoding,errors,what='defaults') + b'\n'
 		if _legacyReplace(fileName, kept, blank, encoding, dlm, errors, payload, expect=stamp) is None:
 			continue  # another writer changed the file after the read
@@ -4299,7 +4344,7 @@ class _TSVZedLoad(OrderedDict):
 
 	Its rows are shaped as ``TSVZed.__setitem__`` shapes them in memory only (keys and cells right-stripped,
 	the column count and the defaults applied), with a column count and defaults of its own. ``dirty`` tells
-	that the file holds a key twice.
+	that the file holds a key twice (two values, or a row and its deletion).
 	"""
 
 	def __init__(self, owner, columns, defaults):
@@ -4349,6 +4394,7 @@ class _TSVZedLoad(OrderedDict):
 	def __delitem__(self, key):
 		key = str(key)
 		OrderedDict.__delitem__(self, key if OrderedDict.__contains__(self, key) else key.rstrip())
+		self.dirty = True  # the file holds the key twice: its row, and this deletion
 
 	def replay(self, rows):
 		"""Apply rows queued for the file, as reading them back will: a row whose values are all empty deletes."""
@@ -6106,8 +6152,8 @@ memoryOnly:{self.memoryOnly}
 	
 	def get_file_obj(self,modes = 'ab'):
 		"""The store's file opened in ``modes`` and locked, with writeLock held until release_file_obj() (3.39 API; the
-		store's own writes do not use it). 'wb' empties it once locked, a compressed one then written through its
-		codec. None when it cannot be opened or locked (reported)."""
+		store's own writes do not use it). 'wb' empties it once locked; for a compressed file 'w' and 'a' modes give
+		a writer through its codec over the locked file (a new member at the end for 'a'). None when it cannot be opened or locked (reported)."""
 		self.writeLock.acquire()
 		file = None
 		try:
@@ -6116,12 +6162,21 @@ memoryOnly:{self.memoryOnly}
 				file = _openUntruncated(self._fileName)
 			elif modes == 'a+b':
 				file = open(self._fileName, 'a+b')
+			elif 'a' in modes and '+' not in modes and _fileCodec(self._fileName):
+				_requireCodec(_fileCodec(self._fileName))
+				file = open(self._fileName, 'ab')  # the codec writes into this locked file, a new member at its end
 			else:
 				file = openFileAsCompressed(self._fileName, mode=modes, encoding=self.encoding, teeLogger=self.teeLogger)
 			_lockFile(file)
 			if 'w' in modes:
 				file.truncate(0)
 				file = _compressedWriter(self._fileName, file) or file
+			elif 'a' in modes and '+' not in modes and _fileCodec(self._fileName):
+				raw = file
+				file = _compressedWriter(self._fileName, raw)  # (release_file_obj closes it, trailer and all, before the unlock)
+				if 'b' not in modes:
+					file = io.TextIOWrapper(file, encoding=self.encoding)
+					file._tsvzRaw = raw
 			if self.verbose:
 				self.__teePrintOrNot(f"File {self._fileName} locked with mode {modes}")
 			return file
@@ -7090,12 +7145,13 @@ class _ServeStore(object):
 	takes ``self.lock``; ``log`` is the teeLogger its messages go to.
 
 	``header`` and ``createDefaults`` (default ``defaults``) are what a missing store is created with
-	(``create``). ``checkHeader``: writes and ``clear`` also check ``header`` as 3.39 did (the CLI's
-	``--x-header``). ``lazy``: the store is read at the first request that needs it, not here.
+	(``create``). ``header`` is also the CLI's ``--x-header``: a spec store's writes and ``clear`` check it as
+	3.39 did, and a loose file takes it by one rule for every request (``_legacyHeader``). ``lazy``: the store is
+	read at the first request that needs it, not here.
 	"""
 
 	def __init__(self, fileName, delimiter=..., header='', defaults=None, strict=False, teeLogger=None,
-				 verbose=False, create=False, createDefaults=None, checkHeader=False, lazy=False):
+				 verbose=False, create=False, createDefaults=None, lazy=False):
 		self.path = fileName
 		self.spec = _isSpecPath(fileName)
 		if self.spec:
@@ -7112,7 +7168,9 @@ class _ServeStore(object):
 		self.unreadable = False
 		self.stopped = False
 		self.parts = self.state = None  # not read yet
-		self.header = _formatHeader(header, teeLogger=teeLogger, delimiter=self.delimiter) if checkHeader else ['']
+		self.given = _formatHeader(header, teeLogger=teeLogger, delimiter=self.delimiter) if header else ['']
+		self.header = self.given  # a loose file's: the header in effect, as _legacyHeader last found it
+		self.mismatch = None  # the line 1 last warned of (_legacyHeader)
 		if create:
 			self._create(header, self.defaults if createDefaults is None else createDefaults)
 		if not lazy:
@@ -7138,12 +7196,42 @@ class _ServeStore(object):
 				if self.spec:
 					self._loadSpec(reporter, log)
 				else:
-					self._loadLegacy(reporter)
+					self._loadLegacy(reporter, log)
 				self.unreadable = reporter.incomplete()
 				self.stopped = reporter.has('stopped')  # a stop policy ended the read (spec §4.10)
+			except Exception:
+				self.parts = None  # (a strict header mismatch): the next request reads again, and fails again
+				raise
 			finally:
 				reporter.flush()
 
+	def _legacyHeader(self, line, log):
+		"""The header in effect for a loose file whose first line is ``line``: the one rule of ``--x-header``.
+
+		``line`` is None when the file is missing, holds nothing, or its first line cannot be read. With a header
+		given, line 1 is the header when it is that header (``_isHeaderLine``: the same cells, each without the
+		whitespace around it), or when there is no line 1 yet (a write puts the header first). Otherwise 3.39's
+		"Header mismatch" is printed and strict raises (nothing is written); loose, the file has no header and line 1
+		is a row, for every request, the scrub included. Sets and returns ``self.header``."""
+		header = self.given
+		if any(header) and line is not None and not _isHeaderLine(header, line, self.delimiter):
+			if line != self.mismatch or self.strict:  # (said once for the line, as 3.39 says it)
+				self.mismatch = line
+				_headerMismatch(header, line, log if log is not None else self.teeLogger, self.strict, self.delimiter)
+			header = ['']
+		self.header = header
+		return header
+
+	def _legacyHeaderNow(self, log):
+		"""``_legacyHeader`` for the file as it is now (a write, ``clear`` or ``scrub`` that does not read it)."""
+		line = None
+		if any(self.given) and os.path.isfile(self.path):
+			try:
+				line = _legacyFirstLine(self.path, 'utf8', self.teeLogger) or None
+			except Exception:  # a first line the codec cannot decode: the write or scrub that reads it reports it
+				pass
+		return self._legacyHeader(line, log)
+
 	def _noteDirMtime(self, dirMtime):
 		self.dirMtime = dirMtime
 		self.dirRacy = time.time() - dirMtime / 1e9 < _SERVE_RACY_SECONDS
@@ -7181,19 +7269,33 @@ class _ServeStore(object):
 		self.label = os.path.basename(self.parts[-1]) + ' ' if len(self.parts) > 1 else ''
 		self.check = info.check if info is not None else b''  # as read: the part may be rewritten since
 
-	def _loadLegacy(self, reporter):
+	def _loadLegacy(self, reporter, log):
 		self.parts = [self.path] if os.path.isfile(self.path) else []
 		self.stamps = self._stamps(self.parts)
 		self.data = self.mapping()
 		self.legacyDefaults = list(self.defaults)
 		self.width, self.offset, self.lineNo, self.tailSeen = -1, 0, 0, b''
+		self.header = self.given  # (no line 1: a write puts the header first)
 		# A compressed file with no byte, or only empty streams, is an empty store, as readTabularFile reads it.
 		if self.parts and not (_isCompressedFile(self.path) and _fileHoldsNothing(self.path)):
 			try:
 				with openFileAsCompressed(self.path, mode='rb', teeLogger=self.teeLogger) as f:
+					if any(self.given):
+						try:
+							raw = f.readline()
+						except Exception as e:  # L5: a stream damaged inside line 1 holds nothing more to read
+							if not _isCompressedFile(self.path):
+								raise
+							reporter.note('damaged', None, 'compressed stream is damaged ({}: {}); read up to the '
+										  'damage'.format(type(e).__name__, e))
+							return
+						if raw and any(self._legacyHeader(raw.decode('utf8', errors='replace'), log)):
+							self.width, self.lineNo = len(self.header), 1  # the header: not a row
+						elif raw:
+							self._apply(raw, reporter)  # line 1 is a row
 					self.width, self.offset, self.lineNo = _legacyReadLoop(
-						f, self.path, self.data, -1, 0, self.strict, self.delimiter, self.legacyDefaults, False,
-						'utf8', reporter)
+						f, self.path, self.data, self.width, self.lineNo, self.strict, self.delimiter,
+						self.legacyDefaults, False, 'utf8', reporter)
 			except OSError as e:
 				reporter.note('unreadable', None, 'could not read {} ({})'.format(self.path, e))
 		self.label = ''
@@ -7236,7 +7338,8 @@ class _ServeStore(object):
 		"""Replay what was appended to the active part; False when it changed some other way."""
 		old = self.stamps.get(path)
 		if (self.stopped or _parsePartName(path).codec or old is None or stamp[0] != old[0]
-				or stamp[1] < self.offset or stamp[1] <= old[1]):
+				or stamp[1] < self.offset or stamp[1] <= old[1] or (not self.spec and self.lineNo <= 1 and any(self.given))):
+			# (a loose file's line 1 is read by a reload, which tells whether it is the header, also once it is whole)
 			return False  # an append always grows the part; anything else is a rewrite (and a stopped read is read again)
 		try:
 			with open(path, 'rb') as f:
@@ -7389,6 +7492,7 @@ class _ServeStore(object):
 			if rows and any(self.header):
 				_specCheckHeader(self.path, self.header, self.delimiter, self.strict, self.verbose, log)
 			return _specFormatRecords(rows, self.delimiter).encode('utf-8')
+		self._legacyHeaderNow(log)
 		# Rows for a file that holds nothing yet are fitted to the header it is about to begin with.
 		assume = self.header if any(self.header) and _fileHoldsNothing(self.path) else None
 		return _legacyFormatPayload(self.path, [cells for cells, marker in rows], log, self.header, True, self.verbose,
@@ -7503,9 +7607,12 @@ class _ServeStore(object):
 		with self.lock:
 			self.writer.barrier()
 			try:
-				clear = _specClearTabularFile if self.spec else clearTabularFile
-				return clear(self.path, teeLogger=log, header=self.header, verifyHeader=self.strict,
-							 verbose=self.verbose, delimiter=self.delimiter) is not False
+				if self.spec:
+					return _specClearTabularFile(self.path, teeLogger=log, header=self.header, verifyHeader=self.strict,
+												 verbose=self.verbose, delimiter=self.delimiter) is not False
+				self._legacyHeaderNow(log)  # a strict mismatch is an error; else the file is left with the header given
+				return clearTabularFile(self.path, teeLogger=log, header=self.given, verifyHeader=False,
+										verbose=self.verbose, delimiter=self.delimiter) is not False
 			finally:
 				self.parts = None  # read again at the next request
 
@@ -7518,8 +7625,13 @@ class _ServeStore(object):
 				if self.spec:
 					return _specScrub(self.path, teeLogger=log, verifyHeader=False, verbose=self.verbose,
 									  strict=self.strict, delimiter=self.delimiter, defaults=self.defaults)[1]
-				return _legacyScrub(self.path, teeLogger=log, verifyHeader=False, verbose=self.verbose,
-									strict=self.strict, delimiter=self.delimiter, defaults=self.defaults)[1]
+				# Line 1 is the header exactly when every read takes it for one, so the scrub keeps what they read; in a
+				# file with no header, a row that is the header given is never written as line 1 (reads would take it
+				# for the header).
+				header = self._legacyHeaderNow(log)
+				return _legacyScrub(self.path, teeLogger=log, header=header, verifyHeader=True, verbose=self.verbose,
+									strict=self.strict, delimiter=self.delimiter, defaults=self.defaults,
+									notFirst=self.given if any(self.given) and not any(header) else None)[1]
 			finally:
 				self.parts = None  # read again at the next request
 
@@ -7543,9 +7655,9 @@ class _ServeStore(object):
 			else:
 				codec = _fileCodec(self.path)
 				_requireCodec(codec)
-				header = None  # a file that holds nothing gets the header first (_appendOpen)
-				if any(self.header):
-					header = functools.partial(_encodeRow, _sanitize(self.header, delimiter=self.delimiter),
+				header = None  # a file that holds nothing gets the header given first (_appendOpen)
+				if any(self.given):
+					header = functools.partial(_encodeRow, _sanitize(self.given, delimiter=self.delimiter),
 											   self.delimiter, 'utf8', 'strict', what='header')
 				_lockedAppend(self.path, payload, 'newline', reporter, fsync=fsync, header=header, codec=codec)
 		finally:
@@ -8614,7 +8726,8 @@ options:
   -q, --quiet                  print errors only
   -v, --verbose                print more detail
   -d, --delimiter D            .tsv-family delimiter: tab, comma, pipe, null or one character
-  --x-header H                 header to check, or to create a new file with
+  --x-header H                 a .tsv's header: line 1 when it is H, else a row
+                               (--x-strict: an error); a new file begins with H
   --x-defaults D               defaults row KEY D1 D2 ... joined by the delimiter (KEY is ignored)
   --x-strict, --x-force        turn the 3.39 column and header checks on or off (default: off)
   --x-direct                   work on the files even when a server runs
@@ -8814,14 +8927,17 @@ def _cliTableCells(rows):
 	return [[_CLI_CONTROL_RE.sub(escape, str(cell)) for cell in row] for row in rows]
 
 
-def _cliEmitRows(rows, args, stdout, spec, delimiter, headerRow=False):
+def _cliEmitRows(rows, args, stdout, spec, delimiter, headerRow=False, header=None):
 	"""Print store rows for ``read`` / ``get`` (spec §20.4): §13 records, 3.39 records or a table.
 
 	The table shows the rows only, each padded to the widest and none truncated to the
 	terminal's width. ``headerRow`` keeps 3.39's ``read`` table of a loose file, whose
-	first row is the file's header line.
+	first row is the file's header line; ``header``, a loose file's header line that is not
+	a row (``--x-header``), is shown there instead. Records never hold it.
 	"""
 	if _cliFormat(args, stdout) == 'table':
+		if headerRow and header:
+			rows = [header] + rows
 		rows = _cliTableCells(rows)
 		if headerRow or not rows:
 			_cliWrite(stdout, pretty_format_table(rows, delimiter=delimiter) + '\n')
@@ -8952,16 +9068,16 @@ def _cliRequestLine(args):
 	return '\t'.join([args.operation] + [_serveEncode(arg) for arg in args.args])
 
 
-def _cliEmitServed(args, delimiter, logger, stdout, response):
+def _cliEmitServed(args, delimiter, logger, stdout, response, header=None):
 	"""Print a response (``(rows, diagnostics, status, message)``) as the operation prints on the files (§20.8);
-	return the exit status."""
+	return the exit status. ``header``: a loose file's header line in effect (``read``'s table shows it)."""
 	rows, diagnostics, status, message = response
 	for text in diagnostics:
 		logger.teelog(text, 'warning')
 	spec = _isSpecPath(args.store)
 	operation = args.operation
 	if operation in ('read', 'get', 'keys', 'pop', 'popitem', 'setdefault'):
-		_cliEmitRows(rows, args, stdout, spec, delimiter, headerRow=operation == 'read' and not spec)
+		_cliEmitRows(rows, args, stdout, spec, delimiter, headerRow=operation == 'read' and not spec, header=header)
 	elif operation == 'len':
 		_cliWrite(stdout, ''.join(row[0] + '\n' for row in rows))
 	elif operation == 'verify':
@@ -9026,8 +9142,7 @@ def _cliEngineOp(args, delimiter, logger, stdin, stdout):
 	store = _ServeStore(args.store, delimiter=delimiter, header=args.header, defaults=args.defaults,
 						strict=args.strict, teeLogger=logger, verbose=args.verbose,
 						create=not exists and operation != 'clear',  # clear creates the store itself
-						createDefaults=None if operation == 'setdefault' else [],
-						checkHeader=operation in ('set', 'append', 'delete', 'clear'), lazy=True)  # as 3.39 did
+						createDefaults=None if operation == 'setdefault' else [], lazy=True)
 
 	def request(line, bulk=None):
 		"""The response to ``line``; the engine's messages go to stderr as they come, at their own level."""
@@ -9043,7 +9158,7 @@ def _cliEngineOp(args, delimiter, logger, stdin, stdout):
 		response = request('\t'.join(written + [_cliRequestLine(args)]), bulk)
 	finally:
 		store.close()
-	return _cliEmitServed(args, delimiter, logger, stdout, response)
+	return _cliEmitServed(args, delimiter, logger, stdout, response, header=store.header if any(store.header) else None)
 
 
 def _cliServe(args, delimiter, logger, stdin, stdout):
@@ -9073,7 +9188,7 @@ def _cliServe(args, delimiter, logger, stdin, stdout):
 	restore = _serveSignals()  # before the pointer file announces this handler
 	try:
 		store = _ServeStore(args.store, delimiter=delimiter, header=args.header, createDefaults=args.defaults,
-							teeLogger=logger, verbose=args.verbose, create=True)
+							strict=args.strict, teeLogger=logger, verbose=args.verbose, create=True)
 		handle.start(store)
 		logger.teelog('tsvz: serving {} at {} (pid {})'.format(args.store, handle.address, os.getpid()))
 		handle.serve()