TSVZ 5.0__tar.gz → 5.2__tar.gz

{tsvz-5.0 → tsvz-5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 5.0
+Version: 5.2
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/tsvz-org/TSVZ
 Author: Yufei Pan
@@ -39,7 +39,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 5.0**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 5.1**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -163,6 +163,20 @@ named like an operation then needs a path prefix (`./read`).
   with a warning.
 - **TSVZ extensions:**
   - `--x-header H` and `--x-defaults KEY<d>D1<d>...` decode backslash escapes such as `\t`;
+  - `--x-header H` makes line 1 of a `.tsv`-family file its header for every operation
+    when line 1 is H: the same number of cells, each equal to H's once the whitespace
+    around it is taken off (a trailing empty cell counts: `id\tv\t` is not `id\tv`; a UTF-8
+    byte order mark before line 1, as Excel writes one, is skipped). It
+    is then never a key, row or count, `scrub` and `clear` keep it, and
+    `read --format table` shows it above the rows. When line 1 differs, the file has no
+    header and line 1 is a row, with 3.39's "Header mismatch" warning (`--x-strict`: an
+    error, exit 1, nothing written). A missing or empty file gets H first. Without
+    `--x-header`, line 1 is a row, as in 3.39. (The library's own header check, a
+    `header=` given to `readTabularFile`, `appendTabularFile`, `clearTabularFile`,
+    `scrubTabularFile` or `TSVZed`, is 3.39's, looser one: each cell of H within the
+    cell of line 1 under it, so `idaho\tvalue` passes for `id\tv`.) A rewrite without
+    `--x-header` can put a row equal to H on line 1
+    ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
   - `--x-strict` / `--x-force` turn 3.39's column and header checks on or off.
 
 ## Serving a store
@@ -193,7 +207,13 @@ status stay the same.
   and `fsync`s everything and removes `STORE.serve`.
 - **Bypassing it.** `--x-direct` makes a `tsvz` command use the files. Command
   lines with `--x-header`, `--x-defaults` or `--x-strict`, or another delimiter
-  than the server's, use the files too.
+  than the server's, use the files too. A server started with `--x-header H`
+  answers every request by the `--x-header` rule above, and one started with
+  `--x-strict` too answers as a direct `--x-strict` command does: a line 1 that is
+  not H fails the request (status 1, nothing written), and reads and scrubs drop
+  the rows of another width. It does not start when line 1 is not H then. The
+  table of a routed `read --format table` shows no H line: its first row is set
+  as the table's header, as with no `--x-header`.
 
 ```python
 import TSVZ
@@ -261,7 +281,9 @@ microseconds as a write starts is followed as a rotation instead); while that re
 fails or is refused, the rows are appended to a new file there, and memory keeps all of
 them for the rewrite that gets through. A compressed file another writer left damaged
 is made again too (L5: every read stops at the damage, so rows
-appended after it would be read by no one); the bytes after the damage go. While that
+appended after it would be read by no one): what the read did not get goes, the bytes after
+the damage and also the lines of the damaged member before it that the codec never handed
+over (Python's gzip gives none of a first member cut short, where `zcat` shows them). While that
 rewrite fails, the rows are appended after the damage and memory keeps them for the
 next rewrite that gets through; while it is refused (a row changed in place to text
 the encoding cannot hold), they are appended after it as in 4.9, and the next read
@@ -301,12 +323,184 @@ empty; the keys it sets or deletes then take its values, and the keys it left ou
 memory's. An explicit `reload()` while the file is damaged replaces memory with the
 damaged file, as asked, and warns.
 
+## Upgrading to 5.2 from 5.1
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.2 writes.
+
+- **A scrub of a `.tsv` whose rows were all deleted shrinks it.** `scrubTabularFile`,
+  `tsvz scrub` (also through a `tsvz serve` handler) of a `.tsv`-family file (`.tsv`,
+  `.csv`, `.nsv`, `.psv` and their compressed forms) in which every key has been deleted
+  leaves the file as an empty store: the header line it has, and its `#_defaults_#`
+  lines as a scrub of a file with rows keeps them (an empty one after the header, the
+  last non-empty one after it, both as wide as the header, or as the column count the
+  read took, which caller defaults set when given). A file with no header line keeps its
+  empty first line, or becomes 0 bytes when its first line was a row; a first line that
+  is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an
+  empty first line, which the scrub then writes. Comment lines go, as a scrub of a file
+  with rows already drops them. A scrub that is given
+  a header (and checks it, the default) keeps the file's first line as its header, an
+  empty one included, with a "Header mismatch" warning when it differs; the header it is
+  given is written only with `verifyHeader=False`, or when the file's first line cannot be
+  decoded (plain text named `.gz`); `tsvz scrub --x-header` follows the `--x-header` rule
+  below instead. A scrub of a path that is not there
+  makes no file and returns `{}`, as in 5.1, unless `createIfNotExist=True`. A compressed
+  file whose read decodes no row (plain text named `.gz`, a first member cut short) is
+  made again as an empty store, as a scrub of a damaged file with rows already was in
+  5.1: what the read did not get goes, the bytes after the damage and also the lines of the
+  damaged member before it that the codec never handed over (Python's gzip gives none of
+  a first member cut short, where `zcat` shows them) (see the damaged-stream notes above), and
+  `tsvz clear --x-header H --x-strict` (`clearTabularFile(verifyHeader=True)`) now clears
+  such a file, where 5.1 exited 1 with the codec's error.
+  5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that
+  appended keys, deleted them all and scrubbed, over and over, saw the file grow without
+  limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after
+  every scrub. A file with no byte (or a compressed one with only empty streams) is
+  still left alone, and so is a file whose strict scrub (`strict=True`, `--x-strict`) read
+  dropped a line for its width and found no row: that line is not a deletion, as 5.1 left
+  it. A scrub that finds the file changed since its read still reads
+  again and writes nothing if it keeps changing, as for a file with rows.
+- **`get_file_obj('ab')` on a compressed store writes its trailer before the lock is let
+  go.** A program that calls it on a `.tsv.gz`, `.bz2`, `.xz` or `.zst` store and writes
+  through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included,
+  in the file when `release_file_obj()` unlocks it. 5.1 and earlier wrote them after the
+  unlock, where another writer could append in between and corrupt the stream.
+- **A `TSVZed` no longer lets a file grow with every session of sets and deletes.** It
+  rewrites a `.tsv`-family file at open when the file holds a key's row and the deletion
+  of it, as 3.39 did only for a key with two values, so a program that sets keys and
+  deletes them in each session (default settings, or `rewrite_interval`) keeps the file
+  the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session
+  that opens such a file now rewrites it once. With `rewrite_on_load=False` the open
+  does not, and the next exit or periodic rewrite does it (`rewrite_on_exit=True`,
+  `rewrite_interval`). What such a rewrite does, it now does at a plain open:
+  - it cuts a row wider than the header, and the cut stays
+    ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)):
+    `id\tv\na\t1\t2\nb\t1\nb\t\n` becomes `id\tv\na\t1  \n`, and
+    `id\tv\na\t\t7\nb\t1\nb\t\n` becomes `id\tv\na\t  \n`, the 7 lost;
+  - it writes over the file in place and pads with spaces (`a\t1  ` above): TSVZ readers
+    strip them, tools that read the raw bytes (`awk`, `cut`) see them;
+  - on a file it cannot write (read-only), it prints `Failed to open file … Permission
+    denied` twice and leaves the file as it is, as 5.1 did for a key with two values.
+- **`--x-header H` makes line 1 the header for every `tsvz` operation.** On a
+  `.tsv`-family file whose line 1 is H (an exact match: the same cells, each without the
+  whitespace around it, a trailing empty cell included), line 1 is the header: `keys`,
+  `len`, `read`, `get`, `has`, `pop`, `popitem` and `delete` never list, count, print,
+  pop or delete it, and `scrub` and `clear` keep it. When line 1 differs, the file has no
+  header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and
+  the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases
+  below), and in such a file it never writes a row that is H as line 1: it puts an empty
+  line first (`a\t1\nid\tv\na\t\n` becomes `\nid\tv\n`), so the reads still take that row
+  for a row. With `--x-strict` a line 1 that is not H is an error (exit 1, nothing
+  written) for every operation, through a handler too. A missing or empty file gets H
+  first at its first write, as in 5.1. A `tsvz serve STORE --x-header H` handler applies
+  the rule to every request it answers, routed `tsvz` commands included; with
+  `--x-strict` it is strict for them as above (5.1 ignored `--x-strict` at `serve`): such
+  a handler refuses to start on a file whose line 1 is not H, its routed `.tsv` reads and
+  scrubs drop lines of another column count, and its `.tsvz` reads are strict, as for a
+  direct `--x-strict` command. The library's own header check (`header=` to
+  `readTabularFile`, `appendTabularFile`, `clearTabularFile`, `scrubTabularFile`,
+  `TSVZed`) is unchanged, 3.39's containment check. The CLI's writes check line 1 more
+  strictly than 5.1's did: 5.1's `set`, `append`, `delete` and `clear` with `--x-header`
+  silently took a line 1 that merely held H's cells (`id\tvalue` for `id\tv`) for the
+  header and fitted rows to H's width; now they warn and write the row at its own width,
+  and with `--x-strict` exit 1. A rewrite without `--x-header` (a scrub without it,
+  `scrubTabularFile`, a `TSVZed` rewrite) can still put a row equal to H on line 1 of a
+  header-less file
+  ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
+  In 5.1 only `set`, `append`, `delete` and `clear` used `--x-header`: the reads and
+  `scrub` took line 1 as a row (`tsvz keys`
+  printed `id a b` for two keys, `len` said 3, `popitem first` popped the header), and
+  so did every request to a handler. Without `--x-header` nothing changed: line 1 is a
+  row, as in 5.1 and 3.39. In a `.tsvz` store `pop`, `popitem` and `setdefault` now check
+  `--x-header` as `set` does; nothing else changed there.
+- **Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row,
+  and can lose cells or keys.** A scrub given defaults narrower than the rows, a row wider
+  than the header, and a `.tsv` that begins with a bare `#_defaults_#` line (read without
+  defaults) are examples; the entry lists each, and which is new in 5.2
+  ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)).
+  A strict scrub of a file whose read dropped a line for its width and found no row leaves it,
+  so it grows again over append, delete-all and scrub cycles
+  ([limitations](docs/limitations.md#a-strict-scrub-leaves-a-file-whose-read-dropped-a-line-for-its-width)).
+  Both wait for the column-width design after 5.2.
+
+## Upgrading to 5.1 from 5.0
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.1 writes. Most of the changes are in scrub and clear.
+
+- **Scrub and clear of a `.tsv` write over the file.** `scrubTabularFile`,
+  `clearTabularFile`, `tsvz scrub` and `tsvz clear` (on the command line or through a
+  `tsvz serve` handler), `TSVZClient.clear()` and `TSVZedLite.clear_file()` of a
+  `.tsv`-family file write the new content over the file and then cut it, as
+  `TSVZed`'s rewrites do, so a reader that takes no lock never meets the file emptied.
+  They take the file's lock, so they wait while another writer holds it. 5.0 emptied
+  the file first, with no lock (`TSVZedLite` truncated its own descriptor).
+- **A scrub keeps another process's rows.** A scrub of a `.tsv` writes only if the
+  file is still what its read saw. Otherwise it reads again, up to three times, and if
+  the file kept changing it writes nothing and warns: `tsvz scrub` exits 1 (`scrub
+  refused; nothing was written`), as for a `.tsvz` (C6), and a handler answers refused.
+  `scrubTabularFile` still returns the rows it read. 5.0 wrote over a row appended
+  after its read.
+- **`scrubTabularFile(lastLineOnly=True)` is a read.** It returns the last row and
+  writes nothing, as for a `.tsvz`. 5.0 and 3.39 wrote that row back as deletion lines
+  (`id\tv\na\t\n3\t\n`), and the store was gone.
+- **A scrub keeps the file's `#_defaults_#` lines.** The defaults in effect at the end
+  of the read are written after the rows: the last non-empty defaults line, its empty
+  cells filled from the line before it as a reader fills them. An empty defaults line
+  that was read (it turns defaults off) is written before the rows, padded to their
+  width. Every row reads as it did to a reader given the scrub's defaults and
+  `strict`. Defaults a caller passes still fill the rows' empty cells, as in 5.0, but
+  are not written as a defaults line, and the caller's list is not changed. 5.0 and
+  3.39 dropped the lines, so rows appended later got no defaults.
+- **The header stays on line 1.** A scrub given no header (`tsvz scrub` of a `.tsv`,
+  `scrubTabularFile(path)`) and a `TSVZed` given none, of a file
+  that has a header line, keep that line first, with a `TSVZed`'s defaults line right
+  after it. 5.0 and 3.39 put an empty line in front of it, so a reader given the header
+  read the header as a row. A header-less file keeps the empty first line it begins
+  with, and new files and clears given no header still get one, as in 5.0.
+- **Interrupts.** A scrub or clear of either dialect keeps a copy of only the bytes it
+  overwrites, to put back if it is interrupted before the file is cut to its new
+  length; once the cut is made the rewrite is done and the new content stays. 5.0 kept
+  a copy of the whole `.tsvz` part and put it all back. (A scrub still reads the whole
+  store, and a `.tsvz` clear reads it too, to keep its markers.)
+- **A compressed store cleared to nothing is one empty stream.** A `.tsvz.gz` (or
+  `.bz2`, `.xz`, `.zst`) cleared or scrubbed to nothing holds one empty compressed
+  stream, which `bzcat`, `xzcat` and Python's `bz2` and `lzma` can read; 5.0 left 0
+  bytes.
+- **A header-checking clear of an empty file writes the header it was given.** A clear
+  that checks the header (`verifyHeader=True`, `tsvz clear --x-header H --x-strict`) of a
+  0-byte `.tsv`, or of a compressed file with no byte or only empty streams (another
+  tool's), takes it for an empty store with that header, as every reader does (L9), and
+  writes H without a "Header mismatch" warning. 5.0 wrote an empty line and warned of a
+  mismatch, so a strict `TSVZed` given H then refused the file, and raised on a 0-byte
+  `.bz2`, `.xz` or `.zst`.
+- **`tsvz --help`** says what `clear` keeps: a `.tsvz` its header and markers, a
+  `.tsv` the `--x-header` it is given. (With `--x-strict` too, a `.tsv` clear keeps the
+  file's own header line.)
+  `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
+  a program asks for.
+- **Known in 5.1, fixed in 5.2: a scrub of a `.tsv` whose rows were all deleted did not
+  compact it** when the scrub was given the file's header (`scrubTabularFile(path, header=H)`)
+  or the file had no header line: it left the file as it was, deletion lines included;
+  reads were right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file read the header
+  line as a row, so it did compact it.)
+- **Known: a scrub run with other defaults than the store's program can change how
+  early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
+  cells before any defaults line, the reset line the scrub writes before the rows keeps
+  those cells empty for a program that opens the store with its own defaults; run the
+  scrub with the defaults that program uses
+  ([limitations](docs/limitations.md#a-scrub-run-with-other-defaults-than-the-stores-program-can-change-how-early-rows-read)).
+- **Running programs keep 5.0** until they are restarted: long-running programs that
+  import TSVZ, and running `tsvz serve` handlers.
+
 ## Upgrading to 5.0 from 4.9
 
 - **Writes refuse text they cannot encode.** A key, value, header, defaults line or
   marker the file's encoding cannot hold (UTF-8 for `.tsvz`; `encoding=` for a
   `.tsv`), such as the lone surrogate `os.fsdecode` gives for a non-UTF-8 file name,
-  raises `TSVZEncodeError` (a `ValueError`) and writes nothing; 4.9 stored `?`.
+  raises `TSVZEncodeError` (a `ValueError`) and writes nothing; 4.9 stored `?` (its
+  `tsvz` stored U+FFFD for a non-UTF-8 argument, and its `.tsvz` file functions raised
+  `UnicodeEncodeError`).
   `errors='replace'` brings `?` back. Under a non-UTF-8 `encoding=`, a key read as
   U+FFFD cannot be deleted or updated. `tsvz` exits 1 with the message. See Fault
   tolerance.
@@ -324,8 +518,7 @@ damaged file, as asked, and warns.
   merged into it (Fault tolerance).
 - **`.tsvz` stores** see another process's append made right after their load, a
   commit or a clear (4.9 missed it until the next change), keep the rows set while
-  they reload, refuse `#_snapshot_#` (spec §19.10.2), and raise a strict header
-  mismatch at the first read.
+  they reload, and raise a strict header mismatch at the first read.
 - **`.tsv` files:** an append to an existing empty file writes the header first, and
   reading one gives an empty store; see L8 to L14 for the rest.
 - **`tsvz`** runs every operation through the engine `tsvz serve` uses, with the same
@@ -334,8 +527,15 @@ damaged file, as asked, and warns.
   file. The 3.39 warning that a file name does not end in `.tsv` is printed only
   when a command creates such a file, and a write that cannot be made reports the
   operating system's error.
-- **macOS and other systems that lock with `lockf`:** 4.9's close gate is gone, so
-  a read in another thread of the process can drop a write's lock
+- **Running programs keep 4.9.** Long-running programs that import TSVZ, and running
+  `tsvz serve` handlers, keep 4.9 until they are restarted. 5.0's `tsvz stop` cannot
+  stop a 4.9 handler of a `.tsv` (or other tab- or NUL-delimited) file: 4.9 wrote the
+  delimiter into the pointer file's `x-delimiter` as `<sep>` (a NUL as `<NUL>`), which
+  5.0 decodes to the TAB or NUL itself and refuses as a control character. Send that
+  handler SIGTERM instead: its pid is on the `pid` line of its pointer file
+  (`STORE.serve`), and it finishes its writes and exits.
+- **macOS and other systems that lock with `lockf`:** as in 4.9, a read in another
+  thread of the process can still drop a write's lock
   ([docs/limitations.md](docs/limitations.md)).
 
 ## Breaking changes from 3.39
@@ -358,6 +558,11 @@ damaged file, as asked, and warns.
 | L12 | A rewrite of a file whose header line has no newline writes the rows on lines of their own; 3.39 wrote the first one onto the header line. |
 | L13 | A compressed file is rewritten in members of at most 1 MiB, as appends already were: the bytes differ, the rows read are the same, and 3.39 reads them. |
 | L14 | `TSVZed` opening a compressed file in a codec this Python lacks (zstd before Python 3.14) raises `ImportError` instead of reading the compressed bytes as text. |
+| L15 | `scrubTabularFile`, `clearTabularFile`, `tsvz scrub`, `tsvz clear` and `TSVZedLite.clear_file()` write the new content over the file and then cut it, under the file's lock; 3.39 emptied the file first, with no lock, so a reader could meet it empty. |
+| L16 | A scrub writes only if the file is still what its read saw: it reads again up to three times, then writes nothing, with a warning (`tsvz scrub` exits 1, C6); 3.39 wrote over rows appended after its read. |
+| L17 | `scrubTabularFile(lastLineOnly=True)` returns the last row and writes nothing; 3.39 wrote that row back as deletion lines, and the store was gone. |
+| L18 | A scrub keeps the file's `#_defaults_#` lines (an empty one before the rows, the defaults in effect after them), so every row reads as it did; 3.39 dropped them, so rows appended later got no defaults. |
+| L19 | A scrub given no header, and a `TSVZed` given none, keep a file's header line first; 3.39 put an empty line in front of it, and a reader given the header then read the header as a row. |
 
 ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
 

{tsvz-5.0 → tsvz-5.2}/README.md

@@ -7,7 +7,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 5.0**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 5.1**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -131,6 +131,20 @@ named like an operation then needs a path prefix (`./read`).
   with a warning.
 - **TSVZ extensions:**
   - `--x-header H` and `--x-defaults KEY<d>D1<d>...` decode backslash escapes such as `\t`;
+  - `--x-header H` makes line 1 of a `.tsv`-family file its header for every operation
+    when line 1 is H: the same number of cells, each equal to H's once the whitespace
+    around it is taken off (a trailing empty cell counts: `id\tv\t` is not `id\tv`; a UTF-8
+    byte order mark before line 1, as Excel writes one, is skipped). It
+    is then never a key, row or count, `scrub` and `clear` keep it, and
+    `read --format table` shows it above the rows. When line 1 differs, the file has no
+    header and line 1 is a row, with 3.39's "Header mismatch" warning (`--x-strict`: an
+    error, exit 1, nothing written). A missing or empty file gets H first. Without
+    `--x-header`, line 1 is a row, as in 3.39. (The library's own header check, a
+    `header=` given to `readTabularFile`, `appendTabularFile`, `clearTabularFile`,
+    `scrubTabularFile` or `TSVZed`, is 3.39's, looser one: each cell of H within the
+    cell of line 1 under it, so `idaho\tvalue` passes for `id\tv`.) A rewrite without
+    `--x-header` can put a row equal to H on line 1
+    ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
   - `--x-strict` / `--x-force` turn 3.39's column and header checks on or off.
 
 ## Serving a store
@@ -161,7 +175,13 @@ status stay the same.
   and `fsync`s everything and removes `STORE.serve`.
 - **Bypassing it.** `--x-direct` makes a `tsvz` command use the files. Command
   lines with `--x-header`, `--x-defaults` or `--x-strict`, or another delimiter
-  than the server's, use the files too.
+  than the server's, use the files too. A server started with `--x-header H`
+  answers every request by the `--x-header` rule above, and one started with
+  `--x-strict` too answers as a direct `--x-strict` command does: a line 1 that is
+  not H fails the request (status 1, nothing written), and reads and scrubs drop
+  the rows of another width. It does not start when line 1 is not H then. The
+  table of a routed `read --format table` shows no H line: its first row is set
+  as the table's header, as with no `--x-header`.
 
 ```python
 import TSVZ
@@ -229,7 +249,9 @@ microseconds as a write starts is followed as a rotation instead); while that re
 fails or is refused, the rows are appended to a new file there, and memory keeps all of
 them for the rewrite that gets through. A compressed file another writer left damaged
 is made again too (L5: every read stops at the damage, so rows
-appended after it would be read by no one); the bytes after the damage go. While that
+appended after it would be read by no one): what the read did not get goes, the bytes after
+the damage and also the lines of the damaged member before it that the codec never handed
+over (Python's gzip gives none of a first member cut short, where `zcat` shows them). While that
 rewrite fails, the rows are appended after the damage and memory keeps them for the
 next rewrite that gets through; while it is refused (a row changed in place to text
 the encoding cannot hold), they are appended after it as in 4.9, and the next read
@@ -269,12 +291,184 @@ empty; the keys it sets or deletes then take its values, and the keys it left ou
 memory's. An explicit `reload()` while the file is damaged replaces memory with the
 damaged file, as asked, and warns.
 
+## Upgrading to 5.2 from 5.1
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.2 writes.
+
+- **A scrub of a `.tsv` whose rows were all deleted shrinks it.** `scrubTabularFile`,
+  `tsvz scrub` (also through a `tsvz serve` handler) of a `.tsv`-family file (`.tsv`,
+  `.csv`, `.nsv`, `.psv` and their compressed forms) in which every key has been deleted
+  leaves the file as an empty store: the header line it has, and its `#_defaults_#`
+  lines as a scrub of a file with rows keeps them (an empty one after the header, the
+  last non-empty one after it, both as wide as the header, or as the column count the
+  read took, which caller defaults set when given). A file with no header line keeps its
+  empty first line, or becomes 0 bytes when its first line was a row; a first line that
+  is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an
+  empty first line, which the scrub then writes. Comment lines go, as a scrub of a file
+  with rows already drops them. A scrub that is given
+  a header (and checks it, the default) keeps the file's first line as its header, an
+  empty one included, with a "Header mismatch" warning when it differs; the header it is
+  given is written only with `verifyHeader=False`, or when the file's first line cannot be
+  decoded (plain text named `.gz`); `tsvz scrub --x-header` follows the `--x-header` rule
+  below instead. A scrub of a path that is not there
+  makes no file and returns `{}`, as in 5.1, unless `createIfNotExist=True`. A compressed
+  file whose read decodes no row (plain text named `.gz`, a first member cut short) is
+  made again as an empty store, as a scrub of a damaged file with rows already was in
+  5.1: what the read did not get goes, the bytes after the damage and also the lines of the
+  damaged member before it that the codec never handed over (Python's gzip gives none of
+  a first member cut short, where `zcat` shows them) (see the damaged-stream notes above), and
+  `tsvz clear --x-header H --x-strict` (`clearTabularFile(verifyHeader=True)`) now clears
+  such a file, where 5.1 exited 1 with the codec's error.
+  5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that
+  appended keys, deleted them all and scrubbed, over and over, saw the file grow without
+  limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after
+  every scrub. A file with no byte (or a compressed one with only empty streams) is
+  still left alone, and so is a file whose strict scrub (`strict=True`, `--x-strict`) read
+  dropped a line for its width and found no row: that line is not a deletion, as 5.1 left
+  it. A scrub that finds the file changed since its read still reads
+  again and writes nothing if it keeps changing, as for a file with rows.
+- **`get_file_obj('ab')` on a compressed store writes its trailer before the lock is let
+  go.** A program that calls it on a `.tsv.gz`, `.bz2`, `.xz` or `.zst` store and writes
+  through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included,
+  in the file when `release_file_obj()` unlocks it. 5.1 and earlier wrote them after the
+  unlock, where another writer could append in between and corrupt the stream.
+- **A `TSVZed` no longer lets a file grow with every session of sets and deletes.** It
+  rewrites a `.tsv`-family file at open when the file holds a key's row and the deletion
+  of it, as 3.39 did only for a key with two values, so a program that sets keys and
+  deletes them in each session (default settings, or `rewrite_interval`) keeps the file
+  the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session
+  that opens such a file now rewrites it once. With `rewrite_on_load=False` the open
+  does not, and the next exit or periodic rewrite does it (`rewrite_on_exit=True`,
+  `rewrite_interval`). What such a rewrite does, it now does at a plain open:
+  - it cuts a row wider than the header, and the cut stays
+    ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)):
+    `id\tv\na\t1\t2\nb\t1\nb\t\n` becomes `id\tv\na\t1  \n`, and
+    `id\tv\na\t\t7\nb\t1\nb\t\n` becomes `id\tv\na\t  \n`, the 7 lost;
+  - it writes over the file in place and pads with spaces (`a\t1  ` above): TSVZ readers
+    strip them, tools that read the raw bytes (`awk`, `cut`) see them;
+  - on a file it cannot write (read-only), it prints `Failed to open file … Permission
+    denied` twice and leaves the file as it is, as 5.1 did for a key with two values.
+- **`--x-header H` makes line 1 the header for every `tsvz` operation.** On a
+  `.tsv`-family file whose line 1 is H (an exact match: the same cells, each without the
+  whitespace around it, a trailing empty cell included), line 1 is the header: `keys`,
+  `len`, `read`, `get`, `has`, `pop`, `popitem` and `delete` never list, count, print,
+  pop or delete it, and `scrub` and `clear` keep it. When line 1 differs, the file has no
+  header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and
+  the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases
+  below), and in such a file it never writes a row that is H as line 1: it puts an empty
+  line first (`a\t1\nid\tv\na\t\n` becomes `\nid\tv\n`), so the reads still take that row
+  for a row. With `--x-strict` a line 1 that is not H is an error (exit 1, nothing
+  written) for every operation, through a handler too. A missing or empty file gets H
+  first at its first write, as in 5.1. A `tsvz serve STORE --x-header H` handler applies
+  the rule to every request it answers, routed `tsvz` commands included; with
+  `--x-strict` it is strict for them as above (5.1 ignored `--x-strict` at `serve`): such
+  a handler refuses to start on a file whose line 1 is not H, its routed `.tsv` reads and
+  scrubs drop lines of another column count, and its `.tsvz` reads are strict, as for a
+  direct `--x-strict` command. The library's own header check (`header=` to
+  `readTabularFile`, `appendTabularFile`, `clearTabularFile`, `scrubTabularFile`,
+  `TSVZed`) is unchanged, 3.39's containment check. The CLI's writes check line 1 more
+  strictly than 5.1's did: 5.1's `set`, `append`, `delete` and `clear` with `--x-header`
+  silently took a line 1 that merely held H's cells (`id\tvalue` for `id\tv`) for the
+  header and fitted rows to H's width; now they warn and write the row at its own width,
+  and with `--x-strict` exit 1. A rewrite without `--x-header` (a scrub without it,
+  `scrubTabularFile`, a `TSVZed` rewrite) can still put a row equal to H on line 1 of a
+  header-less file
+  ([limitations](docs/limitations.md#a-rewrite-without---x-header-can-put-a-row-equal-to-h-on-line-1)).
+  In 5.1 only `set`, `append`, `delete` and `clear` used `--x-header`: the reads and
+  `scrub` took line 1 as a row (`tsvz keys`
+  printed `id a b` for two keys, `len` said 3, `popitem first` popped the header), and
+  so did every request to a handler. Without `--x-header` nothing changed: line 1 is a
+  row, as in 5.1 and 3.39. In a `.tsvz` store `pop`, `popitem` and `setdefault` now check
+  `--x-header` as `set` does; nothing else changed there.
+- **Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row,
+  and can lose cells or keys.** A scrub given defaults narrower than the rows, a row wider
+  than the header, and a `.tsv` that begins with a bare `#_defaults_#` line (read without
+  defaults) are examples; the entry lists each, and which is new in 5.2
+  ([limitations](docs/limitations.md#scrubs-and-rewrites-read-rows-at-one-column-width-deferred-to-the-width-design)).
+  A strict scrub of a file whose read dropped a line for its width and found no row leaves it,
+  so it grows again over append, delete-all and scrub cycles
+  ([limitations](docs/limitations.md#a-strict-scrub-leaves-a-file-whose-read-dropped-a-line-for-its-width)).
+  Both wait for the column-width design after 5.2.
+
+## Upgrading to 5.1 from 5.0
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.1 writes. Most of the changes are in scrub and clear.
+
+- **Scrub and clear of a `.tsv` write over the file.** `scrubTabularFile`,
+  `clearTabularFile`, `tsvz scrub` and `tsvz clear` (on the command line or through a
+  `tsvz serve` handler), `TSVZClient.clear()` and `TSVZedLite.clear_file()` of a
+  `.tsv`-family file write the new content over the file and then cut it, as
+  `TSVZed`'s rewrites do, so a reader that takes no lock never meets the file emptied.
+  They take the file's lock, so they wait while another writer holds it. 5.0 emptied
+  the file first, with no lock (`TSVZedLite` truncated its own descriptor).
+- **A scrub keeps another process's rows.** A scrub of a `.tsv` writes only if the
+  file is still what its read saw. Otherwise it reads again, up to three times, and if
+  the file kept changing it writes nothing and warns: `tsvz scrub` exits 1 (`scrub
+  refused; nothing was written`), as for a `.tsvz` (C6), and a handler answers refused.
+  `scrubTabularFile` still returns the rows it read. 5.0 wrote over a row appended
+  after its read.
+- **`scrubTabularFile(lastLineOnly=True)` is a read.** It returns the last row and
+  writes nothing, as for a `.tsvz`. 5.0 and 3.39 wrote that row back as deletion lines
+  (`id\tv\na\t\n3\t\n`), and the store was gone.
+- **A scrub keeps the file's `#_defaults_#` lines.** The defaults in effect at the end
+  of the read are written after the rows: the last non-empty defaults line, its empty
+  cells filled from the line before it as a reader fills them. An empty defaults line
+  that was read (it turns defaults off) is written before the rows, padded to their
+  width. Every row reads as it did to a reader given the scrub's defaults and
+  `strict`. Defaults a caller passes still fill the rows' empty cells, as in 5.0, but
+  are not written as a defaults line, and the caller's list is not changed. 5.0 and
+  3.39 dropped the lines, so rows appended later got no defaults.
+- **The header stays on line 1.** A scrub given no header (`tsvz scrub` of a `.tsv`,
+  `scrubTabularFile(path)`) and a `TSVZed` given none, of a file
+  that has a header line, keep that line first, with a `TSVZed`'s defaults line right
+  after it. 5.0 and 3.39 put an empty line in front of it, so a reader given the header
+  read the header as a row. A header-less file keeps the empty first line it begins
+  with, and new files and clears given no header still get one, as in 5.0.
+- **Interrupts.** A scrub or clear of either dialect keeps a copy of only the bytes it
+  overwrites, to put back if it is interrupted before the file is cut to its new
+  length; once the cut is made the rewrite is done and the new content stays. 5.0 kept
+  a copy of the whole `.tsvz` part and put it all back. (A scrub still reads the whole
+  store, and a `.tsvz` clear reads it too, to keep its markers.)
+- **A compressed store cleared to nothing is one empty stream.** A `.tsvz.gz` (or
+  `.bz2`, `.xz`, `.zst`) cleared or scrubbed to nothing holds one empty compressed
+  stream, which `bzcat`, `xzcat` and Python's `bz2` and `lzma` can read; 5.0 left 0
+  bytes.
+- **A header-checking clear of an empty file writes the header it was given.** A clear
+  that checks the header (`verifyHeader=True`, `tsvz clear --x-header H --x-strict`) of a
+  0-byte `.tsv`, or of a compressed file with no byte or only empty streams (another
+  tool's), takes it for an empty store with that header, as every reader does (L9), and
+  writes H without a "Header mismatch" warning. 5.0 wrote an empty line and warned of a
+  mismatch, so a strict `TSVZed` given H then refused the file, and raised on a 0-byte
+  `.bz2`, `.xz` or `.zst`.
+- **`tsvz --help`** says what `clear` keeps: a `.tsvz` its header and markers, a
+  `.tsv` the `--x-header` it is given. (With `--x-strict` too, a `.tsv` clear keeps the
+  file's own header line.)
+  `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
+  a program asks for.
+- **Known in 5.1, fixed in 5.2: a scrub of a `.tsv` whose rows were all deleted did not
+  compact it** when the scrub was given the file's header (`scrubTabularFile(path, header=H)`)
+  or the file had no header line: it left the file as it was, deletion lines included;
+  reads were right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file read the header
+  line as a row, so it did compact it.)
+- **Known: a scrub run with other defaults than the store's program can change how
+  early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
+  cells before any defaults line, the reset line the scrub writes before the rows keeps
+  those cells empty for a program that opens the store with its own defaults; run the
+  scrub with the defaults that program uses
+  ([limitations](docs/limitations.md#a-scrub-run-with-other-defaults-than-the-stores-program-can-change-how-early-rows-read)).
+- **Running programs keep 5.0** until they are restarted: long-running programs that
+  import TSVZ, and running `tsvz serve` handlers.
+
 ## Upgrading to 5.0 from 4.9
 
 - **Writes refuse text they cannot encode.** A key, value, header, defaults line or
   marker the file's encoding cannot hold (UTF-8 for `.tsvz`; `encoding=` for a
   `.tsv`), such as the lone surrogate `os.fsdecode` gives for a non-UTF-8 file name,
-  raises `TSVZEncodeError` (a `ValueError`) and writes nothing; 4.9 stored `?`.
+  raises `TSVZEncodeError` (a `ValueError`) and writes nothing; 4.9 stored `?` (its
+  `tsvz` stored U+FFFD for a non-UTF-8 argument, and its `.tsvz` file functions raised
+  `UnicodeEncodeError`).
   `errors='replace'` brings `?` back. Under a non-UTF-8 `encoding=`, a key read as
   U+FFFD cannot be deleted or updated. `tsvz` exits 1 with the message. See Fault
   tolerance.
@@ -292,8 +486,7 @@ damaged file, as asked, and warns.
   merged into it (Fault tolerance).
 - **`.tsvz` stores** see another process's append made right after their load, a
   commit or a clear (4.9 missed it until the next change), keep the rows set while
-  they reload, refuse `#_snapshot_#` (spec §19.10.2), and raise a strict header
-  mismatch at the first read.
+  they reload, and raise a strict header mismatch at the first read.
 - **`.tsv` files:** an append to an existing empty file writes the header first, and
   reading one gives an empty store; see L8 to L14 for the rest.
 - **`tsvz`** runs every operation through the engine `tsvz serve` uses, with the same
@@ -302,8 +495,15 @@ damaged file, as asked, and warns.
   file. The 3.39 warning that a file name does not end in `.tsv` is printed only
   when a command creates such a file, and a write that cannot be made reports the
   operating system's error.
-- **macOS and other systems that lock with `lockf`:** 4.9's close gate is gone, so
-  a read in another thread of the process can drop a write's lock
+- **Running programs keep 4.9.** Long-running programs that import TSVZ, and running
+  `tsvz serve` handlers, keep 4.9 until they are restarted. 5.0's `tsvz stop` cannot
+  stop a 4.9 handler of a `.tsv` (or other tab- or NUL-delimited) file: 4.9 wrote the
+  delimiter into the pointer file's `x-delimiter` as `<sep>` (a NUL as `<NUL>`), which
+  5.0 decodes to the TAB or NUL itself and refuses as a control character. Send that
+  handler SIGTERM instead: its pid is on the `pid` line of its pointer file
+  (`STORE.serve`), and it finishes its writes and exits.
+- **macOS and other systems that lock with `lockf`:** as in 4.9, a read in another
+  thread of the process can still drop a write's lock
   ([docs/limitations.md](docs/limitations.md)).
 
 ## Breaking changes from 3.39
@@ -326,6 +526,11 @@ damaged file, as asked, and warns.
 | L12 | A rewrite of a file whose header line has no newline writes the rows on lines of their own; 3.39 wrote the first one onto the header line. |
 | L13 | A compressed file is rewritten in members of at most 1 MiB, as appends already were: the bytes differ, the rows read are the same, and 3.39 reads them. |
 | L14 | `TSVZed` opening a compressed file in a codec this Python lacks (zstd before Python 3.14) raises `ImportError` instead of reading the compressed bytes as text. |
+| L15 | `scrubTabularFile`, `clearTabularFile`, `tsvz scrub`, `tsvz clear` and `TSVZedLite.clear_file()` write the new content over the file and then cut it, under the file's lock; 3.39 emptied the file first, with no lock, so a reader could meet it empty. |
+| L16 | A scrub writes only if the file is still what its read saw: it reads again up to three times, then writes nothing, with a warning (`tsvz scrub` exits 1, C6); 3.39 wrote over rows appended after its read. |
+| L17 | `scrubTabularFile(lastLineOnly=True)` returns the last row and writes nothing; 3.39 wrote that row back as deletion lines, and the store was gone. |
+| L18 | A scrub keeps the file's `#_defaults_#` lines (an empty one before the rows, the defaults in effect after them), so every row reads as it did; 3.39 dropped them, so rows appended later got no defaults. |
+| L19 | A scrub given no header, and a `TSVZed` given none, keep a file's header line first; 3.39 put an empty line in front of it, and a reader given the header then read the header as a row. |
 
 ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1