TSVZ 4.9__tar.gz → 5.1__tar.gz

{tsvz-4.9 → tsvz-5.1}/PKG-INFO

@@ -1,13 +1,16 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 4.9
+Version: 5.1
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
-Home-page: https://github.com/yufei-pan/TSVZ
+Home-page: https://github.com/tsvz-org/TSVZ
 Author: Yufei Pan
 Author-email: pan@zopyr.us
 License: GPLv3+
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.14
 Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.6
@@ -36,7 +39,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.9**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 5.1**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -48,7 +51,7 @@ pip install -e .          # editable, from this directory
 
 | Extension | Rules |
 |---|---|
-| `.tsv` `.csv` `.nsv` `.psv` (and any other) | The TSVZ **3.39** format, unchanged. A 3.39 host can read every file 4.x writes. |
+| `.tsv` `.csv` `.nsv` `.psv` (and any other) | The TSVZ **3.39** format, unchanged. A 3.39 host can read every file this version writes. |
 | `.tsvz` `.csvz` `.nsvz` `.psvz` | The **[tsvz-spec-v1](tsvz-spec-v1.md)** append-only key-value log. |
 
 The delimiter follows the extension: tab, comma, NUL, pipe. A trailing
@@ -94,7 +97,7 @@ tsvz people.tsvz append carol Carol 5     # the 3.39 form still works
 tsvz -V
 ```
 
-## What TSVZ 4 does with `.tsvz` files
+## What TSVZ does with `.tsvz` files
 
 - **Reads** everything a conformant writer can produce: markers
   (`#_defaults_#`, `#_strip_trailing_whites_#`, `#_fill_empty_with_default_#`,
@@ -121,7 +124,12 @@ tsvz -V
   `setdefault` and `pop` behave like a dict.
 - **`#` keys** are stored (`<#>key`). Keys of the form `#_name_#` are marker
   writes: `t['#_defaults_#'] = [...]` sets the defaults, and so does
-  `setDefaults([...])` once the object is constructed.
+  `setDefaults([...])` once the object is constructed. `#_snapshot_#` is
+  refused (spec §19.10.2).
+- **Other writers:** a `TSVZed` notices another process's change by the active
+  part's inode, size and times and by the list of parts, and reloads. Rows it
+  has queued but not yet written stay in memory across that reload, and a live
+  key is never missing while it runs.
 
 ## Command line
 
@@ -130,6 +138,9 @@ tsvz -V
 lone `--`. The 3.39 form `tsvz STORE [OPERATION] [ARG ...]` works too; a store
 named like an operation then needs a path prefix (`./read`).
 
+- **One engine.** Every operation runs through the engine `tsvz serve` uses,
+  in its own process when no server runs, so a command prints the same with or
+  without a server.
 - **stdout carries only rows.** It gets a table on a terminal and records
   otherwise; `--format table|records` overrides. Records use the store's
   delimiter and escaping, so `tsvz read A | tsvz set B -` copies a store of
@@ -198,7 +209,8 @@ with TSVZ.TSVZClient('people.tsvz') as people:  # served, or in this process whe
 `setDefaults`. Its writes return once they are written (raising `OSError` if they
 cannot be), one client may be shared between threads, and without a server it
 warns once and works on the files in-process, finishing its writes when the
-program exits.
+program exits. A server of a `.tsv`-family file names its delimiter in
+`STORE.serve`, and `TSVZClient` splits string values on it.
 
 ## Fault tolerance
 
@@ -219,9 +231,195 @@ recovered, skipped or stops the read, as the store's `#_on_corruption_#` says,
 part keeps the original as `<part>.damaged-<time>`), an unreadable part, and a
 delimiter or encoding argument that conflicts with the extension.
 
+Writing is stricter than reading: a key, value, header or marker that cannot be
+encoded as UTF-8, such as the lone surrogate `os.fsdecode` gives for a
+non-UTF-8 file name, raises a `ValueError` (`TSVZEncodeError`) that names it
+and writes nothing, instead of storing `?`. Pass `errors='replace'` (or any codec
+error handler) to the writing functions or to `TSVZed`, `TSVZedLite` and `TSVZClient`
+to get the old `?`. `TSVZed` and `TSVZedLite` refuse such a row when it is set. A
+rewrite that meets such text leaves the file as it is: `mapToFile()` and
+`hardMapToFile()` raise the error, and the rewrites `TSVZed` makes itself report it
+once. With a non-UTF-8 `encoding=`, U+FFFD (what the reader puts for a byte it could
+not decode) is text that encoding may lack, and it is refused like any other: such a
+store is not rewritten (`rewrite_on_load` leaves the file as it is, without failing),
+appends of rows that can be written still work, and the delete or update of a key that
+holds U+FFFD raises. Such a key stays until the file is edited, the
+store cleared, or the store opened with the encoding it was written in; `errors='replace'`
+would write `?` for U+FFFD, losing the bytes the file holds there. The `tsvz` command
+exits 1 with the message.
+
+A `TSVZed` writes a `.tsv`, `.csv`, `.nsv` or `.psv` file (compressed or not) from
+its background thread only: `rewrite()`, `mapToFile()`, `hardMapToFile()`,
+`commitAppendToFile()`, `clear()`, `clear_file()`, `reload()`, `close()` and the
+constructor's load hand it the job and wait, so a Ctrl-C stops the wait, never the
+write (the call still raises `KeyboardInterrupt`). At exit a write in progress is
+finished: a Ctrl-C then says so and keeps waiting, and a second one abandons it. A
+closed store is freed. (`get_file_obj()` and `release_file_obj()`, kept from 3.39,
+hand the locked file to the calling thread.) A file another process removed is made
+again from memory by the store's next write, an append included (a removal in the
+microseconds as a write starts is followed as a rotation instead); while that rewrite
+fails or is refused, the rows are appended to a new file there, and memory keeps all of
+them for the rewrite that gets through. A compressed file another writer left damaged
+is made again too (L5: every read stops at the damage, so rows
+appended after it would be read by no one); the bytes after the damage go. While that
+rewrite fails, the rows are appended after the damage and memory keeps them for the
+next rewrite that gets through; while it is refused (a row changed in place to text
+the encoding cannot hold), they are appended after it as in 4.9, and the next read
+drops them. Either way `close()` tries the rewrite once more, whatever
+`rewrite_on_exit` says. Appends that do not read the store first (`tsvz set`,
+`appendTabularFile`, `TSVZedLite`, and a `TSVZed` with `monitor_external_changes=False`
+for damage that lands after it loaded the file) still land after it, and so do a
+`TSVZed`'s rows when a writer that takes no lock leaves the damage between the store's read
+and its append ([docs/limitations.md](docs/limitations.md)).
+
+It knows the size and SHA-256 of the bytes it last read or wrote. When another process
+has changed the file, and it is the same file (device and inode) and still begins with
+those bytes, nothing this store set is still waiting to be written, and
+`setDefaults()` has not changed its defaults since, only what follows is read and
+applied to memory, as a read of the file applies it: memory keeps an order set by
+`move_to_end`, and its `#` keys. Any other change reloads the file, changing memory
+in place, so a live key is never missing meanwhile.
+
+A rewrite writes over the file and then cuts it to its new length, so a reader never
+meets it emptied. One that fails part way (a full disk, an I/O error), or that cannot
+make again a file that was removed, leaves the file damaged: the new bytes it wrote,
+then the old ones after them. Memory then holds the only good copy, as it does when
+the store appends after damage another writer left that it cannot make the file again
+over yet (that rewrite failed, or waits out a failed one's back-off). Until a rewrite gets through, the file is not
+reloaded over memory; the background writer still appends the rows set meanwhile, and
+the next rewrite (the background writer's, `rewrite()`, `close()`) makes the file
+again from memory. A change another process
+made meanwhile is merged into memory first, as above, when the file is still the one
+the damage was left in (for a file that was removed, any file made there since) and
+still begins with exactly the bytes it left; after a damage that left the file empty
+(or removed it), what follows must also begin with this store's header.
+Otherwise that change is overwritten, with the warning `Overwriting external changes`.
+What cannot be told from an append is merged as one: another process's rewrite of the
+same file in place that begins with exactly the bytes the damage left, and, for a store
+with no header (or `verifyHeader=False`), anything written to a file the damage left
+empty; the keys it sets or deletes then take its values, and the keys it left out keep
+memory's. An explicit `reload()` while the file is damaged replaces memory with the
+damaged file, as asked, and warns.
+
+## Upgrading to 5.1 from 5.0
+
+The file formats are unchanged, and a 3.39 host still reads every `.tsv`-family file
+5.1 writes. Most of the changes are in scrub and clear.
+
+- **Scrub and clear of a `.tsv` write over the file.** `scrubTabularFile`,
+  `clearTabularFile`, `tsvz scrub` and `tsvz clear` (on the command line or through a
+  `tsvz serve` handler), `TSVZClient.clear()` and `TSVZedLite.clear_file()` of a
+  `.tsv`-family file write the new content over the file and then cut it, as
+  `TSVZed`'s rewrites do, so a reader that takes no lock never meets the file emptied.
+  They take the file's lock, so they wait while another writer holds it. 5.0 emptied
+  the file first, with no lock (`TSVZedLite` truncated its own descriptor).
+- **A scrub keeps another process's rows.** A scrub of a `.tsv` writes only if the
+  file is still what its read saw. Otherwise it reads again, up to three times, and if
+  the file kept changing it writes nothing and warns: `tsvz scrub` exits 1 (`scrub
+  refused; nothing was written`), as for a `.tsvz` (C6), and a handler answers refused.
+  `scrubTabularFile` still returns the rows it read. 5.0 wrote over a row appended
+  after its read.
+- **`scrubTabularFile(lastLineOnly=True)` is a read.** It returns the last row and
+  writes nothing, as for a `.tsvz`. 5.0 and 3.39 wrote that row back as deletion lines
+  (`id\tv\na\t\n3\t\n`), and the store was gone.
+- **A scrub keeps the file's `#_defaults_#` lines.** The defaults in effect at the end
+  of the read are written after the rows: the last non-empty defaults line, its empty
+  cells filled from the line before it as a reader fills them. An empty defaults line
+  that was read (it turns defaults off) is written before the rows, padded to their
+  width. Every row reads as it did to a reader given the scrub's defaults and
+  `strict`. Defaults a caller passes still fill the rows' empty cells, as in 5.0, but
+  are not written as a defaults line, and the caller's list is not changed. 5.0 and
+  3.39 dropped the lines, so rows appended later got no defaults.
+- **The header stays on line 1.** A scrub given no header (`tsvz scrub` of a `.tsv`,
+  `scrubTabularFile(path)`) and a `TSVZed` given none, of a file
+  that has a header line, keep that line first, with a `TSVZed`'s defaults line right
+  after it. 5.0 and 3.39 put an empty line in front of it, so a reader given the header
+  read the header as a row. A header-less file keeps the empty first line it begins
+  with, and new files and clears given no header still get one, as in 5.0.
+- **Interrupts.** A scrub or clear of either dialect keeps a copy of only the bytes it
+  overwrites, to put back if it is interrupted before the file is cut to its new
+  length; once the cut is made the rewrite is done and the new content stays. 5.0 kept
+  a copy of the whole `.tsvz` part and put it all back. (A scrub still reads the whole
+  store, and a `.tsvz` clear reads it too, to keep its markers.)
+- **A compressed store cleared to nothing is one empty stream.** A `.tsvz.gz` (or
+  `.bz2`, `.xz`, `.zst`) cleared or scrubbed to nothing holds one empty compressed
+  stream, which `bzcat`, `xzcat` and Python's `bz2` and `lzma` can read; 5.0 left 0
+  bytes.
+- **A header-checking clear of an empty file writes the header it was given.** A clear
+  that checks the header (`verifyHeader=True`, `tsvz clear --x-header H --x-strict`) of a
+  0-byte `.tsv`, or of a compressed file with no byte or only empty streams (another
+  tool's), takes it for an empty store with that header, as every reader does (L9), and
+  writes H without a "Header mismatch" warning. 5.0 wrote an empty line and warned of a
+  mismatch, so a strict `TSVZed` given H then refused the file, and raised on a 0-byte
+  `.bz2`, `.xz` or `.zst`.
+- **`tsvz --help`** says what `clear` keeps: a `.tsvz` its header and markers, a
+  `.tsv` the `--x-header` it is given. (With `--x-strict` too, a `.tsv` clear keeps the
+  file's own header line.)
+  `TSVZed`'s message for a refused rewrite lists `hardMapToFile()` among the rewrites
+  a program asks for.
+- **Known: a scrub of a `.tsv` whose rows were all deleted does not compact it** when
+  the scrub is given the file's header (`scrubTabularFile(path, header=H)`) or the file
+  has no header line: it leaves the file as it is, deletion lines included; reads are
+  right, as in 3.39 and 5.0. (`tsvz scrub` of a headed file reads the header line as a
+  row, so it does compact it.)
+  ([limitations](docs/limitations.md#a-scrub-of-a-tsv-whose-rows-were-all-deleted-does-not-compact-it)).
+- **Known: a scrub run with other defaults than the store's program can change how
+  early rows read.** In a `.tsv` with an empty `#_defaults_#` line and rows with empty
+  cells before any defaults line, the reset line the scrub writes before the rows keeps
+  those cells empty for a program that opens the store with its own defaults; run the
+  scrub with the defaults that program uses
+  ([limitations](docs/limitations.md#a-scrub-run-with-other-defaults-than-the-stores-program-can-change-how-early-rows-read)).
+- **Running programs keep 5.0** until they are restarted: long-running programs that
+  import TSVZ, and running `tsvz serve` handlers.
+
+## Upgrading to 5.0 from 4.9
+
+- **Writes refuse text they cannot encode.** A key, value, header, defaults line or
+  marker the file's encoding cannot hold (UTF-8 for `.tsvz`; `encoding=` for a
+  `.tsv`), such as the lone surrogate `os.fsdecode` gives for a non-UTF-8 file name,
+  raises `TSVZEncodeError` (a `ValueError`) and writes nothing; 4.9 stored `?` (its
+  `tsvz` stored U+FFFD for a non-UTF-8 argument, and its `.tsvz` file functions raised
+  `UnicodeEncodeError`).
+  `errors='replace'` brings `?` back. Under a non-UTF-8 `encoding=`, a key read as
+  U+FFFD cannot be deleted or updated. `tsvz` exits 1 with the message. See Fault
+  tolerance.
+- **A Ctrl-C no longer cuts a `.tsv` write.** Every write of a `.tsv`-family file
+  runs on the store's background thread; a call waits for it, a Ctrl-C raises in the
+  caller and the write still finishes, and at exit a write in progress is finished
+  (a second Ctrl-C abandons it). In 4.9 a Ctrl-C while `TSVZed()` opened a
+  compressed store (its rewrite on load) could leave the file with no readable row.
+  A closed `TSVZed` or `TSVZedLite` is freed; 4.9 kept every one until exit.
+- **Another process's change is taken in, not reloaded over memory.** An append to
+  a `.tsv`-family file is read from where the store left off, so an order set by
+  `move_to_end` and `#` keys survive it; any reload (both dialects) changes memory
+  in place, so another thread never finds a live key missing. After a rewrite of
+  the store's own fails part way, memory is kept and another process's appends are
+  merged into it (Fault tolerance).
+- **`.tsvz` stores** see another process's append made right after their load, a
+  commit or a clear (4.9 missed it until the next change), keep the rows set while
+  they reload, and raise a strict header mismatch at the first read.
+- **`.tsv` files:** an append to an existing empty file writes the header first, and
+  reading one gives an empty store; see L8 to L14 for the rest.
+- **`tsvz`** runs every operation through the engine `tsvz serve` uses, with the same
+  output. A server of a `.tsv` (or other tab- or NUL-delimited) file is now found by
+  `tsvz` commands, `TSVZClient` and `tsvz stop`; 4.9's clients rejected its pointer
+  file. The 3.39 warning that a file name does not end in `.tsv` is printed only
+  when a command creates such a file, and a write that cannot be made reports the
+  operating system's error.
+- **Running programs keep 4.9.** Long-running programs that import TSVZ, and running
+  `tsvz serve` handlers, keep 4.9 until they are restarted. 5.0's `tsvz stop` cannot
+  stop a 4.9 handler of a `.tsv` (or other tab- or NUL-delimited) file: 4.9 wrote the
+  delimiter into the pointer file's `x-delimiter` as `<sep>` (a NUL as `<NUL>`), which
+  5.0 decodes to the TAB or NUL itself and refuses as a control character. Send that
+  handler SIGTERM instead: its pid is on the `pid` line of its pointer file
+  (`STORE.serve`), and it finishes its writes and exits.
+- **macOS and other systems that lock with `lockf`:** as in 4.9, a read in another
+  thread of the process can still drop a write's lock
+  ([docs/limitations.md](docs/limitations.md)).
+
 ## Breaking changes from 3.39
 
-### `.tsv`, `.csv`, `.nsv`, `.psv`: bug fixes only
+### `.tsv`, `.csv`, `.nsv`, `.psv`: the format is unchanged
 
 | # | Change |
 |---|---|
@@ -232,6 +430,18 @@ delimiter or encoding argument that conflicts with the extension.
 | L5 | A damaged compressed file is read up to the damage instead of raising. |
 | L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
 | L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
+| L8 | Appending (`TSVZed`, `TSVZedLite`, `appendTabularFile`, `appendLinesTabularFile`) to an existing empty file (no byte, or a compressed file holding only empty streams) writes the store's header first, as creating the file does; 3.39 wrote the rows alone (a read then took the first one for the header), and `appendTabularFile` with `strict` refused the file. |
+| L9 | Reading such an empty file with a header (and `verifyHeader`) gives an empty store with that header: `readTabularFile`, `TSVZed` and `TSVZedLite` with `strict` no longer refuse it with `Header mismatch` (a `TSVZed` restarted after a rewrite that left the file empty failed in its constructor), and without `strict` no longer print a header mismatch. |
+| L10 | Writing text the file's encoding cannot hold (a lone surrogate from `os.fsdecode`, a character the `encoding=` lacks) raises `TSVZEncodeError`, a `ValueError`, and writes nothing; 3.39 and 4.9 wrote `?`, which `errors='replace'` brings back. |
+| L11 | `pop()` and `popitem()` of a `#` key, which lives in memory only, write no tombstone line to the file. |
+| L12 | A rewrite of a file whose header line has no newline writes the rows on lines of their own; 3.39 wrote the first one onto the header line. |
+| L13 | A compressed file is rewritten in members of at most 1 MiB, as appends already were: the bytes differ, the rows read are the same, and 3.39 reads them. |
+| L14 | `TSVZed` opening a compressed file in a codec this Python lacks (zstd before Python 3.14) raises `ImportError` instead of reading the compressed bytes as text. |
+| L15 | `scrubTabularFile`, `clearTabularFile`, `tsvz scrub`, `tsvz clear` and `TSVZedLite.clear_file()` write the new content over the file and then cut it, under the file's lock; 3.39 emptied the file first, with no lock, so a reader could meet it empty. |
+| L16 | A scrub writes only if the file is still what its read saw: it reads again up to three times, then writes nothing, with a warning (`tsvz scrub` exits 1, C6); 3.39 wrote over rows appended after its read. |
+| L17 | `scrubTabularFile(lastLineOnly=True)` returns the last row and writes nothing; 3.39 wrote that row back as deletion lines, and the store was gone. |
+| L18 | A scrub keeps the file's `#_defaults_#` lines (an empty one before the rows, the defaults in effect after them), so every row reads as it did; 3.39 dropped them, so rows appended later got no defaults. |
+| L19 | A scrub given no header, and a `TSVZed` given none, keep a file's header line first; 3.39 put an empty line in front of it, and a reader given the header then read the header as a row. |
 
 ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
 
@@ -281,7 +491,7 @@ Conformance testing).
 
 - §17.5: a numbered part opened directly is read alone, with a warning.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
-- §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's `lockf`; elsewhere (macOS and other systems, or a Linux file system that refuses those locks) it is `lockf`, which closing any descriptor of the file in the process drops: there a read in another thread of the same process, such as a handler's or a `TSVZed`'s, can drop a write's lock while it is held. A known limitation, to be fixed separately.
+- §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's `lockf`; elsewhere (macOS and other systems, or a Linux file system that refuses those locks) it is `lockf`, which closing any descriptor of the file in the process drops: there a read in another thread of the same process, such as a handler's or a `TSVZed`'s, can drop a write's lock while it is held. A known limitation, deferred: see [docs/limitations.md](docs/limitations.md).
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.
@@ -296,16 +506,12 @@ Conformance testing).
 
 ## Website
 
-`website/` holds the tsvz.org site. `website/tsvz_site.py` (standard library
-only, Python 3.8+) renders `website/index.md`, `website/spec-glance.md` and
-`tsvz-spec-v1.md`, and serves HTML to browsers and markdown to agents.
-
-```bash
-python3 website/tsvz_site.py serve -p 8765     # http://127.0.0.1:8765/
-python3 website/tsvz_site.py build /tmp/site   # static files for any host
-```
-
-`deploy/tsvz-site.service` runs it under systemd; restart it to publish an edit.
+The tsvz.org site lives in [tsvz-org/tsvz.org](https://github.com/tsvz-org/tsvz.org).
+It renders `tsvz-spec-v1.md` from a checkout of this repository next to its own,
+and its tests also check the spec: every § reference names a real section, and
+the spec uses only markdown the site renders. Run them after editing the spec
+(see Tests); CI runs them on every push here, and the site publishes `main`'s
+spec each morning.
 
 ## Conformance testing
 
@@ -365,14 +571,16 @@ runs, because a single run can land in a slower mode as a whole.
 
 ```bash
 python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
-python3 -m pytest website/tsvz_site_test.py -q            # the tsvz.org site
+python3 -m pytest ../tsvz.org/tsvz_site_test.py -q        # the spec as tsvz.org renders it (needs the tsvz.org checkout next to this one)
 python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
 python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
 python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)
 ```
 
 `TSVZ_old.py` is the frozen 3.39 module the differential tests compare against;
-it is not installed.
+it is not installed. CI (`.github/workflows/test.yml`) runs the three test files on
+Python 3.6, 3.9, 3.12 and 3.14. Accepted limitations are listed in
+[docs/limitations.md](docs/limitations.md).
 
 ## License