TSVZ 4.2__tar.gz → 4.3__tar.gz

{tsvz-4.2 → tsvz-4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 4.2
+Version: 4.3
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/yufei-pan/TSVZ
 Author: Yufei Pan
@@ -12,7 +12,6 @@ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (G
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.6
 Description-Content-Type: text/markdown
-Requires-Dist: argparse
 Provides-Extra: completion
 Requires-Dist: argcomplete; extra == "completion"
 Dynamic: author
@@ -23,7 +22,6 @@ Dynamic: description-content-type
 Dynamic: home-page
 Dynamic: license
 Dynamic: provides-extra
-Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
@@ -36,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.2**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -258,16 +256,14 @@ and `-d=,`, except the ones C5 and C9 list.
 
 ## Spec deviations and interpretations
 
-- §19.2.3c: scrub keeps the header comment (3.39 header verification needs it).
+None of these fails a conformance case: `conformance/reference-baseline.json`,
+which lists the cases this implementation does not pass, is empty (see
+Conformance testing).
+
 - §17.5: a numbered part opened directly is read alone, with a warning.
-- §4.1: a leading byte order mark is tolerated (stripped, with a warning).
-- A leading byte order mark is not part of any `#_checksum_*_#` segment.
-- §12.4.1: boolean marker values also accept `yes/no/on/off/1/0`; TSVZ writes `true`/`false`.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
 - §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
-- Invalid UTF-8 is replaced with U+FFFD (and reported).
-- v1 has no `<CR>` token, so a value whose last cell ends in `\r` loses it; values set through `TSVZed` are right-stripped, so this only affects rows passed raw to the append helpers.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.
 - §20.2.4: on `.tsv`-family files `get` prints nothing for a missing key, `verify` checks nothing, and `parts` lists the file.
@@ -276,7 +272,6 @@ and `-d=,`, except the ones C5 and C9 list.
 - §21: a served `.tsv` file with a header line shows that line as a row, as `tsvz read` does (`TSVZed` hides it when given the header).
 - §21.8: the server notices changes with `stat()`. It re-reads the part list while the store's directory changed less than 2 s ago, and reloads a part whose size stayed the same but whose inode or modification time changed. A rewrite in place that keeps a part's size and modification time is not seen until the part changes again.
 - §21.9: `tsvz` commands and `TSVZClient` ask the server to answer writes once they are written (`--x-written`), so a write the server cannot make fails as it would on the files. While writes fail, every answer carries a warning.
-- §17.7: the server appends to the store's current part; it does not start a new part when it starts.
 - §21.3: `tsvz serve` needs Unix-domain sockets. Where Python has none (Windows), it exits 1, and `tsvz` commands and `TSVZClient` work on the files.
 - §20.7: on `.tsv`-family files a `-d` longer than one character is used, as in 3.39, with a warning; a `-d` that does not decode is a usage error.
 
@@ -293,11 +288,67 @@ python3 website/tsvz_site.py build /tmp/site   # static files for any host
 
 `deploy/tsvz-site.service` runs it under systemd; restart it to publish an edit.
 
+## Conformance testing
+
+`tsvzconform.py` tests how closely an implementation of tsvz-spec-v1, in any
+language, follows the spec. Its cases are JSON files in `conformance/cases/`,
+each naming the § clauses it tests, and it drives an implementation only
+through the spec's command line (§20) and handler socket (§21).
+
+```bash
+python3 tsvzconform.py --impl ./tsvz --feature crc32,sha256,handler
+python3 tsvzconform.py --impl ./tsvz --class reader --only 13
+```
+
+Each failure is printed with its input, what was expected, what came back and
+a command line that reruns it alone; the summary gives a verdict for each
+conformance class (reader, writer, cli, handler). `conformance/README.md`
+explains the options and the features to claim, and how to load the cases into
+another language's tests.
+
+## Benchmarking implementations
+
+`tsvzbench.py` compares implementations of tsvz-spec-v1, in any language, side
+by side on one machine. It drives each one only through the spec's command
+line (§20) and, when it has one, its handler socket (§21), so any conformant
+`tsvz` tool can be measured as it is.
+
+```bash
+python3 tsvzbench.py --impl ref='python3 TSVZ.py' --impl new='../tsvz-rs/target/release/tsvz' \
+                     --suite quick --work-dir /mnt/fast-disk/tsvzbench
+```
+
+- `--impl NAME=COMMAND`, repeatable: the first is the baseline, and every other
+  implementation gets a `×` ratio against it (below 1 is faster). A relative
+  COMMAND runs from the directory you start tsvzbench in.
+- `--suite smoke|quick|standard|large`: `smoke` checks a setup in seconds,
+  `quick` takes about a minute, `standard` (the default, up to 1M rows) about
+  five, and `large` (up to 10M rows) twenty or more and ~5 GB of memory for the
+  harness itself.
+- `--only cold,bulk,perop,maint` picks families: one-shot replays, `set -` and
+  `delete -` batches, single operations (CLI and handler), and scrub/clear.
+- `--work-dir DIR` keeps the generated fixtures between runs. Put it on the disk
+  you mean to measure: not a tmpfs, and not a FUSE file system such as mergerfs
+  (the harness warns about those). The default is a `tsvzbench-*` directory here.
+- `--json PATH` also writes every raw sample.
+
+The harness writes its own input stores, so every implementation reads the same
+bytes, and checks every result against its own replay model: `✓` right, `✗`
+wrong output or files, `! N` an unexpected exit status, `⏱` timed out, `refused`
+an allowed refusal. It exits 1 when any cell failed. Each command runs under a
+small launcher that measures it with `wait4`; a peak RSS shown as `≤8.0 MiB` is
+the launcher's own size, so the true peak is no higher. Compare an
+implementation with itself to see the noise: a few percent per cell, but the
+socket rows, whose latencies are tens of microseconds, can move 10–20% between
+runs, because a single run can land in a slower mode as a whole.
+
 ## Tests
 
 ```bash
 python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
 python3 -m pytest website/tsvz_site_test.py -q            # the tsvz.org site
+python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
+python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
 python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)
 ```
 

{tsvz-4.2 → tsvz-4.3}/README.md

@@ -7,7 +7,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.2**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -229,16 +229,14 @@ and `-d=,`, except the ones C5 and C9 list.
 
 ## Spec deviations and interpretations
 
-- §19.2.3c: scrub keeps the header comment (3.39 header verification needs it).
+None of these fails a conformance case: `conformance/reference-baseline.json`,
+which lists the cases this implementation does not pass, is empty (see
+Conformance testing).
+
 - §17.5: a numbered part opened directly is read alone, with a warning.
-- §4.1: a leading byte order mark is tolerated (stripped, with a warning).
-- A leading byte order mark is not part of any `#_checksum_*_#` segment.
-- §12.4.1: boolean marker values also accept `yes/no/on/off/1/0`; TSVZ writes `true`/`false`.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
 - §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
-- Invalid UTF-8 is replaced with U+FFFD (and reported).
-- v1 has no `<CR>` token, so a value whose last cell ends in `\r` loses it; values set through `TSVZed` are right-stripped, so this only affects rows passed raw to the append helpers.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.
 - §20.2.4: on `.tsv`-family files `get` prints nothing for a missing key, `verify` checks nothing, and `parts` lists the file.
@@ -247,7 +245,6 @@ and `-d=,`, except the ones C5 and C9 list.
 - §21: a served `.tsv` file with a header line shows that line as a row, as `tsvz read` does (`TSVZed` hides it when given the header).
 - §21.8: the server notices changes with `stat()`. It re-reads the part list while the store's directory changed less than 2 s ago, and reloads a part whose size stayed the same but whose inode or modification time changed. A rewrite in place that keeps a part's size and modification time is not seen until the part changes again.
 - §21.9: `tsvz` commands and `TSVZClient` ask the server to answer writes once they are written (`--x-written`), so a write the server cannot make fails as it would on the files. While writes fail, every answer carries a warning.
-- §17.7: the server appends to the store's current part; it does not start a new part when it starts.
 - §21.3: `tsvz serve` needs Unix-domain sockets. Where Python has none (Windows), it exits 1, and `tsvz` commands and `TSVZClient` work on the files.
 - §20.7: on `.tsv`-family files a `-d` longer than one character is used, as in 3.39, with a warning; a `-d` that does not decode is a usage error.
 
@@ -264,11 +261,67 @@ python3 website/tsvz_site.py build /tmp/site   # static files for any host
 
 `deploy/tsvz-site.service` runs it under systemd; restart it to publish an edit.
 
+## Conformance testing
+
+`tsvzconform.py` tests how closely an implementation of tsvz-spec-v1, in any
+language, follows the spec. Its cases are JSON files in `conformance/cases/`,
+each naming the § clauses it tests, and it drives an implementation only
+through the spec's command line (§20) and handler socket (§21).
+
+```bash
+python3 tsvzconform.py --impl ./tsvz --feature crc32,sha256,handler
+python3 tsvzconform.py --impl ./tsvz --class reader --only 13
+```
+
+Each failure is printed with its input, what was expected, what came back and
+a command line that reruns it alone; the summary gives a verdict for each
+conformance class (reader, writer, cli, handler). `conformance/README.md`
+explains the options and the features to claim, and how to load the cases into
+another language's tests.
+
+## Benchmarking implementations
+
+`tsvzbench.py` compares implementations of tsvz-spec-v1, in any language, side
+by side on one machine. It drives each one only through the spec's command
+line (§20) and, when it has one, its handler socket (§21), so any conformant
+`tsvz` tool can be measured as it is.
+
+```bash
+python3 tsvzbench.py --impl ref='python3 TSVZ.py' --impl new='../tsvz-rs/target/release/tsvz' \
+                     --suite quick --work-dir /mnt/fast-disk/tsvzbench
+```
+
+- `--impl NAME=COMMAND`, repeatable: the first is the baseline, and every other
+  implementation gets a `×` ratio against it (below 1 is faster). A relative
+  COMMAND runs from the directory you start tsvzbench in.
+- `--suite smoke|quick|standard|large`: `smoke` checks a setup in seconds,
+  `quick` takes about a minute, `standard` (the default, up to 1M rows) about
+  five, and `large` (up to 10M rows) twenty or more and ~5 GB of memory for the
+  harness itself.
+- `--only cold,bulk,perop,maint` picks families: one-shot replays, `set -` and
+  `delete -` batches, single operations (CLI and handler), and scrub/clear.
+- `--work-dir DIR` keeps the generated fixtures between runs. Put it on the disk
+  you mean to measure: not a tmpfs, and not a FUSE file system such as mergerfs
+  (the harness warns about those). The default is a `tsvzbench-*` directory here.
+- `--json PATH` also writes every raw sample.
+
+The harness writes its own input stores, so every implementation reads the same
+bytes, and checks every result against its own replay model: `✓` right, `✗`
+wrong output or files, `! N` an unexpected exit status, `⏱` timed out, `refused`
+an allowed refusal. It exits 1 when any cell failed. Each command runs under a
+small launcher that measures it with `wait4`; a peak RSS shown as `≤8.0 MiB` is
+the launcher's own size, so the true peak is no higher. Compare an
+implementation with itself to see the noise: a few percent per cell, but the
+socket rows, whose latencies are tens of microseconds, can move 10–20% between
+runs, because a single run can land in a slower mode as a whole.
+
 ## Tests
 
 ```bash
 python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
 python3 -m pytest website/tsvz_site_test.py -q            # the tsvz.org site
+python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
+python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
 python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)
 ```
 

{tsvz-4.2 → tsvz-4.3}/TSVZ.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 4.2
+Version: 4.3
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/yufei-pan/TSVZ
 Author: Yufei Pan
@@ -12,7 +12,6 @@ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (G
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.6
 Description-Content-Type: text/markdown
-Requires-Dist: argparse
 Provides-Extra: completion
 Requires-Dist: argcomplete; extra == "completion"
 Dynamic: author
@@ -23,7 +22,6 @@ Dynamic: description-content-type
 Dynamic: home-page
 Dynamic: license
 Dynamic: provides-extra
-Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
@@ -36,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.2**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -258,16 +256,14 @@ and `-d=,`, except the ones C5 and C9 list.
 
 ## Spec deviations and interpretations
 
-- §19.2.3c: scrub keeps the header comment (3.39 header verification needs it).
+None of these fails a conformance case: `conformance/reference-baseline.json`,
+which lists the cases this implementation does not pass, is empty (see
+Conformance testing).
+
 - §17.5: a numbered part opened directly is read alone, with a warning.
-- §4.1: a leading byte order mark is tolerated (stripped, with a warning).
-- A leading byte order mark is not part of any `#_checksum_*_#` segment.
-- §12.4.1: boolean marker values also accept `yes/no/on/off/1/0`; TSVZ writes `true`/`false`.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
 - §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
-- Invalid UTF-8 is replaced with U+FFFD (and reported).
-- v1 has no `<CR>` token, so a value whose last cell ends in `\r` loses it; values set through `TSVZed` are right-stripped, so this only affects rows passed raw to the append helpers.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.
 - §20.2.4: on `.tsv`-family files `get` prints nothing for a missing key, `verify` checks nothing, and `parts` lists the file.
@@ -276,7 +272,6 @@ and `-d=,`, except the ones C5 and C9 list.
 - §21: a served `.tsv` file with a header line shows that line as a row, as `tsvz read` does (`TSVZed` hides it when given the header).
 - §21.8: the server notices changes with `stat()`. It re-reads the part list while the store's directory changed less than 2 s ago, and reloads a part whose size stayed the same but whose inode or modification time changed. A rewrite in place that keeps a part's size and modification time is not seen until the part changes again.
 - §21.9: `tsvz` commands and `TSVZClient` ask the server to answer writes once they are written (`--x-written`), so a write the server cannot make fails as it would on the files. While writes fail, every answer carries a warning.
-- §17.7: the server appends to the store's current part; it does not start a new part when it starts.
 - §21.3: `tsvz serve` needs Unix-domain sockets. Where Python has none (Windows), it exits 1, and `tsvz` commands and `TSVZClient` work on the files.
 - §20.7: on `.tsv`-family files a `-d` longer than one character is used, as in 3.39, with a warning; a `-d` that does not decode is a usage error.
 
@@ -293,11 +288,67 @@ python3 website/tsvz_site.py build /tmp/site   # static files for any host
 
 `deploy/tsvz-site.service` runs it under systemd; restart it to publish an edit.
 
+## Conformance testing
+
+`tsvzconform.py` tests how closely an implementation of tsvz-spec-v1, in any
+language, follows the spec. Its cases are JSON files in `conformance/cases/`,
+each naming the § clauses it tests, and it drives an implementation only
+through the spec's command line (§20) and handler socket (§21).
+
+```bash
+python3 tsvzconform.py --impl ./tsvz --feature crc32,sha256,handler
+python3 tsvzconform.py --impl ./tsvz --class reader --only 13
+```
+
+Each failure is printed with its input, what was expected, what came back and
+a command line that reruns it alone; the summary gives a verdict for each
+conformance class (reader, writer, cli, handler). `conformance/README.md`
+explains the options and the features to claim, and how to load the cases into
+another language's tests.
+
+## Benchmarking implementations
+
+`tsvzbench.py` compares implementations of tsvz-spec-v1, in any language, side
+by side on one machine. It drives each one only through the spec's command
+line (§20) and, when it has one, its handler socket (§21), so any conformant
+`tsvz` tool can be measured as it is.
+
+```bash
+python3 tsvzbench.py --impl ref='python3 TSVZ.py' --impl new='../tsvz-rs/target/release/tsvz' \
+                     --suite quick --work-dir /mnt/fast-disk/tsvzbench
+```
+
+- `--impl NAME=COMMAND`, repeatable: the first is the baseline, and every other
+  implementation gets a `×` ratio against it (below 1 is faster). A relative
+  COMMAND runs from the directory you start tsvzbench in.
+- `--suite smoke|quick|standard|large`: `smoke` checks a setup in seconds,
+  `quick` takes about a minute, `standard` (the default, up to 1M rows) about
+  five, and `large` (up to 10M rows) twenty or more and ~5 GB of memory for the
+  harness itself.
+- `--only cold,bulk,perop,maint` picks families: one-shot replays, `set -` and
+  `delete -` batches, single operations (CLI and handler), and scrub/clear.
+- `--work-dir DIR` keeps the generated fixtures between runs. Put it on the disk
+  you mean to measure: not a tmpfs, and not a FUSE file system such as mergerfs
+  (the harness warns about those). The default is a `tsvzbench-*` directory here.
+- `--json PATH` also writes every raw sample.
+
+The harness writes its own input stores, so every implementation reads the same
+bytes, and checks every result against its own replay model: `✓` right, `✗`
+wrong output or files, `! N` an unexpected exit status, `⏱` timed out, `refused`
+an allowed refusal. It exits 1 when any cell failed. Each command runs under a
+small launcher that measures it with `wait4`; a peak RSS shown as `≤8.0 MiB` is
+the launcher's own size, so the true peak is no higher. Compare an
+implementation with itself to see the noise: a few percent per cell, but the
+socket rows, whose latencies are tens of microseconds, can move 10–20% between
+runs, because a single run can land in a slower mode as a whole.
+
 ## Tests
 
 ```bash
 python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
 python3 -m pytest website/tsvz_site_test.py -q            # the tsvz.org site
+python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
+python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
 python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)
 ```
 

{tsvz-4.2 → tsvz-4.3}/TSVZ.egg-info/requires.txt

@@ -1,4 +1,3 @@
-argparse
 
 [completion]
 argcomplete

{tsvz-4.2 → tsvz-4.3}/TSVZ.py

@@ -40,10 +40,10 @@ if os.name == 'nt':
 elif os.name == 'posix':
 	import fcntl
 
-version = '4.2'
+version = '4.3'
 __version__ = version
 author = 'pan@zopyr.us'
-COMMIT_DATE = '2026-10-09'
+COMMIT_DATE = '2026-10-11'
 
 DEFAULT_DELIMITER = '\t'
 DEFAULTS_INDICATOR_KEY = '#_defaults_#'
@@ -121,6 +121,11 @@ def eprint(*args, **kwargs):
 
 def openFileAsCompressed(fileName,mode = 'rb',encoding = 'utf8',teeLogger = None,compressLevel = 1):
 	lowerFileName = fileName.lower()
+	if any(c in mode for c in 'wax+'):
+		# Writing plain bytes into a compressed file destroys it: refuse when this Python lacks the codec.
+		suffix = lowerFileName.rpartition('.')[2]
+		if suffix in COMPRESSED_FILE_EXTENSIONS:
+			_requireCodec(suffix)
 	if 'b' not in mode:
 		mode += 't'
 	kwargs = {}
@@ -683,6 +688,29 @@ def _lockedAppend(path, payload, tailPolicy, reporter, fsync=False):
 				_unlockFile(f)
 
 
+#: Compression suffix -> codec name (spec §16.2), as ``parts`` and error messages say it.
+_CODEC_NAMES = {'gz': 'gzip', 'gzip': 'gzip', 'bz2': 'bzip2', 'bzip2': 'bzip2', 'xz': 'xz', 'lzma': 'xz', 'zst': 'zstd', 'zstd': 'zstd'}
+_CODEC_SUPPORT = {}
+
+
+def _codecSupported(codec):
+	"""True when this Python has ``codec`` (zstd needs Python 3.14; a minimal build may lack lzma or bz2). Checked once per codec."""
+	name = _CODEC_NAMES.get(codec, codec)
+	if name not in _CODEC_SUPPORT:
+		try:
+			_newDecompressor(codec)
+			_CODEC_SUPPORT[name] = True
+		except ImportError:
+			_CODEC_SUPPORT[name] = False
+	return _CODEC_SUPPORT[name]
+
+
+def _requireCodec(codec):
+	"""Raise ImportError before any write to a part in a codec this Python lacks (spec §16.5): nothing is written."""
+	if codec and not _codecSupported(codec):
+		raise ImportError('no {} support in this Python'.format(_CODEC_NAMES.get(codec, codec)))
+
+
 def _compressBytes(codec, data, level=1):
 	"""Compress ``data`` as one complete member / stream for ``codec``."""
 	if codec in ('gz', 'gzip'):
@@ -694,6 +722,7 @@ def _compressBytes(codec, data, level=1):
 	if codec in ('xz', 'lzma'):
 		import lzma
 		return lzma.compress(data, preset=level)
+	_requireCodec(codec)
 	from compression import zstd  # Python 3.14+
 	return zstd.compress(data, level=level)
 
@@ -1488,8 +1517,8 @@ MAX_SPEC_VERSION = 1
 _MARKER_RE = re.compile(r'^#_[A-Za-z0-9_-]+_#$')
 _CHECKSUM_RE = re.compile(r'^#_checksum_([A-Za-z0-9_-]+)_#$', re.IGNORECASE)
 _ASCII_DIGITS_RE = re.compile(r'^[0-9]+$')
-_SPEC_DECODE_RE = re.compile(r'<(?:sep|LF|lt|#)>')
-_SPEC_TOKENS = {'<LF>': '\n', '<lt>': '<', '<#>': '#'}
+_SPEC_DECODE_RE = re.compile(r'<(?:sep|LF|CR|lt|#)>')
+_SPEC_TOKENS = {'<LF>': '\n', '<CR>': '\r', '<lt>': '<', '<#>': '#'}
 _RowState = namedtuple('_RowState', 'defaults strip fillEmpty')
 _BOOL_VALUES = {'true': True, 'yes': True, 'on': True, '1': True,
 				'false': False, 'no': False, 'off': False, '0': False}
@@ -1518,6 +1547,8 @@ def _specEncodeField(value, delimiter, isKey=False):
 		value = value.replace(delimiter, '<sep>')
 	if '\n' in value:
 		value = value.replace('\n', '<LF>')
+	if '\r' in value:
+		value = value.replace('\r', '<CR>')
 	if isKey and value.startswith('#'):
 		value = '<#>' + value[1:]
 	return value
@@ -1563,7 +1594,7 @@ class _SpecState(object):
 
 def _specApplyMarker(state, keyLower, values, reporter, where):
 	"""Apply an official marker (spec §12.4). Returns False when its value was rejected."""
-	value = values[0].strip() if values else ''
+	value = values[0] if values else ''  # marker lines are not stripped (spec §10.4)
 	if keyLower == DEFAULTS_INDICATOR_KEY:
 		state.defaults = [DEFAULTS_INDICATOR_KEY] + list(values)
 		state.sawDefaults = True
@@ -1741,11 +1772,7 @@ def _decompressChunks(f, info):
 	(``'corrupt (...)'``). ``info.goodRawEnd`` is the raw offset after the last
 	complete member.
 	"""
-	try:
-		decompressor = _newDecompressor(info.codec)
-	except ImportError as e:
-		info.damaged = 'cannot decompress ({})'.format(e)
-		return
+	decompressor = _newDecompressor(info.codec)
 	isGzip = info.codec in ('gz', 'gzip')
 	rawPos = 0
 	inMember = False
@@ -1792,6 +1819,11 @@ def _iterPartLines(info, reporter):
 	except OSError as e:
 		reporter.note('unreadable', None, 'skipped unreadable part {} ({})'.format(info.path, e))
 		return
+	if info.codec and not _codecSupported(info.codec):
+		# Spec §16.5: a codec this Python lacks makes the part unreadable; it is not damage to repair.
+		f.close()
+		reporter.note('unreadable', None, 'cannot read part {}: no {} support in this Python'.format(info.path, _CODEC_NAMES.get(info.codec, info.codec)))
+		return
 	rest = b''
 	with f:
 		chunks = _decompressChunks(f, info) if info.codec else iter(lambda: f.read(_CHUNK), b'')
@@ -1842,7 +1874,7 @@ def _specChecksum(state, algo, text, delimiter, reporter, where, position=None):
 	"""
 	if algo in state.digests:
 		fields = text.split(delimiter)
-		expected = fields[1].strip().lower() if len(fields) > 1 else ''
+		expected = fields[1].lower() if len(fields) > 1 else ''  # not stripped (spec §10.4)
 		if expected:
 			actual = state.digests[algo].hexdigest()
 			if actual != expected:
@@ -2109,6 +2141,13 @@ def _specEnsureStore(fileName, createIfNotExist, header, defaults, strict, teeLo
 		lines.append(_specFormatRecord(defaults, delimiter, marker=True))
 	content = ''.join(line + '\n' for line in lines).encode('utf-8')
 	codec = _parsePartName(fileName).codec
+	try:
+		_requireCodec(codec)  # not even an empty file in a codec this Python lacks
+	except ImportError as e:
+		__teePrintOrNot('Failed to create {}: {}'.format(fileName, e),'error',teeLogger=teeLogger)
+		if strict:
+			raise
+		return False
 	try:
 		with open(fileName, 'xb') as f:
 			f.write(_compressBytes(codec, content) if codec and content else content)
@@ -2163,6 +2202,7 @@ def _specRepairCompressed(path, reporter):
 	intact.
 	"""
 	info = _PartInfo(path)
+	_requireCodec(info.codec)  # a part this Python cannot decode is not damaged; it is left alone
 	with _pathLock(path):
 		with open(path, 'r+b') as f:
 			_lockFile(f)
@@ -2208,6 +2248,7 @@ def _specAppendPayload(path, payload, reporter, repair=False, fsync=False):
 	if not codec:
 		_lockedAppend(path, payload, 'truncate', reporter, fsync=fsync)
 		return
+	_requireCodec(codec)
 	if repair and os.path.exists(path):
 		_specRepairCompressed(path, reporter)
 	with _pathLock(path):
@@ -2252,6 +2293,7 @@ def _specRewriteInPlace(path, content, expect=None):
 	writer committed in between, so nothing is written and None is returned.
 	"""
 	codec = _parsePartName(path).codec
+	_requireCodec(codec)
 	raw = _compressBytes(codec, content) if codec and content else content
 	with _pathLock(path):
 		with open(path, 'r+b') as f:
@@ -2398,6 +2440,11 @@ def _specClearTabularFile(fileName, teeLogger=None, header='', verifyHeader=Fals
 			parts, stamp = _specSnapshot(fileName)
 			load = _specLoad(fileName, delimiter, header=header, verifyHeader=False, strict=False,
 							 reporter=attempt, teeLogger=teeLogger, verbose=verbose)
+			if attempt.has('unreadable'):
+				if _storeParts(fileName)[0] != parts:
+					continue  # a part went away during the read (a rotation): read again
+				attempt.note('refused', None, 'clear skipped: a part of the store could not be read; nothing was written')
+				return False
 			if len(load.parts) > 1:
 				tombstones = ''.join(_specFormatRecord([key], delimiter) + '\n' for key in load.data)
 				if tombstones:
@@ -2513,6 +2560,11 @@ def _specScrub(fileName, teeLogger=None, header='', createIfNotExist=False, veri
 			load = _specLoad(fileName, delimiter, header=header, verifyHeader=verifyHeader, strict=strict,
 							 defaults=initial, correctColumnNum=correctColumnNum, taskDic=taskDic,
 							 reporter=attempt, teeLogger=teeLogger, verbose=verbose)
+			if attempt.has('unreadable'):
+				if _storeParts(fileName)[0] != parts:
+					continue  # a part went away during the read (a rotation): read again
+				attempt.note('refused', None, 'scrub skipped: a part of the store could not be read; nothing was written')
+				return taskDic, False
 			if len(load.parts) != 1 or name.ordinal is not None or name.rotated:
 				attempt.note('multipart', None, 'scrub skipped: TSVZ does not compact multi-part stores or single parts of them; nothing was written')
 				return taskDic, False
@@ -2826,6 +2878,7 @@ class TSVZed(OrderedDict):
 				load = _specLoad(self._fileName, self.delimiter, header=self.header, verifyHeader=self.verifyHeader,
 								 strict=self.strict, defaults=self._constructorDefaults, taskDic=OrderedDict(),
 								 reporter=reporter, teeLogger=self.teeLogger, verbose=self.verbose)
+			unreadable = reporter.has('unreadable')
 		finally:
 			reporter.flush()
 		OrderedDict.clear(self)
@@ -2849,7 +2902,7 @@ class TSVZed(OrderedDict):
 		self.lastUpdateTime = self.externalFileUpdateTime
 		self._dirMtimeNs = _dirMtimeNs(self._fileName)
 		if (load is not None and not load.state.sawDefaults and any(self._constructorDefaults[1:])
-				and not self.memoryOnly):
+				and not self.memoryOnly and not unreadable):  # the store's own defaults may be in the unread part
 			# Design §5.3: persist constructor defaults by appending, as 3.39 did by rewriting.
 			self.appendQueue.append(_specFormatRecord(self._constructorDefaults, self.delimiter, marker=True))
 			self._specState.sawDefaults = True
@@ -3865,6 +3918,7 @@ class TSVZedLite(MutableMapping):
 							 strict=self.strict, defaults=self._constructorDefaults,
 							 correctColumnNum=self.correctColumnNum, storeOffset=True, taskDic=target,
 							 reporter=reporter, verbose=self.verbose)
+			unreadable = reporter.has('unreadable')
 		finally:
 			reporter.flush()
 		self._specState = load.state
@@ -3881,7 +3935,7 @@ class TSVZedLite(MutableMapping):
 		held = sum(1 for value in target.values() if isinstance(value, list))
 		if held:
 			_warnOnce(self, 'held', self._fileName, f'{held} keys from older or compressed parts are held in memory (offsets cannot address them)')
-		if not load.state.sawDefaults and any(self._constructorDefaults[1:]):
+		if not load.state.sawDefaults and any(self._constructorDefaults[1:]) and not unreadable:
 			self._pendingDefaults = True
 		return self
 
@@ -4474,7 +4528,7 @@ def _partRows(store, reporter):
 			if name.codec:
 				base = base[:-len(name.codec) - 1]
 			ordinal = base.rpartition('.')[2]
-		flags = (['active'] if path == active else []) + ([name.codec] if name.codec else [])
+		flags = (['active'] if path == active else []) + ([_CODEC_NAMES.get(name.codec, name.codec)] if name.codec else [])
 		rows.append([str(index), ordinal, path, ','.join(flags)])
 	return rows
 
@@ -4890,6 +4944,8 @@ class _ServeStore(object):
 		"""Atomically return the row of ``key`` and append its tombstone; None when it is missing."""
 		with self.lock:
 			self._fresh(log)
+			if self.unreadable:
+				return _SERVE_PARTIAL
 			key = self.resolve(key)
 			row = self.data.get(key)
 			if row is None:
@@ -4903,6 +4959,8 @@ class _ServeStore(object):
 		"""Atomically return the row of the last (or first) live key and delete it; None when empty."""
 		with self.lock:
 			self._fresh(log)
+			if self.unreadable:
+				return _SERVE_PARTIAL
 			if not self.data:
 				return None
 			key = next(reversed(self.data)) if last else next(iter(self.data))
@@ -4915,6 +4973,8 @@ class _ServeStore(object):
 		"""Atomically return the row of ``cells[0]``, appending ``cells`` first when the key is missing."""
 		with self.lock:
 			self._fresh(log)
+			if self.unreadable:
+				return _SERVE_PARTIAL
 			key = self.resolve(cells[0])
 			row = self.data.get(key)
 			if row is not None:
@@ -4999,6 +5059,8 @@ _SERVE_ARITY = {
 	'delete': (1, None), 'has': (1, 1), 'pop': (1, 1), 'popitem': (0, 1), 'setdefault': (2, None),
 }
 _SERVE_UNREADABLE = 'could not read every part of the store'
+#: What pop, popitem and setdefault return instead of acting on a store they could not read whole.
+_SERVE_PARTIAL = object()
 #: Operations that fail on a missing store (spec §20.2.2).
 _SERVE_NEED_STORE = ('read', 'get', 'has', 'len', 'keys', 'pop', 'popitem', 'scrub', 'verify', 'parts')
 #: A pointer file larger than this is not one (spec §21.2).
@@ -5122,11 +5184,15 @@ def _serveDispatch(store, line, bulk, log):
 			if args and args[0] not in ('first', 'last'):
 				raise _ServeUsage('popitem takes first or last')
 			row = store.popitem(not args or args[0] == 'last', sync, log)
+		if row is _SERVE_PARTIAL:
+			return 1, [], _SERVE_UNREADABLE
 		return (3, [], '') if row is None else (0, [_specFormatRecord(row, '\t')], '')
 	if operation == 'setdefault':
 		if _MARKER_RE.match(raw[0]):
 			raise _ServeUsage('setdefault takes a data KEY, not a marker')
 		row = store.setdefault(args, sync, log)
+		if row is _SERVE_PARTIAL:
+			return 1, [], _SERVE_UNREADABLE
 		return (3, [], '') if row is None else (0, [_specFormatRecord(row, '\t')], '')
 	if operation in ('clear', 'scrub'):
 		done = store.clear(log) if operation == 'clear' else store.scrub(log)

{tsvz-4.2 → tsvz-4.3}/setup.py

@@ -15,9 +15,6 @@ setup(
 			'tsvz = TSVZ:__main__',
         ],
     },
-    install_requires=[
-        'argparse',
-    ],
     extras_require={
         'completion': [
             'argcomplete',