TSVZ 4.3__tar.gz → 4.5__tar.gz

{tsvz-4.3 → tsvz-4.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 4.3
+Version: 4.5
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/yufei-pan/TSVZ
 Author: Yufei Pan
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.5**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -51,7 +51,9 @@ pip install -e .          # editable, from this directory
 
 The delimiter follows the extension: tab, comma, NUL, pipe. A trailing
 compression suffix (`.gz`, `.bz2`, `.xz`, `.zst`) is allowed on both; `.zst`
-needs Python 3.14.
+needs Python 3.14. For important or sensitive data, keep the part you write to
+uncompressed and compress parts once they no longer change: a damaged
+compressed member can lose every record in it (spec §16.4.2).
 
 `TSVZed`, `TSVZedLite`, `readTabularFile`, `appendTabularFile`,
 `appendLinesTabularFile`, `clearTabularFile`, `scrubTabularFile` (and the
@@ -203,9 +205,12 @@ TSVZ warning: data.tsvz: invalid UTF-8 replaced with U+FFFD (3 occurrences, firs
 Examples: bytes after the last newline (ignored on read; truncated before the
 next append, and quoted), invalid UTF-8, a byte order mark, a newer
 `#_version_#`, a marker with a bad value, a checksum mismatch, a damaged
-compressed part (read up to the damage; repaired in place before the next
-append, with the original saved as `<part>.damaged-<time>`), an unreadable
-part, and a delimiter or encoding argument that conflicts with the extension.
+compressed member or a zero-filled line (a member cut short by a crash keeps
+the whole lines it decoded; a corrupt member or a zero-filled line is
+recovered, skipped or stops the read, as the store's `#_on_corruption_#` says,
+`recover` by default, spec §4.9, §4.10, §16.6; a scrub that rewrites such a
+part keeps the original as `<part>.damaged-<time>`), an unreadable part, and a
+delimiter or encoding argument that conflicts with the extension.
 
 ## Breaking changes from 3.39
 
@@ -235,6 +240,9 @@ part, and a delimiter or encoding argument that conflicts with the extension.
 | S9 | `clear` keeps the active markers; scrub keeps the header comment and drops other comments, custom markers and checksums. |
 | S10 | Files named `x.tsvz.<hex>` beside `x.tsvz` are read as parts of the store. |
 | S11 | `lastLineOnly` reads forward (slower on large files). |
+| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
+| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
+| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
 
 ### The `tsvz` command
 
@@ -253,6 +261,7 @@ and `-d=,`, except the ones C5 and C9 list.
 | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
 | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
 | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
+| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
 
 ## Spec deviations and interpretations
 
@@ -262,7 +271,7 @@ Conformance testing).
 
 - §17.5: a numbered part opened directly is read alone, with a warning.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
-- §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
+- §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's `lockf`; elsewhere it is `lockf`.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.

{tsvz-4.3 → tsvz-4.5}/README.md

@@ -7,7 +7,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.5**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -24,7 +24,9 @@ pip install -e .          # editable, from this directory
 
 The delimiter follows the extension: tab, comma, NUL, pipe. A trailing
 compression suffix (`.gz`, `.bz2`, `.xz`, `.zst`) is allowed on both; `.zst`
-needs Python 3.14.
+needs Python 3.14. For important or sensitive data, keep the part you write to
+uncompressed and compress parts once they no longer change: a damaged
+compressed member can lose every record in it (spec §16.4.2).
 
 `TSVZed`, `TSVZedLite`, `readTabularFile`, `appendTabularFile`,
 `appendLinesTabularFile`, `clearTabularFile`, `scrubTabularFile` (and the
@@ -176,9 +178,12 @@ TSVZ warning: data.tsvz: invalid UTF-8 replaced with U+FFFD (3 occurrences, firs
 Examples: bytes after the last newline (ignored on read; truncated before the
 next append, and quoted), invalid UTF-8, a byte order mark, a newer
 `#_version_#`, a marker with a bad value, a checksum mismatch, a damaged
-compressed part (read up to the damage; repaired in place before the next
-append, with the original saved as `<part>.damaged-<time>`), an unreadable
-part, and a delimiter or encoding argument that conflicts with the extension.
+compressed member or a zero-filled line (a member cut short by a crash keeps
+the whole lines it decoded; a corrupt member or a zero-filled line is
+recovered, skipped or stops the read, as the store's `#_on_corruption_#` says,
+`recover` by default, spec §4.9, §4.10, §16.6; a scrub that rewrites such a
+part keeps the original as `<part>.damaged-<time>`), an unreadable part, and a
+delimiter or encoding argument that conflicts with the extension.
 
 ## Breaking changes from 3.39
 
@@ -208,6 +213,9 @@ part, and a delimiter or encoding argument that conflicts with the extension.
 | S9 | `clear` keeps the active markers; scrub keeps the header comment and drops other comments, custom markers and checksums. |
 | S10 | Files named `x.tsvz.<hex>` beside `x.tsvz` are read as parts of the store. |
 | S11 | `lastLineOnly` reads forward (slower on large files). |
+| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
+| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
+| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
 
 ### The `tsvz` command
 
@@ -226,6 +234,7 @@ and `-d=,`, except the ones C5 and C9 list.
 | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
 | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
 | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
+| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
 
 ## Spec deviations and interpretations
 
@@ -235,7 +244,7 @@ Conformance testing).
 
 - §17.5: a numbered part opened directly is read alone, with a warning.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
-- §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
+- §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's `lockf`; elsewhere it is `lockf`.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.

{tsvz-4.3 → tsvz-4.5}/TSVZ.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: TSVZ
-Version: 4.3
+Version: 4.5
 Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
 Home-page: https://github.com/yufei-pan/TSVZ
 Author: Yufei Pan
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
 row is the key; the file behaves like an ordered dictionary that is persisted to
 disk as you change it.
 
-**Version 4.3**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
+**Version 4.5**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
 next to your code and `import TSVZ`, or install it.
 
 ```bash
@@ -51,7 +51,9 @@ pip install -e .          # editable, from this directory
 
 The delimiter follows the extension: tab, comma, NUL, pipe. A trailing
 compression suffix (`.gz`, `.bz2`, `.xz`, `.zst`) is allowed on both; `.zst`
-needs Python 3.14.
+needs Python 3.14. For important or sensitive data, keep the part you write to
+uncompressed and compress parts once they no longer change: a damaged
+compressed member can lose every record in it (spec §16.4.2).
 
 `TSVZed`, `TSVZedLite`, `readTabularFile`, `appendTabularFile`,
 `appendLinesTabularFile`, `clearTabularFile`, `scrubTabularFile` (and the
@@ -203,9 +205,12 @@ TSVZ warning: data.tsvz: invalid UTF-8 replaced with U+FFFD (3 occurrences, firs
 Examples: bytes after the last newline (ignored on read; truncated before the
 next append, and quoted), invalid UTF-8, a byte order mark, a newer
 `#_version_#`, a marker with a bad value, a checksum mismatch, a damaged
-compressed part (read up to the damage; repaired in place before the next
-append, with the original saved as `<part>.damaged-<time>`), an unreadable
-part, and a delimiter or encoding argument that conflicts with the extension.
+compressed member or a zero-filled line (a member cut short by a crash keeps
+the whole lines it decoded; a corrupt member or a zero-filled line is
+recovered, skipped or stops the read, as the store's `#_on_corruption_#` says,
+`recover` by default, spec §4.9, §4.10, §16.6; a scrub that rewrites such a
+part keeps the original as `<part>.damaged-<time>`), an unreadable part, and a
+delimiter or encoding argument that conflicts with the extension.
 
 ## Breaking changes from 3.39
 
@@ -235,6 +240,9 @@ part, and a delimiter or encoding argument that conflicts with the extension.
 | S9 | `clear` keeps the active markers; scrub keeps the header comment and drops other comments, custom markers and checksums. |
 | S10 | Files named `x.tsvz.<hex>` beside `x.tsvz` are read as parts of the store. |
 | S11 | `lastLineOnly` reads forward (slower on large files). |
+| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
+| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
+| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
 
 ### The `tsvz` command
 
@@ -253,6 +261,7 @@ and `-d=,`, except the ones C5 and C9 list.
 | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
 | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
 | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
+| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
 
 ## Spec deviations and interpretations
 
@@ -262,7 +271,7 @@ Conformance testing).
 
 - §17.5: a numbered part opened directly is read alone, with a warning.
 - §14.5: the defaults-on-missing read applies to `t[key]` only.
-- §18.6: locking is the 3.39 exclusive `lockf` per write plus an in-process lock; readers do not lock.
+- §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's `lockf`; elsewhere it is `lockf`.
 - §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
 - §20.7: `-q` also silences informational messages ("Created …"); only errors remain.
 - §20.2: `scrub` of a multi-part store, and `clear` of one part of one, are refused with exit 1; `read`, `get` and `parts` of a numbered part read that part alone, with a warning.