TSVZ 4.6__tar.gz → 4.8__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: TSVZ
3
- Version: 4.6
3
+ Version: 4.8
4
4
  Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
5
5
  Home-page: https://github.com/yufei-pan/TSVZ
6
6
  Author: Yufei Pan
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
34
34
  row is the key; the file behaves like an ordered dictionary that is persisted to
35
35
  disk as you change it.
36
36
 
37
- **Version 4.6**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
37
+ **Version 4.8**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
38
38
  next to your code and `import TSVZ`, or install it.
39
39
 
40
40
  ```bash
@@ -224,6 +224,7 @@ delimiter or encoding argument that conflicts with the extension.
224
224
  | L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
225
225
  | L5 | A damaged compressed file is read up to the damage instead of raising. |
226
226
  | L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
227
+ | L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
227
228
 
228
229
  ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
229
230
 
@@ -243,6 +244,7 @@ delimiter or encoding argument that conflicts with the extension.
243
244
  | S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
244
245
  | S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
245
246
  | S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
247
+ | S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
246
248
 
247
249
  ### The `tsvz` command
248
250
 
@@ -256,12 +258,13 @@ and `-d=,`, except the ones C5 and C9 list.
256
258
  | C3 | `read` and `scrub` of a missing store exit 1 with a message (3.39: exit 0, and `read` printed "File not found" on stdout). |
257
259
  | C4 | `delete` deletes every KEY it is given (3.39: only the first). |
258
260
  | C5 | `append` or `delete` without a KEY, or with an empty KEY, is a usage error with exit 2 (3.39 wrote an empty line). |
259
- | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing) exits 1. |
261
+ | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing, a part that declares a spec version newer than 1, per spec §19.4 and §19.8) exits 1. |
260
262
  | C7 | `-c/--header`, `--defaults`, `-s/--strict` and `-f/--force` are undocumented aliases of the `--x-` options. `--x-header` and `--x-defaults` keep non-ASCII text. |
261
263
  | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
262
264
  | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
263
265
  | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
264
266
  | C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
267
+ | C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
265
268
 
266
269
  ## Spec deviations and interpretations
267
270
 
@@ -7,7 +7,7 @@ key-value store in a delimiter-separated text file**. The first column of each
7
7
  row is the key; the file behaves like an ordered dictionary that is persisted to
8
8
  disk as you change it.
9
9
 
10
- **Version 4.6**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
10
+ **Version 4.8**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
11
11
  next to your code and `import TSVZ`, or install it.
12
12
 
13
13
  ```bash
@@ -197,6 +197,7 @@ delimiter or encoding argument that conflicts with the extension.
197
197
  | L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
198
198
  | L5 | A damaged compressed file is read up to the damage instead of raising. |
199
199
  | L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
200
+ | L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
200
201
 
201
202
  ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
202
203
 
@@ -216,6 +217,7 @@ delimiter or encoding argument that conflicts with the extension.
216
217
  | S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
217
218
  | S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
218
219
  | S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
220
+ | S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
219
221
 
220
222
  ### The `tsvz` command
221
223
 
@@ -229,12 +231,13 @@ and `-d=,`, except the ones C5 and C9 list.
229
231
  | C3 | `read` and `scrub` of a missing store exit 1 with a message (3.39: exit 0, and `read` printed "File not found" on stdout). |
230
232
  | C4 | `delete` deletes every KEY it is given (3.39: only the first). |
231
233
  | C5 | `append` or `delete` without a KEY, or with an empty KEY, is a usage error with exit 2 (3.39 wrote an empty line). |
232
- | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing) exits 1. |
234
+ | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing, a part that declares a spec version newer than 1, per spec §19.4 and §19.8) exits 1. |
233
235
  | C7 | `-c/--header`, `--defaults`, `-s/--strict` and `-f/--force` are undocumented aliases of the `--x-` options. `--x-header` and `--x-defaults` keep non-ASCII text. |
234
236
  | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
235
237
  | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
236
238
  | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
237
239
  | C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
240
+ | C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
238
241
 
239
242
  ## Spec deviations and interpretations
240
243
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: TSVZ
3
- Version: 4.6
3
+ Version: 4.8
4
4
  Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
5
5
  Home-page: https://github.com/yufei-pan/TSVZ
6
6
  Author: Yufei Pan
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
34
34
  row is the key; the file behaves like an ordered dictionary that is persisted to
35
35
  disk as you change it.
36
36
 
37
- **Version 4.6**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
37
+ **Version 4.8**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
38
38
  next to your code and `import TSVZ`, or install it.
39
39
 
40
40
  ```bash
@@ -224,6 +224,7 @@ delimiter or encoding argument that conflicts with the extension.
224
224
  | L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
225
225
  | L5 | A damaged compressed file is read up to the damage instead of raising. |
226
226
  | L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
227
+ | L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
227
228
 
228
229
  ### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
229
230
 
@@ -243,6 +244,7 @@ delimiter or encoding argument that conflicts with the extension.
243
244
  | S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
244
245
  | S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
245
246
  | S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
247
+ | S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
246
248
 
247
249
  ### The `tsvz` command
248
250
 
@@ -256,12 +258,13 @@ and `-d=,`, except the ones C5 and C9 list.
256
258
  | C3 | `read` and `scrub` of a missing store exit 1 with a message (3.39: exit 0, and `read` printed "File not found" on stdout). |
257
259
  | C4 | `delete` deletes every KEY it is given (3.39: only the first). |
258
260
  | C5 | `append` or `delete` without a KEY, or with an empty KEY, is a usage error with exit 2 (3.39 wrote an empty line). |
259
- | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing) exits 1. |
261
+ | C6 | A `clear` or `scrub` that writes nothing (one part of a multi-part store, a store that kept changing, a part that declares a spec version newer than 1, per spec §19.4 and §19.8) exits 1. |
260
262
  | C7 | `-c/--header`, `--defaults`, `-s/--strict` and `-f/--force` are undocumented aliases of the `--x-` options. `--x-header` and `--x-defaults` keep non-ASCII text. |
261
263
  | C8 | Usage errors print tsvz's own message, exit 2; the `-h` text is new. |
262
264
  | C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
263
265
  | C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
264
266
  | C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
267
+ | C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
265
268
 
266
269
  ## Spec deviations and interpretations
267
270
 
@@ -40,10 +40,10 @@ if os.name == 'nt':
40
40
  elif os.name == 'posix':
41
41
  import fcntl
42
42
 
43
- version = '4.6'
43
+ version = '4.8'
44
44
  __version__ = version
45
45
  author = 'pan@zopyr.us'
46
- COMMIT_DATE = '2026-10-12'
46
+ COMMIT_DATE = '2026-10-14'
47
47
 
48
48
  DEFAULT_DELIMITER = '\t'
49
49
  DEFAULTS_INDICATOR_KEY = '#_defaults_#'
@@ -1605,8 +1605,10 @@ def scrubTabularFile(fileName,teeLogger = None,header = '',createIfNotExist = Fa
1605
1605
  # ===========================================================================
1606
1606
  MAX_SPEC_VERSION = 1
1607
1607
  _MARKER_RE = re.compile(r'^#_[A-Za-z0-9_-]+_#$')
1608
+ #: What a lookup gets for a key that is not live (spec §20.2.7: then the stripped key is tried).
1609
+ _NO_ROW = object()
1608
1610
  _CHECKSUM_RE = re.compile(r'^#_checksum_([A-Za-z0-9_-]+)_#$', re.IGNORECASE)
1609
- _ASCII_DIGITS_RE = re.compile(r'^[0-9]+$')
1611
+ _ASCII_DIGITS_RE = re.compile(r'^[0-9]+\Z') # \Z: '$' would also match before a final newline
1610
1612
  _SPEC_DECODE_RE = re.compile(r'<(?:sep|LF|CR|NUL|lt|#)>')
1611
1613
  _SPEC_TOKENS = {'<LF>': '\n', '<CR>': '\r', '<NUL>': '\0', '<lt>': '<', '<#>': '#'}
1612
1614
  _RowState = namedtuple('_RowState', 'defaults strip fillEmpty')
@@ -1669,8 +1671,7 @@ def _specTombstones(keys, delimiter, strip):
1669
1671
  (spec §9.6).
1670
1672
  """
1671
1673
  rows = [([key], False) for key in keys]
1672
- if strip and any(key.endswith((' ', '\t')) and _specEncodeField(key, delimiter, isKey=True).endswith((' ', '\t'))
1673
- for key in keys):
1674
+ if strip and any(_specStrippable(key, delimiter) for key in keys):
1674
1675
  rows.insert(0, (['#_strip_trailing_whites_#', 'false'], True))
1675
1676
  rows.append((['#_strip_trailing_whites_#', 'true'], True))
1676
1677
  return rows
@@ -1687,11 +1688,61 @@ def _specTombstoneLines(key, delimiter, strip):
1687
1688
  for cells, marker in _specTombstones([key], delimiter, strip))
1688
1689
 
1689
1690
 
1691
+ def _specStrippable(key, delimiter):
1692
+ """True when a reader that strips (spec §7.7) would change ``key``: its encoded form ends in a space or tab."""
1693
+ return key.endswith((' ', '\t')) and _specEncodeField(key, delimiter, isKey=True).endswith((' ', '\t'))
1694
+
1695
+
1696
+ def _specNameKey(key, delimiter, strip, live):
1697
+ """The key that a requested ``key`` names (spec §20.2.7) in a store whose stripping is ``strip``.
1698
+
1699
+ While stripping is on, a key a reader would strip names the live key equal
1700
+ to it (``live`` holds the live keys), else the key a reader resolves it to.
1701
+ """
1702
+ if not strip or key in live or not _specStrippable(key, delimiter):
1703
+ return key
1704
+ return _specDecodeField(_specEncodeField(key, delimiter, isKey=True).rstrip(' \t'), delimiter)
1705
+
1706
+
1707
+ def _specNamesKeys(rows, delimiter):
1708
+ """True when naming the keys of ``delete`` rows (``[(cells, marker), ...]``) needs the store (spec §20.2.7)."""
1709
+ return any(not marker and _specStrippable(cells[0], delimiter) for cells, marker in rows)
1710
+
1711
+
1712
+ def _specDeleteRows(rows, delimiter, strip, live):
1713
+ """The batch ``delete`` appends for ``rows`` (``[(cells, marker), ...]``, spec §20.2, §20.3).
1714
+
1715
+ Every key names a key of the store as it was before the batch, whose
1716
+ stripping was ``strip`` and whose live keys are ``live`` (§20.2.7). The
1717
+ tombstone of a key a reader would strip is written with stripping off
1718
+ (§9.6). A reset of ``#_strip_trailing_whites_#`` in the batch turns
1719
+ stripping back on (§10.3, §12.1).
1720
+ """
1721
+ batch = []
1722
+ named = strip
1723
+ guarded = False
1724
+ for cells, marker in rows:
1725
+ if marker:
1726
+ if cells[0].lower() == '#_strip_trailing_whites_#':
1727
+ strip, guarded = True, False
1728
+ batch.append((cells, marker))
1729
+ continue
1730
+ key = _specNameKey(cells[0], delimiter, named, live)
1731
+ if strip and not guarded and _specStrippable(key, delimiter):
1732
+ batch.append((['#_strip_trailing_whites_#', 'false'], True))
1733
+ guarded = True
1734
+ batch.append(([key], False))
1735
+ if guarded:
1736
+ batch.append((['#_strip_trailing_whites_#', 'true'], True))
1737
+ return batch
1738
+
1739
+
1690
1740
  class _SpecState(object):
1691
1741
  """Reader state of spec §12, carried across records and parts."""
1692
1742
 
1693
1743
  def __init__(self, defaults=None):
1694
1744
  self.version = 1
1745
+ self.newestVersion = 1 # the highest version a #_version_# line declared (spec §19.4, §19.8)
1695
1746
  self.defaults = list(defaults) if defaults and len(defaults) > 1 else [DEFAULTS_INDICATOR_KEY]
1696
1747
  self.strip = True
1697
1748
  self.fillEmpty = False
@@ -1736,16 +1787,17 @@ def _specApplyMarker(state, keyLower, values, reporter, where):
1736
1787
  else:
1737
1788
  number = 0
1738
1789
  if _ASCII_DIGITS_RE.match(value):
1739
- try:
1740
- number = int(value)
1741
- except ValueError: # more digits than int() converts (Python 3.11+ limit)
1742
- pass
1790
+ digits = value.lstrip('0') or '0'
1791
+ # int() refuses very long values (Python 3.11+); any such value is above every implemented version.
1792
+ number = int(digits) if len(digits) <= 64 else MAX_SPEC_VERSION + 1
1743
1793
  if number < 1:
1744
1794
  reporter.note('bad-marker', where, 'ignored #_version_# with invalid value {!r}'.format(value))
1745
1795
  return False
1746
1796
  if number > MAX_SPEC_VERSION:
1747
- reporter.note('newer-version', where, 'declares spec version {}; read as version {}'.format(value, MAX_SPEC_VERSION))
1797
+ shown = value if len(value) <= 20 else '{}… ({} digits)'.format(value[:20], len(value))
1798
+ reporter.note('newer-version', where, 'declares spec version {}; read as version {}'.format(shown, MAX_SPEC_VERSION))
1748
1799
  state.version = number
1800
+ state.newestVersion = max(state.newestVersion, number)
1749
1801
  elif keyLower in _BOOL_MARKERS:
1750
1802
  attr, default = _BOOL_MARKERS[keyLower]
1751
1803
  if not value:
@@ -2990,7 +3042,8 @@ def _specClearTabularFile(fileName, teeLogger=None, header='', verifyHeader=Fals
2990
3042
  A single file is read without a lock and rewritten in place only if no
2991
3043
  other writer changed it since the read (else it is read again, up to
2992
3044
  ``_SPEC_REWRITE_ATTEMPTS`` times). A path naming one part of a multi-part
2993
- store is refused.
3045
+ store is refused, and so is a single file declaring a spec version newer
3046
+ than ``MAX_SPEC_VERSION``.
2994
3047
  """
2995
3048
  reporter = _Reporter(fileName, teeLogger)
2996
3049
  attempt = None
@@ -3024,6 +3077,10 @@ def _specClearTabularFile(fileName, teeLogger=None, header='', verifyHeader=Fals
3024
3077
  break
3025
3078
  if stamp is None or load.parts != parts:
3026
3079
  continue # the set of parts changed under the read
3080
+ if load.state.newestVersion > MAX_SPEC_VERSION:
3081
+ attempt.note('refused', None, 'clear skipped: the part declares a spec version newer than {} (#_version_#), whose '
3082
+ 'markers this TSVZ cannot keep (spec §19.8); nothing was written'.format(MAX_SPEC_VERSION))
3083
+ return False
3027
3084
  if any(header) and verifyHeader and load.headerLine is not None and not headerChecked:
3028
3085
  headerChecked = True
3029
3086
  if not _lineContainHeader(header, load.headerLine[1:], verbose=verbose, teeLogger=teeLogger, strict=strict, delimiter=delimiter):
@@ -3106,8 +3163,8 @@ def _specScrub(fileName, teeLogger=None, header='', createIfNotExist=False, veri
3106
3163
  writer changed it since the read (else it is read again, up to
3107
3164
  ``_SPEC_REWRITE_ATTEMPTS`` times), so a record committed meanwhile is
3108
3165
  never compacted away. ``done`` is False when nothing was written because
3109
- the store is missing, has several parts (or the path names one part), or
3110
- kept changing. It is None when the read stopped at damaged data (spec
3166
+ the store is missing, has several parts (or the path names one part),
3167
+ declares a spec version newer than ``MAX_SPEC_VERSION``, or kept changing. It is None when the read stopped at damaged data (spec
3111
3168
  §4.10): the part was rewritten from what was read, the original kept.
3112
3169
  """
3113
3170
  if taskDic is None:
@@ -3143,6 +3200,11 @@ def _specScrub(fileName, teeLogger=None, header='', createIfNotExist=False, veri
3143
3200
  return taskDic, False
3144
3201
  if stamp is None or load.parts != parts:
3145
3202
  continue # the set of parts changed under the read
3203
+ if load.state.newestVersion > MAX_SPEC_VERSION:
3204
+ attempt.note('refused', None, 'scrub skipped: the store declares a spec version newer than {0} (#_version_#), which '
3205
+ 'this TSVZ reads as version {0} only (spec §6.3); a rewrite would make that reading permanent; '
3206
+ 'nothing was written'.format(MAX_SPEC_VERSION))
3207
+ return taskDic, False
3146
3208
  info = load.infos[0]
3147
3209
  if (info.lost or info.damaged) and backedUp != stamp:
3148
3210
  # Damaged data is not in what is written, or only unverified (spec §4.10, §16.6): keep the original
@@ -3628,14 +3690,37 @@ class TSVZed(OrderedDict):
3628
3690
  # else:
3629
3691
  # self.appendEvent.set()
3630
3692
 
3693
+ def _nameKey(self, key):
3694
+ """The key a requested ``key`` names: the live key equal to it, else ``key`` right-stripped (spec §20.2.7).
3695
+
3696
+ The fallback strips every trailing whitespace whatever the stripping
3697
+ marker says, as ``__setitem__`` strips every key it writes.
3698
+ """
3699
+ key = str(key)
3700
+ return key if OrderedDict.__contains__(self, key) else key.rstrip()
3701
+
3631
3702
  def __getitem__(self, key):
3632
- return super().__getitem__(str(key).rstrip())
3633
-
3703
+ key = str(key)
3704
+ row = dict.get(self, key, _NO_ROW) # one probe for the usual hit
3705
+ return row if row is not _NO_ROW else OrderedDict.__getitem__(self, key.rstrip())
3706
+
3707
+ def __contains__(self, key):
3708
+ # OrderedDict.setdefault asks this before it reads or writes, so it names keys as t[key] does.
3709
+ key = str(key)
3710
+ return dict.__contains__(self, key) or dict.__contains__(self, key.rstrip())
3711
+
3712
+ def get(self, key, default=None):
3713
+ key = str(key)
3714
+ row = dict.get(self, key, _NO_ROW)
3715
+ if row is _NO_ROW:
3716
+ row = dict.get(self, key.rstrip(), _NO_ROW)
3717
+ return default if row is _NO_ROW else row
3718
+
3634
3719
 
3635
3720
  def __delitem__(self,key):
3636
3721
  if self._spec:
3637
3722
  return self._specDelItem(key)
3638
- key = str(key).rstrip()
3723
+ key = self._nameKey(key)
3639
3724
  if key == DEFAULTS_INDICATOR_KEY:
3640
3725
  self.defaults = [DEFAULTS_INDICATOR_KEY]
3641
3726
  if self.verbose:
@@ -3739,7 +3824,7 @@ class TSVZed(OrderedDict):
3739
3824
  self.lastUpdateTime = get_time_ns()
3740
3825
 
3741
3826
  def _specDelItem(self, key):
3742
- key = str(key).rstrip()
3827
+ key = self._nameKey(key)
3743
3828
  if _MARKER_RE.match(key):
3744
3829
  # A lone marker key resets that marker (spec §12.1).
3745
3830
  self._specSetMarker(key, [key])
@@ -3914,12 +3999,14 @@ memoryOnly:{self.memoryOnly}
3914
3999
  is raised.
3915
4000
 
3916
4001
  '''
3917
- key = str(key).rstrip()
3918
- if key not in self:
4002
+ key = self._nameKey(key)
4003
+ if not OrderedDict.__contains__(self, key):
3919
4004
  if default is self.__marker:
3920
4005
  raise KeyError(key)
3921
4006
  return default
3922
- value = super().pop(key)
4007
+ # Not OrderedDict.pop: before Python 3.11 it goes through del self[key], which writes a tombstone too.
4008
+ value = OrderedDict.__getitem__(self, key)
4009
+ OrderedDict.__delitem__(self, key)
3923
4010
  if not self.memoryOnly:
3924
4011
  self.__appendEmptyLine(key)
3925
4012
  self.lastUpdateTime = get_time_ns()
@@ -3929,7 +4016,7 @@ memoryOnly:{self.memoryOnly}
3929
4016
  '''Move an existing element to the end (or beginning if last is false).
3930
4017
  Raise KeyError if the element does not exist.
3931
4018
  '''
3932
- key = str(key).rstrip()
4019
+ key = self._nameKey(key)
3933
4020
  super().move_to_end(key, last)
3934
4021
  if self._spec:
3935
4022
  _warnOnce(self, 'move_to_end', self._fileName,
@@ -4452,9 +4539,18 @@ class TSVZedLite(MutableMapping):
4452
4539
  """'tsvz' for tsvz-spec-v1 files (.tsvz/.csvz/.nsvz/.psvz), 'tsv' for 3.39-format files."""
4453
4540
  return 'tsvz' if self._spec else 'tsv'
4454
4541
 
4542
+ def _nameKey(self, key):
4543
+ """The key a requested ``key`` names: the live key equal to it, else ``key`` right-stripped (spec §20.2.7).
4544
+
4545
+ The fallback strips every trailing whitespace whatever the stripping
4546
+ marker says, as ``__setitem__`` strips every key it writes.
4547
+ """
4548
+ key = str(key)
4549
+ return key if key in self.indexes else key.rstrip()
4550
+
4455
4551
  def __contains__(self, key):
4456
4552
  if self._spec:
4457
- return str(key).rstrip() in self.indexes
4553
+ return self._nameKey(key) in self.indexes
4458
4554
  return MutableMapping.__contains__(self, key)
4459
4555
 
4460
4556
  def get(self, key, default=None):
@@ -4832,7 +4928,7 @@ class TSVZedLite(MutableMapping):
4832
4928
 
4833
4929
  # Implement basic __getitem__, __setitem__, __delitem__, __iter__, and __len__. needed for MutableMapping
4834
4930
  def __getitem__(self,key):
4835
- key = str(key).rstrip()
4931
+ key = self._nameKey(key)
4836
4932
  if key not in self.indexes:
4837
4933
  if key == DEFAULTS_INDICATOR_KEY:
4838
4934
  return self.defaults
@@ -4911,7 +5007,7 @@ class TSVZedLite(MutableMapping):
4911
5007
  self.indexes[key] = self.__writeValues(value)
4912
5008
 
4913
5009
  def __delitem__(self,key):
4914
- key = str(key).rstrip()
5010
+ key = self._nameKey(key)
4915
5011
  if self._spec:
4916
5012
  if _MARKER_RE.match(key):
4917
5013
  # A lone marker key resets that marker (spec §12.1).
@@ -4969,7 +5065,7 @@ class TSVZedLite(MutableMapping):
4969
5065
 
4970
5066
  __marker = object()
4971
5067
  def pop(self, key, default=__marker, return_pos = False):
4972
- key = str(key).rstrip()
5068
+ key = self._nameKey(key)
4973
5069
  try:
4974
5070
  pos = self.indexes.pop(key)
4975
5071
  except KeyError:
@@ -5430,9 +5526,9 @@ class _ServeStore(object):
5430
5526
  return not self.parts
5431
5527
 
5432
5528
  def resolve(self, key):
5433
- """A requested key as a reader resolves it (spec §7.7; 3.39 rstrip for loose files)."""
5529
+ """The key a requested key names (spec §20.2.7; 3.39 rstrip for loose files)."""
5434
5530
  if self.spec:
5435
- return key.rstrip(' \t') if self.state.strip else key
5531
+ return _specNameKey(key, self.delimiter, self.state.strip, self.data)
5436
5532
  return key.rstrip()
5437
5533
 
5438
5534
  def read(self, log):
@@ -5528,11 +5624,28 @@ class _ServeStore(object):
5528
5624
  self.writer.wait(queued[0])
5529
5625
 
5530
5626
  def write(self, rows, sync, log, written=False):
5531
- """Append ``rows`` (``[(cells, marker), ...]``) as one batch: ``set``, ``delete`` and bulk input."""
5627
+ """Append ``rows`` (``[(cells, marker), ...]``) as one batch: ``set`` and its bulk input."""
5532
5628
  with self.lock:
5533
5629
  queued = self._submit(rows, sync, log)
5534
5630
  self._ack(queued, written)
5535
5631
 
5632
+ def delete(self, rows, sync, log, written=False):
5633
+ """Append the tombstones of ``delete`` (``[(cells, marker), ...]``) as one batch (spec §20.2.7).
5634
+
5635
+ Naming a key a reader would strip needs the store as it is now, read
5636
+ under the lock, so that delete is atomic (spec §21.8.2). Returns False,
5637
+ having written nothing, when that read could not be completed.
5638
+ """
5639
+ with self.lock:
5640
+ if self.spec and _specNamesKeys(rows, self.delimiter):
5641
+ self._fresh(log)
5642
+ if self.unreadable:
5643
+ return False
5644
+ rows = _specDeleteRows(rows, self.delimiter, self.state.strip, self.data)
5645
+ queued = self._submit(rows, sync, log)
5646
+ self._ack(queued, written)
5647
+ return True
5648
+
5536
5649
  def pop(self, key, sync, log):
5537
5650
  """Atomically return the row of ``key`` and append its tombstone; None when it is missing."""
5538
5651
  with self.lock:
@@ -5781,7 +5894,10 @@ def _serveDispatch(store, line, bulk, log):
5781
5894
  rows = [([key], bool(_MARKER_RE.match(field))) for key, field in zip(args, raw)]
5782
5895
  else:
5783
5896
  rows = [(args, bool(_MARKER_RE.match(raw[0])))]
5784
- store.write(rows, sync, log, written)
5897
+ if not keysOnly:
5898
+ store.write(rows, sync, log, written)
5899
+ elif not store.delete(rows, sync, log, written):
5900
+ return 1, [], _SERVE_UNREADABLE + '; nothing was written'
5785
5901
  return 0, [], ''
5786
5902
  if operation in ('pop', 'popitem'):
5787
5903
  if operation == 'pop':
@@ -6482,7 +6598,10 @@ class TSVZClient(MutableMapping):
6482
6598
 
6483
6599
  def __delitem__(self, key):
6484
6600
  """Delete ``key``; a missing key is not an error (as in ``TSVZed``)."""
6485
- self._request(['delete', _serveEncode(str(key))], write=True)
6601
+ key = str(key)
6602
+ # Resending the delete of a key ending in whitespace could name, and delete, the stripped key (spec §20.2.7).
6603
+ retry = not (self._spec and _specStrippable(key, self.delimiter))
6604
+ self._request(['delete', _serveEncode(key)], write=True, retry=retry)
6486
6605
 
6487
6606
  _marker = object()
6488
6607
 
@@ -6508,6 +6627,11 @@ class TSVZClient(MutableMapping):
6508
6627
  cells = self._cells(key, default)
6509
6628
  if len(cells) < 2:
6510
6629
  return self.get(key)
6630
+ if str(key) != cells[0]:
6631
+ # cells[0] is stripped as set strips it; a live key equal to ``key`` wins first (spec §20.2.7).
6632
+ row = self.get(key)
6633
+ if row is not None:
6634
+ return row
6511
6635
  lines = self._request(['setdefault'] + self._fields(cells), write=True, retry=False)[0]
6512
6636
  return _serveDecodeLine(lines[0]) if lines else None
6513
6637
 
@@ -6953,7 +7077,7 @@ def _cliRead(args, delimiter, logger, stdin, stdout):
6953
7077
  def _cliGet(args, delimiter, logger, stdin, stdout):
6954
7078
  """``get STORE KEY ...`` (spec §20.2): each KEY's row in argument order; exit 3 when one is missing.
6955
7079
 
6956
- Keys resolve as a reader resolves them (§7.7). For a missing key of a
7080
+ Each KEY names a key as §20.2.7 says. For a missing key of a
6957
7081
  spec store this prints what ``TSVZed`` returns for it (§14.5): the key and
6958
7082
  the active defaults while ``#_return_defaults_when_missing_#`` is true,
6959
7083
  else nothing. A missing key of a loose file prints nothing.
@@ -6972,7 +7096,7 @@ def _cliGet(args, delimiter, logger, stdin, stdout):
6972
7096
  missing = False
6973
7097
  for key in args.args:
6974
7098
  if spec:
6975
- key = key.rstrip(' \t') if state.strip else key
7099
+ key = _specNameKey(key, delimiter, state.strip, data)
6976
7100
  else:
6977
7101
  key = key.rstrip()
6978
7102
  row = data.get(key)
@@ -7018,15 +7142,20 @@ def _cliStdinBatch(args, delimiter, logger, stdin, keysOnly):
7018
7142
  so is a line with an empty key (reported). ``keysOnly`` keeps the first
7019
7143
  field only: a tombstone, or a marker reset.
7020
7144
  """
7021
- spec = _isSpecPath(args.store)
7145
+ rows = _cliStdinRows(args, delimiter, logger, stdin, keysOnly)
7146
+ if not _isSpecPath(args.store):
7147
+ return [cells for cells, marker in rows]
7148
+ return [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows]
7149
+
7150
+
7151
+ def _cliStdinRows(args, delimiter, logger, stdin, keysOnly):
7152
+ """Standard input for ``set STORE -`` / ``delete STORE -`` as ``[(cells, marker), ...]`` (spec §20.3)."""
7022
7153
  source = _Reporter('<stdin>', logger)
7023
7154
  try:
7024
- rows = _parseRecordLines(_cliStdinLines(stdin, source), delimiter, spec, keysOnly, source)
7155
+ return _parseRecordLines(_cliStdinLines(stdin, source), delimiter, _isSpecPath(args.store), keysOnly,
7156
+ source)
7025
7157
  finally:
7026
7158
  source.flush()
7027
- if not spec:
7028
- return [cells for cells, marker in rows]
7029
- return [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows]
7030
7159
 
7031
7160
 
7032
7161
  def _parseRecordLines(texts, delimiter, spec, keysOnly, reporter):
@@ -7091,12 +7220,33 @@ def _cliDelete(args, delimiter, logger, stdin, stdout):
7091
7220
  """``delete STORE KEY [KEY ...]`` (spec §20.2): one tombstone per KEY, appended in one write.
7092
7221
 
7093
7222
  ``delete STORE -`` deletes the first field of every line on standard input (§20.3).
7223
+ A key that a reader would strip names a key of the store (§20.2.7), so the
7224
+ store is read first; when it cannot be read whole nothing is written and
7225
+ the exit status is 1.
7094
7226
  """
7095
- if args.args == ['-']:
7096
- _cliAppendBatch(args, delimiter, logger, _cliStdinBatch(args, delimiter, logger, stdin, keysOnly=True))
7227
+ if not _isSpecPath(args.store):
7228
+ if args.args == ['-']:
7229
+ _cliAppendBatch(args, delimiter, logger, _cliStdinBatch(args, delimiter, logger, stdin, keysOnly=True))
7230
+ else:
7231
+ appendLinesTabularFile(args.store, [[key] for key in args.args], teeLogger=logger, header=args.header,
7232
+ createIfNotExist=True, verbose=args.verbose, strict=args.strict,
7233
+ delimiter=delimiter)
7097
7234
  return _cliWritten(args, logger)
7098
- appendLinesTabularFile(args.store, [[key] for key in args.args], teeLogger=logger, header=args.header,
7099
- createIfNotExist=True, verbose=args.verbose, strict=args.strict, delimiter=delimiter)
7235
+ if args.args == ['-']:
7236
+ rows = _cliStdinRows(args, delimiter, logger, stdin, keysOnly=True)
7237
+ else:
7238
+ rows = [([key], bool(_MARKER_RE.match(key))) for key in args.args]
7239
+ if _specNamesKeys(rows, delimiter):
7240
+ strip, live = True, {} # a store delete creates starts with stripping on (spec §10.3)
7241
+ if _cliStoreExists(args.store):
7242
+ load, readable = _cliSpecLoad(args, delimiter, logger)
7243
+ if not readable:
7244
+ logger.teelog('tsvz: {}: could not read every part of the store; nothing was written'.format(
7245
+ args.store), 'error')
7246
+ return 1
7247
+ strip, live = load.state.strip, load.data
7248
+ rows = _specDeleteRows(rows, delimiter, strip, live)
7249
+ _cliAppendBatch(args, delimiter, logger, [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows])
7100
7250
  return _cliWritten(args, logger)
7101
7251
 
7102
7252
 
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes