TSVZ 4.6__tar.gz → 4.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {tsvz-4.6 → tsvz-4.7}/PKG-INFO +5 -2
- {tsvz-4.6 → tsvz-4.7}/README.md +4 -1
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/PKG-INFO +5 -2
- {tsvz-4.6 → tsvz-4.7}/TSVZ.py +170 -32
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/SOURCES.txt +0 -0
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/dependency_links.txt +0 -0
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/entry_points.txt +0 -0
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/requires.txt +0 -0
- {tsvz-4.6 → tsvz-4.7}/TSVZ.egg-info/top_level.txt +0 -0
- {tsvz-4.6 → tsvz-4.7}/setup.cfg +0 -0
- {tsvz-4.6 → tsvz-4.7}/setup.py +0 -0
{tsvz-4.6 → tsvz-4.7}/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: TSVZ
|
|
3
|
-
Version: 4.
|
|
3
|
+
Version: 4.7
|
|
4
4
|
Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
|
|
5
5
|
Home-page: https://github.com/yufei-pan/TSVZ
|
|
6
6
|
Author: Yufei Pan
|
|
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
|
|
|
34
34
|
row is the key; the file behaves like an ordered dictionary that is persisted to
|
|
35
35
|
disk as you change it.
|
|
36
36
|
|
|
37
|
-
**Version 4.
|
|
37
|
+
**Version 4.7**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
|
|
38
38
|
next to your code and `import TSVZ`, or install it.
|
|
39
39
|
|
|
40
40
|
```bash
|
|
@@ -224,6 +224,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
224
224
|
| L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
|
|
225
225
|
| L5 | A damaged compressed file is read up to the damage instead of raising. |
|
|
226
226
|
| L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
|
|
227
|
+
| L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
|
|
227
228
|
|
|
228
229
|
### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
|
|
229
230
|
|
|
@@ -243,6 +244,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
243
244
|
| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
|
|
244
245
|
| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
|
|
245
246
|
| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
|
|
247
|
+
| S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
|
|
246
248
|
|
|
247
249
|
### The `tsvz` command
|
|
248
250
|
|
|
@@ -262,6 +264,7 @@ and `-d=,`, except the ones C5 and C9 list.
|
|
|
262
264
|
| C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
|
|
263
265
|
| C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
|
|
264
266
|
| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
|
|
267
|
+
| C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
|
|
265
268
|
|
|
266
269
|
## Spec deviations and interpretations
|
|
267
270
|
|
{tsvz-4.6 → tsvz-4.7}/README.md
RENAMED
|
@@ -7,7 +7,7 @@ key-value store in a delimiter-separated text file**. The first column of each
|
|
|
7
7
|
row is the key; the file behaves like an ordered dictionary that is persisted to
|
|
8
8
|
disk as you change it.
|
|
9
9
|
|
|
10
|
-
**Version 4.
|
|
10
|
+
**Version 4.7**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
|
|
11
11
|
next to your code and `import TSVZ`, or install it.
|
|
12
12
|
|
|
13
13
|
```bash
|
|
@@ -197,6 +197,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
197
197
|
| L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
|
|
198
198
|
| L5 | A damaged compressed file is read up to the damage instead of raising. |
|
|
199
199
|
| L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
|
|
200
|
+
| L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
|
|
200
201
|
|
|
201
202
|
### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
|
|
202
203
|
|
|
@@ -216,6 +217,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
216
217
|
| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
|
|
217
218
|
| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
|
|
218
219
|
| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
|
|
220
|
+
| S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
|
|
219
221
|
|
|
220
222
|
### The `tsvz` command
|
|
221
223
|
|
|
@@ -235,6 +237,7 @@ and `-d=,`, except the ones C5 and C9 list.
|
|
|
235
237
|
| C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
|
|
236
238
|
| C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
|
|
237
239
|
| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
|
|
240
|
+
| C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
|
|
238
241
|
|
|
239
242
|
## Spec deviations and interpretations
|
|
240
243
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: TSVZ
|
|
3
|
-
Version: 4.
|
|
3
|
+
Version: 4.7
|
|
4
4
|
Summary: Key-value store kept in a TSV / CSV / NSV / PSV file, with tsvz-spec-v1 support
|
|
5
5
|
Home-page: https://github.com/yufei-pan/TSVZ
|
|
6
6
|
Author: Yufei Pan
|
|
@@ -34,7 +34,7 @@ key-value store in a delimiter-separated text file**. The first column of each
|
|
|
34
34
|
row is the key; the file behaves like an ordered dictionary that is persisted to
|
|
35
35
|
disk as you change it.
|
|
36
36
|
|
|
37
|
-
**Version 4.
|
|
37
|
+
**Version 4.7**, Python **3.6+**, one self-contained module (`TSVZ.py`): drop it
|
|
38
38
|
next to your code and `import TSVZ`, or install it.
|
|
39
39
|
|
|
40
40
|
```bash
|
|
@@ -224,6 +224,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
224
224
|
| L4 | New warnings on stderr (invalid encoding, lines dropped by `strict`); data unchanged. |
|
|
225
225
|
| L5 | A damaged compressed file is read up to the damage instead of raising. |
|
|
226
226
|
| L6 | `mapToFile` no longer glues a row it adds at the end of the file to the next one. |
|
|
227
|
+
| L7 | `TSVZed`'s `in` and `get` find a key as `t[key]` does: the live key equal to it, else the key right-stripped. 3.39 matched them exactly, so `'x ' in t` was false while `t['x ']` read `x`, and `t.setdefault('x ', …)` overwrote `x`. |
|
|
227
228
|
|
|
228
229
|
### `.tsvz`, `.csvz`, `.nsvz`, `.psvz`: now tsvz-spec-v1
|
|
229
230
|
|
|
@@ -243,6 +244,7 @@ delimiter or encoding argument that conflicts with the extension.
|
|
|
243
244
|
| S12 | `TSVZedLite.clear()` and `clear_file()` raise `OSError` when the clear is refused (a part could not be read, the path names one part of a store, or the store kept changing). Memory is reloaded from the unchanged store first. |
|
|
244
245
|
| S13 | Compressed parts are written in members of at most 1 MiB of records (`#_member_size_#`), Zstandard frames with a checksum; an append never reads or repairs the part. |
|
|
245
246
|
| S14 | A zero byte in a value is written `<NUL>`, and a raw zero byte in a record is read as zero-fill damage (spec §4.9): under the default `recover` only the bytes after it are read. **A value that 4.2 or 4.3 wrote with a raw zero byte therefore reads differently in 4.4.** Such values are rare; before upgrading, `grep -c -P '\x00' store.tsvz` (any `.tsvz`, `.csvz` or `.psvz` part) finds them. Stop 4.2 and 4.3 handlers before upgrading: they read `<NUL>` as text, as they read `<CR>`. |
|
|
247
|
+
| S15 | A lookup (`t[key]`, `in`, `get`, `del`, `pop`, `setdefault`) takes the live key equal to the one given before the right-stripped one, so a key ending in whitespace (written while `#_strip_trailing_whites_#` was false) can be read and deleted by name (spec §20.2.7). Setting a key still right-strips it. |
|
|
246
248
|
|
|
247
249
|
### The `tsvz` command
|
|
248
250
|
|
|
@@ -262,6 +264,7 @@ and `-d=,`, except the ones C5 and C9 list.
|
|
|
262
264
|
| C9 | Arguments after STORE for `read`, `clear` and `scrub` are a usage error, exit 2 (3.39 ignored them, so `clear junk` cleared the file). |
|
|
263
265
|
| C10 | The table (a terminal, or `--format table`) of `get`, `keys`, `pop`, `popitem`, `setdefault`, and of `read` on a `.tsvz`-family store, shows the rows only: no row is drawn as a header, and no row is cut to the first row's or the terminal's width. `read` of a `.tsv`-family file keeps 3.39's table, whose first row is the file's header line. |
|
|
264
266
|
| C11 | The table shows control characters, bidi controls and line separators as escapes (`\r`, `\n`, `\t`, `\xNN`, `\uNNNN`), so no value can move the cursor, reorder the line or restyle the terminal. It sizes columns in terminal columns, so CJK text (two columns a character) lines up. The table is for reading: a backslash in a value is shown as is, so use records for exact values. |
|
|
267
|
+
| C12 | While stripping is on, `get`, `has`, `pop`, `delete` and `setdefault` take the live key equal to a KEY ending in a space or tab before the stripped one (spec §20.2.7), so `tsvz keys s \| tsvz delete s -` deletes every key. `delete` of such a KEY reads the store first, and exits 1 without writing when the store cannot be read whole. |
|
|
265
268
|
|
|
266
269
|
## Spec deviations and interpretations
|
|
267
270
|
|
{tsvz-4.6 → tsvz-4.7}/TSVZ.py
RENAMED
|
@@ -40,10 +40,10 @@ if os.name == 'nt':
|
|
|
40
40
|
elif os.name == 'posix':
|
|
41
41
|
import fcntl
|
|
42
42
|
|
|
43
|
-
version = '4.
|
|
43
|
+
version = '4.7'
|
|
44
44
|
__version__ = version
|
|
45
45
|
author = 'pan@zopyr.us'
|
|
46
|
-
COMMIT_DATE = '2026-10-
|
|
46
|
+
COMMIT_DATE = '2026-10-13'
|
|
47
47
|
|
|
48
48
|
DEFAULT_DELIMITER = '\t'
|
|
49
49
|
DEFAULTS_INDICATOR_KEY = '#_defaults_#'
|
|
@@ -1605,6 +1605,8 @@ def scrubTabularFile(fileName,teeLogger = None,header = '',createIfNotExist = Fa
|
|
|
1605
1605
|
# ===========================================================================
|
|
1606
1606
|
MAX_SPEC_VERSION = 1
|
|
1607
1607
|
_MARKER_RE = re.compile(r'^#_[A-Za-z0-9_-]+_#$')
|
|
1608
|
+
#: What a lookup gets for a key that is not live (spec §20.2.7: then the stripped key is tried).
|
|
1609
|
+
_NO_ROW = object()
|
|
1608
1610
|
_CHECKSUM_RE = re.compile(r'^#_checksum_([A-Za-z0-9_-]+)_#$', re.IGNORECASE)
|
|
1609
1611
|
_ASCII_DIGITS_RE = re.compile(r'^[0-9]+$')
|
|
1610
1612
|
_SPEC_DECODE_RE = re.compile(r'<(?:sep|LF|CR|NUL|lt|#)>')
|
|
@@ -1669,8 +1671,7 @@ def _specTombstones(keys, delimiter, strip):
|
|
|
1669
1671
|
(spec §9.6).
|
|
1670
1672
|
"""
|
|
1671
1673
|
rows = [([key], False) for key in keys]
|
|
1672
|
-
if strip and any(key
|
|
1673
|
-
for key in keys):
|
|
1674
|
+
if strip and any(_specStrippable(key, delimiter) for key in keys):
|
|
1674
1675
|
rows.insert(0, (['#_strip_trailing_whites_#', 'false'], True))
|
|
1675
1676
|
rows.append((['#_strip_trailing_whites_#', 'true'], True))
|
|
1676
1677
|
return rows
|
|
@@ -1687,6 +1688,55 @@ def _specTombstoneLines(key, delimiter, strip):
|
|
|
1687
1688
|
for cells, marker in _specTombstones([key], delimiter, strip))
|
|
1688
1689
|
|
|
1689
1690
|
|
|
1691
|
+
def _specStrippable(key, delimiter):
|
|
1692
|
+
"""True when a reader that strips (spec §7.7) would change ``key``: its encoded form ends in a space or tab."""
|
|
1693
|
+
return key.endswith((' ', '\t')) and _specEncodeField(key, delimiter, isKey=True).endswith((' ', '\t'))
|
|
1694
|
+
|
|
1695
|
+
|
|
1696
|
+
def _specNameKey(key, delimiter, strip, live):
|
|
1697
|
+
"""The key that a requested ``key`` names (spec §20.2.7) in a store whose stripping is ``strip``.
|
|
1698
|
+
|
|
1699
|
+
While stripping is on, a key a reader would strip names the live key equal
|
|
1700
|
+
to it (``live`` holds the live keys), else the key a reader resolves it to.
|
|
1701
|
+
"""
|
|
1702
|
+
if not strip or key in live or not _specStrippable(key, delimiter):
|
|
1703
|
+
return key
|
|
1704
|
+
return _specDecodeField(_specEncodeField(key, delimiter, isKey=True).rstrip(' \t'), delimiter)
|
|
1705
|
+
|
|
1706
|
+
|
|
1707
|
+
def _specNamesKeys(rows, delimiter):
|
|
1708
|
+
"""True when naming the keys of ``delete`` rows (``[(cells, marker), ...]``) needs the store (spec §20.2.7)."""
|
|
1709
|
+
return any(not marker and _specStrippable(cells[0], delimiter) for cells, marker in rows)
|
|
1710
|
+
|
|
1711
|
+
|
|
1712
|
+
def _specDeleteRows(rows, delimiter, strip, live):
|
|
1713
|
+
"""The batch ``delete`` appends for ``rows`` (``[(cells, marker), ...]``, spec §20.2, §20.3).
|
|
1714
|
+
|
|
1715
|
+
Every key names a key of the store as it was before the batch, whose
|
|
1716
|
+
stripping was ``strip`` and whose live keys are ``live`` (§20.2.7). The
|
|
1717
|
+
tombstone of a key a reader would strip is written with stripping off
|
|
1718
|
+
(§9.6). A reset of ``#_strip_trailing_whites_#`` in the batch turns
|
|
1719
|
+
stripping back on (§10.3, §12.1).
|
|
1720
|
+
"""
|
|
1721
|
+
batch = []
|
|
1722
|
+
named = strip
|
|
1723
|
+
guarded = False
|
|
1724
|
+
for cells, marker in rows:
|
|
1725
|
+
if marker:
|
|
1726
|
+
if cells[0].lower() == '#_strip_trailing_whites_#':
|
|
1727
|
+
strip, guarded = True, False
|
|
1728
|
+
batch.append((cells, marker))
|
|
1729
|
+
continue
|
|
1730
|
+
key = _specNameKey(cells[0], delimiter, named, live)
|
|
1731
|
+
if strip and not guarded and _specStrippable(key, delimiter):
|
|
1732
|
+
batch.append((['#_strip_trailing_whites_#', 'false'], True))
|
|
1733
|
+
guarded = True
|
|
1734
|
+
batch.append(([key], False))
|
|
1735
|
+
if guarded:
|
|
1736
|
+
batch.append((['#_strip_trailing_whites_#', 'true'], True))
|
|
1737
|
+
return batch
|
|
1738
|
+
|
|
1739
|
+
|
|
1690
1740
|
class _SpecState(object):
|
|
1691
1741
|
"""Reader state of spec §12, carried across records and parts."""
|
|
1692
1742
|
|
|
@@ -3628,14 +3678,37 @@ class TSVZed(OrderedDict):
|
|
|
3628
3678
|
# else:
|
|
3629
3679
|
# self.appendEvent.set()
|
|
3630
3680
|
|
|
3681
|
+
def _nameKey(self, key):
|
|
3682
|
+
"""The key a requested ``key`` names: the live key equal to it, else ``key`` right-stripped (spec §20.2.7).
|
|
3683
|
+
|
|
3684
|
+
The fallback strips every trailing whitespace whatever the stripping
|
|
3685
|
+
marker says, as ``__setitem__`` strips every key it writes.
|
|
3686
|
+
"""
|
|
3687
|
+
key = str(key)
|
|
3688
|
+
return key if OrderedDict.__contains__(self, key) else key.rstrip()
|
|
3689
|
+
|
|
3631
3690
|
def __getitem__(self, key):
|
|
3632
|
-
|
|
3633
|
-
|
|
3691
|
+
key = str(key)
|
|
3692
|
+
row = dict.get(self, key, _NO_ROW) # one probe for the usual hit
|
|
3693
|
+
return row if row is not _NO_ROW else OrderedDict.__getitem__(self, key.rstrip())
|
|
3694
|
+
|
|
3695
|
+
def __contains__(self, key):
|
|
3696
|
+
# OrderedDict.setdefault asks this before it reads or writes, so it names keys as t[key] does.
|
|
3697
|
+
key = str(key)
|
|
3698
|
+
return dict.__contains__(self, key) or dict.__contains__(self, key.rstrip())
|
|
3699
|
+
|
|
3700
|
+
def get(self, key, default=None):
|
|
3701
|
+
key = str(key)
|
|
3702
|
+
row = dict.get(self, key, _NO_ROW)
|
|
3703
|
+
if row is _NO_ROW:
|
|
3704
|
+
row = dict.get(self, key.rstrip(), _NO_ROW)
|
|
3705
|
+
return default if row is _NO_ROW else row
|
|
3706
|
+
|
|
3634
3707
|
|
|
3635
3708
|
def __delitem__(self,key):
|
|
3636
3709
|
if self._spec:
|
|
3637
3710
|
return self._specDelItem(key)
|
|
3638
|
-
key =
|
|
3711
|
+
key = self._nameKey(key)
|
|
3639
3712
|
if key == DEFAULTS_INDICATOR_KEY:
|
|
3640
3713
|
self.defaults = [DEFAULTS_INDICATOR_KEY]
|
|
3641
3714
|
if self.verbose:
|
|
@@ -3739,7 +3812,7 @@ class TSVZed(OrderedDict):
|
|
|
3739
3812
|
self.lastUpdateTime = get_time_ns()
|
|
3740
3813
|
|
|
3741
3814
|
def _specDelItem(self, key):
|
|
3742
|
-
key =
|
|
3815
|
+
key = self._nameKey(key)
|
|
3743
3816
|
if _MARKER_RE.match(key):
|
|
3744
3817
|
# A lone marker key resets that marker (spec §12.1).
|
|
3745
3818
|
self._specSetMarker(key, [key])
|
|
@@ -3914,12 +3987,14 @@ memoryOnly:{self.memoryOnly}
|
|
|
3914
3987
|
is raised.
|
|
3915
3988
|
|
|
3916
3989
|
'''
|
|
3917
|
-
key =
|
|
3918
|
-
if
|
|
3990
|
+
key = self._nameKey(key)
|
|
3991
|
+
if not OrderedDict.__contains__(self, key):
|
|
3919
3992
|
if default is self.__marker:
|
|
3920
3993
|
raise KeyError(key)
|
|
3921
3994
|
return default
|
|
3922
|
-
|
|
3995
|
+
# Not OrderedDict.pop: before Python 3.11 it goes through del self[key], which writes a tombstone too.
|
|
3996
|
+
value = OrderedDict.__getitem__(self, key)
|
|
3997
|
+
OrderedDict.__delitem__(self, key)
|
|
3923
3998
|
if not self.memoryOnly:
|
|
3924
3999
|
self.__appendEmptyLine(key)
|
|
3925
4000
|
self.lastUpdateTime = get_time_ns()
|
|
@@ -3929,7 +4004,7 @@ memoryOnly:{self.memoryOnly}
|
|
|
3929
4004
|
'''Move an existing element to the end (or beginning if last is false).
|
|
3930
4005
|
Raise KeyError if the element does not exist.
|
|
3931
4006
|
'''
|
|
3932
|
-
key =
|
|
4007
|
+
key = self._nameKey(key)
|
|
3933
4008
|
super().move_to_end(key, last)
|
|
3934
4009
|
if self._spec:
|
|
3935
4010
|
_warnOnce(self, 'move_to_end', self._fileName,
|
|
@@ -4452,9 +4527,18 @@ class TSVZedLite(MutableMapping):
|
|
|
4452
4527
|
"""'tsvz' for tsvz-spec-v1 files (.tsvz/.csvz/.nsvz/.psvz), 'tsv' for 3.39-format files."""
|
|
4453
4528
|
return 'tsvz' if self._spec else 'tsv'
|
|
4454
4529
|
|
|
4530
|
+
def _nameKey(self, key):
|
|
4531
|
+
"""The key a requested ``key`` names: the live key equal to it, else ``key`` right-stripped (spec §20.2.7).
|
|
4532
|
+
|
|
4533
|
+
The fallback strips every trailing whitespace whatever the stripping
|
|
4534
|
+
marker says, as ``__setitem__`` strips every key it writes.
|
|
4535
|
+
"""
|
|
4536
|
+
key = str(key)
|
|
4537
|
+
return key if key in self.indexes else key.rstrip()
|
|
4538
|
+
|
|
4455
4539
|
def __contains__(self, key):
|
|
4456
4540
|
if self._spec:
|
|
4457
|
-
return
|
|
4541
|
+
return self._nameKey(key) in self.indexes
|
|
4458
4542
|
return MutableMapping.__contains__(self, key)
|
|
4459
4543
|
|
|
4460
4544
|
def get(self, key, default=None):
|
|
@@ -4832,7 +4916,7 @@ class TSVZedLite(MutableMapping):
|
|
|
4832
4916
|
|
|
4833
4917
|
# Implement basic __getitem__, __setitem__, __delitem__, __iter__, and __len__. needed for MutableMapping
|
|
4834
4918
|
def __getitem__(self,key):
|
|
4835
|
-
key =
|
|
4919
|
+
key = self._nameKey(key)
|
|
4836
4920
|
if key not in self.indexes:
|
|
4837
4921
|
if key == DEFAULTS_INDICATOR_KEY:
|
|
4838
4922
|
return self.defaults
|
|
@@ -4911,7 +4995,7 @@ class TSVZedLite(MutableMapping):
|
|
|
4911
4995
|
self.indexes[key] = self.__writeValues(value)
|
|
4912
4996
|
|
|
4913
4997
|
def __delitem__(self,key):
|
|
4914
|
-
key =
|
|
4998
|
+
key = self._nameKey(key)
|
|
4915
4999
|
if self._spec:
|
|
4916
5000
|
if _MARKER_RE.match(key):
|
|
4917
5001
|
# A lone marker key resets that marker (spec §12.1).
|
|
@@ -4969,7 +5053,7 @@ class TSVZedLite(MutableMapping):
|
|
|
4969
5053
|
|
|
4970
5054
|
__marker = object()
|
|
4971
5055
|
def pop(self, key, default=__marker, return_pos = False):
|
|
4972
|
-
key =
|
|
5056
|
+
key = self._nameKey(key)
|
|
4973
5057
|
try:
|
|
4974
5058
|
pos = self.indexes.pop(key)
|
|
4975
5059
|
except KeyError:
|
|
@@ -5430,9 +5514,9 @@ class _ServeStore(object):
|
|
|
5430
5514
|
return not self.parts
|
|
5431
5515
|
|
|
5432
5516
|
def resolve(self, key):
|
|
5433
|
-
"""
|
|
5517
|
+
"""The key a requested key names (spec §20.2.7; 3.39 rstrip for loose files)."""
|
|
5434
5518
|
if self.spec:
|
|
5435
|
-
return key.
|
|
5519
|
+
return _specNameKey(key, self.delimiter, self.state.strip, self.data)
|
|
5436
5520
|
return key.rstrip()
|
|
5437
5521
|
|
|
5438
5522
|
def read(self, log):
|
|
@@ -5528,10 +5612,27 @@ class _ServeStore(object):
|
|
|
5528
5612
|
self.writer.wait(queued[0])
|
|
5529
5613
|
|
|
5530
5614
|
def write(self, rows, sync, log, written=False):
|
|
5531
|
-
"""Append ``rows`` (``[(cells, marker), ...]``) as one batch: ``set
|
|
5615
|
+
"""Append ``rows`` (``[(cells, marker), ...]``) as one batch: ``set`` and its bulk input."""
|
|
5616
|
+
with self.lock:
|
|
5617
|
+
queued = self._submit(rows, sync, log)
|
|
5618
|
+
self._ack(queued, written)
|
|
5619
|
+
|
|
5620
|
+
def delete(self, rows, sync, log, written=False):
|
|
5621
|
+
"""Append the tombstones of ``delete`` (``[(cells, marker), ...]``) as one batch (spec §20.2.7).
|
|
5622
|
+
|
|
5623
|
+
Naming a key a reader would strip needs the store as it is now, read
|
|
5624
|
+
under the lock, so that delete is atomic (spec §21.8.2). Returns False,
|
|
5625
|
+
having written nothing, when that read could not be completed.
|
|
5626
|
+
"""
|
|
5532
5627
|
with self.lock:
|
|
5628
|
+
if self.spec and _specNamesKeys(rows, self.delimiter):
|
|
5629
|
+
self._fresh(log)
|
|
5630
|
+
if self.unreadable:
|
|
5631
|
+
return False
|
|
5632
|
+
rows = _specDeleteRows(rows, self.delimiter, self.state.strip, self.data)
|
|
5533
5633
|
queued = self._submit(rows, sync, log)
|
|
5534
5634
|
self._ack(queued, written)
|
|
5635
|
+
return True
|
|
5535
5636
|
|
|
5536
5637
|
def pop(self, key, sync, log):
|
|
5537
5638
|
"""Atomically return the row of ``key`` and append its tombstone; None when it is missing."""
|
|
@@ -5781,7 +5882,10 @@ def _serveDispatch(store, line, bulk, log):
|
|
|
5781
5882
|
rows = [([key], bool(_MARKER_RE.match(field))) for key, field in zip(args, raw)]
|
|
5782
5883
|
else:
|
|
5783
5884
|
rows = [(args, bool(_MARKER_RE.match(raw[0])))]
|
|
5784
|
-
|
|
5885
|
+
if not keysOnly:
|
|
5886
|
+
store.write(rows, sync, log, written)
|
|
5887
|
+
elif not store.delete(rows, sync, log, written):
|
|
5888
|
+
return 1, [], _SERVE_UNREADABLE + '; nothing was written'
|
|
5785
5889
|
return 0, [], ''
|
|
5786
5890
|
if operation in ('pop', 'popitem'):
|
|
5787
5891
|
if operation == 'pop':
|
|
@@ -6482,7 +6586,10 @@ class TSVZClient(MutableMapping):
|
|
|
6482
6586
|
|
|
6483
6587
|
def __delitem__(self, key):
|
|
6484
6588
|
"""Delete ``key``; a missing key is not an error (as in ``TSVZed``)."""
|
|
6485
|
-
|
|
6589
|
+
key = str(key)
|
|
6590
|
+
# Resending the delete of a key ending in whitespace could name, and delete, the stripped key (spec §20.2.7).
|
|
6591
|
+
retry = not (self._spec and _specStrippable(key, self.delimiter))
|
|
6592
|
+
self._request(['delete', _serveEncode(key)], write=True, retry=retry)
|
|
6486
6593
|
|
|
6487
6594
|
_marker = object()
|
|
6488
6595
|
|
|
@@ -6508,6 +6615,11 @@ class TSVZClient(MutableMapping):
|
|
|
6508
6615
|
cells = self._cells(key, default)
|
|
6509
6616
|
if len(cells) < 2:
|
|
6510
6617
|
return self.get(key)
|
|
6618
|
+
if str(key) != cells[0]:
|
|
6619
|
+
# cells[0] is stripped as set strips it; a live key equal to ``key`` wins first (spec §20.2.7).
|
|
6620
|
+
row = self.get(key)
|
|
6621
|
+
if row is not None:
|
|
6622
|
+
return row
|
|
6511
6623
|
lines = self._request(['setdefault'] + self._fields(cells), write=True, retry=False)[0]
|
|
6512
6624
|
return _serveDecodeLine(lines[0]) if lines else None
|
|
6513
6625
|
|
|
@@ -6953,7 +7065,7 @@ def _cliRead(args, delimiter, logger, stdin, stdout):
|
|
|
6953
7065
|
def _cliGet(args, delimiter, logger, stdin, stdout):
|
|
6954
7066
|
"""``get STORE KEY ...`` (spec §20.2): each KEY's row in argument order; exit 3 when one is missing.
|
|
6955
7067
|
|
|
6956
|
-
|
|
7068
|
+
Each KEY names a key as §20.2.7 says. For a missing key of a
|
|
6957
7069
|
spec store this prints what ``TSVZed`` returns for it (§14.5): the key and
|
|
6958
7070
|
the active defaults while ``#_return_defaults_when_missing_#`` is true,
|
|
6959
7071
|
else nothing. A missing key of a loose file prints nothing.
|
|
@@ -6972,7 +7084,7 @@ def _cliGet(args, delimiter, logger, stdin, stdout):
|
|
|
6972
7084
|
missing = False
|
|
6973
7085
|
for key in args.args:
|
|
6974
7086
|
if spec:
|
|
6975
|
-
key = key
|
|
7087
|
+
key = _specNameKey(key, delimiter, state.strip, data)
|
|
6976
7088
|
else:
|
|
6977
7089
|
key = key.rstrip()
|
|
6978
7090
|
row = data.get(key)
|
|
@@ -7018,15 +7130,20 @@ def _cliStdinBatch(args, delimiter, logger, stdin, keysOnly):
|
|
|
7018
7130
|
so is a line with an empty key (reported). ``keysOnly`` keeps the first
|
|
7019
7131
|
field only: a tombstone, or a marker reset.
|
|
7020
7132
|
"""
|
|
7021
|
-
|
|
7133
|
+
rows = _cliStdinRows(args, delimiter, logger, stdin, keysOnly)
|
|
7134
|
+
if not _isSpecPath(args.store):
|
|
7135
|
+
return [cells for cells, marker in rows]
|
|
7136
|
+
return [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows]
|
|
7137
|
+
|
|
7138
|
+
|
|
7139
|
+
def _cliStdinRows(args, delimiter, logger, stdin, keysOnly):
|
|
7140
|
+
"""Standard input for ``set STORE -`` / ``delete STORE -`` as ``[(cells, marker), ...]`` (spec §20.3)."""
|
|
7022
7141
|
source = _Reporter('<stdin>', logger)
|
|
7023
7142
|
try:
|
|
7024
|
-
|
|
7143
|
+
return _parseRecordLines(_cliStdinLines(stdin, source), delimiter, _isSpecPath(args.store), keysOnly,
|
|
7144
|
+
source)
|
|
7025
7145
|
finally:
|
|
7026
7146
|
source.flush()
|
|
7027
|
-
if not spec:
|
|
7028
|
-
return [cells for cells, marker in rows]
|
|
7029
|
-
return [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows]
|
|
7030
7147
|
|
|
7031
7148
|
|
|
7032
7149
|
def _parseRecordLines(texts, delimiter, spec, keysOnly, reporter):
|
|
@@ -7091,12 +7208,33 @@ def _cliDelete(args, delimiter, logger, stdin, stdout):
|
|
|
7091
7208
|
"""``delete STORE KEY [KEY ...]`` (spec §20.2): one tombstone per KEY, appended in one write.
|
|
7092
7209
|
|
|
7093
7210
|
``delete STORE -`` deletes the first field of every line on standard input (§20.3).
|
|
7211
|
+
A key that a reader would strip names a key of the store (§20.2.7), so the
|
|
7212
|
+
store is read first; when it cannot be read whole nothing is written and
|
|
7213
|
+
the exit status is 1.
|
|
7094
7214
|
"""
|
|
7095
|
-
if args.
|
|
7096
|
-
|
|
7215
|
+
if not _isSpecPath(args.store):
|
|
7216
|
+
if args.args == ['-']:
|
|
7217
|
+
_cliAppendBatch(args, delimiter, logger, _cliStdinBatch(args, delimiter, logger, stdin, keysOnly=True))
|
|
7218
|
+
else:
|
|
7219
|
+
appendLinesTabularFile(args.store, [[key] for key in args.args], teeLogger=logger, header=args.header,
|
|
7220
|
+
createIfNotExist=True, verbose=args.verbose, strict=args.strict,
|
|
7221
|
+
delimiter=delimiter)
|
|
7097
7222
|
return _cliWritten(args, logger)
|
|
7098
|
-
|
|
7099
|
-
|
|
7223
|
+
if args.args == ['-']:
|
|
7224
|
+
rows = _cliStdinRows(args, delimiter, logger, stdin, keysOnly=True)
|
|
7225
|
+
else:
|
|
7226
|
+
rows = [([key], bool(_MARKER_RE.match(key))) for key in args.args]
|
|
7227
|
+
if _specNamesKeys(rows, delimiter):
|
|
7228
|
+
strip, live = True, {} # a store delete creates starts with stripping on (spec §10.3)
|
|
7229
|
+
if _cliStoreExists(args.store):
|
|
7230
|
+
load, readable = _cliSpecLoad(args, delimiter, logger)
|
|
7231
|
+
if not readable:
|
|
7232
|
+
logger.teelog('tsvz: {}: could not read every part of the store; nothing was written'.format(
|
|
7233
|
+
args.store), 'error')
|
|
7234
|
+
return 1
|
|
7235
|
+
strip, live = load.state.strip, load.data
|
|
7236
|
+
rows = _specDeleteRows(rows, delimiter, strip, live)
|
|
7237
|
+
_cliAppendBatch(args, delimiter, logger, [_specFormatRecord(cells, delimiter, marker=marker) for cells, marker in rows])
|
|
7100
7238
|
return _cliWritten(args, logger)
|
|
7101
7239
|
|
|
7102
7240
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{tsvz-4.6 → tsvz-4.7}/setup.cfg
RENAMED
|
File without changes
|
{tsvz-4.6 → tsvz-4.7}/setup.py
RENAMED
|
File without changes
|