TSVZ 3.38__tar.gz → 3.39__tar.gz

tsvz-3.39/PKG-INFO

@@ -0,0 +1,396 @@
+Metadata-Version: 2.4
+Name: TSVZ
+Version: 3.39
+Summary: An simple in memory wrapper around a TSV file to function as a database
+Home-page: https://github.com/yufei-pan/TSVZ
+Author: Yufei Pan
+Author-email: pan@zopyr.us
+License: GPLv3+
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+Requires-Dist: argparse
+Provides-Extra: completion
+Requires-Dist: argcomplete; extra == "completion"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# TSVZ
+
+TSVZ is a tiny, dependency-free library and CLI for treating a **tab-separated
+values file as a key-value store**. The first column of every row is a unique
+key; the file behaves like an ordered dictionary that is transparently
+persisted to disk.
+
+Two front-ends are provided:
+
+- **`TSVZed`** — an in-memory `OrderedDict` backed by the file, with a
+  background worker that does **non-blocking, append-only** writes, optional
+  periodic / on-exit rewrites, file locking for multi-process access, and
+  transparent (de)compression.
+- **`TSVZedLite`** — a `MutableMapping` that keeps only a **key → byte-offset
+  index** in memory and reads values from disk on demand. Minimal footprint,
+  single-process, append-only.
+
+It is a single self-contained module (`TSVZ.py`) — drop it next to your code and
+`import TSVZ`, or install it.
+
+```bash
+pip install tsvz          # from PyPI
+pip install -e .          # editable, from this directory
+```
+
+---
+
+## Quick start
+
+### As a library
+
+```python
+import TSVZ
+
+# Opens (creating if needed), loads into memory, starts the async writer.
+db = TSVZ.TSVZed('data.tsv', header='id\tname\tscore')
+
+db['alice'] = ['alice', 'Alice', '10']   # value as a list
+db['bob']   = 'bob\tBob\t20'             # ...or a delimited string
+db['alice'] = ['alice', 'Alice', '11']   # last write wins
+del db['bob']                            # deletes (and persists the deletion)
+
+print(db['alice'])        # ['alice', 'Alice', '11']
+db.close()                # flush the queue and stop the worker (also runs atexit)
+```
+
+There are also stateless helper functions if you just want to touch a file once:
+`readTabularFile`, `appendTabularFile`, `appendLinesTabularFile`,
+`clearTabularFile`, `scrubTabularFile` (and `readTSV` / `appendTSV` / … aliases).
+
+### From the command line
+
+```bash
+tsvz data.tsv                          # read + pretty-print
+tsvz data.tsv append alice Alice 10    # append/update a row keyed by "alice"
+tsvz data.tsv delete alice             # delete the row keyed by "alice"
+tsvz data.tsv clear                    # truncate to just the header
+tsvz data.tsv scrub                    # rewrite compactly (drops comments, applies last-wins)
+
+tsvz data.csv -d comma append k v1 v2  # pick a delimiter explicitly
+tsvz -h
+```
+
+---
+
+## The TSVZ file format
+
+> **Status: draft / evolving spec.** The reference code in this repo does not
+> fully implement it yet (see *Implementation notes*).
+
+`.tsvz` / `.csvz` / `.nsvz` / `.psvz` are a **format in their own right** — not
+merely a `.tsv` that TSVZ happens to manage. They define a **strict,
+append-only, write-ahead-log (WAL) style key-value store** that is *also*
+readable by ordinary CSV/TSV tools (which see the rows fine but may
+misinterpret escaped values; see *Escaping*).
+
+The plain, un-suffixed `.tsv` / `.csv` / `.nsv` / `.psv` extensions are **not**
+this format: they remain loose delimited tables with no semantic guarantees. The
+same delimiter mapping carries over to both; only the `z` forms carry the
+stricter contract below.
+
+The format is defined by the rules below; they are written to be implementable by
+any reader/writer, not just this library.
+
+### 0. Model — the file *is* a write-ahead log
+
+- **Every mutation is an append.** There is no in-place update during normal
+  operation. Current state is reconstructed by **replaying the file forward**,
+  **last-wins**.
+- **Key-value, not relational.** First column is the globally-unique key; the
+  remaining columns are the value (variable width; a virtual infinite tail of
+  empty columns; a defaults line as per-column fallback). No schema, no header
+  semantics beyond "a `#` comment."
+- **Delete is a tombstone** (a lone key / all-empty values), never a physical
+  removal — consistent with a log.
+- **Minimal overhead vs. an embedded DB.** Plain UTF-8 text, line-oriented,
+  human-readable and greppable; no page cache, b-tree, or separate journal. You
+  trade random-update efficiency and bounded file size for near-zero write
+  overhead and trivial crash recovery (truncate the last partial line). Fits
+  logging / event streams / transactional state where overwrites are rare, write
+  latency matters, or the full history is worth keeping.
+- **Compaction is occasional maintenance, not normal operation** — the one place
+  the append-only rule is deliberately, explicitly broken. See *Compaction*.
+
+### 1. Encoding & lines
+
+1. The file is **UTF-8** encoded (no BOM).
+2. Records are separated by a line feed (`\n`). A reader **must** tolerate a
+   trailing `\r` (i.e. read CRLF files); writers **should** emit bare `\n`.
+3. The file **should** end with a trailing newline. A reader **must** tolerate a
+   missing final newline.
+4. An empty file (zero bytes) — or a file containing only comments — is a valid,
+   empty TSVZ.
+
+### 2. Delimiter & variants
+
+5. The variants differ only in the field delimiter. The `z` extension marks the
+   strict WAL format; the plain extension is the loose tabular fallback:
+
+   | Variant | Format ext. | Plain ext. | Delimiter |
+   |---------|-------------|------------|-----------|
+   | TSVZ    | `.tsvz`     | `.tsv`     | tab `\t`  |
+   | CSVZ    | `.csvz`     | `.csv`     | comma `,` |
+   | NSVZ    | `.nsvz`     | `.nsv`     | NUL `\0`  |
+   | PSVZ    | `.psvz`     | `.psv`     | pipe `\|` |
+
+6. Each line is a list of fields joined by the delimiter. **Column count is not
+   fixed** — a reader must accept arbitrarily short or long lines. Conceptually
+   every row is its values followed by an infinite tail of empty-string columns,
+   so reading a field beyond a row's length yields `''` (subject to the defaults
+   rule below). A returned row is therefore **not guaranteed to be a fixed
+   length**; a reader is free to additionally validate or pad/trim if it wants.
+
+### 3. Keys & values
+
+7. The **first column is the key** and is **globally unique** within the file.
+8. **Last-wins:** when the same key appears on multiple lines, the **last**
+   occurrence determines the current value. (This is what makes append-only
+   updates work.)
+9. A key's **logical position** is the position of its *first* appearance; later
+   updates change its value but not its order.
+10. **Append-only:** during normal operation a TSVZ file is only ever appended
+    to — an update or a delete is a new line at the end of the file, never an
+    in-place edit. (Compaction is a separate, explicit operation; see *scrub*.)
+
+### 4. Deletion (tombstones)
+
+11. A line consisting of **a lone key with no following values, or whose
+    following values are all empty**, is a **tombstone**: it removes that key if
+    it currently exists, and is tolerated (no-op) if it does not.
+    - e.g. `mykey\n` or `mykey\t\t\n` both delete `mykey`.
+    - Edge case: in a strictly single-column file a lone key *is* the value, so
+      deletion cannot be represented; such files cannot tombstone.
+
+### 5. Whitespace
+
+12. **Trailing whitespace is stripped from every field on read** (including the
+    key) — trailing whitespace is treated as visual padding and is not
+    significant. Leading whitespace **is** significant and preserved. (This does
+    not apply to comment lines, which are not parsed.)
+
+### 6. Comments & header
+
+13. Any line whose first field starts with `#` is a **comment** and is discarded
+    by the reader (with the exception of the reserved `#_..._#` keys below).
+14. There is **no formal header**. A header row, if present, is simply a comment
+    (a line starting with `#`, e.g. `#id\tname\tscore`) and carries no semantics
+    for the data — it is purely human documentation.
+
+### 7. Defaults line
+
+15. A line whose key is **`#_defaults_#`** (case-insensitive — `#_DEFAULTS_#` is
+    equally valid) is a **defaults line**. It is not data; it supplies fallback
+    values per column index.
+16. Defaults have an **unbounded** number of columns and are applied **forward
+    only**: as a reader streams the file, a field that is empty (or absent) in a
+    data row takes the value at the same column index from the **currently
+    active** defaults line.
+    - Rows that appear *before* a defaults line are unaffected by it (the reader
+      does not backtrack).
+    - Example: rows have 5 fields and there is no defaults line ⇒ reading field 6
+      yields `''`. Then a defaults line with 10 fields appears ⇒ for *subsequent*
+      5-field rows, reading field 6 yields the defaults line's field 6.
+17. **Multiple defaults lines are allowed.** A later defaults line **completely
+    replaces** the previous one — if the later line is shorter, the extra columns
+    of the previous line are discarded, not merged.
+18. A `#_defaults_#` line with **no values** clears the active defaults.
+
+### 8. Reserved namespace
+
+19. Keys matching **`#_<ascii letters/digits>_#`** are **reserved** for current
+    and future internal TSVZ semantics (`#_defaults_#` is the first). Don't use
+    that pattern for your own comments. (Conformance is only enforced where a
+    given reserved key is actually defined; unknown `#_..._#` keys are treated as
+    ordinary comments today.)
+
+### 9. Escaping / sanitization
+
+The delimiter and newline can't appear literally inside a field, so they are
+encoded with reversible tokens. Reading inverts writing:
+
+| In a field (in memory) | Stored on disk | Read back as |
+|---|---|---|
+| the delimiter char (e.g. `\t`) | `<sep>` | the delimiter char |
+| line feed `\n` | `<LF>` | line feed `\n` |
+| literal `<sep>` | `</sep/>` | literal `<sep>` |
+| literal `<LF>` | `</LF/>` | literal `<LF>` |
+
+20. Consequently `<sep>` / `<LF>` appearing **in the file** are interpreted as a
+    delimiter / line feed on read. The double-wrapped forms `</sep/>` / `</LF/>`
+    exist so that the literal strings `<sep>` / `<LF>` (e.g. HTML-ish tags) can
+    round-trip.
+21. To keep the mapping finite, the literal strings **`</sep/>` and `</LF/>` are
+    invalid as field data**: if they occur in a file they are read back as the
+    literal `<sep>` / `<LF>` (i.e. they have no escape of their own and are
+    *lossy* — they collapse one level). Writers should avoid emitting them as
+    real data.
+
+### 10. Compression
+
+22. Compression is an **orthogonal stream filter**, not part of the format
+    semantics: any TSVZ file (or any single part of a multi-part set, below) may
+    be wrapped by appending a compression extension — `.gz`/`.gzip`,
+    `.bz2`/`.bzip2`, `.xz`/`.lzma`, `.zst`/`.zstd` (e.g. `log.tsvz.zst`).
+    Delimiter/format inference ignores the compression suffix. A compressed part
+    is rewritten as a whole, so the append-only optimization applies only to
+    uncompressed parts.
+
+### 11. Multi-part files
+
+23. A logical TSVZ store may be **split across multiple parts**. Parts share the
+    same path up to the format extension and add a `.<hex>` ordinal suffix, e.g.
+    `events.tsvz.0`, `events.tsvz.1`, … or with timestamps
+    `events.tsvz.6650a3c0`, `events.tsvz.66518f10`, ….
+24. Parts are ordered by interpreting the suffix as a **hexadecimal (and hence
+    also plain-integer) number, ascending** — smaller = earlier in the log. The
+    logical store is the concatenation of the parts in that order; replay forward
+    / last-wins simply spans part boundaries.
+25. **Each part is itself a complete, valid TSVZ file** (comments, defaults, etc.
+    all legal per part). Multi-part is purely an addressing convention layered on
+    top; a single-file store is just the one-part case.
+26. **Recommended suffix:** the hex (or integer) Unix timestamp of when the part
+    was created — naturally chronological and collision-resistant.
+
+Operational recommendations for multi-part writers:
+
+- On startup, **open a fresh part** named with the current timestamp and append
+  new records there; leave older parts untouched (they become immutable).
+- **Do not compress the current (active) part** — it is being appended to.
+- Provide a processor/flag that, on the next launch, **compresses prior parts**
+  that aren't already compressed (now immutable, so it's safe and frees space).
+  Different parts may use different codecs.
+
+### Additional ideas worth considering (not yet normative)
+
+These came up while writing the spec; flagging them for your call:
+
+- **Format-version marker.** Reserve a `#_tsvz1_#`-style first-line comment so
+  future readers can detect the format/version explicitly. Cheap insurance.
+- **Key constraints.** State explicitly that a *data* key must be non-empty
+  (after trailing-whitespace strip) and must not start with `#` (that prefix is
+  reserved for comments/internal keys).
+- **Append atomicity.** Recommend that a conformant appender write each record
+  as a *single* `write()` of a complete, newline-terminated line (open with
+  `O_APPEND`) so concurrent appenders never interleave partial lines. This is
+  what makes multi-writer last-wins safe without a global rewrite.
+- **Compaction definition.** Define "scrub/compact" formally: produce an
+  equivalent file containing, for each live key, exactly one line in
+  first-appearance order, dropping comments and tombstones, optionally
+  re-emitting a single `#_defaults_#` line. (This is lossy w.r.t. comments — say
+  so.)
+- **NUL caveat.** Note that NSVZ (`\0` delimiter) is not friendly to most text
+  tools; it's meant for programmatic use.
+
+---
+
+## API
+
+### `TSVZed(fileName, …)`
+
+An `OrderedDict` subclass that auto-syncs to `fileName`. Selected options:
+
+| Option | Default | Meaning |
+|---|---|---|
+| `header` | `''` | Header line (string or list) for creation/validation. |
+| `createIfNotExist` | `True` | Create the file if missing. |
+| `verifyHeader` | `True` | Check the file's first line against `header`. |
+| `delimiter` | auto | Delimiter; inferred from the extension if omitted. |
+| `defaults` | `None` | Per-column default values. |
+| `strict` | `False` | Enforce column counts / header; reject malformed rows. |
+| `rewrite_on_load` | `True` | Compact the file once when opened. |
+| `rewrite_on_exit` | `False` | Compact the file when closed. |
+| `rewrite_interval` | `0` | Min seconds between periodic compactions (`0` = never). |
+| `monitor_external_changes` | `True` | Detect / merge other processes' writes. |
+| `encoding` | `'utf8'` | File encoding. |
+
+Writes are queued and flushed by a background thread; call `close()` (or use it
+as a context manager) to flush and stop cleanly. Keys starting with `#` are kept
+in memory only (handy for scratch state) and are never written.
+
+### `TSVZedLite(fileName, …)`
+
+A `MutableMapping` that stores only a key→offset index in RAM and seeks into the
+file for each read. Append-only; single-process; no compression, no background
+thread, no external-change monitoring. Good for very large, mostly-write,
+seek-cheap (SSD) workloads. An external index dict can be supplied to skip the
+initial full scan (turning it into a small key-value store).
+
+### Module functions
+
+`readTabularFile`, `appendTabularFile`, `appendLinesTabularFile`,
+`clearTabularFile`, `scrubTabularFile`, `get_delimiter`, `pretty_format_table`,
+plus the legacy `readTSV` / `appendTSV` / `clearTSV` / `scrubTSV` aliases.
+
+---
+
+## Implementation notes (reference behavior vs. the format spec)
+
+The format above is the target definition. A few places where the current
+`TSVZed` / `TSVZedLite` implementation differs or only partially implements it —
+useful to know, and candidates for a future alignment pass:
+
+- **Column normalization.** Rather than treating rows as variable-length with an
+  infinite empty tail (rule 6), the classes detect a column count (from the
+  header or first row) and, in non-strict mode, pad/trim rows to it; `strict`
+  mode rejects mismatches. Reads therefore return fixed-width rows in practice.
+- **Defaults application.** Empty cells *present* in a row are filled from the
+  active defaults at parse time, but rows are not virtually extended, so "read a
+  field beyond the row length" (rule 16) isn't exposed as an API yet.
+- **Header.** The reference classes still use a *non*-`#` first line as the
+  header (validated via `verifyHeader`), whereas rule 14 defines the header as an
+  ordinary `#` comment. Existing files written by older versions use the non-`#`
+  header.
+- **`#_DEFAULTS_#` alias.** Only the lowercase `#_defaults_#` key is currently
+  recognized by the parser.
+- **No `z` vs. plain distinction, no multi-part.** `get_delimiter` infers from
+  `.tsv`/`.csv`/`.nsv`/`.psv` only, so the `z` extensions aren't recognized for
+  delimiter inference yet (`.csvz`/`.nsvz`/`.psvz` currently fall back to tab;
+  `.tsvz` happens to land on tab correctly). The code does not yet enforce the
+  strict append-only WAL contract differently for the `z` forms, and there is no
+  multi-part read/write support. These are spec items, not implemented.
+
+### Open design questions
+
+Raised by the spec clarification, not yet decided:
+
+1. **Where does the strict contract live?** Should `TSVZed`/`TSVZedLite` key the
+   append-only WAL behavior off the `z` extension (`.tsv` = lenient table,
+   `.tsvz` = this format), or treat all forms the same and let the suffix be
+   purely documentary?
+2. **Append-only vs. the existing rewrite machinery.** A strict `.tsvz` arguably
+   forbids in-place rewrite (`rewrite_on_load` / `mapToFile`) entirely except via
+   explicit compaction — which would simplify the class considerably.
+3. **Compaction for history-retention users.** If "keep the full log" is a goal,
+   compaction must be an explicit, separate action — and perhaps emit a `.tsv`
+   snapshot rather than mutating the `.tsvz` log in place.
+
+## Tests
+
+```bash
+python -m pytest test_TSVZ.py            # regression suite
+python test_TSVZ.py                      # or run directly
+python benchTSVZ.py data.tsv -n 100000   # throughput benchmark (not a test)
+```
+
+## License
+
+GPL-3.0-or-later © Yufei Pan (pan@zopyr.us)