kvgit 0.3.6__tar.gz → 0.3.8__tar.gz

{kvgit-0.3.6 → kvgit-0.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kvgit
-Version: 0.3.6
+Version: 0.3.8
 Summary: Versioned key-value store with git-like commit, branch, and merge semantics.
 Author: ashenfad
 License: MIT
@@ -94,6 +94,89 @@ main.tag("v1")
 print(main.peek("score", tag="v1"))  # 0
 ```
 
+## Merging
+
+`Staged.merge()` merges another head -- usually another branch's HEAD --
+into the current branch: lowest common ancestor, three-way resolve, and
+a two-parent merge commit guarded on your own head:
+
+```python
+dev["score"] = 500
+dev.commit()
+
+result = main.merge(dev.current_commit)  # True when merged
+print(main["score"])  # 500 (fast-forward: main hadn't diverged)
+```
+
+Overlapping changes need a merge function per key, or a `default_merge`
+fallback. `text_merge()` resolves line-oriented text with git-style
+`<<<<<<<` markers (its `ours_label` / `theirs_label` arguments name
+them); anything it cannot mark -- binary, non-UTF-8, oversized -- raises
+`CantMark`, filed as an ordinary conflict:
+
+```python
+from kvgit import text_merge
+
+main["notes"] = "alpha\nbeta\n"
+main.commit()
+
+edits = main.create_branch("edits")
+edits["notes"] = "alpha\nBETA\n"
+edits.commit()
+
+main["notes"] = "ALPHA\nbeta\n"
+main.commit()
+
+main.merge(edits.current_commit, default_merge=text_merge())
+print(main["notes"])  # "ALPHA\nBETA\n" -- both edits kept
+```
+
+`kvgit.merges.text` is the same merge one level down, over raw bytes,
+for `VersionedKV`. A `Staged` decodes each side before calling a merge
+function, so register `text_merge()` there unless the key's values are
+already `bytes`.
+
+Registrations resolve most-specific-first: a key takes its exact-key
+registration, else the longest registered prefix it starts with, else
+`default_merge`. Prefixes cover keys whose names are not known when the
+policy is set:
+
+```python
+from kvgit import MergeChoice, text_merge
+
+main.set_merge_prefix("runs/", MergeChoice.OURS)   # this branch owns runs/
+main.set_merge_fn("runs/index", text_merge())      # except this one key
+```
+
+A registration holds either a merge function or a `MergeChoice`, and the
+two reach differently:
+
+* A **merge function** is consulted only where both sides changed a key.
+  A change only one side made is applied as it always was.
+* A **`MergeChoice`** is a standing policy: it hands that side every key
+  either side changed under it. So `OURS` also drops a key the other
+  side added and keeps one the other side removed -- what a branch that
+  owns a whole namespace needs. Nothing is read or decoded for those
+  keys.
+
+`kvgit.merges.ours` and `kvgit.merges.theirs` are the merge-function
+form, for keys where one side wins only when both sides collide. They
+keep that side's stored value as it stands rather than rewriting it, so
+the merge writes no new blob -- and if the chosen side removed the key,
+the merge removes it.
+
+`commit()` and `merge()` take `merge_fns=` / `merge_prefixes=` /
+`default_merge=` for one call, layered over what is registered.
+
+Keys both sides changed to the same bytes merge cleanly with no merge
+function, even though each side wrote its own copy.
+
+A `post_check(key, merged_bytes)` predicate runs over every
+merge-produced value; returning `False` files that key as conflicted.
+`on_conflict="abandon"` leaves the branch untouched instead of raising.
+Merging refuses with `ValueError` when the staging buffer holds
+uncommitted changes -- commit or reset first.
+
 ## Chunked codecs (numpy / pandas)
 
 Large numpy arrays and pandas DataFrames -- and any sliced views of them -- can be stored once and shared across keys, commits, and branches:

{kvgit-0.3.6 → kvgit-0.3.8}/README.md

@@ -45,6 +45,89 @@ main.tag("v1")
 print(main.peek("score", tag="v1"))  # 0
 ```
 
+## Merging
+
+`Staged.merge()` merges another head -- usually another branch's HEAD --
+into the current branch: lowest common ancestor, three-way resolve, and
+a two-parent merge commit guarded on your own head:
+
+```python
+dev["score"] = 500
+dev.commit()
+
+result = main.merge(dev.current_commit)  # True when merged
+print(main["score"])  # 500 (fast-forward: main hadn't diverged)
+```
+
+Overlapping changes need a merge function per key, or a `default_merge`
+fallback. `text_merge()` resolves line-oriented text with git-style
+`<<<<<<<` markers (its `ours_label` / `theirs_label` arguments name
+them); anything it cannot mark -- binary, non-UTF-8, oversized -- raises
+`CantMark`, filed as an ordinary conflict:
+
+```python
+from kvgit import text_merge
+
+main["notes"] = "alpha\nbeta\n"
+main.commit()
+
+edits = main.create_branch("edits")
+edits["notes"] = "alpha\nBETA\n"
+edits.commit()
+
+main["notes"] = "ALPHA\nbeta\n"
+main.commit()
+
+main.merge(edits.current_commit, default_merge=text_merge())
+print(main["notes"])  # "ALPHA\nBETA\n" -- both edits kept
+```
+
+`kvgit.merges.text` is the same merge one level down, over raw bytes,
+for `VersionedKV`. A `Staged` decodes each side before calling a merge
+function, so register `text_merge()` there unless the key's values are
+already `bytes`.
+
+Registrations resolve most-specific-first: a key takes its exact-key
+registration, else the longest registered prefix it starts with, else
+`default_merge`. Prefixes cover keys whose names are not known when the
+policy is set:
+
+```python
+from kvgit import MergeChoice, text_merge
+
+main.set_merge_prefix("runs/", MergeChoice.OURS)   # this branch owns runs/
+main.set_merge_fn("runs/index", text_merge())      # except this one key
+```
+
+A registration holds either a merge function or a `MergeChoice`, and the
+two reach differently:
+
+* A **merge function** is consulted only where both sides changed a key.
+  A change only one side made is applied as it always was.
+* A **`MergeChoice`** is a standing policy: it hands that side every key
+  either side changed under it. So `OURS` also drops a key the other
+  side added and keeps one the other side removed -- what a branch that
+  owns a whole namespace needs. Nothing is read or decoded for those
+  keys.
+
+`kvgit.merges.ours` and `kvgit.merges.theirs` are the merge-function
+form, for keys where one side wins only when both sides collide. They
+keep that side's stored value as it stands rather than rewriting it, so
+the merge writes no new blob -- and if the chosen side removed the key,
+the merge removes it.
+
+`commit()` and `merge()` take `merge_fns=` / `merge_prefixes=` /
+`default_merge=` for one call, layered over what is registered.
+
+Keys both sides changed to the same bytes merge cleanly with no merge
+function, even though each side wrote its own copy.
+
+A `post_check(key, merged_bytes)` predicate runs over every
+merge-produced value; returning `False` files that key as conflicted.
+`on_conflict="abandon"` leaves the branch untouched instead of raising.
+Merging refuses with `ValueError` when the staging buffer holds
+uncommitted changes -- commit or reset first.
+
 ## Chunked codecs (numpy / pandas)
 
 Large numpy arrays and pandas DataFrames -- and any sliced views of them -- can be stored once and shared across keys, commits, and branches:

{kvgit-0.3.6 → kvgit-0.3.8}/kvgit/__init__.py

@@ -1,19 +1,28 @@
 """kvgit: Versioned key-value store."""
 
-from .content_types import MergeFn, counter, last_writer_wins
-from .errors import ConcurrencyError, MergeConflict
-from .merges import CantMark, TextMergeFn, make_text_merge, text
+from .content_types import MergeFn, counter, last_writer_wins, text_merge
+from .errors import ConcurrencyError, GcBusy, MergeConflict
+from .merges import CantMark, TextMergeFn, make_text_merge, ours, text, theirs
 from .namespaced import Namespaced
 from .staged import Staged
 from .store import delete_branches, delete_tags, store
 from .versioned.kv import VersionedKV
-from .versioned.protocol import MergeResult, TagInfo, Versioned
+from .versioned.protocol import (
+    MergeChoice,
+    MergePolicy,
+    MergeResult,
+    TagInfo,
+    Versioned,
+)
 
 __all__ = [
     "CantMark",
     "ConcurrencyError",
+    "GcBusy",
+    "MergeChoice",
     "MergeConflict",
     "MergeFn",
+    "MergePolicy",
     "MergeResult",
     "Namespaced",
     "Staged",
@@ -26,6 +35,9 @@ __all__ = [
     "delete_tags",
     "last_writer_wins",
     "make_text_merge",
+    "ours",
     "store",
     "text",
+    "text_merge",
+    "theirs",
 ]

kvgit-0.3.8/kvgit/content_types.py

@@ -0,0 +1,69 @@
+"""Merge functions for typed values."""
+
+from collections.abc import Callable
+from typing import Any
+
+from .merges import CantMark, make_text_merge
+
+MergeFn = Callable[[Any | None, Any, Any], Any]
+"""Merge function: (old_value | None, our_value, their_value) -> merged_value.
+
+Any argument can be None (key absent or removed on that side).
+
+Returning a :class:`~kvgit.versioned.protocol.MergeChoice` instead of a
+value keeps that side's committed value as it stands: no value is
+encoded and the merge writes no new blob for the key.
+"""
+
+
+def counter() -> MergeFn:
+    """Counter merge: ours + theirs - old."""
+
+    def merge(old: int | None, ours: int, theirs: int) -> int:
+        base = old if old is not None else 0
+        return ours + theirs - base
+
+    return merge
+
+
+def last_writer_wins() -> MergeFn:
+    """Last-writer-wins: always returns theirs."""
+    return lambda old, ours, theirs: theirs
+
+
+def text_merge(*, ours_label: str = "ours", theirs_label: str = "theirs") -> MergeFn:
+    """Marker merge for ``str`` (or ``bytes``) values, for use with ``Staged``.
+
+    The value-level counterpart to :func:`kvgit.merges.text`. ``Staged``
+    decodes both sides before calling a merge function, so a key holding
+    ``str`` reaches a bytes-level function as ``str`` and fails there;
+    this one encodes ``str`` sides as UTF-8, marker-merges, and decodes
+    the result back to ``str`` when any side was ``str``. A key whose
+    values are ``bytes`` merges as bytes and comes back as ``bytes``.
+
+    Disjoint line changes merge cleanly and overlapping ones come back
+    with git-style ``<<<<<<<`` markers (labelled by ``ours_label`` /
+    ``theirs_label``) rather than raising. A value that is neither
+    ``str`` nor ``bytes``, and anything the marker merge cannot handle —
+    undecodable bytes, NUL bytes, inputs over the size cap — raises
+    :class:`~kvgit.merges.CantMark`, which the merge machinery files as
+    an ordinary conflict.
+    """
+    merge_bytes = make_text_merge(ours_label=ours_label, theirs_label=theirs_label)
+
+    def encode(value: Any) -> bytes | None:
+        if value is None or isinstance(value, bytes):
+            return value
+        if isinstance(value, str):
+            return value.encode("utf-8")
+        raise CantMark(f"not text: {type(value).__name__}")
+
+    def merge(old: Any, ours: Any, theirs: Any) -> str | bytes:
+        # One side holding str is enough to make the merged value str:
+        # the sides are the same key's value at different commits, so a
+        # bytes side is the same text in another spelling.
+        as_str = any(isinstance(v, str) for v in (old, ours, theirs))
+        merged = merge_bytes(encode(old), encode(ours), encode(theirs))
+        return merged.decode("utf-8") if as_str else merged
+
+    return merge

{kvgit-0.3.6 → kvgit-0.3.8}/kvgit/errors.py

@@ -26,3 +26,13 @@ class MergeConflict(Exception):
         self.merge_errors = merge_errors or {}
         keys_str = ", ".join(sorted(conflicting_keys))
         super().__init__(f"Merge conflict on keys: {keys_str}")
+
+
+class GcBusy(Exception):
+    """Raised when a deep clean cannot take the store's GC lease.
+
+    Another process or thread holds an unexpired lease, so a namespace
+    scan started now would run beside a sweep that is already deleting.
+    Retry later; the lease carries an expiry, so a holder that dies
+    without releasing it blocks nothing past that point.
+    """

{kvgit-0.3.6 → kvgit-0.3.8}/kvgit/kv/composite.py

@@ -20,15 +20,45 @@ def _is_bug(exc: BaseException) -> bool:
     return isinstance(exc, _BUG_EXCEPTIONS)
 
 
+MUTABLE_PREFIX = "__"
+"""Key prefix marking a value that may change under a fixed key.
+
+Every mutable key kvgit writes starts with it — branch heads, their
+prev-HEAD backups, the storage version stamp, the GC lease — and so does
+commit metadata, which is immutable but small enough that reading it
+from the authoritative tier costs little. Everything else is derived
+from its own content (``kvgit:keyset:``, ``kvgit:chunk:``,
+``<commit>:<key>``): the same key always holds the same bytes, which is
+what makes it cacheable.
+"""
+
+
+def _is_mutable(key: str) -> bool:
+    return isinstance(key, str) and key.startswith(MUTABLE_PREFIX)
+
+
 class Composite(KVStore):
     """N-tier cache composing any number of KV stores.
 
-    On get: check L1, L2, ..., Ln in order. On hit at tier i,
-    populate L1..L(i-1) and return.
+    **Cache tiers serve only immutable, content-derived keys.** A key
+    starting with ``__`` names a value that changes under a fixed key —
+    a branch head, its recovery backup, the storage version stamp, the
+    GC lease — so ``get``, ``get_many`` and ``__contains__`` read those
+    from Ln alone and never populate a cache with them. Cached, they
+    would let a process keep serving a branch head that another process
+    has already moved: the handle would take a ``ConcurrencyError`` on
+    commit, call ``refresh()``, and read the same stale head back out of
+    L1. Everything else is keyed by its own content
+    (``kvgit:keyset:``, ``kvgit:chunk:``, ``<commit>:<key>``), so a hit
+    at any tier is the right answer forever.
+
+    On get: for a ``__`` key, read Ln. Otherwise check L1, L2, ..., Ln
+    in order; on hit at tier i, populate L1..L(i-1) and return.
 
     On set: write to all tiers (most durable first).
 
-    On cas: delegate to Ln (authoritative), update caches on success.
+    On cas: delegate to Ln (authoritative), update caches on success —
+    except for ``__`` keys, which no cache tier serves.
 
     Tier failures (``OSError``, network errors, etc.) are logged at
     WARNING and the next tier is tried; programming-error exceptions
@@ -54,8 +84,20 @@ class Composite(KVStore):
                     raise
                 logger.warning("Composite cache populate failed at tier %d: %s", j, e)
 
+    def _first_read_tier(self, key: str) -> int:
+        """Index of the highest tier allowed to answer for ``key``.
+
+        A ``__``-prefixed key names a value that can change under a
+        fixed key, so only the authoritative tier may answer it;
+        everything else is content-derived and may be served by any
+        tier.
+        """
+        return len(self._stores) - 1 if _is_mutable(key) else 0
+
     def get(self, key: str) -> bytes | None:
-        for i, store in enumerate(self._stores):
+        start = self._first_read_tier(key)
+        for i in range(start, len(self._stores)):
+            store = self._stores[i]
             try:
                 value = store.get(key)
             except Exception as e:
@@ -64,17 +106,21 @@ class Composite(KVStore):
                 logger.warning("Composite get failed at tier %d for %r: %s", i, key, e)
                 continue
             if value is not None:
-                if i > 0:
+                # Nothing to populate when the read started at the
+                # authoritative tier, which is where a mutable key's
+                # read starts and ends.
+                if i > start:
                     self._populate_caches(i, {key: value})
                 return value
         return None
 
-    def get_many(self, *args) -> Mapping[str, bytes]:
+    def _get_many_from(self, remaining: set[str], start: int) -> dict[str, bytes]:
+        """Bulk read ``remaining``, consulting tiers ``start``..Ln."""
         result: dict[str, bytes] = {}
-        remaining = set(self._normalize_keys(args))
-        for i, store in enumerate(self._stores):
+        for i in range(start, len(self._stores)):
             if not remaining:
                 break
+            store = self._stores[i]
             try:
                 # Delegate to the tier's bulk get — backends with high
                 # per-call latency (Disk, IndexedDB) collapse N round-trips
@@ -86,16 +132,28 @@ class Composite(KVStore):
                     raise
                 logger.warning("Composite get_many failed at tier %d: %s", i, e)
                 continue
-            if tier_values and i > 0:
+            if tier_values and i > start:
                 self._populate_caches(i, tier_values)
             result.update(tier_values)
-            remaining -= tier_values.keys()
+            remaining = remaining - tier_values.keys()
+        return result
+
+    def get_many(self, *args) -> Mapping[str, bytes]:
+        keys = set(self._normalize_keys(args))
+        mutable = {key for key in keys if _is_mutable(key)}
+        if not mutable:
+            return self._get_many_from(keys, 0)
+        # Two passes rather than one: mutable keys must come from the
+        # authoritative tier and must not be written into a cache, so
+        # they cannot share a batch with content-derived keys.
+        result = self._get_many_from(mutable, len(self._stores) - 1)
+        result.update(self._get_many_from(keys - mutable, 0))
         return result
 
     def __contains__(self, key: str) -> bool:
-        for i, store in enumerate(self._stores):
+        for i in range(self._first_read_tier(key), len(self._stores)):
             try:
-                if key in store:
+                if key in self._stores[i]:
                     return True
             except Exception as e:
                 if _is_bug(e):
@@ -175,7 +233,10 @@ class Composite(KVStore):
 
     def cas(self, key: str, value: bytes, expected: bytes | None) -> bool:
         success = self._stores[-1].cas(key, value, expected)
-        if success:
+        # A ``__``-prefixed key is read from the authoritative tier
+        # only, so writing it into a cache stores bytes nothing will
+        # ever read, and a CAS is almost always against such a key.
+        if success and not _is_mutable(key):
             for i, store in enumerate(self._stores[:-1]):
                 try:
                     store.set(key, value)

{kvgit-0.3.6 → kvgit-0.3.8}/kvgit/merges.py

@@ -9,6 +9,9 @@ Anything that cannot be marked — undecodable bytes, NUL bytes, inputs
 over the size cap — raises :class:`CantMark`, which the merge machinery
 files as an ordinary conflict. Callers that need to tell "no encoding"
 from "too big" catch it themselves; callers that don't let it flow.
+
+Also here: :func:`ours` and :func:`theirs`, the pick-a-side merges, for
+keys where one branch simply owns the value.
 """
 
 from __future__ import annotations
@@ -16,7 +19,7 @@ from __future__ import annotations
 import difflib
 from collections.abc import Callable
 
-from .versioned.protocol import BytesMergeFn
+from .versioned.protocol import BytesMergeFn, MergeChoice
 
 #: Inputs whose combined size exceeds this are refused with
 #: :class:`CantMark` rather than merged. Line-diffing is quadratic-ish
@@ -245,3 +248,24 @@ def text(old: bytes | None, ours: bytes | None, theirs: bytes | None) -> bytes:
 
 TextMergeFn = Callable[[bytes | None, bytes | None, bytes | None], bytes]
 """Type of what :func:`make_text_merge` builds (== ``BytesMergeFn``)."""
+
+
+def ours(old: bytes | None, our: bytes | None, their: bytes | None) -> MergeChoice:
+    """Take our side: the merged value is our committed value.
+
+    Keeps our existing blob pointer rather than rewriting the value, so
+    the merge writes nothing for this key. If we removed the key, the
+    merge removes it. Usable as a per-key, per-prefix, or default merge.
+    """
+    return MergeChoice.OURS
+
+
+def theirs(old: bytes | None, our: bytes | None, their: bytes | None) -> MergeChoice:
+    """Take their side: the merged value is their committed value.
+
+    Keeps their existing blob pointer rather than rewriting the value,
+    so the merge writes nothing for this key. If they removed the key,
+    the merge removes it. Usable as a per-key, per-prefix, or default
+    merge.
+    """
+    return MergeChoice.THEIRS