snkv 0.2.0__tar.gz

snkv-0.2.0/PKG-INFO

@@ -0,0 +1,15 @@
+Metadata-Version: 2.4
+Name: snkv
+Version: 0.2.0
+Summary: Python bindings for SNKV - crash-safe embedded key-value store
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/hash-anu/snkv
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: C
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8

snkv-0.2.0/README.md

@@ -0,0 +1,349 @@
+# SNKV Python Bindings
+
+[![Build](https://github.com/hash-anu/snkv/actions/workflows/c-cpp.yml/badge.svg)](https://github.com/hash-anu/snkv/actions/workflows/c-cpp.yml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](https://github.com/hash-anu/snkv/blob/master/LICENSE)
+
+Idiomatic Python 3.8+ bindings for [SNKV](https://github.com/hash-anu/snkv) — a lightweight,
+ACID-compliant embedded key-value store built directly on SQLite's B-Tree engine.
+
+---
+
+## Features
+
+- **Dict-style API** — `db["key"] = value`, `val = db["key"]`, `del db["key"]`, `"key" in db`
+- **Context managers** — `with KVStore(...) as db` and `with db.create_column_family(...) as cf` for guaranteed cleanup
+- **Prefix iterators** — efficient namespace scans with `db.prefix_iterator(b"user:")`
+- **WAL checkpoint control** — PASSIVE / FULL / RESTART / TRUNCATE modes via `db.checkpoint()`
+- **Auto-checkpoint** — set `wal_size_limit=N` to checkpoint automatically after every N WAL frames
+- **Typed exceptions** — `NotFoundError`, `BusyError`, `LockedError`, `ReadOnlyError`, `CorruptError` all subclass `snkv.Error`
+- **No Python dependencies** — pure CPython C extension; only requires a C compiler and `python3-dev`
+- **247 tests** — full pytest suite covering ACID, WAL, crash recovery, concurrency, column families, JSON, and more
+
+---
+
+## Installation
+
+### Linux
+
+```bash
+# System dependencies
+sudo apt-get install -y build-essential python3-dev python3-pip
+
+# Python build dependencies
+pip3 install setuptools wheel pytest
+
+# Build
+cd python
+python3 setup.py build_ext --inplace
+```
+
+#### macOS
+
+```bash
+# Compiler (skip if already installed)
+xcode-select --install
+
+# Python build dependencies
+pip3 install setuptools wheel pytest
+
+# Build
+cd python
+python3 setup.py build_ext --inplace
+```
+
+#### Windows — Native Python (recommended)
+
+1. Install [Python 3.8+](https://python.org/downloads) — check **"Add Python to PATH"**
+2. Install [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) — select **"Desktop development with C++"**
+3. Open **"x64 Native Tools Command Prompt for VS 2022"** from the Start Menu (required for 64-bit Python; "Developer PowerShell for VS" defaults to 32-bit and will fail)
+
+```cmd
+:: Python build dependencies
+pip install setuptools wheel pytest
+
+:: Build
+cd python
+python setup.py build_ext --inplace
+```
+
+#### Windows — MSYS2 MinGW64 shell
+
+Open the **MSYS2 MinGW64** shell (not plain MSYS2, not cmd.exe):
+
+```bash
+# System + Python dependencies (one-time)
+pacman -S --needed mingw-w64-x86_64-python \
+                   mingw-w64-x86_64-python-pip \
+                   mingw-w64-x86_64-python-setuptools \
+                   mingw-w64-x86_64-python-pytest
+
+# Build
+cd python
+python3 setup.py build_ext --inplace
+```
+
+> On all platforms, `setup.py` automatically locates `snkv.h` — no manual
+> header step needed. On Linux/macOS it regenerates it via `make snkv.h`;
+> on Windows it falls back to the pre-built `snkv.h` included in the repo.
+
+---
+
+## Quick Start
+
+```python
+from snkv import KVStore
+
+with KVStore("mydb.db") as db:
+    db["hello"] = "world"
+    print(db["hello"].decode())   # world
+```
+
+---
+
+## API Reference
+
+### Opening a store
+
+```python
+from snkv import KVStore, JOURNAL_WAL, JOURNAL_DELETE, SYNC_NORMAL, SYNC_OFF, SYNC_FULL
+
+with KVStore(
+    "mydb.db",
+    journal_mode=JOURNAL_WAL,   # JOURNAL_WAL (default) or JOURNAL_DELETE
+    sync_level=SYNC_NORMAL,     # SYNC_NORMAL (default), SYNC_OFF, SYNC_FULL
+    cache_size=2000,            # pages (~8 MB default)
+    page_size=4096,             # bytes; new databases only
+    busy_timeout=5000,          # ms to retry on SQLITE_BUSY (default 0)
+    read_only=False,            # open read-only
+    wal_size_limit=100,         # auto-checkpoint every 100 WAL frames (0 = off)
+) as db:
+    ...
+```
+
+### CRUD
+
+```python
+# Write
+db["key"] = b"value"          # bytes or str keys/values are both accepted
+db["key"] = "value"           # str is UTF-8 encoded automatically
+
+# Read
+val = db["key"]               # returns bytes; raises NotFoundError if missing
+val = db.get("key")           # returns bytes or None
+val = db.get("key", b"def")   # with default
+
+# Check existence
+exists = "key" in db
+exists = db.exists(b"key")
+
+# Delete
+del db["key"]
+db.delete(b"key")             # same as del; no error if key absent
+
+# Upsert
+db.put(b"key", b"value")      # identical to db["key"] = value
+```
+
+### Transactions
+
+```python
+db.begin(write=True)
+db["a"] = "1"
+db["b"] = "2"
+db.commit()          # persist
+
+db.begin(write=True)
+db["c"] = "3"
+db.rollback()        # discard — "c" is never written
+```
+
+Auto-commit is the default: each `db["key"] = value` outside an explicit transaction is
+committed immediately.
+
+### Column Families
+
+Logical namespaces within a single database file. Always close `cf` before `db`.
+
+```python
+# Create (first use)
+with db.create_column_family("users") as cf:
+    cf[b"alice"] = b"admin"
+    cf[b"bob"]   = b"viewer"
+
+# Open (subsequent uses)
+with db.open_column_family("users") as cf:
+    print(cf[b"alice"])       # b"admin"
+
+# List all column families
+names = db.list_column_families()   # ["users", ...]
+
+# Drop
+db.drop_column_family("users")
+```
+
+### Iterators
+
+```python
+# Full scan — yields (key, value) tuples in key order
+for key, value in db.iterator():
+    print(key, value)
+
+# Prefix scan
+for key, value in db.prefix_iterator(b"user:"):
+    print(key, value)
+
+# Manual control
+it = db.iterator()
+it.first()
+while not it.eof:
+    print(it.key, it.value)
+    it.next()
+it.close()
+
+# As a context manager
+with db.iterator() as it:
+    for key, value in it:
+        ...
+```
+
+### WAL Checkpoint
+
+```python
+from snkv import CHECKPOINT_PASSIVE, CHECKPOINT_FULL, CHECKPOINT_RESTART, CHECKPOINT_TRUNCATE
+
+# Returns (nLog, nCkpt) — WAL frames total / frames written to DB
+nlog, nckpt = db.checkpoint(CHECKPOINT_PASSIVE)    # copy frames without blocking
+nlog, nckpt = db.checkpoint(CHECKPOINT_FULL)       # wait for writers, flush all
+nlog, nckpt = db.checkpoint(CHECKPOINT_RESTART)    # like FULL, reset write position
+nlog, nckpt = db.checkpoint(CHECKPOINT_TRUNCATE)   # like RESTART, truncate WAL file
+```
+
+Must be called outside an active write transaction. Use `wal_size_limit` to auto-checkpoint
+instead.
+
+### Maintenance
+
+```python
+db.sync()                 # flush OS write buffers (fsync)
+db.vacuum(100)            # reclaim up to 100 unused pages incrementally
+db.integrity_check()      # raises CorruptError if database is corrupt
+stats = db.stats()        # dict: {"puts": N, "gets": N, "deletes": N, "iterations": N}
+```
+
+---
+
+## Error Hierarchy
+
+```
+snkv.Error (base)
+├── snkv.NotFoundError   (also KeyError — raised by db["missing"])
+├── snkv.BusyError       (SQLITE_BUSY — another writer holds the lock)
+├── snkv.LockedError     (SQLITE_LOCKED)
+├── snkv.ReadOnlyError   (write attempted on read-only store)
+└── snkv.CorruptError    (database file is corrupt)
+```
+
+```python
+import snkv
+
+try:
+    val = db["missing_key"]
+except snkv.NotFoundError:
+    val = b"default"
+
+try:
+    db["key"] = b"value"
+except snkv.BusyError:
+    # retry after a delay
+    ...
+```
+
+---
+
+## Running Tests
+
+**Linux / macOS**
+```bash
+cd python
+python3 -m pytest tests/ -v
+```
+
+**Windows — Native Python (x64 Native Tools Command Prompt for VS 2022)**
+```cmd
+cd python
+set PYTHONPATH=.
+python -m pytest tests\ -v
+```
+
+**Windows — MSYS2 MinGW64 shell**
+```bash
+cd python
+PYTHONPATH=. python3 -m pytest tests/ -v
+```
+
+All 247 tests should pass.
+
+---
+
+## Running Examples
+
+**Linux / macOS**
+```bash
+cd python
+PYTHONPATH=. python3 examples/basic.py           # CRUD, binary data, in-memory store
+PYTHONPATH=. python3 examples/transactions.py    # begin/commit/rollback
+PYTHONPATH=. python3 examples/column_families.py # logical namespaces
+PYTHONPATH=. python3 examples/iterators.py       # ordered scan, prefix scan
+PYTHONPATH=. python3 examples/config.py          # journal mode, sync, cache, WAL limit
+PYTHONPATH=. python3 examples/checkpoint.py      # manual + auto WAL checkpoint
+PYTHONPATH=. python3 examples/session_store.py   # real-world session store pattern
+```
+
+**Windows — Native Python (x64 Native Tools Command Prompt for VS 2022)**
+```cmd
+cd python
+set PYTHONPATH=.
+python examples\basic.py
+python examples\transactions.py
+python examples\column_families.py
+python examples\iterators.py
+python examples\config.py
+python examples\checkpoint.py
+python examples\session_store.py
+```
+
+**Windows — MSYS2 MinGW64 shell**
+```bash
+cd python
+PYTHONPATH=. python3 examples/basic.py
+PYTHONPATH=. python3 examples/transactions.py
+# ... same pattern for all examples
+```
+
+---
+
+## Thread Safety
+
+Each thread must use its own `KVStore` instance. WAL mode serialises concurrent writers
+at the SQLite level — a `BusyError` is raised (or retried up to `busy_timeout` ms) when
+two writers collide. Multiple readers always make progress concurrently in WAL mode.
+
+```python
+import threading
+from snkv import KVStore, JOURNAL_WAL
+
+def worker(db_path, worker_id):
+    # Each thread opens its own connection
+    with KVStore(db_path, journal_mode=JOURNAL_WAL, busy_timeout=5000) as db:
+        db[f"key_{worker_id}".encode()] = b"value"
+
+threads = [threading.Thread(target=worker, args=("mydb.db", i)) for i in range(4)]
+for t in threads: t.start()
+for t in threads: t.join()
+```
+
+---
+
+## License
+
+Apache License 2.0 © 2025 Hash Anu

snkv-0.2.0/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "snkv"
+version = "0.2.0"
+requires-python = ">=3.8"
+description = "Python bindings for SNKV - crash-safe embedded key-value store"
+license = {text = "Apache-2.0"}
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: C",
+    "License :: OSI Approved :: Apache Software License",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
+    "Operating System :: Microsoft :: Windows",
+    "Topic :: Database",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+
+[project.urls]
+Homepage = "https://github.com/hash-anu/snkv"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["snkv*"]
+
+[tool.setuptools.package-data]
+snkv = ["py.typed", "_snkv.pyi"]
+
+# ── cibuildwheel ────────────────────────────────────────────────────────────
+[tool.cibuildwheel]
+# Build CPython 3.8 – 3.13; skip 32-bit and musl builds
+build  = "cp38-* cp39-* cp310-* cp311-* cp312-* cp313-*"
+skip   = "*-win32 *-manylinux_i686 *-musllinux*"
+
+[tool.cibuildwheel.linux]
+archs = ["x86_64"]
+
+[tool.cibuildwheel.macos]
+archs = ["x86_64", "arm64"]
+
+[tool.cibuildwheel.windows]
+archs = ["AMD64"]

snkv-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

snkv-0.2.0/setup.py

@@ -0,0 +1,207 @@
+"""
+Build script for the snkv Python extension.
+
+Before compiling the C extension this script:
+  1. Runs `make snkv.h` in the repo root to regenerate the amalgamated header.
+  2. Copies the freshly built snkv.h into this directory so the compiler
+     finds it alongside snkv_module.c.
+
+Run from the repo root:
+    pip install -e python/
+or from this directory:
+    python setup.py build_ext --inplace
+"""
+
+import os
+import shutil
+import subprocess
+import sys
+import urllib.request
+from setuptools import setup, Extension
+from setuptools.command.build_ext import build_ext as _build_ext
+
+# Absolute path of the repository root (one level above this file).
+REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+HERE      = os.path.abspath(os.path.dirname(__file__))
+
+
+class BuildExtWithHeader(_build_ext):
+    """Custom build_ext that regenerates snkv.h before compiling."""
+
+    def run(self) -> None:
+        self._regenerate_snkv_header()
+        # Ensure the package directory exists so the inplace copy succeeds
+        # on fresh clones where snkv/ may not yet contain any built artifact.
+        os.makedirs(os.path.join(HERE, "snkv"), exist_ok=True)
+        super().run()
+
+    def build_extension(self, ext: Extension) -> None:
+        self._regenerate_snkv_header()
+        super().build_extension(ext)
+
+    def _regenerate_snkv_header(self) -> None:
+        target = os.path.join(HERE, "snkv.h")
+
+        # Only regenerate once per build run.
+        if getattr(self, "_snkv_h_ready", False):
+            return
+
+        # Try `make snkv.h` first (works on Linux, macOS, and Windows MSYS2).
+        # If make is not on PATH, fall back to downloading from GitHub — same
+        # on all platforms.
+        try:
+            result = subprocess.run(
+                ["make", "snkv.h"],
+                cwd=REPO_ROOT,
+                check=True,
+                capture_output=True,
+                text=True,
+            )
+            if result.stdout:
+                print(result.stdout, end="")
+
+            # Copy the generated header from repo root into python/
+            src = os.path.join(REPO_ROOT, "snkv.h")
+            if not os.path.exists(src):
+                raise FileNotFoundError(
+                    f"`make snkv.h` succeeded but {src} was not found."
+                )
+            shutil.copy2(src, target)
+            print(f"-- snkv: copied {src} -> {target}")
+
+        except FileNotFoundError:
+            # make not on PATH — download from GitHub (same on all platforms)
+            if os.path.exists(target):
+                print(f"-- snkv: `make` not found; using existing {target}")
+            else:
+                print("-- snkv: `make` not found; downloading snkv.h from GitHub...")
+                self._download_snkv_header(target)
+
+        except subprocess.CalledProcessError as e:
+            raise RuntimeError(
+                f"`make snkv.h` failed (exit {e.returncode}):\n{e.stderr}"
+            ) from e
+
+        self._snkv_h_ready = True
+
+    def _download_snkv_header(self, target: str) -> None:
+        import json
+        import tempfile
+        import zipfile
+        from urllib.error import HTTPError
+
+        headers = {"Accept": "application/vnd.github+json"}
+
+        # 1a. Try GitHub Releases API first (works once releases are published).
+        tag = None
+        release_zip_ok = False
+        api_url = "https://api.github.com/repos/hash-anu/snkv/releases/latest"
+        try:
+            req = urllib.request.Request(api_url, headers=headers)
+            with urllib.request.urlopen(req) as resp:
+                tag = json.loads(resp.read())["tag_name"]   # e.g. "v0.1.3"
+        except HTTPError as exc:
+            if exc.code == 404:
+                pass  # No releases published yet — fall through to tags API
+            else:
+                raise RuntimeError(
+                    f"GitHub API error ({api_url}): {exc}\n"
+                    "Either install make (Linux/macOS) or ensure internet access."
+                ) from exc
+
+        # 1b. If no releases, fall back to the latest git tag.
+        if tag is None:
+            tags_url = "https://api.github.com/repos/hash-anu/snkv/tags"
+            try:
+                req = urllib.request.Request(tags_url, headers=headers)
+                with urllib.request.urlopen(req) as resp:
+                    tags = json.loads(resp.read())
+                if not tags:
+                    raise RuntimeError("No tags found in the snkv repository.")
+                tag = tags[0]["name"]   # most-recent tag first
+            except Exception as exc:
+                raise RuntimeError(
+                    f"Failed to fetch tags from GitHub ({tags_url}):\n{exc}\n"
+                    "Either install make (Linux/macOS) or ensure internet access."
+                ) from exc
+
+        version = tag.lstrip("v")   # "0.1.3"
+
+        # 2a. Try the release asset zip (requires a published GitHub Release).
+        zip_url = (
+            f"https://github.com/hash-anu/snkv/releases/download/"
+            f"{tag}/snkv-{version}.zip"
+        )
+        print(f"-- snkv: downloading {zip_url} ...")
+        with tempfile.TemporaryDirectory() as tmp:
+            zip_path = os.path.join(tmp, f"snkv-{version}.zip")
+            try:
+                urllib.request.urlretrieve(zip_url, zip_path)
+                release_zip_ok = True
+            except HTTPError as exc:
+                if exc.code == 404:
+                    release_zip_ok = False  # asset not uploaded yet
+                else:
+                    raise RuntimeError(
+                        f"Failed to download release zip ({zip_url}): {exc}\n"
+                        "Either install make (Linux/macOS) or ensure internet access."
+                    ) from exc
+
+            if release_zip_ok:
+                # Extract snkv.h from release/snkv-{version}/include/
+                inner_path = f"snkv-{version}/include/snkv.h"
+                with zipfile.ZipFile(zip_path) as zf:
+                    if inner_path not in zf.namelist():
+                        raise RuntimeError(
+                            f"Expected '{inner_path}' inside {zip_url} but it was not found.\n"
+                            f"Available entries: {zf.namelist()[:10]}"
+                        )
+                    with zf.open(inner_path) as src_f:
+                        with open(target, "wb") as dst_f:
+                            dst_f.write(src_f.read())
+                print(f"-- snkv: extracted snkv.h {tag} -> {target}")
+                return
+
+        # 2b. Release asset not available — download raw snkv.h from the tag.
+        raw_url = (
+            f"https://raw.githubusercontent.com/hash-anu/snkv/{tag}/snkv.h"
+        )
+        print(f"-- snkv: release asset not found; downloading raw {raw_url} ...")
+        try:
+            urllib.request.urlretrieve(raw_url, target)
+            print(f"-- snkv: downloaded snkv.h {tag} from GitHub -> {target}")
+        except Exception as exc:
+            raise RuntimeError(
+                f"Failed to download snkv.h from GitHub ({raw_url}):\n{exc}\n"
+                "Either install make (Linux/macOS) or ensure internet access."
+            ) from exc
+
+
+# ---------------------------------------------------------------------------
+# Platform-specific linker flags
+# ---------------------------------------------------------------------------
+
+extra_link_args: list = []
+
+if sys.platform == "darwin":
+    extra_link_args = ["-lpthread", "-lm"]
+
+# ---------------------------------------------------------------------------
+# C extension
+# ---------------------------------------------------------------------------
+
+ext = Extension(
+    "snkv._snkv",
+    sources=["snkv_module.c"],
+    include_dirs=[HERE],
+    extra_link_args=extra_link_args,
+)
+
+# ---------------------------------------------------------------------------
+# setup()
+# ---------------------------------------------------------------------------
+
+setup(
+    ext_modules=[ext],
+    cmdclass={"build_ext": BuildExtWithHeader},
+)