hotdata-materialized 0.1.0__tar.gz

hotdata_materialized-0.1.0/PKG-INFO

@@ -0,0 +1,227 @@
+Metadata-Version: 2.4
+Name: hotdata-materialized
+Version: 0.1.0
+Summary: Materialize expensive Django query results into Hotdata: miss runs locally, hit returns a queryable frame.
+Author: Hotdata
+License: MIT
+Project-URL: Homepage, https://hotdata.dev
+Keywords: django,hotdata,cache,materialized,analytics,parquet
+Classifier: Framework :: Django
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: Django>=3.2
+Requires-Dist: hotdata[arrow]>=0.8
+Requires-Dist: pyarrow>=14.0.1
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: duckdb>=1.0; extra == "test"
+Requires-Dist: flake8>=4.0; extra == "test"
+Requires-Dist: mypy>=1.5; extra == "test"
+Requires-Dist: build>=1.0; extra == "test"
+Provides-Extra: demo
+Requires-Dist: psycopg[binary]>=3.1; extra == "demo"
+
+# hotdata-materialized
+
+Materialize expensive Django query results into [Hotdata](https://hotdata.dev).
+On a cache **miss** your code runs in your environment as usual and the result
+is captured into Hotdata as parquet — in the background, after your response
+has already returned. On a **hit** your database is never touched: the data
+comes back as a `pyarrow.Table` you can turn into a dataframe, or query with
+SQL server-side.
+
+Think materialized views, not Redis: entries are snapshots of expensive
+analytical results, with a TTL lifecycle, each living in its own Hotdata
+managed database. The host application needs **no migrations and no Redis** —
+entry metadata lives in a small Hotdata managed database (the registry),
+created on first use.
+
+Measured against TPC-H on a Neon Postgres (see [demo/](demo/)):
+
+| | Direct on Postgres | Miss (perceived) | Hit |
+|---|---|---|---|
+| Q1 pricing summary (full scan, 4 rows) | ~1.4 s | ~1.4 s | **~80 ms** |
+| Revenue by part (50,000 rows) | ~2.1 s | ~2.0 s | **~260 ms** |
+
+A miss costs the same as not caching (the persist runs write-behind); a hit
+is 8–18× faster with zero load on your database.
+
+## Install
+
+```bash
+pip install hotdata-materialized
+```
+
+```python
+# settings.py
+HOTDATA_MATERIALIZED = {
+    "API_KEY": env("HOTDATA_API_KEY"),      # hd_... key from app.hotdata.dev
+    "WORKSPACE_ID": "work...",
+    # optional:
+    # "API_URL": "https://api.hotdata.dev",
+    # "REGISTRY_DATABASE": "materialized_registry",
+    # "INLINE_THRESHOLD_BYTES": 2 * 1024 * 1024,
+    # "BACKGROUND": True,                   # write-behind persists (default)
+}
+```
+
+## Using it in a Django view
+
+Wrap the expensive part in `@materialize`; call it from your view:
+
+```python
+# reports.py
+from django.db.models import Count, Sum
+from hotdata_materialized import materialize
+
+from .models import Order
+
+
+@materialize(ttl=3600)
+def revenue_by_region():
+    return (
+        Order.objects
+        .filter(status="complete")
+        .values("region")
+        .annotate(orders=Count("id"), revenue=Sum("total"))
+        .order_by("-revenue")
+    )
+```
+
+```python
+# views.py
+from django.http import JsonResponse
+from .reports import revenue_by_region
+
+
+def revenue_view(request):
+    frame = revenue_by_region()
+    return JsonResponse({"rows": frame.to_pylist(), "cached": frame.cached})
+```
+
+On a **hit** the function never runs and your database is never touched. On a
+**miss** the function runs exactly as it would uncached, the caller gets the
+result immediately, and the persist happens in a background thread. Either
+way you get a `MaterializedFrame`:
+
+- `frame.arrow()` — the data as a `pyarrow.Table` (`frame.df()` for pandas,
+  `frame.to_pylist()` for dicts, `len(frame)` for the row count)
+- `frame.sql("SELECT region, revenue FROM this ORDER BY revenue DESC LIMIT 5")`
+  — SQL runs server-side against the cached entry; `this` names the data.
+  Needs a persisted entry: available on hits (on a fresh miss the
+  write-behind persist has to land first; `background=False` avoids that)
+- `frame.cached` — which path served it; `frame.entry` — the registry record
+
+The wrapped function can return an iterable of dicts (a `.values()` queryset),
+a `pyarrow.Table`, or a pandas DataFrame. Decorator knobs: `ttl=` (seconds,
+`None` = never expires), `version=` (bump to bust the cache on code changes),
+`key=` (human-readable label), `key_fn=` (stable identity for arguments that
+aren't JSON-serializable), `background=False` (block until persisted).
+Fail-open by design: if Hotdata is unreachable, the function runs and its
+result is served uncached — the cache degrades to "no cache," never to
+"no page."
+
+## Search
+
+Declare a search index on the decorator and the entry gets it when it
+persists (builds run server-side; a freshly missed entry may error on search
+for a few seconds until its index is ready):
+
+```python
+from hotdata_materialized import BM25, Vector, materialize
+
+
+@materialize(ttl=3600, index=BM25("notes"))
+def incidents():
+    return Incident.objects.values("id", "notes", "severity")
+
+
+incidents().search("checkout errors outage", column="notes", limit=5)
+```
+
+Vector (semantic) search embeds your text server-side via a workspace
+embedding provider — pass the indexed source column and a natural-language
+query:
+
+```python
+@materialize(ttl=3600, index=Vector("notes", provider="emb_...", metric="cosine"))
+def incidents_semantic():
+    return Incident.objects.values("id", "notes")
+
+
+incidents_semantic().vector_search("the site was down", column="notes", limit=5)
+```
+
+Both return the matching rows as a `pyarrow.Table` with a relevance column
+(`score` / `distance`). One platform rule to know: an embedding-backed vector
+index cannot share an entry with other indexes — the decorator rejects that
+combination up front.
+
+## Evicting entries
+
+The decorator composes public primitives (`fingerprint_call` /
+`fingerprint_queryset`, `Registry`, `EntryStore`) that you can use directly —
+for example to drop an entry explicitly:
+
+```python
+from hotdata_materialized import fingerprint_call
+from hotdata_materialized.decorator import get_runtime
+
+registry, store = get_runtime()
+store.evict(fingerprint_call(revenue_by_region))
+```
+
+Expired entries stop being served immediately (the `is_expired` check) and
+their databases carry a server-side expiry backstop; a sweep command that
+deletes them proactively is on the roadmap below.
+
+## How it works
+
+One managed database (the **registry**) holds one row per entry: fingerprint,
+status, TTL, the entry's database id, and — for small results — the data
+itself as an inline Arrow payload, so a chart-sized hit is served in a single
+API round trip. Larger results live in a per-entry managed database and are
+fetched as an Arrow IPC stream via a result id minted at persist time (no
+re-query on hits). Failures are loud and fail toward "no cache": a failed
+persist leaves no registry row, and the next request simply misses again.
+
+See [DESIGN.md](DESIGN.md) for the architecture and the accepted
+trade-offs.
+
+## Roadmap
+
+- [x] Content-addressed fingerprinting (querysets and function calls)
+- [x] Remote registry — no migrations or local state in the host app
+- [x] Entry store with write-behind persists
+- [x] Arrow-native reads (inline payloads, persisted-result fetch)
+- [x] `@materialize` decorator and `MaterializedFrame`
+      (`.arrow()/.df()/.to_pylist()/.sql()`)
+- [x] TPC-H demo and benchmark
+- [x] CI: pytest, mypy, flake8
+- [ ] Stale-while-revalidate refresh (rebuild protocol)
+- [ ] Sweep command for expired entries
+- [ ] Chainable queryset facade on the frame (`.filter()/.order_by()`)
+- [x] Vector/BM25 index declarations, `frame.search()` and
+      `frame.vector_search()`
+- [ ] Async view support (`await frame.aarrow()`)
+- [ ] PyPI release
+
+## Demo
+
+[demo/](demo/) is a small Django project that benchmarks TPC-H queries on a
+Neon Postgres directly vs. through the cache:
+
+```bash
+cd demo
+python manage.py compare                   # Q1: tiny result, inline hit
+python manage.py compare --scenario parts  # 50k rows: remote Arrow hit
+```
+
+## Development
+
+```bash
+uv venv && uv pip install -e '.[test]'
+uv run pytest
+```

hotdata_materialized-0.1.0/README.md

@@ -0,0 +1,202 @@
+# hotdata-materialized
+
+Materialize expensive Django query results into [Hotdata](https://hotdata.dev).
+On a cache **miss** your code runs in your environment as usual and the result
+is captured into Hotdata as parquet — in the background, after your response
+has already returned. On a **hit** your database is never touched: the data
+comes back as a `pyarrow.Table` you can turn into a dataframe, or query with
+SQL server-side.
+
+Think materialized views, not Redis: entries are snapshots of expensive
+analytical results, with a TTL lifecycle, each living in its own Hotdata
+managed database. The host application needs **no migrations and no Redis** —
+entry metadata lives in a small Hotdata managed database (the registry),
+created on first use.
+
+Measured against TPC-H on a Neon Postgres (see [demo/](demo/)):
+
+| | Direct on Postgres | Miss (perceived) | Hit |
+|---|---|---|---|
+| Q1 pricing summary (full scan, 4 rows) | ~1.4 s | ~1.4 s | **~80 ms** |
+| Revenue by part (50,000 rows) | ~2.1 s | ~2.0 s | **~260 ms** |
+
+A miss costs the same as not caching (the persist runs write-behind); a hit
+is 8–18× faster with zero load on your database.
+
+## Install
+
+```bash
+pip install hotdata-materialized
+```
+
+```python
+# settings.py
+HOTDATA_MATERIALIZED = {
+    "API_KEY": env("HOTDATA_API_KEY"),      # hd_... key from app.hotdata.dev
+    "WORKSPACE_ID": "work...",
+    # optional:
+    # "API_URL": "https://api.hotdata.dev",
+    # "REGISTRY_DATABASE": "materialized_registry",
+    # "INLINE_THRESHOLD_BYTES": 2 * 1024 * 1024,
+    # "BACKGROUND": True,                   # write-behind persists (default)
+}
+```
+
+## Using it in a Django view
+
+Wrap the expensive part in `@materialize`; call it from your view:
+
+```python
+# reports.py
+from django.db.models import Count, Sum
+from hotdata_materialized import materialize
+
+from .models import Order
+
+
+@materialize(ttl=3600)
+def revenue_by_region():
+    return (
+        Order.objects
+        .filter(status="complete")
+        .values("region")
+        .annotate(orders=Count("id"), revenue=Sum("total"))
+        .order_by("-revenue")
+    )
+```
+
+```python
+# views.py
+from django.http import JsonResponse
+from .reports import revenue_by_region
+
+
+def revenue_view(request):
+    frame = revenue_by_region()
+    return JsonResponse({"rows": frame.to_pylist(), "cached": frame.cached})
+```
+
+On a **hit** the function never runs and your database is never touched. On a
+**miss** the function runs exactly as it would uncached, the caller gets the
+result immediately, and the persist happens in a background thread. Either
+way you get a `MaterializedFrame`:
+
+- `frame.arrow()` — the data as a `pyarrow.Table` (`frame.df()` for pandas,
+  `frame.to_pylist()` for dicts, `len(frame)` for the row count)
+- `frame.sql("SELECT region, revenue FROM this ORDER BY revenue DESC LIMIT 5")`
+  — SQL runs server-side against the cached entry; `this` names the data.
+  Needs a persisted entry: available on hits (on a fresh miss the
+  write-behind persist has to land first; `background=False` avoids that)
+- `frame.cached` — which path served it; `frame.entry` — the registry record
+
+The wrapped function can return an iterable of dicts (a `.values()` queryset),
+a `pyarrow.Table`, or a pandas DataFrame. Decorator knobs: `ttl=` (seconds,
+`None` = never expires), `version=` (bump to bust the cache on code changes),
+`key=` (human-readable label), `key_fn=` (stable identity for arguments that
+aren't JSON-serializable), `background=False` (block until persisted).
+Fail-open by design: if Hotdata is unreachable, the function runs and its
+result is served uncached — the cache degrades to "no cache," never to
+"no page."
+
+## Search
+
+Declare a search index on the decorator and the entry gets it when it
+persists (builds run server-side; a freshly missed entry may error on search
+for a few seconds until its index is ready):
+
+```python
+from hotdata_materialized import BM25, Vector, materialize
+
+
+@materialize(ttl=3600, index=BM25("notes"))
+def incidents():
+    return Incident.objects.values("id", "notes", "severity")
+
+
+incidents().search("checkout errors outage", column="notes", limit=5)
+```
+
+Vector (semantic) search embeds your text server-side via a workspace
+embedding provider — pass the indexed source column and a natural-language
+query:
+
+```python
+@materialize(ttl=3600, index=Vector("notes", provider="emb_...", metric="cosine"))
+def incidents_semantic():
+    return Incident.objects.values("id", "notes")
+
+
+incidents_semantic().vector_search("the site was down", column="notes", limit=5)
+```
+
+Both return the matching rows as a `pyarrow.Table` with a relevance column
+(`score` / `distance`). One platform rule to know: an embedding-backed vector
+index cannot share an entry with other indexes — the decorator rejects that
+combination up front.
+
+## Evicting entries
+
+The decorator composes public primitives (`fingerprint_call` /
+`fingerprint_queryset`, `Registry`, `EntryStore`) that you can use directly —
+for example to drop an entry explicitly:
+
+```python
+from hotdata_materialized import fingerprint_call
+from hotdata_materialized.decorator import get_runtime
+
+registry, store = get_runtime()
+store.evict(fingerprint_call(revenue_by_region))
+```
+
+Expired entries stop being served immediately (the `is_expired` check) and
+their databases carry a server-side expiry backstop; a sweep command that
+deletes them proactively is on the roadmap below.
+
+## How it works
+
+One managed database (the **registry**) holds one row per entry: fingerprint,
+status, TTL, the entry's database id, and — for small results — the data
+itself as an inline Arrow payload, so a chart-sized hit is served in a single
+API round trip. Larger results live in a per-entry managed database and are
+fetched as an Arrow IPC stream via a result id minted at persist time (no
+re-query on hits). Failures are loud and fail toward "no cache": a failed
+persist leaves no registry row, and the next request simply misses again.
+
+See [DESIGN.md](DESIGN.md) for the architecture and the accepted
+trade-offs.
+
+## Roadmap
+
+- [x] Content-addressed fingerprinting (querysets and function calls)
+- [x] Remote registry — no migrations or local state in the host app
+- [x] Entry store with write-behind persists
+- [x] Arrow-native reads (inline payloads, persisted-result fetch)
+- [x] `@materialize` decorator and `MaterializedFrame`
+      (`.arrow()/.df()/.to_pylist()/.sql()`)
+- [x] TPC-H demo and benchmark
+- [x] CI: pytest, mypy, flake8
+- [ ] Stale-while-revalidate refresh (rebuild protocol)
+- [ ] Sweep command for expired entries
+- [ ] Chainable queryset facade on the frame (`.filter()/.order_by()`)
+- [x] Vector/BM25 index declarations, `frame.search()` and
+      `frame.vector_search()`
+- [ ] Async view support (`await frame.aarrow()`)
+- [ ] PyPI release
+
+## Demo
+
+[demo/](demo/) is a small Django project that benchmarks TPC-H queries on a
+Neon Postgres directly vs. through the cache:
+
+```bash
+cd demo
+python manage.py compare                   # Q1: tiny result, inline hit
+python manage.py compare --scenario parts  # 50k rows: remote Arrow hit
+```
+
+## Development
+
+```bash
+uv venv && uv pip install -e '.[test]'
+uv run pytest
+```

hotdata_materialized-0.1.0/hotdata_materialized/__init__.py

@@ -0,0 +1,44 @@
+"""hotdata-materialized: offload expensive Django query results to Hotdata.
+
+Public surface: the @materialize decorator and MaterializedFrame handle,
+built on fingerprinting, the remote registry, and the entry store.
+"""
+
+from .conf import Config
+from .client import HotdataClients, get_clients, reset_clients
+from .decorator import MaterializedFrame, materialize
+from .exceptions import (
+    ConfigurationError,
+    FingerprintError,
+    MaterializedError,
+    RegistryError,
+    StoreError,
+)
+from .fingerprint import fingerprint_call, fingerprint_queryset
+from .indexes import BM25, Vector
+from .registry import Registry, RegistryEntry
+from .store import EntryStore
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "materialize",
+    "MaterializedFrame",
+    "BM25",
+    "Vector",
+    "Config",
+    "HotdataClients",
+    "get_clients",
+    "reset_clients",
+    "MaterializedError",
+    "ConfigurationError",
+    "FingerprintError",
+    "RegistryError",
+    "StoreError",
+    "fingerprint_call",
+    "fingerprint_queryset",
+    "Registry",
+    "RegistryEntry",
+    "EntryStore",
+    "__version__",
+]

hotdata_materialized-0.1.0/hotdata_materialized/_arrow.py

@@ -0,0 +1,44 @@
+"""Arrow/parquet encoding helpers shared by the registry and the entry store."""
+
+from __future__ import annotations
+
+import base64
+import io
+import json
+from typing import Optional
+
+import pyarrow as pa
+import pyarrow.ipc as pa_ipc
+import pyarrow.parquet as pq
+
+
+def table_to_parquet_bytes(table: pa.Table) -> bytes:
+    buffer = io.BytesIO()
+    pq.write_table(table, buffer)
+    return buffer.getvalue()
+
+
+def table_to_ipc_bytes(table: pa.Table) -> bytes:
+    sink = io.BytesIO()
+    with pa_ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table)
+    return sink.getvalue()
+
+
+def encode_inline_payload(table: pa.Table, threshold_bytes: int) -> Optional[str]:
+    # threshold bounds the payload as stored (post-base64), not the raw IPC
+    payload = base64.b64encode(table_to_ipc_bytes(table)).decode("ascii")
+    if len(payload) > threshold_bytes:
+        return None
+    return payload
+
+
+def decode_inline_payload(payload: str) -> pa.Table:
+    with pa_ipc.open_stream(io.BytesIO(base64.b64decode(payload))) as reader:
+        return reader.read_all()
+
+
+def schema_to_json(table: pa.Table) -> str:
+    return json.dumps(
+        [{"name": field.name, "type": str(field.type)} for field in table.schema]
+    )

hotdata_materialized-0.1.0/hotdata_materialized/_sql.py

@@ -0,0 +1,42 @@
+"""SQL literal and identifier encoding.
+
+POST /v1/queries takes SQL text only — there are no bind parameters — so
+every value written into SQL must go through quote_literal(), and every
+caller-supplied identifier (column names in search) through quote_ident().
+"""
+
+from __future__ import annotations
+
+import datetime as _dt
+import math
+import re
+
+from .exceptions import RegistryError
+
+_IDENT = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
+
+
+def quote_ident(name: str) -> str:
+    if not _IDENT.match(name):
+        raise ValueError(f"invalid SQL identifier: {name!r}")
+    return name
+
+
+def quote_literal(value) -> str:
+    if value is None:
+        return "NULL"
+    if isinstance(value, bool):
+        return "TRUE" if value else "FALSE"
+    if isinstance(value, int):
+        return str(value)
+    if isinstance(value, float):
+        if math.isnan(value) or math.isinf(value):
+            raise RegistryError(f"cannot encode non-finite float: {value!r}")
+        return repr(value)
+    if isinstance(value, (_dt.datetime, _dt.date)):
+        return quote_literal(value.isoformat())
+    if isinstance(value, str):
+        if "\x00" in value:
+            raise RegistryError("cannot encode string containing NUL byte")
+        return "'" + value.replace("'", "''") + "'"
+    raise RegistryError(f"cannot encode SQL literal of type {type(value).__name__}")

hotdata_materialized-0.1.0/hotdata_materialized/client.py

@@ -0,0 +1,50 @@
+"""Hotdata SDK client construction.
+
+One ApiClient (and its connection pool) per process, shared by the query,
+databases, and uploads resource APIs.
+"""
+
+from __future__ import annotations
+
+import threading
+from typing import Optional
+
+import hotdata
+from hotdata.arrow import ResultsApi
+from hotdata.query import QueryApi
+from hotdata.uploads import UploadsApi
+
+from .conf import Config
+
+
+class HotdataClients:
+    def __init__(self, config: Config):
+        sdk_config = hotdata.Configuration(
+            host=config.api_url,
+            api_key=config.api_key,
+            workspace_id=config.workspace_id,
+        )
+        self.api_client = hotdata.ApiClient(sdk_config)
+        self.query = QueryApi(self.api_client)
+        self.databases = hotdata.DatabasesApi(self.api_client)
+        self.uploads = UploadsApi(self.api_client)
+        self.results = ResultsApi(self.api_client)
+        self.indexes = hotdata.IndexesApi(self.api_client)
+
+
+_lock = threading.Lock()
+_clients: Optional[HotdataClients] = None
+
+
+def get_clients(config: Optional[Config] = None) -> HotdataClients:
+    global _clients
+    with _lock:
+        if _clients is None:
+            _clients = HotdataClients(config or Config.from_django())
+        return _clients
+
+
+def reset_clients() -> None:
+    global _clients
+    with _lock:
+        _clients = None

hotdata_materialized-0.1.0/hotdata_materialized/conf.py

@@ -0,0 +1,70 @@
+"""Configuration, loaded from the HOTDATA_MATERIALIZED dict in Django settings.
+
+The Config dataclass is plain and injectable so the library (and its tests)
+can run without a configured Django project.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Dict
+
+from .exceptions import ConfigurationError
+
+DEFAULT_API_URL = "https://api.hotdata.dev"
+
+_REQUIRED_KEYS = ("API_KEY", "WORKSPACE_ID")
+
+
+@dataclass(frozen=True)
+class Config:
+    api_key: str
+    workspace_id: str
+    api_url: str = DEFAULT_API_URL
+    registry_database: str = "materialized_registry"
+    inline_threshold_bytes: int = 2 * 1024 * 1024
+    entry_prefix: str = "mz_"
+    # Server-side entry-database expiry is set to ttl + grace: a safety net
+    # behind the sweep, never the primary eviction path.
+    entry_expiry_grace_seconds: int = 24 * 3600
+    # Write-behind is the default: a miss returns as soon as the caller's data
+    # is in hand and the persist runs on a small background pool.
+    background: bool = True
+    background_max_workers: int = 2
+
+    @classmethod
+    def from_dict(cls, raw: Dict[str, Any]) -> "Config":
+        missing = [key for key in _REQUIRED_KEYS if not raw.get(key)]
+        if missing:
+            raise ConfigurationError(
+                "HOTDATA_MATERIALIZED is missing required keys: " + ", ".join(missing)
+            )
+        return cls(
+            api_key=raw["API_KEY"],
+            workspace_id=raw["WORKSPACE_ID"],
+            api_url=raw.get("API_URL", DEFAULT_API_URL),
+            registry_database=raw.get("REGISTRY_DATABASE", cls.registry_database),
+            inline_threshold_bytes=raw.get(
+                "INLINE_THRESHOLD_BYTES", cls.inline_threshold_bytes
+            ),
+            entry_prefix=raw.get("ENTRY_PREFIX", cls.entry_prefix),
+            entry_expiry_grace_seconds=raw.get(
+                "ENTRY_EXPIRY_GRACE_SECONDS", cls.entry_expiry_grace_seconds
+            ),
+            background=raw.get("BACKGROUND", cls.background),
+            background_max_workers=raw.get(
+                "BACKGROUND_MAX_WORKERS", cls.background_max_workers
+            ),
+        )
+
+    @classmethod
+    def from_django(cls) -> "Config":
+        from django.conf import settings
+
+        raw = getattr(settings, "HOTDATA_MATERIALIZED", None)
+        if not isinstance(raw, dict):
+            raise ConfigurationError(
+                "Define a HOTDATA_MATERIALIZED dict in Django settings "
+                "(required keys: API_KEY, WORKSPACE_ID)."
+            )
+        return cls.from_dict(raw)