cached-duckdb 0.2.0__py3-none-any.whl

cached_duckdb/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-06-01
+
+### Changed
+- Standardized packaging to match deployment rules
+- Added `license-files`, `classifiers`, and `cibuildwheel` config to pyproject.toml
+- Replaced legacy setup.py with Cython-gated build pattern
+- Added version lookup via `importlib.metadata` in `__init__.py`
+- Added GitHub Actions publish workflow (`.github/workflows/publish.yml`)
+- Updated MANIFEST.in to include all documentation files
+- Bumped `requires-python` to `>=3.10`
+
+### Added
+- PersistenceCoordinator for external DB persistence
+- `DuckDbCachePersistenceError` exception
+- `cache_persistence.py` module
+
+## [0.1.0] - 2026-05-13
+
+### Added
+- Initial release of cached_duckdb library
+- Generic database/table API for in-memory DataFrame caching
+- Two storage modes: single_db and per_table_db
+- Atomic swap for safe concurrent writes
+- TTL-based expiry with background cleanup thread
+- Lazy stale flagging for active readers
+- Scheduler-managed table support (bypass TTL)
+- SQL query interface with WHERE clause filtering
+- Cross-table JOIN support (single_db mode)
+- Per-database configuration via JSON file
+- Environment variable configuration
+- Thread-safe operations with per-database/table locking
+- Comprehensive error handling with custom exceptions
+- Metadata queries (row count, columns, types, last updated)
+- Manual cache invalidation (per table or entire database)
+- Raw DuckDB connection access for advanced queries
+- Graceful shutdown with connection cleanup
+
+### Features
+- Zero disk usage - pure in-memory storage
+- Columnar format - 20-30% less RAM than pandas
+- Single-pass SQL queries - filter + aggregate in one operation
+- Safe concurrent reads during writes
+- Configurable TTL per table or database
+- Priority-based configuration resolution
+- Background cleanup thread with configurable interval

cached_duckdb/ENVIRONMENT_VARIABLES.md

@@ -0,0 +1,179 @@
+# Environment Variables
+
+This document lists all environment variables supported by cached_duckdb.
+
+All variables use the `CACHED_DUCKDB_` prefix by default.
+
+## Configuration Variables
+
+### CACHED_DUCKDB_DEFAULT_MODE
+- **Type:** String
+- **Default:** `single_db`
+- **Options:** `single_db` | `per_table_db`
+- **Description:** Default storage mode for all databases
+  - `single_db`: One connection per database (enables JOINs)
+  - `per_table_db`: One connection per (database, table) pair (parallel writes)
+
+### CACHED_DUCKDB_DEFAULT_TTL_MINUTES
+- **Type:** Integer
+- **Default:** `30`
+- **Description:** Default time-to-live in minutes for cached data
+- **Note:** Can be overridden per database/table in config file
+
+### CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES
+- **Type:** Integer
+- **Default:** `5`
+- **Description:** How often the background cleanup thread runs (in minutes)
+
+### CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS
+- **Type:** Float
+- **Default:** `30.0`
+- **Description:** Timeout for acquiring write locks (in seconds)
+- **Note:** Raises `DuckDbCacheLockError` if timeout exceeded
+
+### CACHED_DUCKDB_CONFIG_FILE_PATH
+- **Type:** String (file path)
+- **Default:** `None`
+- **Description:** Path to connector_config.json for per-database settings
+- **Example:** `/path/to/connector_config.json`
+
+### CACHED_DUCKDB_LOG_NAME
+- **Type:** String
+- **Default:** `cached_duckdb`
+- **Description:** Logger name for this library
+- **Note:** Use this to configure logging for cached_duckdb specifically
+
+## Persistence Variables (v0.2.0+)
+
+### CACHED_DUCKDB_PERSIST_BASE_PATH
+- **Type:** String (directory path)
+- **Default:** `None` (in-memory only, no persistence)
+- **Description:** Base directory for file-based persistence. When set:
+  - **Scenario 1 (DB-level):** Databases with a `persist_path` in connector_config.json (or all databases if no per-DB override) are stored as `{path}/{db_name}.duckdb` files instead of in-memory.
+  - **Scenario 2 (Table-level):** Tables marked with `"persist": true` in connector_config.json are saved as `{path}/{db_name}/{table_name}.parquet` files after each `store()`.
+- **Example:** `/data/cache` or `C:\data\cache`
+
+### CACHED_DUCKDB_SERVICE_NAME
+- **Type:** String
+- **Default:** `default`
+- **Description:** Service identifier used to namespace rows in the external DB snapshot table (`cached_duckdb_snapshots`). Allows multiple services to share the same external snapshot table.
+- **Example:** `order_service`, `analytics_pipeline`
+
+## Example Configuration
+
+### Linux/macOS (.env file)
+```bash
+CACHED_DUCKDB_DEFAULT_MODE=single_db
+CACHED_DUCKDB_DEFAULT_TTL_MINUTES=30
+CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES=5
+CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS=30
+CACHED_DUCKDB_CONFIG_FILE_PATH=/opt/config/connector_config.json
+CACHED_DUCKDB_PERSIST_BASE_PATH=/data/cache
+CACHED_DUCKDB_SERVICE_NAME=my_service
+CACHED_DUCKDB_LOG_NAME=cached_duckdb
+```
+
+### Windows (PowerShell)
+```powershell
+$env:CACHED_DUCKDB_DEFAULT_MODE="single_db"
+$env:CACHED_DUCKDB_DEFAULT_TTL_MINUTES="30"
+$env:CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES="5"
+$env:CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS="30"
+$env:CACHED_DUCKDB_CONFIG_FILE_PATH="C:\config\connector_config.json"
+$env:CACHED_DUCKDB_PERSIST_BASE_PATH="C:\data\cache"
+$env:CACHED_DUCKDB_SERVICE_NAME="my_service"
+$env:CACHED_DUCKDB_LOG_NAME="cached_duckdb"
+```
+
+### Python Code
+```python
+import os
+
+os.environ['CACHED_DUCKDB_DEFAULT_MODE'] = 'per_table_db'
+os.environ['CACHED_DUCKDB_DEFAULT_TTL_MINUTES'] = '60'
+os.environ['CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES'] = '10'
+os.environ['CACHED_DUCKDB_PERSIST_BASE_PATH'] = '/data/cache'
+os.environ['CACHED_DUCKDB_SERVICE_NAME'] = 'order_service'
+
+from cached_duckdb import DuckDbCacheConfig, DuckDbCacheManager
+
+config = DuckDbCacheConfig.from_env()
+cache = DuckDbCacheManager(config)
+```
+
+## Configuration Priority
+
+Configuration is resolved in this order (highest to lowest):
+
+1. **Per-table config** in connector_config.json (database → table → setting)
+2. **Per-database config** in connector_config.json (database → setting)
+3. **Environment variables** (CACHED_DUCKDB_*)
+4. **Hardcoded defaults** in DuckDbCacheConfig
+
+### Example Priority Resolution
+
+For TTL of `database="client_abc"`, `table="sales_data"`:
+
+```json
+// connector_config.json
+{
+  "client_abc": {
+    "default_cache_ttl_minutes": 45,  // Database-level
+    "sales_data": {
+      "cache_ttl_minutes": 60         // Table-level (highest priority)
+    }
+  }
+}
+```
+
+### Persistence Configuration in connector_config.json (v0.2.0+)
+
+```json
+{
+  "client_abc": {
+    "persist_path": "/data/cache",         // Scenario 1: DB saved as /data/cache/client_abc.duckdb
+    "sales_data": {
+      "persist": true                      // Scenario 2: table saved as {persist_base_path}/client_abc/sales_data.parquet
+    }
+  }
+}
+```
+
+**Note:** If `persist_path` is set on the database AND `persist: true` is set on a table within that database, the file-based DB takes precedence — the table is already on disk inside the `.duckdb` file, so no separate Parquet file is created.
+```
+
+```bash
+# .env
+CACHED_DUCKDB_DEFAULT_TTL_MINUTES=30    # Environment (lower priority)
+```
+
+**Result:** `cache_ttl_minutes = 60` (from table-level config)
+
+## Logging Configuration
+
+To configure logging for cached_duckdb:
+
+```python
+import logging
+
+# Set log level
+logging.getLogger('cached_duckdb').setLevel(logging.DEBUG)
+
+# Add handler
+handler = logging.StreamHandler()
+handler.setFormatter(logging.Formatter(
+    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+))
+logging.getLogger('cached_duckdb').addHandler(handler)
+```
+
+Or use environment-based log name:
+
+```bash
+export CACHED_DUCKDB_LOG_NAME=my_cache_logger
+```
+
+```python
+import logging
+logging.getLogger('my_cache_logger').setLevel(logging.INFO)
+```

cached_duckdb/VERSION

@@ -0,0 +1 @@
+0.2.0

cached_duckdb/__init__.py

@@ -0,0 +1,48 @@
+"""cached_duckdb - Fast in-memory DataFrame cache using DuckDB."""
+
+try:
+    from importlib.metadata import version
+    __version__ = version("cached-duckdb")
+except Exception:
+    __version__ = "0.0.0"
+
+from .config import DuckDbCacheConfig, CacheConfigResolver
+from .cache_manager import DuckDbCacheManager
+from .cache_persistence import PersistenceCoordinator
+from .errors import (
+    DuckDbCacheError,
+    DuckDbCacheConfigError,
+    DuckDbCacheLockError,
+    DuckDbCacheNotFoundError,
+    DuckDbCacheStaleError,
+    DuckDbCachePersistenceError
+)
+
+
+def get_cache_manager(config: DuckDbCacheConfig = None, external_connector=None) -> DuckDbCacheManager:
+    """Get DuckDB cache manager instance (singleton).
+    
+    Args:
+        config: Optional configuration. If None, loads from environment.
+        external_connector: Optional duck-typed connector for external DB
+            persistence. Must implement execute(sql, params), fetchone(), fetchall().
+        
+    Returns:
+        DuckDbCacheManager instance
+    """
+    return DuckDbCacheManager(config, external_connector=external_connector)
+
+
+__all__ = [
+    "DuckDbCacheConfig",
+    "CacheConfigResolver",
+    "DuckDbCacheManager",
+    "PersistenceCoordinator",
+    "DuckDbCacheError",
+    "DuckDbCacheConfigError",
+    "DuckDbCacheLockError",
+    "DuckDbCacheNotFoundError",
+    "DuckDbCacheStaleError",
+    "DuckDbCachePersistenceError",
+    "get_cache_manager",
+]