cached-duckdb 0.2.0__tar.gz

cached_duckdb-0.2.0/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-06-01
+
+### Changed
+- Standardized packaging to match deployment rules
+- Added `license-files`, `classifiers`, and `cibuildwheel` config to pyproject.toml
+- Replaced legacy setup.py with Cython-gated build pattern
+- Added version lookup via `importlib.metadata` in `__init__.py`
+- Added GitHub Actions publish workflow (`.github/workflows/publish.yml`)
+- Updated MANIFEST.in to include all documentation files
+- Bumped `requires-python` to `>=3.10`
+
+### Added
+- PersistenceCoordinator for external DB persistence
+- `DuckDbCachePersistenceError` exception
+- `cache_persistence.py` module
+
+## [0.1.0] - 2026-05-13
+
+### Added
+- Initial release of cached_duckdb library
+- Generic database/table API for in-memory DataFrame caching
+- Two storage modes: single_db and per_table_db
+- Atomic swap for safe concurrent writes
+- TTL-based expiry with background cleanup thread
+- Lazy stale flagging for active readers
+- Scheduler-managed table support (bypass TTL)
+- SQL query interface with WHERE clause filtering
+- Cross-table JOIN support (single_db mode)
+- Per-database configuration via JSON file
+- Environment variable configuration
+- Thread-safe operations with per-database/table locking
+- Comprehensive error handling with custom exceptions
+- Metadata queries (row count, columns, types, last updated)
+- Manual cache invalidation (per table or entire database)
+- Raw DuckDB connection access for advanced queries
+- Graceful shutdown with connection cleanup
+
+### Features
+- Zero disk usage - pure in-memory storage
+- Columnar format - 20-30% less RAM than pandas
+- Single-pass SQL queries - filter + aggregate in one operation
+- Safe concurrent reads during writes
+- Configurable TTL per table or database
+- Priority-based configuration resolution
+- Background cleanup thread with configurable interval

cached_duckdb-0.2.0/ENVIRONMENT_VARIABLES.md

@@ -0,0 +1,179 @@
+# Environment Variables
+
+This document lists all environment variables supported by cached_duckdb.
+
+All variables use the `CACHED_DUCKDB_` prefix by default.
+
+## Configuration Variables
+
+### CACHED_DUCKDB_DEFAULT_MODE
+- **Type:** String
+- **Default:** `single_db`
+- **Options:** `single_db` | `per_table_db`
+- **Description:** Default storage mode for all databases
+  - `single_db`: One connection per database (enables JOINs)
+  - `per_table_db`: One connection per (database, table) pair (parallel writes)
+
+### CACHED_DUCKDB_DEFAULT_TTL_MINUTES
+- **Type:** Integer
+- **Default:** `30`
+- **Description:** Default time-to-live in minutes for cached data
+- **Note:** Can be overridden per database/table in config file
+
+### CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES
+- **Type:** Integer
+- **Default:** `5`
+- **Description:** How often the background cleanup thread runs (in minutes)
+
+### CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS
+- **Type:** Float
+- **Default:** `30.0`
+- **Description:** Timeout for acquiring write locks (in seconds)
+- **Note:** Raises `DuckDbCacheLockError` if timeout exceeded
+
+### CACHED_DUCKDB_CONFIG_FILE_PATH
+- **Type:** String (file path)
+- **Default:** `None`
+- **Description:** Path to connector_config.json for per-database settings
+- **Example:** `/path/to/connector_config.json`
+
+### CACHED_DUCKDB_LOG_NAME
+- **Type:** String
+- **Default:** `cached_duckdb`
+- **Description:** Logger name for this library
+- **Note:** Use this to configure logging for cached_duckdb specifically
+
+## Persistence Variables (v0.2.0+)
+
+### CACHED_DUCKDB_PERSIST_BASE_PATH
+- **Type:** String (directory path)
+- **Default:** `None` (in-memory only, no persistence)
+- **Description:** Base directory for file-based persistence. When set:
+  - **Scenario 1 (DB-level):** Databases with a `persist_path` in connector_config.json (or all databases if no per-DB override) are stored as `{path}/{db_name}.duckdb` files instead of in-memory.
+  - **Scenario 2 (Table-level):** Tables marked with `"persist": true` in connector_config.json are saved as `{path}/{db_name}/{table_name}.parquet` files after each `store()`.
+- **Example:** `/data/cache` or `C:\data\cache`
+
+### CACHED_DUCKDB_SERVICE_NAME
+- **Type:** String
+- **Default:** `default`
+- **Description:** Service identifier used to namespace rows in the external DB snapshot table (`cached_duckdb_snapshots`). Allows multiple services to share the same external snapshot table.
+- **Example:** `order_service`, `analytics_pipeline`
+
+## Example Configuration
+
+### Linux/macOS (.env file)
+```bash
+CACHED_DUCKDB_DEFAULT_MODE=single_db
+CACHED_DUCKDB_DEFAULT_TTL_MINUTES=30
+CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES=5
+CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS=30
+CACHED_DUCKDB_CONFIG_FILE_PATH=/opt/config/connector_config.json
+CACHED_DUCKDB_PERSIST_BASE_PATH=/data/cache
+CACHED_DUCKDB_SERVICE_NAME=my_service
+CACHED_DUCKDB_LOG_NAME=cached_duckdb
+```
+
+### Windows (PowerShell)
+```powershell
+$env:CACHED_DUCKDB_DEFAULT_MODE="single_db"
+$env:CACHED_DUCKDB_DEFAULT_TTL_MINUTES="30"
+$env:CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES="5"
+$env:CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS="30"
+$env:CACHED_DUCKDB_CONFIG_FILE_PATH="C:\config\connector_config.json"
+$env:CACHED_DUCKDB_PERSIST_BASE_PATH="C:\data\cache"
+$env:CACHED_DUCKDB_SERVICE_NAME="my_service"
+$env:CACHED_DUCKDB_LOG_NAME="cached_duckdb"
+```
+
+### Python Code
+```python
+import os
+
+os.environ['CACHED_DUCKDB_DEFAULT_MODE'] = 'per_table_db'
+os.environ['CACHED_DUCKDB_DEFAULT_TTL_MINUTES'] = '60'
+os.environ['CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES'] = '10'
+os.environ['CACHED_DUCKDB_PERSIST_BASE_PATH'] = '/data/cache'
+os.environ['CACHED_DUCKDB_SERVICE_NAME'] = 'order_service'
+
+from cached_duckdb import DuckDbCacheConfig, DuckDbCacheManager
+
+config = DuckDbCacheConfig.from_env()
+cache = DuckDbCacheManager(config)
+```
+
+## Configuration Priority
+
+Configuration is resolved in this order (highest to lowest):
+
+1. **Per-table config** in connector_config.json (database → table → setting)
+2. **Per-database config** in connector_config.json (database → setting)
+3. **Environment variables** (CACHED_DUCKDB_*)
+4. **Hardcoded defaults** in DuckDbCacheConfig
+
+### Example Priority Resolution
+
+For TTL of `database="client_abc"`, `table="sales_data"`:
+
+```json
+// connector_config.json
+{
+  "client_abc": {
+    "default_cache_ttl_minutes": 45,  // Database-level
+    "sales_data": {
+      "cache_ttl_minutes": 60         // Table-level (highest priority)
+    }
+  }
+}
+```
+
+### Persistence Configuration in connector_config.json (v0.2.0+)
+
+```json
+{
+  "client_abc": {
+    "persist_path": "/data/cache",         // Scenario 1: DB saved as /data/cache/client_abc.duckdb
+    "sales_data": {
+      "persist": true                      // Scenario 2: table saved as {persist_base_path}/client_abc/sales_data.parquet
+    }
+  }
+}
+```
+
+**Note:** If `persist_path` is set on the database AND `persist: true` is set on a table within that database, the file-based DB takes precedence — the table is already on disk inside the `.duckdb` file, so no separate Parquet file is created.
+```
+
+```bash
+# .env
+CACHED_DUCKDB_DEFAULT_TTL_MINUTES=30    # Environment (lower priority)
+```
+
+**Result:** `cache_ttl_minutes = 60` (from table-level config)
+
+## Logging Configuration
+
+To configure logging for cached_duckdb:
+
+```python
+import logging
+
+# Set log level
+logging.getLogger('cached_duckdb').setLevel(logging.DEBUG)
+
+# Add handler
+handler = logging.StreamHandler()
+handler.setFormatter(logging.Formatter(
+    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+))
+logging.getLogger('cached_duckdb').addHandler(handler)
+```
+
+Or use environment-based log name:
+
+```bash
+export CACHED_DUCKDB_LOG_NAME=my_cache_logger
+```
+
+```python
+import logging
+logging.getLogger('my_cache_logger').setLevel(logging.INFO)
+```

cached_duckdb-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 sreeyenan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cached_duckdb-0.2.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include README.md
+include LICENSE
+include pyproject.toml
+include setup.py
+include VERSION
+include requirements.txt
+include CHANGELOG.md
+include ENVIRONMENT_VARIABLES.md
+include cached_duckdb_USER_MANUAL.md
+include *.py

cached_duckdb-0.2.0/PKG-INFO

@@ -0,0 +1,375 @@
+Metadata-Version: 2.4
+Name: cached-duckdb
+Version: 0.2.0
+Summary: Fast in-memory DataFrame cache using DuckDB with SQL query interface
+Author-email: sreeyenan <sreeyenanek@gmail.com>
+Keywords: duckdb,cache,dataframe,sql,in-memory
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: duckdb>=0.10.0
+Requires-Dist: pandas>=1.5.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Provides-Extra: protected
+Requires-Dist: Cython>=3.0; extra == "protected"
+Provides-Extra: all
+Requires-Dist: Cython>=3.0; extra == "all"
+Dynamic: license-file
+
+# cached_duckdb
+
+Fast in-memory DataFrame cache using DuckDB with SQL query interface.
+
+## Overview
+
+`cached_duckdb` replaces pandas dict-based caching with DuckDB in-memory connections for:
+- **Columnar storage** - 20-30% less RAM than pandas
+- **SQL queries** - Single-pass filter+aggregate operations
+- **Concurrency** - Safe parallel reads during writes
+- **Zero disk usage** - Pure in-memory like pandas
+
+## Key Features
+
+- **Generic database/table API** - Works with any cache-based system
+- **Two storage modes:**
+  - `single_db`: One connection per database (enables cross-table JOINs)
+  - `per_table_db`: One connection per (database, table) pair (fully parallel writes)
+- **Atomic swap** - Readers see 100% old or 100% new data, never partial
+- **TTL-based expiry** - Background cleanup with lazy stale flagging
+- **Scheduler-managed tables** - Bypass TTL for scheduled updates
+- **Thread-safe operations** - Per-database or per-table locking
+
+## Installation
+
+### From PyPI (Recommended)
+
+```bash
+pip install cached-duckdb
+```
+
+### Install Specific Version
+
+```bash
+pip install cached-duckdb==0.2.0
+```
+
+### With Optional Protected Build Extras
+
+```bash
+pip install "cached-duckdb[all]"
+```
+
+### From Local Source
+
+```bash
+pip install -r requirements.txt
+```
+
+Or install in development mode:
+
+```bash
+pip install -e .
+```
+
+Verify installed version:
+
+```python
+import cached_duckdb
+print(cached_duckdb.__version__)
+```
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from cached_duckdb import DuckDbCacheManager
+import pandas as pd
+
+# Initialize cache (singleton)
+cache = DuckDbCacheManager()
+
+# Store DataFrame
+df = pd.DataFrame({
+    'date': ['2026-01-01', '2026-01-02'],
+    'amount': [1000, 2000],
+    'country': ['USA', 'UK']
+})
+cache.store(database="client_abc", table="sales_data", df=df)
+
+# Query with SQL filtering
+result = cache.query(
+    database="client_abc",
+    table="sales_data",
+    sql_where="amount > 1000 AND country = 'USA'",
+    columns=["date", "amount"]
+)
+print(result)
+```
+
+### Advanced Queries
+
+```python
+# Get all data
+df = cache.query(database="app_x", table="dataset_1")
+
+# Filter with WHERE
+df = cache.query(
+    database="app_x",
+    table="dataset_1",
+    sql_where="age > 25 AND country = 'USA'"
+)
+
+# Select specific columns
+df = cache.query(
+    database="app_x",
+    table="dataset_1",
+    columns=["name", "age", "salary"]
+)
+
+# Limit results
+df = cache.query(
+    database="app_x",
+    table="dataset_1",
+    sql_where="date >= '2026-01-01'",
+    limit=100
+)
+```
+
+### Cross-Table JOINs (single_db mode)
+
+```python
+# Execute raw SQL for complex queries
+sql = """
+    SELECT s.date, s.amount, o.customer_name
+    FROM sales_data s
+    JOIN orders o ON s.order_id = o.id
+    WHERE s.amount > 1000
+"""
+result = cache.execute_sql(database="client_abc", sql=sql)
+```
+
+### Check if Data Exists
+
+```python
+if cache.exists(database="client_abc", table="sales_data"):
+    # Data is ready and fresh
+    df = cache.query(database="client_abc", table="sales_data")
+else:
+    # Data missing or stale - reload needed
+    df = load_from_source()
+    cache.store(database="client_abc", table="sales_data", df=df)
+```
+
+### TTL and Scheduler-Managed Tables
+
+```python
+# Store with custom TTL
+cache.store(
+    database="client_abc",
+    table="sales_data",
+    df=df,
+    ttl_minutes=60  # Expires after 60 minutes
+)
+
+# Scheduler-managed table (no auto-expiry on reads)
+cache.store(
+    database="client_abc",
+    table="sales_data",
+    df=df,
+    scheduler_managed=True  # Only scheduler updates it
+)
+```
+
+### Invalidate Cache
+
+```python
+# Invalidate one table
+cache.invalidate(database="client_abc", table="sales_data")
+
+# Invalidate all tables for a database
+cache.invalidate(database="client_abc")
+```
+
+### Get Metadata
+
+```python
+# Last updated timestamp
+last_updated = cache.get_last_updated(database="client_abc", table="sales_data")
+print(f"Last updated: {last_updated}")
+
+# Table info
+info = cache.get_table_info(database="client_abc", table="sales_data")
+print(f"Rows: {info['row_count']}")
+print(f"Columns: {info['columns']}")
+print(f"Types: {info['column_types']}")
+```
+
+## Documentation
+
+- [User Manual](cached_duckdb_USER_MANUAL.md)
+- [Environment Variables](ENVIRONMENT_VARIABLES.md)
+- [Changelog](CHANGELOG.md)
+
+## Author
+
+**sreeyenan** (sreeyenanek@gmail.com)
+
+## Version
+
+Current version: **0.2.0**
+
+## Configuration
+
+### Environment Variables
+
+```bash
+# Storage mode: single_db or per_table_db
+CACHED_DUCKDB_DEFAULT_MODE=single_db
+
+# Default TTL in minutes
+CACHED_DUCKDB_DEFAULT_TTL_MINUTES=30
+
+# Cleanup thread interval
+CACHED_DUCKDB_CLEANUP_INTERVAL_MINUTES=5
+
+# Lock timeout in seconds
+CACHED_DUCKDB_LOCK_TIMEOUT_SECONDS=30
+
+# Path to connector config file (optional)
+CACHED_DUCKDB_CONFIG_FILE_PATH=/path/to/connector_config.json
+
+# Logger name
+CACHED_DUCKDB_LOG_NAME=cached_duckdb
+```
+
+### Per-Database Configuration File
+
+Create `connector_config.json` for per-database settings:
+
+```json
+{
+  "client_abc": {
+    "duck_cache_mode": "per_table_db",
+    "default_cache_ttl_minutes": 45,
+    "sales_data": {
+      "cache_ttl_minutes": 60,
+      "scheduler_managed": false
+    },
+    "live_feed": {
+      "cache_ttl_minutes": 0,
+      "scheduler_managed": true
+    }
+  },
+  "client_xyz": {
+    "duck_cache_mode": "single_db"
+  }
+}
+```
+
+**Priority order:**
+1. Per-table config in JSON file (highest)
+2. Per-database config in JSON file
+3. Environment variables
+4. Hardcoded defaults (lowest)
+
+### Load Configuration
+
+```python
+from cached_duckdb import DuckDbCacheConfig, DuckDbCacheManager
+
+# From environment
+config = DuckDbCacheConfig.from_env()
+cache = DuckDbCacheManager(config)
+
+# From dict
+config = DuckDbCacheConfig.from_dict({
+    "default_mode": "single_db",
+    "default_cache_ttl_minutes": 30,
+    "config_file_path": "/path/to/connector_config.json"
+})
+cache = DuckDbCacheManager(config)
+```
+
+## Storage Modes
+
+### Mode A: single_db (Default)
+
+- One DuckDB connection per `database`
+- Multiple tables share the same connection
+- Enables cross-table SQL JOINs
+- Write contention: One lock per database
+
+**Use when:** Database has few tables (< 20) or need cross-table queries
+
+### Mode B: per_table_db
+
+- One DuckDB connection per `(database, table)` pair
+- Each table is fully isolated
+- Zero write contention between tables
+- Fully parallel writes
+
+**Use when:** Database has many tables (20+) or high write concurrency
+
+## Architecture
+
+```
+DuckDbCacheManager (singleton)
+  ├── CacheStore        - Atomic writes, table management
+  ├── CacheQuery        - SQL queries, metadata
+  ├── TTLRegistry       - TTL tracking, cleanup thread
+  └── CacheConfigResolver - Per-database config resolution
+```
+
+## Thread Safety
+
+- **store()**: Write-locked per database or per table
+- **query()**: Lock-free parallel reads
+- **invalidate()**: Write-locked, waits for active readers
+- **Background cleanup**: Minimal locking, uses stale flags
+
+## Use Cases
+
+- **Multi-tenant web APIs** - Cache per tenant with `database=tenant_id`
+- **Analytics dashboards** - Fast in-memory OLAP queries
+- **ETL pipelines** - Store intermediate DataFrames
+- **Session managers** - Replace pandas dict caching
+- **Microservices** - Shared cache library across services
+
+## API Reference
+
+### DuckDbCacheManager
+
+- `store(database, table, df, ttl_minutes=None, scheduler_managed=False)` - Store DataFrame
+- `query(database, table, sql_where=None, columns=None, limit=None)` - Query with filtering
+- `execute_sql(database, sql)` - Execute raw SQL
+- `exists(database, table)` - Check if exists and fresh
+- `invalidate(database, table=None)` - Remove from cache
+- `get_last_updated(database, table)` - Get timestamp
+- `get_table_info(database, table)` - Get metadata
+- `get_raw_connection(database, table=None)` - Get DuckDB connection
+- `shutdown()` - Close all connections
+
+### Exceptions
+
+- `DuckDbCacheError` - Base exception
+- `DuckDbCacheConfigError` - Configuration error
+- `DuckDbCacheLockError` - Lock acquisition failed
+- `DuckDbCacheNotFoundError` - Table not found
+- `DuckDbCacheStaleError` - Data is stale
+
+## License
+
+MIT License - see LICENSE file
+
+## Author
+
+sreeyenan