supertable 2.4.0__tar.gz → 2.5.0__tar.gz

{supertable-2.4.0/supertable.egg-info → supertable-2.5.0}/PKG-INFO

@@ -1,15 +1,15 @@
 Metadata-Version: 2.4
 Name: supertable
-Version: 2.4.0
+Version: 2.5.0
 Summary: SuperTable — versioned data lake library for SQL analytics on Parquet + Redis.
 Author: Levente Kupas
 Author-email: Levente Kupas <lkupas@kladnasoft.com>
-License: Super Table Public Use License (STPUL) v1.0
+License: FSL-1.1-ALv2
 Project-URL: Homepage, https://github.com/kladnasoft/supertable
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: License :: Other/Proprietary License
-Classifier: Operating System :: OS Independent
+Classifier: Operating System :: POSIX
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -18,10 +18,12 @@ Requires-Dist: pandas<2.3,>=2.1
 Requires-Dist: pyarrow<19,>=16
 Requires-Dist: pytz>=2023.3
 Requires-Dist: tzdata>=2023.3
-Requires-Dist: duckdb<2.0,>=1.1
-Requires-Dist: polars<2.0,>=1.6
-Requires-Dist: sqlglot<27,>=26
+Requires-Dist: apscheduler<4.0,>=3.11.3
+Requires-Dist: duckdb<2.0,>=1.5.1
+Requires-Dist: polars<2.0,>=1.36.1
+Requires-Dist: sqlglot<27,>=26.33
 Requires-Dist: colorlog<7.0,>=6.9
+Requires-Dist: tqdm<5.0,>=4.66
 Requires-Dist: python-dotenv<2.0,>=1.0
 Requires-Dist: setuptools>=68
 Requires-Dist: redis<6.0,>=5.0
@@ -45,7 +47,7 @@ Dynamic: requires-python
 
 ![Python](https://img.shields.io/badge/python-3.10%2B-blue)
 ![License](https://img.shields.io/badge/license-FSL--1.1--ALv2-orange)
-![Version](https://img.shields.io/badge/version-2.0.0-brightgreen)
+![Version](https://img.shields.io/badge/version-2.5.0-brightgreen)
 
 **SuperTable — versioned data lake library for SQL analytics.**
 
@@ -66,8 +68,9 @@ pip install "supertable[gcp]"         # Google Cloud Storage
 pip install "supertable[all]"         # everything
 ```
 
-Requirements: Python 3.10+, a reachable Redis 6+, and a configured storage
-backend (or local disk for development). See
+Requirements: Python 3.10+, a POSIX host (the durable monitoring spool uses
+`fcntl` locking), a reachable Redis 6+, and a configured storage backend (or
+local disk for development). See
 [docs/02_configuration.md](docs/02_configuration.md) for environment
 variables.
 
@@ -140,7 +143,7 @@ df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
 
 ## Demos
 
-The package ships two runnable demos under `supertable.demo`:
+The package ships three runnable demos under `supertable.demo`:
 
 ```bash
 # Numbered tutorial — runs the full lifecycle end-to-end.
@@ -152,9 +155,13 @@ python -m supertable.demo.quickstart
 supertable-demo-webshop-generate    # build ~1.2M rows on disk
 supertable-demo-webshop-load        # load them into SuperTable
 supertable-demo-webshop-topup       # continuous incremental refresh
+
+# Medical-center finance demo (raw → staging → mart, fully synthetic).
+supertable-demo-medcenter-run       # generate → load → transform → tests →
+                                    # accounting export → idempotency proof
 ```
 
-Both demos are also runnable as module steps. Examples:
+All demos are also runnable as module steps. Examples:
 
 ```bash
 python -m supertable.demo.quickstart.s01_01_01_create_super_table
@@ -171,7 +178,7 @@ script index.
 
 - **Versioned tables** with snapshot isolation, upsert (`overwrite_columns`),
   soft deletes (`delete_only=True`), schema evolution, and staleness filtering
-- **DuckDB query engine** — embedded, zero-copy reads from object storage
+- **DuckDB and IslandDB query engines** — embedded reads with adaptive selection
 - **Spark SQL via Thrift** — for queries exceeding DuckDB memory limits
 - **RBAC** — role types (superadmin, admin, writer, reader, meta) with
   row-level and column-level security enforced through view chains
@@ -202,13 +209,15 @@ See [docs/00_index.md](docs/00_index.md) for the full table of contents.
 | 06 | [Data Writer](docs/06_data_writer.md) | Write pipeline, locking, dedup, tombstones |
 | 07 | [Ingestion & Pipes](docs/07_ingestion.md) | Staging areas, automated ingestion pipes |
 | 08 | [Distributed Locking](docs/08_locking.md) | Redis locks, file locks, deadlock prevention |
-| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB Lite/Pro, Spark SQL, auto selection |
+| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB, IslandDB, Spark SQL, auto selection |
 | 10 | [Data Reader](docs/10_data_reader.md) | Read facade, snapshot history, view chain |
 | 11 | [RBAC & Access Control](docs/11_rbac.md) | Roles, users, row/column security |
 | 12 | [Audit Logging](docs/12_audit.md) | SHA-256 hash chain, DORA/SOC 2, SIEM |
 | 13 | [Table Mirroring](docs/13_mirroring.md) | Delta Lake, Iceberg, Parquet export |
 | 14 | [Monitoring](docs/14_monitoring.md) | Metrics writer, structured logging |
 | 15 | [Python SDK](docs/15_python_sdk.md) | Core classes, demos, example index |
+| 16 | [Redis Key Layout](docs/16_redis_layout.md) | Canonical key namespaces and validation rules |
+| 17 | [Privileged Audit Worker](docs/17_privileged_audit_worker.md) | Supervised deployment, durability preflight, safe trimming, probes, failure policy |
 
 ---
 

{supertable-2.4.0 → supertable-2.5.0}/README.md

@@ -2,7 +2,7 @@
 
 ![Python](https://img.shields.io/badge/python-3.10%2B-blue)
 ![License](https://img.shields.io/badge/license-FSL--1.1--ALv2-orange)
-![Version](https://img.shields.io/badge/version-2.0.0-brightgreen)
+![Version](https://img.shields.io/badge/version-2.5.0-brightgreen)
 
 **SuperTable — versioned data lake library for SQL analytics.**
 
@@ -23,8 +23,9 @@ pip install "supertable[gcp]"         # Google Cloud Storage
 pip install "supertable[all]"         # everything
 ```
 
-Requirements: Python 3.10+, a reachable Redis 6+, and a configured storage
-backend (or local disk for development). See
+Requirements: Python 3.10+, a POSIX host (the durable monitoring spool uses
+`fcntl` locking), a reachable Redis 6+, and a configured storage backend (or
+local disk for development). See
 [docs/02_configuration.md](docs/02_configuration.md) for environment
 variables.
 
@@ -97,7 +98,7 @@ df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
 
 ## Demos
 
-The package ships two runnable demos under `supertable.demo`:
+The package ships three runnable demos under `supertable.demo`:
 
 ```bash
 # Numbered tutorial — runs the full lifecycle end-to-end.
@@ -109,9 +110,13 @@ python -m supertable.demo.quickstart
 supertable-demo-webshop-generate    # build ~1.2M rows on disk
 supertable-demo-webshop-load        # load them into SuperTable
 supertable-demo-webshop-topup       # continuous incremental refresh
+
+# Medical-center finance demo (raw → staging → mart, fully synthetic).
+supertable-demo-medcenter-run       # generate → load → transform → tests →
+                                    # accounting export → idempotency proof
 ```
 
-Both demos are also runnable as module steps. Examples:
+All demos are also runnable as module steps. Examples:
 
 ```bash
 python -m supertable.demo.quickstart.s01_01_01_create_super_table
@@ -128,7 +133,7 @@ script index.
 
 - **Versioned tables** with snapshot isolation, upsert (`overwrite_columns`),
   soft deletes (`delete_only=True`), schema evolution, and staleness filtering
-- **DuckDB query engine** — embedded, zero-copy reads from object storage
+- **DuckDB and IslandDB query engines** — embedded reads with adaptive selection
 - **Spark SQL via Thrift** — for queries exceeding DuckDB memory limits
 - **RBAC** — role types (superadmin, admin, writer, reader, meta) with
   row-level and column-level security enforced through view chains
@@ -159,13 +164,15 @@ See [docs/00_index.md](docs/00_index.md) for the full table of contents.
 | 06 | [Data Writer](docs/06_data_writer.md) | Write pipeline, locking, dedup, tombstones |
 | 07 | [Ingestion & Pipes](docs/07_ingestion.md) | Staging areas, automated ingestion pipes |
 | 08 | [Distributed Locking](docs/08_locking.md) | Redis locks, file locks, deadlock prevention |
-| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB Lite/Pro, Spark SQL, auto selection |
+| 09 | [Query Engine](docs/09_query_engine.md) | DuckDB, IslandDB, Spark SQL, auto selection |
 | 10 | [Data Reader](docs/10_data_reader.md) | Read facade, snapshot history, view chain |
 | 11 | [RBAC & Access Control](docs/11_rbac.md) | Roles, users, row/column security |
 | 12 | [Audit Logging](docs/12_audit.md) | SHA-256 hash chain, DORA/SOC 2, SIEM |
 | 13 | [Table Mirroring](docs/13_mirroring.md) | Delta Lake, Iceberg, Parquet export |
 | 14 | [Monitoring](docs/14_monitoring.md) | Metrics writer, structured logging |
 | 15 | [Python SDK](docs/15_python_sdk.md) | Core classes, demos, example index |
+| 16 | [Redis Key Layout](docs/16_redis_layout.md) | Canonical key namespaces and validation rules |
+| 17 | [Privileged Audit Worker](docs/17_privileged_audit_worker.md) | Supervised deployment, durability preflight, safe trimming, probes, failure policy |
 
 ---
 

supertable-2.5.0/pyproject.toml

@@ -0,0 +1,89 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "supertable"
+version = "2.5.0"
+description = "SuperTable — versioned data lake library for SQL analytics on Parquet + Redis."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "FSL-1.1-ALv2" }
+authors = [{ name = "Levente Kupas", email = "lkupas@kladnasoft.com" }]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "License :: Other/Proprietary License",
+  "Operating System :: POSIX"
+]
+
+# Read runtime deps from requirements.txt to keep a single source of truth
+dynamic = ["dependencies"]
+
+[tool.setuptools.dynamic]
+dependencies = { file = ["requirements.txt"] }
+
+[project.optional-dependencies]
+# Storage backends (pick what you deploy on)
+s3    = ["boto3>=1.34,<2.0"]
+minio = ["minio>=7.2,<8.0"]
+azure = ["azure-storage-blob>=12.26.0"]
+gcp   = ["google-cloud-storage>=3.1.0"]
+all-cloud = ["supertable[s3,minio,azure,gcp]"]
+
+# Everything
+all = ["supertable[all-cloud]"]
+
+[project.urls]
+Homepage = "https://github.com/kladnasoft/supertable"
+
+[project.scripts]
+supertable-privileged-audit-worker = "supertable.audit.privileged_worker:main"
+supertable-redis-recovery = "supertable.recovery.redis_rebuild:main"
+supertable-demo-quickstart       = "supertable.demo.quickstart.controller:main"
+supertable-demo-webshop-generate = "supertable.demo.webshop.generate:main"
+supertable-demo-webshop-load     = "supertable.demo.webshop.load:main"
+supertable-demo-webshop-topup    = "supertable.demo.webshop.topup:main"
+supertable-demo-medcenter-generate  = "supertable.demo.medcenter.generate:main"
+supertable-demo-medcenter-load      = "supertable.demo.medcenter.load:main"
+supertable-demo-medcenter-transform = "supertable.demo.medcenter.transform:main"
+supertable-demo-medcenter-quality   = "supertable.demo.medcenter.quality:main"
+supertable-demo-medcenter-export    = "supertable.demo.medcenter.export_accounting:main"
+supertable-demo-medcenter-run       = "supertable.demo.medcenter.run:main"
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.packages.find]
+include = ["supertable", "supertable.*"]
+
+[tool.pytest.ini_options]
+asyncio_default_fixture_loop_scope = "function"
+markers = [
+  "integration: requires an explicitly configured disposable external service",
+]
+
+[tool.ruff.lint]
+# Start with correctness rules that can never be waived by formatting taste:
+# syntax errors, invalid control flow, and undefined names. Broader rule groups
+# can be enabled incrementally without weakening this release-blocking baseline.
+select = ["E9", "F63", "F7", "F82"]
+
+[tool.mypy]
+python_version = "3.10"
+files = ["supertable"]
+# Tests and benchmarks have their own runtime gates.  Every shipped production
+# module (including demos/console entry points) remains inside the mypy target.
+exclude = [
+  "^supertable/(?:[^/]+/)*(?:tests|benchmarks)/",
+]
+follow_imports = "skip"
+ignore_missing_imports = true
+# Keep the debt baseline reproducible across the minimum/latest dependency
+# jobs; runtime compatibility is exercised by pytest, while this gate checks
+# SuperTable's own modules against one pinned mypy/typeshed version.
+no_site_packages = true
+check_untyped_defs = true
+incremental = false
+show_error_codes = true
+warn_unused_ignores = true

{supertable-2.4.0 → supertable-2.5.0}/requirements.txt

@@ -4,12 +4,14 @@ pandas>=2.1,<2.3
 pyarrow>=16,<19
 pytz>=2023.3
 tzdata>=2023.3
+apscheduler>=3.11.3,<4.0
 
 # -------- Engine / query / utils --------
-duckdb>=1.1,<2.0
-polars>=1.6,<2.0
-sqlglot>=26,<27
+duckdb>=1.5.1,<2.0
+polars>=1.36.1,<2.0
+sqlglot>=26.33,<27
 colorlog>=6.9,<7.0
+tqdm>=4.66,<5.0
 python-dotenv>=1.0,<2.0
 setuptools>=68
 

{supertable-2.4.0 → supertable-2.5.0}/setup.py

@@ -19,13 +19,13 @@ long_description = readme.read_text(encoding="utf-8") if readme.exists() else ""
 
 setup(
     name="supertable",
-    version="2.4.0",
+    version="2.5.0",
     description="SuperTable — versioned data lake library for SQL analytics on Parquet + Redis.",
     long_description=long_description,
     long_description_content_type="text/markdown",
     author="Levente Kupas",
     author_email="lkupas@kladnasoft.com",
-    license="Super Table Public Use License (STPUL) v1.0",
+    license="FSL-1.1-ALv2",
     python_requires=">=3.10",
     packages=find_packages(include=["supertable", "supertable.*"]),
     include_package_data=True,
@@ -34,7 +34,7 @@ setup(
         "Programming Language :: Python :: 3",
         "Programming Language :: Python :: 3 :: Only",
         "License :: Other/Proprietary License",
-        "Operating System :: OS Independent",
+        "Operating System :: POSIX",
     ],
     extras_require={
         "s3": ["boto3>=1.34,<2.0"],
@@ -56,10 +56,18 @@ setup(
     },
     entry_points={
         "console_scripts": [
+            "supertable-privileged-audit-worker=supertable.audit.privileged_worker:main",
+            "supertable-redis-recovery=supertable.recovery.redis_rebuild:main",
             "supertable-demo-quickstart=supertable.demo.quickstart.controller:main",
             "supertable-demo-webshop-generate=supertable.demo.webshop.generate:main",
             "supertable-demo-webshop-load=supertable.demo.webshop.load:main",
             "supertable-demo-webshop-topup=supertable.demo.webshop.topup:main",
+            "supertable-demo-medcenter-generate=supertable.demo.medcenter.generate:main",
+            "supertable-demo-medcenter-load=supertable.demo.medcenter.load:main",
+            "supertable-demo-medcenter-transform=supertable.demo.medcenter.transform:main",
+            "supertable-demo-medcenter-quality=supertable.demo.medcenter.quality:main",
+            "supertable-demo-medcenter-export=supertable.demo.medcenter.export_accounting:main",
+            "supertable-demo-medcenter-run=supertable.demo.medcenter.run:main",
         ],
     },
 )

supertable-2.5.0/supertable/__init__.py

@@ -0,0 +1,112 @@
+"""SuperTable — versioned data lake library for SQL analytics.
+
+SuperTable stores structured data as versioned Parquet snapshots on object
+storage (S3, MinIO, Azure Blob, GCP Cloud Storage, or local disk), keeps
+metadata and locks in Redis, and queries everything through DuckDB or
+IslandDB, with Spark SQL for fleet-scale workloads.
+
+Quick reference
+---------------
+
+>>> from supertable import SuperTable, DataWriter, DataReader, engine
+>>> st = SuperTable(super_name="example", organization="my-org")
+>>> dw = DataWriter(super_name="example", organization="my-org")
+>>> _, _, inserted, deleted = dw.write(
+...     role_name="superadmin",
+...     simple_name="facts",
+...     data=arrow_table,
+...     overwrite_columns=["day"],
+... )
+>>> dr = DataReader(super_name="example", organization="my-org",
+...                 query="SELECT count(*) FROM facts")
+>>> df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)
+
+See the ``supertable.demo`` package for runnable end-to-end demos and the
+project documentation for the full API surface.
+"""
+
+__version__ = "2.5.0"
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+# This public alias predates the ``supertable.engine`` implementation package.
+# Importing the tiny enum after the now-lazy package is initialized ensures
+# later imports of engine submodules cannot replace ``supertable.engine`` with
+# the package object and break ``engine.AUTO``.
+from supertable.engine.engine_enum import Engine as engine
+
+
+# Importing the package is intentionally cheap and side-effect-free.  The
+# public convenience exports are loaded on first use (PEP 562) so importing
+# ``supertable`` does not import PyArrow/Polars, connect to Redis, probe the
+# filesystem, or initialise an application home.
+_LAZY_EXPORTS = {
+    "SuperTable": ("supertable.super_table", "SuperTable"),
+    "SimpleTable": ("supertable.simple_table", "SimpleTable"),
+    "DataWriter": ("supertable.data_writer", "DataWriter"),
+    "DataReader": ("supertable.data_reader", "DataReader"),
+    "MetaReader": ("supertable.meta_reader", "MetaReader"),
+    "list_supers": ("supertable.meta_reader", "list_supers"),
+    "list_tables": ("supertable.meta_reader", "list_tables"),
+    "Staging": ("supertable.staging_area", "Staging"),
+    "SuperPipe": ("supertable.super_pipe", "SuperPipe"),
+    "RedisCatalog": ("supertable.redis_catalog", "RedisCatalog"),
+    "RoleManager": ("supertable.rbac.role_manager", "RoleManager"),
+    "UserManager": ("supertable.rbac.user_manager", "UserManager"),
+    "SupertableLookupError": ("supertable.errors", "SupertableLookupError"),
+    "SuperTableNotFoundError": ("supertable.errors", "SuperTableNotFoundError"),
+    "TableNotFoundError": ("supertable.errors", "TableNotFoundError"),
+}
+
+
+def __getattr__(name: str) -> Any:
+    target = _LAZY_EXPORTS.get(name)
+    if target is None:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+    module_name, attribute = target
+    value = getattr(import_module(module_name), attribute)
+    globals()[name] = value
+    return value
+
+
+def __dir__() -> list[str]:
+    return sorted(set(globals()) | set(_LAZY_EXPORTS))
+
+
+if TYPE_CHECKING:
+    from supertable.data_reader import DataReader
+    from supertable.data_writer import DataWriter
+    from supertable.errors import (
+        SuperTableNotFoundError,
+        SupertableLookupError,
+        TableNotFoundError,
+    )
+    from supertable.meta_reader import MetaReader, list_supers, list_tables
+    from supertable.rbac.role_manager import RoleManager
+    from supertable.rbac.user_manager import UserManager
+    from supertable.redis_catalog import RedisCatalog
+    from supertable.simple_table import SimpleTable
+    from supertable.staging_area import Staging
+    from supertable.super_pipe import SuperPipe
+    from supertable.super_table import SuperTable
+
+__all__ = [
+    "__version__",
+    "SuperTable",
+    "SimpleTable",
+    "DataWriter",
+    "DataReader",
+    "engine",
+    "MetaReader",
+    "list_supers",
+    "list_tables",
+    "Staging",
+    "SuperPipe",
+    "RedisCatalog",
+    "RoleManager",
+    "UserManager",
+    "SupertableLookupError",
+    "SuperTableNotFoundError",
+    "TableNotFoundError",
+]

{supertable-2.4.0 → supertable-2.5.0}/supertable/audit/__init__.py

@@ -2,9 +2,16 @@
 """
 Audit logging module for Data Island Core.
 
-Provides an immutable, append-only audit trail for every security-relevant
-action. Dual-tier storage (Redis Streams + Parquet), tamper-evident hash
-chaining, and external SIEM integration via consumer groups.
+Provides two deliberately different audit lanes:
+
+* general security/telemetry events use the non-blocking, configurable logger;
+* RBAC role, user, and assignment mutations plus organization auth-token
+  create/delete use a mandatory Redis outbox committed atomically with the
+  state change and a verified Parquet archive. Token IDs and metadata digests,
+  never plaintext tokens, enter this ledger.
+
+The second lane is never disabled by general audit configuration and never
+silently drops a privileged state transition.
 
 Quick start:
 
@@ -48,6 +55,18 @@ from supertable.audit.logger import (
     get_audit_logger,
     shutdown_all,
 )
+from supertable.audit.privileged import (
+    PrivilegedActionContext,
+    PrivilegedAuditRecord,
+    build_record as build_privileged_record,
+    verify_records as verify_privileged_records,
+)
+from supertable.audit.privileged_outbox import (
+    CascadeEvidenceRow,
+    CascadeManifest,
+    PrivilegedAuditOutbox,
+    PrivilegedOutboxError,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -63,9 +82,47 @@ __all__ = [
     "ActorType",
     "Actions",
     "make_detail",
+    "PrivilegedActionContext",
+    "PrivilegedAuditRecord",
+    "PrivilegedAuditOutbox",
+    "CascadeEvidenceRow",
+    "CascadeManifest",
+    "PrivilegedOutboxError",
+    "build_privileged_record",
+    "verify_privileged_records",
+    "get_privileged_audit_outbox",
 ]
 
 
+def get_privileged_audit_outbox(
+    organization: str,
+    *,
+    redis_client=None,
+    storage=None,
+) -> PrivilegedAuditOutbox:
+    """Construct the mandatory privileged control-plane WAL reader/archiver.
+
+    This is intentionally separate from :func:`get_audit_logger`: runtime
+    audit configuration can disable the best-effort event logger, but it can
+    never disable or redirect the privileged mutation ledger.
+    """
+    if not organization:
+        raise ValueError("organization is required")
+    if redis_client is None:
+        from supertable.redis_connector import RedisConnector
+
+        redis_client = RedisConnector().r
+    from supertable import redis_keys as RK
+    from supertable.audit.writer_parquet import ParquetAuditWriter
+
+    return PrivilegedAuditOutbox(
+        redis_client,
+        stream_key=RK.audit_privileged_outbox(organization),
+        delivery_ledger_key=RK.audit_privileged_delivery(organization),
+        parquet_writer=ParquetAuditWriter(storage=storage),
+    )
+
+
 # ---------------------------------------------------------------------------
 # Convenience emit() — the primary public API
 # ---------------------------------------------------------------------------

{supertable-2.4.0 → supertable-2.5.0}/supertable/audit/events.py

@@ -51,6 +51,7 @@ class Outcome(str, Enum):
     SUCCESS = "success"
     FAILURE = "failure"
     DENIED = "denied"
+    NO_CHANGE = "no_change"
 
 
 class ActorType(str, Enum):