ducklens 0.1.0__tar.gz

ducklens-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+.venv/
+__pycache__/
+*.pyc
+*.duckdb
+*.duckdb.wal
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.DS_Store

ducklens-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Munim Zafar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ducklens-0.1.0/PKG-INFO

@@ -0,0 +1,132 @@
+Metadata-Version: 2.4
+Name: ducklens
+Version: 0.1.0
+Summary: Score how much of your Snowflake or BigQuery bill fits on one DuckDB machine
+Project-URL: Repository, https://github.com/munimdev/ducklens
+Author: Munim Zafar
+License-Expression: MIT
+License-File: LICENSE
+Keywords: bigquery,cost,duckdb,finops,migration,snowflake
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: duckdb>=1.5.0
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: pytz>=2024.1
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Provides-Extra: bigquery
+Requires-Dist: google-cloud-bigquery>=3.0; extra == 'bigquery'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: snowflake
+Requires-Dist: snowflake-connector-python>=3.0; extra == 'snowflake'
+Description-Content-Type: text/markdown
+
+# ducklens
+
+ducklens reads your Snowflake or BigQuery query history and works out how much of the bill could run on a single DuckDB machine. It scores every query, rolls the scores up per warehouse into a move, split, or keep decision, and reconciles the totals to your metered invoice.
+
+This is worth measuring because DuckDB is an out-of-core engine. A query that scans two terabytes but streams through a filter and an aggregate runs fine on a box that holds a few gigabytes in memory. What breaks a single machine is the working set, not the scan. So the real question is whether a query spilled, and ducklens answers it from observed spill instead of guessing from scan size.
+
+The scorer is one SQL file, `ducklens/scoring.sql`. Every threshold is a named key you can override. Read it, disagree with a number, change it, and re-run.
+
+## Install
+
+```bash
+uv venv -p 3.12 .venv
+uv pip install --python .venv/bin/python -e '.[dev]'
+.venv/bin/ducklens --help
+```
+
+## Try it without an account
+
+`demo` generates synthetic history and runs a full audit:
+
+```bash
+ducklens demo
+ducklens demo --source bigquery
+```
+
+For a real analytical workload, `scripts/tpch_to_history.py` runs TPC-H locally, captures the actual execution times and scan footprints, and replays them across a few warehouses:
+
+```bash
+python scripts/tpch_to_history.py --sf 8 --days 90 --ram-gb 8 --out ./tpch
+ducklens audit --source snowflake \
+  --query-history ./tpch/query_history.parquet \
+  --metering ./tpch/warehouse_metering_history.parquet \
+  --metering-daily ./tpch/metering_daily_history.parquet \
+  --ram-gb 8 --db audit.duckdb
+```
+
+Example output:
+
+```
+43% of your total Snowflake bill is movable query compute
+    = $31,379/yr of $72,178/yr invoice
+
+GROSS ANNUAL RUN-RATE DELTA   $18,410 - $24,908
+
+HYBRID SPLIT
+  BI_SERVING_WH   100%   $19,769   MOVE
+  DBT_WH           78%   $14,386   SPLIT
+  AD_HOC_WH         6%    $6,804    KEEP
+```
+
+## Audit your own account
+
+You run a read-only export and ducklens reads the files locally. Nothing leaves your machine and it never sees a credential. The export SQL is in `ducklens/export_sql/`. `snowflake_export.sql` copies the three Account Usage views to a temp stage as parquet, with a switch to drop query text if you would rather not share it.
+
+```bash
+ducklens audit --source snowflake \
+  --query-history 'query_history*.parquet' \
+  --metering 'warehouse_metering*.parquet' \
+  --metering-daily 'metering_daily*.parquet' \
+  --db audit.duckdb
+
+ducklens audit ... --format html -o report.html
+ducklens explain <query_id> --db audit.duckdb
+```
+
+`ducklens pull` runs the read-only export for you if you would rather hand it credentials. It needs the `[snowflake]` or `[bigquery]` extra.
+
+## How it scores
+
+A query fits unless a flag says otherwise. The flags, in priority order:
+
+- remote spill, or local spill past the box's memory
+- warehouse-specific SQL that would not port
+- sustained high concurrency, which is a serving workload rather than a batch one
+- multi-cluster scale-out
+- long queue times
+- stored procedures, multi-statement write transactions, and high-frequency writes
+
+Spill leads and scan size never blocks a query on its own. Concurrency counts only sustained overlap: sixteen 200ms dashboard pings score zero, while sixteen overlapping 30-second queries score sixteen. Each blocked query is attributed to a single flag, so the residual dollars never double-count.
+
+Cost comes from your metered credits, spread across queries by runtime and calibrated so the per-query numbers sum back to what you were billed. The headline anchors to `METERING_DAILY_HISTORY`. Idle warehouse time and serverless spend are separate lines, never folded into the movable number. On BigQuery the model switches to bytes billed.
+
+The report prints the movable share of the bill, the per-warehouse split, the costliest queries keeping each warehouse in place, a saving range, and a table of how fit changes with box size. `--format` gives you rich, html, markdown, or json.
+
+## On real data
+
+Snowset is a public trace of about 70 million real Snowflake queries, released with the NSDI 2020 paper on Snowflake's architecture. It records real spill bytes per query, which is what the scorer needs. On a 5.8 million query sample across 1,290 warehouses, ducklens scores in about 20 seconds and puts 81% of the query compute in the single-machine range, holding the genuinely spill-heavy warehouses back. `ducklens/export_sql/snowset_to_history.sql` maps the trace onto the input schema so you can reproduce this.
+
+## When it says stay
+
+The report is built to talk you out of a migration when the numbers do not hold. Below roughly a $40k/yr bill, the machine and object storage cost more than you save. Serving workloads, sustained writes, multi-petabyte estates, and regulated governance surfaces stay on the warehouse. The report says so per warehouse, with the dollars attached.
+
+## Development
+
+```bash
+.venv/bin/pytest
+```
+
+## License
+
+MIT.

ducklens-0.1.0/README.md

@@ -0,0 +1,101 @@
+# ducklens
+
+ducklens reads your Snowflake or BigQuery query history and works out how much of the bill could run on a single DuckDB machine. It scores every query, rolls the scores up per warehouse into a move, split, or keep decision, and reconciles the totals to your metered invoice.
+
+This is worth measuring because DuckDB is an out-of-core engine. A query that scans two terabytes but streams through a filter and an aggregate runs fine on a box that holds a few gigabytes in memory. What breaks a single machine is the working set, not the scan. So the real question is whether a query spilled, and ducklens answers it from observed spill instead of guessing from scan size.
+
+The scorer is one SQL file, `ducklens/scoring.sql`. Every threshold is a named key you can override. Read it, disagree with a number, change it, and re-run.
+
+## Install
+
+```bash
+uv venv -p 3.12 .venv
+uv pip install --python .venv/bin/python -e '.[dev]'
+.venv/bin/ducklens --help
+```
+
+## Try it without an account
+
+`demo` generates synthetic history and runs a full audit:
+
+```bash
+ducklens demo
+ducklens demo --source bigquery
+```
+
+For a real analytical workload, `scripts/tpch_to_history.py` runs TPC-H locally, captures the actual execution times and scan footprints, and replays them across a few warehouses:
+
+```bash
+python scripts/tpch_to_history.py --sf 8 --days 90 --ram-gb 8 --out ./tpch
+ducklens audit --source snowflake \
+  --query-history ./tpch/query_history.parquet \
+  --metering ./tpch/warehouse_metering_history.parquet \
+  --metering-daily ./tpch/metering_daily_history.parquet \
+  --ram-gb 8 --db audit.duckdb
+```
+
+Example output:
+
+```
+43% of your total Snowflake bill is movable query compute
+    = $31,379/yr of $72,178/yr invoice
+
+GROSS ANNUAL RUN-RATE DELTA   $18,410 - $24,908
+
+HYBRID SPLIT
+  BI_SERVING_WH   100%   $19,769   MOVE
+  DBT_WH           78%   $14,386   SPLIT
+  AD_HOC_WH         6%    $6,804    KEEP
+```
+
+## Audit your own account
+
+You run a read-only export and ducklens reads the files locally. Nothing leaves your machine and it never sees a credential. The export SQL is in `ducklens/export_sql/`. `snowflake_export.sql` copies the three Account Usage views to a temp stage as parquet, with a switch to drop query text if you would rather not share it.
+
+```bash
+ducklens audit --source snowflake \
+  --query-history 'query_history*.parquet' \
+  --metering 'warehouse_metering*.parquet' \
+  --metering-daily 'metering_daily*.parquet' \
+  --db audit.duckdb
+
+ducklens audit ... --format html -o report.html
+ducklens explain <query_id> --db audit.duckdb
+```
+
+`ducklens pull` runs the read-only export for you if you would rather hand it credentials. It needs the `[snowflake]` or `[bigquery]` extra.
+
+## How it scores
+
+A query fits unless a flag says otherwise. The flags, in priority order:
+
+- remote spill, or local spill past the box's memory
+- warehouse-specific SQL that would not port
+- sustained high concurrency, which is a serving workload rather than a batch one
+- multi-cluster scale-out
+- long queue times
+- stored procedures, multi-statement write transactions, and high-frequency writes
+
+Spill leads and scan size never blocks a query on its own. Concurrency counts only sustained overlap: sixteen 200ms dashboard pings score zero, while sixteen overlapping 30-second queries score sixteen. Each blocked query is attributed to a single flag, so the residual dollars never double-count.
+
+Cost comes from your metered credits, spread across queries by runtime and calibrated so the per-query numbers sum back to what you were billed. The headline anchors to `METERING_DAILY_HISTORY`. Idle warehouse time and serverless spend are separate lines, never folded into the movable number. On BigQuery the model switches to bytes billed.
+
+The report prints the movable share of the bill, the per-warehouse split, the costliest queries keeping each warehouse in place, a saving range, and a table of how fit changes with box size. `--format` gives you rich, html, markdown, or json.
+
+## On real data
+
+Snowset is a public trace of about 70 million real Snowflake queries, released with the NSDI 2020 paper on Snowflake's architecture. It records real spill bytes per query, which is what the scorer needs. On a 5.8 million query sample across 1,290 warehouses, ducklens scores in about 20 seconds and puts 81% of the query compute in the single-machine range, holding the genuinely spill-heavy warehouses back. `ducklens/export_sql/snowset_to_history.sql` maps the trace onto the input schema so you can reproduce this.
+
+## When it says stay
+
+The report is built to talk you out of a migration when the numbers do not hold. Below roughly a $40k/yr bill, the machine and object storage cost more than you save. Serving workloads, sustained writes, multi-petabyte estates, and regulated governance surfaces stay on the warehouse. The report says so per warehouse, with the dollars attached.
+
+## Development
+
+```bash
+.venv/bin/pytest
+```
+
+## License
+
+MIT.

ducklens-0.1.0/ducklens/__init__.py

@@ -0,0 +1,6 @@
+"""ducklens: see what fits.
+
+DuckDB migration fit assessment for Snowflake & BigQuery warehouse spend.
+"""
+
+__version__ = "0.1.0"

ducklens-0.1.0/ducklens/cli.py

@@ -0,0 +1,138 @@
+"""ducklens CLI: the four commands audit, pull, demo, explain.
+
+This module owns the command-line surface: flags, help text, and exit codes.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from pathlib import Path
+
+import typer
+
+from . import __version__
+
+app = typer.Typer(
+    add_completion=False,
+    no_args_is_help=True,
+    help=(
+        "Assess what percentage of your Snowflake or BigQuery spend fits on one "
+        "DuckDB machine."
+    ),
+)
+
+
+class Source(str, Enum):
+    snowflake = "snowflake"
+    bigquery = "bigquery"
+
+
+class Edition(str, Enum):
+    standard = "standard"
+    enterprise = "enterprise"
+    business_critical = "business_critical"
+
+
+class OutFormat(str, Enum):
+    rich = "rich"
+    json = "json"
+    md = "md"
+    html = "html"
+
+
+class DemoProfile(str, Enum):
+    balanced = "balanced"
+    serving_heavy = "serving-heavy"
+    batch_heavy = "batch-heavy"
+    already_small = "already-small"
+
+
+def _version_callback(value: bool) -> None:
+    if value:
+        typer.echo(f"ducklens {__version__}")
+        raise typer.Exit()
+
+
+@app.callback()
+def _root(
+    _version: bool = typer.Option(
+        False, "--version", callback=_version_callback, is_eager=True, help="Show version and exit."
+    ),
+) -> None:
+    """See what fits on one DuckDB machine."""
+
+
+@app.command()
+def audit(
+    source: Source = typer.Option(None, help="Warehouse source (auto-detected from columns if omitted)."),
+    query_history: Path = typer.Option(None, help="Snowflake QUERY_HISTORY export (parquet/csv; globs ok)."),
+    metering: Path = typer.Option(None, help="Snowflake WAREHOUSE_METERING_HISTORY export."),
+    metering_daily: Path = typer.Option(None, help="Snowflake METERING_DAILY_HISTORY export (invoice anchor)."),
+    jobs: Path = typer.Option(None, help="BigQuery JOBS_BY_PROJECT export (parquet preferred)."),
+    edition: Edition = typer.Option(None, help="Snowflake edition, sets the $/credit rate (default enterprise)."),
+    credit_rate: float = typer.Option(None, help="Override $/credit with your effective negotiated rate."),
+    ram_gb: int = typer.Option(None, help="Target DuckDB machine RAM in GB (default 128)."),
+    concurrency_max: int = typer.Option(None, help="Concurrency above which a workload is 'serving' (default 16)."),
+    machine_monthly: float = typer.Option(None, help="Assumed DuckDB box $/month (default 800)."),
+    migration_cost: float = typer.Option(None, help="One-time migration cost in dollars, used to compute the payback period."),
+    config: Path = typer.Option(None, exists=True, help="TOML overriding any scoring threshold (CLI flags win when set)."),
+    fmt: OutFormat = typer.Option(OutFormat.rich, "--format", help="Output format."),
+    db: Path = typer.Option(None, help="Persist the scored DuckDB database for follow-up querying."),
+    output: Path = typer.Option(None, "--output", "-o", help="Write the report to a file."),
+) -> None:
+    """Score a warehouse usage export and print the one-page fit report."""
+    from .engine import run_audit  # deferred import keeps --help fast
+
+    run_audit(locals())
+
+
+@app.command()
+def demo(
+    source: Source = typer.Option(Source.snowflake, help="Warehouse model to simulate (snowflake credits or bigquery bytes-billed)."),
+    queries: int = typer.Option(25_000, help="Number of synthetic queries to generate."),
+    days: int = typer.Option(90, help="Days of synthetic history (>=1 month-end recommended)."),
+    seed: int = typer.Option(7, help="RNG seed for deterministic output."),
+    profile: DemoProfile = typer.Option(DemoProfile.balanced, help="Workload shape to simulate (snowflake only)."),
+    fmt: OutFormat = typer.Option(OutFormat.rich, "--format", help="Output format."),
+    db: Path = typer.Option(None, help="Persist the scored DuckDB database (enables `ducklens explain`)."),
+    output: Path = typer.Option(None, "--output", "-o", help="Write the report to a file."),
+) -> None:
+    """Generate synthetic usage and run a full audit, no warehouse account needed."""
+    from .engine import run_demo
+
+    run_demo(locals())
+
+
+@app.command()
+def pull(
+    source: Source = typer.Argument(..., help="Warehouse to export from (read-only)."),
+    out_dir: Path = typer.Option(Path("./export"), help="Where to write the exported parquet files."),
+    account: str = typer.Option(None, help="Snowflake account identifier (or BigQuery project)."),
+    user: str = typer.Option(None, help="Snowflake user (password via $SNOWFLAKE_PASSWORD)."),
+    role: str = typer.Option(None, help="Snowflake role with ACCOUNT_USAGE privileges."),
+    region: str = typer.Option(None, help="BigQuery region (INFORMATION_SCHEMA.JOBS is region-scoped)."),
+    days: int = typer.Option(90, help="Window to export."),
+) -> None:
+    """Run the usage export yourself (read-only) and write parquet you can hand off.
+
+    Live mode needs your OWN credentials and the connector extra
+    (pip install ducklens with the snowflake / bigquery extra); ducklens never sees them.
+    """
+    from .engine import run_pull  # deferred import keeps --help fast and avoids the connector
+
+    run_pull(locals())
+
+
+@app.command()
+def explain(
+    query_id: str = typer.Argument(..., help="The query_id to explain."),
+    db: Path = typer.Option(..., exists=True, help="A scored DuckDB database (from `audit --db`)."),
+) -> None:
+    """Explain why one query was scored as fitting / not fitting."""
+    from .engine import run_explain
+
+    run_explain(query_id, db)
+
+
+if __name__ == "__main__":
+    app()

ducklens-0.1.0/ducklens/config.py

@@ -0,0 +1,95 @@
+"""Scoring configuration. Every threshold is a named, overridable key.
+
+`Config.setup(con)` materializes a one-row `config` table and a `wh_size_credits` table in
+DuckDB, so `scoring.sql` references thresholds by name and holds no magic constants. Read the
+SQL and re-run with `--config thresholds.toml` to change any of them.
+"""
+
+from __future__ import annotations
+
+import tomllib
+from dataclasses import dataclass, fields
+from pathlib import Path
+
+import duckdb
+
+EDITION_CREDIT_RATE = {"standard": 2.0, "enterprise": 3.0, "business_critical": 4.0}
+
+# Snowflake Gen1 warehouse size to credits/hour (doubling table).
+SIZE_CREDITS_HR = {
+    "XS": 1, "S": 2, "M": 4, "L": 8, "XL": 16,
+    "2XL": 32, "3XL": 64, "4XL": 128, "5XL": 256, "6XL": 512,
+}
+
+GB = 1_000_000_000
+
+
+@dataclass
+class Config:
+    # cost
+    edition: str = "enterprise"
+    credit_rate: float | None = None        # override $/credit (effective negotiated rate)
+    machine_monthly: float = 800.0          # assumed DuckDB box $/month
+    scan_per_tib: float = 0.50              # object-storage scan $/TiB (conservative)
+    # bigquery
+    bq_per_tib: float = 6.25
+    bq_free_tib_per_month: float = 1.0
+    # fit thresholds
+    machine_ram_gb: int = 128
+    ram_headroom: float = 0.6               # working set tolerated = ram * headroom
+    concurrency_max: float = 16.0           # sustained queries above which a workload is serving
+    concurrency_min_ms: float = 1000.0      # only queries this long count toward concurrency
+    overload_frac: float = 0.2
+    write_rate_per_hr: float = 120.0
+    # warehouse roll-up
+    warehouse_move_pct: float = 0.80
+    warehouse_split_pct: float = 0.40
+    # one-time migration cost (FTE-months x loaded rate, or a flat dollar amount) for payback
+    migration_cost: float | None = None
+
+    @property
+    def resolved_credit_rate(self) -> float:
+        if self.credit_rate is not None:
+            return self.credit_rate
+        return EDITION_CREDIT_RATE[self.edition]
+
+    @classmethod
+    def load(cls, toml_path: str | Path | None = None, **overrides) -> Config:
+        data: dict = {}
+        if toml_path:
+            data = tomllib.loads(Path(toml_path).read_text())
+        data.update({k: v for k, v in overrides.items() if v is not None})
+        known = {f.name for f in fields(cls)}
+        unknown = set(data) - known
+        if unknown:
+            raise ValueError(f"unknown config keys: {sorted(unknown)}")
+        return cls(**{k: v for k, v in data.items() if k in known})
+
+    def setup(self, con: duckdb.DuckDBPyConnection) -> None:
+        """Materialize `config` (one row) and `wh_size_credits` in the connection."""
+        # All date_trunc/epoch math must be UTC so query timestamps align to the UTC-hour
+        # metering buckets, or +5:30/+5:45 sessions silently miss the metering join.
+        con.execute("SET TimeZone='UTC'")
+        cols = {
+            "credit_rate": self.resolved_credit_rate,
+            "machine_monthly": self.machine_monthly,
+            "scan_per_tib": self.scan_per_tib,
+            "bq_per_tib": self.bq_per_tib,
+            "bq_free_tib_per_month": self.bq_free_tib_per_month,
+            "ram_bytes": self.machine_ram_gb * GB,
+            "ram_headroom": self.ram_headroom,
+            "concurrency_max": self.concurrency_max,
+            "concurrency_min_ms": self.concurrency_min_ms,
+            "overload_frac": self.overload_frac,
+            "write_rate_per_hr": self.write_rate_per_hr,
+            "warehouse_move_pct": self.warehouse_move_pct,
+            "warehouse_split_pct": self.warehouse_split_pct,
+        }
+        select = ", ".join(f"{float(v)} AS {k}" for k, v in cols.items())
+        con.execute(f"CREATE OR REPLACE TABLE config AS SELECT {select}")
+
+        con.execute("CREATE OR REPLACE TABLE wh_size_credits(size VARCHAR, credits_hr DOUBLE)")
+        con.executemany(
+            "INSERT INTO wh_size_credits VALUES (?, ?)",
+            [(s, float(c)) for s, c in SIZE_CREDITS_HR.items()],
+        )