bqlens 0.1.0__tar.gz

bqlens-0.1.0/LICENSE

@@ -0,0 +1,15 @@
+Apache License 2.0
+
+Copyright 2026 bqlens contributors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

bqlens-0.1.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: bqlens
+Version: 0.1.0
+Summary: Find wasted BigQuery spend in one command. Read-only.
+License: Apache-2.0
+Keywords: bigquery,gcp,finops,cost-optimization,data-engineering
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: bigquery
+Requires-Dist: google-cloud-bigquery>=3.11; extra == "bigquery"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Dynamic: license-file
+
+# bqlens
+
+**Find wasted BigQuery spend in one command.**
+
+No signup. No sales call. No agent installed in your VPC. It reads your query
+history, tells you where the money is going, and tells you how to stop it.
+
+```bash
+pip install bqlens
+bqlens scan --project my-project --days 30
+```
+
+```
+  bqlens — my-project
+  1,833 query jobs over the last 30 days
+
+  Spend in window      $     1,946.10
+  Projected monthly    $     1,946.10
+  Recoverable          $       572.50/mo (29% of spend)
+
+  1. [HIGH] Wildcard table scans without _TABLE_SUFFIX  (R004)
+     $179.30/mo recoverable   ·  22 jobs
+     ...
+```
+
+Try it with no credentials at all:
+
+```bash
+bqlens scan --demo
+```
+
+---
+
+## What it looks at
+
+bqlens reads `INFORMATION_SCHEMA.JOBS_BY_PROJECT` — job metadata and the SQL
+text of your queries.
+
+**It never reads a row of your table data.** There is no row access, no
+sampling, no data leaving your project. The whole tool is a read of your own
+query log plus regex and arithmetic, running on your machine.
+
+## What it finds
+
+| Rule | What it catches | Why it costs money |
+|------|-----------------|--------------------|
+| R001 | `SELECT *` on large scans | BigQuery bills per column read |
+| R002 | `LIMIT` with no `WHERE` | LIMIT caps rows returned, not bytes scanned |
+| R003 | Large scans with no filter | No partition pruning |
+| R004 | Wildcard tables without `_TABLE_SUFFIX` | Scans every table in the dataset |
+| R005 | The same query shape run over and over | One materialised refresh replaces all of them |
+| R006 | Failed jobs that were still billed | Money that bought nothing |
+| R007 | Cross joins / joins missing `ON` | Row multiplication |
+
+## Every dollar is counted once
+
+A repeated `SELECT * ... LIMIT` matches three rules at once. If each rule
+counted it, the reported savings would add up to more than your actual bill —
+and the first thing a sceptical engineer does is add up our numbers.
+
+So rules run in priority order and each job is claimed by exactly one rule,
+the one with the most specific fix. The sum of findings can never exceed
+actual spend, and there is a test that fails the build if it ever does.
+
+Savings estimates are deliberately conservative. bqlens counts what a stated
+fix would plausibly recover, not the full cost of the query. If it says $500,
+the intent is that you find $500, not that you find $180 and stop trusting it.
+
+## Usage
+
+```bash
+# Basic scan
+bqlens scan --project my-project
+
+# Non-US region (the JOBS view is region-qualified)
+bqlens scan --project my-project --region region-asia-south1
+
+# Your negotiated rate rather than list price
+bqlens scan --project my-project --price-per-tib 5.00
+
+# A page you can forward to your manager
+bqlens scan --project my-project --format html --out report.html
+
+# Machine-readable
+bqlens scan --project my-project --format json
+
+# Fail CI if waste creeps above a threshold
+bqlens scan --project my-project --fail-over 200
+```
+
+### Options
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--project` | — | GCP project to scan (required unless `--demo`) |
+| `--region` | `region-us` | Region of the JOBS view |
+| `--days` | `30` | Days of history |
+| `--price-per-tib` | `6.25` | On-demand price per TiB scanned |
+| `--min-gib` | `1.0` | Ignore queries smaller than this |
+| `--min-repeats` | `10` | Repeats before a shape is flagged |
+| `--format` | `terminal` | `terminal`, `json`, or `html` |
+| `--out` | — | Write to a file |
+| `--fail-over` | — | Exit 1 if monthly recoverable exceeds this |
+| `--demo` | — | Run on synthetic data, no credentials |
+
+## Permissions
+
+The scanning account needs `bigquery.jobs.listAll` on the project — included
+in `roles/bigquery.resourceAdmin`, or grant it directly. Without `listAll` you
+will only see your own jobs, not the whole project's.
+
+The metadata query is itself capped at 10 GiB billed, so the scanner can never
+become the expensive thing in your bill.
+
+## Install
+
+```bash
+pip install bqlens              # CLI + demo mode
+pip install 'bqlens[bigquery]'  # adds the BigQuery client for real scans
+```
+
+Python 3.10+. Authenticate with `gcloud auth application-default login`.
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest
+```
+
+## Limitations, stated plainly
+
+- Rules work on query text and job metadata. bqlens does not read table schemas,
+  so it cannot know whether a table is partitioned — R003 flags unfiltered scans
+  as a *candidate*, not a certainty.
+- Savings percentages per rule are heuristics based on typical wide analytics
+  tables. Your mileage varies.
+- Slot-based (capacity) pricing is not yet modelled; cost figures assume
+  on-demand. Reservation support is planned.
+- SQL parsing is regex-based, not a full parser. Exotic queries may be missed.
+
+## Licence
+
+Apache-2.0

bqlens-0.1.0/README.md

@@ -0,0 +1,144 @@
+# bqlens
+
+**Find wasted BigQuery spend in one command.**
+
+No signup. No sales call. No agent installed in your VPC. It reads your query
+history, tells you where the money is going, and tells you how to stop it.
+
+```bash
+pip install bqlens
+bqlens scan --project my-project --days 30
+```
+
+```
+  bqlens — my-project
+  1,833 query jobs over the last 30 days
+
+  Spend in window      $     1,946.10
+  Projected monthly    $     1,946.10
+  Recoverable          $       572.50/mo (29% of spend)
+
+  1. [HIGH] Wildcard table scans without _TABLE_SUFFIX  (R004)
+     $179.30/mo recoverable   ·  22 jobs
+     ...
+```
+
+Try it with no credentials at all:
+
+```bash
+bqlens scan --demo
+```
+
+---
+
+## What it looks at
+
+bqlens reads `INFORMATION_SCHEMA.JOBS_BY_PROJECT` — job metadata and the SQL
+text of your queries.
+
+**It never reads a row of your table data.** There is no row access, no
+sampling, no data leaving your project. The whole tool is a read of your own
+query log plus regex and arithmetic, running on your machine.
+
+## What it finds
+
+| Rule | What it catches | Why it costs money |
+|------|-----------------|--------------------|
+| R001 | `SELECT *` on large scans | BigQuery bills per column read |
+| R002 | `LIMIT` with no `WHERE` | LIMIT caps rows returned, not bytes scanned |
+| R003 | Large scans with no filter | No partition pruning |
+| R004 | Wildcard tables without `_TABLE_SUFFIX` | Scans every table in the dataset |
+| R005 | The same query shape run over and over | One materialised refresh replaces all of them |
+| R006 | Failed jobs that were still billed | Money that bought nothing |
+| R007 | Cross joins / joins missing `ON` | Row multiplication |
+
+## Every dollar is counted once
+
+A repeated `SELECT * ... LIMIT` matches three rules at once. If each rule
+counted it, the reported savings would add up to more than your actual bill —
+and the first thing a sceptical engineer does is add up our numbers.
+
+So rules run in priority order and each job is claimed by exactly one rule,
+the one with the most specific fix. The sum of findings can never exceed
+actual spend, and there is a test that fails the build if it ever does.
+
+Savings estimates are deliberately conservative. bqlens counts what a stated
+fix would plausibly recover, not the full cost of the query. If it says $500,
+the intent is that you find $500, not that you find $180 and stop trusting it.
+
+## Usage
+
+```bash
+# Basic scan
+bqlens scan --project my-project
+
+# Non-US region (the JOBS view is region-qualified)
+bqlens scan --project my-project --region region-asia-south1
+
+# Your negotiated rate rather than list price
+bqlens scan --project my-project --price-per-tib 5.00
+
+# A page you can forward to your manager
+bqlens scan --project my-project --format html --out report.html
+
+# Machine-readable
+bqlens scan --project my-project --format json
+
+# Fail CI if waste creeps above a threshold
+bqlens scan --project my-project --fail-over 200
+```
+
+### Options
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--project` | — | GCP project to scan (required unless `--demo`) |
+| `--region` | `region-us` | Region of the JOBS view |
+| `--days` | `30` | Days of history |
+| `--price-per-tib` | `6.25` | On-demand price per TiB scanned |
+| `--min-gib` | `1.0` | Ignore queries smaller than this |
+| `--min-repeats` | `10` | Repeats before a shape is flagged |
+| `--format` | `terminal` | `terminal`, `json`, or `html` |
+| `--out` | — | Write to a file |
+| `--fail-over` | — | Exit 1 if monthly recoverable exceeds this |
+| `--demo` | — | Run on synthetic data, no credentials |
+
+## Permissions
+
+The scanning account needs `bigquery.jobs.listAll` on the project — included
+in `roles/bigquery.resourceAdmin`, or grant it directly. Without `listAll` you
+will only see your own jobs, not the whole project's.
+
+The metadata query is itself capped at 10 GiB billed, so the scanner can never
+become the expensive thing in your bill.
+
+## Install
+
+```bash
+pip install bqlens              # CLI + demo mode
+pip install 'bqlens[bigquery]'  # adds the BigQuery client for real scans
+```
+
+Python 3.10+. Authenticate with `gcloud auth application-default login`.
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest
+```
+
+## Limitations, stated plainly
+
+- Rules work on query text and job metadata. bqlens does not read table schemas,
+  so it cannot know whether a table is partitioned — R003 flags unfiltered scans
+  as a *candidate*, not a certainty.
+- Savings percentages per rule are heuristics based on typical wide analytics
+  tables. Your mileage varies.
+- Slot-based (capacity) pricing is not yet modelled; cost figures assume
+  on-demand. Reservation support is planned.
+- SQL parsing is regex-based, not a full parser. Exotic queries may be missed.
+
+## Licence
+
+Apache-2.0

bqlens-0.1.0/bqlens/__init__.py

@@ -0,0 +1,3 @@
+"""bqlens — find wasted BigQuery spend in one command."""
+
+__version__ = "0.1.0"

bqlens-0.1.0/bqlens/cli.py

@@ -0,0 +1,135 @@
+"""Command line interface for bqlens."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from . import __version__
+from .models import DEFAULT_PRICE_PER_TIB, ScanResult
+from .report import render_html, render_json, render_terminal
+from .rules import run_all
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="bqlens",
+        description="Find wasted BigQuery spend. Read-only; never touches your table data.",
+    )
+    parser.add_argument("--version", action="version", version=f"bqlens {__version__}")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    scan = sub.add_parser("scan", help="Scan query history for recoverable spend")
+    scan.add_argument("--project", help="GCP project ID to scan")
+    scan.add_argument(
+        "--region",
+        default="region-us",
+        help="BigQuery region of the JOBS view (default: region-us)",
+    )
+    scan.add_argument("--days", type=int, default=30, help="Days of history (default: 30)")
+    scan.add_argument(
+        "--price-per-tib",
+        type=float,
+        default=DEFAULT_PRICE_PER_TIB,
+        help=f"On-demand price per TiB scanned (default: {DEFAULT_PRICE_PER_TIB})",
+    )
+    scan.add_argument(
+        "--min-gib",
+        type=float,
+        default=1.0,
+        help="Ignore queries smaller than this many GiB (default: 1.0)",
+    )
+    scan.add_argument(
+        "--min-repeats",
+        type=int,
+        default=10,
+        help="Repeats before a query shape is flagged for materialisation (default: 10)",
+    )
+    scan.add_argument(
+        "--format",
+        choices=["terminal", "json", "html"],
+        default="terminal",
+        help="Output format (default: terminal)",
+    )
+    scan.add_argument("--out", help="Write output to this file instead of stdout")
+    scan.add_argument("--no-color", action="store_true", help="Disable ANSI colour")
+    scan.add_argument(
+        "--demo",
+        action="store_true",
+        help="Run against built-in synthetic history. No credentials needed.",
+    )
+    scan.add_argument(
+        "--fail-over",
+        type=float,
+        default=None,
+        metavar="USD",
+        help="Exit non-zero if monthly recoverable spend exceeds this. For CI.",
+    )
+    return parser
+
+
+def cmd_scan(args: argparse.Namespace) -> int:
+    if args.demo:
+        from .demo import generate
+
+        jobs = generate()
+        project = "demo-project"
+    else:
+        if not args.project:
+            print("error: --project is required (or use --demo)", file=sys.stderr)
+            return 2
+        from .fetch import fetch_jobs
+
+        jobs = fetch_jobs(project=args.project, region=args.region, days=args.days)
+        project = args.project
+
+    if not jobs:
+        print("No query jobs found in that window.", file=sys.stderr)
+        return 0
+
+    min_bytes = int(args.min_gib * 1024**3)
+    findings = run_all(
+        jobs,
+        price_per_tib=args.price_per_tib,
+        min_bytes=min_bytes,
+        min_repeats=args.min_repeats,
+    )
+
+    result = ScanResult(
+        findings=findings,
+        total_jobs=len(jobs),
+        total_spend_usd=sum(j.cost(args.price_per_tib) for j in jobs),
+        days=args.days,
+        project=project,
+        price_per_tib=args.price_per_tib,
+    )
+
+    if args.format == "json":
+        output = render_json(result)
+    elif args.format == "html":
+        output = render_html(result)
+    else:
+        colour = sys.stdout.isatty() and not args.no_color
+        output = render_terminal(result, colour=colour)
+
+    if args.out:
+        Path(args.out).write_text(output, encoding="utf-8")
+        print(f"Wrote {args.out}", file=sys.stderr)
+    else:
+        print(output)
+
+    if args.fail_over is not None and result.monthly_recoverable_usd > args.fail_over:
+        return 1
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    if args.command == "scan":
+        return cmd_scan(args)
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

bqlens-0.1.0/bqlens/demo.py

@@ -0,0 +1,188 @@
+"""Synthetic job history so the scanner can be tried without credentials.
+
+`bqlens scan --demo` runs the full pipeline against this. It is also what the
+test suite asserts against, so the numbers below are fixtures, not decoration.
+"""
+
+from __future__ import annotations
+
+import random
+from datetime import datetime, timedelta
+
+from .models import Job
+
+GIB = 1024**3
+
+_USERS = [
+    "analytics@example.com",
+    "dbt-runner@example.iam.gserviceaccount.com",
+    "looker@example.iam.gserviceaccount.com",
+    "priya@example.com",
+]
+
+
+def _job(
+    idx: int,
+    query: str,
+    gib: float,
+    user: str,
+    *,
+    cache_hit: bool = False,
+    error: str | None = None,
+    when: datetime | None = None,
+) -> Job:
+    billed = int(gib * GIB)
+    return Job(
+        job_id=f"demo_job_{idx:05d}",
+        user_email=user,
+        creation_time=when or (datetime.utcnow() - timedelta(hours=idx % 720)),
+        query=query,
+        total_bytes_billed=0 if cache_hit else billed,
+        total_bytes_processed=billed,
+        cache_hit=cache_hit,
+        statement_type="SELECT",
+        referenced_tables=["demo-project.warehouse.events"],
+        total_slot_ms=int(gib * 1200),
+        error_result=error,
+    )
+
+
+def generate(seed: int = 7) -> list[Job]:
+    """A plausible 30 days of query history for a mid-sized data team."""
+    rng = random.Random(seed)
+    jobs: list[Job] = []
+    idx = 0
+
+    # 1. A Looker dashboard hammering the same unaggregated query all month.
+    dashboard_sql = """
+        SELECT user_id, event_name, event_timestamp, platform, country,
+               session_id, revenue_usd
+        FROM `demo-project.warehouse.events`
+        WHERE event_date >= '2026-08-01'
+    """
+    for _ in range(280):
+        idx += 1
+        jobs.append(_job(idx, dashboard_sql, rng.uniform(7.5, 9.0), "looker@example.iam.gserviceaccount.com"))
+
+    # 2. GA4-style wildcard scans with no _TABLE_SUFFIX filter. The classic.
+    wildcard_sql = """
+        SELECT event_name, COUNT(*) AS n
+        FROM `demo-project.analytics_311.events_*`
+        GROUP BY event_name
+    """
+    for _ in range(22):
+        idx += 1
+        jobs.append(_job(idx, wildcard_sql, rng.uniform(140, 190), "analytics@example.com"))
+
+    # 3. Analysts exploring with SELECT * LIMIT, believing LIMIT is cheap.
+    for _ in range(46):
+        idx += 1
+        jobs.append(
+            _job(
+                idx,
+                "SELECT * FROM `demo-project.warehouse.events` LIMIT 100",
+                rng.uniform(58, 72),
+                "priya@example.com",
+            )
+        )
+
+    # 4. SELECT * feeding a transform that uses four columns.
+    for _ in range(35):
+        idx += 1
+        jobs.append(
+            _job(
+                idx,
+                "SELECT * FROM `demo-project.warehouse.transactions` "
+                "WHERE txn_date BETWEEN '2026-08-01' AND '2026-08-31'",
+                rng.uniform(18, 26),
+                "dbt-runner@example.iam.gserviceaccount.com",
+            )
+        )
+
+    # 5. A broken scheduled query retrying all month and billing every time.
+    for _ in range(64):
+        idx += 1
+        jobs.append(
+            _job(
+                idx,
+                "SELECT customer_id, SUM(amount) FROM `demo-project.warehouse.ledger` "
+                "WHERE posted_date >= '2026-08-01' GROUP BY 1",
+                rng.uniform(11, 15),
+                "dbt-runner@example.iam.gserviceaccount.com",
+                error="Not found: Table demo-project:warehouse.ledger_v2",
+            )
+        )
+
+    # 6. An unfiltered full scan somebody schedules nightly.
+    for _ in range(30):
+        idx += 1
+        jobs.append(
+            _job(
+                idx,
+                "SELECT customer_id, lifetime_value, segment "
+                "FROM `demo-project.warehouse.customer_360`",
+                rng.uniform(34, 44),
+                "analytics@example.com",
+            )
+        )
+
+    # 7. A join missing its ON clause.
+    for _ in range(6):
+        idx += 1
+        jobs.append(
+            _job(
+                idx,
+                "SELECT a.user_id, b.campaign_id "
+                "FROM `demo-project.warehouse.users` a "
+                "JOIN `demo-project.warehouse.campaigns` b "
+                "WHERE a.country = 'IN'",
+                rng.uniform(25, 33),
+                "priya@example.com",
+            )
+        )
+
+    # 8. Plenty of ordinary, well-written, cheap queries — the healthy majority.
+    for _ in range(900):
+        idx += 1
+        col = rng.choice(["event_name", "platform", "country"])
+        jobs.append(
+            _job(
+                idx,
+                f"SELECT {col}, COUNT(*) FROM `demo-project.warehouse.events` "
+                f"WHERE event_date = '2026-09-{rng.randint(1, 17):02d}' GROUP BY 1",
+                rng.uniform(0.05, 0.9),
+                rng.choice(_USERS),
+            )
+        )
+
+    # 9. Legitimate heavy analytical work: large, well-written, distinct
+    #    queries. This is the bulk of a healthy bill and nothing should flag
+    #    it. Without this block the demo would claim an absurd savings rate.
+    _dims = ["country", "platform", "channel", "device_type", "cohort", "tier", "region"]
+    _metrics = ["revenue_usd", "sessions", "orders", "refunds", "margin_usd", "units"]
+    _tables = ["transactions", "sessions_daily", "orders", "subscriptions", "inventory_daily"]
+    for i in range(300):
+        idx += 1
+        dim = _dims[i % len(_dims)]
+        metric = _metrics[(i // 7) % len(_metrics)]
+        table = _tables[(i // 3) % len(_tables)]
+        jobs.append(
+            _job(
+                idx,
+                f"SELECT {dim}, DATE_TRUNC(txn_date, MONTH) AS m, SUM({metric}) AS total_{i} "
+                f"FROM `demo-project.warehouse.{table}` t "
+                f"JOIN `demo-project.warehouse.dim_customer` c ON t.customer_id = c.customer_id "
+                f"WHERE txn_date >= '2026-0{rng.randint(1, 8)}-01' "
+                f"GROUP BY 1, 2",
+                rng.uniform(40, 90),
+                rng.choice(_USERS),
+            )
+        )
+
+    # 10. Cache hits — free, and correctly ignored by every rule.
+    for _ in range(150):
+        idx += 1
+        jobs.append(_job(idx, dashboard_sql, 8.0, "looker@example.iam.gserviceaccount.com", cache_hit=True))
+
+    rng.shuffle(jobs)
+    return jobs