bqcost 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +119 -0
- package/dist/cli.js +504 -0
- package/package.json +52 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Denis Gryazev
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# bqcost
|
|
2
|
+
|
|
3
|
+
**Catch the BigQuery query that would've cost $10k — in your PR.**
|
|
4
|
+
|
|
5
|
+
`sqlfluff` catches your style. **bqcost catches the query that scans 3 TB.** It dry-runs every
|
|
6
|
+
changed `.sql`/dbt model, prices the scan, flags the cost **regression vs your base branch**, and
|
|
7
|
+
comments the number on the PR — before you merge.
|
|
8
|
+
|
|
9
|
+
> BigQuery bills by bytes scanned. One missing partition filter turns a $0.02 query into a $20
|
|
10
|
+
> one (or a $10k one). bqcost surfaces that in review, not on the invoice.
|
|
11
|
+
|
|
12
|
+

|
|
13
|
+
|
|
14
|
+
<sub>Prefer an animated GIF? `vhs docs/demo.tape` renders one from a real run.</sub>
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Quickstart (≈30 seconds)
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
# authenticate once (Application Default Credentials)
|
|
22
|
+
gcloud auth application-default login
|
|
23
|
+
|
|
24
|
+
# lint a file or a directory of .sql
|
|
25
|
+
npx bqcost lint ./models
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
bqcost: scanned 1 file(s), est. $20.20, 2 finding(s)
|
|
30
|
+
|
|
31
|
+
examples/big-scan.sql ($20.20, 3310.3 GB)
|
|
32
|
+
[R2] scans 3310.3 GB ($20.20) of partitioned table bigquery-public-data.crypto_ethereum.transactions — check the partition filter
|
|
33
|
+
fix: narrow the filter on partition column `block_timestamp` (e.g. WHERE block_timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
|
|
34
|
+
[R1] SELECT * — reads every column (3310.3 GB)
|
|
35
|
+
fix: project only the columns you need instead of *
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
That's a real dry-run against a real table — **$20.20, 3.3 TB**, caught before it ran.
|
|
39
|
+
|
|
40
|
+
## In CI — the cost gate on every PR
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
# .github/workflows/bqcost.yml
|
|
44
|
+
name: bqcost
|
|
45
|
+
on: pull_request
|
|
46
|
+
jobs:
|
|
47
|
+
cost:
|
|
48
|
+
runs-on: ubuntu-latest
|
|
49
|
+
permissions: { contents: read, pull-requests: write }
|
|
50
|
+
steps:
|
|
51
|
+
- uses: actions/checkout@v4
|
|
52
|
+
with: { fetch-depth: 0 } # bqcost needs base..head
|
|
53
|
+
- uses: google-github-actions/auth@v2 # your GCP service account
|
|
54
|
+
with: { credentials_json: '${{ secrets.GCP_SA_KEY }}' }
|
|
55
|
+
- uses: dagryazev/bqcost@v1 # dry-runs changed .sql, posts a sticky PR comment
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
On each PR bqcost posts (and updates) one comment: cost per changed file, **Δ vs base**, and the
|
|
59
|
+
findings. It fails the check when a change adds more than your threshold (default $1).
|
|
60
|
+
|
|
61
|
+
## Pre-commit
|
|
62
|
+
|
|
63
|
+
```yaml
|
|
64
|
+
# .pre-commit-config.yaml
|
|
65
|
+
repos:
|
|
66
|
+
- repo: https://github.com/dagryazev/bqcost
|
|
67
|
+
rev: v1
|
|
68
|
+
hooks:
|
|
69
|
+
- id: bqcost
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## What it checks
|
|
73
|
+
|
|
74
|
+
| rule | flags | how |
|
|
75
|
+
|------|-------|-----|
|
|
76
|
+
| **R2** | a big scan of a **partitioned** table (the missing partition filter) | **measured** — real dry-run bytes, not a static guess |
|
|
77
|
+
| **R1** | `SELECT *` on a table that actually costs | AST + cost |
|
|
78
|
+
| **R4** | `CROSS JOIN` fan-out | AST |
|
|
79
|
+
| **R5** | `ORDER BY` without `LIMIT` | AST |
|
|
80
|
+
|
|
81
|
+
R2 is measured on purpose: BigQuery's optimizer prunes many predicates that "look" unprunable
|
|
82
|
+
(`DATE(ts) = …`, `EXTRACT(YEAR …)`), so a static "function wrap ⇒ no pruning" rule would cry wolf.
|
|
83
|
+
bqcost reads what the query **actually** scans instead. (See `docs/decisions.md`.)
|
|
84
|
+
|
|
85
|
+
## Configuration — `.bqcost.yml`
|
|
86
|
+
|
|
87
|
+
```yaml
|
|
88
|
+
pricePerTiB: 6.25 # on-demand $/TiB (US)
|
|
89
|
+
regression:
|
|
90
|
+
absUsd: 1 # fail a PR when a file adds > $1
|
|
91
|
+
rules:
|
|
92
|
+
r1: { minBytes: 1073741824 } # 1 GiB — below this, SELECT * is quiet
|
|
93
|
+
r2: { minScanBytes: 107374182400 } # 100 GiB — flag scans bigger than this
|
|
94
|
+
allowlist: ["reporting.big_daily_rollup"] # skip known-huge tables
|
|
95
|
+
disabledRules: ["R5"]
|
|
96
|
+
dbt: { manifestPath: target/manifest.json } # lint compiled SQL, not Jinja
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Silence one query inline:
|
|
100
|
+
|
|
101
|
+
```sql
|
|
102
|
+
SELECT * FROM `p.d.events` -- bqcost:ignore (all rules)
|
|
103
|
+
SELECT * FROM `p.d.events` -- bqcost:ignore R1 (just R1)
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
## Is it safe? Yes — dry-run only
|
|
107
|
+
|
|
108
|
+
bqcost uses BigQuery **dry-run**: it validates and prices the query **without executing it and
|
|
109
|
+
without reading a single row**. Dry-runs are free and don't consume slots. Give it a service
|
|
110
|
+
account with `roles/bigquery.jobUser` + `roles/bigquery.dataViewer` (read-only) and nothing more.
|
|
111
|
+
|
|
112
|
+
## dbt
|
|
113
|
+
|
|
114
|
+
Point `dbt.manifestPath` at your `target/manifest.json` (after `dbt compile`) and bqcost lints the
|
|
115
|
+
**compiled** SQL, so `{{ ref() }}` is resolved to real tables.
|
|
116
|
+
|
|
117
|
+
## License
|
|
118
|
+
|
|
119
|
+
MIT.
|
package/dist/cli.js
ADDED
|
@@ -0,0 +1,504 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
// src/cli.ts
|
|
4
|
+
import { Command } from "commander";
|
|
5
|
+
|
|
6
|
+
// src/core/config.ts
|
|
7
|
+
import { z } from "zod";
|
|
8
|
+
import { readFileSync, existsSync } from "fs";
|
|
9
|
+
import { join } from "path";
|
|
10
|
+
import { parse as parseYaml } from "yaml";
|
|
11
|
+
var ConfigSchema = z.object({
|
|
12
|
+
pricePerTiB: z.number().positive().default(6.25),
|
|
13
|
+
currency: z.string().default("USD"),
|
|
14
|
+
paths: z.array(z.string()).default(["**/*.sql"]),
|
|
15
|
+
/** Substrings of table refs to skip entirely (e.g. a known-huge reporting table). */
|
|
16
|
+
allowlist: z.array(z.string()).default([]),
|
|
17
|
+
/** Rule ids to turn off (e.g. ["R5"]). */
|
|
18
|
+
disabledRules: z.array(z.string()).default([]),
|
|
19
|
+
/** Cost-regression gate vs the base branch. */
|
|
20
|
+
regression: z.object({ absUsd: z.number().default(1) }).default({}),
|
|
21
|
+
/** dbt: point at a compiled manifest.json to lint compiled SQL instead of Jinja source. */
|
|
22
|
+
dbt: z.object({ manifestPath: z.string().optional() }).default({}),
|
|
23
|
+
rules: z.object({
|
|
24
|
+
r1: z.object({ minBytes: z.number().default(1 * 2 ** 30) }).default({}),
|
|
25
|
+
r2: z.object({ minScanBytes: z.number().default(100 * 2 ** 30) }).default({})
|
|
26
|
+
}).default({})
|
|
27
|
+
});
|
|
28
|
+
function defaultConfig() {
|
|
29
|
+
return ConfigSchema.parse({});
|
|
30
|
+
}
|
|
31
|
+
function loadConfig(cwd = process.cwd()) {
|
|
32
|
+
const path = join(cwd, ".bqcost.yml");
|
|
33
|
+
if (!existsSync(path)) return defaultConfig();
|
|
34
|
+
return ConfigSchema.parse(parseYaml(readFileSync(path, "utf8")) ?? {});
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// src/run.ts
|
|
38
|
+
import { existsSync as existsSync2 } from "fs";
|
|
39
|
+
|
|
40
|
+
// src/core/extract.ts
|
|
41
|
+
import { readFileSync as readFileSync3, readdirSync, statSync } from "fs";
|
|
42
|
+
import { join as join2, extname } from "path";
|
|
43
|
+
|
|
44
|
+
// src/core/parse.ts
|
|
45
|
+
import pkg from "node-sql-parser";
|
|
46
|
+
var { Parser } = pkg;
|
|
47
|
+
var parser = new Parser();
|
|
48
|
+
var OPT = { database: "BigQuery" };
|
|
49
|
+
function tableRefs(sql) {
|
|
50
|
+
try {
|
|
51
|
+
const list = parser.tableList(sql, OPT);
|
|
52
|
+
return list.map((t) => t.split("::").slice(1).filter((x) => x && x !== "null").join(".")).filter(Boolean);
|
|
53
|
+
} catch {
|
|
54
|
+
return [];
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
function parse(sql) {
|
|
58
|
+
try {
|
|
59
|
+
const ast = parser.astify(sql, OPT);
|
|
60
|
+
return { ast, tables: tableRefs(sql) };
|
|
61
|
+
} catch (e) {
|
|
62
|
+
return { parseError: (e?.message ?? String(e)).split("\n")[0], tables: [] };
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
// src/adapters/dbt.ts
|
|
67
|
+
import { readFileSync as readFileSync2 } from "fs";
|
|
68
|
+
function loadCompiledSql(manifestPath) {
|
|
69
|
+
const manifest = JSON.parse(readFileSync2(manifestPath, "utf8"));
|
|
70
|
+
const out = /* @__PURE__ */ new Map();
|
|
71
|
+
for (const node of Object.values(manifest.nodes ?? {})) {
|
|
72
|
+
const sql = node.compiled_code ?? node.raw_code;
|
|
73
|
+
if (node.original_file_path && sql) out.set(node.original_file_path, sql);
|
|
74
|
+
}
|
|
75
|
+
return out;
|
|
76
|
+
}
|
|
77
|
+
function compiledFor(file, compiled) {
|
|
78
|
+
const norm = file.replace(/\\/g, "/");
|
|
79
|
+
for (const [key, sql] of compiled) {
|
|
80
|
+
if (norm === key || norm.endsWith("/" + key) || norm.endsWith(key)) return sql;
|
|
81
|
+
}
|
|
82
|
+
return void 0;
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
// src/core/extract.ts
|
|
86
|
+
function findSqlFiles(target) {
|
|
87
|
+
const st = statSync(target);
|
|
88
|
+
if (st.isFile()) return extname(target) === ".sql" ? [target] : [];
|
|
89
|
+
const out = [];
|
|
90
|
+
for (const entry of readdirSync(target)) out.push(...findSqlFiles(join2(target, entry)));
|
|
91
|
+
return out;
|
|
92
|
+
}
|
|
93
|
+
function extract(files, compiled) {
|
|
94
|
+
return files.map((file) => {
|
|
95
|
+
const dbtSql = compiled && compiledFor(file, compiled);
|
|
96
|
+
const rawText = dbtSql ?? readFileSync3(file, "utf8");
|
|
97
|
+
const p = parse(rawText);
|
|
98
|
+
return {
|
|
99
|
+
id: file,
|
|
100
|
+
sourceFile: file,
|
|
101
|
+
rawText,
|
|
102
|
+
kind: dbtSql ? "dbt_model" : "sql",
|
|
103
|
+
refs: p.tables,
|
|
104
|
+
ast: p.ast,
|
|
105
|
+
parseError: p.parseError
|
|
106
|
+
};
|
|
107
|
+
});
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
// src/core/cost.ts
|
|
111
|
+
var TIB = 2 ** 40;
|
|
112
|
+
function bytesToUsd(bytes, pricePerTiB) {
|
|
113
|
+
return bytes / TIB * pricePerTiB;
|
|
114
|
+
}
|
|
115
|
+
function formatUsd(usd) {
|
|
116
|
+
const neg = usd < 0;
|
|
117
|
+
const a = Math.abs(usd);
|
|
118
|
+
const s = a < 0.01 ? `$${a.toFixed(4)}` : `$${a.toFixed(2)}`;
|
|
119
|
+
return neg ? `-${s}` : s;
|
|
120
|
+
}
|
|
121
|
+
function regress(head, base, absUsd, pricePerTiB) {
|
|
122
|
+
const headBytes = head.totalBytesProcessed;
|
|
123
|
+
const baseBytes = base?.dryRunOk ? base.totalBytesProcessed : null;
|
|
124
|
+
const deltaBytes = headBytes - (baseBytes ?? 0);
|
|
125
|
+
const deltaUsd = bytesToUsd(deltaBytes, pricePerTiB);
|
|
126
|
+
return { headBytes, baseBytes, deltaBytes, deltaUsd, verdict: deltaUsd >= absUsd ? "fail" : "ok" };
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
// src/core/rules/r2-partition-scan.ts
|
|
130
|
+
var r2PartitionScan = {
|
|
131
|
+
id: "R2",
|
|
132
|
+
severity: "warn",
|
|
133
|
+
check(ctx) {
|
|
134
|
+
const { cost, tables, cfg } = ctx;
|
|
135
|
+
if (!cost?.dryRunOk) return [];
|
|
136
|
+
if (cost.totalBytesProcessed < cfg.rules.r2.minScanBytes) return [];
|
|
137
|
+
const partRef = cost.referencedTables.find((r) => tables.get(r)?.isPartitioned);
|
|
138
|
+
if (!partRef) return [];
|
|
139
|
+
const meta = tables.get(partRef);
|
|
140
|
+
const gb2 = (cost.totalBytesProcessed / 2 ** 30).toFixed(1);
|
|
141
|
+
return [
|
|
142
|
+
{
|
|
143
|
+
ruleId: "R2",
|
|
144
|
+
severity: "warn",
|
|
145
|
+
message: `scans ${gb2} GB (${formatUsd(cost.estimatedUsd)}) of partitioned table ${partRef} \u2014 check the partition filter`,
|
|
146
|
+
fixHint: meta.partitionColumn ? `narrow the filter on partition column \`${meta.partitionColumn}\` (e.g. WHERE ${meta.partitionColumn} >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))` : `add a filter on the partition column of ${partRef}`
|
|
147
|
+
}
|
|
148
|
+
];
|
|
149
|
+
}
|
|
150
|
+
};
|
|
151
|
+
|
|
152
|
+
// src/core/rules/ast.ts
|
|
153
|
+
function collectSelects(node, out = []) {
|
|
154
|
+
if (!node || typeof node !== "object") return out;
|
|
155
|
+
if (Array.isArray(node)) {
|
|
156
|
+
for (const n of node) collectSelects(n, out);
|
|
157
|
+
return out;
|
|
158
|
+
}
|
|
159
|
+
if (node.type === "select") out.push(node);
|
|
160
|
+
for (const v of Object.values(node)) {
|
|
161
|
+
if (v && typeof v === "object") collectSelects(v, out);
|
|
162
|
+
}
|
|
163
|
+
return out;
|
|
164
|
+
}
|
|
165
|
+
function selectHasStar(sel) {
|
|
166
|
+
const cols = sel?.columns;
|
|
167
|
+
if (cols === "*") return true;
|
|
168
|
+
if (!Array.isArray(cols)) return false;
|
|
169
|
+
return cols.some((c) => c === "*" || c?.expr?.type === "star" || c?.expr?.column === "*");
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
// src/core/rules/r1-select-star.ts
|
|
173
|
+
var r1SelectStar = {
|
|
174
|
+
id: "R1",
|
|
175
|
+
severity: "info",
|
|
176
|
+
check(ctx) {
|
|
177
|
+
const ast = ctx.unit.ast;
|
|
178
|
+
if (!ast) return [];
|
|
179
|
+
const bytes = ctx.cost?.totalBytesProcessed ?? 0;
|
|
180
|
+
if (ctx.cost?.dryRunOk && bytes < ctx.cfg.rules.r1.minBytes) return [];
|
|
181
|
+
if (!collectSelects(ast).some(selectHasStar)) return [];
|
|
182
|
+
const gb2 = ctx.cost?.dryRunOk ? ` (${(bytes / 2 ** 30).toFixed(1)} GB)` : "";
|
|
183
|
+
return [
|
|
184
|
+
{
|
|
185
|
+
ruleId: "R1",
|
|
186
|
+
severity: "info",
|
|
187
|
+
message: `SELECT * \u2014 reads every column${gb2}`,
|
|
188
|
+
fixHint: "project only the columns you need instead of *"
|
|
189
|
+
}
|
|
190
|
+
];
|
|
191
|
+
}
|
|
192
|
+
};
|
|
193
|
+
|
|
194
|
+
// src/core/rules/r4-cross-join.ts
|
|
195
|
+
var r4CrossJoin = {
|
|
196
|
+
id: "R4",
|
|
197
|
+
severity: "warn",
|
|
198
|
+
check(ctx) {
|
|
199
|
+
const ast = ctx.unit.ast;
|
|
200
|
+
if (!ast) return [];
|
|
201
|
+
for (const sel of collectSelects(ast)) {
|
|
202
|
+
const from = sel.from;
|
|
203
|
+
if (!Array.isArray(from)) continue;
|
|
204
|
+
for (const f of from) {
|
|
205
|
+
if (String(f?.join ?? "").toUpperCase().includes("CROSS")) {
|
|
206
|
+
return [
|
|
207
|
+
{
|
|
208
|
+
ruleId: "R4",
|
|
209
|
+
severity: "warn",
|
|
210
|
+
message: "CROSS JOIN \u2014 cartesian fan-out risk",
|
|
211
|
+
fixHint: "add join keys (ON/USING) or confirm the intent"
|
|
212
|
+
}
|
|
213
|
+
];
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
return [];
|
|
218
|
+
}
|
|
219
|
+
};
|
|
220
|
+
|
|
221
|
+
// src/core/rules/r5-orderby-no-limit.ts
|
|
222
|
+
var r5OrderByNoLimit = {
|
|
223
|
+
id: "R5",
|
|
224
|
+
severity: "info",
|
|
225
|
+
check(ctx) {
|
|
226
|
+
const ast = ctx.unit.ast;
|
|
227
|
+
if (!ast) return [];
|
|
228
|
+
for (const sel of collectSelects(ast)) {
|
|
229
|
+
const hasOrder = Array.isArray(sel.orderby) ? sel.orderby.length > 0 : !!sel.orderby;
|
|
230
|
+
if (hasOrder && !sel.limit) {
|
|
231
|
+
return [
|
|
232
|
+
{
|
|
233
|
+
ruleId: "R5",
|
|
234
|
+
severity: "info",
|
|
235
|
+
message: "ORDER BY without LIMIT \u2014 sorts the whole result",
|
|
236
|
+
fixHint: "add LIMIT, or drop ORDER BY in a subquery/CTE"
|
|
237
|
+
}
|
|
238
|
+
];
|
|
239
|
+
}
|
|
240
|
+
}
|
|
241
|
+
return [];
|
|
242
|
+
}
|
|
243
|
+
};
|
|
244
|
+
|
|
245
|
+
// src/core/rules/index.ts
|
|
246
|
+
var RULES = [r2PartitionScan, r1SelectStar, r4CrossJoin, r5OrderByNoLimit];
|
|
247
|
+
|
|
248
|
+
// src/core/suppress.ts
|
|
249
|
+
function ignoredRules(sql) {
|
|
250
|
+
const m = sql.match(/--\s*bqcost:ignore(.*)$/im);
|
|
251
|
+
if (!m) return null;
|
|
252
|
+
const ids = m[1].trim().split(/[,\s]+/).map((s) => s.toUpperCase()).filter(Boolean);
|
|
253
|
+
return ids.length ? new Set(ids) : "all";
|
|
254
|
+
}
|
|
255
|
+
function applySuppressions(unit, cost, findings, cfg) {
|
|
256
|
+
const refs = [...unit.refs, ...cost?.referencedTables ?? []];
|
|
257
|
+
if (cfg.allowlist.some((a) => refs.some((r) => r.includes(a)))) return [];
|
|
258
|
+
const ign = ignoredRules(unit.rawText);
|
|
259
|
+
if (ign === "all") return [];
|
|
260
|
+
const disabled = new Set(cfg.disabledRules.map((s) => s.toUpperCase()));
|
|
261
|
+
return findings.filter(
|
|
262
|
+
(f) => !(ign && ign.has(f.ruleId)) && !disabled.has(f.ruleId.toUpperCase())
|
|
263
|
+
);
|
|
264
|
+
}
|
|
265
|
+
|
|
266
|
+
// src/run.ts
|
|
267
|
+
async function ruleFindings(unit, cost, bq, config) {
|
|
268
|
+
const tables = /* @__PURE__ */ new Map();
|
|
269
|
+
if (cost.dryRunOk) {
|
|
270
|
+
for (const ref of cost.referencedTables) {
|
|
271
|
+
const meta = await bq.fetchTableMeta(ref);
|
|
272
|
+
if (meta) tables.set(ref, meta);
|
|
273
|
+
}
|
|
274
|
+
}
|
|
275
|
+
const raw = RULES.flatMap((r) => r.check({ unit, cost, tables, cfg: config }));
|
|
276
|
+
return applySuppressions(unit, cost, raw, config);
|
|
277
|
+
}
|
|
278
|
+
async function run(target, config, bq) {
|
|
279
|
+
const mp = config.dbt.manifestPath;
|
|
280
|
+
const compiled = mp && existsSync2(mp) ? loadCompiledSql(mp) : void 0;
|
|
281
|
+
const units = extract(findSqlFiles(target), compiled);
|
|
282
|
+
const results = [];
|
|
283
|
+
for (const unit of units) {
|
|
284
|
+
let cost;
|
|
285
|
+
let findings2 = [];
|
|
286
|
+
if (bq) {
|
|
287
|
+
cost = await bq.estimateCost(unit.rawText);
|
|
288
|
+
findings2 = await ruleFindings(unit, cost, bq, config);
|
|
289
|
+
}
|
|
290
|
+
results.push({ unit, cost, findings: findings2 });
|
|
291
|
+
}
|
|
292
|
+
const findings = results.flatMap((r) => r.findings);
|
|
293
|
+
const totalUsd = results.reduce((s, r) => s + (r.cost?.estimatedUsd ?? 0), 0);
|
|
294
|
+
return { scanned: units.length, results, findings, totalUsd };
|
|
295
|
+
}
|
|
296
|
+
|
|
297
|
+
// src/ci.ts
|
|
298
|
+
import { readFileSync as readFileSync4 } from "fs";
|
|
299
|
+
import { join as join3 } from "path";
|
|
300
|
+
|
|
301
|
+
// src/core/normalize.ts
|
|
302
|
+
function normalize(sql) {
|
|
303
|
+
return sql.replace(/--[^\n]*/g, " ").replace(/\/\*[\s\S]*?\*\//g, " ").replace(/\s+/g, " ").trim();
|
|
304
|
+
}
|
|
305
|
+
function sqlUnchanged(a, b) {
|
|
306
|
+
return normalize(a) === normalize(b);
|
|
307
|
+
}
|
|
308
|
+
|
|
309
|
+
// src/adapters/git.ts
|
|
310
|
+
import { execFileSync } from "child_process";
|
|
311
|
+
function git(args, cwd) {
|
|
312
|
+
return execFileSync("git", args, { cwd, encoding: "utf8" });
|
|
313
|
+
}
|
|
314
|
+
function changedSqlFiles(base, cwd) {
|
|
315
|
+
const out = git(["diff", "--name-only", "--diff-filter=AM", `${base}...HEAD`], cwd);
|
|
316
|
+
return out.split("\n").map((s) => s.trim()).filter((f) => f.endsWith(".sql"));
|
|
317
|
+
}
|
|
318
|
+
function fileAtBase(base, path, cwd) {
|
|
319
|
+
try {
|
|
320
|
+
return git(["show", `${base}:${path}`], cwd);
|
|
321
|
+
} catch {
|
|
322
|
+
return null;
|
|
323
|
+
}
|
|
324
|
+
}
|
|
325
|
+
|
|
326
|
+
// src/ci.ts
|
|
327
|
+
async function runChanged(base, cwd, config, bq) {
|
|
328
|
+
const files = changedSqlFiles(base, cwd);
|
|
329
|
+
const results = [];
|
|
330
|
+
for (const file of files) {
|
|
331
|
+
const headSql = readFileSync4(join3(cwd, file), "utf8");
|
|
332
|
+
const head = await bq.estimateCost(headSql);
|
|
333
|
+
let regression;
|
|
334
|
+
let findings = [];
|
|
335
|
+
if (head.dryRunOk) {
|
|
336
|
+
const baseSql = fileAtBase(base, file, cwd);
|
|
337
|
+
const baseCost = baseSql && !sqlUnchanged(baseSql, headSql) ? await bq.estimateCost(baseSql) : null;
|
|
338
|
+
regression = regress(head, baseCost, config.regression.absUsd, config.pricePerTiB);
|
|
339
|
+
const p = parse(headSql);
|
|
340
|
+
const unit = { id: file, sourceFile: file, rawText: headSql, kind: "sql", refs: p.tables, ast: p.ast };
|
|
341
|
+
findings = await ruleFindings(unit, head, bq, config);
|
|
342
|
+
}
|
|
343
|
+
results.push({ file, head, regression, findings });
|
|
344
|
+
}
|
|
345
|
+
const totalDeltaUsd = results.reduce((s, r) => s + (r.regression?.deltaUsd ?? 0), 0);
|
|
346
|
+
const verdict = results.some((r) => r.regression?.verdict === "fail") ? "fail" : "ok";
|
|
347
|
+
return { base, results, totalDeltaUsd, verdict };
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
// src/core/report.ts
|
|
351
|
+
var gb = (bytes) => (bytes / 2 ** 30).toFixed(1) + " GB";
|
|
352
|
+
function renderCli(report) {
|
|
353
|
+
const lines = [
|
|
354
|
+
`bqcost: scanned ${report.scanned} file(s), est. ${formatUsd(report.totalUsd)}, ${report.findings.length} finding(s)`
|
|
355
|
+
];
|
|
356
|
+
for (const r of report.results) {
|
|
357
|
+
if (!r.findings.length) continue;
|
|
358
|
+
const cost = r.cost?.dryRunOk ? ` (${formatUsd(r.cost.estimatedUsd)}, ${gb(r.cost.totalBytesProcessed)})` : "";
|
|
359
|
+
lines.push(`
|
|
360
|
+
${r.unit.sourceFile}${cost}`);
|
|
361
|
+
for (const f of r.findings) {
|
|
362
|
+
lines.push(` [${f.ruleId}] ${f.message}`);
|
|
363
|
+
if (f.fixHint) lines.push(` fix: ${f.fixHint}`);
|
|
364
|
+
}
|
|
365
|
+
}
|
|
366
|
+
return lines.join("\n");
|
|
367
|
+
}
|
|
368
|
+
function renderCi(ci) {
|
|
369
|
+
const lines = [
|
|
370
|
+
`bqcost CI vs ${ci.base}: ${ci.results.length} changed file(s), \u0394 ${formatUsd(ci.totalDeltaUsd)}, verdict ${ci.verdict.toUpperCase()}`
|
|
371
|
+
];
|
|
372
|
+
for (const r of ci.results) {
|
|
373
|
+
if (!r.head?.dryRunOk) {
|
|
374
|
+
lines.push(`
|
|
375
|
+
${r.file} (not estimated: ${r.head?.error ?? "no estimate"})`);
|
|
376
|
+
continue;
|
|
377
|
+
}
|
|
378
|
+
const reg = r.regression;
|
|
379
|
+
const delta = reg.baseBytes == null ? "new query" : `\u0394 ${formatUsd(reg.deltaUsd)}`;
|
|
380
|
+
const mark = reg.verdict === "fail" ? "\u274C" : "\u2705";
|
|
381
|
+
lines.push(`
|
|
382
|
+
${r.file} ${formatUsd(r.head.estimatedUsd)} (${gb(r.head.totalBytesProcessed)}) ${delta} ${mark}`);
|
|
383
|
+
for (const f of r.findings) {
|
|
384
|
+
lines.push(` [${f.ruleId}] ${f.message}`);
|
|
385
|
+
if (f.fixHint) lines.push(` fix: ${f.fixHint}`);
|
|
386
|
+
}
|
|
387
|
+
}
|
|
388
|
+
return lines.join("\n");
|
|
389
|
+
}
|
|
390
|
+
|
|
391
|
+
// src/adapters/bigquery.ts
|
|
392
|
+
import { BigQuery } from "@google-cloud/bigquery";
|
|
393
|
+
var BqClient = class {
|
|
394
|
+
bq;
|
|
395
|
+
location;
|
|
396
|
+
pricePerTiB;
|
|
397
|
+
metaCache = /* @__PURE__ */ new Map();
|
|
398
|
+
constructor(opts) {
|
|
399
|
+
this.bq = new BigQuery(opts.projectId ? { projectId: opts.projectId } : {});
|
|
400
|
+
this.location = opts.location;
|
|
401
|
+
this.pricePerTiB = opts.pricePerTiB;
|
|
402
|
+
}
|
|
403
|
+
/** Dry-run cost. Fail-soft: on error returns `dryRunOk:false` rather than throwing. */
|
|
404
|
+
async estimateCost(sql) {
|
|
405
|
+
try {
|
|
406
|
+
const [job] = await this.bq.createQueryJob({
|
|
407
|
+
query: sql,
|
|
408
|
+
dryRun: true,
|
|
409
|
+
location: this.location
|
|
410
|
+
});
|
|
411
|
+
const stats = job.metadata?.statistics ?? {};
|
|
412
|
+
const bytes = Number(stats.totalBytesProcessed ?? 0);
|
|
413
|
+
const refs = (stats.query?.referencedTables ?? []).map(
|
|
414
|
+
(t) => `${t.projectId}.${t.datasetId}.${t.tableId}`
|
|
415
|
+
);
|
|
416
|
+
return {
|
|
417
|
+
totalBytesProcessed: bytes,
|
|
418
|
+
estimatedUsd: bytesToUsd(bytes, this.pricePerTiB),
|
|
419
|
+
referencedTables: refs,
|
|
420
|
+
dryRunOk: true
|
|
421
|
+
};
|
|
422
|
+
} catch (e) {
|
|
423
|
+
return {
|
|
424
|
+
totalBytesProcessed: 0,
|
|
425
|
+
estimatedUsd: 0,
|
|
426
|
+
referencedTables: [],
|
|
427
|
+
dryRunOk: false,
|
|
428
|
+
error: e?.message ?? String(e)
|
|
429
|
+
};
|
|
430
|
+
}
|
|
431
|
+
}
|
|
432
|
+
/** Partition/clustering metadata for `project.dataset.table` (cached, fail-soft). */
|
|
433
|
+
async fetchTableMeta(fqName) {
|
|
434
|
+
if (this.metaCache.has(fqName)) return this.metaCache.get(fqName) ?? null;
|
|
435
|
+
const meta = await this.loadTableMeta(fqName);
|
|
436
|
+
this.metaCache.set(fqName, meta);
|
|
437
|
+
return meta;
|
|
438
|
+
}
|
|
439
|
+
async loadTableMeta(fqName) {
|
|
440
|
+
const parts = fqName.split(".");
|
|
441
|
+
if (parts.length !== 3) return null;
|
|
442
|
+
const [project, dataset, table] = parts;
|
|
443
|
+
try {
|
|
444
|
+
const [md] = await this.bq.dataset(dataset, { projectId: project }).table(table).getMetadata();
|
|
445
|
+
const tp = md.timePartitioning;
|
|
446
|
+
const rp = md.rangePartitioning;
|
|
447
|
+
let partitionColumn = null;
|
|
448
|
+
let partitionType = null;
|
|
449
|
+
if (tp) {
|
|
450
|
+
partitionColumn = tp.field ?? "_PARTITIONTIME";
|
|
451
|
+
partitionType = tp.field ? tp.type ?? "DAY" : "INGESTION_TIME";
|
|
452
|
+
} else if (rp) {
|
|
453
|
+
partitionColumn = rp.field ?? null;
|
|
454
|
+
partitionType = "RANGE";
|
|
455
|
+
}
|
|
456
|
+
return {
|
|
457
|
+
fqName,
|
|
458
|
+
sizeBytes: md.numBytes != null ? Number(md.numBytes) : null,
|
|
459
|
+
isPartitioned: !!(tp || rp),
|
|
460
|
+
partitionColumn,
|
|
461
|
+
partitionType,
|
|
462
|
+
clusteringColumns: md.clustering?.fields ?? [],
|
|
463
|
+
isView: md.type === "VIEW"
|
|
464
|
+
};
|
|
465
|
+
} catch {
|
|
466
|
+
return null;
|
|
467
|
+
}
|
|
468
|
+
}
|
|
469
|
+
};
|
|
470
|
+
|
|
471
|
+
// src/cli.ts
|
|
472
|
+
function bqFromEnv(pricePerTiB) {
|
|
473
|
+
return process.env.GOOGLE_APPLICATION_CREDENTIALS ? new BqClient({
|
|
474
|
+
projectId: process.env.BQCOST_PROJECT_ID,
|
|
475
|
+
location: process.env.BQCOST_LOCATION,
|
|
476
|
+
pricePerTiB
|
|
477
|
+
}) : null;
|
|
478
|
+
}
|
|
479
|
+
var program = new Command();
|
|
480
|
+
program.name("bqcost").description("Catch the BigQuery query that would've cost $10k \u2014 in your PR.").version("0.0.0");
|
|
481
|
+
program.command("lint").argument("<paths...>", "files or directories of .sql to lint").action(async (paths) => {
|
|
482
|
+
const config = loadConfig();
|
|
483
|
+
const bq = bqFromEnv(config.pricePerTiB);
|
|
484
|
+
let failed = false;
|
|
485
|
+
for (const p of paths) {
|
|
486
|
+
const report = await run(p, config, bq);
|
|
487
|
+
console.log(renderCli(report));
|
|
488
|
+
if (report.findings.some((f) => f.severity === "error")) failed = true;
|
|
489
|
+
}
|
|
490
|
+
process.exitCode = failed ? 1 : 0;
|
|
491
|
+
});
|
|
492
|
+
program.command("ci").description("cost-regression gate for changed .sql vs a base git ref").option("--base <ref>", "base git ref", "origin/main").action(async (opts) => {
|
|
493
|
+
const config = loadConfig();
|
|
494
|
+
const bq = bqFromEnv(config.pricePerTiB);
|
|
495
|
+
if (!bq) {
|
|
496
|
+
console.error("bqcost ci needs BigQuery creds (GOOGLE_APPLICATION_CREDENTIALS).");
|
|
497
|
+
process.exitCode = 2;
|
|
498
|
+
return;
|
|
499
|
+
}
|
|
500
|
+
const ci = await runChanged(opts.base, process.cwd(), config, bq);
|
|
501
|
+
console.log(renderCi(ci));
|
|
502
|
+
process.exitCode = ci.verdict === "fail" ? 1 : 0;
|
|
503
|
+
});
|
|
504
|
+
program.parseAsync();
|
package/package.json
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "bqcost",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Catch the BigQuery query that would've cost $10k — in your PR. Dry-run cost + regression gate for CI.",
|
|
5
|
+
"keywords": [
|
|
6
|
+
"bigquery",
|
|
7
|
+
"cost",
|
|
8
|
+
"finops",
|
|
9
|
+
"dbt",
|
|
10
|
+
"sql",
|
|
11
|
+
"linter",
|
|
12
|
+
"github-action",
|
|
13
|
+
"pre-commit",
|
|
14
|
+
"ci",
|
|
15
|
+
"dry-run"
|
|
16
|
+
],
|
|
17
|
+
"homepage": "https://github.com/dagryazev/bqcost#readme",
|
|
18
|
+
"bugs": "https://github.com/dagryazev/bqcost/issues",
|
|
19
|
+
"repository": {
|
|
20
|
+
"type": "git",
|
|
21
|
+
"url": "git+https://github.com/dagryazev/bqcost.git"
|
|
22
|
+
},
|
|
23
|
+
"license": "MIT",
|
|
24
|
+
"author": "Denis Gryazev",
|
|
25
|
+
"type": "module",
|
|
26
|
+
"bin": { "bqcost": "dist/cli.js" },
|
|
27
|
+
"files": ["dist"],
|
|
28
|
+
"engines": { "node": ">=18" },
|
|
29
|
+
"scripts": {
|
|
30
|
+
"build": "tsup",
|
|
31
|
+
"test": "vitest run",
|
|
32
|
+
"typecheck": "tsc --noEmit",
|
|
33
|
+
"label:r2": "tsx scripts/label-r2.ts",
|
|
34
|
+
"prepublishOnly": "npm run build"
|
|
35
|
+
},
|
|
36
|
+
"dependencies": {
|
|
37
|
+
"@google-cloud/bigquery": "^7.9.0",
|
|
38
|
+
"commander": "^12.1.0",
|
|
39
|
+
"node-sql-parser": "^5.3.4",
|
|
40
|
+
"yaml": "^2.6.1",
|
|
41
|
+
"zod": "^3.23.8"
|
|
42
|
+
},
|
|
43
|
+
"devDependencies": {
|
|
44
|
+
"@actions/core": "^1.11.1",
|
|
45
|
+
"@actions/github": "^6.0.0",
|
|
46
|
+
"@types/node": "^22.9.0",
|
|
47
|
+
"tsup": "^8.3.5",
|
|
48
|
+
"tsx": "^4.19.2",
|
|
49
|
+
"typescript": "^5.6.3",
|
|
50
|
+
"vitest": "^2.1.5"
|
|
51
|
+
}
|
|
52
|
+
}
|