gulliver-mcp 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. gulliver_mcp-0.1.0/.gitignore +22 -0
  2. gulliver_mcp-0.1.0/AGENTS.md +67 -0
  3. gulliver_mcp-0.1.0/CHANGELOG.md +26 -0
  4. gulliver_mcp-0.1.0/LICENSE +21 -0
  5. gulliver_mcp-0.1.0/PKG-INFO +374 -0
  6. gulliver_mcp-0.1.0/README.md +340 -0
  7. gulliver_mcp-0.1.0/llms.txt +69 -0
  8. gulliver_mcp-0.1.0/pyproject.toml +74 -0
  9. gulliver_mcp-0.1.0/skills/gulliver/SKILL.md +65 -0
  10. gulliver_mcp-0.1.0/src/gulliver/__init__.py +7 -0
  11. gulliver_mcp-0.1.0/src/gulliver/checks.py +173 -0
  12. gulliver_mcp-0.1.0/src/gulliver/cli.py +364 -0
  13. gulliver_mcp-0.1.0/src/gulliver/config.py +209 -0
  14. gulliver_mcp-0.1.0/src/gulliver/core.py +598 -0
  15. gulliver_mcp-0.1.0/src/gulliver/demo.py +40 -0
  16. gulliver_mcp-0.1.0/src/gulliver/diffing.py +138 -0
  17. gulliver_mcp-0.1.0/src/gulliver/engines/__init__.py +38 -0
  18. gulliver_mcp-0.1.0/src/gulliver/engines/base.py +166 -0
  19. gulliver_mcp-0.1.0/src/gulliver/engines/bigquery.py +181 -0
  20. gulliver_mcp-0.1.0/src/gulliver/engines/duckdb_engine.py +283 -0
  21. gulliver_mcp-0.1.0/src/gulliver/engines/postgres.py +207 -0
  22. gulliver_mcp-0.1.0/src/gulliver/engines/snowflake.py +191 -0
  23. gulliver_mcp-0.1.0/src/gulliver/extrapolate.py +126 -0
  24. gulliver_mcp-0.1.0/src/gulliver/features.py +457 -0
  25. gulliver_mcp-0.1.0/src/gulliver/incremental.py +274 -0
  26. gulliver_mcp-0.1.0/src/gulliver/lineage.py +306 -0
  27. gulliver_mcp-0.1.0/src/gulliver/lint.py +260 -0
  28. gulliver_mcp-0.1.0/src/gulliver/mcp_server.py +340 -0
  29. gulliver_mcp-0.1.0/src/gulliver/receipts.py +139 -0
  30. gulliver_mcp-0.1.0/src/gulliver/sampling.py +51 -0
  31. gulliver_mcp-0.1.0/src/gulliver/sqlsafety.py +129 -0
  32. gulliver_mcp-0.1.0/src/gulliver/store.py +154 -0
  33. gulliver_mcp-0.1.0/tests/conftest.py +28 -0
  34. gulliver_mcp-0.1.0/tests/test_cli_and_mcp.py +50 -0
  35. gulliver_mcp-0.1.0/tests/test_config.py +66 -0
  36. gulliver_mcp-0.1.0/tests/test_engines_offline.py +92 -0
  37. gulliver_mcp-0.1.0/tests/test_features.py +207 -0
  38. gulliver_mcp-0.1.0/tests/test_flow_duckdb.py +171 -0
  39. gulliver_mcp-0.1.0/tests/test_lint.py +69 -0
  40. gulliver_mcp-0.1.0/tests/test_postgres_live.py +129 -0
  41. gulliver_mcp-0.1.0/tests/test_sampling_and_extrapolate.py +75 -0
  42. gulliver_mcp-0.1.0/tests/test_sqlsafety.py +51 -0
@@ -0,0 +1,22 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ *.egg-info/
4
+ build/
5
+ dist/
6
+ .venv/
7
+ venv/
8
+ .pytest_cache/
9
+ .ruff_cache/
10
+ .mypy_cache/
11
+ .coverage
12
+ htmlcov/
13
+
14
+ # Gulliver local state and data
15
+ .gulliver/
16
+ *.duckdb
17
+ *.duckdb.wal
18
+ gulliver.toml
19
+
20
+ .DS_Store
21
+ .idea/
22
+ .vscode/
@@ -0,0 +1,67 @@
1
+ # Gulliver for agents
2
+
3
+ Gulliver checks SQL before it touches big data. Use it whenever you are about to run SQL you wrote against a
4
+ real table (DuckDB, BigQuery, Snowflake or Postgres), especially when you do not know how big the tables are.
5
+
6
+ ## Tools (MCP) and the matching CLI commands
7
+
8
+ | MCP tool | CLI | Use it to |
9
+ |---|---|---|
10
+ | `gulliver_status` | `gulliver status --json` | see connections, budget limits and money spent today |
11
+ | `gulliver_inspect` | `gulliver inspect [table] --json` | find the big tables, partitions and columns (free) |
12
+ | `gulliver_lint` | `gulliver lint "SQL" --json` | check SQL for scale problems without a database |
13
+ | `gulliver_preflight` | `gulliver preflight "SQL" --json` | estimate bytes, rows and money without running |
14
+ | `gulliver_rehearse` | `gulliver rehearse "SQL" --json` | run on 1%, 3%, 10% samples and predict the full run |
15
+ | `gulliver_run` | `gulliver run "SQL" --json` | run the full query inside the budget |
16
+ | `gulliver_receipt` | `gulliver receipt latest` | give the human proof of what happened |
17
+ | `gulliver_impact` | `gulliver impact table.column --models DIR` | see which models and columns a change breaks |
18
+ | `gulliver_drift` | `gulliver drift --models DIR` | see schema changes since the last snapshot and what they break |
19
+ | `gulliver_diff` | `gulliver diff --left Q --right Q --key id` | prove two queries/tables return the same rows |
20
+ | `gulliver_translate` | `gulliver translate SQL --to bigquery` | move SQL between DuckDB, Postgres, BigQuery, Snowflake |
21
+ | `gulliver_incremental` | `gulliver incremental SQL --time-column ts` | make a full-rebuild model incremental, with the saving |
22
+ | `gulliver_check` | `gulliver check table --unique id ...` | data quality in one scan (status `fail` if a check fails) |
23
+
24
+ ## The loop
25
+
26
+ 1. `gulliver_inspect` once per task, so you know which tables are big and how they are partitioned.
27
+ 2. Write the SQL. Select only the columns you need. Filter on the partition column. Aggregate before returning rows.
28
+ 3. `gulliver_preflight`. Read `status` and `next_step`.
29
+ 4. If `next_step` asks for it (or the estimate is large or only `heuristic`), `gulliver_rehearse`.
30
+ 5. `gulliver_run` with the exact same SQL. The preview has at most `max_output_rows` rows; `actual.rows` is the total.
31
+ 6. Offer `gulliver_receipt` to the human.
32
+
33
+ ## Pipeline work
34
+
35
+ - Before changing or dropping a column in a model or source: `gulliver_impact` with the models folder.
36
+ - After refactoring a model, migrating it to another engine or making it incremental: `gulliver_diff` between the
37
+ old and the new query with the key. Say "identical" only when `identical` is true.
38
+ - When a model rebuilds everything each run: `gulliver_incremental`; present the strategy, the SQL and the saving,
39
+ and verify with `gulliver_diff`.
40
+ - Before trusting a table or after a load: `gulliver_check` (unique keys, not null, freshness). On partitioned
41
+ tables pass `where` with a partition filter.
42
+ - Periodically, or when a pipeline breaks: `gulliver_drift` with the models folder.
43
+
44
+ ## Status values
45
+
46
+ | status | meaning | what to do |
47
+ |---|---|---|
48
+ | `ok` | within budget | continue |
49
+ | `warn` | allowed, with risks listed in `lint` / `warnings` | fix them if cheap, else continue |
50
+ | `block` | over a limit, a lint error, or a refused statement | change the SQL; do not work around the gate |
51
+ | `needs_rehearsal` | big enough that a rehearsal is required | call `gulliver_rehearse` with the same SQL |
52
+ | `needs_approval` | costs more than the auto-approve limit | tell the human the `receipt_id`; they run `gulliver approve <id>` |
53
+ | `skipped` | rehearsal not useful (tiny data) | call `gulliver_run` |
54
+ | `fail` | a data quality check failed | report which checks failed; do not use the data as if it were fine |
55
+ | `error` | engine or config problem | read `error`, fix, retry once |
56
+
57
+ ## Reading a rehearsal
58
+
59
+ `prediction.<metric>.value` is the expected full-run value; `low` and `high` give the range.
60
+ `exponent` near 1 means it grows with the data; above 1.3 means it grows faster (a join that multiplies rows).
61
+ `confidence` is `high`, `medium` or `low`. Treat `low` as a rough range, not a number.
62
+
63
+ ## Rules you cannot change
64
+
65
+ - Only one statement per call. Only SELECT, or CREATE/INSERT ... SELECT when a human enabled writes.
66
+ - You cannot approve your own runs. Approval is a human CLI command.
67
+ - Credentials and budgets are set by the human in `gulliver.toml` or environment variables.
@@ -0,0 +1,26 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 - 2026-11-01
4
+
5
+ First release.
6
+
7
+ Query rehearsal:
8
+ - Workflow: `status`, `inspect`, `lint`, `preflight`, `rehearse`, `run`, `receipt`, `approve`.
9
+ - Rehearsal ladder (1% → 3% → 10%), power-law extrapolation with confidence ranges, detection of super-linear growth,
10
+ saturation cap for grouped results.
11
+ - 12 scale lint rules (GV001-GV012).
12
+ - Budget gate: per query, per day, bytes, runtime, rehearsal requirement and human approval.
13
+
14
+ Pipeline tools:
15
+ - `impact` / `lineage`: column-level lineage over SQL model folders (dbt Jinja supported) or a dbt manifest.
16
+ - `snapshot` / `drift`: schema drift per connection, mapped to impacted models.
17
+ - `diff`: key-based data diff inside the engines with MD5 buckets and a zoom-in pass; works across engines.
18
+ - `translate`: SQL between DuckDB, Postgres, BigQuery and Snowflake, with warnings and an optional target preflight.
19
+ - `incremental`: append / partition_overwrite / merge_recompute / not_safe advice, generated SQL and dbt config,
20
+ measured savings.
21
+ - `check`: unique, not null, accepted values, range, freshness and row-count change in one scan (exit code 4 on fail).
22
+
23
+ Engines and integration:
24
+ - DuckDB and PostgreSQL tested (Postgres live in CI), BigQuery and Snowflake beta (tested with fakes).
25
+ - MCP server (stdio and Streamable HTTP), CLI with `--json`, OpenAI and Gemini tool export.
26
+ - Claude Code plugin and agent skill.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Roel
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,374 @@
1
+ Metadata-Version: 2.5
2
+ Name: gulliver-mcp
3
+ Version: 0.1.0
4
+ Summary: Test on tiny data before you run on giant data. A dress rehearsal for data agents: estimate the cost, rehearse on 1% -> 10%, predict 100%, and only go full scale within budget.
5
+ Project-URL: Homepage, https://github.com/laaielichter/gulliver
6
+ Project-URL: Issues, https://github.com/laaielichter/gulliver/issues
7
+ Author: laaielichter
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Keywords: ai-agents,bigquery,cost,data-engineering,duckdb,llm,mcp,postgres,snowflake,sql
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Topic :: Database
15
+ Classifier: Topic :: Software Development :: Quality Assurance
16
+ Requires-Python: >=3.10
17
+ Requires-Dist: duckdb>=1.1
18
+ Requires-Dist: mcp>=1.10
19
+ Requires-Dist: sqlglot>=25.0
20
+ Requires-Dist: tomli>=2.0; python_version < '3.11'
21
+ Provides-Extra: all
22
+ Requires-Dist: google-cloud-bigquery>=3.20; extra == 'all'
23
+ Requires-Dist: psycopg[binary]>=3.1; extra == 'all'
24
+ Requires-Dist: snowflake-connector-python>=3.10; extra == 'all'
25
+ Provides-Extra: bigquery
26
+ Requires-Dist: google-cloud-bigquery>=3.20; extra == 'bigquery'
27
+ Provides-Extra: dev
28
+ Requires-Dist: pytest>=8; extra == 'dev'
29
+ Provides-Extra: postgres
30
+ Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
31
+ Provides-Extra: snowflake
32
+ Requires-Dist: snowflake-connector-python>=3.10; extra == 'snowflake'
33
+ Description-Content-Type: text/markdown
34
+
35
+ # Gulliver
36
+
37
+ <!-- mcp-name: io.github.laaielichter/gulliver -->
38
+
39
+ **Test on tiny data before you run on giant data.**
40
+
41
+ Gulliver gives AI agents a dress rehearsal for SQL: estimate the cost, rehearse on 1% → 3% → 10% of the data, predict what 100% will do, and only run the full query inside a budget. Local-first, open source, MCP + CLI for any agent and any LLM.
42
+
43
+ [![ci](https://github.com/laaielichter/gulliver/actions/workflows/ci.yml/badge.svg)](https://github.com/laaielichter/gulliver/actions/workflows/ci.yml)
44
+ ![license](https://img.shields.io/badge/license-MIT-blue)
45
+ ![python](https://img.shields.io/badge/python-3.10%2B-blue)
46
+
47
+ > In *Gulliver's Travels*, Gulliver visits Lilliput, a land of tiny people, and Brobdingnag, a land of giants.
48
+ > Gulliver does the same with your data: it tries your query in Lilliput first, and only walks into Brobdingnag
49
+ > when it knows what will happen there.
50
+
51
+ ## Why
52
+
53
+ Agents write SQL that works on a small table and breaks on a big one. They do not know that `events` has
54
+ 5 billion rows, that a join multiplies rows, or that a forgotten partition filter scans every day since 2019.
55
+ Benchmarks score agents on *is the answer right*, never on *what did it cost* or *does it still work at 100x*.
56
+ One unlucky query can burn a monthly budget.
57
+
58
+ Gulliver sits between the agent and the warehouse:
59
+
60
+ | Step | Tool | What happens |
61
+ |---|---|---|
62
+ | 1 | `gulliver_inspect` | Which tables are big? Rows, bytes, partitions, columns. Metadata only, free. |
63
+ | 2 | `gulliver_preflight` | Bytes, rows and money **before** running, plus **scale lint** (12 rules) and the budget gate. |
64
+ | 3 | `gulliver_rehearse` | Runs the query on 1%, 3% and 10% samples, fits a growth curve and **predicts the full run** with a confidence range. Catches joins that multiply rows. |
65
+ | 4 | `gulliver_run` | Runs the full query only inside the budget. Otherwise answers `needs_rehearsal` or `needs_approval`. |
66
+ | 5 | `gulliver_receipt` | Markdown proof for the human: what was estimated, rehearsed, approved and run, and predicted vs actual. |
67
+
68
+ Every answer has a `status` and a `next_step`, so even small models know what to do next.
69
+
70
+ And for the pipeline around the query (details in [Pipeline tools](#pipeline-tools)):
71
+
72
+ | Tool | Problem it solves |
73
+ |---|---|
74
+ | `gulliver_impact` | "If I change or drop this column, what breaks?" Column-level lineage over SQL models or a dbt manifest. |
75
+ | `gulliver_drift` | Schema changed under you: columns removed, retyped or added since the last snapshot, plus the models they break. |
76
+ | `gulliver_diff` | "Does my refactored / migrated / incremental model return the same rows?" Key-based data diff inside the engine, also across engines. |
77
+ | `gulliver_translate` | SQL between DuckDB, Postgres, BigQuery and Snowflake, with warnings and an optional preflight on the target. |
78
+ | `gulliver_incremental` | A model rebuilds everything every run. Is incremental safe, what is the SQL / dbt config, and how many rows does it skip? |
79
+ | `gulliver_check` | Data quality in one cheap scan: unique keys, not null, accepted values, ranges, freshness, row-count jumps. |
80
+
81
+ ## Quick start (2 minutes, no warehouse needed)
82
+
83
+ ```bash
84
+ pip install git+https://github.com/laaielichter/gulliver
85
+ gulliver demo # creates gulliver_demo.duckdb with 5M events
86
+ export GULLIVER_DSN=gulliver_demo.duckdb
87
+
88
+ gulliver inspect
89
+ gulliver preflight "select country, count(*) from events e join users u on e.user_id = u.user_id group by 1"
90
+ gulliver lint "select * from events e, users u" # GV003: join without a condition -> blocked
91
+ gulliver rehearse "select a.event_id, b.event_id from events a join events b on a.user_id = b.user_id"
92
+ ```
93
+
94
+ The last one is the classic agent mistake: a self-join on a non-unique key. The rehearsal sees the output grow
95
+ with exponent ~1.8 instead of ~1 and warns that the full result has more than 100 million rows, before you run it.
96
+
97
+ ```
98
+ | Sample | Runtime | Output rows |
99
+ |--------|----------|-------------|
100
+ | 1% | 0.0202s | 41,706 |
101
+ | 3% | 0.0484s | 237,422 |
102
+ | 10% | 0.3909s | 2,438,395 |
103
+
104
+ Prediction for 100%: output rows 134,525,654 (range 67,262,827 - 269,051,308)
105
+ Growth exponent output_rows: 1.77 -> grows faster than the data
106
+ ```
107
+
108
+ Add `--json` to any command for machine-readable output. Exit codes: `0` ok, `1` error, `2` blocked,
109
+ `3` needs rehearsal or approval, `4` a data check failed, so it also works as a CI check.
110
+
111
+ ## Use it from your agent (MCP)
112
+
113
+ Gulliver is an MCP server (stdio or Streamable HTTP). Until it is on PyPI, run it straight from GitHub with
114
+ [uv](https://docs.astral.sh/uv/):
115
+
116
+ ```bash
117
+ uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn path/to/warehouse.duckdb
118
+ ```
119
+
120
+ **Claude Code**
121
+
122
+ ```bash
123
+ claude mcp add gulliver -- uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn /abs/path/data.duckdb
124
+ ```
125
+
126
+ Or install it as a plugin (MCP server + skill):
127
+
128
+ ```
129
+ /plugin marketplace add laaielichter/gulliver
130
+ /plugin install gulliver@gulliver
131
+ ```
132
+
133
+ **Claude Desktop, Cursor, Windsurf, VS Code and other MCP clients** (JSON config):
134
+
135
+ ```json
136
+ {
137
+ "mcpServers": {
138
+ "gulliver": {
139
+ "command": "uvx",
140
+ "args": ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
141
+ }
142
+ }
143
+ }
144
+ ```
145
+
146
+ **Codex CLI** (`~/.codex/config.toml`):
147
+
148
+ ```toml
149
+ [mcp_servers.gulliver]
150
+ command = "uvx"
151
+ args = ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
152
+ ```
153
+
154
+ **Gemini CLI** (`~/.gemini/settings.json`): same `mcpServers` block as above.
155
+
156
+ **Any HTTP client or remote agent:** `gulliver-mcp --http --port 8765` serves `http://127.0.0.1:8765/mcp`.
157
+
158
+ **No MCP?** Every tool is also a CLI command with `--json`, and `gulliver tools --format openai` or
159
+ `--format gemini` prints ready-made function-calling definitions for the OpenAI or Gemini APIs.
160
+
161
+ ## Engines
162
+
163
+ | Engine | Estimate before running | Sampling | Money model | Status |
164
+ |---|---|---|---|---|
165
+ | DuckDB (files, Parquet, CSV, in-memory) | EXPLAIN + metadata (heuristic) | `TABLESAMPLE` system/bernoulli, seeded | free (time is the budget) | tested in CI |
166
+ | BigQuery | dry run: **exact** bytes | `TABLESAMPLE SYSTEM` | on-demand $/TiB, plus a hard `maximum_bytes_billed` cap | beta, tested with fakes |
167
+ | Snowflake | `EXPLAIN USING JSON`: bytes and partitions (upper bound) | `SAMPLE SYSTEM ... SEED` | warehouse credits x runtime | beta, tested with fakes |
168
+ | PostgreSQL | `EXPLAIN (FORMAT JSON)` (heuristic), partition-aware sizes | `TABLESAMPLE ... REPEATABLE` | self-hosted (time and bytes are the budget) | tested in CI against a live Postgres 16 |
169
+
170
+ Install the drivers you need: `pip install "gulliver-mcp[bigquery]"`, `[snowflake]`, `[postgres]` or `[all]`.
171
+ Every estimate says how much to trust it: `exact`, `upper_bound` or `heuristic`. A heuristic is never called exact.
172
+
173
+ ## Configuration
174
+
175
+ Copy [`gulliver.example.toml`](gulliver.example.toml) to `gulliver.toml`:
176
+
177
+ ```toml
178
+ default_connection = "local"
179
+
180
+ [connections.local]
181
+ engine = "duckdb"
182
+ path = "warehouse.duckdb"
183
+
184
+ [connections.bq]
185
+ dsn = "bigquery://my-project?location=EU"
186
+ dataset = "analytics"
187
+ price_per_tib_usd = 6.25
188
+
189
+ [budget]
190
+ max_query_usd = 1.00 # hard stop for one query
191
+ approval_over_usd = 0.25 # above this a human runs `gulliver approve <receipt_id>`
192
+ rehearse_over_usd = 0.10 # above this a rehearsal is required first
193
+ day_usd = 10.00 # total per day (local ledger)
194
+ max_runtime_s = 900
195
+ max_output_rows = 200 # rows handed back to the agent
196
+ ```
197
+
198
+ Or skip the file: `GULLIVER_DSN`, `GULLIVER_MAX_QUERY_USD`, `GULLIVER_DAY_USD` and friends.
199
+ Credentials never go in the config: Gulliver uses each engine's own environment variables and credential chains
200
+ (Google ADC, `SNOWFLAKE_PASSWORD`, `PGPASSWORD`, ...). The agent only ever sees connection *names*.
201
+
202
+ ## Safety model
203
+
204
+ - **Read-first.** Only `SELECT` (and `WITH ... SELECT`) runs by default. `DELETE`, `UPDATE`, `DROP`, `TRUNCATE`,
205
+ `ALTER`, `MERGE`, several statements at once and unparsable SQL are refused by a parser, not a regex.
206
+ - **Writes need two keys.** `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` run only when the call says
207
+ `allow_write` **and** a human set `allow_write = true` on that connection. Rehearsals never write.
208
+ - **Budgets fail closed.** Over the per-query, per-day, bytes or runtime limit means `block`. Money above the
209
+ approval limit needs `gulliver approve <receipt_id>`, a CLI command for humans that is not exposed as an MCP tool.
210
+ - **Engine-side caps** where they exist: BigQuery `maximum_bytes_billed`, Snowflake and Postgres statement timeouts,
211
+ a DuckDB interrupt timer.
212
+ - **Local state only.** Receipts, the spend ledger and approvals live in `./.gulliver/` as JSON. No telemetry.
213
+
214
+ ## How the prediction works
215
+
216
+ Each rung measures runtime, output rows and (where the engine reports it) bytes. Gulliver fits
217
+ `metric = a * fraction^k` in log-log space and reads off the value at 100%.
218
+
219
+ - `k ≈ 1`: grows with the data (scans, filters, key joins against a full table).
220
+ - `k ≈ 0`: flat (aggregations). Group counts saturate, so grouped results are capped by what was observed and the planner estimate.
221
+ - `k > 1.3`: grows faster than the data: a join that multiplies rows, a big sort, or both join sides sampled. Flagged.
222
+
223
+ Runtime first subtracts the fixed per-query overhead (measured with `SELECT 1`). Confidence is `high`, `medium`
224
+ or `low`, with a range of x0.7-1.5, x0.5-2 or x0.25-4. After a real run, the receipt shows predicted vs actual.
225
+
226
+ By default only the **largest** table is sampled and the others stay complete, which keeps key joins honest.
227
+ `--sample all` samples every table. For BigQuery the money prediction uses the exact dry-run bytes of the full query.
228
+
229
+ **Limits, honestly:** block (system) sampling is cheap but can be skewed on clustered data; use `--method bernoulli`
230
+ when that matters. Extrapolating 10x always has uncertainty; Gulliver shows it instead of hiding it. Databricks,
231
+ ClickHouse and Spark are not supported yet.
232
+
233
+ ## Pipeline tools
234
+
235
+ These reuse the same connections, budget gate and receipts. Every query they run is generated, parsed,
236
+ estimated and gated like your own queries.
237
+
238
+ ### Impact: what breaks if a column changes
239
+
240
+ ```bash
241
+ gulliver impact raw.events.amount --models models/ # or --manifest target/manifest.json
242
+ ```
243
+ ```
244
+ Changing raw.events.amount affects 4 model(s):
245
+ stg_events: amount <- raw.events.amount
246
+ fct_revenue: revenue <- stg_events.amount
247
+ user_ltv: ltv <- stg_events.amount
248
+ country_report: revenue <- fct_revenue.revenue
249
+ ```
250
+
251
+ Plain `*.sql` files and dbt models (`{{ ref() }}`, `{{ source() }}`, `{{ config() }}`, `{% if %}`) both work.
252
+ Columns used only in filters or joins are reported too, and `SELECT *` models are flagged because their shape
253
+ follows the source. `gulliver lineage --models models/` prints the whole graph.
254
+
255
+ ### Drift: schema changes since last time
256
+
257
+ ```bash
258
+ gulliver snapshot # remember tables, columns and row counts
259
+ gulliver drift --models models/ # later: what changed, and which models it breaks
260
+ gulliver drift --save # accept the new schema
261
+ ```
262
+
263
+ Removed and retyped columns are `warn` and are mapped to impacted models; row counts that move more than 20%
264
+ are listed too.
265
+
266
+ ### Diff: prove two queries return the same rows
267
+
268
+ ```bash
269
+ gulliver diff --left "select * from fct_orders" --right "select * from fct_orders_v2" --key order_id
270
+ gulliver diff --left orders --right orders --key id -c postgres_prod --right-connection snowflake_new
271
+ ```
272
+
273
+ Each side is reduced inside its engine to 256 buckets of (row count, sum of MD5 row hashes) keyed by the key's
274
+ hash, so it scales to billions of rows and costs about one scan per side. Only buckets that differ are opened
275
+ (with a finer second pass when they are big) to list keys that are missing, extra or changed. MD5 is the same on
276
+ every engine, so you can diff across engines; values are compared as text, so cast or round in your queries where
277
+ engines format numbers or timestamps differently. Use it after a refactor, a migration or an incremental switch.
278
+
279
+ ### Translate: SQL between dialects
280
+
281
+ ```bash
282
+ gulliver translate --from snowflake --to bigquery -f model.sql
283
+ gulliver translate --from duckdb --to postgres --check -c pg "select ..." # also preflight on the target
284
+ ```
285
+
286
+ Functions the translator does not know are copied as-is and listed as warnings, so nothing is silently wrong.
287
+
288
+ ### Incremental: stop rebuilding everything
289
+
290
+ ```bash
291
+ gulliver incremental -f models/user_ltv.sql --time-column ts --unique-key user_id --period 1d
292
+ ```
293
+
294
+ | Strategy | When | What it generates |
295
+ |---|---|---|
296
+ | `append` | no aggregation | read only rows newer than the target's max (merge on `unique_key` for late updates) |
297
+ | `partition_overwrite` | grouped by a time grain | dbt `microbatch` config: recompute the recent periods (+ lookback) |
298
+ | `merge_recompute` | grouped (or windowed) by an entity key | recompute only the keys that got new rows, merge on the key |
299
+ | `not_safe` | LIMIT, totals over everything, windows without PARTITION BY | keeps the full rebuild and says why |
300
+
301
+ It also measures the saving on the real source ("each run would read about 172,800 of 5,000,000 rows, 96.5% fewer")
302
+ and, on billed engines, compares dry-run bytes, telling you when you need to partition or cluster the source to
303
+ turn fewer rows into less money. Prove the result with `gulliver diff`.
304
+
305
+ ### Check: data quality in one scan
306
+
307
+ ```bash
308
+ gulliver check events --unique event_id --not-null user_id,amount \
309
+ --accepted "event_type=view|click|cart|buy" --range "amount=0:10000" --fresh ts=24h --max-change 30
310
+ ```
311
+
312
+ All checks become one aggregate query (one scan), budget-gated like any other. Status `fail` (exit code 4)
313
+ when a check fails, so it drops into CI or an orchestrator. Row-count history is kept per target, so
314
+ `--max-change` catches a load that suddenly lost half its rows.
315
+
316
+ ## Scale lint rules
317
+
318
+ | Rule | Name | Severity |
319
+ |---|---|---|
320
+ | GV001 | select-star | warn |
321
+ | GV002 | unbounded-output | warn |
322
+ | GV003 | cartesian-join | error (no condition) / warn (CROSS JOIN) |
323
+ | GV004 | missing-partition-filter | error |
324
+ | GV005 | function-on-filter-column | warn |
325
+ | GV006 | order-without-limit | warn |
326
+ | GV007 | exact-count-distinct | info |
327
+ | GV008 | non-equi-join | warn |
328
+ | GV009 | union-distinct | info |
329
+ | GV010 | scalar-subquery-in-select | info |
330
+ | GV011 | window-without-partition | warn |
331
+ | GV012 | not-in-subquery | info |
332
+
333
+ `gulliver lint` works without any database.
334
+
335
+ ## CLI reference
336
+
337
+ ```
338
+ gulliver status connections, budget, spend today
339
+ gulliver inspect [TABLE] biggest tables, or one table's columns
340
+ gulliver lint SQL [--dialect D] scale lint only
341
+ gulliver preflight SQL estimate + lint + gate, no execution
342
+ gulliver rehearse SQL [--fractions 0.01,0.03,0.1] [--sample largest|all] [--method auto|system|bernoulli]
343
+ gulliver run SQL [--max-rows N] [--allow-write]
344
+ gulliver receipt [ID|latest] [--format markdown|json]
345
+ gulliver approve ID [--note TEXT] (human only)
346
+ gulliver demo [PATH] create a demo DuckDB database
347
+ gulliver lineage --models DIR | --manifest FILE
348
+ gulliver impact TABLE[.COLUMN] --models DIR | --manifest FILE
349
+ gulliver snapshot | drift [--models DIR] [--save]
350
+ gulliver diff --left SQL --right SQL --key k1,k2 [--right-connection NAME] [--columns a,b]
351
+ gulliver translate SQL --to D [--from D] [--check]
352
+ gulliver incremental SQL [--time-column C] [--unique-key K] [--period 1d] [--lookback 1]
353
+ gulliver check TARGET [--where W] [--unique K] [--not-null C] [--accepted C=a|b] [--range C=min:max] [--fresh C=24h] [--min-rows N] [--max-change PCT]
354
+ gulliver tools --format mcp|openai|gemini
355
+ gulliver mcp [--http] start the MCP server
356
+ ```
357
+
358
+ Exit codes: `0` ok, `1` error, `2` blocked, `3` needs rehearsal or approval, `4` a data check failed.
359
+
360
+ SQL can be an argument, `-f file.sql`, or `-` for stdin. Global flags: `--dsn`, `--config`, `-c NAME`, `--json`.
361
+
362
+ ## Development
363
+
364
+ ```bash
365
+ git clone https://github.com/laaielichter/gulliver && cd gulliver
366
+ pip install -e ".[dev]"
367
+ pytest
368
+ ```
369
+
370
+ See [AGENTS.md](AGENTS.md) for the agent-facing guide and [SECURITY.md](SECURITY.md) to report a vulnerability.
371
+
372
+ ## License
373
+
374
+ MIT, see [LICENSE](LICENSE).