gulliver-mcp 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- gulliver_mcp-0.1.0/.gitignore +22 -0
- gulliver_mcp-0.1.0/AGENTS.md +67 -0
- gulliver_mcp-0.1.0/CHANGELOG.md +26 -0
- gulliver_mcp-0.1.0/LICENSE +21 -0
- gulliver_mcp-0.1.0/PKG-INFO +374 -0
- gulliver_mcp-0.1.0/README.md +340 -0
- gulliver_mcp-0.1.0/llms.txt +69 -0
- gulliver_mcp-0.1.0/pyproject.toml +74 -0
- gulliver_mcp-0.1.0/skills/gulliver/SKILL.md +65 -0
- gulliver_mcp-0.1.0/src/gulliver/__init__.py +7 -0
- gulliver_mcp-0.1.0/src/gulliver/checks.py +173 -0
- gulliver_mcp-0.1.0/src/gulliver/cli.py +364 -0
- gulliver_mcp-0.1.0/src/gulliver/config.py +209 -0
- gulliver_mcp-0.1.0/src/gulliver/core.py +598 -0
- gulliver_mcp-0.1.0/src/gulliver/demo.py +40 -0
- gulliver_mcp-0.1.0/src/gulliver/diffing.py +138 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/__init__.py +38 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/base.py +166 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/bigquery.py +181 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/duckdb_engine.py +283 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/postgres.py +207 -0
- gulliver_mcp-0.1.0/src/gulliver/engines/snowflake.py +191 -0
- gulliver_mcp-0.1.0/src/gulliver/extrapolate.py +126 -0
- gulliver_mcp-0.1.0/src/gulliver/features.py +457 -0
- gulliver_mcp-0.1.0/src/gulliver/incremental.py +274 -0
- gulliver_mcp-0.1.0/src/gulliver/lineage.py +306 -0
- gulliver_mcp-0.1.0/src/gulliver/lint.py +260 -0
- gulliver_mcp-0.1.0/src/gulliver/mcp_server.py +340 -0
- gulliver_mcp-0.1.0/src/gulliver/receipts.py +139 -0
- gulliver_mcp-0.1.0/src/gulliver/sampling.py +51 -0
- gulliver_mcp-0.1.0/src/gulliver/sqlsafety.py +129 -0
- gulliver_mcp-0.1.0/src/gulliver/store.py +154 -0
- gulliver_mcp-0.1.0/tests/conftest.py +28 -0
- gulliver_mcp-0.1.0/tests/test_cli_and_mcp.py +50 -0
- gulliver_mcp-0.1.0/tests/test_config.py +66 -0
- gulliver_mcp-0.1.0/tests/test_engines_offline.py +92 -0
- gulliver_mcp-0.1.0/tests/test_features.py +207 -0
- gulliver_mcp-0.1.0/tests/test_flow_duckdb.py +171 -0
- gulliver_mcp-0.1.0/tests/test_lint.py +69 -0
- gulliver_mcp-0.1.0/tests/test_postgres_live.py +129 -0
- gulliver_mcp-0.1.0/tests/test_sampling_and_extrapolate.py +75 -0
- gulliver_mcp-0.1.0/tests/test_sqlsafety.py +51 -0
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
*.egg-info/
|
|
4
|
+
build/
|
|
5
|
+
dist/
|
|
6
|
+
.venv/
|
|
7
|
+
venv/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.ruff_cache/
|
|
10
|
+
.mypy_cache/
|
|
11
|
+
.coverage
|
|
12
|
+
htmlcov/
|
|
13
|
+
|
|
14
|
+
# Gulliver local state and data
|
|
15
|
+
.gulliver/
|
|
16
|
+
*.duckdb
|
|
17
|
+
*.duckdb.wal
|
|
18
|
+
gulliver.toml
|
|
19
|
+
|
|
20
|
+
.DS_Store
|
|
21
|
+
.idea/
|
|
22
|
+
.vscode/
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Gulliver for agents
|
|
2
|
+
|
|
3
|
+
Gulliver checks SQL before it touches big data. Use it whenever you are about to run SQL you wrote against a
|
|
4
|
+
real table (DuckDB, BigQuery, Snowflake or Postgres), especially when you do not know how big the tables are.
|
|
5
|
+
|
|
6
|
+
## Tools (MCP) and the matching CLI commands
|
|
7
|
+
|
|
8
|
+
| MCP tool | CLI | Use it to |
|
|
9
|
+
|---|---|---|
|
|
10
|
+
| `gulliver_status` | `gulliver status --json` | see connections, budget limits and money spent today |
|
|
11
|
+
| `gulliver_inspect` | `gulliver inspect [table] --json` | find the big tables, partitions and columns (free) |
|
|
12
|
+
| `gulliver_lint` | `gulliver lint "SQL" --json` | check SQL for scale problems without a database |
|
|
13
|
+
| `gulliver_preflight` | `gulliver preflight "SQL" --json` | estimate bytes, rows and money without running |
|
|
14
|
+
| `gulliver_rehearse` | `gulliver rehearse "SQL" --json` | run on 1%, 3%, 10% samples and predict the full run |
|
|
15
|
+
| `gulliver_run` | `gulliver run "SQL" --json` | run the full query inside the budget |
|
|
16
|
+
| `gulliver_receipt` | `gulliver receipt latest` | give the human proof of what happened |
|
|
17
|
+
| `gulliver_impact` | `gulliver impact table.column --models DIR` | see which models and columns a change breaks |
|
|
18
|
+
| `gulliver_drift` | `gulliver drift --models DIR` | see schema changes since the last snapshot and what they break |
|
|
19
|
+
| `gulliver_diff` | `gulliver diff --left Q --right Q --key id` | prove two queries/tables return the same rows |
|
|
20
|
+
| `gulliver_translate` | `gulliver translate SQL --to bigquery` | move SQL between DuckDB, Postgres, BigQuery, Snowflake |
|
|
21
|
+
| `gulliver_incremental` | `gulliver incremental SQL --time-column ts` | make a full-rebuild model incremental, with the saving |
|
|
22
|
+
| `gulliver_check` | `gulliver check table --unique id ...` | data quality in one scan (status `fail` if a check fails) |
|
|
23
|
+
|
|
24
|
+
## The loop
|
|
25
|
+
|
|
26
|
+
1. `gulliver_inspect` once per task, so you know which tables are big and how they are partitioned.
|
|
27
|
+
2. Write the SQL. Select only the columns you need. Filter on the partition column. Aggregate before returning rows.
|
|
28
|
+
3. `gulliver_preflight`. Read `status` and `next_step`.
|
|
29
|
+
4. If `next_step` asks for it (or the estimate is large or only `heuristic`), `gulliver_rehearse`.
|
|
30
|
+
5. `gulliver_run` with the exact same SQL. The preview has at most `max_output_rows` rows; `actual.rows` is the total.
|
|
31
|
+
6. Offer `gulliver_receipt` to the human.
|
|
32
|
+
|
|
33
|
+
## Pipeline work
|
|
34
|
+
|
|
35
|
+
- Before changing or dropping a column in a model or source: `gulliver_impact` with the models folder.
|
|
36
|
+
- After refactoring a model, migrating it to another engine or making it incremental: `gulliver_diff` between the
|
|
37
|
+
old and the new query with the key. Say "identical" only when `identical` is true.
|
|
38
|
+
- When a model rebuilds everything each run: `gulliver_incremental`; present the strategy, the SQL and the saving,
|
|
39
|
+
and verify with `gulliver_diff`.
|
|
40
|
+
- Before trusting a table or after a load: `gulliver_check` (unique keys, not null, freshness). On partitioned
|
|
41
|
+
tables pass `where` with a partition filter.
|
|
42
|
+
- Periodically, or when a pipeline breaks: `gulliver_drift` with the models folder.
|
|
43
|
+
|
|
44
|
+
## Status values
|
|
45
|
+
|
|
46
|
+
| status | meaning | what to do |
|
|
47
|
+
|---|---|---|
|
|
48
|
+
| `ok` | within budget | continue |
|
|
49
|
+
| `warn` | allowed, with risks listed in `lint` / `warnings` | fix them if cheap, else continue |
|
|
50
|
+
| `block` | over a limit, a lint error, or a refused statement | change the SQL; do not work around the gate |
|
|
51
|
+
| `needs_rehearsal` | big enough that a rehearsal is required | call `gulliver_rehearse` with the same SQL |
|
|
52
|
+
| `needs_approval` | costs more than the auto-approve limit | tell the human the `receipt_id`; they run `gulliver approve <id>` |
|
|
53
|
+
| `skipped` | rehearsal not useful (tiny data) | call `gulliver_run` |
|
|
54
|
+
| `fail` | a data quality check failed | report which checks failed; do not use the data as if it were fine |
|
|
55
|
+
| `error` | engine or config problem | read `error`, fix, retry once |
|
|
56
|
+
|
|
57
|
+
## Reading a rehearsal
|
|
58
|
+
|
|
59
|
+
`prediction.<metric>.value` is the expected full-run value; `low` and `high` give the range.
|
|
60
|
+
`exponent` near 1 means it grows with the data; above 1.3 means it grows faster (a join that multiplies rows).
|
|
61
|
+
`confidence` is `high`, `medium` or `low`. Treat `low` as a rough range, not a number.
|
|
62
|
+
|
|
63
|
+
## Rules you cannot change
|
|
64
|
+
|
|
65
|
+
- Only one statement per call. Only SELECT, or CREATE/INSERT ... SELECT when a human enabled writes.
|
|
66
|
+
- You cannot approve your own runs. Approval is a human CLI command.
|
|
67
|
+
- Credentials and budgets are set by the human in `gulliver.toml` or environment variables.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 - 2026-11-01
|
|
4
|
+
|
|
5
|
+
First release.
|
|
6
|
+
|
|
7
|
+
Query rehearsal:
|
|
8
|
+
- Workflow: `status`, `inspect`, `lint`, `preflight`, `rehearse`, `run`, `receipt`, `approve`.
|
|
9
|
+
- Rehearsal ladder (1% → 3% → 10%), power-law extrapolation with confidence ranges, detection of super-linear growth,
|
|
10
|
+
saturation cap for grouped results.
|
|
11
|
+
- 12 scale lint rules (GV001-GV012).
|
|
12
|
+
- Budget gate: per query, per day, bytes, runtime, rehearsal requirement and human approval.
|
|
13
|
+
|
|
14
|
+
Pipeline tools:
|
|
15
|
+
- `impact` / `lineage`: column-level lineage over SQL model folders (dbt Jinja supported) or a dbt manifest.
|
|
16
|
+
- `snapshot` / `drift`: schema drift per connection, mapped to impacted models.
|
|
17
|
+
- `diff`: key-based data diff inside the engines with MD5 buckets and a zoom-in pass; works across engines.
|
|
18
|
+
- `translate`: SQL between DuckDB, Postgres, BigQuery and Snowflake, with warnings and an optional target preflight.
|
|
19
|
+
- `incremental`: append / partition_overwrite / merge_recompute / not_safe advice, generated SQL and dbt config,
|
|
20
|
+
measured savings.
|
|
21
|
+
- `check`: unique, not null, accepted values, range, freshness and row-count change in one scan (exit code 4 on fail).
|
|
22
|
+
|
|
23
|
+
Engines and integration:
|
|
24
|
+
- DuckDB and PostgreSQL tested (Postgres live in CI), BigQuery and Snowflake beta (tested with fakes).
|
|
25
|
+
- MCP server (stdio and Streamable HTTP), CLI with `--json`, OpenAI and Gemini tool export.
|
|
26
|
+
- Claude Code plugin and agent skill.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Roel
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,374 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: gulliver-mcp
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Test on tiny data before you run on giant data. A dress rehearsal for data agents: estimate the cost, rehearse on 1% -> 10%, predict 100%, and only go full scale within budget.
|
|
5
|
+
Project-URL: Homepage, https://github.com/laaielichter/gulliver
|
|
6
|
+
Project-URL: Issues, https://github.com/laaielichter/gulliver/issues
|
|
7
|
+
Author: laaielichter
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: ai-agents,bigquery,cost,data-engineering,duckdb,llm,mcp,postgres,snowflake,sql
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Topic :: Database
|
|
15
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Requires-Dist: duckdb>=1.1
|
|
18
|
+
Requires-Dist: mcp>=1.10
|
|
19
|
+
Requires-Dist: sqlglot>=25.0
|
|
20
|
+
Requires-Dist: tomli>=2.0; python_version < '3.11'
|
|
21
|
+
Provides-Extra: all
|
|
22
|
+
Requires-Dist: google-cloud-bigquery>=3.20; extra == 'all'
|
|
23
|
+
Requires-Dist: psycopg[binary]>=3.1; extra == 'all'
|
|
24
|
+
Requires-Dist: snowflake-connector-python>=3.10; extra == 'all'
|
|
25
|
+
Provides-Extra: bigquery
|
|
26
|
+
Requires-Dist: google-cloud-bigquery>=3.20; extra == 'bigquery'
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
29
|
+
Provides-Extra: postgres
|
|
30
|
+
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
|
|
31
|
+
Provides-Extra: snowflake
|
|
32
|
+
Requires-Dist: snowflake-connector-python>=3.10; extra == 'snowflake'
|
|
33
|
+
Description-Content-Type: text/markdown
|
|
34
|
+
|
|
35
|
+
# Gulliver
|
|
36
|
+
|
|
37
|
+
<!-- mcp-name: io.github.laaielichter/gulliver -->
|
|
38
|
+
|
|
39
|
+
**Test on tiny data before you run on giant data.**
|
|
40
|
+
|
|
41
|
+
Gulliver gives AI agents a dress rehearsal for SQL: estimate the cost, rehearse on 1% → 3% → 10% of the data, predict what 100% will do, and only run the full query inside a budget. Local-first, open source, MCP + CLI for any agent and any LLM.
|
|
42
|
+
|
|
43
|
+
[](https://github.com/laaielichter/gulliver/actions/workflows/ci.yml)
|
|
44
|
+

|
|
45
|
+

|
|
46
|
+
|
|
47
|
+
> In *Gulliver's Travels*, Gulliver visits Lilliput, a land of tiny people, and Brobdingnag, a land of giants.
|
|
48
|
+
> Gulliver does the same with your data: it tries your query in Lilliput first, and only walks into Brobdingnag
|
|
49
|
+
> when it knows what will happen there.
|
|
50
|
+
|
|
51
|
+
## Why
|
|
52
|
+
|
|
53
|
+
Agents write SQL that works on a small table and breaks on a big one. They do not know that `events` has
|
|
54
|
+
5 billion rows, that a join multiplies rows, or that a forgotten partition filter scans every day since 2019.
|
|
55
|
+
Benchmarks score agents on *is the answer right*, never on *what did it cost* or *does it still work at 100x*.
|
|
56
|
+
One unlucky query can burn a monthly budget.
|
|
57
|
+
|
|
58
|
+
Gulliver sits between the agent and the warehouse:
|
|
59
|
+
|
|
60
|
+
| Step | Tool | What happens |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| 1 | `gulliver_inspect` | Which tables are big? Rows, bytes, partitions, columns. Metadata only, free. |
|
|
63
|
+
| 2 | `gulliver_preflight` | Bytes, rows and money **before** running, plus **scale lint** (12 rules) and the budget gate. |
|
|
64
|
+
| 3 | `gulliver_rehearse` | Runs the query on 1%, 3% and 10% samples, fits a growth curve and **predicts the full run** with a confidence range. Catches joins that multiply rows. |
|
|
65
|
+
| 4 | `gulliver_run` | Runs the full query only inside the budget. Otherwise answers `needs_rehearsal` or `needs_approval`. |
|
|
66
|
+
| 5 | `gulliver_receipt` | Markdown proof for the human: what was estimated, rehearsed, approved and run, and predicted vs actual. |
|
|
67
|
+
|
|
68
|
+
Every answer has a `status` and a `next_step`, so even small models know what to do next.
|
|
69
|
+
|
|
70
|
+
And for the pipeline around the query (details in [Pipeline tools](#pipeline-tools)):
|
|
71
|
+
|
|
72
|
+
| Tool | Problem it solves |
|
|
73
|
+
|---|---|
|
|
74
|
+
| `gulliver_impact` | "If I change or drop this column, what breaks?" Column-level lineage over SQL models or a dbt manifest. |
|
|
75
|
+
| `gulliver_drift` | Schema changed under you: columns removed, retyped or added since the last snapshot, plus the models they break. |
|
|
76
|
+
| `gulliver_diff` | "Does my refactored / migrated / incremental model return the same rows?" Key-based data diff inside the engine, also across engines. |
|
|
77
|
+
| `gulliver_translate` | SQL between DuckDB, Postgres, BigQuery and Snowflake, with warnings and an optional preflight on the target. |
|
|
78
|
+
| `gulliver_incremental` | A model rebuilds everything every run. Is incremental safe, what is the SQL / dbt config, and how many rows does it skip? |
|
|
79
|
+
| `gulliver_check` | Data quality in one cheap scan: unique keys, not null, accepted values, ranges, freshness, row-count jumps. |
|
|
80
|
+
|
|
81
|
+
## Quick start (2 minutes, no warehouse needed)
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
pip install git+https://github.com/laaielichter/gulliver
|
|
85
|
+
gulliver demo # creates gulliver_demo.duckdb with 5M events
|
|
86
|
+
export GULLIVER_DSN=gulliver_demo.duckdb
|
|
87
|
+
|
|
88
|
+
gulliver inspect
|
|
89
|
+
gulliver preflight "select country, count(*) from events e join users u on e.user_id = u.user_id group by 1"
|
|
90
|
+
gulliver lint "select * from events e, users u" # GV003: join without a condition -> blocked
|
|
91
|
+
gulliver rehearse "select a.event_id, b.event_id from events a join events b on a.user_id = b.user_id"
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The last one is the classic agent mistake: a self-join on a non-unique key. The rehearsal sees the output grow
|
|
95
|
+
with exponent ~1.8 instead of ~1 and warns that the full result has more than 100 million rows, before you run it.
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
| Sample | Runtime | Output rows |
|
|
99
|
+
|--------|----------|-------------|
|
|
100
|
+
| 1% | 0.0202s | 41,706 |
|
|
101
|
+
| 3% | 0.0484s | 237,422 |
|
|
102
|
+
| 10% | 0.3909s | 2,438,395 |
|
|
103
|
+
|
|
104
|
+
Prediction for 100%: output rows 134,525,654 (range 67,262,827 - 269,051,308)
|
|
105
|
+
Growth exponent output_rows: 1.77 -> grows faster than the data
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Add `--json` to any command for machine-readable output. Exit codes: `0` ok, `1` error, `2` blocked,
|
|
109
|
+
`3` needs rehearsal or approval, `4` a data check failed, so it also works as a CI check.
|
|
110
|
+
|
|
111
|
+
## Use it from your agent (MCP)
|
|
112
|
+
|
|
113
|
+
Gulliver is an MCP server (stdio or Streamable HTTP). Until it is on PyPI, run it straight from GitHub with
|
|
114
|
+
[uv](https://docs.astral.sh/uv/):
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn path/to/warehouse.duckdb
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
**Claude Code**
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
claude mcp add gulliver -- uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn /abs/path/data.duckdb
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Or install it as a plugin (MCP server + skill):
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
/plugin marketplace add laaielichter/gulliver
|
|
130
|
+
/plugin install gulliver@gulliver
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
**Claude Desktop, Cursor, Windsurf, VS Code and other MCP clients** (JSON config):
|
|
134
|
+
|
|
135
|
+
```json
|
|
136
|
+
{
|
|
137
|
+
"mcpServers": {
|
|
138
|
+
"gulliver": {
|
|
139
|
+
"command": "uvx",
|
|
140
|
+
"args": ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
**Codex CLI** (`~/.codex/config.toml`):
|
|
147
|
+
|
|
148
|
+
```toml
|
|
149
|
+
[mcp_servers.gulliver]
|
|
150
|
+
command = "uvx"
|
|
151
|
+
args = ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
**Gemini CLI** (`~/.gemini/settings.json`): same `mcpServers` block as above.
|
|
155
|
+
|
|
156
|
+
**Any HTTP client or remote agent:** `gulliver-mcp --http --port 8765` serves `http://127.0.0.1:8765/mcp`.
|
|
157
|
+
|
|
158
|
+
**No MCP?** Every tool is also a CLI command with `--json`, and `gulliver tools --format openai` or
|
|
159
|
+
`--format gemini` prints ready-made function-calling definitions for the OpenAI or Gemini APIs.
|
|
160
|
+
|
|
161
|
+
## Engines
|
|
162
|
+
|
|
163
|
+
| Engine | Estimate before running | Sampling | Money model | Status |
|
|
164
|
+
|---|---|---|---|---|
|
|
165
|
+
| DuckDB (files, Parquet, CSV, in-memory) | EXPLAIN + metadata (heuristic) | `TABLESAMPLE` system/bernoulli, seeded | free (time is the budget) | tested in CI |
|
|
166
|
+
| BigQuery | dry run: **exact** bytes | `TABLESAMPLE SYSTEM` | on-demand $/TiB, plus a hard `maximum_bytes_billed` cap | beta, tested with fakes |
|
|
167
|
+
| Snowflake | `EXPLAIN USING JSON`: bytes and partitions (upper bound) | `SAMPLE SYSTEM ... SEED` | warehouse credits x runtime | beta, tested with fakes |
|
|
168
|
+
| PostgreSQL | `EXPLAIN (FORMAT JSON)` (heuristic), partition-aware sizes | `TABLESAMPLE ... REPEATABLE` | self-hosted (time and bytes are the budget) | tested in CI against a live Postgres 16 |
|
|
169
|
+
|
|
170
|
+
Install the drivers you need: `pip install "gulliver-mcp[bigquery]"`, `[snowflake]`, `[postgres]` or `[all]`.
|
|
171
|
+
Every estimate says how much to trust it: `exact`, `upper_bound` or `heuristic`. A heuristic is never called exact.
|
|
172
|
+
|
|
173
|
+
## Configuration
|
|
174
|
+
|
|
175
|
+
Copy [`gulliver.example.toml`](gulliver.example.toml) to `gulliver.toml`:
|
|
176
|
+
|
|
177
|
+
```toml
|
|
178
|
+
default_connection = "local"
|
|
179
|
+
|
|
180
|
+
[connections.local]
|
|
181
|
+
engine = "duckdb"
|
|
182
|
+
path = "warehouse.duckdb"
|
|
183
|
+
|
|
184
|
+
[connections.bq]
|
|
185
|
+
dsn = "bigquery://my-project?location=EU"
|
|
186
|
+
dataset = "analytics"
|
|
187
|
+
price_per_tib_usd = 6.25
|
|
188
|
+
|
|
189
|
+
[budget]
|
|
190
|
+
max_query_usd = 1.00 # hard stop for one query
|
|
191
|
+
approval_over_usd = 0.25 # above this a human runs `gulliver approve <receipt_id>`
|
|
192
|
+
rehearse_over_usd = 0.10 # above this a rehearsal is required first
|
|
193
|
+
day_usd = 10.00 # total per day (local ledger)
|
|
194
|
+
max_runtime_s = 900
|
|
195
|
+
max_output_rows = 200 # rows handed back to the agent
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Or skip the file: `GULLIVER_DSN`, `GULLIVER_MAX_QUERY_USD`, `GULLIVER_DAY_USD` and friends.
|
|
199
|
+
Credentials never go in the config: Gulliver uses each engine's own environment variables and credential chains
|
|
200
|
+
(Google ADC, `SNOWFLAKE_PASSWORD`, `PGPASSWORD`, ...). The agent only ever sees connection *names*.
|
|
201
|
+
|
|
202
|
+
## Safety model
|
|
203
|
+
|
|
204
|
+
- **Read-first.** Only `SELECT` (and `WITH ... SELECT`) runs by default. `DELETE`, `UPDATE`, `DROP`, `TRUNCATE`,
|
|
205
|
+
`ALTER`, `MERGE`, several statements at once and unparsable SQL are refused by a parser, not a regex.
|
|
206
|
+
- **Writes need two keys.** `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` run only when the call says
|
|
207
|
+
`allow_write` **and** a human set `allow_write = true` on that connection. Rehearsals never write.
|
|
208
|
+
- **Budgets fail closed.** Over the per-query, per-day, bytes or runtime limit means `block`. Money above the
|
|
209
|
+
approval limit needs `gulliver approve <receipt_id>`, a CLI command for humans that is not exposed as an MCP tool.
|
|
210
|
+
- **Engine-side caps** where they exist: BigQuery `maximum_bytes_billed`, Snowflake and Postgres statement timeouts,
|
|
211
|
+
a DuckDB interrupt timer.
|
|
212
|
+
- **Local state only.** Receipts, the spend ledger and approvals live in `./.gulliver/` as JSON. No telemetry.
|
|
213
|
+
|
|
214
|
+
## How the prediction works
|
|
215
|
+
|
|
216
|
+
Each rung measures runtime, output rows and (where the engine reports it) bytes. Gulliver fits
|
|
217
|
+
`metric = a * fraction^k` in log-log space and reads off the value at 100%.
|
|
218
|
+
|
|
219
|
+
- `k ≈ 1`: grows with the data (scans, filters, key joins against a full table).
|
|
220
|
+
- `k ≈ 0`: flat (aggregations). Group counts saturate, so grouped results are capped by what was observed and the planner estimate.
|
|
221
|
+
- `k > 1.3`: grows faster than the data: a join that multiplies rows, a big sort, or both join sides sampled. Flagged.
|
|
222
|
+
|
|
223
|
+
Runtime first subtracts the fixed per-query overhead (measured with `SELECT 1`). Confidence is `high`, `medium`
|
|
224
|
+
or `low`, with a range of x0.7-1.5, x0.5-2 or x0.25-4. After a real run, the receipt shows predicted vs actual.
|
|
225
|
+
|
|
226
|
+
By default only the **largest** table is sampled and the others stay complete, which keeps key joins honest.
|
|
227
|
+
`--sample all` samples every table. For BigQuery the money prediction uses the exact dry-run bytes of the full query.
|
|
228
|
+
|
|
229
|
+
**Limits, honestly:** block (system) sampling is cheap but can be skewed on clustered data; use `--method bernoulli`
|
|
230
|
+
when that matters. Extrapolating 10x always has uncertainty; Gulliver shows it instead of hiding it. Databricks,
|
|
231
|
+
ClickHouse and Spark are not supported yet.
|
|
232
|
+
|
|
233
|
+
## Pipeline tools
|
|
234
|
+
|
|
235
|
+
These reuse the same connections, budget gate and receipts. Every query they run is generated, parsed,
|
|
236
|
+
estimated and gated like your own queries.
|
|
237
|
+
|
|
238
|
+
### Impact: what breaks if a column changes
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
gulliver impact raw.events.amount --models models/ # or --manifest target/manifest.json
|
|
242
|
+
```
|
|
243
|
+
```
|
|
244
|
+
Changing raw.events.amount affects 4 model(s):
|
|
245
|
+
stg_events: amount <- raw.events.amount
|
|
246
|
+
fct_revenue: revenue <- stg_events.amount
|
|
247
|
+
user_ltv: ltv <- stg_events.amount
|
|
248
|
+
country_report: revenue <- fct_revenue.revenue
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
Plain `*.sql` files and dbt models (`{{ ref() }}`, `{{ source() }}`, `{{ config() }}`, `{% if %}`) both work.
|
|
252
|
+
Columns used only in filters or joins are reported too, and `SELECT *` models are flagged because their shape
|
|
253
|
+
follows the source. `gulliver lineage --models models/` prints the whole graph.
|
|
254
|
+
|
|
255
|
+
### Drift: schema changes since last time
|
|
256
|
+
|
|
257
|
+
```bash
|
|
258
|
+
gulliver snapshot # remember tables, columns and row counts
|
|
259
|
+
gulliver drift --models models/ # later: what changed, and which models it breaks
|
|
260
|
+
gulliver drift --save # accept the new schema
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Removed and retyped columns are `warn` and are mapped to impacted models; row counts that move more than 20%
|
|
264
|
+
are listed too.
|
|
265
|
+
|
|
266
|
+
### Diff: prove two queries return the same rows
|
|
267
|
+
|
|
268
|
+
```bash
|
|
269
|
+
gulliver diff --left "select * from fct_orders" --right "select * from fct_orders_v2" --key order_id
|
|
270
|
+
gulliver diff --left orders --right orders --key id -c postgres_prod --right-connection snowflake_new
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
Each side is reduced inside its engine to 256 buckets of (row count, sum of MD5 row hashes) keyed by the key's
|
|
274
|
+
hash, so it scales to billions of rows and costs about one scan per side. Only buckets that differ are opened
|
|
275
|
+
(with a finer second pass when they are big) to list keys that are missing, extra or changed. MD5 is the same on
|
|
276
|
+
every engine, so you can diff across engines; values are compared as text, so cast or round in your queries where
|
|
277
|
+
engines format numbers or timestamps differently. Use it after a refactor, a migration or an incremental switch.
|
|
278
|
+
|
|
279
|
+
### Translate: SQL between dialects
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
gulliver translate --from snowflake --to bigquery -f model.sql
|
|
283
|
+
gulliver translate --from duckdb --to postgres --check -c pg "select ..." # also preflight on the target
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
Functions the translator does not know are copied as-is and listed as warnings, so nothing is silently wrong.
|
|
287
|
+
|
|
288
|
+
### Incremental: stop rebuilding everything
|
|
289
|
+
|
|
290
|
+
```bash
|
|
291
|
+
gulliver incremental -f models/user_ltv.sql --time-column ts --unique-key user_id --period 1d
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
| Strategy | When | What it generates |
|
|
295
|
+
|---|---|---|
|
|
296
|
+
| `append` | no aggregation | read only rows newer than the target's max (merge on `unique_key` for late updates) |
|
|
297
|
+
| `partition_overwrite` | grouped by a time grain | dbt `microbatch` config: recompute the recent periods (+ lookback) |
|
|
298
|
+
| `merge_recompute` | grouped (or windowed) by an entity key | recompute only the keys that got new rows, merge on the key |
|
|
299
|
+
| `not_safe` | LIMIT, totals over everything, windows without PARTITION BY | keeps the full rebuild and says why |
|
|
300
|
+
|
|
301
|
+
It also measures the saving on the real source ("each run would read about 172,800 of 5,000,000 rows, 96.5% fewer")
|
|
302
|
+
and, on billed engines, compares dry-run bytes, telling you when you need to partition or cluster the source to
|
|
303
|
+
turn fewer rows into less money. Prove the result with `gulliver diff`.
|
|
304
|
+
|
|
305
|
+
### Check: data quality in one scan
|
|
306
|
+
|
|
307
|
+
```bash
|
|
308
|
+
gulliver check events --unique event_id --not-null user_id,amount \
|
|
309
|
+
--accepted "event_type=view|click|cart|buy" --range "amount=0:10000" --fresh ts=24h --max-change 30
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
All checks become one aggregate query (one scan), budget-gated like any other. Status `fail` (exit code 4)
|
|
313
|
+
when a check fails, so it drops into CI or an orchestrator. Row-count history is kept per target, so
|
|
314
|
+
`--max-change` catches a load that suddenly lost half its rows.
|
|
315
|
+
|
|
316
|
+
## Scale lint rules
|
|
317
|
+
|
|
318
|
+
| Rule | Name | Severity |
|
|
319
|
+
|---|---|---|
|
|
320
|
+
| GV001 | select-star | warn |
|
|
321
|
+
| GV002 | unbounded-output | warn |
|
|
322
|
+
| GV003 | cartesian-join | error (no condition) / warn (CROSS JOIN) |
|
|
323
|
+
| GV004 | missing-partition-filter | error |
|
|
324
|
+
| GV005 | function-on-filter-column | warn |
|
|
325
|
+
| GV006 | order-without-limit | warn |
|
|
326
|
+
| GV007 | exact-count-distinct | info |
|
|
327
|
+
| GV008 | non-equi-join | warn |
|
|
328
|
+
| GV009 | union-distinct | info |
|
|
329
|
+
| GV010 | scalar-subquery-in-select | info |
|
|
330
|
+
| GV011 | window-without-partition | warn |
|
|
331
|
+
| GV012 | not-in-subquery | info |
|
|
332
|
+
|
|
333
|
+
`gulliver lint` works without any database.
|
|
334
|
+
|
|
335
|
+
## CLI reference
|
|
336
|
+
|
|
337
|
+
```
|
|
338
|
+
gulliver status connections, budget, spend today
|
|
339
|
+
gulliver inspect [TABLE] biggest tables, or one table's columns
|
|
340
|
+
gulliver lint SQL [--dialect D] scale lint only
|
|
341
|
+
gulliver preflight SQL estimate + lint + gate, no execution
|
|
342
|
+
gulliver rehearse SQL [--fractions 0.01,0.03,0.1] [--sample largest|all] [--method auto|system|bernoulli]
|
|
343
|
+
gulliver run SQL [--max-rows N] [--allow-write]
|
|
344
|
+
gulliver receipt [ID|latest] [--format markdown|json]
|
|
345
|
+
gulliver approve ID [--note TEXT] (human only)
|
|
346
|
+
gulliver demo [PATH] create a demo DuckDB database
|
|
347
|
+
gulliver lineage --models DIR | --manifest FILE
|
|
348
|
+
gulliver impact TABLE[.COLUMN] --models DIR | --manifest FILE
|
|
349
|
+
gulliver snapshot | drift [--models DIR] [--save]
|
|
350
|
+
gulliver diff --left SQL --right SQL --key k1,k2 [--right-connection NAME] [--columns a,b]
|
|
351
|
+
gulliver translate SQL --to D [--from D] [--check]
|
|
352
|
+
gulliver incremental SQL [--time-column C] [--unique-key K] [--period 1d] [--lookback 1]
|
|
353
|
+
gulliver check TARGET [--where W] [--unique K] [--not-null C] [--accepted C=a|b] [--range C=min:max] [--fresh C=24h] [--min-rows N] [--max-change PCT]
|
|
354
|
+
gulliver tools --format mcp|openai|gemini
|
|
355
|
+
gulliver mcp [--http] start the MCP server
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
Exit codes: `0` ok, `1` error, `2` blocked, `3` needs rehearsal or approval, `4` a data check failed.
|
|
359
|
+
|
|
360
|
+
SQL can be an argument, `-f file.sql`, or `-` for stdin. Global flags: `--dsn`, `--config`, `-c NAME`, `--json`.
|
|
361
|
+
|
|
362
|
+
## Development
|
|
363
|
+
|
|
364
|
+
```bash
|
|
365
|
+
git clone https://github.com/laaielichter/gulliver && cd gulliver
|
|
366
|
+
pip install -e ".[dev]"
|
|
367
|
+
pytest
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
See [AGENTS.md](AGENTS.md) for the agent-facing guide and [SECURITY.md](SECURITY.md) to report a vulnerability.
|
|
371
|
+
|
|
372
|
+
## License
|
|
373
|
+
|
|
374
|
+
MIT, see [LICENSE](LICENSE).
|