lazarillo 0.1.0__tar.gz

lazarillo-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+          cache: pip
+      - run: pip install -e ".[dev]"
+      - run: pytest -q
+
+  package:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build twine
+      - run: python -m build
+      - run: twine check --strict dist/*
+      - name: Install the wheel in a clean venv
+        run: |
+          python -m venv /tmp/clean
+          /tmp/clean/bin/pip install dist/*.whl
+          /tmp/clean/bin/lazarillo --version

lazarillo-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,63 @@
+name: Release
+
+# Push a tag like v0.1.0: builds once, publishes to TestPyPI, then to PyPI.
+# Uses PyPI Trusted Publishing, so no tokens are stored. Setup in docs/releasing.md.
+
+on:
+  push:
+    tags: ["v*"]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build twine
+      - name: Check the tag matches the package version
+        run: |
+          version=$(python -c "import re; print(re.search(r'__version__ = \"(.+)\"', open('src/lazarillo/__init__.py').read())[1])")
+          test "v$version" = "$GITHUB_REF_NAME" || { echo "tag $GITHUB_REF_NAME != v$version"; exit 1; }
+      - run: python -m build
+      - run: twine check --strict dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  testpypi:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: testpypi
+      url: https://test.pypi.org/p/lazarillo
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+
+  pypi:
+    needs: testpypi
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/lazarillo
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

lazarillo-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+.lazarillo/
+# example artefacts (regenerated by each example's build script)
+examples/*/data/
+examples/*/*/target/
+examples/*/*/target_dev/
+examples/*/*/logs/
+examples/*/*/dbt_packages/
+.user.yml

lazarillo-0.1.0/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+## 0.1.0
+
+First public release.
+
+- `lazarillo map`, `describe` and `impact`: a data map from the dbt manifest, local or from a
+  dbt Cloud job, with materializations, incremental strategies, tests, lineage and exposures.
+- `lazarillo query`: one read-only statement per call, row-capped, with PII masking and no
+  filesystem access.
+- `lazarillo diff`: row-level diff between warehouse tables and lake files (Delta, Parquet,
+  Iceberg), locally or on S3, AWS Glue and DuckLake.
+- `lazarillo verify`: builds a model in dev, diffs it against prod and reports the blast radius.
+- Warehouses: DuckDB and Redshift (password or IAM, read-only session).
+- `lazarillo init` and `lazarillo mcp`: the same tools over MCP, returning the same Markdown.
+- Tarima Tickets demo in `examples/`, with three planted data problems.

lazarillo-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Guillermo Vizcaíno Román
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lazarillo-0.1.0/PKG-INFO

@@ -0,0 +1,219 @@
+Metadata-Version: 2.5
+Name: lazarillo
+Version: 0.1.0
+Summary: A data harness that guides AI agents through your lakehouse: context, guardrails and verification.
+Project-URL: Homepage, https://github.com/guille-vizcaino/lazarillo
+Project-URL: Documentation, https://github.com/guille-vizcaino/lazarillo#readme
+Project-URL: Issues, https://github.com/guille-vizcaino/lazarillo/issues
+Author-email: Guillermo Vizcaíno Román <hi@guille.me>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai-agents,data-diff,data-engineering,dbt,delta-lake,harness,iceberg,mcp,redshift
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Requires-Python: >=3.11
+Requires-Dist: click>=8.1
+Requires-Dist: duckdb>=1.1
+Requires-Dist: pyarrow>=16
+Requires-Dist: pyyaml>=6
+Provides-Extra: all
+Requires-Dist: boto3>=1.34; extra == 'all'
+Requires-Dist: dbt-duckdb>=1.8; extra == 'all'
+Requires-Dist: deltalake>=0.20; extra == 'all'
+Requires-Dist: mcp>=2; extra == 'all'
+Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
+Requires-Dist: pyiceberg[glue]>=0.8; extra == 'all'
+Requires-Dist: pyiceberg[pyarrow,sql-sqlite]>=0.8; extra == 'all'
+Provides-Extra: dbt
+Requires-Dist: dbt-duckdb>=1.8; extra == 'dbt'
+Provides-Extra: delta
+Requires-Dist: deltalake>=0.20; extra == 'delta'
+Provides-Extra: dev
+Requires-Dist: boto3>=1.34; extra == 'dev'
+Requires-Dist: dbt-duckdb>=1.8; extra == 'dev'
+Requires-Dist: deltalake>=0.20; extra == 'dev'
+Requires-Dist: mcp>=2; extra == 'dev'
+Requires-Dist: moto[server]>=5; extra == 'dev'
+Requires-Dist: psycopg2-binary>=2.9; extra == 'dev'
+Requires-Dist: pyiceberg[glue]>=0.8; extra == 'dev'
+Requires-Dist: pyiceberg[pyarrow,sql-sqlite]>=0.8; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: glue
+Requires-Dist: boto3>=1.34; extra == 'glue'
+Requires-Dist: pyiceberg[glue]>=0.8; extra == 'glue'
+Requires-Dist: pyiceberg[pyarrow,sql-sqlite]>=0.8; extra == 'glue'
+Provides-Extra: iceberg
+Requires-Dist: pyiceberg[pyarrow,sql-sqlite]>=0.8; extra == 'iceberg'
+Provides-Extra: mcp
+Requires-Dist: mcp>=2; extra == 'mcp'
+Provides-Extra: redshift
+Requires-Dist: boto3>=1.34; extra == 'redshift'
+Requires-Dist: psycopg2-binary>=2.9; extra == 'redshift'
+Provides-Extra: s3
+Requires-Dist: boto3>=1.34; extra == 's3'
+Description-Content-Type: text/markdown
+
+# Lazarillo
+
+> **Lazarillo** (n.): the boy who guides a blind man across 16th-century Spain in
+> *Lazarillo de Tormes*. Your AI agent can't see your warehouse. Lazarillo guides it.
+
+Lazarillo is a **data harness**: the layer between an AI agent and your lakehouse that
+gives the agent **context**, enforces **guardrails** and **verifies** every change before it
+reaches production.
+
+SaaS products used to wrap a database in a UI. The equivalent today is a harness: it wraps
+your data, your conventions and your workflows so that an agent can work on them
+safely. Lazarillo is an opinionated harness for analytics engineering. It is local-first,
+vendor-neutral, and exposed as a CLI and an MCP server.
+
+## Why
+
+Agents already write decent SQL. They fail in three other ways:
+
+| The agent… | Lazarillo gives it… |
+|---|---|
+| doesn't know what `fct_orders` means, how it's built, or who reads it | **Context**: a data map compiled from the dbt manifest, covering materializations, incremental strategies, tests, lineage and exposures such as Power BI dashboards. |
+| can `DROP` a table, scan 3 TB or print customer emails | **Guardrails** in code, not in the prompt: read-only, one statement per call, row caps, PII masking, no filesystem access. |
+| says "done ✅" without proof | **Verification**: build the change in a dev schema, diff it row by row against prod, and show the blast radius. |
+
+The picaresque twist: Lázaro famously tricks the blind man. So the harness **never trusts
+the agent's word**. Every claim comes with a diff.
+
+## Use it on your project
+
+```bash
+pip install "lazarillo[all]"
+cd your-project
+lazarillo init                # writes lazarillo.yml and finds your dbt project, if any
+lazarillo query "select 42"
+```
+
+`lazarillo init` prints the `.mcp.json` block that hands the harness to your agent. The core
+only needs DuckDB; pick extras for the rest of your stack: `delta`, `iceberg`, `s3`, `glue`,
+`redshift`, `dbt`, `mcp`, or `all`. Without dbt you still get `query` and `diff`; add a dbt project to unlock `map`,
+`describe`, `impact` and `verify`. If production runs in dbt Cloud, the map can come from
+your production job instead ([docs/dbt-cloud.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/dbt-cloud.md)).
+
+## Try the demo (2 minutes, no cloud account)
+
+```bash
+git clone https://github.com/guille-vizcaino/lazarillo && cd lazarillo
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[all]"
+python examples/tarima-tickets/build_demo.py   # builds Tarima Tickets, a fictional concert ticketing company
+cd examples/tarima-tickets && lazarillo map
+```
+
+The demo lives in [`examples/tarima-tickets`](https://github.com/guille-vizcaino/lazarillo/tree/main/examples/tarima-tickets) and uses Lazarillo
+exactly like your project would: its own `lazarillo.yml`, dbt project and data. It mirrors a common AWS setup (ticketing system → S3 landing in Delta, migrating
+to Iceberg → warehouse → dbt → Power BI) on DuckDB. Three problems are planted in it,
+the kind that reach production every week:
+
+### 1. Source vs landing: "are we extracting everything?"
+
+```console
+$ lazarillo diff src.orders landing.orders -k order_id
+| rows       | 3,996 | 3,961 |
+| only here  |    35 |     0 |
+Schema drift
+- `sales_channel` only in left
+```
+
+The extract uses an `updated_at` watermark. Box office sales reach the ticketing system
+days later, when the venue syncs, so they fall behind the watermark and are never landed.
+The source also grew a column the extract ignores.
+
+### 2. Delta → Iceberg migration: "is the new table identical?"
+
+```console
+$ lazarillo diff delta:data/landing/delta/orders iceberg:landing.orders -k order_id
+| changed (same key) | 1,081 |
+- `ordered_at`: 1,081 rows
+| 2 | 2026-09-11 05:47:14 | 2026-09-11 03:47:14 |
+```
+
+Same row count and same keys, so a count-based check passes. The new writer stores
+timestamps in UTC, and a row-level diff catches the shift.
+
+### 3. Incremental drift: "is prod what a full rebuild would give?"
+
+```console
+$ lazarillo verify fct_orders
+Built `dev.fct_orders` and compared it with `analytics.fct_orders`.
+- `status`: 38 rows      (paid → refunded)
+### Blast radius
+- downstream models: fct_event_revenue
+- **Revenue by event** (dashboard, owner: Finance)
+⚠️ Dev and prod differ. Review the diff above before merging; 1 exposure(s) will see the change.
+```
+
+The incremental filter only picks up *new* orders, so later refunds never reach
+production. The revenue-by-event dashboard overstates what each promoter is owed.
+`verify` is also the loop an agent uses after editing a model: change the filter to
+`updated_at`, verify, and show the diff.
+
+## Use it with an agent
+
+```bash
+lazarillo -c examples/tarima-tickets/lazarillo.yml mcp     # MCP over stdio
+```
+
+This repo ships a `.mcp.json` wired to the demo, so opening it in Claude Code gives the agent these tools:
+`data_map`, `describe_model`, `impact`, `query`, `diff` and `verify`. The server also sends
+instructions that tell the agent to verify before it claims success.
+
+Try: *"Finance says the payout for Los Tejados in Barcelona looks too high. Find out why and propose a fix."*
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `lazarillo init [DIR]` | Write a starter `lazarillo.yml` and print the MCP config |
+| `lazarillo map` | Sources, models, materializations, exposures |
+| `lazarillo describe MODEL` | Columns, tests, SQL, upstream and downstream |
+| `lazarillo impact MODEL` | Downstream models and dashboards |
+| `lazarillo query "SQL"` | One read-only statement, capped and masked |
+| `lazarillo diff LEFT RIGHT -k KEY [--where]` | Row-level diff between any two relations |
+| `lazarillo verify MODEL [--where]` | Build in dev, diff against prod, report blast radius |
+| `lazarillo mcp` | Serve all of the above over MCP |
+
+Relations can be `schema.table`, `attached_db.table`, `delta:<path>`,
+`iceberg:<namespace.table>` or `parquet:<glob>`. Paths can be local or `s3://`, Iceberg
+catalogs can be SQL, AWS Glue or REST, and DuckLakes can be attached next to the
+warehouse. See [docs/aws.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/aws.md). The warehouse itself can be a DuckDB file or
+Redshift, with IAM auth; see [docs/redshift.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/redshift.md).
+
+## Scope
+
+**In scope:** everything from landing to consumption, i.e. landing tables, dbt models and
+the dashboards that read them.
+
+**Out of scope:** orchestration and scheduling (Step Functions, Airflow, EventBridge…).
+Lazarillo doesn't run your pipelines. It checks what they produce.
+
+## Roadmap
+
+- [x] Data map from the dbt manifest, including exposures
+- [x] Read-only guardrails, row caps and PII masking
+- [x] Row-level diff across warehouse, Delta, Iceberg and Parquet
+- [x] `verify`: dev build, diff and blast radius
+- [x] MCP server
+- [x] Lake in S3 (Delta, Parquet, Iceberg), AWS Glue catalog and DuckLake
+- [x] Redshift as the warehouse, with IAM auth ([docs/redshift.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/redshift.md))
+- [x] dbt Cloud: read the manifest of the production job
+- [ ] `--defer` builds so `verify` doesn't rebuild parents
+- [ ] Cost guardrails (`EXPLAIN`-based scan budget)
+- [ ] **`lazarillo checkride`**: a benchmark of real data-engineering tasks, scored with
+      and without the harness ([docs/checkride.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/checkride.md))
+
+## License
+
+MIT © Guillermo Vizcaíno Román. The demo company and all its data are fictional.

lazarillo-0.1.0/README.md

@@ -0,0 +1,157 @@
+# Lazarillo
+
+> **Lazarillo** (n.): the boy who guides a blind man across 16th-century Spain in
+> *Lazarillo de Tormes*. Your AI agent can't see your warehouse. Lazarillo guides it.
+
+Lazarillo is a **data harness**: the layer between an AI agent and your lakehouse that
+gives the agent **context**, enforces **guardrails** and **verifies** every change before it
+reaches production.
+
+SaaS products used to wrap a database in a UI. The equivalent today is a harness: it wraps
+your data, your conventions and your workflows so that an agent can work on them
+safely. Lazarillo is an opinionated harness for analytics engineering. It is local-first,
+vendor-neutral, and exposed as a CLI and an MCP server.
+
+## Why
+
+Agents already write decent SQL. They fail in three other ways:
+
+| The agent… | Lazarillo gives it… |
+|---|---|
+| doesn't know what `fct_orders` means, how it's built, or who reads it | **Context**: a data map compiled from the dbt manifest, covering materializations, incremental strategies, tests, lineage and exposures such as Power BI dashboards. |
+| can `DROP` a table, scan 3 TB or print customer emails | **Guardrails** in code, not in the prompt: read-only, one statement per call, row caps, PII masking, no filesystem access. |
+| says "done ✅" without proof | **Verification**: build the change in a dev schema, diff it row by row against prod, and show the blast radius. |
+
+The picaresque twist: Lázaro famously tricks the blind man. So the harness **never trusts
+the agent's word**. Every claim comes with a diff.
+
+## Use it on your project
+
+```bash
+pip install "lazarillo[all]"
+cd your-project
+lazarillo init                # writes lazarillo.yml and finds your dbt project, if any
+lazarillo query "select 42"
+```
+
+`lazarillo init` prints the `.mcp.json` block that hands the harness to your agent. The core
+only needs DuckDB; pick extras for the rest of your stack: `delta`, `iceberg`, `s3`, `glue`,
+`redshift`, `dbt`, `mcp`, or `all`. Without dbt you still get `query` and `diff`; add a dbt project to unlock `map`,
+`describe`, `impact` and `verify`. If production runs in dbt Cloud, the map can come from
+your production job instead ([docs/dbt-cloud.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/dbt-cloud.md)).
+
+## Try the demo (2 minutes, no cloud account)
+
+```bash
+git clone https://github.com/guille-vizcaino/lazarillo && cd lazarillo
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[all]"
+python examples/tarima-tickets/build_demo.py   # builds Tarima Tickets, a fictional concert ticketing company
+cd examples/tarima-tickets && lazarillo map
+```
+
+The demo lives in [`examples/tarima-tickets`](https://github.com/guille-vizcaino/lazarillo/tree/main/examples/tarima-tickets) and uses Lazarillo
+exactly like your project would: its own `lazarillo.yml`, dbt project and data. It mirrors a common AWS setup (ticketing system → S3 landing in Delta, migrating
+to Iceberg → warehouse → dbt → Power BI) on DuckDB. Three problems are planted in it,
+the kind that reach production every week:
+
+### 1. Source vs landing: "are we extracting everything?"
+
+```console
+$ lazarillo diff src.orders landing.orders -k order_id
+| rows       | 3,996 | 3,961 |
+| only here  |    35 |     0 |
+Schema drift
+- `sales_channel` only in left
+```
+
+The extract uses an `updated_at` watermark. Box office sales reach the ticketing system
+days later, when the venue syncs, so they fall behind the watermark and are never landed.
+The source also grew a column the extract ignores.
+
+### 2. Delta → Iceberg migration: "is the new table identical?"
+
+```console
+$ lazarillo diff delta:data/landing/delta/orders iceberg:landing.orders -k order_id
+| changed (same key) | 1,081 |
+- `ordered_at`: 1,081 rows
+| 2 | 2026-09-11 05:47:14 | 2026-09-11 03:47:14 |
+```
+
+Same row count and same keys, so a count-based check passes. The new writer stores
+timestamps in UTC, and a row-level diff catches the shift.
+
+### 3. Incremental drift: "is prod what a full rebuild would give?"
+
+```console
+$ lazarillo verify fct_orders
+Built `dev.fct_orders` and compared it with `analytics.fct_orders`.
+- `status`: 38 rows      (paid → refunded)
+### Blast radius
+- downstream models: fct_event_revenue
+- **Revenue by event** (dashboard, owner: Finance)
+⚠️ Dev and prod differ. Review the diff above before merging; 1 exposure(s) will see the change.
+```
+
+The incremental filter only picks up *new* orders, so later refunds never reach
+production. The revenue-by-event dashboard overstates what each promoter is owed.
+`verify` is also the loop an agent uses after editing a model: change the filter to
+`updated_at`, verify, and show the diff.
+
+## Use it with an agent
+
+```bash
+lazarillo -c examples/tarima-tickets/lazarillo.yml mcp     # MCP over stdio
+```
+
+This repo ships a `.mcp.json` wired to the demo, so opening it in Claude Code gives the agent these tools:
+`data_map`, `describe_model`, `impact`, `query`, `diff` and `verify`. The server also sends
+instructions that tell the agent to verify before it claims success.
+
+Try: *"Finance says the payout for Los Tejados in Barcelona looks too high. Find out why and propose a fix."*
+
+## Commands
+
+| command | what it does |
+|---|---|
+| `lazarillo init [DIR]` | Write a starter `lazarillo.yml` and print the MCP config |
+| `lazarillo map` | Sources, models, materializations, exposures |
+| `lazarillo describe MODEL` | Columns, tests, SQL, upstream and downstream |
+| `lazarillo impact MODEL` | Downstream models and dashboards |
+| `lazarillo query "SQL"` | One read-only statement, capped and masked |
+| `lazarillo diff LEFT RIGHT -k KEY [--where]` | Row-level diff between any two relations |
+| `lazarillo verify MODEL [--where]` | Build in dev, diff against prod, report blast radius |
+| `lazarillo mcp` | Serve all of the above over MCP |
+
+Relations can be `schema.table`, `attached_db.table`, `delta:<path>`,
+`iceberg:<namespace.table>` or `parquet:<glob>`. Paths can be local or `s3://`, Iceberg
+catalogs can be SQL, AWS Glue or REST, and DuckLakes can be attached next to the
+warehouse. See [docs/aws.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/aws.md). The warehouse itself can be a DuckDB file or
+Redshift, with IAM auth; see [docs/redshift.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/redshift.md).
+
+## Scope
+
+**In scope:** everything from landing to consumption, i.e. landing tables, dbt models and
+the dashboards that read them.
+
+**Out of scope:** orchestration and scheduling (Step Functions, Airflow, EventBridge…).
+Lazarillo doesn't run your pipelines. It checks what they produce.
+
+## Roadmap
+
+- [x] Data map from the dbt manifest, including exposures
+- [x] Read-only guardrails, row caps and PII masking
+- [x] Row-level diff across warehouse, Delta, Iceberg and Parquet
+- [x] `verify`: dev build, diff and blast radius
+- [x] MCP server
+- [x] Lake in S3 (Delta, Parquet, Iceberg), AWS Glue catalog and DuckLake
+- [x] Redshift as the warehouse, with IAM auth ([docs/redshift.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/redshift.md))
+- [x] dbt Cloud: read the manifest of the production job
+- [ ] `--defer` builds so `verify` doesn't rebuild parents
+- [ ] Cost guardrails (`EXPLAIN`-based scan budget)
+- [ ] **`lazarillo checkride`**: a benchmark of real data-engineering tasks, scored with
+      and without the harness ([docs/checkride.md](https://github.com/guille-vizcaino/lazarillo/blob/main/docs/checkride.md))
+
+## License
+
+MIT © Guillermo Vizcaíno Román. The demo company and all its data are fictional.

lazarillo-0.1.0/docs/aws.md

@@ -0,0 +1,101 @@
+# Reading a lake in S3, Glue and DuckLake
+
+Lazarillo reads lake tables from local disk or from S3. Iceberg tables come from any
+catalog pyiceberg supports (SQL, AWS Glue, REST...), and DuckLakes can be attached next
+to the warehouse. None of this needs a cloud account to try; see *Testing without AWS*.
+
+```bash
+pip install "lazarillo[delta,glue]"   # glue brings iceberg and boto3
+```
+
+## Configuration
+
+```yaml
+warehouse:
+  path: warehouse.duckdb
+
+attach:
+  src: source.duckdb                       # another DuckDB file, as before
+  lake: ducklake:lake/metadata.ducklake    # a DuckLake (files local or in S3)
+  # lake: {type: ducklake, path: "postgres:dbname=lake", data_path: s3://my-lake/ducklake/}
+
+landing:
+  # Where `delta:` and `parquet:` refs may read. Anything else is refused.
+  # Defaults to the folder holding lazarillo.yml.
+  locations:
+    - s3://my-lake/landing/
+  s3:                         # every key is optional
+    region: eu-west-1
+    profile: lake-readonly    # or leave it out and use the AWS default chain
+    # endpoint: http://localhost:9000    # MinIO, LocalStack...
+  iceberg_catalog:            # passed to pyiceberg's load_catalog
+    name: glue
+    type: glue
+    # glue.id: "123456789012"            # another account's catalog
+```
+
+Then refer to tables as usual:
+
+```console
+$ lazarillo diff delta:s3://my-lake/landing/delta/orders iceberg:landing.orders -k order_id
+$ lazarillo diff parquet:s3://my-lake/landing/raw/dt=2026-09-*/*.parquet landing.orders -k order_id
+$ lazarillo query "select status, count(*) from lake.orders group by 1"
+```
+
+## Credentials
+
+Lazarillo resolves credentials once with boto3 and hands the same identity to every
+reader (deltalake, pyarrow, pyiceberg and DuckDB for DuckLake), so SSO and assume-role
+profiles work everywhere. The order is:
+
+1. `access_key_id` / `secret_access_key` in `landing.s3` (avoid committing them).
+2. `profile` in `landing.s3`.
+3. The AWS default chain: environment variables, `AWS_PROFILE`, ~/.aws, instance or task role.
+
+Keys in `landing.s3` also fill in the Glue catalog's credentials and region. Anything
+set on `iceberg_catalog` itself (`glue.region`, `s3.endpoint`...) wins.
+
+Lazarillo only reads. A policy like this one is enough:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {"Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"],
+     "Resource": ["arn:aws:s3:::my-lake", "arn:aws:s3:::my-lake/landing/*"]},
+    {"Effect": "Allow", "Action": ["glue:GetDatabase", "glue:GetTable", "glue:GetTables"],
+     "Resource": "*"}
+  ]
+}
+```
+
+## Guardrails
+
+- SQL still cannot touch files or the network (`enable_external_access = false`).
+  Delta, Parquet and Iceberg tables are opened in Python and handed to DuckDB as Arrow.
+- `delta:` and `parquet:` paths must sit inside `landing.locations`. `..` segments,
+  symlinks that leave the folder and other buckets are refused.
+- Iceberg tables are reached by name through the catalog, never by path.
+- A DuckLake is attached read-only. DuckDB reads its Parquet files itself, so the
+  lake's data folder, and only that folder, stays readable from SQL.
+
+## Limits
+
+- Iceberg tables are loaded into memory before the diff. Keep them to what fits, or
+  compare a snapshot or partition you copy elsewhere, until pushdown lands.
+- DuckLake needs DuckDB's `ducklake` extension, and `httpfs` when its files live in S3.
+  DuckDB downloads them on first use; install them once if you work offline.
+- The warehouse can be a DuckDB file or Redshift; see [redshift.md](redshift.md).
+
+## Testing without AWS
+
+`pytest` runs every S3 and Glue test against [moto](https://github.com/getmoto/moto)'s
+server, in process. To run them against a real S3-compatible server too:
+
+```bash
+docker run -d -p 9000:9000 -e MINIO_ROOT_USER=lazarillo -e MINIO_ROOT_PASSWORD=lazarillo-secret bitnamilegacy/minio
+LAZARILLO_TEST_S3_ENDPOINT=http://127.0.0.1:9000 LAZARILLO_TEST_S3_KEY=lazarillo \
+  LAZARILLO_TEST_S3_SECRET=lazarillo-secret pytest -m minio
+```
+
+MinIO has no Glue, so there the Iceberg tests use a SQL catalog whose data lives in the bucket.

lazarillo-0.1.0/docs/checkride.md

@@ -0,0 +1,26 @@
+# checkride (design notes)
+
+A *checkride* is the practical exam a pilot flies with an examiner before getting a rating.
+`lazarillo checkride` will put agents through the same thing on data work.
+
+## Idea
+
+Every task starts from a broken but realistic platform (the Tarima Tickets demo in
+`examples/tarima-tickets` plus a scenario patch). The agent has to reach a verifiable end state. Each task is scored by
+checks the harness can run, never by an LLM judge:
+
+| task | the agent must… | pass when |
+|---|---|---|
+| `incremental-drift` | find why payouts are overstated and fix `fct_orders` | `verify fct_orders` reports identical |
+| `late-arrivals` | explain the source/landing gap | answer names the watermark and the 35 missing box office orders |
+| `iceberg-parity` | decide whether the Iceberg table can replace Delta | answer is "no" and names the timestamp shift |
+| `safe-refactor` | rename a column in staging without breaking the dashboard | dbt build passes, exposure columns unchanged |
+
+Each task runs twice, **with** the harness (MCP tools) and **without** it (raw SQL
+access), for every model under test. The headline number is how much the harness helps.
+
+## Open questions
+
+- Scenario format: a YAML file plus a patch applied to the demo project?
+- Isolation: a fresh copy of `examples/tarima-tickets/data` per run.
+- Reporting: Markdown table plus JSON, easy to publish.