metabase-sync 0.1.0__tar.gz

metabase_sync-0.1.0/.gitignore

@@ -0,0 +1,34 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+*.egg
+.venv/
+.python-version
+
+# Build
+build/
+dist/
+*.whl
+*.tar.gz
+
+# uv
+.uv-cache/
+
+# Pytest / coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+
+# Local state / secrets
+.env
+.env.local
+state/.plan.json
+state/.last-apply.json
+
+# Editors
+.vscode/
+.idea/
+*.swp
+.DS_Store

metabase_sync-0.1.0/CHANGELOG.md

@@ -0,0 +1,57 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] — 2026-06-20
+
+### Commands
+
+- `metabase-sync export` — pull the live Metabase instance into a YAML + SQL state tree under `state/`.
+- `metabase-sync plan` — itemised per-resource diff between the on-disk state and the live instance. Writes a human-readable report to stdout and a machine-readable `state/.plan.json` for CI consumption.
+- `metabase-sync apply` — re-derives the diff and executes the necessary POST/PUT calls. Idempotent (a second run is a no-op). Writes `state/.last-apply.json` as an audit log.
+- `metabase-sync index` — debug command listing remote resource counts.
+- `metabase-sync --version` prints the package version.
+
+### Resources covered
+
+- Collections, native and GUI cards (including dashboard-internal cards), dashboards (tabs + dashcards), snippets (with collection routing), pulses (dashboard subscriptions, recipients matched by email), and database manifests (name + engine only — credentials never serialised).
+- Personal collections are intentionally excluded.
+- Out-of-scope resources (alerts, segments, legacy metrics) produce a warning on export so they aren't silently missed.
+
+### Code-first authoring
+
+- Dashboards and pulses reference cards by relative file path (`card_path: ../../cards/foo.sql`) and dashboards by `dashboard_path` — brand-new items can be authored without knowing any Metabase ids ahead of time. Apply allocates ids and writes the new `entity_id` back into the source file.
+
+### Round-trip stability
+
+- Serialises Metabase's classic `legacy_query` form for native cards and accepts either classic or MBQL5 form for GUI cards. The diff path strips volatile keys (`lib/uuid`, `info`, `lib.convert/converted?`) so MBQL5 cards don't churn across exports.
+- SQL bodies are stored byte-faithfully (trailing whitespace, template-tag UUIDs all survive).
+- Atomic file writes (tempfile + `os.replace`) — Ctrl+C never leaves a corrupted YAML.
+
+### Production safety
+
+- **HTTP retries with exponential backoff** for connection errors, 408, 429, 502, 503, 504. Configurable via `HTTP_MAX_RETRIES` (default 3) and `HTTP_RETRY_BACKOFF_S` (default 1.0). `HTTP_TIMEOUT_S` default is 120s so `result_metadata` recomputation on large cards doesn't trip.
+- **Single PUT for dashboard updates** — metadata + tabs + dashcards in one request, so a dashboard can never be left half-applied.
+- **`result_metadata` preserved** for metadata-only card updates so Metabase doesn't re-run the underlying query on every cosmetic edit.
+- **Reference preflight** — apply walks every dashboard and pulse and resolves every `card_path` / `dashboard_path` against the filesystem before any HTTP call.
+- **Optimistic-concurrency check** — plan captures `updated_at` per item; apply refuses to overwrite anything that changed since plan unless `--force` is set.
+- **Pre-apply confirmation prompt** — `apply` re-prints the plan and asks `[y/N]` unless `--yes`, `CI=true`, or stdout is not a TTY.
+- **Loud error on missing pulse recipients** — apply errors with the list of bad emails unless `--allow-missing-recipients` is set.
+- **`--delete` rejected at CLI parse time** with exit code 2 (planned for a future release).
+- **Terraform exit codes**: `plan` exits 2 when changes are detected, 0 when nothing to do, 1 on error.
+
+### Compatibility
+
+- Built and tested against Metabase OSS v0.62.2 (pinned for CI). A non-blocking CI job runs the integration suite against `metabase:latest` to surface upcoming breakage.
+- Supports Python 3.11, 3.12, 3.13.
+
+### Known limitations
+
+- `--delete` (opt-in archival of items absent from disk) is not yet implemented.
+- Permissions, users, and groups are out of scope.
+- Alerts, legacy metrics, and segments are not yet supported (export warns when they exist).

metabase_sync-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 novucs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

metabase_sync-0.1.0/PKG-INFO

@@ -0,0 +1,315 @@
+Metadata-Version: 2.4
+Name: metabase-sync
+Version: 0.1.0
+Summary: Metabase as code: export, plan, and apply dashboards, cards, collections, snippets and pulses via the Metabase REST API.
+Project-URL: Homepage, https://github.com/novucs/metabase-sync
+Project-URL: Repository, https://github.com/novucs/metabase-sync
+Project-URL: Issues, https://github.com/novucs/metabase-sync/issues
+Project-URL: Changelog, https://github.com/novucs/metabase-sync/blob/main/CHANGELOG.md
+Author: novucs
+License-Expression: MIT
+License-File: LICENSE
+Keywords: as-code,cli,dashboards,gitops,iac,metabase
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Version Control
+Classifier: Topic :: System :: Systems Administration
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pydantic-settings>=2.6.0
+Requires-Dist: pydantic>=2.8.0
+Requires-Dist: python-slugify>=8.0.4
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: rich>=15.0.0
+Requires-Dist: tenacity>=9.1.4
+Requires-Dist: typer>=0.12.5
+Description-Content-Type: text/markdown
+
+# metabase-sync
+
+[![PyPI](https://img.shields.io/pypi/v/metabase-sync.svg)](https://pypi.org/project/metabase-sync/)
+[![Python](https://img.shields.io/pypi/pyversions/metabase-sync.svg)](https://pypi.org/project/metabase-sync/)
+[![CI](https://github.com/novucs/metabase-sync/actions/workflows/ci.yml/badge.svg)](https://github.com/novucs/metabase-sync/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Metabase as code. `export`, `plan`, and `apply` your dashboards, cards, collections, snippets and pulses via the regular Metabase REST API. Works with Metabase **OSS** (no Enterprise license required).
+
+```
+$ metabase-sync plan
+
+plan ← state/
+
+collections: 0 create, 0 update, 12 skip
+cards:       1 create, 1 update, 48 skip
+  CREATE  collections/finance/cards/q4-revenue.sql                  Q4 Revenue — SELECT SUM(revenue) FROM finance.q4
+  UPDATE  collections/finance/cards/profit-margin.sql               SQL: 23 → 25 lines, +2 −0
+dashboards:  1 create, 0 update, 8 skip
+  CREATE  collections/finance/dashboards/q4-review/dashboard.yaml   Q4 Review (1 dashcards)
+
+2 create, 1 update, 68 skip
+run `metabase-sync apply` to apply this plan.
+```
+
+## Why this exists
+
+Metabase's [official serialization](https://www.metabase.com/docs/latest/installation-and-operation/serialization) and Remote Sync are gated behind Enterprise / Pro. If you self-host the open-source edition, the `/api/ee/serialization/*` endpoints return 404 and you have no first-party way to put your dashboards under version control.
+
+This tool wraps the plain REST API and gives you the same workflow without the licence wall:
+
+- One YAML/SQL tree per Metabase instance, suitable for git.
+- A `plan` / `apply` loop that reads exactly like Terraform.
+- Round-trip determinism: `export` → `apply` → `export` produces zero diff.
+- Code-first authoring: drop a new `.sql` and a new `dashboard.yaml`, run `apply`, done. No need to know any Metabase ids ahead of time.
+
+## Install
+
+```bash
+uv tool install metabase-sync         # recommended: isolated install
+# or
+pipx install metabase-sync
+# or
+pip install metabase-sync
+```
+
+## Quickstart
+
+```bash
+mkdir my-metabase && cd my-metabase
+cat > .env <<EOF
+METABASE_URL=https://your-instance.example.com
+METABASE_API_KEY=mb_...
+EOF
+metabase-sync export        # creates ./state/
+git init && git add state && git commit -m "import metabase state"
+```
+
+Then your day-to-day:
+
+```bash
+# Edit a .sql or dashboard.yaml in your editor.
+metabase-sync plan          # see exactly what will change
+metabase-sync apply         # write changes
+```
+
+## Commands
+
+| Command | Purpose |
+| --- | --- |
+| `metabase-sync export` | Pull the live instance into the on-disk state tree. |
+| `metabase-sync plan` | Read-only diff. Prints a per-item report and writes `state/.plan.json`. |
+| `metabase-sync apply` | Re-derives the diff and executes it. Idempotent. |
+| `metabase-sync apply --only cards` | Restrict to one resource (`collections,snippets,cards,dashboards,pulses`). |
+| `metabase-sync index` | Debug: print remote resource counts. |
+
+Both `plan` and `apply` re-fetch the live instance, so you can't apply a stale plan against a moved instance.
+
+## Authoring new cards and dashboards in code
+
+You don't need to know any Metabase ids ahead of time. Create the files; apply allocates the ids and writes them back.
+
+### A new card
+
+`state/collections/finance/cards/q4-revenue.sql`:
+
+```sql
+---
+entity_id: null
+name: Q4 Revenue
+description: null
+type: question
+display: scalar
+database: bigquery
+parameters: []
+visualization_settings: {}
+enable_embedding: false
+embedding_params: null
+cache_ttl: null
+archived: false
+template_tags: {}
+---
+---body---
+SELECT SUM(revenue) FROM finance.q4
+```
+
+Run `plan` to see the CREATE line, then `apply`. The `entity_id: null` becomes the server-assigned nanoid in your local file after apply.
+
+### A new dashboard referencing a new card
+
+`state/collections/finance/dashboards/q4-review/dashboard.yaml`:
+
+```yaml
+entity_id: null
+name: Q4 Review
+description: null
+archived: false
+auto_apply_filters: true
+cache_ttl: null
+enable_embedding: false
+embedding_params: null
+position: null
+width: fixed
+parameters: []
+tabs: []
+dashcards:
+  - entity_id: null
+    card_path: ../../cards/q4-revenue.sql       # path relative to this dashboard dir
+    tab_position: null
+    row: 0
+    col: 0
+    size_x: 12
+    size_y: 6
+    parameter_mappings: []
+    visualization_settings: {}
+    series: []
+```
+
+`card_path` is the file path from this dashboard's directory to the card's `.sql` file. Apply resolves it after creating the card, then writes the new entity_id back to your file.
+
+## On-disk layout
+
+```
+state/
+  databases/<name>.yaml         # manifest only — name + engine, NO credentials
+  snippets/<slug>.sql           # YAML frontmatter + raw SQL body
+  collections/
+    <slug>/_collection.yaml
+           /<nested-slug>/_collection.yaml
+                         /cards/<slug>.sql or .yaml
+                         /dashboards/<slug>/dashboard.yaml
+                                           /cards/<slug>.sql   # dashboard-internal cards
+    root/cards/...              # cards directly under the root collection
+  pulses/<slug>.yaml            # dashboard subscriptions
+```
+
+- Native SQL cards: `.sql` file with YAML frontmatter + raw query body.
+- GUI / MBQL cards: `.yaml` file with the full `dataset_query` (classic or MBQL5) inlined.
+- Dashcards: reference cards by `card_path` (relative).
+- Pulses: reference cards by `card_path`, the target dashboard by `dashboard_path`. Recipients are stored by email.
+- Snippets without a collection live in `state/snippets/`; snippets inside a collection live in `state/collections/<...>/snippets/`.
+- Personal collections are filtered out.
+
+## Configuration
+
+| Env var | Required | Default | Notes |
+| --- | --- | --- | --- |
+| `METABASE_URL` | yes | — | e.g. `https://metabase.example.com` |
+| `METABASE_API_KEY` | yes | — | Admin API key (see below) |
+| `STATE_DIR` | no | `state/` | Relative to CWD |
+| `HTTP_TIMEOUT_S` | no | `120` | Per HTTP request (large cards' `result_metadata` recompute can take time) |
+| `HTTP_MAX_RETRIES` | no | `3` | Retries on 408 / 429 / 502 / 503 / 504 / connection errors |
+| `HTTP_RETRY_BACKOFF_S` | no | `1.0` | Base delay; doubles per retry (1s, 2s, 4s) |
+
+A `.env` file in the current working directory is read automatically.
+
+API keys are minted from the Metabase admin UI (Settings → Admin settings → Authentication → API keys). The key needs admin permissions to fetch + write everything `export` and `apply` touch.
+
+## Exit codes
+
+`plan` and `apply` follow the terraform convention so CI pipelines can fan out:
+
+| Code | Meaning |
+| --- | --- |
+| 0 | Success and no changes (or apply finished cleanly) |
+| 1 | Error (HTTP failure, preflight failure, missing recipient, concurrency drift) |
+| 2 | `plan` detected pending changes (informational; not an error) |
+
+## CI/CD recipes
+
+See [`examples/`](examples/) for two GitHub Actions templates:
+
+- [`github-actions-plan-on-pr.yml`](examples/github-actions-plan-on-pr.yml) — comment the plan output on every PR that touches `state/`.
+- [`github-actions-apply-on-merge.yml`](examples/github-actions-apply-on-merge.yml) — apply automatically on merge to `main`, gated by an environment approval.
+
+## Round-trip guarantees
+
+- `export` is deterministic: byte-identical output across runs.
+- `plan` against a freshly-exported tree reports `nothing to do.`.
+- SQL bodies are byte-faithful — trailing whitespace and template-tag UUIDs survive the round-trip.
+
+## Caveats
+
+Before running this on a production Metabase, you should know:
+
+- **Apply overwrites concurrent UI edits unless you re-plan first.** `apply` runs a fresh `plan` and checks the captured `updated_at` for every item it touched. If a UI user has edited an item since you ran `plan`, apply aborts with the list and tells you to re-plan. Pass `--force` to overwrite anyway.
+- **Dashboard contents are a full replacement.** Tabs and dashcards are PUT in one go with client-assigned negative temp ids; the server replaces existing rows and allocates new `dashcard.entity_id`s. Two simultaneous applies race; wrap CI in a concurrency group to serialise them.
+- **Out-of-scope resources are silently NOT synced.** Alerts, segments, legacy metrics, permissions, users and groups are not part of the state tree. `export` prints a warning if it finds any so you don't discover this in production.
+- **Personal collections are excluded.** Cards/dashboards inside a user's personal collection don't appear in the export. Dashboards in shared collections that reference a personal-collection card will fail the reference preflight at plan time, not at apply time.
+- **`--delete` is not yet implemented.** Items removed from `state/` are not auto-archived on apply. Archive them through the UI or directly via the API. Passing `--delete` is rejected with exit code 2.
+
+### Pre-apply backup
+
+`apply --backup-dir <path>` re-exports the live instance to `<path>` before mutating. If apply messes something up, `metabase-sync apply --state-dir <path>` rolls forward against the backup.
+
+```bash
+metabase-sync apply --backup-dir /tmp/pre-apply-$(date +%Y%m%d-%H%M%S)
+```
+
+## Troubleshooting / FAQ
+
+**`401 Unauthorized` on the first call.** Your API key was revoked or you're pointed at the wrong instance. Run `metabase-sync diagnose` — it prints the URL and key length.
+
+**`plan` reports updates I didn't make.** Most often a Metabase version mismatch — server-generated `lib/uuid` values on GUI cards regenerate on every UI save. The tool strips them at diff time; if you still see noise, `metabase-sync export` once to refresh and try again.
+
+**How do I rename a card?** Edit the `name:` in the frontmatter. The `entity_id` stays the same so the rename round-trips cleanly.
+
+**How do I move a card to another collection?** Move the file. The collection that contains the card is determined by its on-disk path; on apply the collection_id is rebound and Metabase moves it.
+
+**How do I delete a card?** Archive it via the Metabase UI for now. The `--delete` flag is not implemented in this release. After archival, re-export and commit.
+
+**`metabase-sync diagnose` to file a bug.** It captures everything we'd ask for in a triage thread. Paste the output into the GitHub issue template.
+
+**My Metabase is on a really old version.** Check the [compatibility band](#compatibility). Versions older than v0.45 are explicitly refused; v0.45–v0.55 will warn but try; v0.55–v0.62 are in our tested band; newer versions warn but proceed.
+
+## Compatibility
+
+| Metabase | Status |
+| --- | --- |
+| <v0.45 | **Refused** — collection API shape too different |
+| v0.45–v0.55 | Warns; proceed at your own risk |
+| v0.55–v0.62 | **Tested** in CI integration matrix |
+| >v0.62 | Warns; we want to hear about issues |
+| `latest` | Non-blocking CI job runs against it to surface upcoming breakage |
+
+We pin `v0.62.2` for the blocking CI job; the matrix in `.github/workflows/ci.yml` runs the same integration suite against `v0.55`, `v0.58`, `v0.60`, and `latest` non-blockingly.
+
+Python: 3.11, 3.12, 3.13.
+
+If something goes wrong, you can `cp -r` the backup over `state/` and re-apply to roll back.
+
+## Security
+
+- The API key is read from the environment (or an `.env` file). Anything that captures the environment — CI logs, shell history (`printenv`), error stack traces from third-party deps that `repr()` settings — can leak it. Tools like [sops](https://github.com/getsops/sops) + `sops exec-env encrypted.env 'metabase-sync apply'` work well.
+- HTTPS verification follows httpx's defaults (CA bundle from `certifi`). The tool does not disable cert verification.
+- `state/.plan.json` and `state/.last-apply.json` include full SQL bodies — they're written under `state/` and should be `.gitignore`d. The default `.gitignore` snippet:
+  ```
+  state/.plan.json
+  state/.last-apply.json
+  ```
+- The tool never serialises database connection credentials (`details` is stripped from `databases/<name>.yaml`).
+
+## Limitations
+
+- `--delete` (opt-in archival of items absent from disk) is not yet implemented.
+- Permissions, users, and groups are out of scope.
+- Alerts, legacy metrics, and segments are not yet supported.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for dev setup, running tests, and the release process.
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Disclaimer
+
+This is an unofficial, community-built tool. Not affiliated with or endorsed by Metabase, Inc. "Metabase" is a trademark of Metabase, Inc.