knack-elt 0.2.1__tar.gz

knack_elt-0.2.1/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.3
+Name: knack-elt
+Version: 0.2.1
+Summary: Generic ELT pipeline: extract any Knack application into DuckDB or MotherDuck with dlt, keeping full SCD2 history.
+Requires-Dist: dlt[duckdb]>=1.30.0
+Requires-Dist: httpx[http2]>=0.28.1
+Requires-Dist: knack-sleuth>=0.7.1
+Requires-Dist: pydantic-settings>=2.15.0
+Requires-Dist: rich>=15.0.0
+Requires-Dist: typer>=0.27.1
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# KnackELT
+
+**Get your data out of Knack and into a real database — on a schedule, read-only, with full history.**
+
+Knack is a good place to *run* a business and a poor place to *remember* one. Every plan caps
+how many records you can hold, there is no SQL and no aggregates, and once a record is deleted
+it is gone.
+
+KnackELT copies every record out through Knack's own REST API and keeps **every version of every
+row**. Nothing is ever overwritten, so the warehouse can still answer questions about records
+your app no longer has. It is built on [dlt](https://dlthub.com), and it never writes back to
+Knack.
+
+## How it fits together
+
+```mermaid
+flowchart TB
+    subgraph src["SOURCE OF RECORD"]
+        app["<b>Knack App</b><br/>where your team works"]
+        api["<b>Knack REST API</b><br/>Knack's own data API"]
+        app --> api
+    end
+
+    subgraph ing["INGESTION — this repo"]
+        elt["<b>KnackELT</b><br/>pulls every record,<br/>never writes back"]
+    end
+
+    subgraph wh["DATA WAREHOUSE / DATABASE"]
+        hist["<b>Complete history</b><br/>every version of every record —<br/>including ones deleted in Knack"]
+        rep["<b>Reporting tables</b><br/>tidied into a shape you can<br/>filter, sort and add up"]
+        hist -->|"modeled for reporting"| rep
+    end
+
+    subgraph bi["BI / DATA TOOLS"]
+        dash["<b>Dashboards</b><br/>look up, drill down"]
+        adhoc["<b>Ad-hoc + export</b><br/>new questions,<br/>Excel and CSV out"]
+    end
+
+    cron["<b>Scheduled run</b><br/>daily cron or CI job"]
+    backup["<b>Offsite backup</b> — optional<br/>S3-compatible object storage:<br/>a third copy, outside both<br/>Knack and the warehouse"]
+
+    api -->|"read-only"| elt
+    cron -.->|"triggers"| elt
+    elt -->|"keeps every version"| hist
+    rep -->|"SQL"| dash
+    rep -->|"SQL"| adhoc
+    hist -.->|"optional"| backup
+
+    classDef keep stroke:#d97706,stroke-width:3px
+    classDef opt stroke-dasharray:5 5
+    class hist keep
+    class backup opt
+    linkStyle 4 stroke:#d97706,stroke-width:2px
+```
+
+**This repo is the `INGESTION` box.** Everything flows one way: KnackELT reads through the same
+REST API your app already exposes, so it cannot alter or break anything in Knack. The amber box
+is the point of the exercise — your app deletes records to stay under its limit, and the
+warehouse keeps them anyway.
+
+The other boxes are deliberately generic. KnackELT loads into anything
+[dlt supports as a destination](https://dlthub.com/docs/dlt-ecosystem/destinations/), and any
+BI tool that speaks SQL to that destination will do. For a concrete, working combination of all
+four — MotherDuck, dbt and Preset, orchestrated by a daily GitHub Actions job — see
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
+
+## What it does
+
+- **Discovers your schema.** Reads the Knack application metadata and builds a resource per
+  object, so there is no table list to maintain. Add an object in Knack and the next run picks
+  it up.
+- **Gives you readable column names.** `field_43` becomes `event_name`, slugified from the field
+  label you already chose in the builder. Knack's own row id is loaded as `record_id`, so a
+  field you named `id` keeps the `id` column it was named for.
+- **Cleans what the API hands back.** Empty strings become `NULL` in numeric fields, boolean
+  fields get the default declared in Knack, and malformed JSON becomes `NULL` instead of
+  failing the load.
+- **Keeps history.** Loads with dlt's SCD2 merge strategy keyed on the Knack record id, so an
+  edit retires the old row and appends a new one. Tables are kept flat
+  (`max_table_nesting=0`) — one table per Knack object, no nested child tables.
+
+## Quick start
+
+Requires Python 3.13+ and [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync
+
+export KNACK_APP_ID=your_app_id
+export KNACK_API_KEY=your_rest_api_key
+
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID"
+```
+
+Names are derived from your app's slug, so a second app never lands on the first one's tables:
+database `knack_{slug}_data`, dataset `{slug}`, pipeline `knack_{slug}_pipeline`.
+
+### Destinations
+
+`--destination local` (the default) writes a DuckDB file — nothing to sign up for, so a fresh
+clone can be pointed at a Knack app and produce a queryable warehouse immediately. The file
+lands at `./tests/data/knack_{slug}_data.duckdb` unless you pass `--db-path`; the resolved
+path is printed on every run.
+
+```bash
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID" --db-path ~/knack.duckdb
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID" --destination motherduck
+```
+
+`--destination motherduck` loads to `md:///knack_{slug}_data` and needs `motherduck_api_key`
+in the environment. Both destinations also write the run's `_load_info` and `_trace` tables.
+
+### Other flags
+
+| Flag | What it does |
+| --- | --- |
+| `--api-key` | Knack REST API key, if you would rather not set `KNACK_API_KEY` |
+| `--refresh-metadata` | Re-fetch app metadata instead of reusing knack-sleuth's 24h on-disk cache |
+| `--skip-unreadable` | Log and continue past objects that fail *before yielding any row* (typically no read permission). An object that fails partway through still aborts the run — loading a partial batch would retire live SCD2 rows as if the missing records had been deleted in Knack. |
+
+## Configuration
+
+Read from the environment or a `.env` file via `pydantic-settings`
+([`src/knack_elt/config.py`](src/knack_elt/config.py)):
+
+| Variable | Purpose |
+| --- | --- |
+| `KNACK_APP_ID` | Knack application id — also the default for `--app-id` |
+| `KNACK_API_KEY` | Knack REST API key, sent as `X-Knack-REST-API-Key` |
+| `motherduck_api_key` | MotherDuck token, when the destination is MotherDuck |
+
+## Querying what you get
+
+Because loads are SCD2, a record's history is several rows sharing one `record_id`, tagged with
+`_dlt_valid_from` and `_dlt_valid_to`. Two flags are worth deriving up front — conflating them
+is the most common way to get a wrong answer:
+
+```sql
+with flagged as (
+    select
+        *,
+        row_number() over (partition by record_id order by _dlt_valid_from desc) = 1
+            as latest_version,      -- one row per record
+        _dlt_valid_to is null as is_live_in_knack   -- still in the app?
+    from your_dataset.some_table
+)
+select * from flagged where latest_version
+```
+
+A record deleted in Knack survives only as a *retired* row, so filtering on
+`_dlt_valid_to is null` alone silently drops exactly the history you built the warehouse for.
+And aggregating without `latest_version` double-counts, because every past version is still a
+row. The [architecture doc](docs/ARCHITECTURE.md#4-scd2-row-lifecycle) works through both.
+
+> **One caveat on `is_live_in_knack`.** If an object returns *zero* records, dlt has nothing to
+> load for that table and the merge never runs, so rows loaded earlier keep `_dlt_valid_to is
+> null` and still read as live. Emptying an object in Knack is therefore invisible to the flag —
+> a table whose row count stops moving is worth checking against the app.
+
+## Documentation
+
+- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — the reference architecture in plain language
+  and in technical detail, the pipeline internals, a run sequence, and the SCD2 row model with
+  the query patterns it requires. Also available as a [PDF](docs/ARCHITECTURE.pdf).
+
+The PDF is generated from the markdown rather than maintained alongside it. After editing
+the diagrams, rebuild it with `uv run scripts/build_architecture_pdf.py` (needs node and
+Chrome) so the two don't drift apart.
+
+## Related
+
+- [dlt](https://dlthub.com) — the load framework this is built on
+- `knack-sleuth` — Knack application metadata models and schema export, used here to read your
+  app's structure
+
+## License
+
+GPL-3.0. See [LICENSE](LICENSE).

knack_elt-0.2.1/README.md

@@ -0,0 +1,178 @@
+# KnackELT
+
+**Get your data out of Knack and into a real database — on a schedule, read-only, with full history.**
+
+Knack is a good place to *run* a business and a poor place to *remember* one. Every plan caps
+how many records you can hold, there is no SQL and no aggregates, and once a record is deleted
+it is gone.
+
+KnackELT copies every record out through Knack's own REST API and keeps **every version of every
+row**. Nothing is ever overwritten, so the warehouse can still answer questions about records
+your app no longer has. It is built on [dlt](https://dlthub.com), and it never writes back to
+Knack.
+
+## How it fits together
+
+```mermaid
+flowchart TB
+    subgraph src["SOURCE OF RECORD"]
+        app["<b>Knack App</b><br/>where your team works"]
+        api["<b>Knack REST API</b><br/>Knack's own data API"]
+        app --> api
+    end
+
+    subgraph ing["INGESTION — this repo"]
+        elt["<b>KnackELT</b><br/>pulls every record,<br/>never writes back"]
+    end
+
+    subgraph wh["DATA WAREHOUSE / DATABASE"]
+        hist["<b>Complete history</b><br/>every version of every record —<br/>including ones deleted in Knack"]
+        rep["<b>Reporting tables</b><br/>tidied into a shape you can<br/>filter, sort and add up"]
+        hist -->|"modeled for reporting"| rep
+    end
+
+    subgraph bi["BI / DATA TOOLS"]
+        dash["<b>Dashboards</b><br/>look up, drill down"]
+        adhoc["<b>Ad-hoc + export</b><br/>new questions,<br/>Excel and CSV out"]
+    end
+
+    cron["<b>Scheduled run</b><br/>daily cron or CI job"]
+    backup["<b>Offsite backup</b> — optional<br/>S3-compatible object storage:<br/>a third copy, outside both<br/>Knack and the warehouse"]
+
+    api -->|"read-only"| elt
+    cron -.->|"triggers"| elt
+    elt -->|"keeps every version"| hist
+    rep -->|"SQL"| dash
+    rep -->|"SQL"| adhoc
+    hist -.->|"optional"| backup
+
+    classDef keep stroke:#d97706,stroke-width:3px
+    classDef opt stroke-dasharray:5 5
+    class hist keep
+    class backup opt
+    linkStyle 4 stroke:#d97706,stroke-width:2px
+```
+
+**This repo is the `INGESTION` box.** Everything flows one way: KnackELT reads through the same
+REST API your app already exposes, so it cannot alter or break anything in Knack. The amber box
+is the point of the exercise — your app deletes records to stay under its limit, and the
+warehouse keeps them anyway.
+
+The other boxes are deliberately generic. KnackELT loads into anything
+[dlt supports as a destination](https://dlthub.com/docs/dlt-ecosystem/destinations/), and any
+BI tool that speaks SQL to that destination will do. For a concrete, working combination of all
+four — MotherDuck, dbt and Preset, orchestrated by a daily GitHub Actions job — see
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
+
+## What it does
+
+- **Discovers your schema.** Reads the Knack application metadata and builds a resource per
+  object, so there is no table list to maintain. Add an object in Knack and the next run picks
+  it up.
+- **Gives you readable column names.** `field_43` becomes `event_name`, slugified from the field
+  label you already chose in the builder. Knack's own row id is loaded as `record_id`, so a
+  field you named `id` keeps the `id` column it was named for.
+- **Cleans what the API hands back.** Empty strings become `NULL` in numeric fields, boolean
+  fields get the default declared in Knack, and malformed JSON becomes `NULL` instead of
+  failing the load.
+- **Keeps history.** Loads with dlt's SCD2 merge strategy keyed on the Knack record id, so an
+  edit retires the old row and appends a new one. Tables are kept flat
+  (`max_table_nesting=0`) — one table per Knack object, no nested child tables.
+
+## Quick start
+
+Requires Python 3.13+ and [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync
+
+export KNACK_APP_ID=your_app_id
+export KNACK_API_KEY=your_rest_api_key
+
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID"
+```
+
+Names are derived from your app's slug, so a second app never lands on the first one's tables:
+database `knack_{slug}_data`, dataset `{slug}`, pipeline `knack_{slug}_pipeline`.
+
+### Destinations
+
+`--destination local` (the default) writes a DuckDB file — nothing to sign up for, so a fresh
+clone can be pointed at a Knack app and produce a queryable warehouse immediately. The file
+lands at `./tests/data/knack_{slug}_data.duckdb` unless you pass `--db-path`; the resolved
+path is printed on every run.
+
+```bash
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID" --db-path ~/knack.duckdb
+uv run knack-elt run-pipeline --app-id "$KNACK_APP_ID" --destination motherduck
+```
+
+`--destination motherduck` loads to `md:///knack_{slug}_data` and needs `motherduck_api_key`
+in the environment. Both destinations also write the run's `_load_info` and `_trace` tables.
+
+### Other flags
+
+| Flag | What it does |
+| --- | --- |
+| `--api-key` | Knack REST API key, if you would rather not set `KNACK_API_KEY` |
+| `--refresh-metadata` | Re-fetch app metadata instead of reusing knack-sleuth's 24h on-disk cache |
+| `--skip-unreadable` | Log and continue past objects that fail *before yielding any row* (typically no read permission). An object that fails partway through still aborts the run — loading a partial batch would retire live SCD2 rows as if the missing records had been deleted in Knack. |
+
+## Configuration
+
+Read from the environment or a `.env` file via `pydantic-settings`
+([`src/knack_elt/config.py`](src/knack_elt/config.py)):
+
+| Variable | Purpose |
+| --- | --- |
+| `KNACK_APP_ID` | Knack application id — also the default for `--app-id` |
+| `KNACK_API_KEY` | Knack REST API key, sent as `X-Knack-REST-API-Key` |
+| `motherduck_api_key` | MotherDuck token, when the destination is MotherDuck |
+
+## Querying what you get
+
+Because loads are SCD2, a record's history is several rows sharing one `record_id`, tagged with
+`_dlt_valid_from` and `_dlt_valid_to`. Two flags are worth deriving up front — conflating them
+is the most common way to get a wrong answer:
+
+```sql
+with flagged as (
+    select
+        *,
+        row_number() over (partition by record_id order by _dlt_valid_from desc) = 1
+            as latest_version,      -- one row per record
+        _dlt_valid_to is null as is_live_in_knack   -- still in the app?
+    from your_dataset.some_table
+)
+select * from flagged where latest_version
+```
+
+A record deleted in Knack survives only as a *retired* row, so filtering on
+`_dlt_valid_to is null` alone silently drops exactly the history you built the warehouse for.
+And aggregating without `latest_version` double-counts, because every past version is still a
+row. The [architecture doc](docs/ARCHITECTURE.md#4-scd2-row-lifecycle) works through both.
+
+> **One caveat on `is_live_in_knack`.** If an object returns *zero* records, dlt has nothing to
+> load for that table and the merge never runs, so rows loaded earlier keep `_dlt_valid_to is
+> null` and still read as live. Emptying an object in Knack is therefore invisible to the flag —
+> a table whose row count stops moving is worth checking against the app.
+
+## Documentation
+
+- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — the reference architecture in plain language
+  and in technical detail, the pipeline internals, a run sequence, and the SCD2 row model with
+  the query patterns it requires. Also available as a [PDF](docs/ARCHITECTURE.pdf).
+
+The PDF is generated from the markdown rather than maintained alongside it. After editing
+the diagrams, rebuild it with `uv run scripts/build_architecture_pdf.py` (needs node and
+Chrome) so the two don't drift apart.
+
+## Related
+
+- [dlt](https://dlthub.com) — the load framework this is built on
+- `knack-sleuth` — Knack application metadata models and schema export, used here to read your
+  app's structure
+
+## License
+
+GPL-3.0. See [LICENSE](LICENSE).

knack_elt-0.2.1/pyproject.toml

@@ -0,0 +1,61 @@
+[project]
+name = "knack-elt"
+version = "0.2.1"
+description = "Generic ELT pipeline: extract any Knack application into DuckDB or MotherDuck with dlt, keeping full SCD2 history."
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "dlt[duckdb]>=1.30.0",
+    "httpx[http2]>=0.28.1",
+    "knack-sleuth>=0.7.1",
+    "pydantic-settings>=2.15.0",
+    "rich>=15.0.0",
+    "typer>=0.27.1",
+]
+
+[project.scripts]
+knack-elt = "knack_elt.cli:cli"
+
+[build-system]
+requires = ["uv_build>=0.9.5,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.ruff]
+target-version = "py313"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+    "B",
+    "SIM",
+    "RUF",
+]
+ignore = ["E501"]
+
+[tool.ruff.lint.per-file-ignores]
+"src/knack_elt/cli.py" = ["B008"]
+
+[tool.bumpversion]
+current_version = "0.2.1"
+commit = true
+tag = true
+
+[[tool.bumpversion.files]]
+filename = "pyproject.toml"
+search = 'version = "{current_version}"'
+replace = 'version = "{new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "src/knack_elt/__init__.py"
+search = '__version__ = "{current_version}"'
+replace = '__version__ = "{new_version}"'
+
+[dependency-groups]
+dev = [
+    "bump-my-version>=1.5.1",
+    "pytest>=9.1.1",
+    "ruff>=0.16.4",
+]

knack_elt-0.2.1/pyproject.toml.orig

@@ -0,0 +1,61 @@
+[project]
+name = "knack-elt"
+version = "0.2.1"
+description = "Generic ELT pipeline: extract any Knack application into DuckDB or MotherDuck with dlt, keeping full SCD2 history."
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "dlt[duckdb]>=1.30.0",
+    "httpx[http2]>=0.28.1",
+    "knack-sleuth>=0.7.1",
+    "pydantic-settings>=2.15.0",
+    "rich>=15.0.0",
+    "typer>=0.27.1",
+]
+
+
+[project.scripts]
+knack-elt = "knack_elt.cli:cli"
+
+[build-system]
+requires = ["uv_build>=0.9.5,<0.13.0"]
+build-backend = "uv_build"
+
+
+[tool.ruff]
+target-version = "py313"
+
+[tool.ruff.lint]
+# Pinned explicitly: ruff's default rule set widens between releases, and CI
+# should fail on our choices rather than on ruff's changing defaults.
+select = ["E", "F", "I", "UP", "B", "SIM", "RUF"]
+ignore = [
+    "E501",  # line length - not enforced, comments here carry reasoning
+]
+
+[tool.ruff.lint.per-file-ignores]
+# Typer's API *is* function calls in argument defaults.
+"src/knack_elt/cli.py" = ["B008"]
+
+[tool.bumpversion]
+current_version = "0.2.1"
+commit = true
+tag = true
+
+[[tool.bumpversion.files]]
+filename = "pyproject.toml"
+search = 'version = "{current_version}"'
+replace = 'version = "{new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "src/knack_elt/__init__.py"
+search = '__version__ = "{current_version}"'
+replace = '__version__ = "{new_version}"'
+
+
+[dependency-groups]
+dev = [
+    "bump-my-version>=1.5.1",
+    "pytest>=9.1.1",
+    "ruff>=0.16.4",
+]

knack_elt-0.2.1/src/knack_elt/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.1"

knack_elt-0.2.1/src/knack_elt/cli.py

@@ -0,0 +1,149 @@
+from pathlib import Path
+
+import dlt
+import typer
+from knack_sleuth import load_app_metadata
+from rich.console import Console
+from rich.table import Table
+
+from knack_elt import __version__
+from knack_elt.config import settings
+from knack_elt.knack_dlt import build_knack_resources, create_rest_client
+
+cli = typer.Typer()
+console = Console()
+
+DEFAULT_LOCAL_DB_DIR = Path("tests/data")
+
+
+def version_callback(value: bool):
+    """Display version and exit."""
+    if value:
+        console.print(f"knack-elt version {__version__}")
+        raise typer.Exit()
+
+
+@cli.callback()
+def main(
+    version: bool = typer.Option(
+        False,
+        "--version",
+        "-v",
+        callback=version_callback,
+        is_eager=True,
+        help="Show version and exit."
+    )
+):
+    pass
+
+
+@cli.command()
+def run_pipeline(
+    app_id: str = typer.Option(
+        None,
+        "--app-id",
+        help="Knack application ID to extract data from. Defaults to $KNACK_APP_ID."
+    ),
+    api_key: str = typer.Option(
+        None,
+        "--api-key",
+        help="Knack REST API key. Defaults to $KNACK_API_KEY.",
+    ),
+    destination: str = typer.Option(
+        "local",
+        "--destination",
+        "-d",
+        help="Where to load: 'local' (a DuckDB file, no account needed) or 'motherduck'.",
+    ),
+    db_path: Path = typer.Option(
+        None,
+        "--db-path",
+        help=f"Local DuckDB file. Defaults to ./{DEFAULT_LOCAL_DB_DIR}/knack_{{slug}}_data.duckdb.",
+    ),
+    refresh_metadata: bool = typer.Option(
+        False,
+        "--refresh-metadata",
+        help="Re-fetch app metadata instead of using knack-sleuth's 24h on-disk cache.",
+    ),
+    skip_unreadable: bool = typer.Option(
+        False,
+        "--skip-unreadable",
+        help="Log and skip objects that fail before yielding any row (e.g. no read "
+             "permission) instead of aborting the run. An object that fails partway "
+             "through still aborts: a partial batch would retire live SCD2 rows.",
+    ),
+):
+    """Run the ELT pipeline for a Knack application."""
+    final_app_id = app_id or settings.knack_app_id
+    final_api_key = api_key or settings.knack_api_key
+
+    if not final_app_id:
+        console.print("[bold red]Error:[/bold red] app_id is required. Provide it via --app-id option or set KNACK_APP_ID environment variable.")
+        raise typer.Exit(code=1)
+
+    if not final_api_key:
+        console.print("[bold red]Error:[/bold red] a Knack REST API key is required to read records. Provide it via --api-key or set KNACK_API_KEY.")
+        raise typer.Exit(code=1)
+
+    if destination not in ("local", "motherduck"):
+        console.print(f"[bold red]Error:[/bold red] unknown destination {destination!r}; expected 'local' or 'motherduck'.")
+        raise typer.Exit(code=1)
+
+    if destination == "motherduck" and not settings.motherduck_api_key:
+        console.print("[bold red]Error:[/bold red] --destination motherduck requires motherduck_api_key in the environment or .env.")
+        raise typer.Exit(code=1)
+
+    kn_app = load_app_metadata(app_id=final_app_id, refresh=refresh_metadata).application
+
+    dest_db_name = f"knack_{kn_app.slug}_data"
+    dlt_pipeline_name = f"knack_{kn_app.slug}_pipeline"
+    dataset_name = kn_app.slug.replace('-', '_')
+
+    if destination == "local":
+        local_db_path = (db_path or DEFAULT_LOCAL_DB_DIR / f"{dest_db_name}.duckdb").resolve()
+        local_db_path.parent.mkdir(parents=True, exist_ok=True)
+        dlt_destination = dlt.destinations.duckdb(str(local_db_path))
+        destination_label = str(local_db_path)
+    else:
+        local_db_path = None
+        dlt_destination = dlt.destinations.motherduck(
+            f"md:///{dest_db_name}?token={settings.motherduck_api_key}"
+        )
+        destination_label = f"MotherDuck md:///{dest_db_name}"
+
+    summary = Table(show_header=False, box=None)
+    summary.add_column(style="bold")
+    summary.add_column(style="cyan bold")
+    summary.add_row("App", f"{kn_app.name} ({final_app_id})")
+    summary.add_row("Slug", kn_app.slug)
+    summary.add_row("Objects", str(len(kn_app.objects)))
+    summary.add_row("Destination", destination_label)
+    summary.add_row("Dataset", dataset_name)
+    summary.add_row("dlt pipeline", dlt_pipeline_name)
+    console.print(summary)
+
+    dlt.config["load.workers"] = 3
+    dlt.config["truncate_staging_dataset"] = True
+
+    knack_dlt_pipeline = dlt.pipeline(
+        pipeline_name=dlt_pipeline_name,
+        dataset_name=dataset_name,
+        dev_mode=False,
+        destination=dlt_destination,
+    )
+
+    client = create_rest_client(app_id=final_app_id, api_key=final_api_key)
+    load_info = knack_dlt_pipeline.run(
+        build_knack_resources(kn_app, client, skip_unreadable=skip_unreadable)
+    )
+
+    console.print(load_info)
+    console.print(f"Elapsed: {(load_info.finished_at - load_info.started_at).in_words()}")
+
+    # Keep the run's own bookkeeping alongside the data.
+    knack_dlt_pipeline.run([load_info], table_name="_load_info")
+    knack_dlt_pipeline.run([knack_dlt_pipeline.last_trace], table_name="_trace")
+
+
+if __name__ == "__main__":
+    cli()

knack_elt-0.2.1/src/knack_elt/config.py

@@ -0,0 +1,23 @@
+import os
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class Settings(BaseSettings):
+    # Lowercase deliberately: this is the documented variable name and the one
+    # existing deployments set. os.environ is case-sensitive, so capitalising it
+    # would be a breaking config change.
+    motherduck_api_key: str = os.environ.get('motherduck_api_key', '')  # noqa: SIM112
+    knack_app_id: str = Field(default='', alias='KNACK_APP_ID')
+    knack_api_key: str = Field(default='', alias='KNACK_API_KEY')
+    
+    model_config = SettingsConfigDict(
+        env_file='.env',
+        env_file_encoding='utf-8',
+        case_sensitive=False,
+    )
+    
+settings = Settings()
+
+

knack_elt-0.2.1/src/knack_elt/knack_dlt.py

@@ -0,0 +1,182 @@
+"""dlt resources and transformers for extracting a Knack application into DuckDB/MotherDuck.
+
+Nothing in this module is app-specific: every name, object key, and credential is
+passed in by the caller (see `cli.py`).
+"""
+import json
+import logging
+from collections.abc import Iterable
+
+import dlt
+from dlt.sources.helpers.rest_client import RESTClient
+from dlt.sources.helpers.rest_client.auth import APIKeyAuth
+from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
+from knack_sleuth import Application
+
+from .mapping import create_app_mappings, remap_keys
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+# Columns the pipeline stamps onto every row for lineage. Underscore-prefixed so a
+# user-defined Knack field can never slugify onto one of them and clobber the value.
+LINEAGE_TABLE_NAME = "_kn_table_name"
+LINEAGE_OBJECT_ID = "_kn_object_id"
+
+# Knack returns the row identifier as the top-level key "id". The pipeline renames it to
+# "record_id" and merges on that, which frees the "id" column for a user-defined field of
+# that name (Knack apps commonly have one). Sourcing the key from the payload rather than
+# from Knack's auto-added "Record ID" field means this holds for every app, including ones
+# that predate that field.
+RECORD_KEY = "record_id"
+
+
+def create_rest_client(app_id: str, api_key: str) -> RESTClient:
+    """Build a Knack REST client for a specific application.
+
+    Both the application id and the API key are explicit: the record endpoints
+    authenticate per-app, so they must match the app whose metadata was loaded.
+    """
+    auth = APIKeyAuth(
+        name="X-Knack-REST-API-Key",
+        api_key=api_key,
+        location="header"
+    )
+    return RESTClient(
+        base_url="https://api.knack.com/v1",
+        auth=auth,
+        headers={"X-Knack-Application-ID": app_id},
+        paginator=PageNumberPaginator(
+            base_page=1,
+            total_path="total_pages"
+        ),
+        data_selector="records"
+    )
+
+
+def get_knack_table_data(table_name, object_id, client, json_fields=(), skip_unreadable=False):
+    @dlt.resource(name=f"table_{object_id}",
+                  write_disposition={"disposition": "merge", "strategy": "scd2"},
+                  primary_key=RECORD_KEY,
+                  columns={RECORD_KEY: {"merge_key": False}}  # to work around a possible bug in DLT
+                  )
+    def table_data():
+        logger.info(f"Processing table: {table_name} ({object_id})")
+        kn_params = {'rows_per_page': 1000, 'format': 'raw'}
+        url = f"/objects/{object_id}/records"
+        yielded = 0
+        try:
+            for page in client.paginate(url, params=kn_params):
+                logger.debug(f"Fetched {len(page)} records from {table_name}")
+                for row in page:
+                    if row.get('id') is None:
+                        logger.warning(f"Missing Primary Key in table {table_name}: {row}")
+                        continue
+                    # Rename before anything else so the merge key is set even if a
+                    # user-defined field also wants the "id" column.
+                    row[RECORD_KEY] = row.pop('id')
+                    row[LINEAGE_TABLE_NAME] = table_name
+                    row[LINEAGE_OBJECT_ID] = object_id
+
+                    row = clean_json_fields(row, json_fields)
+                    yielded += 1
+                    yield row
+        except Exception as e:
+            # Swallowing an error *after* rows were yielded would hand dlt a
+            # successful partial extraction, and the SCD2 merge would retire every
+            # row missing from that partial batch. Only a zero-yield failure is safe
+            # to skip.
+            if skip_unreadable and yielded == 0:
+                logger.error(f"Skipping unreadable object {table_name} ({object_id}): {e}")
+                return
+            logger.error(f"Error fetching data for table {table_name}: {e}")
+            raise
+
+    return table_data()
+
+
+def clean_empty_strings(row, numeric_fields):
+    """Converts empty strings to None for specified numeric fields."""
+    for field in numeric_fields:
+        if field in row and row[field] == "":
+            row[field] = None
+    return row
+
+
+def assign_default_values(row, default_values):
+    """Assign default values to fields that are None."""
+    for fk in row:
+        if fk in default_values and (row[fk] is None or row[fk] == ""):
+            row[fk] = default_values[fk]
+    return row
+
+
+def clean_json_fields(row, json_fields: Iterable[str] = ()):
+    """Ensure JSON-in-string fields are valid, or convert empty/invalid JSON to None.
+
+    Knack's `format=raw` returns rich fields (file, image, connection) as dicts, not
+    JSON strings, so non-string values are left untouched.
+    """
+    for field in json_fields:
+        if field in row:
+            value = row[field]
+            if not isinstance(value, str):
+                continue
+            if value.strip() == "":
+                row[field] = None  # Replace empty JSON with None
+            else:
+                try:
+                    json.loads(value)  # Validate JSON
+                except json.JSONDecodeError:
+                    row[field] = None  # Replace invalid JSON with None
+    return row
+
+
+def get_remap_transformer(table_name, object_id, field_mappings, numeric_fields, default_values):
+    @dlt.transformer(name=f"remap_{object_id}", table_name=table_name,
+                     write_disposition={"disposition": "merge", "strategy": "scd2"},
+                     primary_key=RECORD_KEY,
+                     columns={RECORD_KEY: {"merge_key": False}}  # to work around a possible bug in DLT
+                     )
+    def remap_knack_field_id_to_name(row):
+        # Cleaning runs before the remap, so it matches on raw Knack field keys.
+        row = clean_empty_strings(row, numeric_fields)
+        row = assign_default_values(row, default_values)
+
+        field_mapping = field_mappings.get(object_id)
+        if field_mapping:
+            row = remap_keys(row, field_mapping)
+        else:
+            logger.info(f"No field mapping found for object_id {object_id}. Keeping original field names.")
+        return row
+
+    return remap_knack_field_id_to_name
+
+
+@dlt.source(max_table_nesting=0)
+def build_knack_resources(kn_app: Application, client: RESTClient, skip_unreadable: bool = False):
+    """One resource+transformer pair per Knack object, chained with the pipe operator."""
+    resources = []
+    field_mappings, _object_mappings, numeric_fields, default_values = create_app_mappings(kn_app)
+
+    # Resource names key off obj.key (globally unique in Knack); destination table
+    # names come from the object name, which is NOT guaranteed unique, so dedupe.
+    seen_tables = {}
+    for obj in kn_app.objects:
+        table_name = obj.name
+        if table_name in seen_tables:
+            table_name = f"{obj.name}_{obj.key}"
+            logger.warning(
+                f"Duplicate object name {obj.name!r} ({obj.key} and {seen_tables[obj.name]}); "
+                f"loading it as {table_name!r}"
+            )
+        seen_tables.setdefault(obj.name, obj.key)
+
+        table_resource = get_knack_table_data(table_name, obj.key, client, skip_unreadable=skip_unreadable)
+        transformer_resource = get_remap_transformer(
+            table_name, obj.key, field_mappings, numeric_fields, default_values
+        )
+        resources.append(table_resource | transformer_resource)
+
+    logger.info(f"Built {len(resources)} resources for {kn_app.name}")
+    return resources

knack_elt-0.2.1/src/knack_elt/mapping.py

@@ -0,0 +1,135 @@
+"""
+The following functions are used to create a field mapping and remap keys for a JSON record based on
+knack metadata for objects.
+
+
+Goal is to keep this as lightweight and simple as possible.  To lean into dlt functionality where possible.
+
+The input to these functions is the Application Metadata object from the Knack API, (https://api.knack.com/v1/applications/{app_id}))
+"""
+import logging
+import re
+from typing import Any
+
+from knack_sleuth.models import Application
+
+logger = logging.getLogger(__name__)
+
+
+# Field types whose values are numeric (or, for date_time, non-textual) and so must
+# have empty strings nulled before load — otherwise the column types as VARCHAR.
+# The aggregate types (sum/min/max/average/count) are connection roll-ups.
+NUMERIC_FIELD_TYPES = [
+    'number', 'currency', 'link', 'date_time', 'auto_increment',
+    'count', 'sum', 'min', 'max', 'average', 'equation', 'rating',
+]
+
+
+def slugify_field_name(field_name: str) -> str:
+    """Lowercase snake_case a Knack field name. May return '' for a name with no
+    ASCII alphanumerics (e.g. "%" or a fully non-Latin name) - callers must handle it."""
+    return re.sub(r'[^a-z0-9]+', '_', field_name.lower()).strip('_')
+
+
+def create_app_mappings(app_metadata: Application) -> tuple[
+    dict[Any, dict[Any, Any]], dict[Any, Any], list[str | Any], dict[Any, Any]]:
+    """
+    Creates both field mappings and object mappings from the Knack app metadata.
+    
+    Uses KnackAppMetadata Pydantic model for validated, type-safe parsing.
+
+    Returns:
+    - field_mappings: A dictionary of dictionaries. The outer dictionary keys are object_ids,
+      and the inner dictionary maps original field keys to new slugified field names.
+    - object_mappings: A dictionary mapping table names to object_ids.
+    - numeric_fields: List of field identifiers that should be treated as numeric.
+    - default_values: Dictionary of default values for fields (primarily boolean fields).
+    """
+    
+    # Columns the pipeline owns. A user-defined field slugifying onto one of these would
+    # silently replace it, so it is renamed <singular>_<name> instead.
+    #
+    # 'id' is deliberately NOT reserved: the pipeline renames Knack's row id to
+    # 'record_id' (see knack_dlt.RECORD_KEY), so a user field named "ID" keeps the plain
+    # 'id' column it was named for.
+    #
+    # 'record_id' IS reserved, because Knack now auto-adds a short_text field named
+    # "Record ID" to every object holding a copy of the row id (verified live 2026-08-25
+    # across three apps; 3,562 Illinois records, all non-blank and all equal to 'id').
+    # Left unreserved it would slugify onto the merge key and overwrite it. Reserving it
+    # costs one redundant <singular>_record_id column per table, which is preferable to a
+    # schema whose shape depends on whether any row's values happen to diverge.
+    #
+    # Not covered: user objects also return account_status, approval_status, utility_key,
+    # profile_keys and profile_keys_raw. Renaming e.g. an "Account Status" field on a
+    # non-user object would be gratuitous, so those are left alone.
+    restricted_field_names = ['record_id']
+    field_mappings = {}
+    object_mappings = {}
+    default_values = {}
+    numeric_fields = []
+    
+    for obj in app_metadata.objects:
+        object_id = obj.key
+        object_name = obj.name
+        
+        # Get singular form safely with fallback
+        singular = obj.inflections.singular if obj.inflections else object_name
+
+        # Create object mapping
+        object_mappings[object_name] = object_id
+
+        # Create field mapping for this object
+        field_mappings[object_id] = {}
+        used_slugs = {}
+
+        for field in obj.fields:
+            field_key = field.key
+            field_name = field.name
+            
+            # Slugify field name. Two distinct Knack fields can legally slugify to
+            # the same name ("Total ($)" and "Total (%)" both -> "total"), and a name
+            # with no ASCII alphanumerics slugifies to "". Either case would collapse
+            # columns in remap_keys, where the last field silently wins - so fall back
+            # to the Knack field key, which is unique app-wide.
+            new_key = slugify_field_name(field_name)
+            if not new_key:
+                new_key = field_key
+            # Compare the *slug*, not the raw name: Knack's auto-added field is named
+            # "Record ID", which lowercases to "record id" and only becomes "record_id"
+            # after slugification.
+            if new_key in restricted_field_names:
+                new_key = slugify_field_name(f"{singular} {new_key}") or f"{field_key}_{new_key}"
+            if new_key in used_slugs:
+                collided_with = used_slugs[new_key]
+                new_key = f"{new_key}_{field_key}"
+                logger.warning(
+                    f"Field name {field_name!r} ({field_key}) on {object_name!r} slugifies onto "
+                    f"{collided_with}'s column; loading it as {new_key!r} instead."
+                )
+            used_slugs[new_key] = field_key
+            field_mappings[object_id][field_key] = new_key
+            
+            # Track numeric fields
+            if field.type in NUMERIC_FIELD_TYPES:
+                numeric_fields.append(new_key)
+                numeric_fields.append(field_key)
+                numeric_fields.append(field_name)
+
+            # Handle boolean fields with defaults
+            if field.type == 'boolean' and field.format and hasattr(field.format, '__dict__'):
+                # Access format as Pydantic model with extra fields allowed
+                format_dict = field.format.model_dump()
+                if 'default' in format_dict:
+                    field_default_value = format_dict['default']
+                    default_values[field_key] = field_default_value
+                    default_values[new_key] = field_default_value
+                    default_values[field_name] = field_default_value
+
+    return field_mappings, object_mappings, numeric_fields, default_values
+    
+
+def remap_keys(record: dict[str, Any], field_mapping: dict[str, str]) -> dict[str, Any]:
+    """Remaps the keys of a single record using the provided field mapping."""
+    return {field_mapping.get(key, key): value for key, value in record.items()}
+