umf-tablespec 0.0.14__tar.gz

umf_tablespec-0.0.14/.agents/skills/tablespec/SKILL.md

@@ -0,0 +1,175 @@
+---
+name: tablespec
+description: Working with tablespec and Universal Metadata Format (UMF) table schemas - bootstrapping UMF from Spark/Databricks tables, profiling, validation expectations, sample data, Excel table specs, and Spark/LDP/dbt pipeline artifacts. Use when a task involves UMF YAML/JSON, the tablespec CLI or Python API, or Databricks table-spec workflows.
+---
+
+# Tablespec
+
+Use this skill as a routing layer. Prefer the documented public workflow first,
+then drop to internals only when the task explicitly requires it. Do not duplicate
+the project's requirements or design docs here; read them when changing
+requirements.
+
+## Task Routing
+
+Focused sibling skills carry the mechanics. Route by task:
+
+| Task | Skill |
+| --- | --- |
+| Author or edit UMF split YAML, columns, relationships, domain types | `tablespec-umf-authoring` |
+| Bootstrap UMF, compile the artifact tree, run the raw-to-ingested backbone | `tablespec-pipeline` |
+| Great Expectations suites: baseline, sync, preview, apply, staged execution | `tablespec-validation` |
+| Derived/gold tables and SQL plans (derivations, base table strategies) | `tablespec-sql-plans` |
+| The data-profiling Databricks App (deploy, provision, operate) | `tablespec-profiling-app` |
+
+## Getting tablespec
+
+tablespec is published to a GitHub Pages index, so the `--index-url` flag is
+required: `pip install umf-tablespec --index-url
+https://documentdrivendx.github.io/tablespec/simple/` (use `umf-tablespec[spark]` for
+the Spark-backed bootstrap, profiling, and validation features). Installing
+provides a `tablespec` CLI. Full documentation:
+https://documentdrivendx.github.io/tablespec/
+
+## Default Workflow
+
+1. Read the local code and governing spec before changing behavior.
+2. Keep user-facing examples on public APIs and commands. Internal modules may be
+composable, but should not be the first path shown to users or coding agents.
+
+## The End-to-End Story
+
+UMF specs are the single source of truth. The happy path: bootstrap UMF from
+existing tables (`bootstrap_from_tables`) or author specs by hand, compile them
+into a pinned artifact tree (`bootstrap_from_specs` / `compile_umfs`), review via
+Excel and sample data, then run the compiled artifacts — locally or on
+Databricks. Development builds artifacts; production runs installed pipeline
+artifacts and wheels, never re-derives them. Mechanics live in
+`tablespec-pipeline`.
+
+Split YAML directories (`tables/<table>/table.yaml` + `columns/<column>.yaml` +
+`expectations.yaml`) are the editable authoring format; JSON is
+artifact/interchange; inline whole-UMF YAML (`table.umf.yaml`) is
+legacy/migration-only. Layout rules live in `tablespec-umf-authoring`.
+
+## CLI Index
+
+One line per command; the named skill has flags and gotchas.
+
+| Command | Purpose | Skill |
+| --- | --- | --- |
+| `validate` | Validate UMF specs (schema, naming, relationships) | `tablespec-umf-authoring` |
+| `info` | Rich summary of a UMF spec | `tablespec-umf-authoring` |
+| `convert` / `batch-convert` | Split-dir ↔ JSON format conversion | `tablespec-umf-authoring` |
+| `column-add` / `column-remove` / `column-modify` / `column-rename` | Column mutations | `tablespec-umf-authoring` |
+| `domains-list` / `domains-show` / `domains-infer` / `domains-set` | Domain-type registry and assignment | `tablespec-umf-authoring` |
+| `explore` | Textual TUI browser/editor (`[tui]` extra) | `tablespec-umf-authoring` |
+| `bootstrap` | Authored specs → full compiled artifact tree | `tablespec-pipeline` |
+| `generate` | Emit ONE artifact format (sql/pyspark/json/ingest) to stdout | `tablespec-pipeline` |
+| `emit` | Materialize a runnable dbt project | `tablespec-pipeline` |
+| `validation-sync` | Regenerate baseline expectations, preserving customizations | `tablespec-validation` |
+| `validation-remove` | Delete expectations from the suite | `tablespec-validation` |
+| `preview` | Classify expectations by stage (merged view) | `tablespec-validation` |
+| `apply-response` | Apply reviewed expectation JSON | `tablespec-validation` |
+| `export-excel` / `import-excel` | Excel round trip for domain-expert review | (docs: guide/excel) |
+| `guidebook` | Static HTML data-catalog site from a UMF directory | (docs: guide/guidebook) |
+
+## Dialects
+
+Databricks uses Spark-family SQL for tablespec cast expressions. Until the
+project's Databricks dialect handling is fully canonicalized, prefer
+`dialect="spark"` in Databricks examples unless the code path being changed
+explicitly accepts `dialect="databricks"`. If changing dialect support, make code,
+errors, CLI help, docs, and tests agree on the same accepted values.
+
+## Production Pipeline Contract
+
+Keep development and production boundaries explicit:
+
+- Development/bootstrap may infer UMF, profile data, compile artifacts, and build
+  packages.
+- Production should run installed pipeline JSON artifacts and installed wheels.
+- Do not present notebook-local Python orchestration as the production runtime.
+- Dev pipeline work should build wheel artifacts for installation and produce or
+  validate the JSON pipeline definitions consumed by production.
+
+## Databricks Testing
+
+On Databricks, run Spark-dependent tests in the notebook kernel process. Use
+`ipytest` or `pytest.main([...])`, not subprocess pytest. Do not use `uv run
+pytest` for Spark runtime tests on Databricks because the isolated environment
+cannot see the runtime PySpark/Spark Connect session.
+
+For local work, use your project's normal pytest invocation (in the tablespec repo
+that is `uv run pytest ...`) unless a Databricks-specific instruction says
+otherwise.
+
+## Public Surface Discipline
+
+When agents or users need a "how do I do this?" answer, point them at the public
+facade or CLI. The public facade includes `bootstrap_from_tables`,
+`bootstrap_from_specs`, `compile_umfs`, `run_backbone`, `generate_sql_plan`, and
+`umf_from_information_schema`. Reserve these lower-level pieces for
+implementation work:
+
+- `tablespec.e2e.paths.umfs_from_tables`
+- `tablespec.profiling.spark_mapper.SparkToUmfMapper`
+- `tablespec.profiling.native_profiler.NativeSparkProfiler`
+- `tablespec.profiling.gx_expectation_builder.ProfileToGxMapper`
+
+If no public facade covers a workflow, compose the documented public pieces
+rather than reaching into internals.
+
+## When contributing to tablespec itself
+
+Everything above applies to using tablespec. When the change is to tablespec's own
+code, docs, or specs, these also apply:
+
+- Read the governing requirements and design docs before changing behavior. The
+  tablespec repository keeps HELIX-style specs under `docs/helix/`.
+- Track implementation work in whatever the project uses. If the repository tracks
+  work with DDx beads (a `.ddx/` directory exists), use the `ddx` skill for bead
+  create, claim, close, worker, and review commands; otherwise use the project's
+  normal issue tracker.
+- When adding or changing behavior, add a command-based acceptance test and cite
+  the relevant spec or tracked issue in the implementation notes.
+- Preserve the end-to-end happy path as the product story when writing docs,
+  examples, APIs, or tracked work items. If a step is not yet one-shot, show the
+  current supported composition and file or cite a tracked issue for the missing
+  facade.
+- Do not create new public examples that use `table.umf.yaml` as the canonical
+  authoring format. If a task touches inline whole-UMF YAML support, check the
+  tracked issue or bead covering its quarantine before deciding whether to
+  preserve, reject, or migrate it.
+- If no public facade exists for a common workflow, file an issue (or a bead, in
+  DDx repositories) to add one instead of normalizing hand-written orchestration
+  in docs.
+
+## Fabricated sample-data loading
+
+Use `tablespec sample-data load --umf <dir> --target <catalog.schema>
+--domain legal --scale small --dry-run` to review local generation, DDL and
+verification. Healthcare is the default pack. Read `docs/guide/sample-data.md`
+for registration, custom scale configuration and supported constraints. Only
+perform a workspace load when explicitly authorized; supply warehouse/profile
+names, never credentials. Replacement stages batches then overwrites each
+target table; it is not a transaction across tables. `--drop-existing` requires
+explicit operator intent. Praxis specs belong in praxis, not this library.
+
+Legal sample data is tabular; ontology modeling belongs in truss and ashlar.
+The bounded legal path coordinates teams/walls before entries and audits
+entitlements. Optional `nda_issues`/`vague_entry` columns expose evaluation ground
+truth. `--bulk --volume /Volumes/catalog/schema/volume/path` uses CSV + Files API
++ COPY INTO; normal loading runs read-back audits and `--verify-only` reads
+existing tables. Preserve dry-run local-only behavior. See the sample-data guide.
+
+UMF owns portable pack/source metadata, schemas and schema export tooling;
+tablespec owns data generation, CSV ZIP export, ingestion and data tests.
+`sample-data export --umf <dir> --domain legal --scale small --output <file.zip>`
+creates a verified local archive without authentication. `--domain-pack` selects
+UMF metadata per run; resolve generator IDs through explicit trusted registration.
+Retain external source origin, declared data kind, revision/checksum, rights and
+provenance. Never fetch references or substitute generated rows for external input
+implicitly. Use the typed local CSV reader to preserve null/empty distinctions.
+Run the Sail and Spark ingestion lanes; live warehouse tests require both
+`TABLESPEC_LIVE_WAREHOUSE_ID` and `TABLESPEC_LIVE_PROFILE` and are opt-in only.

umf_tablespec-0.0.14/.agents/skills/tablespec/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "tablespec"
+  short_description: "UMF table-schema workflows for Codex."
+  default_prompt: "Use the tablespec skill when working with Universal Metadata Format (UMF) table schemas: bootstrapping UMF from Spark/Databricks tables, profiling, validation expectations, sample data, Excel table specs, and Spark/LDP/dbt pipeline artifacts."

umf_tablespec-0.0.14/.agents/skills/tablespec-pipeline/SKILL.md

@@ -0,0 +1,165 @@
+---
+name: tablespec-pipeline
+description: Running the UMF-to-pipeline flow - bootstrapping UMF from existing Spark/Databricks tables or authored specs (bootstrap_from_tables, bootstrap_from_specs, tablespec bootstrap), compiling the pinned artifact tree with compile_umfs (ingest SQL, schemas, validation suites, dbt, LDP, gold plans), and executing artifacts with run_backbone. Use when generating pipeline artifacts from UMF or running the raw-to-ingested backbone locally or on Databricks.
+---
+
+# Tablespec Pipeline
+
+## Mental model
+
+The flow is spec → compile → pinned artifact tree → backbone run. UMF (from
+reflection or authored specs) is compiled once into a persisted artifact tree
+with a `manifest.json`; the runtime backbone then consumes those compiled
+artifacts — never the UMF directly. Development builds the artifacts;
+production installs and runs them (installed pipeline JSON artifacts plus
+installed wheels). Do not re-derive UMF or transforms from source at runtime.
+
+## Path A: bootstrap from existing tables (Spark required)
+
+`bootstrap_from_tables` is the one-shot facade: reflect table schemas into
+UMF, optionally profile the data, and compile the full artifact tree.
+
+```python
+from tablespec import bootstrap_from_tables
+
+artifacts = bootstrap_from_tables(
+    spark,
+    ["catalog.schema.member"],
+    out_dir="/tmp/tablespec-bootstrap",
+    profile=True,
+    dialect="databricks",
+)
+print(artifacts.manifest_path)
+```
+
+Signature: `bootstrap_from_tables(spark, table_names, out_dir, *, profile=True,
+dialect="duckdb", gold_targets=None, infer_keys="none",
+key_promotion_min_score=0.9, key_promotion_min_gap=0.05)` returning
+`CompiledArtifacts`.
+
+- Path A is Python-only; there is no CLI equivalent (the `tablespec bootstrap`
+  command is Path B).
+- Do not describe the profiler as creating UMF. Schema reflection creates the
+  UMF; the native profiler enriches the compiled validation suites with
+  profile-derived expectations.
+- Keep `profile=True` (the recommended default). `profile=False` degrades the
+  compiled suites to schema-only structural/type checks.
+- `infer_keys` accepts `"none"`, `"candidates"`, or `"auto"`. Candidate mode
+  writes an advisory `validation/<t>.keycandidates.json` sidecar only. Auto
+  mode may promote one clear verified candidate into the compiled UMF
+  snapshot's `primary_key` — which changes dbt incremental MERGE `unique_key`,
+  LDP APPLY CHANGES KEYS, and sample-data uniqueness downstream, so prefer
+  `"candidates"` unless promotion is intended.
+
+## Path B: bootstrap from authored specs (no Spark)
+
+`bootstrap_from_specs(spec_paths, out_dir, *, dialect="duckdb",
+gold_targets=None)` loads split table directories, `*.umf.yaml`, or JSON
+interchange paths and compiles the same tree. The CLI covers this path:
+
+```bash
+tablespec bootstrap tables/member tables/claims -o build/artifacts
+tablespec bootstrap tables/ -o out/ --dialect databricks
+```
+
+Flags: one or more spec paths as arguments, `--out`/`-o` for the output
+directory (required), `--dialect` choosing from `duckdb` (default), `spark`,
+or `databricks`.
+
+## compile_umfs is the orchestrator
+
+There is NO single "generate everything" command. `tablespec generate <source>
+-f <format>` emits exactly ONE format to stdout — `sql`, `pyspark`, `json`, or
+`ingest` — for piping to a file. The composition seam that persists every
+runtime artifact is `compile_umfs`:
+
+```python
+from tablespec.e2e import compile_umfs, umfs_from_tables
+
+umfs, suites = umfs_from_tables(spark, ["member"], profile=True)
+artifacts = compile_umfs(
+    umfs,
+    "build/tablespec",
+    source="tables",
+    profile_enriched=True,
+    dialect="databricks",
+    gold_targets=["Claims_Summary"],
+    suites=suites,
+)
+```
+
+Key parameters: `source` is `"tables"` (Path A) or `"specs"` (Path B) —
+manifest provenance only; `profile_enriched` is recorded on the manifest and
+does not itself run profiling — the caller supplies enriched `suites`;
+`suites` (expectation lists keyed by table name) are persisted verbatim when
+present, otherwise the baseline suite is generated from the UMF;
+`gold_targets` names tables that additionally get a single-target gold SQL
+plan; `infer_keys` / `key_candidates` carry the key-inference sidecar inputs.
+The bootstrap facades wrap this call — prefer them unless you need the seam.
+
+## The pinned artifact tree
+
+`compile_umfs` writes a pinned layout under `out_dir` and serializes a
+`CompiledArtifacts` manifest enumerating every path:
+
+```text
+<out_dir>/
+  manifest.json                  # serialized CompiledArtifacts — the production handoff
+  umf/<table>.umf.yaml           # UMF snapshot the compile ran against
+  ingest/<table>.ingest.sql      # raw DDL + typed DDL + raw->ingested transform
+  schemas/<table>.ddl.sql        # SQL DDL
+  schemas/<table>.schema.py      # PySpark StructType source
+  schemas/<table>.schema.json    # JSON Schema
+  validation/<table>.suite.json  # compiled expectation suite (raw + ingested stages)
+  validation/<table>.keycandidates.json  # optional advisory key-candidate sidecar
+  dbt_ingest/<table>/            # single-table ingest dbt project
+  dbt_gold/                      # multi-table gold dbt DAG project (when well-formed)
+  ldp/                           # Lakeflow Declarative Pipelines project
+  gold_plan/<target>.plan.sql    # single-target gold plan (distinct from dbt_gold/)
+```
+
+`manifest.json` stores paths relative to the root, so the tree is relocatable;
+`CompiledArtifacts.load(root)` re-absolutizes them. Production jobs locate
+artifacts through the manifest, never by re-deriving names.
+
+## Dialects
+
+Accepted cast dialects are `duckdb`, `spark`, and `databricks`. Use `duckdb`
+for local compile/run checks and `databricks` (or `spark`) for warehouse legs.
+`databricks` is an accepted public spelling that Spark-family emitters
+normalize to `spark` internally because the rendered cast SQL is identical;
+the manifest preserves the caller's spelling.
+
+## run_backbone
+
+`run_backbone(artifacts, *, spark, raw_batches, run_transforms=True,
+backend="spark", engine=None)` executes the compiled artifacts and returns a
+`BackboneResult` (a list of per-stage `StageOutcome`s; `result.ok` is True iff
+every stage succeeded). `raw_batches` maps table name to ordered raw batch
+file paths; `backend` is `"spark"` (classic), `"sail"` (Spark Connect), or
+`"duckdb"`.
+
+Stage ordering is strict: raw load (all-STRING columns plus `_source_file` and
+`_load_ts` audit columns) → raw validate → cast/MERGE into the typed ingested
+table → ingested validate → transforms (dbt parse over every compiled project,
+gold-plan check, LDP structure check). Validation runs
+`GXSuiteExecutor.execute_staged(raw_df, ingested_df, expectations)` with the
+compiled suite JSON — not `TableValidator`, which is the UMF-level row
+validator outside the backbone.
+
+The real-Databricks execution leg is opt-in, gated by
+`tablespec.e2e.gating.databricks_e2e_availability()`: `DATABRICKS_HOST` is the
+opt-in switch, and `DATABRICKS_HTTP_PATH` plus `DATABRICKS_TOKEN` must also be
+set (with the dbt-databricks adapter and Databricks SDK/SQL connector
+installed). When unset the leg is skipped with a precise reason — local
+success never depends on a remote workspace.
+
+## Related
+
+- Expectations lifecycle (baselines, GX suites, staged execution): see the
+  `tablespec-validation` skill.
+- Derived/gold tables and plan generation: see the `tablespec-sql-plans` skill.
+- Authoring and editing UMF specs: see the `tablespec-umf-authoring` skill.
+
+Docs: `docs/guide/happy-path.md`, `docs/guide/bootstrap.md`,
+`docs/guide/databricks-e2e.md`, and https://documentdrivendx.github.io/tablespec/

umf_tablespec-0.0.14/.agents/skills/tablespec-profiling-app/SKILL.md

@@ -0,0 +1,150 @@
+---
+name: tablespec-profiling-app
+description: Deploying and operating the data-profiling Databricks App (apps/data-profiling) - provisioning the metadata home, app.yaml configuration, service-principal grants, deploying from repo root or app subfolder, Spark-less UMF reflection via umf_from_information_schema, and local development with the mock runtime. Use when working on the Streamlit profiling/guidebook app or its Unity Catalog deployment.
+---
+
+# Tablespec Profiling App
+
+A Streamlit **Databricks App** at `apps/data-profiling/` that profiles Unity
+Catalog tables and renders the tablespec guidebook over the same UMF metadata.
+Its tabs, in display order: **Guidebook**, **Compare two tables**,
+**Profile table(s)**, **Load Results**, and **Ask Genie** (hidden when no Genie
+space is configured). Prefer `docs/guide/data-profiling-app.md` as the source of
+truth over `apps/data-profiling/README.md`.
+
+## No Spark inside the App container
+
+App containers get a SQL warehouse and the workspace SDK, but **no
+`SparkSession`**. Do not use `SparkToUmfMapper` (needs a DataFrame) or
+`JdbcToUmfMapper` (needs `spark.read.format("jdbc")`) in app code. Reflect UMFs
+from `INFORMATION_SCHEMA` rows instead — the function takes rows and opens no
+connection of its own; the caller owns connectivity:
+
+```python
+from tablespec.profiling import umf_from_information_schema
+
+rows = warehouse.query(
+    "SELECT column_name, data_type, is_nullable, character_maximum_length, "
+    "       numeric_precision, numeric_scale, comment, ordinal_position "
+    "FROM `dev`.information_schema.columns "
+    "WHERE table_schema = 'prod_main_clinical' AND table_name = 'encounter' "
+    "ORDER BY ordinal_position"
+)
+umf = umf_from_information_schema("encounter", rows)
+```
+
+Signature: `umf_from_information_schema(table_name, columns, *,
+table_type="inferred", description=None) -> UMF`. Behavior worth knowing:
+
+- Rows may be raw mappings or `ColumnMeta`; keys match case-insensitively, so
+  Databricks (`column_name`) and SQL Server (`COLUMN_NAME`) rows both work.
+- `is_nullable` accepts `"YES"`/`"NO"` or a bool and becomes `nullable.default`.
+- `DECIMAL` carries `precision`/`scale`; `VARCHAR`/`CHAR` carry `length`.
+- Columns are ordered by `ordinal_position` when every row has one.
+- A type with no UMF equivalent (`ARRAY`, `MAP`, `STRUCT`, `VARIANT`) becomes
+  `VARCHAR` and is logged — never dropped.
+
+## Metadata home
+
+Each deployment declares a `(catalog, schema, volume)` where its governance
+tables and run artifacts live. The app is *told* this address — it never scans
+for a naming convention. It writes Delta tables `profiler_runs`,
+`dataset_profiles`, `column_profiles`, `column_alerts`, and
+`column_comparisons` there, and the Load Results tab reads the nightly-load
+tables in the same home. There is **no
+lazy creation**: the app does not create its own tables on first write, so an
+unprovisioned target fails deliberately with a message pointing at the
+provision step, instead of creating objects at a mistyped address.
+
+## Provision — once per environment
+
+```bash
+cd apps/data-profiling
+export DATABRICKS_CONFIG_PROFILE=<profile>   # or DATABRICKS_HOST/TOKEN
+
+python scripts/provision.py --dry-run \
+    --catalog <catalog> --schema <schema> --volume <volume> \
+    --warehouse-id <sql-warehouse-id>
+```
+
+Run with `--dry-run` first — it prints the resolved target and where each value
+came from, changing nothing. Then re-run without it. With no flags it
+provisions whatever the resolved config points at, so a shell that already
+exports `PROFILER_METADATA_*` needs no arguments. Provisioning is idempotent (a
+second run reports no changes) and additive (absent tables and columns are
+added; an extra column is reported, never dropped). A failed `GRANT` is a
+warning carrying the exact statement an administrator must run, not a failure.
+
+## Configure
+
+Declare the inputs in the `env:` block of `app.yaml`:
+
+```yaml
+env:
+  - name: "PROFILER_METADATA_CATALOG"
+    value: "<catalog>"
+  - name: "PROFILER_METADATA_SCHEMA"
+    value: "<schema>"
+  - name: "PROFILER_OUTPUT_VOLUME"
+    value: "<volume>"
+  - name: "DATABRICKS_WAREHOUSE_ID"
+    value: "<sql-warehouse-id>"
+  - name: "PROFILER_RUNTIME"
+    value: "databricks"            # "mock" for local development
+```
+
+Keep the `resources:` block (`output-volume`, `sql-warehouse`) naming the same
+location. Settings resolve through one precedence: **deployment environment >
+`connections.yaml` `metadata:` block > built-in default** — and the built-in
+defaults are intentionally generic (`main.tablespec_profiler`), so a manifest
+input that didn't take effect shows up as the generic default, not a silent
+write elsewhere. `apps/data-profiling/connections.yaml` also declares the
+source connections (catalogs) the app can read from and the environment labels
+offered in the UI. Optional inputs each disable only their surface:
+`GENIE_SPACE_ID` (Ask Genie tab), `PROFILER_DASHBOARD_URL` (sidebar button),
+`PROFILER_SPEC_VOLUME` (guidebook falls back to a path under the output
+volume).
+
+## Deploy
+
+The Apps **source root must be the repository root**: Databricks Apps installs
+the `requirements.txt` found at the source root, and only the root one runs
+`pip install .` so the Guidebook tab can `import tablespec`. The root
+`app.yaml` command then `cd`s into `apps/data-profiling` before launching
+Streamlit, because the app resolves `connections.yaml` and `assets/` relative
+to the working directory. Do not deploy `apps/data-profiling/` expecting the
+working-tree library — its own `requirements.txt` pins tablespec to a git ref.
+
+```bash
+databricks sync --profile <profile> . /Workspace/Users/<you>/<repo>
+databricks apps deploy <app-name> --profile <profile> \
+    --source-code-path /Workspace/Users/<you>/<repo>
+```
+
+A subfolder deploy (`--source-code-path …/<repo>/apps/data-profiling`) is the
+faster fallback when the root deploy times out, at the cost of the git-ref pin.
+Keep the two manifests' `resources`/`env` blocks in step; whichever you deploy
+from takes effect. Grant the app's service principal access to the metadata
+home, the warehouse, and every catalog it should profile — the grant table is
+in `docs/guide/data-profiling-app.md` ("Grant the app's service principal").
+
+## Verify and local development
+
+After start, the sidebar **Environment** panel lists every setting, its value,
+and which tier supplied it — source `default` next to the catalog or schema
+means the manifest input did not take effect. Configuration faults appear as a
+banner naming the setting and the fix; a stopped warehouse is a deferred check,
+not a startup failure.
+
+Locally, `make app-smoke` runs the app's config/provision/diagnostics tests
+under `PROFILER_RUNTIME=mock` with no workspace, and `make app-typecheck` runs
+the scoped pyright config in `apps/data-profiling/`. Never add an
+`__init__.py` to `apps/data-profiling/tests/` — it would create a second
+top-level `tests` package that shadows the library's and break collection.
+
+## Related
+
+For generating UMF, validation suites, and pipeline artifacts outside the app,
+use the `tablespec-pipeline` skill. Full guide:
+`docs/guide/data-profiling-app.md` and
+https://documentdrivendx.github.io/tablespec/

umf_tablespec-0.0.14/.agents/skills/tablespec-sql-plans/SKILL.md

@@ -0,0 +1,175 @@
+---
+name: tablespec-sql-plans
+description: Authoring derived/gold tables and generating SQL plans - table_type generated UMFs with derivation candidates, generate_sql_plan in views or cte mode, base_table_strategy (unpivot, union_sources, union_branches), per-branch filters and dedup, alternative join paths, and DuckDB/Spark byte-identical SQL. Use when defining a derived table from source UMFs or shaping the SQL plan a dbt or LDP gold model consumes.
+---
+
+# Tablespec SQL Plans
+
+Use this skill for authoring derived (gold) tables as UMF and shaping the SQL
+plans generated from them. It covers behavior and gotchas; the exhaustive
+strategy and metadata tables live in `docs/guide/sql-plans.md`.
+
+## What a SQL plan is
+
+`generate_sql_plan(target_umf, related_umfs, mode=...)` compiles a *generated*
+table's UMF — derivation candidates, relationships, metadata — into executable
+SQL. `mode="views"` (the default) emits a sequence of
+`CREATE OR REPLACE TEMPORARY VIEW` statements; `mode="cte"` emits a single
+`WITH ... SELECT`, the form dbt and LDP gold models consume.
+
+```python
+from tablespec import generate_sql_plan
+
+sql = generate_sql_plan(target_umf, related_umfs, mode="cte")
+```
+
+The emitted SQL is engine-agnostic and must stay that way: conformance tests
+pin DuckDB↔Spark byte-parity on every gold path. Do not emit engine-specific
+syntax — e.g. spell null-safe equality as `(a = b OR (a IS NULL AND b IS
+NULL))`, not `<=>` (Spark-only), and avoid `SELECT * EXCEPT/EXCLUDE`
+(dialect-divergent). Both filters and expressions run through
+`{{template_var}}` substitution via the `template_vars` argument.
+
+## Authoring a generated table
+
+Derived tables are `table_type: "generated"` UMFs whose columns carry a
+`derivation` (`UMFColumnDerivation`) with prioritized `candidates`
+(`DerivationCandidate`):
+
+```python
+UMFColumn(
+    name="claim_id",
+    data_type="VARCHAR",
+    length=50,
+    nullable=Nullable(MD=False, MP=False, ME=False),
+    derivation=UMFColumnDerivation(
+        strategy="primary_key",
+        candidates=[
+            DerivationCandidate(table="Medical_Claims", column="claim_id", priority=1),
+        ],
+    ),
+)
+```
+
+Candidate field semantics (each optional unless noted):
+
+- `table` (required) and `column` or `expression` (one required) — where the
+  value comes from; `expression` is a SQL expression instead of a bare column.
+- `priority` (required, `>= 1`) — survivorship order; 1 wins.
+- `join_filter` — extra predicate ANDed into the JOIN ON clause for this
+  table; used when joining the same table multiple times with different filters.
+- `table_instance` — unique alias for a table+filter combination; required to
+  disambiguate self-joins / multi-instance joins of the same table.
+- `row_filter` — WHERE applied to source rows before aggregation or union
+  branching.
+- `order_by` — window ORDER BY columns (all DESC); switches aggregation to
+  `ROW_NUMBER()` picking the full max row, and drives per-branch dedup.
+- `union_value` — literal emitted for this column in the UNION branch for this
+  table (typically a source discriminator flag).
+
+Do not author an empty `candidates: []` list — omit `derivation` entirely for
+columns with no source; they compile to `CAST(NULL AS <type>)`.
+
+## Base table strategies
+
+`metadata.base_table_strategy` selects how the plan's base view is built.
+
+**Unset** — plain base: `SELECT <required columns> FROM base_table`, filtered
+by `base_table_filter` if set.
+
+**`unpivot`** — UNPIVOTs the base table's `unpivot_columns` into rows; requires
+both `unpivot_columns` and `unpivot_value_column` in metadata. With
+`dedup_strategy: latest` and a `primary_key`, the *wide* row is deduped before
+the UNPIVOT — deduping after would collapse all unpivoted rows per key to one.
+
+**`union_sources`** — key-only universe: UNION of each source table's join key
+across `source_tables`, with per-source pre-aggregation views joined back.
+Unlike `union_branches`, it unions keys, not full rows.
+
+**`union_branches`** — one full SELECT branch per source table (base plus
+`union_base_tables`, falling back to `source_tables`), combined with
+`union_type: union_all` (default) or `union` (dedupes exact rows). Requires a
+resolvable `base_table`.
+
+**`aggregate_source`** — the base view is itself a GROUP BY over `base_table`;
+requires `metadata.base_table`.
+
+## union_branches specifics
+
+Every branch projects the full target column set, mapped through that source
+table's own candidates:
+
+- a candidate with `union_value` emits `CAST(<literal> AS <type>)` — a
+  per-branch constant;
+- otherwise the branch table's lowest-priority-number candidate supplies the
+  `expression` or `column`;
+- a column with no candidate for a branch emits `CAST(NULL AS <type>)`, so the
+  UNION stays column-aligned when sources have different columns.
+
+A branch's WHERE clause comes from the single distinct `row_filter` among that
+table's candidates — this is how generation cutovers are expressed (legacy feed
+`< DATE`, daily feed `>=`). Candidates of one branch carrying *different*
+`row_filter` values is a plan-time error, as are conflicting `order_by` lists.
+`base_table_filter` additionally applies to the base branch (ANDed).
+
+With `dedup_strategy: latest` and a candidate `order_by`, each branch is
+deduplicated before the union via `ROW_NUMBER() OVER (PARTITION BY <primary
+key> ORDER BY <order_by> DESC NULLS LAST)`; every `primary_key` column must be
+branch-projected. `NULLS LAST` is pinned because DuckDB and Spark default NULL
+placement differently.
+
+Overlap handling: `union_exclude_base: true` anti-joins union branches against
+the base branch's post-filter, post-dedup rows on the primary key;
+`union_coalesce_base: true` instead merges overlapping rows with
+`COALESCE(base.col, union.col)` (base wins; supports exactly one union table).
+Both require a primary key and raise without one. Joins to other tables still
+work after a union base view — join keys are projected into every branch.
+
+## Filters and joins
+
+- `base_table_filter` — WHERE on the base view, before any joins; bare
+  base-table columns only. `final_filter` — WHERE after final assembly, wrapped
+  as a subquery so it can reference derived output columns. Put predicates in
+  `base_table_filter` when they can go there; it filters earliest and cheapest.
+- `final_dedup: distinct` collapses exact-duplicate output rows;
+  `final_dedup: latest` requires `final_dedup_keys` (or `primary_key`) plus
+  `final_dedup_order_by`.
+- `base_join_column` overrides the auto-inferred base join key *and* every
+  relationship declared outgoing from the base table — set it only when every
+  join out of the base should use that one key.
+- Candidate-level `join_filter` takes precedence over `ForeignKey.join_filter`
+  because candidate filters are keyed by `(table, table_instance)` and can
+  disambiguate multi-instance joins; FK-level filters fill the gaps.
+- `OutgoingRelationship.alternative_joins` — additional join paths tried in
+  declared priority order (the relationship's own columns are priority 1).
+  Compiled as a **UNION-of-joins**, never `ON (a = b OR c = d)`: Spark plans
+  OR-joins as a BroadcastNestedLoopJoin, a known join-explosion hazard. Each
+  path becomes an inner-join branch, one match per base key survives by branch
+  priority, and the result joins back null-safely. Alternative-join columns
+  are validated against the actual tables at plan time.
+
+## Error philosophy
+
+Misconfiguration fails at *plan time* with `ValueError` — missing union
+tables, conflicting row_filters or order_bys, exclude/coalesce without a
+primary key, alternative-join columns that don't exist — rather than emitting
+broken SQL. Declarative fields the selected strategy does not consume (e.g.
+`union_base_tables` without `base_table_strategy: union_branches`, or
+`base_table_filter` under `unpivot`/`union_sources`) log a warning and are
+ignored so foreign-authored specs still load; the warning names the missing
+switch.
+
+## Where plans land
+
+`compile_umfs(..., gold_targets=["Claims_Summary"])` writes a single-target
+plan artifact per listed table to `gold_plan/<target>.plan.sql` (views mode).
+The dbt gold and LDP gold model renderers invoke the same generator in `cte`
+mode, so the model body is the plan — do not hand-edit generated gold SQL;
+change the UMF and regenerate.
+
+## Related
+
+- `tablespec-pipeline` — compiling UMFs into runtime artifacts and running them.
+- `tablespec-umf-authoring` — split-YAML spec layout and UMF authoring.
+- Full reference: `docs/guide/sql-plans.md` in the repository, and
+  https://documentdrivendx.github.io/tablespec/