vibedata-odcs 0.1.0__tar.gz

vibedata_odcs-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+.env
+.pytest_cache/

vibedata_odcs-0.1.0/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+The public API is the `odcs` command line: its commands and options, the JSON diagnostics on stdout (each diagnostic's `code` included), the `results` record shape, and the exit codes. `vibedata.odcs.cli.main`, `vibedata.odcs.cli.app` and `vibedata.odcs.warehouse.set_destination_resolver` (re-exported, with `DestinationResolver`, from `vibedata.odcs.cli`) are the Python API. Everything else is internal. While the version is 0.x, a breaking change bumps the minor version.
+
+## 0.1.0
+
+- First release. The console script is `vibedata-odcs` (also `python -m vibedata.odcs`). Ported from the plugin's `skills/data-contract/scripts/odcs.py` at vibedata-data-engineering `f73ce0b`, with no change to any command's behaviour or output.
+- `version` reports the package version beside the standard and schema pins.
+- The Intent's sandbox comes from a resolver the caller installs (`main(destination_resolver=...)`), not from the plugin's `platforms` module. With neither a resolver nor `--catalog`, a warehouse command refuses with `catalog/missing`.
+- `duckdb` moves behind the `warehouse` extra.

vibedata_odcs-0.1.0/PKG-INFO

@@ -0,0 +1,64 @@
+Metadata-Version: 2.5
+Name: vibedata-odcs
+Version: 0.1.0
+Summary: VibeData data-contract tooling: check, generate, reverse, diff, profile and results for Open Data Contract Standard (ODCS v3.2.0) contracts at a Domain's boundary. One library and one `odcs` command shared by the Build agent's skills, Domain CI and Studio.
+Project-URL: Repository, https://github.com/accelerate-data/vibedata-python-packages
+Project-URL: Changelog, https://github.com/accelerate-data/vibedata-python-packages/blob/main/packages/vibedata-odcs/CHANGELOG.md
+Author-email: Vibedata <eng@acceleratedata.ai>
+License: MIT
+Requires-Python: >=3.11
+Requires-Dist: jsonschema>=4.18
+Requires-Dist: pyyaml>=6
+Requires-Dist: typer>=0.13
+Provides-Extra: dev
+Requires-Dist: duckdb; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: warehouse
+Requires-Dist: duckdb; extra == 'warehouse'
+Description-Content-Type: text/markdown
+
+# vibedata-odcs
+
+Data-contract tooling for [Open Data Contract Standard](https://github.com/bitol-io/open-data-contract-standard) (ODCS v3.2.0) contracts at a VibeData Domain's boundary.
+One library and one `vibedata-odcs` command, shared by the Build agent's `data-contract` skills, Domain CI's logical-model gate, and Studio. It replaces the `odcs.py` script that was copied into each of them.
+
+```bash
+pip install vibedata-odcs              # check, generate, diff, results, version
+pip install "vibedata-odcs[warehouse]" # adds duckdb for check --relations, reverse and profile
+```
+
+Run from the Domain's repository root:
+
+| Command | What it does | Reads a warehouse |
+| --- | --- | --- |
+| `vibedata-odcs check` | schema and structure of the committed contracts, one per boundary relation, agreement with the canonical model | no |
+| `vibedata-odcs check --relations` | adds the comparison against the built relation (or dbt's `catalog.json`) | yes |
+| `vibedata-odcs reverse [--force]` | drafts the boundary's contracts from a built warehouse, with the open questions as findings | yes |
+| `vibedata-odcs generate` | writes the dbt keys and the dlt `COLUMNS` hints from the contracts | no |
+| `vibedata-odcs diff <base-ref>` | classifies each changed contract against a git base and requires the version bump it needs | no |
+| `vibedata-odcs profile [--relation S.T]` | tests each contract's key, `required` and value-set claims against the built relation | yes |
+| `vibedata-odcs results <run_results.json>` | maps a dbt run's test results to the contract assertions they enforce (JSON lines) | no |
+| `vibedata-odcs version` | package, standard and pinned-schema versions | no |
+
+Diagnostics are JSON on stdout. The exit code is 0 when there are no errors (warnings are allowed) and 1 otherwise. That JSON, its diagnostic `code` values and the exit codes are the public API this package versions (see `CHANGELOG.md`).
+
+The console script is `vibedata-odcs`, prefixed so it cannot collide with another tool's `odcs` in a shared environment; `python -m vibedata.odcs` is the same command.
+
+## The warehouse
+
+`--catalog PATH` names the local database file (or dbt's `catalog.json` for `check --relations`).
+Without it, the command asks the installed destination resolver. The library ships none, because it knows no platform. A caller that knows the Intent's platform passes one:
+
+```python
+from vibedata.odcs.cli import main
+
+main(destination_resolver=lambda repo_map: {"family": "duckdb", "path": "/sandbox.duckdb", "platform": "duckdb_local"})
+```
+
+With no resolver and no `--catalog`, a warehouse command refuses with `catalog/missing` rather than passing silently.
+
+The resolver is process-wide state: every `main()` call sets it, `None` included, so a long-lived process that calls `main()` repeatedly passes the resolver it wants on each call.
+
+## Vendored schema
+
+`src/vibedata/odcs/schemas/` holds the pinned ODCS JSON schema; see its README for provenance. Validation never fetches, so `check` runs offline.

vibedata_odcs-0.1.0/README.md

@@ -0,0 +1,45 @@
+# vibedata-odcs
+
+Data-contract tooling for [Open Data Contract Standard](https://github.com/bitol-io/open-data-contract-standard) (ODCS v3.2.0) contracts at a VibeData Domain's boundary.
+One library and one `vibedata-odcs` command, shared by the Build agent's `data-contract` skills, Domain CI's logical-model gate, and Studio. It replaces the `odcs.py` script that was copied into each of them.
+
+```bash
+pip install vibedata-odcs              # check, generate, diff, results, version
+pip install "vibedata-odcs[warehouse]" # adds duckdb for check --relations, reverse and profile
+```
+
+Run from the Domain's repository root:
+
+| Command | What it does | Reads a warehouse |
+| --- | --- | --- |
+| `vibedata-odcs check` | schema and structure of the committed contracts, one per boundary relation, agreement with the canonical model | no |
+| `vibedata-odcs check --relations` | adds the comparison against the built relation (or dbt's `catalog.json`) | yes |
+| `vibedata-odcs reverse [--force]` | drafts the boundary's contracts from a built warehouse, with the open questions as findings | yes |
+| `vibedata-odcs generate` | writes the dbt keys and the dlt `COLUMNS` hints from the contracts | no |
+| `vibedata-odcs diff <base-ref>` | classifies each changed contract against a git base and requires the version bump it needs | no |
+| `vibedata-odcs profile [--relation S.T]` | tests each contract's key, `required` and value-set claims against the built relation | yes |
+| `vibedata-odcs results <run_results.json>` | maps a dbt run's test results to the contract assertions they enforce (JSON lines) | no |
+| `vibedata-odcs version` | package, standard and pinned-schema versions | no |
+
+Diagnostics are JSON on stdout. The exit code is 0 when there are no errors (warnings are allowed) and 1 otherwise. That JSON, its diagnostic `code` values and the exit codes are the public API this package versions (see `CHANGELOG.md`).
+
+The console script is `vibedata-odcs`, prefixed so it cannot collide with another tool's `odcs` in a shared environment; `python -m vibedata.odcs` is the same command.
+
+## The warehouse
+
+`--catalog PATH` names the local database file (or dbt's `catalog.json` for `check --relations`).
+Without it, the command asks the installed destination resolver. The library ships none, because it knows no platform. A caller that knows the Intent's platform passes one:
+
+```python
+from vibedata.odcs.cli import main
+
+main(destination_resolver=lambda repo_map: {"family": "duckdb", "path": "/sandbox.duckdb", "platform": "duckdb_local"})
+```
+
+With no resolver and no `--catalog`, a warehouse command refuses with `catalog/missing` rather than passing silently.
+
+The resolver is process-wide state: every `main()` call sets it, `None` included, so a long-lived process that calls `main()` repeatedly passes the resolver it wants on each call.
+
+## Vendored schema
+
+`src/vibedata/odcs/schemas/` holds the pinned ODCS JSON schema; see its README for provenance. Validation never fetches, so `check` runs offline.

vibedata_odcs-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "vibedata-odcs"
+version = "0.1.0"
+description = "VibeData data-contract tooling: check, generate, reverse, diff, profile and results for Open Data Contract Standard (ODCS v3.2.0) contracts at a Domain's boundary. One library and one `odcs` command shared by the Build agent's skills, Domain CI and Studio."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [{ name = "Vibedata", email = "eng@acceleratedata.ai" }]
+
+dependencies = [
+    # >=4.18: the referencing-based resolver the vendored v3.2.0 schema's $defs need.
+    "jsonschema>=4.18",
+    "pyyaml>=6",
+    # >=0.13: 0.12 cannot run against the current click ("Secondary flag is not valid
+    # for non-boolean flag"); proven by the lowest-direct CI job.
+    "typer>=0.13",
+]
+
+[project.optional-dependencies]
+# Read only by `check --relations`, `reverse` and `profile`. Its version is the
+# runtime's own (the managed constraints pin it), so no floor is stated here.
+warehouse = ["duckdb"]
+dev = ["pytest>=8.0", "duckdb"]
+
+[project.urls]
+Repository = "https://github.com/accelerate-data/vibedata-python-packages"
+Changelog = "https://github.com/accelerate-data/vibedata-python-packages/blob/main/packages/vibedata-odcs/CHANGELOG.md"
+
+[project.scripts]
+vibedata-odcs = "vibedata.odcs.cli:main"
+
+# PEP 420 namespace package: `vibedata` carries no __init__.py, so this
+# distribution contributes `vibedata.odcs` beside `vibedata.dlt.*`.
+[tool.hatch.build.targets.wheel]
+packages = ["src/vibedata"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

vibedata_odcs-0.1.0/src/vibedata/odcs/__init__.py

@@ -0,0 +1,13 @@
+"""VibeData data-contract tooling for Open Data Contract Standard (ODCS) contracts.
+
+The command line is the public interface: `odcs <command>` (or `python -m vibedata.odcs`)
+prints JSON diagnostics and exits 0 or 1. `vibedata.odcs.cli.main` runs it in-process and
+accepts a destination resolver for callers that know the Intent's platform.
+"""
+
+from importlib.metadata import PackageNotFoundError, version as _version
+
+try:
+    __version__ = _version("vibedata-odcs")
+except PackageNotFoundError:  # a source checkout run without installing the distribution
+    __version__ = "0.0.0+unknown"

vibedata_odcs-0.1.0/src/vibedata/odcs/__main__.py

@@ -0,0 +1,3 @@
+from vibedata.odcs.cli import main
+
+main()

vibedata_odcs-0.1.0/src/vibedata/odcs/boundary.py

@@ -0,0 +1,297 @@
+"""Where a contract is owed: the Domain's repo map, its layers' roles, and the boundary relations."""
+
+import json
+from pathlib import Path
+
+import yaml
+
+from vibedata.odcs.diagnostics import _Refused, _diag
+
+# Relations the installed loader library writes into every dlt schema. No dlt resource declares
+# them, so none is on the boundary, a contract on one is no orphan, and nothing is generated
+# from it.
+AUDIT_RELATIONS = ("audit_runs", "audit_table_loads")
+
+
+def _repo_root(root: str) -> Path:
+    """The Domain root: `--root`, else the nearest directory above the working one with a map."""
+    start = Path(root).resolve() if root else Path.cwd().resolve()
+    for candidate in [start, *start.parents]:
+        if (candidate / "repo-map.json").is_file():
+            return candidate
+    raise _Refused(_diag("error", "map/missing",
+                         f"no repo-map.json at or above {start}; run from the Domain's root"))
+
+
+def _repo_map(root: Path) -> dict:
+    return json.loads((root / "repo-map.json").read_text(encoding="utf-8"))
+
+
+def _declared(root: Path, raw: str) -> Path:
+    """The path an artifact pattern names, up to its first placeholder."""
+    prefix = raw.split("{")[0].split("*")[0].rstrip("/")
+    return root / prefix
+
+
+def declared_relations(root: Path, repo_map: dict) -> dict[tuple[str, str], str]:
+    """Every relation the map declares, as `(schema, table)` mapped to what declares it.
+
+    dbt models and seeds come from the layer paths. Raw relations come from each source's
+    resource docs. The loader's audit relations are written by the installed library and appear
+    in none of these: a contract on one is no orphan, and none is owed.
+    """
+    expected: dict[tuple[str, str], str] = {}
+    for name, layer in (repo_map.get("layers") or {}).items():
+        path = root / layer["path"] if layer.get("path") else None
+        if path is None or not path.is_dir() or layer.get("built_by") != "dbt":
+            continue
+        # A dbt layer builds whatever it holds: a model from a `.sql`, a seed from a `.csv`.
+        # Which suffix belongs to which is the file's fact, not the layer's name.
+        for suffix in (".sql", ".csv"):
+            for file in sorted(path.rglob(f"*{suffix}")):
+                expected[(layer["schema"], file.stem)] = f"layer {name}"
+    for name, source in (repo_map.get("sources") or {}).items():
+        docs = source.get("docs")
+        if not docs:
+            continue
+        entries = yaml.safe_load((root / docs).read_text(encoding="utf-8")) or {}
+        for resource in sorted(entries.get("resources") or {}):
+            expected[(source["dataset"], resource)] = f"source {name}"
+    return expected
+
+
+def _unknown_gold_layers(repo_map: dict) -> list[dict]:
+    """A legacy `gold_layers` entry that names no layer declares no boundary; warn, so a stale entry never fails an existing Domain."""
+    named = repo_map.get("gold_layers")
+    if named is None:
+        return []
+    if not isinstance(named, list):
+        return [_diag("warning", "map/gold-layers",
+                      "repo-map.json `gold_layers` is not a list of layer names")]
+    layers = repo_map.get("layers") or {}
+    return [
+        _diag("warning", "map/gold-layers",
+              f"repo-map.json `gold_layers` names `{name}`, which `layers` does not declare")
+        for name in named
+        if name not in layers
+    ]
+
+
+def _unknown_contract_layers(repo_map: dict) -> list[dict]:
+    """A `contract_layers` entry that names no layer scopes nothing; saying so is the report."""
+    named = repo_map.get("contract_layers")
+    if named is None:
+        return []
+    if not isinstance(named, list):
+        return [_diag("error", "map/contract-layers",
+                      "repo-map.json `contract_layers` is not a list of layer names")]
+    layers = repo_map.get("layers") or {}
+    return [
+        _diag("error", "map/contract-layers",
+              f"repo-map.json `contract_layers` names `{name}`, which `layers` does not declare")
+        for name in named
+        if name not in layers
+    ]
+
+
+def _exclusions(repo_map: dict) -> tuple[dict[tuple[str, str], str], list[dict]]:
+    """The relations the canonical model deliberately leaves out, with the reason each states.
+
+    An entry the tool cannot read excludes nothing; saying so is the report.
+    """
+    entries = repo_map.get("semantic_exclusions")
+    if entries is None:
+        return {}, []
+    if not isinstance(entries, list):
+        return {}, [_diag("error", "map/semantic-exclusions",
+                          "repo-map.json `semantic_exclusions` is not a list of "
+                          "{relation, reason} entries")]
+    excluded: dict[tuple[str, str], str] = {}
+    diags = []
+    for entry in entries:
+        relation = entry.get("relation") if isinstance(entry, dict) else None
+        reason = entry.get("reason") if isinstance(entry, dict) else None
+        if (isinstance(relation, str) and relation.count(".") == 1 and all(relation.split("."))
+                and isinstance(reason, str) and reason.strip()):
+            schema, table = relation.split(".")
+            excluded[(schema, table)] = reason.strip()
+        else:
+            diags.append(_diag("error", "map/semantic-exclusions",
+                               f"repo-map.json `semantic_exclusions` entry {entry!r} needs a "
+                               f"`relation` as `schema.table` and a `reason`"))
+    return excluded, diags
+
+
+LAYER_ROLES = ("input", "internal", "boundary")
+
+
+def _role_for(repo_map: dict, name: str) -> str:
+    """A layer's role: `layers.<name>.role` when valid, else derived from the legacy keys.
+
+    Derived: `built_by == "dlt"` is `input` (even when the layer is also named in `gold_layers`);
+    a name in `gold_layers` (the legacy way to declare a
+    boundary layer) is `boundary`; anything else is `internal`. An invalid explicit role falls
+    back to the derivation; `check` reports it as `map/layer-role`.
+    """
+    layer = (repo_map.get("layers") or {}).get(name)
+    layer = layer if isinstance(layer, dict) else {}
+    if layer.get("role") in LAYER_ROLES:
+        return layer["role"]
+    if layer.get("built_by") == "dlt":
+        return "input"
+    if name in (repo_map.get("gold_layers") or []):
+        return "boundary"
+    return "internal"
+
+
+def _roles(repo_map: dict) -> dict[str, str]:
+    """The role of every layer the map declares."""
+    return {name: _role_for(repo_map, name) for name in repo_map.get("layers") or {}}
+
+
+def _role_schemas(repo_map: dict, *roles: str) -> set[str]:
+    """Schemas of the layers whose role is one of `roles`."""
+    layers = repo_map.get("layers") or {}
+    return {layers[name]["schema"] for name, role in _roles(repo_map).items()
+            if role in roles and "schema" in layers[name]}
+
+
+def _layer_role_diags(repo_map: dict) -> list[dict]:
+    """A `role` outside `input`, `internal` and `boundary` is reported, not guessed at."""
+    return [
+        _diag("error", "map/layer-role",
+              f"repo-map.json layers.{name}.role must be input, internal or boundary")
+        for name, layer in (repo_map.get("layers") or {}).items()
+        if isinstance(layer, dict) and "role" in layer and layer["role"] not in LAYER_ROLES
+    ]
+
+
+def boundary_relations(repo_map: dict,
+                       declared: dict[tuple[str, str], str]) -> dict[tuple[str, str], str]:
+    """The relations a contract is owed for: the inputs and outputs of the Domain.
+
+    By default, every relation a dlt source loads (a consumer of the load sees its shape first)
+    and every relation a layer with role `input` or `boundary` builds (what a consumer outside
+    the repo reads). Layers with role `internal` are guarded by their dbt tests and owe none.
+    `contract_layers`, when the map declares it, replaces that default with the layers it names, a dlt layer
+    contributing the resources its sources land in that schema.
+    """
+    layers = repo_map.get("layers") or {}
+    named = repo_map.get("contract_layers")
+    if isinstance(named, list):
+        schemas = {layers[name]["schema"] for name in named if name in layers}
+        return {relation: by for relation, by in declared.items() if relation[0] in schemas}
+    owed = _role_schemas(repo_map, "input", "boundary")
+    # Only an explicit `role: internal` stops a source from owing a contract; a layer whose role
+    # is merely derived as internal still owes one for the sources that land in it.
+    internal = {layer["schema"] for layer in layers.values()
+                if isinstance(layer, dict) and layer.get("role") == "internal" and "schema" in layer}
+    return {
+        relation: by
+        for relation, by in declared.items()
+        if (by.startswith("source ") and relation[0] not in internal)
+        or (relation[0] in owed and by.startswith("layer "))
+    }
+
+
+def _boundary_rank(relation: tuple[str, str], repo_map: dict,
+                   declared: dict[tuple[str, str], str]) -> int:
+    """Boundary first, then inputs, then anything else: the order a reviewer triages in."""
+    if relation[0] in _role_schemas(repo_map, "boundary"):
+        return 0
+    if (declared.get(relation) or "").startswith("source "):
+        return 1
+    return 2
+
+
+def _boundary_schemas(repo_map: dict, expected: dict[tuple[str, str], str]) -> set[str]:
+    """Schemas holding boundary relations: the layers with role `boundary`, where they are built.
+
+    The map is the only source. Another Domain names its layers differently, so a layer name is
+    something this tool cannot know; guessing one — by exclusion, or by the shape of the
+    directory — would gate relations the Domain never called boundary and pass over the ones it did.
+    An undeclared set is therefore empty and names nothing, and `check` warns about the gap
+    instead of inventing a boundary set to fill it.
+    """
+    built = {schema for schema, _ in expected}
+    return _role_schemas(repo_map, "boundary") & built
+
+
+def _layer_for(repo_map: dict, schema: str) -> dict | None:
+    """The layer that declares a schema; the map is the only place a schema is named."""
+    for layer in (repo_map.get("layers") or {}).values():
+        if layer.get("schema") == schema:
+            return layer
+    return None
+
+
+def _is_dlt(relation: tuple[str, str], repo_map: dict) -> bool:
+    """Whether a dlt pipeline builds the relation: the only place the loader adds `_dlt_*`."""
+    layer = _layer_for(repo_map, relation[0])
+    return layer is not None and layer.get("built_by") == "dlt"
+
+
+def _is_dbt(relation: tuple[str, str], repo_map: dict) -> bool:
+    """Whether a dbt layer builds the relation, as opposed to a dlt pipeline or the loader."""
+    layer = _layer_for(repo_map, relation[0])
+    return layer is not None and layer.get("built_by") == "dbt"
+
+
+def _built_models(root: Path, layer: dict) -> set[str] | None:
+    """The models dbt built successfully, from the `target/run_results.json` of the project
+    holding the layer's path; None when there is no such file (no evidence either way)."""
+    folder = (root / layer["path"]).resolve()
+    for project in (folder, *folder.parents):
+        if (project / "dbt_project.yml").is_file():
+            break
+        if project == root.resolve():
+            return None
+    else:
+        return None
+    path = project / "target" / "run_results.json"
+    if not path.is_file():
+        return None
+    try:
+        found = json.loads(path.read_text(encoding="utf-8")).get("results") or []
+    except (OSError, ValueError, AttributeError):
+        return None
+    return {str(result.get("unique_id", "")).split(".")[-1] for result in found
+            if isinstance(result, dict) and result.get("status") in ("success", "pass")
+            and str(result.get("unique_id", "")).split(".")[0] in ("model", "seed", "snapshot")}
+
+
+def _active_unbuilt(root: Path, repo_map: dict, relation: tuple[str, str], path: Path,
+                    doc: dict, excluded: dict) -> list[dict]:
+    """An `active` contract whose dbt relation the last run did not build: still a proposal."""
+    if doc.get("status") != "active" or relation in excluded or not _is_dbt(relation, repo_map):
+        return []
+    built = _built_models(root, _layer_for(repo_map, relation[0]))
+    if built is None or relation[1] in built:
+        return []
+    return [_diag("warning", "contract/active-unbuilt",
+                  f"{relation[0]}.{relation[1]}: status is active but dbt has not built "
+                  f"{relation[0]}.{relation[1]}; mark it proposed until the table exists", path)]
+
+
+def _is_loader_column(relation: tuple[str, str], name: str, repo_map: dict) -> bool:
+    """A `_dlt_*` column dlt adds to a relation it loaded.
+
+    A dbt model that selects the column propagates it deliberately, and its enforced contract
+    must declare every output column, so only a raw relation excludes the loader's own.
+    """
+    return name.startswith("_dlt") and not _is_dbt(relation, repo_map)
+
+
+def _is_audit(relation: tuple[str, str], repo_map: dict) -> bool:
+    """A relation the installed loader writes into a dlt layer's schema, not a dlt resource."""
+    layer = _layer_for(repo_map, relation[0])
+    return (
+        layer is not None
+        and layer.get("built_by") == "dlt"
+        and relation[1] in AUDIT_RELATIONS
+    )
+
+
+def _platform(repo_map: dict) -> str:
+    """The platform the map declares. One Domain runs on one, so the map is the only source."""
+    return str(repo_map.get("data_platform") or "")