rocky-sdk 0.1.0__tar.gz

rocky_sdk-0.1.0/.gitignore

@@ -0,0 +1,62 @@
+# Root .gitignore — cross-cutting safety nets for the Rocky monorepo.
+# Each subproject (engine/, integrations/dagster/, editors/vscode/, examples/playground/)
+# has its own .gitignore tuned to its build system. Patterns here are universal.
+
+# --- Secrets — never commit ---
+.env
+.env.*
+!.env.example
+*.pem
+*.key
+id_rsa*
+secrets.*
+credentials.*
+
+# --- OS / editor noise ---
+.DS_Store
+Thumbs.db
+*.swp
+*.swo
+*~
+.idea/
+
+# --- Rocky runtime state (any subproject) ---
+*.redb
+*.duckdb
+*.duckdb.wal
+.rocky-state*
+
+# --- Vendored binaries (downloaded by scripts/vendor_*.sh) ---
+vendor/
+
+# --- Per-user Claude state (skills + shared settings.json are committed) ---
+.claude/projects/
+.claude/worktrees/
+.claude/settings.local.json
+.claude/scheduled_tasks.lock
+
+# --- Defense-in-depth duplicates of subproject ignores ---
+# (in case someone runs cargo / npm / uv from the wrong directory)
+target/
+target-linux/
+node_modules/
+dist/
+out/
+build/
+*.vsix
+.vscode-test/
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+
+# --- Astro build cache ---
+docs/.astro/
+
+# --- Logs and scratch ---
+*.log
+logs/
+scratch/
+tmp/

rocky_sdk-0.1.0/CLAUDE.md

@@ -0,0 +1,92 @@
+# rocky-sdk
+
+Standalone, framework-agnostic Python client for the Rocky SQL transformation
+engine. `RockyClient` wraps the `rocky` CLI binary (subprocess + `--output json`)
+behind typed methods and raises a `RockyError` hierarchy. For notebooks, scripts,
+and orchestrators. The `dagster-rocky` integration is a thin Dagster adapter over
+this client.
+
+**Positioning:** the SDK is a typed, ergonomic client over the CLI for *human
+Python callers*. It complements — does not replace — `rocky mcp` (the surface for
+AI agents) and `rocky serve` (the language-agnostic HTTP surface).
+
+## Architecture
+
+Three layers, each independently usable:
+
+1. **Types** (`types.py` + `types_generated/`) — Pydantic models for every Rocky
+   CLI command's JSON output. The *generated* models in `types_generated/` are
+   autogenerated from `../../schemas/*.schema.json` via `datamodel-code-generator`
+   (run `just codegen-sdk` from the monorepo root). `types.py` re-exports them
+   under both generated and legacy names and defines `parse_rocky_output()`, a
+   table-driven dispatch that auto-detects the command and returns the matching
+   model. **Do not hand-edit `types_generated/`** — it's clobbered on regen.
+2. **Client** (`client.py`) — `RockyClient`. Binary resolution, lazy version gate
+   (`MIN_ROCKY_VERSION`), argv builders, the subprocess core (`run_cli`), the
+   `rocky serve` HTTP fallback, the governance pre-flight, and one typed method
+   per CLI command. `mirror_stderr` and `logger` are constructor settings so an
+   adapter (dagster-rocky) can preserve its own stderr-capture + logger provenance.
+3. **Exceptions** (`exceptions.py`) — `RockyError` base + subclasses
+   (`RockyBinaryNotFoundError`, `RockyVersionError`, `RockyTimeoutError`,
+   `RockyCommandError`, `RockyPartialFailure`, `RockyOutputParseError`,
+   `RockyServerError`, `RockyGovernanceError`). Each carries structured fields so
+   a caller can rebuild a rich error (the dagster adapter rebuilds `dagster.Failure`).
+
+Subprocess plumbing (`_subprocess.py`): single reader thread per pipe plus an
+external watchdog that kills the process group on timeout — never
+`communicate(timeout=)`, which races with a concurrent pipe reader. This pattern
+is load-bearing; preserve it.
+
+## Project structure
+
+```
+src/rocky_sdk/
+├── __init__.py        # Public API: RockyClient, exceptions, all result types
+├── client.py          # RockyClient + module-level parse/governance helpers
+├── exceptions.py      # RockyError hierarchy
+├── _subprocess.py     # reader threads, watchdog, kill, redact (framework-agnostic)
+├── types.py           # parse_rocky_output + dispatch + hand-written models
+└── types_generated/   # Autogenerated from ../../schemas/*.schema.json (do not edit)
+tests/
+├── test_client.py     # argv / version / subprocess / governance / HTTP (mocked binary)
+└── test_types.py      # parse_rocky_output dispatch + parse-helper error paths
+```
+
+## Coding standards
+
+- Python 3.11+, `from __future__ import annotations` in all modules.
+- Pydantic `BaseModel` for data structures. Line length 100.
+- Ruff rules: E, F, I, N, UP, B, SIM. `types_generated/` is excluded from lint/format.
+- Runtime dependency: `pydantic>=2.0` only — keep the SDK free of heavy deps.
+
+## Common commands
+
+```bash
+uv sync --dev
+uv run pytest -v                       # all tests (no binary or creds needed)
+uv run ruff check src/ tests/
+uv run ruff format --check src/ tests/  # CI gate
+uv build                               # wheel + sdist
+```
+
+## Adding support for a new Rocky CLI command
+
+1. Add the typed `*Output` struct in `engine/crates/rocky-cli/src/output.rs` and
+   register it in `commands/export_schemas.rs::schemas()`.
+2. From the monorepo root, run `just codegen-sdk` (or `just codegen`).
+3. Re-export the new type from `types.py` (bridge section) if needed and add a
+   `parse_rocky_output()` dispatch entry.
+4. Add a typed method to `RockyClient` and a delegating method to
+   `RockyResource` (dagster-rocky), translating any new error shape.
+
+## Release
+
+Tag-namespaced: `sdk-v*` → `rocky-sdk` wheel on PyPI (`sdk-release.yml`).
+`just release-sdk <version> [--publish]`. **Release the SDK before any
+`dagster-rocky` release that raises its `rocky-sdk>=…` floor** — the published
+dagster wheel resolves the SDK from PyPI, not the monorepo path source.
+
+## Git conventions
+
+- **Never** include `Co-Authored-By` trailers. Conventional commits, scoped
+  `feat(sdk): …` / `fix(sdk): …`.

rocky_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: rocky-sdk
+Version: 0.1.0
+Summary: Typed Python client for the Rocky SQL transformation engine
+Project-URL: Homepage, https://rocky-data.dev/
+Project-URL: Repository, https://github.com/rocky-data/rocky
+Project-URL: Documentation, https://rocky-data.dev/
+Project-URL: Bug Tracker, https://github.com/rocky-data/rocky/issues
+Author-email: Hugo Correia <hello@rocky-data.dev>
+License-Expression: Apache-2.0
+Keywords: client,data-engineering,rocky,sdk,sql,transformation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Requires-Dist: pydantic>=2.0
+Description-Content-Type: text/markdown
+
+# rocky-sdk
+
+A typed Python client for the [Rocky](https://rocky-data.dev/) SQL transformation engine.
+
+`rocky-sdk` wraps the `rocky` CLI binary (subprocess + `--output json`) behind a
+typed `RockyClient`. Each method builds the right argv, runs the binary, parses
+the JSON output, and returns a Pydantic model. Failures surface as `RockyError`
+subclasses carrying structured fields (exit code, stderr tail, version strings)
+rather than opaque messages.
+
+It is for **human Python callers** — notebooks, scripts, and orchestrators. The
+[`dagster-rocky`](https://pypi.org/project/dagster-rocky/) integration is a thin
+Dagster adapter built on this client. For AI agents, use `rocky mcp`; for a
+language-agnostic HTTP surface, use `rocky serve`.
+
+## Install
+
+```bash
+pip install rocky-sdk
+```
+
+The `rocky` binary is not bundled — install it separately and put it on `$PATH`
+(or pass `binary_path=`). See the
+[releases page](https://github.com/rocky-data/rocky/releases). The SDK requires
+engine **v1.34.0 or newer**.
+
+## Usage
+
+```python
+from rocky_sdk import RockyClient
+
+client = RockyClient(config_path="rocky.toml")
+
+# Read-only inspection — all return typed Pydantic models
+result = client.compile()
+for diag in result.diagnostics:
+    print(diag.severity, diag.message)
+
+lineage = client.lineage("customer_orders", column="email")
+catalog = client.catalog()
+
+# Execute a pipeline; stream live progress to a callback
+run = client.run(filter="tenant=acme", log_callback=print)
+print(run.summary)
+```
+
+### Errors
+
+```python
+from rocky_sdk import RockyClient
+from rocky_sdk.exceptions import RockyVersionError, RockyCommandError, RockyTimeoutError
+
+client = RockyClient(config_path="rocky.toml", timeout_seconds=600)
+try:
+    client.run(filter="tenant=acme")
+except RockyTimeoutError as exc:
+    print("timed out after", exc.timeout_seconds, "s")
+    print(exc.stderr_tail)
+except RockyCommandError as exc:
+    print("exit", exc.returncode)
+    print(exc.stderr_tail)
+```
+
+`RockyError` is the base of the hierarchy:
+
+| Exception | Raised when |
+|---|---|
+| `RockyBinaryNotFoundError` | the `rocky` binary is missing |
+| `RockyVersionError` | the binary is older than the SDK's minimum |
+| `RockyTimeoutError` | a command exceeds `timeout_seconds` |
+| `RockyCommandError` | a command exits non-zero |
+| `RockyPartialFailure` | a non-zero run still returned a parseable partial result (only with `allow_partial=False`) |
+| `RockyOutputParseError` | stdout was not the expected JSON shape |
+| `RockyServerError` | a `rocky serve` HTTP request failed |
+| `RockyGovernanceError` | a `governance_override` would silently full-revoke |
+
+## License
+
+Apache-2.0

rocky_sdk-0.1.0/README.md

@@ -0,0 +1,79 @@
+# rocky-sdk
+
+A typed Python client for the [Rocky](https://rocky-data.dev/) SQL transformation engine.
+
+`rocky-sdk` wraps the `rocky` CLI binary (subprocess + `--output json`) behind a
+typed `RockyClient`. Each method builds the right argv, runs the binary, parses
+the JSON output, and returns a Pydantic model. Failures surface as `RockyError`
+subclasses carrying structured fields (exit code, stderr tail, version strings)
+rather than opaque messages.
+
+It is for **human Python callers** — notebooks, scripts, and orchestrators. The
+[`dagster-rocky`](https://pypi.org/project/dagster-rocky/) integration is a thin
+Dagster adapter built on this client. For AI agents, use `rocky mcp`; for a
+language-agnostic HTTP surface, use `rocky serve`.
+
+## Install
+
+```bash
+pip install rocky-sdk
+```
+
+The `rocky` binary is not bundled — install it separately and put it on `$PATH`
+(or pass `binary_path=`). See the
+[releases page](https://github.com/rocky-data/rocky/releases). The SDK requires
+engine **v1.34.0 or newer**.
+
+## Usage
+
+```python
+from rocky_sdk import RockyClient
+
+client = RockyClient(config_path="rocky.toml")
+
+# Read-only inspection — all return typed Pydantic models
+result = client.compile()
+for diag in result.diagnostics:
+    print(diag.severity, diag.message)
+
+lineage = client.lineage("customer_orders", column="email")
+catalog = client.catalog()
+
+# Execute a pipeline; stream live progress to a callback
+run = client.run(filter="tenant=acme", log_callback=print)
+print(run.summary)
+```
+
+### Errors
+
+```python
+from rocky_sdk import RockyClient
+from rocky_sdk.exceptions import RockyVersionError, RockyCommandError, RockyTimeoutError
+
+client = RockyClient(config_path="rocky.toml", timeout_seconds=600)
+try:
+    client.run(filter="tenant=acme")
+except RockyTimeoutError as exc:
+    print("timed out after", exc.timeout_seconds, "s")
+    print(exc.stderr_tail)
+except RockyCommandError as exc:
+    print("exit", exc.returncode)
+    print(exc.stderr_tail)
+```
+
+`RockyError` is the base of the hierarchy:
+
+| Exception | Raised when |
+|---|---|
+| `RockyBinaryNotFoundError` | the `rocky` binary is missing |
+| `RockyVersionError` | the binary is older than the SDK's minimum |
+| `RockyTimeoutError` | a command exceeds `timeout_seconds` |
+| `RockyCommandError` | a command exits non-zero |
+| `RockyPartialFailure` | a non-zero run still returned a parseable partial result (only with `allow_partial=False`) |
+| `RockyOutputParseError` | stdout was not the expected JSON shape |
+| `RockyServerError` | a `rocky serve` HTTP request failed |
+| `RockyGovernanceError` | a `governance_override` would silently full-revoke |
+
+## License
+
+Apache-2.0

rocky_sdk-0.1.0/pyproject.toml

@@ -0,0 +1,59 @@
+[project]
+name = "rocky-sdk"
+version = "0.1.0"
+description = "Typed Python client for the Rocky SQL transformation engine"
+readme = "README.md"
+license = "Apache-2.0"
+requires-python = ">=3.11"
+authors = [{ name = "Hugo Correia", email = "hello@rocky-data.dev" }]
+keywords = ["rocky", "data-engineering", "sql", "transformation", "client", "sdk"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Database",
+    "Topic :: Software Development :: Libraries",
+]
+dependencies = [
+    "pydantic>=2.0",
+]
+
+[project.urls]
+Homepage = "https://rocky-data.dev/"
+Repository = "https://github.com/rocky-data/rocky"
+Documentation = "https://rocky-data.dev/"
+"Bug Tracker" = "https://github.com/rocky-data/rocky/issues"
+
+[dependency-groups]
+dev = [
+    "datamodel-code-generator>=0.56.0",
+    "pytest>=8.0",
+    "ruff>=0.4",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/rocky_sdk"]
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+# Generated code is produced by datamodel-code-generator from
+# ../../schemas/*.schema.json via `just codegen-sdk`. Hand-editing is
+# forbidden (it gets clobbered on next regen) and the generator's output
+# style doesn't align with our ruff ruleset — linting it produces noise
+# without fixable signal. Exclude from both lint and format.
+extend-exclude = ["src/rocky_sdk/types_generated"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "B", "SIM"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

rocky_sdk-0.1.0/src/rocky_sdk/__init__.py

@@ -0,0 +1,48 @@
+"""rocky-sdk: a typed Python client for the Rocky SQL transformation engine.
+
+The SDK wraps the ``rocky`` CLI binary (subprocess + ``--output json``) behind
+a typed :class:`~rocky_sdk.client.RockyClient`. Each method builds the right
+argv, runs the binary, parses the JSON output, and returns a Pydantic model
+from :mod:`rocky_sdk.types`. Errors surface as
+:class:`~rocky_sdk.exceptions.RockyError` subclasses.
+
+For human Python callers — notebooks, scripts, and orchestrators. The
+``dagster-rocky`` integration is a thin Dagster adapter over this client.
+For AI agents use ``rocky mcp``; for a language-agnostic HTTP surface use
+``rocky serve``.
+
+    from rocky_sdk import RockyClient
+
+    client = RockyClient(config_path="rocky.toml")
+    result = client.compile()
+    run = client.run(filter="tenant=acme", log_callback=print)
+"""
+
+from __future__ import annotations
+
+from rocky_sdk.client import RockyClient
+from rocky_sdk.exceptions import (
+    RockyBinaryNotFoundError,
+    RockyCommandError,
+    RockyError,
+    RockyOutputParseError,
+    RockyPartialFailure,
+    RockyTimeoutError,
+    RockyVersionError,
+)
+from rocky_sdk.types import *  # noqa: F401,F403  (re-export the full typed surface)
+from rocky_sdk.types import __all__ as _types_all
+from rocky_sdk.types import parse_rocky_output
+
+__all__ = [
+    "RockyClient",
+    "RockyError",
+    "RockyBinaryNotFoundError",
+    "RockyCommandError",
+    "RockyPartialFailure",
+    "RockyTimeoutError",
+    "RockyVersionError",
+    "RockyOutputParseError",
+    "parse_rocky_output",
+    *_types_all,
+]

rocky_sdk-0.1.0/src/rocky_sdk/_subprocess.py

@@ -0,0 +1,121 @@
+"""Low-level subprocess plumbing shared by :class:`rocky_sdk.client.RockyClient`.
+
+Single-reader-per-pipe + external watchdog. Two dedicated threads are the *sole*
+readers of ``proc.stdout`` / ``proc.stderr`` while a third watchdog thread
+enforces the wall-clock timeout by killing the process group — never
+``communicate(timeout=)``, which races with a concurrent pipe reader and was the
+root cause of intermittent multi-hour hangs in production. See
+:meth:`RockyClient._run_with_log_sink` for how the pieces fit together.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import logging
+import os
+import signal
+import subprocess
+import sys
+from collections.abc import Callable, Iterable
+from typing import IO
+
+_log = logging.getLogger("rocky_sdk")
+
+#: Argv flags whose immediately-following value is credential-bearing or
+#: otherwise sensitive. When the constructed argv is logged, the value of any
+#: matching flag is masked. The subprocess itself still receives the real value
+#: — only the log line is redacted.
+_REDACTED_ARGV_FLAGS = frozenset({"--governance-override", "--idempotency-key"})
+
+
+def redact_argv(argv: list[str]) -> list[str]:
+    """Return a copy of ``argv`` with credential-bearing flag values masked.
+
+    Walks left-to-right; whenever a token is in :data:`_REDACTED_ARGV_FLAGS`,
+    the *next* token (its value) is replaced with ``"***"``. For log output
+    only — never for the argv handed to the subprocess.
+    """
+    out: list[str] = []
+    redact_next = False
+    for token in argv:
+        if redact_next:
+            out.append("***")
+            redact_next = False
+            continue
+        out.append(token)
+        if token in _REDACTED_ARGV_FLAGS:
+            redact_next = True
+    return out
+
+
+def forward_stderr_to_sink(
+    stderr: Iterable[str] | None,
+    log_line: Callable[[str], None],
+    sink: list[str],
+    *,
+    mirror_to_stderr: bool = False,
+) -> None:
+    """Reader-thread body: forward rocky stderr lines to ``log_line`` and ``sink``.
+
+    Reads ``stderr`` line-by-line until EOF. Each non-empty line is appended to
+    ``sink`` (shared with the parent for failure-tail metadata) and handed to
+    ``log_line`` (the caller's destination — a logger, ``print``, Dagster's
+    ``context.log.info``, …). This is the **sole reader** of ``proc.stderr``.
+
+    When ``mirror_to_stderr`` is set, each line is additionally written to this
+    process's ``sys.stderr`` so an outer capture that only sees the real fds
+    (e.g. Dagster's compute-log capture) preserves rocky's tracing output.
+    Mirroring is best-effort: ``OSError`` / ``ValueError`` from a closed fd are
+    swallowed so the reader thread never dies on teardown.
+    """
+    if stderr is None:
+        return
+    try:
+        for raw in stderr:
+            line = raw.rstrip("\n")
+            if not line:
+                continue
+            sink.append(line)
+            log_line(line)
+            if mirror_to_stderr:
+                with contextlib.suppress(OSError, ValueError):
+                    print(line, file=sys.stderr, flush=True)
+    except (OSError, ValueError) as exc:
+        _log.warning("rocky stderr reader terminated: %s", exc)
+
+
+def accumulate_stdout(stdout: IO[str] | None, sink: list[str]) -> None:
+    """Reader-thread body: accumulate every rocky stdout line into ``sink``.
+
+    Sole reader of ``proc.stdout``. Appends every line **including blank ones**
+    so the final concatenation reconstructs rocky's exact JSON output
+    byte-for-byte. On an unexpected read error logs at WARN and exits cleanly;
+    the parent surfaces a parse failure if the payload was truncated.
+    """
+    if stdout is None:
+        return
+    try:
+        for line in stdout:
+            sink.append(line)
+    except (OSError, ValueError) as exc:
+        _log.warning("rocky stdout accumulator terminated: %s", exc)
+
+
+def kill_process_group(proc: subprocess.Popen[str]) -> None:
+    """Terminate ``proc`` and any children via its POSIX process group.
+
+    Called from the watchdog when the wall-clock timeout elapses. On POSIX,
+    ``os.killpg(os.getpgid(pid), SIGKILL)`` reaps children too (requires the
+    Popen was launched with ``start_new_session=True``). On Windows, falls back
+    to ``proc.kill()``. ``ProcessLookupError`` / ``OSError`` are swallowed: the
+    process may have exited between ``wait()`` returning and the kill call.
+    """
+    try:
+        if os.name == "nt":
+            proc.kill()
+        else:
+            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+    except (ProcessLookupError, OSError):
+        # Process already reaped, pgid lookup raced with exit, or the kernel
+        # refused — nothing useful to do from the watchdog. Swallow and move on.
+        pass