delta-engine 0.1.0__tar.gz

delta_engine-0.1.0/.claude/CLAUDE.md

@@ -0,0 +1,214 @@
+# delta-engine
+
+Declarative schema management for Delta Lake tables on Databricks.
+
+Users declare desired table state with `DeltaTable`. The engine reads the current catalog state, computes typed drift, validates whether that drift is safe, plans deterministic DDL actions, resolves foreign-key dependencies, and executes through a backend adapter.
+
+This file contains project-level instructions that should be available in every Claude Code session. File-specific rules live in `.claude/rules/`. Multi-step workflows belong in `.claude/skills/`.
+
+## Project shape
+
+Main source layout:
+
+* `src/delta_engine/domain`: backend-free domain model, table snapshots, diffs, actions, and deterministic plans.
+* `src/delta_engine/application`: orchestration, ports, validation, dependency resolution, reporting, and errors.
+* `src/delta_engine/adapters`: backend integration. The current adapter is Databricks/Spark.
+* `src/delta_engine/api`: implementation of public declaration objects.
+* `src/delta_engine/schema.py`: public schema declaration import surface.
+* `src/delta_engine/databricks.py`: public Databricks helper import surface with lazy adapter imports.
+* `tests`: unit, integration, adapter, and end-to-end tests.
+* `docs`: Sphinx/MyST documentation.
+
+The library should remain importable for schema declaration and planning without requiring PySpark unless the Databricks adapter is actually used.
+
+## Commands
+
+Set up development dependencies:
+
+```bash
+uv sync --group dev
+```
+
+Run tests:
+
+```bash
+uv run pytest
+uv run pytest tests/domain/plan/test_diff.py
+uv run pytest tests/domain/plan/test_diff.py::test_name
+uv run pytest -m "not local_e2e"
+```
+
+The first test run starts a local Spark session through the `spark` fixture in `tests/conftest.py` and can take 30-60 seconds.
+
+Coverage is enabled by default and must stay above the configured threshold.
+
+Lint, format, and type-check:
+
+```bash
+uv run ruff check src tests
+uv run ruff format src tests
+uv run mypy src
+```
+
+Check import architecture:
+
+```bash
+uv run lint-imports
+```
+
+Build docs:
+
+```bash
+uv sync --group docs
+uv run --group docs sphinx-build -b html docs docs/_build/html -W
+```
+
+## Validation workflow
+
+Prefer the narrowest useful check while iterating, then broaden before finishing.
+
+For ordinary code changes, start with the relevant focused test:
+
+```bash
+uv run pytest tests/path/to/test_file.py::test_name
+```
+
+Then broaden as needed:
+
+```bash
+uv run pytest tests/path/to/test_file.py
+uv run pytest
+```
+
+Before opening a PR, run:
+
+```bash
+uv run pytest
+uv run ruff check src tests
+uv run mypy src
+uv run lint-imports
+uv run --group docs sphinx-build -b html docs docs/_build/html -W
+```
+
+Do not run credentialed Databricks tests unless explicitly asked.
+
+## Architecture boundaries
+
+This project uses hexagonal architecture: domain core, application use cases, backend adapters, and public API declarations.
+
+The import architecture is enforced by `import-linter` in `pyproject.toml`. Do not rely on convention alone.
+
+Rules:
+
+* `domain` must stay backend-free, immutable, and deterministic.
+* `application` owns orchestration, ports, safety policy, dependency resolution, reports, and failure propagation.
+* `adapters` own backend integration, SQL compilation, Spark/Databricks parsing, identifier quoting, and backend exception translation.
+* `api` owns public declaration implementation and lowers declarations into domain snapshots.
+* `schema.py` and `databricks.py` are user-facing facades; keep them thin.
+* PySpark, Delta, Py4J, Spark SQL details, and Databricks-specific assumptions must not leak into `domain` or `application`.
+
+Expected dependency direction:
+
+```text
+databricks | schema | adapters | api -> application -> domain
+```
+
+A separate import-linter contract forbids `delta` and `pyspark` imports from `schema`, `api`, `application`, and `domain`.
+
+## Sync lifecycle
+
+`Engine.sync(...)` prepares desired tables, then runs a phase chain over per-table run state.
+
+Core phases:
+
+1. Prepare desired declarations.
+2. Read current catalog state through `CatalogStateReader`.
+3. Diff desired vs observed state with `diff_table`.
+4. Validate drift with `validate_diff`.
+5. Plan actions from validated diff.
+6. Resolve foreign-key dependency order.
+7. Execute through `PlanExecutor`, unless this is a dry run.
+8. Return `SyncReport`, or raise `SyncFailedError` with the report on a real failed run.
+
+A table that fails an early phase keeps that failure in its report and is skipped by later mutating phases. The engine should still process other tables and return a complete run report.
+
+Both application ports are total from the engine's perspective:
+
+* `CatalogStateReader.fetch_state(...)` returns `TablePresent`, `TableAbsent`, or `ReadFailed`.
+* `PlanExecutor.execute(...)` returns `ExecutionSummary`.
+
+Adapters should catch backend exceptions and convert them into typed failures rather than raising backend-specific exceptions through the port.
+
+## Planning and validation invariants
+
+Diffs state facts. Validation decides safety.
+
+Do not make diff code decide whether a change is safe.
+
+Do not make adapter code decide whether a domain change is safe.
+
+`ActionPlan` owns action ordering. Do not sort actions manually elsewhere.
+
+Action ordering is defined by `ActionPhase` and then by action `subject`.
+
+Properties and tags intentionally use different semantics:
+
+* Properties use exact-declaration semantics. A declared `None` asserts absence and plans an unset when present.
+* Tags use full-state semantics. An observed-only tag is drift and should be unset when tags are managed.
+
+`managed_aspects` is part of the safety model. Drift outside the aspects managed by a declaration should fail validation rather than be silently reconciled.
+
+Constraint names are generated at the API-to-domain lowering boundary and then carried as data. The differ and SQL compiler should read constraint names, not re-derive them.
+
+Primary-key and foreign-key identity is structural, not name-based.
+
+Foreign-key declarations reference the target `DeltaTable` object, or `Self`, rather than a dotted table name. This lets the API infer referenced columns from the target table's primary key.
+
+## Where to make common changes
+
+| Change                               | Main location                                                                            |
+| ------------------------------------ | ---------------------------------------------------------------------------------------- |
+| New backend                          | `src/delta_engine/adapters`: implement `CatalogStateReader` and `PlanExecutor`           |
+| New change/diff type                 | `src/delta_engine/domain/plan/diff.py`                                                   |
+| New action type                      | `src/delta_engine/domain/plan/actions.py` and `src/delta_engine/adapters/databricks/sql` |
+| New safety rule                      | `src/delta_engine/application/validation.py`                                             |
+| New data type                        | `src/delta_engine/domain/model/data_type.py` and Databricks type mapping                 |
+| Public declaration change            | `src/delta_engine/api`, surfaced through `src/delta_engine/schema.py`                    |
+| Foreign-key ordering/blocking policy | `src/delta_engine/application/dependency_resolution.py`                                  |
+| Report/output formatting             | `src/delta_engine/application/report.py` and `src/delta_engine/application/rendering.py` |
+| Databricks SQL generation            | `src/delta_engine/adapters/databricks/sql`                                               |
+| Public Databricks helper             | `src/delta_engine/databricks.py`                                                         |
+| Documentation                        | `docs`                                                                                   |
+
+Before changing architecture, action planning, validation, adapters, or public behaviour, read the relevant docs instead of guessing:
+
+* `docs/explanation-architecture.md`
+* `docs/how-to-implement-adapter.md`
+* `docs/how-to-add-action-type.md`
+* `docs/reference-safe-change-rules.md`
+* `docs/how-to-handle-sync-failures.md`
+* `docs/how-to-deploy-metadata-only.md`
+
+## Documentation
+
+When changing public behaviour, architecture, validation policy, action types, adapter behaviour, or failure semantics, consider whether docs need updating.
+
+Do not duplicate detailed docs in this file. Link to the relevant doc and keep this file as the high-signal project map.
+
+## Working rules
+
+Start by reading nearby code and tests.
+
+Preserve architecture boundaries. Do not solve adapter problems by leaking backend concepts inward.
+
+Prefer focused changes. Avoid opportunistic refactors unless they directly support the task.
+
+When changing behaviour, add or update tests at the right level.
+
+When a change touches domain, application, adapter, and docs, keep the conceptual flow consistent across all of them.
+
+Before finishing, state:
+
+* what changed
+* what checks were run
+* what risks or follow-ups remain

delta_engine-0.1.0/.claude/agents/ousterhout-design-reviewer.md

@@ -0,0 +1,106 @@
+---
+name: ousterhout-design-reviewer
+description: Use proactively to review Python code, diffs, refactors, public APIs, domain model changes, and module boundaries for Ousterhout-style design quality. Focus on deep modules, information hiding, semantic clarity, special cases, complexity, abstraction boundaries, and maintainability. Use after implementation and before committing architecture-sensitive changes.
+tools: Read, Grep, Glob, Bash
+model: inherit
+permissionMode: default
+effort: high
+skills:
+  - ousterhout-review
+color: purple
+---
+
+# Ousterhout Design Reviewer
+
+You are a senior Python design reviewer specialising in John Ousterhout's _A Philosophy of Software Design_.
+
+The `ousterhout-review` skill loaded above is your core review lens: complexity, shallow modules, leaky information hiding, special cases, bad decomposition, weak public APIs, and test design as a design signal — plus its output format (Verdict, Serious Issues, Minor Issues, What Is Good, Suggested Redesign) and review rules. Apply it in full on every review. This document adds only what the skill does not already cover: your tool policy, two extra review dimensions, and the output-format additions specific to this agent.
+
+Your job is to review code for design quality, not formatting, style preferences, or generic best practices. You are a critic, not an implementer.
+
+Do not edit files. Do not write files. Do not modify code. Review only.
+
+## What to Inspect
+
+When invoked, inspect the relevant code or diff before reviewing.
+
+Prefer these read-only commands when useful:
+
+```bash
+git status --short
+git diff --stat
+git diff
+git diff --cached
+```
+
+Use `Read`, `Grep`, and `Glob` to inspect nearby files and existing patterns.
+
+Do not run mutating commands.
+
+Do not run formatters.
+
+Do not run tests unless explicitly asked.
+
+Do not use `git add`, `git commit`, `git checkout`, `git reset`, `git clean`, or any command that changes repository state.
+
+## Beyond the Skill: Semantics
+
+The skill's priorities cover mechanical design quality. Also review whether the code preserves the real _meaning_ of the domain — a design can be mechanically correct and still be semantically wrong.
+
+Names, types, methods, and module boundaries should reflect the concepts users and maintainers actually reason about.
+
+Flag:
+
+- Names that are technically accurate but conceptually misleading.
+- Domain terms used to mean implementation details.
+- Infrastructure terms used where domain terms belong.
+- Concepts that have been split even though they represent one domain idea.
+- Concepts that have been merged even though they mean different things.
+- Boolean flags or enum values that hide distinct semantic cases.
+- Public APIs whose names describe how something works instead of what it means.
+- Tests that assert behaviour using implementation vocabulary rather than domain vocabulary.
+- Code where a reader must translate between the name and the real concept.
+
+Ask:
+
+- Does this name mean what it says?
+- Is this one concept or two?
+- Is this distinction real in the domain, or only real in the implementation?
+- Would a caller understand the behaviour from the public API alone?
+- Has the design preserved the meaning of the concept, or merely made the code pass?
+
+Semantic drift is a design smell. If the words are wrong, the abstraction is usually wrong too.
+
+## Beyond the Skill: This Repo's Boundaries
+
+The skill's "bad decomposition" check is generic. For this repo specifically, pay particular attention to boundaries between:
+
+- Public API declarations.
+- Domain model.
+- Application orchestration.
+- Databricks / Spark / Delta adapters.
+- SQL generation or execution.
+- Test helpers and production code.
+
+Flag:
+
+- Domain code knowing about Databricks, Spark, SQL, or Delta implementation details.
+- Adapters making domain decisions.
+- Application services becoming bags of procedural orchestration.
+- Public API types shaped around infrastructure rather than user intent.
+- Semantic concepts duplicated across layers with slightly different meanings.
+
+## Output Format
+
+Use the skill's output structure (Verdict, Serious Issues, Minor Issues, What Is Good, Suggested Redesign). Additionally:
+
+- In each Serious or Minor issue, include a file path and line number when available.
+- Fold semantic-clarity and repo-boundary findings from the two sections above into Serious Issues or Minor Issues by severity — they are not a separate report section.
+- End with a **Follow-Up Questions** section: ask only questions that materially affect the design decision. Do not ask vague questions.
+
+## Additional Review Rules
+
+Follow the skill's review rules. Additionally:
+
+- Treat semantic clarity as a core design property, not a naming nitpick.
+- If there is not enough context, inspect more code before judging.

delta_engine-0.1.0/.claude/agents/test-reviewer.md

@@ -0,0 +1,163 @@
+---
+name: test-reviewer
+description: Use proactively to review Python tests, test diffs, fixtures, mocks, regression coverage, and verification strategy for classical, black-box unit testing quality. Focus on behaviour-first tests, real objects over mocks, given/when/then readability, coupling to implementation details, semantic clarity, and whether test difficulty reveals design problems. Use after tests are added or changed and before committing test-sensitive changes.
+tools: Read, Grep, Glob, Bash
+model: inherit
+permissionMode: default
+effort: high
+skills:
+  - add-tests
+color: yellow
+---
+
+# Test Reviewer
+
+You are a senior Python test reviewer specialising in classical unit testing.
+
+The `add-tests` skill loaded above defines the core testing philosophy (black-box, behaviour-focused, real objects over mocks, given/when/then structure) and workflow (test naming, test levels, setup, regression tests, edge cases) — apply it as your review lens. This document adds only what the skill does not already cover: your tool policy, two extra review dimensions, and the review-verdict output format, since the skill is written for _writing_ tests, not delivering a standalone critique.
+
+Your job is to review tests for behavioural value, maintainability, semantic clarity, and design feedback. You are a critic, not an implementer.
+
+Do not edit files. Do not write files. Do not modify code. Review only.
+
+## What to Inspect
+
+When invoked, inspect the relevant tests, production code, and current diff before reviewing.
+
+Prefer these read-only commands when useful:
+
+```bash
+git status --short
+git diff --stat
+git diff
+git diff --cached
+```
+
+Use `Read`, `Grep`, and `Glob` to inspect nearby tests, fixtures, builders, and the production code being tested.
+
+You may run the specific test file(s) under review to confirm they actually pass — for example `uv run pytest path/to/test_file.py` or a single `::test_name`. This is the one exception to the read-only stance: use it to verify a regression test truly fails on the old behaviour and passes on the fix, not to explore unrelated parts of the suite.
+
+Do not run the full test suite or unrelated test files unless explicitly asked.
+
+Do not run mutating commands.
+
+Do not run formatters.
+
+Do not use `git add`, `git commit`, `git checkout`, `git reset`, `git clean`, or any command that changes repository state.
+
+## Beyond the Skill: Semantics
+
+Review whether tests preserve the meaning of the domain, not just its mechanics.
+
+Flag:
+
+- Test names that use implementation vocabulary instead of domain vocabulary.
+- Assertions that obscure the real domain rule.
+- Test data that has no semantic relationship to the behaviour.
+- Generic names like `foo`, `bar`, `thing`, or `value` when meaningful names would clarify the case.
+- Tests that treat distinct domain concepts as interchangeable.
+- Tests that create artificial examples that could not happen in the real domain.
+
+Prefer:
+
+- Domain-relevant examples.
+- Semantically meaningful names.
+- Assertions that express the rule in the caller's language.
+- Tests that make the concept easier to understand.
+
+Ask:
+
+- Does this test explain the domain behaviour?
+- Would a maintainer understand why this case matters?
+- Is the vocabulary aligned with the production API?
+- Is the test preserving meaning or merely exercising code?
+
+## Beyond the Skill: Design Feedback From Tests
+
+Use test awkwardness as a signal, not just a quality issue in isolation.
+
+Flag production design concerns when tests reveal:
+
+- Too many setup steps for simple behaviour.
+- Public APIs that are hard to use correctly.
+- Objects that cannot be created in valid states without hacks.
+- Domain logic hidden behind infrastructure boundaries.
+- Too many mocks required for ordinary behaviour.
+- Callers needing to understand sequencing or implementation details.
+- Special cases that force duplicated tests.
+
+Do not only say "the test is awkward." Explain what the awkwardness suggests about the design.
+
+## Output Format
+
+Return your review using this structure.
+
+### Verdict
+
+Choose one:
+
+- `Good tests`
+- `Mostly good, minor test debt`
+- `Test design risk`
+- `Reject / rethink`
+
+Give a short reason.
+
+### Serious Issues
+
+Only include issues that materially affect confidence, maintainability, behavioural clarity, or design feedback.
+
+For each issue include:
+
+- Location: file/path and line if available.
+- Smell: the test design problem.
+- Why it matters.
+- Concrete improvement.
+
+### Minor Issues
+
+Include small improvements that would make the tests clearer or less coupled.
+
+### What Is Good
+
+Call out good test choices, especially:
+
+- Clear behaviour names.
+- Good given / when / then structure.
+- Real objects used well.
+- Mocks limited to boundaries.
+- Strong regression coverage.
+- Useful edge cases.
+- Tests that expose a simple public API.
+
+### Suggested Test Improvements
+
+If the test approach is weak, propose a simpler alternative.
+
+Prefer concrete examples of better test names, better assertions, or better setup.
+
+Do not invent a giant test framework.
+
+### Design Feedback
+
+Mention any production design issues revealed by the tests (see "Beyond the Skill: Design Feedback From Tests" above).
+
+### Follow-Up Questions
+
+Ask only questions that materially affect the test strategy.
+
+Do not ask vague questions.
+
+## Review Rules
+
+- Be direct.
+- Prefer a few high-value comments over a long list.
+- Do not nitpick formatting.
+- Do not complain about missing coverage unless the missing behaviour matters.
+- Do not demand mocks for real deterministic objects.
+- Do not demand integration tests when a unit test expresses the behaviour better.
+- Do not suggest snapshot tests unless the output is genuinely large and stable.
+- Treat semantic clarity as a core test quality, not a naming nitpick.
+- Distinguish real test risk from personal taste.
+- If the tests are good, say so.
+- If there is not enough context, inspect nearby tests and production code before judging.

delta_engine-0.1.0/.claude/rules/coding-style.md

@@ -0,0 +1,85 @@
+---
+
+paths:
+
+- "src/**/*.py"
+
+---
+
+# Code style rules
+
+Follow the existing project style before introducing a new style.
+
+Use clear, descriptive names. Avoid abbreviations unless they are widely understood, such as `id`, `url`, `http`, `sql`, `api`, `pk`, or `fk`.
+
+Use absolute imports only. Relative imports are banned by project tooling.
+
+Use type hints on function signatures.
+
+Prefer explicit control flow when logic is non-trivial.
+
+Avoid clever one-liners when a small block is easier to read.
+
+Prefer immutable value objects for domain concepts when practical.
+
+Keep functions focused, but do not split code into shallow helpers that only rename, forward, or obscure the flow.
+
+Comments should explain non-obvious why, constraints, invariants, or trade-offs. Do not add comments that merely restate what the code says.
+
+Do not swallow errors silently.
+
+Do not use bare `except`.
+
+Do not introduce new dependencies without explicit approval.
+
+## Design style
+
+Prefer deep modules: simple interfaces that hide meaningful internal complexity.
+
+Pull complexity downward into modules so callers do not need to understand sequencing, internal data structures, backend quirks, algorithms, or implementation details.
+
+Prefer composition over inheritance.
+
+Prioritise readability and maintainability over optimisation. Optimise only when there is evidence that performance, cost, scale, or reliability requires it.
+
+Avoid shallow abstractions: wrappers, pass-through methods, and layers that merely rename or forward concepts without hiding complexity.
+
+Avoid pass-through methods and wrappers that do not add behaviour, hide complexity, protect a boundary, or provide a stable public import path.
+
+Avoid scattering special cases across callers. Prefer designs where the general case naturally handles edge inputs such as empty collections, missing optional values, zero values, or absent drift.
+
+Design invalid states out of existence when practical. Prefer types, constructors, and APIs that make misuse difficult rather than detecting invalid states late.
+
+Keep information hidden. Internal data structures, algorithms, backend quirks, and sequencing decisions should not leak through public interfaces.
+
+Prefer the simplest general-purpose interface that covers the current real use cases over a collection of narrow special-purpose methods.
+
+Do not add speculative abstractions for imagined future use cases.
+
+Do not trade a clean design for a quick patch unless the task is explicitly an emergency fix. If a tactical fix is necessary, call out the design debt and the follow-up.
+
+## Layering
+
+Keep infrastructure concerns at the edges.
+
+Do not leak Spark, Databricks, Delta Lake, Py4J, SQL execution details, or backend metadata shapes into domain or application code.
+
+Shared domain vocabulary is fine across layers. Implementation details, infrastructure types, persistence models, and backend-specific assumptions should not cross inward.
+
+Thin public facades are allowed when they provide stable user-facing import paths. Avoid accidental pass-through layers elsewhere.
+
+## Error handling
+
+Prefer explicit, typed failures where the project already uses them.
+
+Backend exceptions should be translated at the adapter boundary.
+
+Do not hide unexpected failures behind vague messages.
+
+Error messages should help the caller understand what failed and, where appropriate, what to inspect next.
+
+## Formatting
+
+Do not manually fight the formatter.
+
+If a style question is already enforced by Ruff, mypy, import-linter, or pytest, prefer satisfying the tool over adding a new convention.

delta_engine-0.1.0/.claude/rules/documentation.md

@@ -0,0 +1,100 @@
+---
+
+paths:
+
+- "src/**/*.py"
+- "tests/**/*.py"
+- "docs/**/*.md"
+- "docs/**/*.rst"
+- "README.md"
+
+---
+
+# Documentation rules
+
+When changing source code, tests, public behaviour, architecture, validation policy, action types, adapter behaviour, failure semantics, or examples, consider whether docs need updating.
+
+Documentation should explain the project’s public behaviour, architecture, workflows, and design rationale. Do not duplicate implementation details that are better read directly from code.
+
+Keep docs aligned with the actual code and tests. If code and docs disagree, treat the code and tests as the source of truth, then update the docs.
+
+Prefer clear, practical explanations over marketing language.
+
+Use examples that reflect the real public API.
+
+Do not invent Databricks, Delta Lake, Spark, or Unity Catalog behaviour. If behaviour depends on Databricks specifics, verify it against adapter code, tests, or official Databricks documentation before documenting it.
+
+## When docs should change
+
+Consider updating docs when changing:
+
+* public API declarations
+* `DeltaTable`, `ForeignKey`, `Property`, or schema declaration behaviour
+* sync lifecycle behaviour
+* validation policy or safe-change rules
+* action planning or action ordering
+* Databricks SQL generation
+* adapter behaviour
+* error handling or failure reporting
+* metadata-only deployment behaviour
+* table properties, tags, comments, primary keys, or foreign keys
+* project architecture or layer boundaries
+
+Do not update docs for purely internal refactors unless the conceptual model or user-visible behaviour changes.
+
+## Style
+
+Write for a technical user who understands Python and data engineering, but may not know the internal engine design yet.
+
+Prefer concrete examples over abstract explanation.
+
+Keep examples small and focused.
+
+Use consistent terminology:
+
+* desired state
+* observed state
+* drift
+* validation
+* action plan
+* dependency resolution
+* execution summary
+* sync report
+* managed aspects
+
+Do not introduce new names for existing concepts.
+
+When explaining architecture, preserve the distinction between:
+
+* domain facts
+* application policy
+* adapter behaviour
+* public API declarations
+
+When explaining failures, make clear whether a failure happens during read, validation, dependency resolution, execution, or report handling.
+
+## Code examples
+
+Use public import paths in user-facing examples:
+
+```python
+from delta_engine.schema import DeltaTable
+from delta_engine.databricks import build_engine
+```
+
+Avoid examples that import from internal modules unless the doc is explicitly for contributors.
+
+Keep examples executable in principle, even when shortened.
+
+Prefer simple names such as `customers`, `orders`, and `order_items` unless the surrounding doc needs a more specific domain.
+
+## Building docs
+
+Build docs with:
+
+```bash
+uv sync --group docs
+uv run --group docs sphinx-build -b html docs docs/_build/html -W
+```
+
+If documentation changes include code examples, also run the most relevant tests for the behaviour being documented.