aqueduct-core 1.0.0__tar.gz

aqueduct_core-1.0.0/.gitignore

@@ -0,0 +1,157 @@
+# --- Spark / Hadoop / Data ---
+.spark-staging/
+*.parquet
+*.avro
+*.csv
+# DuckDB
+*.db
+*.duckdb
+*.duckdb.wal
+**/spark-warehouse/
+artifacts/
+
+# --- Virtual Environments & Secrets ---
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+.pdm-python
+.pdm-build/
+.pixi
+
+# --- Python ---
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+*.so
+
+# --- Distribution / Packaging ---
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+*.manifest
+*.spec
+
+# --- Testing / Coverage ---
+.pytest_cache/
+pytest_*.*/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.nox/
+.nosepy
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+cover/
+.cache
+
+# --- IDEs, OS & AI Assistants ---
+# Visual Studio Code
+.vscode/
+# JetBrains
+.idea/
+# Spyder
+.spyderproject
+.spyproject
+# Rope
+.ropeproject
+# macOS
+.DS_Store
+# Windows
+Thumbs.db
+# Claude / Cursor
+.claude/
+CLAUDE.md
+.cursorignore
+.cursorindexingignore
+.prompts/
+# Vim
+*.swp
+*.swo
+*~
+
+# --- Tools & Frameworks ---
+# PyPI config
+.pypirc
+# Ruff
+.ruff_cache/
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+# Pyre
+.pyre/
+# pytype
+.pytype/
+# Cython
+cython_debug/
+# Jupyter / IPython
+.ipynb_checkpoints
+profile_default/
+ipython_config.py
+# Django / Flask / Scrapy
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+instance/
+.webassets-cache
+.scrapy
+# Celery
+celerybeat-schedule
+celerybeat.pid
+# Sphinx / mkdocs
+docs/_build/
+/site
+# Translations
+*.mo
+*.pot
+# PyBuilder
+.pybuilder/
+target/
+# SageMath
+*.sage.py
+# Abstra
+.abstra/
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# --- Logs & Misc ---
+*.log
+*.bak
+pip-log.txt
+pip-delete-this-directory.txt
+__pypackages__/
+
+# --- Aqueduct Project ---
+# Local testing, examples, and patches
+.aqueduct/
+tmp/
+patches/
+**/data/output/**
+!**/data/input/*

aqueduct_core-1.0.0/CHANGELOG.md

@@ -0,0 +1,210 @@
+# Changelog
+
+All notable, consumer-facing changes to Aqueduct (`aqueduct-core`) are
+recorded here.
+
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
+versioning follows [SemVer](https://semver.org/). The stability contract
+applies from v1.0.0 — during alpha/RC, breaking changes may land in any
+release and are marked **BREAKING**.
+
+## [1.0.0] — 2026-05-18
+
+First stable release. The stability contract (`docs/STABILITY.md`, exit
+codes, frozen public API) is now in force; subsequent breaking changes
+follow SemVer. Consolidates everything previously staged as Unreleased
+and the `1.0.0a2` pre-release (which never shipped separately).
+
+### Added
+- `aqueduct benchmark` accepts a single `.aqscenario.yml` (positional or
+  `--scenarios`), not only a directory.
+- `aqueduct benchmark --provider` / `--base-url` / `--timeout` to override
+  the agent connection per run (precedence: flag > `aqueduct.yml` agent >
+  default). `--timeout 0` = unbounded read (connect still fails fast).
+- `aqueduct init` now writes a `.gitignore` (Spark/Aqueduct runtime
+  artifacts, `.env`, ephemeral `patches/{pending,rejected}/`); existing
+  files, including a user `.gitignore`, are never overwritten.
+- `aqueduct doctor` `--aqtest` / `--aqscenario` pre-flight, combinable
+  with a config/blueprint probe in one pass.
+- `aqueduct benchmark --format json` now includes the generated
+  `patch` (PatchSpec) per result, so a failure can be diagnosed without
+  re-running. Table mode prints a `(N failed — rerun with --format
+  json …)` hint when any scenario fails.
+- Stable exit codes (`aqueduct/exit_codes.py`) and `docs/STABILITY.md`;
+  downstream tooling can branch on `$?`.
+- `aqueduct schema --target {blueprint,config,patch}` emits the JSON
+  Schema for IDE autocomplete / CI gating.
+- `--format json` on `aqueduct runs` and `aqueduct patch list`.
+- Two-tier suppressible warning system with stable `AQ-WARN [rule_id]`
+  IDs and `warnings.suppress` / `--suppress-warning`. New rules:
+  `kafka_checkpoint_stale`, `nondeterministic_fanout`,
+  `count_col_likely_count_star`, `file_format_no_repartition`,
+  `jdbc_missing_partition`, `jar_availability`.
+- Post-patch `explain()` regression check and
+  `agent.block_on_explain_regression` (warn-only by default).
+- `aqueduct patch preview` with lineage + sandbox validation gates;
+  `agent.patch_validation: full_run | sandbox`.
+- Pluggable observability/lineage/depot store backends (Postgres;
+  Redis for depot) with `aqueduct stores info|migrate` and
+  `[postgres]` / `[redis]` extras.
+- Secrets providers `aws` / `gcp` / `azure` / `custom` with
+  `[aws]` / `[gcp]` / `[azure]` extras.
+- Global `--log-format {text,json}` for structured log shipping.
+- `agent.max_heal_attempts_per_hour` spend-cap on self-healing.
+- `aqueduct compile --show {manifest,provenance,inputs,all}`.
+- Ingress `partition_filters`; Egress `maintenance:` (Delta
+  OPTIMIZE/VACUUM); Channel `metrics_boundary`; Channel
+  `materialize: incremental` + `watermark_column`; `Manifest`
+  input fingerprinting.
+- Assert `error_type` plus `agent.guardrails.heal_on_errors` /
+  `never_heal_errors` pre-trigger guards.
+
+### Changed
+- `aqueduct benchmark` scoring is now two-tier: **correctness gates**
+  PASS/FAIL (`patch_is_valid`, `patch_applies`, `expected_patch`), while
+  **diagnosis quality** (`root_cause_contains`, `expected_category`,
+  `max_attempts`, `min_confidence`) is recorded as a diagnosis score and
+  reported (`diag_score`/`soft_failures` in JSON, `d%` + `Diag score` in
+  the table) but **never fails an otherwise-correct fix**.
+- Gallery `aqscenarios` benchmark suite reworked for fidelity: each
+  scenario now has its own blueprint carrying exactly one real defect,
+  `inject_failure.error_message` mirrors authentic Spark output (error
+  class, `SQLSTATE`, suggestion list), and scoring is op-agnostic
+  (outcome + diagnosis, not a hard-coded patch op). Previously 4/5
+  scenarios injected an error unrelated to the shared clean blueprint
+  (unsolvable/ungradable).
+- `.env` is now auto-loaded by **every** command from the directory of
+  the config/blueprint passed (previously only `run`/`doctor`/`validate`).
+  A one-line stderr notice reports what was loaded; `-e KEY=VAL`
+  (docker-style, highest precedence) and `AQ_NO_ENV_FILE=1` added.
+- `aqueduct benchmark` default `--workers` is now `1` (serial); set `>1`
+  to parallelize scenario×model pairs.
+- `aqueduct test` always runs on `local[*]` and ignores
+  `deployment.master_url` (isolated unit tests); `--master` overrides for
+  cluster-runtime-dependent modules.
+- `aqueduct doctor` default view shows only actionable rows; skip/green
+  low-signal rows collapse into one line, `--verbose` expands. The Spark
+  check is a fast bounded reachability probe by default; `--preflight`
+  runs a full unbounded session check. `agent` no longer warns when
+  self-healing is simply unconfigured (opt-in). Convention: ✗ = "will
+  break", ⚠ = "runs but fragile". doctor stays advisory.
+- Init scaffold directories renamed for consistency: `tests/` → `aqtests/`,
+  `benchmarks/` → `aqscenarios/`.
+- Default `agent.model` is now `claude-sonnet-4-6`.
+- Incremental-Channel watermark is computed from materialized Egress
+  output instead of re-scanning the upstream DAG twice.
+- Cluster/cloud deployments with relative store paths now error in
+  `doctor` and warn at run start.
+- Public API frozen to a small `__all__` (`parse`, `ParseError`,
+  `AqueductWarning`, `__version__`); everything else is internal.
+- **BREAKING:** `aqueduct benchmark --output` renamed to `--format`
+  (data-shape selector; consistent with `runs`/`report`/`patch list`).
+  `-o`/`--output` is now reserved exclusively for file destinations
+  (`schema`); `compile --show` keeps its artefact-slice meaning.
+
+### Deprecated
+- _None._
+
+### Removed
+- **BREAKING:** `heal --scenario` removed. `heal` is production-only
+  (heal a real failed `run_id`); scenario evaluation is
+  `aqueduct benchmark <file-or-dir>`.
+- **BREAKING:** `heal --print-prompt-format` removed — folded into
+  `--print-prompt [text|json]` (bare = text, `--print-prompt json` for
+  JSON).
+- **BREAKING:** `warnings.silence_all` config field and `--no-warnings`
+  flag removed. Use `warnings.suppress: ["*"]` or
+  `--suppress-warning '*'`.
+- **BREAKING:** `aqueduct check-config` removed — use `aqueduct validate`
+  (auto-detects blueprint vs engine config by header, accepts multiple
+  files).
+- **BREAKING:** `aqueduct doctor --config` / `--blueprint` removed — pass
+  the file positionally (header-sniffed).
+- **BREAKING:** `agent.llm_timeout` → `agent.timeout`,
+  `agent.llm_max_reprompts` → `agent.max_reprompts` (the self-healing
+  subsystem is uniformly "the agent"; no aliases).
+- **BREAKING:** `probes.block_full_actions_in_prod` →
+  `danger.allow_full_probe_actions` (inverted polarity).
+- `ollama_options` renamed to `provider_options`.
+
+### Fixed
+- Postgres store backend (`stores.*.backend: postgres`) no longer crashes
+  on DuckDB-only DDL/upsert SQL; non-DuckDB store `path` (a DSN) is no
+  longer mis-created as a local directory. (Full multi-backend
+  certification still pending.)
+- `--parallel` Probe race that silently dropped Probe signals
+  (ISSUE-042).
+- Tier-0 tokens (`${VAR:-default}`, `${ctx.*}`) are now resolved in
+  top-level `spark_config` and `macros` (ISSUE-027).
+- `materialize: incremental` blueprints referencing `${ctx._watermark}`
+  no longer fail `validate`/`run`.
+- Duplicate missing-env-var error lines collapsed to one.
+- pyspark `DataFrame.sql_ctx` deprecation warning from the explain gate.
+- Misleading agent "failed after N attempts" log now reports the actual
+  attempt count.
+- Bundled `aqtest.yml.template` rewritten to match the real
+  `aqueduct test` runner schema (it previously documented a
+  non-functional format).
+
+## [1.0.0a1] — 2026-05-12
+
+### Added
+- `danger:` block (`allow_aggressive_patching`,
+  `allow_full_probe_actions`), defaulting `false`, with a startup
+  warning when any is enabled.
+- `approval_mode: ci` with `agent.ci_webhook_url`; webhooks
+  `on_patch_pending` / `on_ci_patch` make `human`/`ci` review viable
+  for teams.
+- `PatchSpec.confidence|category|root_cause`; `confidence < 0.7`
+  auto-escalates to human review regardless of `approval_mode`.
+- `healing_outcomes` table persisting every healing attempt.
+- Per-pipeline store paths (`.aqueduct/obs/{blueprint_id}/`),
+  removing DuckDB write contention across pipelines.
+- Java/Scala UDFs (`lang: java|scala`, `jar:`, `entry:`).
+- `schema_hint` `strict | additive | subset` modes.
+- Git-integrated patch lifecycle: `aqueduct patch
+  commit|discard|log|rollback|list`; surgical `set_module_config_key`
+  op.
+- Provenance layer — smaller/cleaner LLM prompts; `doctor` recurses
+  into Arcades.
+- `aqueduct init` project scaffold (writes `.gitignore` ignoring the
+  ephemeral patch dirs); `aqueduct runs`; `aqueduct report`;
+  `aqueduct lineage`; `aqueduct signal`; `aqueduct heal`;
+  `aqueduct test`.
+- Probe signal types: `value_distribution`, `distinct_count`,
+  `data_freshness`, `partition_stats`.
+- Compile-time SQL macros; Channel `op: join` with broadcast hint.
+
+### Changed
+- `runs.db` + `signals.db` consolidated into a single
+  `observability.db`.
+
+### Removed
+- **BREAKING:** `agent.validate_patch` field removed — `aggressive`
+  mode now always validates in-memory before writing.
+
+## [1.0.0a0] — 2026-04-27
+
+First alpha on PyPI (`aqueduct-core`). Phases 1–9: the core engine.
+
+### Added
+- Declarative blueprint pipeline: Parser → Compiler → Executor with
+  `aqueduct validate | compile | run`.
+- Modules: Ingress, Egress, Channel (SQL), Junction, Funnel, Probe;
+  Spillway error-row routing; Assert data-quality module
+  (`abort|warn|webhook|quarantine|trigger_agent`).
+- Resolution: Tier-0 (`${ENV:-default}`, `${ctx.*}`, env/profile/CLI
+  overrides) and Tier-1 (`@aq.date.*`, `@aq.run.*`, `@aq.depot.*`,
+  `@aq.secret()`); Arcades (reusable sub-blueprints).
+- Depot KV store; Python/Java/Scala UDFs; `RetryPolicy` (exponential/
+  linear/fixed + jitter + deadline); per-module `on_failure`;
+  opt-in checkpoint/resume.
+- LLM self-healing: `approval_mode: disabled|human|auto|aggressive`,
+  deterministic guardrails (`allowed_paths`, `forbidden_ops`), patch
+  staging/rollback; Anthropic + Ollama + OpenAI-compatible providers
+  over plain HTTP (no `anthropic` SDK dependency).
+- Observability: Surveyor (DuckDB) run records / failure contexts;
+  SparkListener per-module metrics; column-level lineage via sqlglot;
+  `aqueduct doctor`.
+- Remote Spark (`local[*]`, `spark://`, `yarn`, `k8s://`); partial DAG
+  (`--from`/`--to`); `--execution-date` for backfills.

aqueduct_core-1.0.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Sadig Akhund
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.