sparkforensics-operator 0.2.0__tar.gz → 0.3.0__tar.gz

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.3.0}/AGENTS.md

@@ -9,8 +9,8 @@ Python 3.9+, apache-airflow 2.6+ or 3.0+, pytest. Runtime dependency: Node.js
 18+ + the `sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`)
 wherever the analysis runs: the worker for `SubprocessAnalyzeHook`, the SSH
 host for `SSHAnalyzeHook`. Log sources resolve an `EventLogRef` (`log_ref.py`)
-that backends consume; see `docs/architecture.md`. `deferrable=True` runs the
-SSH analysis as the SSH provider's detached remote job (architecture.md's
+that backends consume; see `ARCHITECTURE.md`. `deferrable=True` runs the
+SSH analysis as the SSH provider's detached remote job (`ARCHITECTURE.md`'s
 "Deferrable execution").
 
 ## Commands
@@ -44,9 +44,10 @@ of going public. The original multi-PR history (42 commits, 4 PRs) lives in
 indefinitely). `git blame`/`git log -p` on any line predating that date stop
 at the squash commit; check the archive repo instead.
 
-## Design docs
-- `docs/architecture.md`, `docs/api-reference.md`, `docs/runbook.md`,
-  `docs/glossary.md`, see below.
+## Docs
+- User docs: `docs/index.md` (getting started, configuration,
+  troubleshooting). Contributor docs: `CONTRIBUTING.md` (dev, tox matrix,
+  release), `ARCHITECTURE.md` (design notes).
 
 ## Maintaining this file
 

sparkforensics_operator-0.2.0/docs/architecture.md → sparkforensics_operator-0.3.0/ARCHITECTURE.md

@@ -30,8 +30,22 @@ Each backend declares the reference kinds it can read from where it runs
 (`AnalyzeHook.supported_log_refs`); `AnalyzeHook.analyze()` rejects any
 other kind with an error naming what it reads, before running anything, so
 a mismatched pairing (say, `SSHAnalyzeHook` with a log the worker
-downloaded) fails fast instead of analyzing the wrong file. Both trigger shapes (the standalone `SparkForensicsOperator` and the `spark_forensics_callback`
-`on_success_callback` factory) converge on one function,
+downloaded) fails fast instead of analyzing the wrong file.
+
+A run can also be compared against an earlier one: `baseline_log_source`
+is a second `LogSourceHook`, located and cleaned up next to `log_source`,
+and the backend passes its reference to the CLI as `--baseline`. The CLI
+reads a baseline only from a path, never from a History Server, so each
+backend declares the kinds it can pass that way
+(`AnalyzeHook.supported_baseline_refs`: `LocalEventLog` on the worker,
+`RemoteEventLog` on the SSH host) and `analyze()` or the deferrable
+submit step rejects any other before running anything. The comparison
+itself, and the budgets on it, are the CLI's. Which run is the baseline
+is the DAG author's choice, made through that log source: the operator
+keeps no state across DAG runs.
+
+Both trigger shapes (the standalone `SparkForensicsOperator` and the
+`spark_forensics_callback` `on_success_callback` factory) converge on one function,
 `operator.run_spark_forensics(context, ...)`, so there is exactly one
 execute-analyze-persist-notify-threshold code path to maintain. Notify
 always runs before the threshold check's fail/warn/ignore branching, so a
@@ -39,19 +53,24 @@ breach that raises still gets a notification out first.
 
 ## Components
 
-- `_compat.py`, the single place this package imports Airflow's public
-  API from. `BaseOperator`/`BaseOperatorLink` live in `airflow.sdk` on
+- `_compat.py`, where this package imports Airflow's core API from (the
+  one exception is `links.py`'s lazy `XCom` import on the Airflow 2
+  branch; provider hooks are imported where they are used). `BaseOperator`/`BaseOperatorLink` live in `airflow.sdk` on
   3.x and `airflow.models.*` on 2.x. The Task SDK also re-homed `BaseHook`
   (3.1), and the exceptions, `TaskDeferred` and `conf` (3.2); their old
   paths still work there, as the same objects, behind deprecation shims.
   `_compat.py` tries the SDK path first and falls back to the old one, and
   `tests/test_compat.py` imports every module of the package in a fresh
-  interpreter and fails on any deprecation warning they emit. The repo has
+  interpreter and fails if any import fails or emits a
+  `DeprecationWarning`, `PendingDeprecationWarning` or `FutureWarning`
+  from inside the package. The repo has
   no ruff configuration, so there are no ruff `AIR3` rules to enable; that
   test covers the same ground.
 - `hooks/_templating.py`, `TemplatedHookMixin`, the base of every hook:
-  an empty `template_fields` and a `__repr__` that shows them. See
-  "Templated hook arguments" below.
+  an empty `template_fields` and a `__repr__` that shows them. Also
+  `task_renderer()` and `render_with_task_env()`, which render a value
+  with a task's Jinja environment for the callback and the tunnel hook.
+  See "Templated hook arguments" below.
 - `log_ref.py`, the `EventLogRef` kinds (`LocalEventLog`,
   `RemoteEventLog`, `HistoryServerApp`): frozen dataclasses saying where a
   log is, the only thing a log source and a backend share.
@@ -68,8 +87,9 @@ breach that raises still gets a notification out first.
   cleaning up after itself in that case; currently `HistoryServerLogSourceHook`
   and `SFTPLogSourceHook` do this, via their own internal try/except
   around the download/transfer that removes their owned temp dir on any
-  exception. See `docs/runbook.md`'s "Disk fills up" entry for which
-  hook/config combinations actually clean up.
+  exception. See the "Disk fills up on a worker" entry in
+  `docs/troubleshooting.md` for which hook/config combinations actually
+  clean up.
 - `hooks/log_source/{sftp,tunnel}.py`, `SFTPLogSourceHook` and
   `SSHTunneledLogSourceHook`, for on-prem log sources reachable only over
   the SSH connection an `SSHOperator` already has configured (no cloud
@@ -89,22 +109,23 @@ breach that raises still gets a notification out first.
   `HistoryServerApp` and do no I/O. Per-run values come from Jinja, the
   same way as in the fetching hooks (see "Templated hook arguments").
   `hooks/log_source/_path_template.py` checks a rendered path (rendered,
-  non-empty, no `..` segment) and `_history_server_args.py` checks a
+  non-empty, no `..` segment, no old `{run_id}`-style placeholder) and `_history_server_args.py` checks a
   rendered app id and attempt id; `_rolling_log.py` holds the
   rolling-log-entry-matching logic shared between `history_server.py`
   and `sftp.py`. `_dest_root.py` holds the `dest_root/<basename>` staging
   convention shared by `filesystem.py`/`sftp.py`, and the owned-temp-dir
   bookkeeping (create if `dest_dir` is unset, `rmtree` on cleanup/error if
   owned) shared by `sftp.py`/`history_server.py`.
-- `hooks/analyze/base.py`, `AnalyzeHook`: `analyze(log_ref, thresholds)`
-  checks the reference against `supported_log_refs` (`check_log_ref()`),
-  then calls the backend's `_analyze()`. `DeferrableAnalyzeHook` adds the
+- `hooks/analyze/base.py`, `AnalyzeHook`: `analyze(log_ref, thresholds,
+  baseline_ref=None)` checks the reference against `supported_log_refs`
+  (`check_log_ref()`) and any baseline against `supported_baseline_refs`
+  (`check_baseline_ref()`), then calls the backend's `_analyze()`. `DeferrableAnalyzeHook` adds the
   detached-job interface the deferrable mode drives (`submit`,
   `trigger_for`, `defer_timeout`, `collect`, `abandon`); see "Deferrable
   execution" below.
 - `hooks/analyze/_cli.py`, the CLI contract every backend shares: argument
   building for each reference kind (a positional path, or
-  `--shs-base-url`/`--app-id`/`--attempt-id`), threshold flags, exit-code
+  `--shs-base-url`/`--app-id`/`--attempt-id`), `--baseline`, threshold flags, exit-code
   handling (0/1/3 usable, 2 and anything else raise), and building the
   `Report` from the JSON report and stderr threshold lines. Backends only
   differ in how they run the command and collect its output.
@@ -139,6 +160,8 @@ breach that raises still gets a notification out first.
   threshold-evaluation *logic* lives here: sparkforensics's own CLI decides
   pass/violation/inconclusive, this module only parses what it printed, so
   it can never drift from what the CLI actually enforces.
+- `exceptions.py`, `ThresholdBreached`, an `AirflowFailException` so a
+  breach fails the task without consuming its retries.
 - `sinks.py`, persists the report JSON (local path, `file://`, `s3://`).
 - `notify.py`, the `Notifier` base class: subclasses implement `_send()`
   for whatever messaging/paging system they target, and `notify()` makes
@@ -176,6 +199,7 @@ breach that raises still gets a notification out first.
   deferred run's `execute_complete()` calls.
 - `callback.py`, `spark_forensics_callback(**kwargs)`, an
   `on_success_callback` factory sharing the same core.
+- `__init__.py`, the public exports and `__version__`.
 
 ## Deferrable execution
 
@@ -185,7 +209,7 @@ worker slot while the CLI runs on the SSH host:
 ```
 worker:     execute()  locate log_ref -> backend.submit() -> defer(trigger_for(job), kwargs)
 triggerer:  SSHRemoteJobTrigger polls exit_code, streams stdout.log
-worker:     execute_complete(event, job, log_ref, report_dest, thresholds)
+worker:     execute_complete(event, job, log_ref, report_dest, thresholds, baseline_log_ref)
               -> backend.collect()  read stderr + report.json, remove the job dir
               -> handle_report()    same persist/notify/threshold code as a sync run
 ```
@@ -207,8 +231,10 @@ from the DAG file, so nothing set on the instance before `defer()`
 survives. Everything `execute_complete()` needs travels in the resume
 kwargs, as JSON-native values that Airflow 2's `BaseSerialization` and
 Airflow 3's serde both round-trip: the job dict (connection id, CLI
-binary, `timeout`, every remote path), the log reference as a dict, the
-rendered `report_dest` and the thresholds the job was submitted with. The
+binary, `timeout`, `poll_interval`, the job id, the submission time and
+every remote path), the log reference as a dict (and the baseline's, when
+there is one), the rendered `report_dest` and the thresholds the job was
+submitted with. The
 job dict and the log reference are built from the rendered hooks, so
 templated hook arguments cross the deferral as the values the job ran
 with. The notifier, `on_threshold_breach` and `report_url_template` come
@@ -243,9 +269,10 @@ session ids in `/proc/<pid>/stat`. A pid is only signalled while its
 command line still names its job directory, so a pid reused after a
 reboot is left alone.
 
-Failure paths. A trigger error event makes `collect()` stop and remove
-the job before raising. A deferral that times out (the backend's
-`defer_timeout`, `timeout` plus 120 seconds, or the task's
+Failure paths. A trigger error event without an exit code makes
+`collect()` stop and remove the job before raising. A deferral that times
+out (the backend's `defer_timeout`, `timeout` plus 120 seconds counted
+from submission, or the task's
 `execution_timeout`) resumes with `next_method="__fail__"`, which never
 reaches `execute_complete()`; `resume_execution()` is overridden to call
 `backend.abandon(context, job)` first, with the job from XCom. Airflow 2
@@ -256,7 +283,8 @@ anything on the resumed operator, then calls `on_kill()`, which is why
 between `submit()` and `defer()` (building the trigger, the timeout, the
 deferral itself) abandons the job before the error propagates, and a
 task timeout or kill there reaches `on_kill()`, which does the same. The
-runbook's "Deferred runs" section lists the behaviour for each case.
+"Deferred runs" section of `docs/troubleshooting.md` lists the behaviour
+for each case.
 
 Killing a task. `SparkForensicsOperator.on_kill()` uses the context of the
 `execute()` or `execute_complete()` running in the same process, or,
@@ -306,7 +334,7 @@ synchronous.
 Hook arguments are Jinja templates, rendered by Airflow's own nested
 template-field mechanism, the same one used for any object assigned to a
 templated operator field. `SparkForensicsOperator.template_fields` is
-`("report_dest", "log_source", "backend")`, and each hook class declares
+`("report_dest", "log_source", "backend", "baseline_log_source")`, and each hook class declares
 the attributes it templates in its own `template_fields`; Airflow renders
 those in place, on Airflow 2 and 3 alike. There is one mechanism, and no
 hook formats a string itself. The earlier `{run_id}`-style `str.format`
@@ -316,7 +344,7 @@ one is rejected with an error naming the Jinja replacement.
 Three details keep this safe.
 
 - `render_template_fields()` gives the task its own shallow copy of
-  `log_source` and `backend` before rendering, because Airflow renders in
+  `log_source`, `baseline_log_source` and `backend` before rendering, because Airflow renders in
   place and the same hook instance is often shared by several tasks or by
   every mapped instance.
 - The log-source method is `locate()`, not `resolve()`. Airflow 3's
@@ -335,7 +363,8 @@ Three details keep this safe.
 when Airflow renders the task. `spark_forensics_callback` runs outside
 any template rendering, so it renders `report_dest`, `log_source` and
 `backend` itself, on copies, with the upstream task's Jinja environment.
-Both render through a copy of the task with an empty `template_ext`, so a
+Both render through the task, or a copy of it with an empty
+`template_ext` when it has one, so a
 value ending in `.json` is rendered as a string, not looked up as a
 template file on an EMR or Spark-on-Kubernetes upstream.
 
@@ -345,7 +374,12 @@ Before notifying and before any threshold breach raises,
 `handle_report()` pushes a summary under the `sparkforensics_summary`
 XCom key: the schema version, the destination, the report URL, the
 impact-band counts, whether the run violated a threshold, the breached
-and inconclusive threshold names, and the CLI exit code. Downstream tasks
+and inconclusive threshold names, and the CLI exit code. A run compared
+against a baseline adds a `comparison` entry: the direction the CLI
+computed for one metric (`regression_metric`, wall-clock by default) as
+the verdict, the comparison's confidence and reason, and that metric's
+values and change in percent, taken from the CLI's comparison section.
+The full section is persisted with the report. Downstream tasks
 can branch on it without reading the report. It is built from the same
 `Report` on both the synchronous and the deferred path, and
 `return_value` stays the destination.
@@ -358,7 +392,7 @@ presigns anything: the URL points at a place the viewer's own
 credentials open. `ReportLink` shows that URL when it is set and the raw
 destination when it is not.
 
-## Deferred
+## Not implemented
 
 **http/MCP AnalyzeHook backend.** The spec originally described an "http"
 backend calling a deployed sparkforensics `server/` instance. `server/`

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.3.0}/CHANGELOG.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-09-28
+
+### Added
+
+- Run comparison: `baseline_log_source` on `SparkForensicsOperator` and
+  `spark_forensics_callback` resolves an earlier run's event log, the same
+  way `log_source` resolves the run under analysis, and passes it to
+  `sparkforensics-analyze --baseline`. It must resolve to a file path
+  readable where the analysis runs: a `LocalEventLog` for
+  `SubprocessAnalyzeHook`, a `RemoteEventLog` on the same SSH connection
+  for `SSHAnalyzeHook`. A History Server application is rejected, since
+  the CLI reads a baseline only from a path. A baseline that is, contains
+  or sits inside the log under analysis fails the task before analysis.
+  Works with `deferrable=True`.
+- Comparison thresholds `max_regression_pct`, `regression_metric` and
+  `fail_on_introduced`, mirroring the CLI's flags and its rules: all need
+  `baseline_log_source`, and `regression_metric` needs
+  `max_regression_pct`. A breach raises `ThresholdBreached` like any other
+  threshold.
+- The `sparkforensics_summary` XCom gains a `comparison` key when a
+  baseline ran: the checked metric's verdict, the comparison's confidence
+  and reason, and the metric's baseline and candidate values and change in
+  percent. The persisted report carries the CLI's full comparison section.
+
+### Changed
+
+- The operator reads report schema 5, where a finding's text figure is in
+  `valueText` and its `value` is always a number or `null`. A report with
+  an older `schemaVersion` fails the task.
+- The persisted report and `Report` carry the CLI's `verdict` and
+  `notRunChecks` sections (`Report.verdict`, `Report.not_run_checks`).
+  Before this, the operator persisted only the sections it listed, so
+  these two would have been dropped.
+
 ## [0.2.0] - 2026-09-25
 
 ### Added
@@ -154,7 +188,8 @@ Initial release.
 - Generic `Notifier` interface for threshold-breach alerting.
 - PyPI trusted-publishing GitHub Actions workflow (OIDC, no stored token).
 
-[Unreleased]: https://github.com/shuffle-works/sparkforensics-operator/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/shuffle-works/sparkforensics-operator/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.3.0
 [0.2.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.2.0
 [0.1.1]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.1.1
 [0.1.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.1.0

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.3.0}/CLAUDE.md

@@ -9,7 +9,7 @@ Python 3.9+, apache-airflow 2.6+ or 3.0+, pytest. Runtime dependency: Node.js
 18+ + the `sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`)
 wherever the analysis runs: the worker for `SubprocessAnalyzeHook`, the SSH
 host for `SSHAnalyzeHook`. Log sources resolve an `EventLogRef` (`log_ref.py`)
-that backends consume; see `docs/architecture.md`.
+that backends consume; see `ARCHITECTURE.md`.
 
 ## Commands
 - `pip install -e ".[test,s3,ssh]"`, install for local dev.
@@ -18,6 +18,6 @@ that backends consume; see `docs/architecture.md`.
 - `tox -e py311-airflow2` / `tox -e py311-airflow3`, run the suite against a
   specific Airflow major version.
 
-## Design docs
-- `docs/architecture.md`, `docs/api-reference.md`, `docs/runbook.md`,
-  `docs/glossary.md`, see below.
+## Docs
+- User docs: `docs/index.md` (getting started, configuration,
+  troubleshooting). Contributor docs: `CONTRIBUTING.md`, `ARCHITECTURE.md`.

sparkforensics_operator-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,77 @@
+# Contributing
+
+Design notes and the component map are in [ARCHITECTURE.md](ARCHITECTURE.md).
+User documentation lives under [docs/](docs/index.md).
+
+## Development install
+
+```bash
+pip install -e ".[test,s3,ssh]"
+```
+
+## Running the tests
+
+```bash
+pytest
+```
+
+The suite needs no live Spark, Airflow or Node.js. The one real process is
+a shell: the deferred-mode tests run the SSH provider's real shell wrapper against a local `sh` standing in
+for the SSH host (`tests/hooks/analyze/_local_ssh.py`), so they need
+`bash`, `setsid` and coreutils `timeout` on the machine running the tests.
+The kill and sweep tests also run with procps hidden
+(`LocalHost.hide_procps()`), as on slim images.
+
+## The tox Airflow matrix
+
+`tox.ini` runs the suite against each supported Airflow major version:
+
+```bash
+tox -e py311-airflow2
+tox -e py311-airflow3
+```
+
+The full env list is `py{39,310,311,312}-airflow2`,
+`py{310,311,312}-airflow3`, and three floor envs:
+
+- `py39-airflow2min` pins Airflow 2.6.3 without the `ssh` extra, and
+  omits the SSH-only modules from its coverage (`coverage-no-ssh.ini`).
+- `py39-airflow2sshmin` runs the `ssh` extra's floor
+  (`apache-airflow-providers-ssh` 3.7.1) on Airflow 2.6.3. The deferrable
+  tests skip there, since that mode needs a newer provider.
+- `py311-airflow2deferrablemin` pins the deferrable mode's floors
+  (`constraints-deferrable-floor.txt`).
+
+The two Airflow 2.6.3 envs install through
+`constraints-airflow-2.6-floor.txt` so the extras step can't upgrade
+Airflow past the floor; `tox.ini` explains why.
+
+CI (`.github/workflows/ci.yml`) runs a subset of these envs on every push
+to `main` and every pull request. Each env writes coverage to
+`coverage-<envname>.lcov`; CI uploads one file per env to Coveralls and
+merges them in a `finish` job. `usedevelop = true` in `tox.ini` keeps the
+lcov `SF:` paths pointing at `src/sparkforensics_operator/...` instead of
+a `.tox` virtualenv, which Coveralls needs to attribute lines.
+
+## Releasing to PyPI
+
+`.github/workflows/publish.yml` builds the sdist and wheel and publishes
+them on every GitHub Release (`release: published`). It uses PyPI trusted
+publishing (OIDC), so no API token is stored in the repository.
+
+One-time setup before the first release:
+
+- On PyPI, add a trusted publisher for this project: owner
+  `shuffle-works`, repo `sparkforensics-operator`, workflow `publish.yml`,
+  environment `pypi`.
+- In the GitHub repository settings, create an environment named `pypi`
+  (optionally with required reviewers) to match.
+
+To cut a release:
+
+1. Bump `version` in `pyproject.toml` and move the `[Unreleased]` entries
+   in `CHANGELOG.md` under the new version. The Airflow provider metadata
+   reads the version from the installed package, so nothing else needs
+   changing.
+2. Tag the release commit (for example `v0.2.0`).
+3. Publish a GitHub Release from that tag. The workflow does the rest.

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sparkforensics-operator
-Version: 0.2.0
+Version: 0.3.0
 Summary: Airflow operator that runs sparkforensics analysis on a Spark job's event log and acts on the result.
 Project-URL: Homepage, https://github.com/shuffle-works/sparkforensics-operator
 Project-URL: Repository, https://github.com/shuffle-works/sparkforensics-operator
@@ -71,6 +71,14 @@ runs as part of the DAG, right where the Spark task just ran.
   you configure are enforced, and a breach raises `ThresholdBreached`
   without consuming the task's retries, since re-running would just reach
   the same verdict.
+- Compare a run against an earlier one. `baseline_log_source` takes any
+  log source that resolves to an event log path the analysis can read
+  (`path_template="/spark-logs/{{ prev_ds }}/eventlog"`, an XCom pull),
+  and `max_regression_pct` fails the DAG when runtime, or the metric
+  `regression_metric` names, regressed past that budget.
+  `fail_on_introduced` fails it when the run gained new findings. The
+  comparison's verdict, confidence and measured regression land in the
+  summary XCom.
 - Reports persist to a local path, `file://`, or `s3://`, and a clickable
   "SparkForensics report" link shows up on the task in the Airflow UI. Set
   `report_url_template` (an S3 console URL, say) to make it open in a
@@ -85,8 +93,8 @@ runs as part of the DAG, right where the Spark task just ran.
 - An optional `Notifier` you implement (Slack, MS Teams, email, PagerDuty,
   ZenDuty, whatever you use) gets a best-effort pass/fail summary; a
   delivery failure never fails the task itself.
-- Tested against both Airflow 2.6+ and Airflow 3.0+ (CI matrix, Python
-  3.9-3.12). No live Spark, Airflow or Node is needed to run the tests:
+- Tested against both Airflow 2.6+ and Airflow 3 (CI runs Python 3.9,
+  3.11 and 3.12; the tox matrix also covers 3.10). No live Spark, Airflow or Node is needed to run the tests:
   everything but the shell on the stand-in SSH host is mocked.
 
 ## Quick start
@@ -162,16 +170,17 @@ the analysis runs: the CLI runs as a detached job on the SSH host and the
 triggerer polls it, which needs a running triggerer with
 `sparkforensics-operator[ssh]` installed and
 `apache-airflow-providers-ssh>=6.0.1`, so Airflow 2.11 or newer. See
-[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md#deferrable-execution).
+[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md#deferrable-execution).
 
 Whichever host runs the analysis (the worker for `SubprocessAnalyzeHook`,
 the SSH host for `SSHAnalyzeHook`) needs Node.js 18+ and the
-`sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`), so
+`sparkforensics-cli` npm package, version 0.4.0 or newer
+(`npm install -g "sparkforensics-cli@>=0.4.0"`), so
 `sparkforensics-analyze` is resolvable on `PATH`. Add the `ssh` extra for
 the SSH hooks (it works from Airflow 2.6, like the package; only
 `deferrable=True` needs Airflow 2.11+), and the `s3` extra if
 `report_dest` is `s3://...`.
-Full details in [`docs/runbook.md`](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md).
+Full details in [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md).
 
 ## Development
 
@@ -189,11 +198,11 @@ tox -e py311-airflow3
 
 ## Learn more
 
-- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/architecture.md), components and design decisions
-- [API reference](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md), the full operator/hook/threshold
-  surface
-- [Runbook](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md), worker and SSH host prerequisites, troubleshooting
-- [Glossary](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/glossary.md), domain terms
+- [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md), install and host prerequisites, and DAG wiring examples
+- [Configuration](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md), the full operator/hook/threshold surface
+- [Troubleshooting](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/troubleshooting.md), error messages and deferred-run failure modes
+- [Contributing](https://github.com/shuffle-works/sparkforensics-operator/blob/main/CONTRIBUTING.md), development setup and the release procedure
+- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/ARCHITECTURE.md), components and design decisions
 
 ## License
 

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.3.0}/README.md

@@ -36,6 +36,14 @@ runs as part of the DAG, right where the Spark task just ran.
   you configure are enforced, and a breach raises `ThresholdBreached`
   without consuming the task's retries, since re-running would just reach
   the same verdict.
+- Compare a run against an earlier one. `baseline_log_source` takes any
+  log source that resolves to an event log path the analysis can read
+  (`path_template="/spark-logs/{{ prev_ds }}/eventlog"`, an XCom pull),
+  and `max_regression_pct` fails the DAG when runtime, or the metric
+  `regression_metric` names, regressed past that budget.
+  `fail_on_introduced` fails it when the run gained new findings. The
+  comparison's verdict, confidence and measured regression land in the
+  summary XCom.
 - Reports persist to a local path, `file://`, or `s3://`, and a clickable
   "SparkForensics report" link shows up on the task in the Airflow UI. Set
   `report_url_template` (an S3 console URL, say) to make it open in a
@@ -50,8 +58,8 @@ runs as part of the DAG, right where the Spark task just ran.
 - An optional `Notifier` you implement (Slack, MS Teams, email, PagerDuty,
   ZenDuty, whatever you use) gets a best-effort pass/fail summary; a
   delivery failure never fails the task itself.
-- Tested against both Airflow 2.6+ and Airflow 3.0+ (CI matrix, Python
-  3.9-3.12). No live Spark, Airflow or Node is needed to run the tests:
+- Tested against both Airflow 2.6+ and Airflow 3 (CI runs Python 3.9,
+  3.11 and 3.12; the tox matrix also covers 3.10). No live Spark, Airflow or Node is needed to run the tests:
   everything but the shell on the stand-in SSH host is mocked.
 
 ## Quick start
@@ -127,16 +135,17 @@ the analysis runs: the CLI runs as a detached job on the SSH host and the
 triggerer polls it, which needs a running triggerer with
 `sparkforensics-operator[ssh]` installed and
 `apache-airflow-providers-ssh>=6.0.1`, so Airflow 2.11 or newer. See
-[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md#deferrable-execution).
+[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md#deferrable-execution).
 
 Whichever host runs the analysis (the worker for `SubprocessAnalyzeHook`,
 the SSH host for `SSHAnalyzeHook`) needs Node.js 18+ and the
-`sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`), so
+`sparkforensics-cli` npm package, version 0.4.0 or newer
+(`npm install -g "sparkforensics-cli@>=0.4.0"`), so
 `sparkforensics-analyze` is resolvable on `PATH`. Add the `ssh` extra for
 the SSH hooks (it works from Airflow 2.6, like the package; only
 `deferrable=True` needs Airflow 2.11+), and the `s3` extra if
 `report_dest` is `s3://...`.
-Full details in [`docs/runbook.md`](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md).
+Full details in [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md).
 
 ## Development
 
@@ -154,11 +163,11 @@ tox -e py311-airflow3
 
 ## Learn more
 
-- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/architecture.md), components and design decisions
-- [API reference](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md), the full operator/hook/threshold
-  surface
-- [Runbook](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md), worker and SSH host prerequisites, troubleshooting
-- [Glossary](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/glossary.md), domain terms
+- [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md), install and host prerequisites, and DAG wiring examples
+- [Configuration](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md), the full operator/hook/threshold surface
+- [Troubleshooting](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/troubleshooting.md), error messages and deferred-run failure modes
+- [Contributing](https://github.com/shuffle-works/sparkforensics-operator/blob/main/CONTRIBUTING.md), development setup and the release procedure
+- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/ARCHITECTURE.md), components and design decisions
 
 ## License