sparkforensics-operator 0.2.0__tar.gz → 0.4.0__tar.gz

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.4.0}/.github/workflows/ci.yml

@@ -5,6 +5,10 @@ on:
     branches: [main]
   pull_request:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   test:
     runs-on: ubuntu-latest
@@ -40,6 +44,32 @@ jobs:
           path-to-lcov: coverage-${{ matrix.tox-env }}.lcov
           parallel: true
 
+  # The rest of the suite simulates the CLI. This job runs the real one on
+  # public corpus logs, so a change in its exit codes or flag rules fails
+  # here. SPARKFORENSICS_REQUIRE_CLI turns a missing CLI or an unreachable
+  # corpus into a failure instead of a skip.
+  cli-integration:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # The release the operator is tested against (TESTED_CLI_VERSION in
+        # report.py), and the newest one.
+        cli: ["0.12.0", "latest"]
+    env:
+      SPARKFORENSICS_REQUIRE_CLI: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+      - run: npm install -g sparkforensics-cli@${{ matrix.cli }}
+      - run: pip install tox
+      - run: tox -e py312-airflow3 -- tests/integration -v
+
   finish:
     needs: test
     runs-on: ubuntu-latest

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.4.0}/AGENTS.md

@@ -9,14 +9,14 @@ Python 3.9+, apache-airflow 2.6+ or 3.0+, pytest. Runtime dependency: Node.js
 18+ + the `sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`)
 wherever the analysis runs: the worker for `SubprocessAnalyzeHook`, the SSH
 host for `SSHAnalyzeHook`. Log sources resolve an `EventLogRef` (`log_ref.py`)
-that backends consume; see `docs/architecture.md`. `deferrable=True` runs the
-SSH analysis as the SSH provider's detached remote job (architecture.md's
+that backends consume; see `ARCHITECTURE.md`. `deferrable=True` runs the
+SSH analysis as the SSH provider's detached remote job (`ARCHITECTURE.md`'s
 "Deferrable execution").
 
 ## Commands
 - `pip install -e ".[test,s3,ssh]"`, install for local dev.
-- `pytest`, run the test suite (mocks everything: no live Spark/Airflow/Node
-  needed for any single test).
+- `pytest`, run the test suite (mocks everything except `tests/integration`,
+  which runs the real CLI on public logs and skips without it).
 - `tox -e py311-airflow2` / `tox -e py311-airflow3`, run the suite against a
   specific Airflow major version.
 
@@ -44,9 +44,10 @@ of going public. The original multi-PR history (42 commits, 4 PRs) lives in
 indefinitely). `git blame`/`git log -p` on any line predating that date stop
 at the squash commit; check the archive repo instead.
 
-## Design docs
-- `docs/architecture.md`, `docs/api-reference.md`, `docs/runbook.md`,
-  `docs/glossary.md`, see below.
+## Docs
+- User docs: `docs/index.md` (getting started, configuration,
+  troubleshooting). Contributor docs: `CONTRIBUTING.md` (dev, tox matrix,
+  release), `ARCHITECTURE.md` (design notes).
 
 ## Maintaining this file
 

sparkforensics_operator-0.2.0/docs/architecture.md → sparkforensics_operator-0.4.0/ARCHITECTURE.md

@@ -30,8 +30,24 @@ Each backend declares the reference kinds it can read from where it runs
 (`AnalyzeHook.supported_log_refs`); `AnalyzeHook.analyze()` rejects any
 other kind with an error naming what it reads, before running anything, so
 a mismatched pairing (say, `SSHAnalyzeHook` with a log the worker
-downloaded) fails fast instead of analyzing the wrong file. Both trigger shapes (the standalone `SparkForensicsOperator` and the `spark_forensics_callback`
-`on_success_callback` factory) converge on one function,
+downloaded) fails fast instead of analyzing the wrong file.
+
+A run can also be compared against an earlier one: `baseline_log_source`
+is a second `LogSourceHook`, located and cleaned up next to `log_source`,
+and the backend passes its reference to the CLI as `--baseline` (an event
+log path) or `--baseline-shs-base-url`/`--baseline-app-id`/
+`--baseline-attempt-id` (a History Server application, which the CLI
+fetches itself, like the run under analysis). Each backend declares the
+kinds it can pass (`AnalyzeHook.supported_baseline_refs`: `LocalEventLog`
+and `HistoryServerApp` on the worker, `RemoteEventLog` on its own host and
+`HistoryServerApp` on the SSH host), and `analyze()` or the deferrable
+submit step rejects any other before running anything. The comparison
+itself, and the budgets on it, are the CLI's. Which run is the baseline
+is the DAG author's choice, made through that log source: the operator
+keeps no state across DAG runs.
+
+Both trigger shapes (the standalone `SparkForensicsOperator` and the
+`spark_forensics_callback` `on_success_callback` factory) converge on one function,
 `operator.run_spark_forensics(context, ...)`, so there is exactly one
 execute-analyze-persist-notify-threshold code path to maintain. Notify
 always runs before the threshold check's fail/warn/ignore branching, so a
@@ -39,19 +55,24 @@ breach that raises still gets a notification out first.
 
 ## Components
 
-- `_compat.py`, the single place this package imports Airflow's public
-  API from. `BaseOperator`/`BaseOperatorLink` live in `airflow.sdk` on
+- `_compat.py`, where this package imports Airflow's core API from (the
+  one exception is `links.py`'s lazy `XCom` import on the Airflow 2
+  branch; provider hooks are imported where they are used). `BaseOperator`/`BaseOperatorLink` live in `airflow.sdk` on
   3.x and `airflow.models.*` on 2.x. The Task SDK also re-homed `BaseHook`
   (3.1), and the exceptions, `TaskDeferred` and `conf` (3.2); their old
   paths still work there, as the same objects, behind deprecation shims.
   `_compat.py` tries the SDK path first and falls back to the old one, and
   `tests/test_compat.py` imports every module of the package in a fresh
-  interpreter and fails on any deprecation warning they emit. The repo has
+  interpreter and fails if any import fails or emits a
+  `DeprecationWarning`, `PendingDeprecationWarning` or `FutureWarning`
+  from inside the package. The repo has
   no ruff configuration, so there are no ruff `AIR3` rules to enable; that
   test covers the same ground.
 - `hooks/_templating.py`, `TemplatedHookMixin`, the base of every hook:
-  an empty `template_fields` and a `__repr__` that shows them. See
-  "Templated hook arguments" below.
+  an empty `template_fields` and a `__repr__` that shows them. Also
+  `task_renderer()` and `render_with_task_env()`, which render a value
+  with a task's Jinja environment for the callback and the tunnel hook.
+  See "Templated hook arguments" below.
 - `log_ref.py`, the `EventLogRef` kinds (`LocalEventLog`,
   `RemoteEventLog`, `HistoryServerApp`): frozen dataclasses saying where a
   log is, the only thing a log source and a backend share.
@@ -68,8 +89,9 @@ breach that raises still gets a notification out first.
   cleaning up after itself in that case; currently `HistoryServerLogSourceHook`
   and `SFTPLogSourceHook` do this, via their own internal try/except
   around the download/transfer that removes their owned temp dir on any
-  exception. See `docs/runbook.md`'s "Disk fills up" entry for which
-  hook/config combinations actually clean up.
+  exception. See the "Disk fills up on a worker" entry in
+  `docs/troubleshooting.md` for which hook/config combinations actually
+  clean up.
 - `hooks/log_source/{sftp,tunnel}.py`, `SFTPLogSourceHook` and
   `SSHTunneledLogSourceHook`, for on-prem log sources reachable only over
   the SSH connection an `SSHOperator` already has configured (no cloud
@@ -89,27 +111,47 @@ breach that raises still gets a notification out first.
   `HistoryServerApp` and do no I/O. Per-run values come from Jinja, the
   same way as in the fetching hooks (see "Templated hook arguments").
   `hooks/log_source/_path_template.py` checks a rendered path (rendered,
-  non-empty, no `..` segment) and `_history_server_args.py` checks a
+  non-empty, no `..` segment, no old `{run_id}`-style placeholder) and `_history_server_args.py` checks a
   rendered app id and attempt id; `_rolling_log.py` holds the
   rolling-log-entry-matching logic shared between `history_server.py`
   and `sftp.py`. `_dest_root.py` holds the `dest_root/<basename>` staging
   convention shared by `filesystem.py`/`sftp.py`, and the owned-temp-dir
   bookkeeping (create if `dest_dir` is unset, `rmtree` on cleanup/error if
   owned) shared by `sftp.py`/`history_server.py`.
-- `hooks/analyze/base.py`, `AnalyzeHook`: `analyze(log_ref, thresholds)`
-  checks the reference against `supported_log_refs` (`check_log_ref()`),
-  then calls the backend's `_analyze()`. `DeferrableAnalyzeHook` adds the
+- `hooks/analyze/base.py`, `AnalyzeHook`: `analyze(log_ref, thresholds,
+  baseline_ref=None)` checks the reference against `supported_log_refs`
+  (`check_log_ref()`) and any baseline against `supported_baseline_refs`
+  (`check_baseline_ref()`), then calls the backend's `_analyze()`. `DeferrableAnalyzeHook` adds the
   detached-job interface the deferrable mode drives (`submit`,
   `trigger_for`, `defer_timeout`, `collect`, `abandon`); see "Deferrable
   execution" below.
 - `hooks/analyze/_cli.py`, the CLI contract every backend shares: argument
   building for each reference kind (a positional path, or
-  `--shs-base-url`/`--app-id`/`--attempt-id`), threshold flags, exit-code
-  handling (0/1/3 usable, 2 and anything else raise), and building the
-  `Report` from the JSON report and stderr threshold lines. Backends only
-  differ in how they run the command and collect its output.
+  `--shs-base-url`/`--app-id`/`--attempt-id`), `--baseline`, threshold flags
+  (a dict of per-metric budgets becomes one `--regression-budget` or
+  `--stage-regression-budget <metric>:<pct>` each), the comparison tuning
+  flags (`--normalize-path` and `--ignore-conf` once per pattern,
+  `--noise-band-pct`), the `--redact` switch, the Markdown run (below),
+  exit-code handling, and building the `Report` from the JSON report and
+  stderr threshold lines. Exit codes 0, 1 and 3 carry a report. Every other
+  code raises a `CliError` with its own message: 2 (usage error), 4
+  (unreadable event log or failed History Server fetch), 5 (unreadable
+  baseline), 6 (internal error), anything else "unexpected". The CLI's
+  stderr separates a History Server failure a retry can clear
+  (`HistoryServerFetchError`) from one it cannot (`CliConfigurationError`,
+  an `AirflowFailException`, so Airflow does not retry it). Backends differ
+  only in how they run the command and collect its output.
 - `hooks/analyze/subprocess.py`, `SubprocessAnalyzeHook`, runs the CLI on
-  the worker with `--out <tmpfile>`.
+  the worker with `--out <tmpfile>`. The CLI writes one report format per
+  run, so for the Markdown report (`thresholds["markdown"]`, set by
+  `markdown_dest`) the backend runs it a second time with `--format md` and
+  no budget flags, and sets `Report.markdown`
+  (`AnalyzeHook.supports_markdown`). The first run's report and thresholds
+  decide the outcome; the second run only adds the text. `SSHAnalyzeHook`
+  does the same on the host: a second synchronous run or, in a deferred
+  job, a second command in the same remote job. Its outputs (report, stderr,
+  exit code) go to their own files, and the job's exit code stays the
+  first run's.
 - `hooks/analyze/ssh.py`, `SSHAnalyzeHook`, runs the CLI on the SSH host
   over `SSHHook` (imported lazily, `ssh` extra) and reads the report from
   stdout, so nothing is written on that host (see "Killing a task" below
@@ -135,15 +177,35 @@ breach that raises still gets a notification out first.
   which reports the CLI's pid, and the command that stops it.
 - `report.py`, `Report`/`ThresholdResult` dataclasses, plus parsing of the
   CLI's JSON report (`--out` file or stdout) and its stderr threshold
-  lines. No
+  lines. Several regression budgets print as `max-regression` lines
+  (`max-stage-regression` for the stage budgets), and
+  `parse_threshold_results()` keeps every line it reads, so a name can have
+  several results. It refuses a `schemaVersion` outside
+  `MIN_SCHEMA_VERSION` to `MAX_SCHEMA_VERSION` (an
+  `UnsupportedReportSchema` that `build_report()` turns into a
+  `CliConfigurationError`), reads the CLI's version from the report's
+  `generator`, and checks it against the release the operator is tested
+  against (`TESTED_CLI_VERSION`). No
   threshold-evaluation *logic* lives here: sparkforensics's own CLI decides
   pass/violation/inconclusive, this module only parses what it printed, so
   it can never drift from what the CLI actually enforces.
-- `sinks.py`, persists the report JSON (local path, `file://`, `s3://`).
+- `exceptions.py`, `ThresholdBreached`, an `AirflowFailException` so a
+  breach fails the task without consuming its retries, and the CLI
+  errors: `CliError` (a CLI exit code with no report, carrying
+  `exit_code`), its `HistoryServerFetchError` (retried) and
+  `CliConfigurationError` (an `AirflowFailException`, not retried).
+- `sinks.py`, persists the report JSON (local path, `file://`, `s3://`),
+  and, through `persist_text()`, the Markdown report. The JSON is the
+  CLI's whole report (`Report.raw`) plus the operator's `thresholdResults`
+  and the comparison section, so a field the CLI adds reaches the file
+  without a change here. `operator._report_json()` builds it; a `Report`
+  built by hand (no `raw`) persists the sections the operator reads.
 - `notify.py`, the `Notifier` base class: subclasses implement `_send()`
   for whatever messaging/paging system they target, and `notify()` makes
-  that best-effort, never raising. No concrete notifier ships in this
-  package.
+  that best-effort, never raising. `_default_message()` writes the
+  one-line summary (application, verdict, run time, finding counts,
+  skipped lines, destination), leaving out what the report lacks. No
+  concrete notifier ships in this package.
 - `summary.py`, the summary XCom (`SUMMARY_XCOM_KEY`,
   `build_summary()`) and the report URL (`render_report_url()`,
   `validate_report_url_template()`). See "Summary XCom and report URL"
@@ -176,6 +238,7 @@ breach that raises still gets a notification out first.
   deferred run's `execute_complete()` calls.
 - `callback.py`, `spark_forensics_callback(**kwargs)`, an
   `on_success_callback` factory sharing the same core.
+- `__init__.py`, the public exports and `__version__`.
 
 ## Deferrable execution
 
@@ -185,7 +248,7 @@ worker slot while the CLI runs on the SSH host:
 ```
 worker:     execute()  locate log_ref -> backend.submit() -> defer(trigger_for(job), kwargs)
 triggerer:  SSHRemoteJobTrigger polls exit_code, streams stdout.log
-worker:     execute_complete(event, job, log_ref, report_dest, thresholds)
+worker:     execute_complete(event, job, log_ref, report_dest, thresholds, baseline_log_ref)
               -> backend.collect()  read stderr + report.json, remove the job dir
               -> handle_report()    same persist/notify/threshold code as a sync run
 ```
@@ -207,8 +270,10 @@ from the DAG file, so nothing set on the instance before `defer()`
 survives. Everything `execute_complete()` needs travels in the resume
 kwargs, as JSON-native values that Airflow 2's `BaseSerialization` and
 Airflow 3's serde both round-trip: the job dict (connection id, CLI
-binary, `timeout`, every remote path), the log reference as a dict, the
-rendered `report_dest` and the thresholds the job was submitted with. The
+binary, `timeout`, `poll_interval`, the job id, the submission time and
+every remote path), the log reference as a dict (and the baseline's, when
+there is one), the rendered `report_dest` and the thresholds the job was
+submitted with. The
 job dict and the log reference are built from the rendered hooks, so
 templated hook arguments cross the deferral as the values the job ran
 with. The notifier, `on_threshold_breach` and `report_url_template` come
@@ -243,10 +308,11 @@ session ids in `/proc/<pid>/stat`. A pid is only signalled while its
 command line still names its job directory, so a pid reused after a
 reboot is left alone.
 
-Failure paths. A trigger error event makes `collect()` stop and remove
-the job before raising. A deferral that times out (the backend's
-`defer_timeout`, `timeout` plus 120 seconds, or the task's
-`execution_timeout`) resumes with `next_method="__fail__"`, which never
+Failure paths. A trigger error event without an exit code makes
+`collect()` stop and remove the job before raising. A deferral that times
+out (the backend's `defer_timeout`, `timeout` per CLI run in the job, so
+twice with `markdown_dest`, plus 120 seconds counted from submission, or
+the task's `execution_timeout`) resumes with `next_method="__fail__"`, which never
 reaches `execute_complete()`; `resume_execution()` is overridden to call
 `backend.abandon(context, job)` first, with the job from XCom. Airflow 2
 takes a different path when `execution_timeout` ran out during the
@@ -256,7 +322,8 @@ anything on the resumed operator, then calls `on_kill()`, which is why
 between `submit()` and `defer()` (building the trigger, the timeout, the
 deferral itself) abandons the job before the error propagates, and a
 task timeout or kill there reaches `on_kill()`, which does the same. The
-runbook's "Deferred runs" section lists the behaviour for each case.
+"Deferred runs" section of `docs/troubleshooting.md` lists the behaviour
+for each case.
 
 Killing a task. `SparkForensicsOperator.on_kill()` uses the context of the
 `execute()` or `execute_complete()` running in the same process, or,
@@ -306,7 +373,7 @@ synchronous.
 Hook arguments are Jinja templates, rendered by Airflow's own nested
 template-field mechanism, the same one used for any object assigned to a
 templated operator field. `SparkForensicsOperator.template_fields` is
-`("report_dest", "log_source", "backend")`, and each hook class declares
+`("report_dest", "log_source", "backend", "baseline_log_source")`, and each hook class declares
 the attributes it templates in its own `template_fields`; Airflow renders
 those in place, on Airflow 2 and 3 alike. There is one mechanism, and no
 hook formats a string itself. The earlier `{run_id}`-style `str.format`
@@ -316,7 +383,7 @@ one is rejected with an error naming the Jinja replacement.
 Three details keep this safe.
 
 - `render_template_fields()` gives the task its own shallow copy of
-  `log_source` and `backend` before rendering, because Airflow renders in
+  `log_source`, `baseline_log_source` and `backend` before rendering, because Airflow renders in
   place and the same hook instance is often shared by several tasks or by
   every mapped instance.
 - The log-source method is `locate()`, not `resolve()`. Airflow 3's
@@ -335,7 +402,8 @@ Three details keep this safe.
 when Airflow renders the task. `spark_forensics_callback` runs outside
 any template rendering, so it renders `report_dest`, `log_source` and
 `backend` itself, on copies, with the upstream task's Jinja environment.
-Both render through a copy of the task with an empty `template_ext`, so a
+Both render through the task, or a copy of it with an empty
+`template_ext` when it has one, so a
 value ending in `.json` is rendered as a string, not looked up as a
 template file on an EMR or Spark-on-Kubernetes upstream.
 
@@ -345,7 +413,15 @@ Before notifying and before any threshold breach raises,
 `handle_report()` pushes a summary under the `sparkforensics_summary`
 XCom key: the schema version, the destination, the report URL, the
 impact-band counts, whether the run violated a threshold, the breached
-and inconclusive threshold names, and the CLI exit code. Downstream tasks
+and inconclusive threshold names, the CLI exit code, the CLI version
+that wrote the report, and the run's verdict title, application name,
+wall-clock time and skipped line count (`Report.verdict_title`,
+`app_name`, `run_time_ms` and `skipped_lines`, read from the CLI's
+summary). A run compared against a baseline adds a `comparison` entry: the direction the CLI
+computed for one metric (`regression_metric`, wall-clock by default) as
+the verdict, the comparison's confidence and reason, and that metric's
+values and change in percent, taken from the CLI's comparison section.
+The full section is persisted with the report. Downstream tasks
 can branch on it without reading the report. It is built from the same
 `Report` on both the synchronous and the deferred path, and
 `return_value` stays the destination.
@@ -358,7 +434,7 @@ presigns anything: the URL points at a place the viewer's own
 credentials open. `ReportLink` shows that URL when it is set and the raw
 destination when it is not.
 
-## Deferred
+## Not implemented
 
 **http/MCP AnalyzeHook backend.** The spec originally described an "http"
 backend calling a deployed sparkforensics `server/` instance. `server/`

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.4.0}/CHANGELOG.md

@@ -7,6 +7,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-11-04
+
+### Upgrade notes
+
+- `sparkforensics-cli` 0.10.2 counts every failed task attempt in `max_failed_task_rate_pct`, so a job with retried tasks can fail where it passed.
+- With 0.10.2, a log with skipped lines gives exit 3 and an inconclusive `parse-incomplete` result. The task does not fail; see `skipped_lines` in the summary XCom.
+- The operator is tested against `sparkforensics-cli` 0.12.0.
+- With CLI 0.12.0 and newer, `findings` lists critical and warning findings only. The persisted report, `report.findings` and the `findings` XCom omit info findings; `notesHidden` in the report counts them. The summary's `impact_band_counts` is unchanged.
+
+### Added
+
+- `markdown_dest` saves the CLI's Markdown report (`--format md`) next to the JSON report. The analysis runs twice.
+- `redact=True` passes `--redact`.
+- `normalize_path`, `ignore_conf` and `noise_band_pct` tune the baseline comparison.
+- `regression_budgets`, `stage_regression_budgets` and `stage_qualities` set several regression budgets at once.
+- A Spark History Server application can be the baseline on `SubprocessAnalyzeHook` and `SSHAnalyzeHook`.
+- The summary XCom has `verdict`, `app_name`, `run_time_ms`, `skipped_lines` and `cli_version`. The default notification names the application, the verdict and the run time.
+- A warning is logged when the CLI is older than the tested version.
+- `CliError`, `CliConfigurationError` and `HistoryServerFetchError` in `sparkforensics_operator.exceptions`.
+
+### Changed
+
+- The operator reads the JSON output shape of `sparkforensics-cli` 0.12.0 (`{schemaVersion, run, comparison, budgets}`) and still reads the older shapes.
+- Each CLI exit code without a report (2, 4, 5, 6) has its own error message. The task still fails.
+- Airflow does not retry usage errors, and retries History Server failures that can clear.
+- A report with a newer `schemaVersion` than the operator reads fails the task.
+- The saved report is the CLI's whole JSON report. `Report.raw` holds it. It includes `effectiveConf`, where the CLI masks secret-looking keys and values; see [Report destinations](docs/configuration.md#report-destinations).
+- `Report.threshold_results` keeps every result of a threshold the CLI reports on several lines.
+
+### Fixed
+
+- Thresholds such as `0.00001` or `1e16` reach the CLI as plain decimals, not exponent form, which the CLI rejects.
+
+## [0.3.0] - 2026-09-28
+
+### Added
+
+- Run comparison: `baseline_log_source` on `SparkForensicsOperator` and
+  `spark_forensics_callback` resolves an earlier run's event log, the same
+  way `log_source` resolves the run under analysis, and passes it to
+  `sparkforensics-analyze --baseline`. It must resolve to a file path
+  readable where the analysis runs: a `LocalEventLog` for
+  `SubprocessAnalyzeHook`, a `RemoteEventLog` on the same SSH connection
+  for `SSHAnalyzeHook`. A History Server application is rejected, since
+  the CLI reads a baseline only from a path. A baseline that is, contains
+  or sits inside the log under analysis fails the task before analysis.
+  Works with `deferrable=True`.
+- Comparison thresholds `max_regression_pct`, `regression_metric` and
+  `fail_on_introduced`, mirroring the CLI's flags and its rules: all need
+  `baseline_log_source`, and `regression_metric` needs
+  `max_regression_pct`. A breach raises `ThresholdBreached` like any other
+  threshold.
+- The `sparkforensics_summary` XCom gains a `comparison` key when a
+  baseline ran: the checked metric's verdict, the comparison's confidence
+  and reason, and the metric's baseline and candidate values and change in
+  percent. The persisted report carries the CLI's full comparison section.
+
+### Changed
+
+- The operator reads report schema 5, where a finding's text figure is in
+  `valueText` and its `value` is always a number or `null`. A report with
+  an older `schemaVersion` fails the task.
+- The persisted report and `Report` carry the CLI's `verdict` and
+  `notRunChecks` sections (`Report.verdict`, `Report.not_run_checks`).
+  Before this, the operator persisted only the sections it listed, so
+  these two would have been dropped.
+
 ## [0.2.0] - 2026-09-25
 
 ### Added
@@ -154,7 +221,9 @@ Initial release.
 - Generic `Notifier` interface for threshold-breach alerting.
 - PyPI trusted-publishing GitHub Actions workflow (OIDC, no stored token).
 
-[Unreleased]: https://github.com/shuffle-works/sparkforensics-operator/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/shuffle-works/sparkforensics-operator/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.4.0
+[0.3.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.3.0
 [0.2.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.2.0
 [0.1.1]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.1.1
 [0.1.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.1.0

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.4.0}/CLAUDE.md

@@ -9,15 +9,15 @@ Python 3.9+, apache-airflow 2.6+ or 3.0+, pytest. Runtime dependency: Node.js
 18+ + the `sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`)
 wherever the analysis runs: the worker for `SubprocessAnalyzeHook`, the SSH
 host for `SSHAnalyzeHook`. Log sources resolve an `EventLogRef` (`log_ref.py`)
-that backends consume; see `docs/architecture.md`.
+that backends consume; see `ARCHITECTURE.md`.
 
 ## Commands
 - `pip install -e ".[test,s3,ssh]"`, install for local dev.
-- `pytest`, run the test suite (mocks everything: no live Spark/Airflow/Node
-  needed for any single test).
+- `pytest`, run the test suite (mocks everything except `tests/integration`,
+  which runs the real CLI on public logs and skips without it).
 - `tox -e py311-airflow2` / `tox -e py311-airflow3`, run the suite against a
   specific Airflow major version.
 
-## Design docs
-- `docs/architecture.md`, `docs/api-reference.md`, `docs/runbook.md`,
-  `docs/glossary.md`, see below.
+## Docs
+- User docs: `docs/index.md` (getting started, configuration,
+  troubleshooting). Contributor docs: `CONTRIBUTING.md`, `ARCHITECTURE.md`.

sparkforensics_operator-0.4.0/CONTRIBUTING.md

@@ -0,0 +1,100 @@
+# Contributing
+
+Design notes and the component map are in [ARCHITECTURE.md](ARCHITECTURE.md).
+User documentation lives under [docs/](docs/index.md).
+
+## Development install
+
+```bash
+pip install -e ".[test,s3,ssh]"
+```
+
+## Running the tests
+
+```bash
+pytest
+```
+
+The suite needs no live Spark or Airflow, and no Node.js apart from
+`tests/integration` (below). The one real process is a shell: the deferred-mode tests run the SSH provider's real shell wrapper against a local `sh` standing in
+for the SSH host (`tests/hooks/analyze/_local_ssh.py`), so they need
+`bash`, `setsid` and coreutils `timeout` on the machine running the tests.
+The kill and sweep tests also run with procps hidden
+(`LocalHost.hide_procps()`), as on slim images.
+
+### Tests against the real CLI
+
+The other tests simulate `sparkforensics-analyze`, so they cannot catch the
+CLI changing an exit code or a flag's rules. `tests/integration` runs the
+real CLI on public logs from the
+[spark-event-corpus-data](https://github.com/shuffle-works/spark-event-corpus-data)
+repository (fetched at a pinned tag), through `SubprocessAnalyzeHook`. It
+covers a clean log, a violation, an unreadable log, an unreadable baseline
+and budgets that print in exponent form.
+
+```bash
+npm install -g sparkforensics-cli
+pytest tests/integration
+```
+
+They need Node.js 18+, the `sparkforensics-cli` package on `PATH` (or
+its binary at `SPARKFORENSICS_ANALYZE_BIN`) and network access to GitHub, and
+skip when the CLI or the network is missing. With `SPARKFORENSICS_REQUIRE_CLI=1`
+a missing CLI or an unreachable corpus fails the run instead, as in CI. The
+`cli-integration` job in `.github/workflows/ci.yml` runs them against the
+release the operator is tested against (`TESTED_CLI_VERSION` in
+`report.py`) and the newest one.
+
+## The tox Airflow matrix
+
+`tox.ini` runs the suite against each supported Airflow major version:
+
+```bash
+tox -e py311-airflow2
+tox -e py311-airflow3
+```
+
+The full env list is `py{39,310,311,312}-airflow2`,
+`py{310,311,312}-airflow3`, and three floor envs:
+
+- `py39-airflow2min` pins Airflow 2.6.3 without the `ssh` extra, and
+  omits the SSH-only modules from its coverage (`coverage-no-ssh.ini`).
+- `py39-airflow2sshmin` runs the `ssh` extra's floor
+  (`apache-airflow-providers-ssh` 3.7.1) on Airflow 2.6.3. The deferrable
+  tests skip there, since that mode needs a newer provider.
+- `py311-airflow2deferrablemin` pins the deferrable mode's floors
+  (`constraints-deferrable-floor.txt`).
+
+The two Airflow 2.6.3 envs install through
+`constraints-airflow-2.6-floor.txt` so the extras step can't upgrade
+Airflow past the floor; `tox.ini` explains why.
+
+CI (`.github/workflows/ci.yml`) runs a subset of these envs on every push
+to `main` and every pull request. Each env writes coverage to
+`coverage-<envname>.lcov`; CI uploads one file per env to Coveralls and
+merges them in a `finish` job. `usedevelop = true` in `tox.ini` keeps the
+lcov `SF:` paths pointing at `src/sparkforensics_operator/...` instead of
+a `.tox` virtualenv, which Coveralls needs to attribute lines.
+
+## Releasing to PyPI
+
+`.github/workflows/publish.yml` builds the sdist and wheel and publishes
+them on every GitHub Release (`release: published`). It uses PyPI trusted
+publishing (OIDC), so no API token is stored in the repository.
+
+One-time setup before the first release:
+
+- On PyPI, add a trusted publisher for this project: owner
+  `shuffle-works`, repo `sparkforensics-operator`, workflow `publish.yml`,
+  environment `pypi`.
+- In the GitHub repository settings, create an environment named `pypi`
+  (optionally with required reviewers) to match.
+
+To cut a release:
+
+1. Bump `version` in `pyproject.toml` and move the `[Unreleased]` entries
+   in `CHANGELOG.md` under the new version. The Airflow provider metadata
+   reads the version from the installed package, so nothing else needs
+   changing.
+2. Tag the release commit (for example `v0.2.0`).
+3. Publish a GitHub Release from that tag. The workflow does the rest.

{sparkforensics_operator-0.2.0 → sparkforensics_operator-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: sparkforensics-operator
-Version: 0.2.0
+Version: 0.4.0
 Summary: Airflow operator that runs sparkforensics analysis on a Spark job's event log and acts on the result.
 Project-URL: Homepage, https://github.com/shuffle-works/sparkforensics-operator
 Project-URL: Repository, https://github.com/shuffle-works/sparkforensics-operator
@@ -71,6 +71,17 @@ runs as part of the DAG, right where the Spark task just ran.
   you configure are enforced, and a breach raises `ThresholdBreached`
   without consuming the task's retries, since re-running would just reach
   the same verdict.
+- Compare a run against an earlier one. `baseline_log_source` takes any
+  log source that resolves to an event log path the analysis can read
+  (`path_template="/spark-logs/{{ prev_ds }}/eventlog"`, an XCom pull) or
+  to a Spark History Server application. `max_regression_pct` fails the
+  DAG when runtime, or the metric `regression_metric` names, regressed past
+  that budget. `regression_budgets` budgets several metrics at once, and
+  `stage_regression_budgets` the paired stages. `fail_on_introduced` fails
+  it when the run gained new findings. The comparison's verdict,
+  confidence and measured regression land in the summary XCom.
+- `markdown_dest` also persists the CLI's Markdown report, for a notifier
+  to post or a person to read.
 - Reports persist to a local path, `file://`, or `s3://`, and a clickable
   "SparkForensics report" link shows up on the task in the Airflow UI. Set
   `report_url_template` (an S3 console URL, say) to make it open in a
@@ -85,8 +96,8 @@ runs as part of the DAG, right where the Spark task just ran.
 - An optional `Notifier` you implement (Slack, MS Teams, email, PagerDuty,
   ZenDuty, whatever you use) gets a best-effort pass/fail summary; a
   delivery failure never fails the task itself.
-- Tested against both Airflow 2.6+ and Airflow 3.0+ (CI matrix, Python
-  3.9-3.12). No live Spark, Airflow or Node is needed to run the tests:
+- Tested against both Airflow 2.6+ and Airflow 3 (CI runs Python 3.9,
+  3.11 and 3.12; the tox matrix also covers 3.10). No live Spark, Airflow or Node is needed to run the tests:
   everything but the shell on the stand-in SSH host is mocked.
 
 ## Quick start
@@ -162,16 +173,17 @@ the analysis runs: the CLI runs as a detached job on the SSH host and the
 triggerer polls it, which needs a running triggerer with
 `sparkforensics-operator[ssh]` installed and
 `apache-airflow-providers-ssh>=6.0.1`, so Airflow 2.11 or newer. See
-[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md#deferrable-execution).
+[Deferrable execution](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md#deferrable-execution).
 
 Whichever host runs the analysis (the worker for `SubprocessAnalyzeHook`,
 the SSH host for `SSHAnalyzeHook`) needs Node.js 18+ and the
-`sparkforensics-cli` npm package (`npm install -g sparkforensics-cli`), so
+`sparkforensics-cli` npm package, version 0.4.0 or newer
+(`npm install -g "sparkforensics-cli@>=0.4.0"`), so
 `sparkforensics-analyze` is resolvable on `PATH`. Add the `ssh` extra for
 the SSH hooks (it works from Airflow 2.6, like the package; only
 `deferrable=True` needs Airflow 2.11+), and the `s3` extra if
 `report_dest` is `s3://...`.
-Full details in [`docs/runbook.md`](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md).
+Full details in [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md).
 
 ## Development
 
@@ -189,11 +201,11 @@ tox -e py311-airflow3
 
 ## Learn more
 
-- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/architecture.md), components and design decisions
-- [API reference](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/api-reference.md), the full operator/hook/threshold
-  surface
-- [Runbook](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/runbook.md), worker and SSH host prerequisites, troubleshooting
-- [Glossary](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/glossary.md), domain terms
+- [Getting started](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/getting-started.md), install and host prerequisites, and DAG wiring examples
+- [Configuration](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/configuration.md), the full operator/hook/threshold surface
+- [Troubleshooting](https://github.com/shuffle-works/sparkforensics-operator/blob/main/docs/troubleshooting.md), error messages and deferred-run failure modes
+- [Contributing](https://github.com/shuffle-works/sparkforensics-operator/blob/main/CONTRIBUTING.md), development setup and the release procedure
+- [Architecture](https://github.com/shuffle-works/sparkforensics-operator/blob/main/ARCHITECTURE.md), components and design decisions
 
 ## License