sparkforensics-operator 0.1.0__tar.gz

sparkforensics_operator-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,31 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - python: "3.11"
+            tox-env: py311-airflow2
+          - python: "3.12"
+            tox-env: py312-airflow2
+          - python: "3.11"
+            tox-env: py311-airflow3
+          - python: "3.12"
+            tox-env: py312-airflow3
+          - python: "3.9"
+            tox-env: py39-airflow2min
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install tox
+      - run: tox -e ${{ matrix.tox-env }}

sparkforensics_operator-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,33 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

sparkforensics_operator-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.worktrees/
+__pycache__/
+*.pyc
+.pytest_cache/
+.tox/
+*.egg-info/
+dist/
+build/
+.venv/
+docs/superpowers/

sparkforensics_operator-0.1.0/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-09-06
+
+Initial release.
+
+### Added
+
+- `SparkForensicsOperator`: after a Spark job runs, fetches its event log
+  and runs it through the `sparkforensics-analyze` CLI, then
+  persists/XComs/threshold-checks/notifies on the result.
+- Log source hooks: `FilesystemLogSourceHook`, `HistoryServerLogSourceHook`,
+  `SFTPLogSourceHook`, `SSHTunneledLogSourceHook`, `XComLogSourceHook`.
+- `SubprocessAnalyzeHook`, shelling out to the `sparkforensics-analyze` CLI.
+- Generic `Notifier` interface for threshold-breach alerting.
+- PyPI trusted-publishing GitHub Actions workflow (OIDC, no stored token).
+
+[0.1.0]: https://github.com/shuffle-works/sparkforensics-operator/releases/tag/v0.1.0

sparkforensics_operator-0.1.0/CLAUDE.md

@@ -0,0 +1,26 @@
+# sparkforensics-operator
+
+Airflow package: after a Spark job runs, fetches its event log and runs it
+through the sparkforensics-analyze CLI, then persists/XComs/threshold-checks/
+notifies on the result.
+
+## Stack
+Python 3.9+, apache-airflow 2.6+ or 3.0+, pytest. Runtime dependency: Node.js
+(engines `>=22.18.0 <23.0.0 || >=23.6.0`) + the `sparkforensics` npm package
+on the worker, for the `sparkforensics-analyze` CLI the subprocess backend
+shells out to.
+
+## Commands
+- `pip install -e ".[test,s3,ssh]"`, install for local dev.
+- `pytest`, run the test suite (mocks everything: no live Spark/Airflow/Node
+  needed for any single test).
+- `tox -e py311-airflow2` / `tox -e py311-airflow3`, run the suite against a
+  specific Airflow major version.
+
+## Design docs
+- `docs/superpowers/specs/2026-09-05-sparkforensics-operator-design.md`, the original design spec.
+- `docs/superpowers/plans/2026-09-05-sparkforensics-operator.md`, the
+  implementation plan, including every upstream sparkforensics CLI/API
+  fact and every Airflow-2-vs-3 compat fact this codebase depends on.
+- `docs/architecture.md`, `docs/api-reference.md`, `docs/runbook.md`,
+  `docs/glossary.md`, see below.

sparkforensics_operator-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Agustin Recoba
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sparkforensics_operator-0.1.0/PKG-INFO

@@ -0,0 +1,147 @@
+Metadata-Version: 2.5
+Name: sparkforensics-operator
+Version: 0.1.0
+Summary: Airflow operator that runs sparkforensics analysis on a Spark job's event log and acts on the result.
+Project-URL: Homepage, https://github.com/shuffle-works/sparkforensics-operator
+Project-URL: Repository, https://github.com/shuffle-works/sparkforensics-operator
+Project-URL: Issues, https://github.com/shuffle-works/sparkforensics-operator/issues
+Author-email: Agustin Recoba <agustin.recoba@outlook.com>
+License: MIT
+License-File: LICENSE
+Keywords: airflow,data-engineering,observability,spark,sparkforensics
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Apache Airflow
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Requires-Dist: apache-airflow>=2.6
+Requires-Dist: requests>=2.28
+Provides-Extra: s3
+Requires-Dist: apache-airflow-providers-amazon>=8.0; extra == 's3'
+Provides-Extra: ssh
+Requires-Dist: apache-airflow-providers-sftp>=4.0; extra == 'ssh'
+Requires-Dist: apache-airflow-providers-ssh>=5.0; extra == 'ssh'
+Provides-Extra: test
+Requires-Dist: pytest-cov>=4.0; extra == 'test'
+Requires-Dist: pytest>=7.0; extra == 'test'
+Description-Content-Type: text/markdown
+
+# sparkforensics-operator
+
+[![CI](https://github.com/shuffle-works/sparkforensics-operator/actions/workflows/ci.yml/badge.svg)](https://github.com/shuffle-works/sparkforensics-operator/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+An Airflow operator that runs [sparkforensics](https://github.com/shuffle-works/sparkforensics)
+analysis on a Spark job's event log right after it finishes, then acts on
+the result: persist the report, push it to XCom, notify (via any
+messaging/paging system you implement), and optionally fail the DAG if a
+budget was breached.
+
+Airflow tells you a Spark task succeeded. It doesn't tell you that stage 14
+spilled 40GB from a skewed join, or that runtime crept 20% past last week's
+run. This package closes that gap without a separate monitoring job: it
+runs as part of the DAG, right where the Spark task just ran.
+
+## Why this exists
+
+- Two ways to wire it in: a standalone `SparkForensicsOperator` that fails
+  the DAG on a threshold breach, or a `spark_forensics_callback` factory to
+  attach as `on_success_callback` on the Spark task itself, log-only.
+- Three ways to get the event log: fetch it from the Spark History Server,
+  read it from a filesystem/mounted-HDFS path template, or pull it straight
+  out of XCom if the upstream task already has it.
+- Budget thresholds mirror `sparkforensics-analyze`'s own CLI flags (max
+  runtime, spill, skew, failed-task rate, min efficiency); only the ones
+  you configure are enforced, and a breach raises `ThresholdBreached`
+  without consuming the task's retries, since re-running would just reach
+  the same verdict.
+- Reports persist to a local path, `file://`, or `s3://`, and a clickable
+  "SparkForensics report" link shows up on the task in the Airflow UI.
+- An optional `Notifier` you implement (Slack, MS Teams, email, PagerDuty,
+  ZenDuty, whatever you use) gets a best-effort pass/fail summary; a
+  delivery failure never fails the task itself.
+- Tested against both Airflow 2.6+ and Airflow 3.0+ (CI matrix, Python
+  3.9-3.12). 97 tests, 96% coverage, no live Spark/Airflow/Node needed to
+  run them: everything is mocked.
+
+## Quick start
+
+```bash
+pip install sparkforensics-operator
+```
+
+Standalone operator, chained after the Spark task, fails the DAG on breach:
+
+```python
+from sparkforensics_operator import FilesystemLogSourceHook, SparkForensicsOperator, SubprocessAnalyzeHook
+
+run_spark_job = SparkSubmitOperator(task_id="run_spark_job", ...)
+
+check_spark_job = SparkForensicsOperator(
+    task_id="check_spark_job",
+    log_source=FilesystemLogSourceHook(path_template="/mnt/spark-logs/{run_id}/eventlog"),
+    backend=SubprocessAnalyzeHook(),
+    report_dest="s3://reports/{{ run_id }}/report.json",
+    max_runtime_ms=3_600_000,
+    max_skew_ratio=3,
+    on_threshold_breach="fail",
+)
+
+run_spark_job >> check_spark_job
+```
+
+Or attach it as a callback instead. Airflow logs and swallows any exception
+raised inside a callback, so a breach here never fails or retries the
+upstream task, whatever `on_threshold_breach` is set to:
+
+```python
+from sparkforensics_operator import SubprocessAnalyzeHook, XComLogSourceHook, spark_forensics_callback
+
+run_spark_job = SparkSubmitOperator(
+    task_id="run_spark_job",
+    on_success_callback=spark_forensics_callback(
+        log_source=XComLogSourceHook(task_id="run_spark_job"),
+        backend=SubprocessAnalyzeHook(),
+        report_dest="s3://reports/run_spark_job/app.json",
+        max_runtime_ms=3_600_000,
+    ),
+    ...,
+)
+```
+
+The worker needs Node.js (`>=22.18.0 <23.0.0 || >=23.6.0`) and the
+`sparkforensics` npm package installed, so `sparkforensics-analyze` is
+resolvable on `PATH`. Add the `s3` extra if `report_dest` is `s3://...`.
+Full details in [`docs/runbook.md`](docs/runbook.md).
+
+## Development
+
+```bash
+pip install -e ".[test,s3]"
+pytest
+```
+
+Test against a specific Airflow major version:
+
+```bash
+tox -e py311-airflow2
+tox -e py311-airflow3
+```
+
+## Learn more
+
+- [Architecture](docs/architecture.md), components and design decisions
+- [API reference](docs/api-reference.md), the full operator/hook/threshold
+  surface
+- [Runbook](docs/runbook.md), worker prerequisites and troubleshooting
+- [Glossary](docs/glossary.md), domain terms
+
+## License
+
+[MIT](LICENSE)

sparkforensics_operator-0.1.0/README.md

@@ -0,0 +1,113 @@
+# sparkforensics-operator
+
+[![CI](https://github.com/shuffle-works/sparkforensics-operator/actions/workflows/ci.yml/badge.svg)](https://github.com/shuffle-works/sparkforensics-operator/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+An Airflow operator that runs [sparkforensics](https://github.com/shuffle-works/sparkforensics)
+analysis on a Spark job's event log right after it finishes, then acts on
+the result: persist the report, push it to XCom, notify (via any
+messaging/paging system you implement), and optionally fail the DAG if a
+budget was breached.
+
+Airflow tells you a Spark task succeeded. It doesn't tell you that stage 14
+spilled 40GB from a skewed join, or that runtime crept 20% past last week's
+run. This package closes that gap without a separate monitoring job: it
+runs as part of the DAG, right where the Spark task just ran.
+
+## Why this exists
+
+- Two ways to wire it in: a standalone `SparkForensicsOperator` that fails
+  the DAG on a threshold breach, or a `spark_forensics_callback` factory to
+  attach as `on_success_callback` on the Spark task itself, log-only.
+- Three ways to get the event log: fetch it from the Spark History Server,
+  read it from a filesystem/mounted-HDFS path template, or pull it straight
+  out of XCom if the upstream task already has it.
+- Budget thresholds mirror `sparkforensics-analyze`'s own CLI flags (max
+  runtime, spill, skew, failed-task rate, min efficiency); only the ones
+  you configure are enforced, and a breach raises `ThresholdBreached`
+  without consuming the task's retries, since re-running would just reach
+  the same verdict.
+- Reports persist to a local path, `file://`, or `s3://`, and a clickable
+  "SparkForensics report" link shows up on the task in the Airflow UI.
+- An optional `Notifier` you implement (Slack, MS Teams, email, PagerDuty,
+  ZenDuty, whatever you use) gets a best-effort pass/fail summary; a
+  delivery failure never fails the task itself.
+- Tested against both Airflow 2.6+ and Airflow 3.0+ (CI matrix, Python
+  3.9-3.12). 97 tests, 96% coverage, no live Spark/Airflow/Node needed to
+  run them: everything is mocked.
+
+## Quick start
+
+```bash
+pip install sparkforensics-operator
+```
+
+Standalone operator, chained after the Spark task, fails the DAG on breach:
+
+```python
+from sparkforensics_operator import FilesystemLogSourceHook, SparkForensicsOperator, SubprocessAnalyzeHook
+
+run_spark_job = SparkSubmitOperator(task_id="run_spark_job", ...)
+
+check_spark_job = SparkForensicsOperator(
+    task_id="check_spark_job",
+    log_source=FilesystemLogSourceHook(path_template="/mnt/spark-logs/{run_id}/eventlog"),
+    backend=SubprocessAnalyzeHook(),
+    report_dest="s3://reports/{{ run_id }}/report.json",
+    max_runtime_ms=3_600_000,
+    max_skew_ratio=3,
+    on_threshold_breach="fail",
+)
+
+run_spark_job >> check_spark_job
+```
+
+Or attach it as a callback instead. Airflow logs and swallows any exception
+raised inside a callback, so a breach here never fails or retries the
+upstream task, whatever `on_threshold_breach` is set to:
+
+```python
+from sparkforensics_operator import SubprocessAnalyzeHook, XComLogSourceHook, spark_forensics_callback
+
+run_spark_job = SparkSubmitOperator(
+    task_id="run_spark_job",
+    on_success_callback=spark_forensics_callback(
+        log_source=XComLogSourceHook(task_id="run_spark_job"),
+        backend=SubprocessAnalyzeHook(),
+        report_dest="s3://reports/run_spark_job/app.json",
+        max_runtime_ms=3_600_000,
+    ),
+    ...,
+)
+```
+
+The worker needs Node.js (`>=22.18.0 <23.0.0 || >=23.6.0`) and the
+`sparkforensics` npm package installed, so `sparkforensics-analyze` is
+resolvable on `PATH`. Add the `s3` extra if `report_dest` is `s3://...`.
+Full details in [`docs/runbook.md`](docs/runbook.md).
+
+## Development
+
+```bash
+pip install -e ".[test,s3]"
+pytest
+```
+
+Test against a specific Airflow major version:
+
+```bash
+tox -e py311-airflow2
+tox -e py311-airflow3
+```
+
+## Learn more
+
+- [Architecture](docs/architecture.md), components and design decisions
+- [API reference](docs/api-reference.md), the full operator/hook/threshold
+  surface
+- [Runbook](docs/runbook.md), worker prerequisites and troubleshooting
+- [Glossary](docs/glossary.md), domain terms
+
+## License
+
+[MIT](LICENSE)