flowsense-engine 0.4.0__tar.gz → 0.5.1__tar.gz

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/CHANGELOG.md

@@ -5,6 +5,78 @@ All notable changes to FlowSense are documented in this file. The project uses
 
 ## [Unreleased]
 
+### Added
+
+### Changed
+
+### Fixed
+
+## [0.5.1] - 2026-09-25
+
+### Added
+
+- Six controlled Airflow demo DAGs, each with 20 seeded runs, bounded live
+  execution waves, and a read-only execution-evidence verifier.
+- Dedicated multi-DAG showcase dashboard with per-pipeline results and separate
+  queued/running activity metrics.
+- FlowSense Engine wordmark in the README and PyPI package description.
+
+### Changed
+
+- Overview dashboard now uses instant summary queries, clearer multi-DAG labels,
+  and a shorter default history window.
+
+### Fixed
+
+- Corrected Airflow 3 GET state filters from `states` to `state`, preventing
+  running runs from shrinking the bounded successful-run analysis window.
+
+## [0.5.0] - 2026-09-23
+
+### Added
+
+- Added a production quickstart and executable bounded batch-analysis example.
+- Added `flowsense doctor` for safe configuration, Airflow API, authentication,
+  and DAG-visibility diagnostics.
+- Added `flowsense dags` for read-only DAG discovery with table and JSON output.
+- Added `flowsense analyze-batch` for bounded multi-DAG CLI analysis with the
+  versioned batch JSON contract.
+- Added severity-based CI exit thresholds to batch analysis.
+- Added full analysis-policy configuration to the batch CLI.
+- Added bounded DAG discovery to batch analysis with `--all-dags`,
+  `--dag-limit`, and optional paused-DAG inclusion.
+- Added CLI access to the versioned batch output JSON Schema through
+  `flowsense schema --document batch`.
+- Added reliable versioned JSON file output to single-DAG analysis through
+  `flowsense analyze --output-file`.
+- Added reusable, versioned JSON analysis policies for single and batch CLI
+  commands, with explicit command-line overrides and JSON Schema export.
+- Added versioned analysis-policy support to the optional Prometheus metrics
+  command and propagated the selected policy through every collection cycle.
+- Added a validated per-cycle history limit to the Prometheus metrics command
+  so Airflow collection remains explicitly bounded.
+- Added expanded Ruff quality rules, a 90% CI coverage gate, and graceful MCP
+  test skipping when the optional dependency is not installed.
+- Reduced Airflow collection overhead with bounded server-side DAG run filters
+  and batched task-instance retrieval, including compatibility fallbacks for
+  deployments that do not support the optimized endpoints.
+- Added explicit drift direction and effective MAD fields to analysis results,
+  plus configurable relative and absolute dispersion floors.
+
+### Changed
+
+- Airflow login tokens are now refreshed once after an unauthorized API
+  response, while static bearer, Basic, and custom authentication lifecycles
+  remain application-owned.
+- Advanced the analysis output schema to version `1.2` for the new drift and
+  policy fields.
+
+### Fixed
+
+- Constant and near-constant baselines no longer classify negligible timing
+  noise as an automatic critical anomaly when raw MAD is zero or extremely
+  small.
+
 ## [0.4.0] - 2026-09-20
 
 ### Added

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/MANIFEST.in

@@ -1,2 +1,3 @@
 include CHANGELOG.md
 recursive-include docs *.md
+recursive-include examples *.py

{flowsense_engine-0.4.0/src/flowsense_engine.egg-info → flowsense_engine-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flowsense-engine
-Version: 0.4.0
+Version: 0.5.1
 Summary: Temporal drift and anomaly detection for Apache Airflow
 Author: Omer Cengiz
 License-Expression: Apache-2.0
@@ -28,6 +28,7 @@ Provides-Extra: dev
 Requires-Dist: build>=1.2; extra == "dev"
 Requires-Dist: pyright>=1.1.400; extra == "dev"
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Provides-Extra: mcp
 Requires-Dist: mcp[cli]>=2.0; extra == "mcp"
@@ -35,12 +36,18 @@ Dynamic: license-file
 
 # FlowSense Engine
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/omercengiz/flowsense-engine/27f7db5e55bfefbba6e355c53c7c84ff34e6f37b/assets/branding/flowsense-engine-logo-v2.png" alt="FlowSense Engine logo" width="640">
+</p>
+
 Temporal drift, anomaly detection, and dependency-aware propagation analysis for Apache Airflow.
 
-Current package release: `0.4.0`. See [CHANGELOG.md](CHANGELOG.md) for release
+Current package release: `0.5.1`. See [CHANGELOG.md](CHANGELOG.md) for release
 notes, [docs/architecture.md](docs/architecture.md) for the system architecture,
 [docs/benchmarks.md](docs/benchmarks.md) for performance measurement, and
-[docs/migrating-to-0.3.md](docs/migrating-to-0.3.md) for migration guidance.
+[docs/production-quickstart.md](docs/production-quickstart.md) for a production
+walkthrough. See [docs/migrating-to-0.5.md](docs/migrating-to-0.5.md) for the
+current migration guidance.
 Maintainers can use [docs/releasing.md](docs/releasing.md) as the release checklist.
 
 FlowSense analyzes historical DAG executions to identify abnormal task behavior and trace how anomalies propagate through downstream dependencies.
@@ -153,6 +160,9 @@ Once published on PyPI, install the distribution with:
 pip install flowsense-engine
 ```
 
+For an end-to-end Airflow setup, follow the
+[production quickstart](docs/production-quickstart.md).
+
 The distribution name is `flowsense-engine`; Python imports and CLI commands
 remain `flowsense`.
 
@@ -218,9 +228,15 @@ export $(grep -v '^#' .env | xargs)
 Then run:
 
 ```bash
+flowsense doctor
+flowsense dags
 flowsense analyze <dag_id>
+flowsense analyze-batch <dag_id> [<dag_id> ...]
 ```
 
+Both single and batch analysis commands support `--fail-on` for CI severity
+gates.
+
 The CLI report includes a DAG summary and separate tables for task drift,
 handoff drift, change points, trends, propagation paths, and diagnostics.
 Results are ordered by severity or subject so repeated analyses remain easy to
@@ -230,14 +246,30 @@ For automation and CI/CD integrations, request the versioned JSON document:
 
 ```bash
 flowsense analyze <dag_id> --output json
+flowsense analyze <dag_id> --output-file flowsense-analysis.json
 ```
 
+`--output-file` always writes the versioned JSON contract and avoids relying on
+shell redirection in CI jobs.
+
 CI jobs can also fail when the analysis reaches a selected severity:
 
 ```bash
 flowsense analyze <dag_id> --output json --fail-on high
 ```
 
+Store reusable analysis settings in a versioned JSON policy file and share it
+between single and batch analysis:
+
+```bash
+flowsense analyze <dag_id> --policy-file flowsense-policy.json
+flowsense analyze-batch --all-dags --dag-limit 10 \
+  --policy-file flowsense-policy.json
+```
+
+Explicit policy options override values loaded from the file. Generate its JSON
+Schema with `flowsense schema --document policy`.
+
 `--fail-on` accepts `medium`, `high`, or `critical`. The report is always
 written before FlowSense exits: code `0` means the severity is below the
 threshold, code `2` means the threshold was reached, and code `1` remains
@@ -265,6 +297,8 @@ flowsense analyze <dag_id> --dag-run-id <dag_run_id>
 
 The MCP tool exposes the same option as `dag_run_id`. Analysis output includes
 `current_dag_run_id`; this field was introduced in output schema version `1.1`.
+Output schema version `1.2` adds drift `direction`, the regularized
+`effective_mad`, and the dispersion-floor policy values.
 
 The JSON document and MCP tool response share the same serialization contract
 and include a `schema_version` field. The serializer is also available from the
@@ -279,8 +313,13 @@ The same schema can be emitted without connecting to Airflow:
 
 ```bash
 flowsense schema > flowsense-analysis.schema.json
+flowsense schema --document batch > flowsense-batch.schema.json
 ```
 
+The default remains the single-DAG analysis schema. Use `--document batch` to
+emit the versioned batch envelope schema, including successful analyses and
+per-DAG failure records.
+
 This output is deterministic and can be used in CI contract checks, editor
 tooling, or client code generation.
 
@@ -332,6 +371,8 @@ policy = AnalysisPolicy(
     medium_threshold=2.5,
     high_threshold=4.0,
     critical_threshold=6.0,
+    minimum_relative_dispersion=0.01,
+    minimum_absolute_dispersion=0.001,
     mapped_task_aggregation=MappedTaskAggregation.MAX,
     change_point_minimum_segment_size=4,
     change_point_score_threshold=4.0,
@@ -344,6 +385,8 @@ policy = AnalysisPolicy(
 `baseline_window` limits the number of historical values used before the current
 run. Mapped task durations can be aggregated with `MAX`, `MEAN`, or `SUM`. The
 same policy options are available through the CLI and MCP tool.
+The dispersion floors prevent constant or nearly constant baselines from
+turning negligible timing noise into an automatic critical anomaly.
 Change-point and trend detection can also be disabled independently with
 `change_point_detection_enabled=False` or `trend_detection_enabled=False`.
 
@@ -383,7 +426,11 @@ information for a DAG.
 Run continuous analysis and expose the latest DAG and bounded task-level metrics:
 
 ```bash
-flowsense serve-metrics example_dag --host 0.0.0.0 --port 9108
+flowsense serve-metrics example_dag \
+  --host 0.0.0.0 \
+  --port 9108 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
 See [docs/prometheus.md](docs/prometheus.md) for the metric contract,

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/README.md

@@ -1,11 +1,17 @@
 # FlowSense Engine
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/omercengiz/flowsense-engine/27f7db5e55bfefbba6e355c53c7c84ff34e6f37b/assets/branding/flowsense-engine-logo-v2.png" alt="FlowSense Engine logo" width="640">
+</p>
+
 Temporal drift, anomaly detection, and dependency-aware propagation analysis for Apache Airflow.
 
-Current package release: `0.4.0`. See [CHANGELOG.md](CHANGELOG.md) for release
+Current package release: `0.5.1`. See [CHANGELOG.md](CHANGELOG.md) for release
 notes, [docs/architecture.md](docs/architecture.md) for the system architecture,
 [docs/benchmarks.md](docs/benchmarks.md) for performance measurement, and
-[docs/migrating-to-0.3.md](docs/migrating-to-0.3.md) for migration guidance.
+[docs/production-quickstart.md](docs/production-quickstart.md) for a production
+walkthrough. See [docs/migrating-to-0.5.md](docs/migrating-to-0.5.md) for the
+current migration guidance.
 Maintainers can use [docs/releasing.md](docs/releasing.md) as the release checklist.
 
 FlowSense analyzes historical DAG executions to identify abnormal task behavior and trace how anomalies propagate through downstream dependencies.
@@ -118,6 +124,9 @@ Once published on PyPI, install the distribution with:
 pip install flowsense-engine
 ```
 
+For an end-to-end Airflow setup, follow the
+[production quickstart](docs/production-quickstart.md).
+
 The distribution name is `flowsense-engine`; Python imports and CLI commands
 remain `flowsense`.
 
@@ -183,9 +192,15 @@ export $(grep -v '^#' .env | xargs)
 Then run:
 
 ```bash
+flowsense doctor
+flowsense dags
 flowsense analyze <dag_id>
+flowsense analyze-batch <dag_id> [<dag_id> ...]
 ```
 
+Both single and batch analysis commands support `--fail-on` for CI severity
+gates.
+
 The CLI report includes a DAG summary and separate tables for task drift,
 handoff drift, change points, trends, propagation paths, and diagnostics.
 Results are ordered by severity or subject so repeated analyses remain easy to
@@ -195,14 +210,30 @@ For automation and CI/CD integrations, request the versioned JSON document:
 
 ```bash
 flowsense analyze <dag_id> --output json
+flowsense analyze <dag_id> --output-file flowsense-analysis.json
 ```
 
+`--output-file` always writes the versioned JSON contract and avoids relying on
+shell redirection in CI jobs.
+
 CI jobs can also fail when the analysis reaches a selected severity:
 
 ```bash
 flowsense analyze <dag_id> --output json --fail-on high
 ```
 
+Store reusable analysis settings in a versioned JSON policy file and share it
+between single and batch analysis:
+
+```bash
+flowsense analyze <dag_id> --policy-file flowsense-policy.json
+flowsense analyze-batch --all-dags --dag-limit 10 \
+  --policy-file flowsense-policy.json
+```
+
+Explicit policy options override values loaded from the file. Generate its JSON
+Schema with `flowsense schema --document policy`.
+
 `--fail-on` accepts `medium`, `high`, or `critical`. The report is always
 written before FlowSense exits: code `0` means the severity is below the
 threshold, code `2` means the threshold was reached, and code `1` remains
@@ -230,6 +261,8 @@ flowsense analyze <dag_id> --dag-run-id <dag_run_id>
 
 The MCP tool exposes the same option as `dag_run_id`. Analysis output includes
 `current_dag_run_id`; this field was introduced in output schema version `1.1`.
+Output schema version `1.2` adds drift `direction`, the regularized
+`effective_mad`, and the dispersion-floor policy values.
 
 The JSON document and MCP tool response share the same serialization contract
 and include a `schema_version` field. The serializer is also available from the
@@ -244,8 +277,13 @@ The same schema can be emitted without connecting to Airflow:
 
 ```bash
 flowsense schema > flowsense-analysis.schema.json
+flowsense schema --document batch > flowsense-batch.schema.json
 ```
 
+The default remains the single-DAG analysis schema. Use `--document batch` to
+emit the versioned batch envelope schema, including successful analyses and
+per-DAG failure records.
+
 This output is deterministic and can be used in CI contract checks, editor
 tooling, or client code generation.
 
@@ -297,6 +335,8 @@ policy = AnalysisPolicy(
     medium_threshold=2.5,
     high_threshold=4.0,
     critical_threshold=6.0,
+    minimum_relative_dispersion=0.01,
+    minimum_absolute_dispersion=0.001,
     mapped_task_aggregation=MappedTaskAggregation.MAX,
     change_point_minimum_segment_size=4,
     change_point_score_threshold=4.0,
@@ -309,6 +349,8 @@ policy = AnalysisPolicy(
 `baseline_window` limits the number of historical values used before the current
 run. Mapped task durations can be aggregated with `MAX`, `MEAN`, or `SUM`. The
 same policy options are available through the CLI and MCP tool.
+The dispersion floors prevent constant or nearly constant baselines from
+turning negligible timing noise into an automatic critical anomaly.
 Change-point and trend detection can also be disabled independently with
 `change_point_detection_enabled=False` or `trend_detection_enabled=False`.
 
@@ -348,7 +390,11 @@ information for a DAG.
 Run continuous analysis and expose the latest DAG and bounded task-level metrics:
 
 ```bash
-flowsense serve-metrics example_dag --host 0.0.0.0 --port 9108
+flowsense serve-metrics example_dag \
+  --host 0.0.0.0 \
+  --port 9108 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
 See [docs/prometheus.md](docs/prometheus.md) for the metric contract,

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/docs/architecture.md

@@ -81,6 +81,14 @@ Authentication is supplied through `AirflowAuthProvider`. Built-in providers
 cover Basic Auth, Airflow login-token exchange, static bearer tokens, and custom
 headers without coupling request collection to a deployment's identity system.
 
+Collection is bounded at the Airflow API boundary whenever the configured API
+supports it. Airflow 2 uses the stable batch POST endpoints, while Airflow 3
+uses filtered collection endpoints for successful DAG runs and task instances.
+Unsupported filter or batch responses fall back to the paginated per-run API,
+preserving compatibility with older Airflow 2 deployments. This optimization
+belongs exclusively to the infrastructure adapter and does not leak transport
+capabilities into application or domain contracts.
+
 ### Delivery adapters
 
 `flowsense.cli`, `flowsense.mcp`, and `flowsense.observability` translate user input into an

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/docs/authentication.md

@@ -16,8 +16,11 @@ AIRFLOW_USERNAME=airflow
 AIRFLOW_PASSWORD=secret
 ```
 
-FlowSense exchanges the credentials once and caches the returned bearer token
-for the lifetime of the client.
+FlowSense exchanges the credentials and caches the returned bearer token for
+the lifetime of the client. If an Airflow API request returns `401`, FlowSense
+invalidates a login-issued token, obtains a fresh token, and retries that API
+request once. A second `401` is returned as `AirflowApiError`; it is never
+retried indefinitely.
 
 ### Basic authentication
 
@@ -42,6 +45,9 @@ AIRFLOW_BEARER_TOKEN=secret-token
 Username and password are not required in bearer mode. Secrets are excluded
 from the `AirflowConfig` representation, but applications must still keep them
 out of logs, shell history, source control, and command-line arguments.
+Static bearer tokens are not automatically replaced after `401`; their
+lifecycle remains owned by the application or identity provider that issued
+them.
 
 ## Custom provider
 
@@ -74,4 +80,6 @@ provider = BearerTokenAuthProvider(load_current_token)
 
 Custom providers should return only authentication material and leave request
 execution, retry, timeout, payload validation, and error translation to
-`AirflowClient`.
+`AirflowClient`. A `401` from a custom provider is not interpreted as permission
+to rotate external credentials; the owning application decides how to refresh
+them.

{flowsense_engine-0.4.0 → flowsense_engine-0.5.1}/docs/grafana.md

@@ -12,9 +12,13 @@ Configure the required `AIRFLOW_*` environment variables, then run:
 flowsense serve-metrics example_dag another_dag \
   --host 0.0.0.0 \
   --port 9108 \
-  --interval-seconds 60
+  --interval-seconds 60 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
+Omit `--policy-file` to use the default analysis policy.
+
 Binding to `0.0.0.0` makes the endpoint reachable from the local Prometheus
 container. Do not expose this port to untrusted networks without an appropriate
 network policy or reverse proxy.
@@ -52,7 +56,21 @@ The dashboard provides:
 - currently firing FlowSense alerts.
 
 Use the DAG and task variables at the top of the dashboard to narrow the view.
-The default time range is six hours and the dashboard refreshes every 30 seconds.
+The default time range is 15 minutes and the dashboard refreshes every 30 seconds.
+Summary cards use instant queries, so they remain populated independently of
+the selected history window. Current task comparisons include both DAG and
+task names to distinguish tasks across multiple DAGs.
+
+The top row summarizes the selected DAGs: monitored count, highest severity,
+analysis health (healthy only when every selected DAG succeeded), mean coverage,
+and total anomalous and affected tasks. Task comparisons and history appear
+below; collection diagnostics and active alerts follow.
+
+Prometheus history starts when the metrics service is scraped; historical
+Airflow runs are analyzed but are not backfilled into these charts. After
+starting the exporter, use a recent time range and allow a few scrapes. An
+existing browser URL can retain an older time range; select Last 15 minutes
+manually in that case.
 See [alerting.md](alerting.md) for rule behavior, validation, and production
 notification configuration.
 
@@ -64,3 +82,8 @@ docker compose -f deploy/observability/docker-compose.yml down
 
 Prometheus and Grafana data remain in named volumes. To explicitly remove those
 local volumes, add `--volumes` to the command.
+
+## Multi-DAG demo with execution evidence
+
+For six real Airflow workloads, 20 seeded runs per DAG, continued live activity,
+and a dedicated presentation dashboard, see [showcase.md](showcase.md).

flowsense_engine-0.5.1/docs/migrating-to-0.5.md

@@ -0,0 +1,63 @@
+# Migrating to FlowSense 0.5
+
+FlowSense 0.5 improves production-scale Airflow collection, authentication
+resilience, and statistical drift accuracy. Existing Python, CLI, and MCP entry
+points remain available, but JSON consumers must account for analysis output
+schema version `1.2`.
+
+## Analysis output schema 1.2
+
+Each task and handoff drift result now includes:
+
+- `direction`: `INCREASE`, `DECREASE`, or `UNCHANGED`;
+- `effective_mad`: the regularized dispersion used to calculate the robust
+  z-score.
+
+The embedded policy also includes `minimum_relative_dispersion` and
+`minimum_absolute_dispersion`. Consumers that validate JSON should regenerate
+their model or schema with:
+
+```bash
+flowsense schema --document analysis
+```
+
+The batch envelope remains at schema version `1.0`; each successful analysis
+inside it now uses schema version `1.2`.
+
+## Drift behavior
+
+Raw MAD is still reported without modification. Score calculation now uses the
+largest of raw MAD, the configured relative floor, and the configured absolute
+floor. This prevents negligible timing noise on constant baselines from being
+classified as an automatic critical anomaly.
+
+The defaults are:
+
+```json
+{
+  "minimum_relative_dispersion": 0.01,
+  "minimum_absolute_dispersion": 0.001
+}
+```
+
+Pin explicit values in a policy file if existing alert thresholds were tuned
+around the previous zero-MAD behavior.
+
+## Airflow collection and authentication
+
+Recent successful DAG runs and task instances use bounded or batch API calls
+when supported. Older Airflow deployments automatically fall back to the
+paginated per-run endpoints.
+
+Airflow login tokens obtained through `AIRFLOW_AUTH_MODE=token` are refreshed
+once after a `401` response. Basic, static bearer, and custom authentication
+providers retain application-owned credential lifecycles.
+
+## Validation checklist
+
+1. Regenerate and compare the analysis JSON Schema.
+2. Validate alerting against representative historical DAG runs.
+3. Confirm the configured Airflow identity can use batch endpoints or fallback
+   endpoints.
+4. Run `flowsense doctor` against the target Airflow environment.
+5. Perform a canary analysis before replacing a pinned 0.4 installation.