flowsense-engine 0.4.0__tar.gz → 0.5.0__tar.gz

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/CHANGELOG.md

@@ -5,6 +5,58 @@ All notable changes to FlowSense are documented in this file. The project uses
 
 ## [Unreleased]
 
+### Added
+
+### Changed
+
+### Fixed
+
+## [0.5.0] - 2026-09-23
+
+### Added
+
+- Added a production quickstart and executable bounded batch-analysis example.
+- Added `flowsense doctor` for safe configuration, Airflow API, authentication,
+  and DAG-visibility diagnostics.
+- Added `flowsense dags` for read-only DAG discovery with table and JSON output.
+- Added `flowsense analyze-batch` for bounded multi-DAG CLI analysis with the
+  versioned batch JSON contract.
+- Added severity-based CI exit thresholds to batch analysis.
+- Added full analysis-policy configuration to the batch CLI.
+- Added bounded DAG discovery to batch analysis with `--all-dags`,
+  `--dag-limit`, and optional paused-DAG inclusion.
+- Added CLI access to the versioned batch output JSON Schema through
+  `flowsense schema --document batch`.
+- Added reliable versioned JSON file output to single-DAG analysis through
+  `flowsense analyze --output-file`.
+- Added reusable, versioned JSON analysis policies for single and batch CLI
+  commands, with explicit command-line overrides and JSON Schema export.
+- Added versioned analysis-policy support to the optional Prometheus metrics
+  command and propagated the selected policy through every collection cycle.
+- Added a validated per-cycle history limit to the Prometheus metrics command
+  so Airflow collection remains explicitly bounded.
+- Added expanded Ruff quality rules, a 90% CI coverage gate, and graceful MCP
+  test skipping when the optional dependency is not installed.
+- Reduced Airflow collection overhead with bounded server-side DAG run filters
+  and batched task-instance retrieval, including compatibility fallbacks for
+  deployments that do not support the optimized endpoints.
+- Added explicit drift direction and effective MAD fields to analysis results,
+  plus configurable relative and absolute dispersion floors.
+
+### Changed
+
+- Airflow login tokens are now refreshed once after an unauthorized API
+  response, while static bearer, Basic, and custom authentication lifecycles
+  remain application-owned.
+- Advanced the analysis output schema to version `1.2` for the new drift and
+  policy fields.
+
+### Fixed
+
+- Constant and near-constant baselines no longer classify negligible timing
+  noise as an automatic critical anomaly when raw MAD is zero or extremely
+  small.
+
 ## [0.4.0] - 2026-09-20
 
 ### Added

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/MANIFEST.in

@@ -1,2 +1,3 @@
 include CHANGELOG.md
 recursive-include docs *.md
+recursive-include examples *.py

{flowsense_engine-0.4.0/src/flowsense_engine.egg-info → flowsense_engine-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flowsense-engine
-Version: 0.4.0
+Version: 0.5.0
 Summary: Temporal drift and anomaly detection for Apache Airflow
 Author: Omer Cengiz
 License-Expression: Apache-2.0
@@ -28,6 +28,7 @@ Provides-Extra: dev
 Requires-Dist: build>=1.2; extra == "dev"
 Requires-Dist: pyright>=1.1.400; extra == "dev"
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Provides-Extra: mcp
 Requires-Dist: mcp[cli]>=2.0; extra == "mcp"
@@ -37,10 +38,12 @@ Dynamic: license-file
 
 Temporal drift, anomaly detection, and dependency-aware propagation analysis for Apache Airflow.
 
-Current package release: `0.4.0`. See [CHANGELOG.md](CHANGELOG.md) for release
+Current package release: `0.5.0`. See [CHANGELOG.md](CHANGELOG.md) for release
 notes, [docs/architecture.md](docs/architecture.md) for the system architecture,
 [docs/benchmarks.md](docs/benchmarks.md) for performance measurement, and
-[docs/migrating-to-0.3.md](docs/migrating-to-0.3.md) for migration guidance.
+[docs/production-quickstart.md](docs/production-quickstart.md) for a production
+walkthrough. See [docs/migrating-to-0.5.md](docs/migrating-to-0.5.md) for the
+current migration guidance.
 Maintainers can use [docs/releasing.md](docs/releasing.md) as the release checklist.
 
 FlowSense analyzes historical DAG executions to identify abnormal task behavior and trace how anomalies propagate through downstream dependencies.
@@ -153,6 +156,9 @@ Once published on PyPI, install the distribution with:
 pip install flowsense-engine
 ```
 
+For an end-to-end Airflow setup, follow the
+[production quickstart](docs/production-quickstart.md).
+
 The distribution name is `flowsense-engine`; Python imports and CLI commands
 remain `flowsense`.
 
@@ -218,9 +224,15 @@ export $(grep -v '^#' .env | xargs)
 Then run:
 
 ```bash
+flowsense doctor
+flowsense dags
 flowsense analyze <dag_id>
+flowsense analyze-batch <dag_id> [<dag_id> ...]
 ```
 
+Both single and batch analysis commands support `--fail-on` for CI severity
+gates.
+
 The CLI report includes a DAG summary and separate tables for task drift,
 handoff drift, change points, trends, propagation paths, and diagnostics.
 Results are ordered by severity or subject so repeated analyses remain easy to
@@ -230,14 +242,30 @@ For automation and CI/CD integrations, request the versioned JSON document:
 
 ```bash
 flowsense analyze <dag_id> --output json
+flowsense analyze <dag_id> --output-file flowsense-analysis.json
 ```
 
+`--output-file` always writes the versioned JSON contract and avoids relying on
+shell redirection in CI jobs.
+
 CI jobs can also fail when the analysis reaches a selected severity:
 
 ```bash
 flowsense analyze <dag_id> --output json --fail-on high
 ```
 
+Store reusable analysis settings in a versioned JSON policy file and share it
+between single and batch analysis:
+
+```bash
+flowsense analyze <dag_id> --policy-file flowsense-policy.json
+flowsense analyze-batch --all-dags --dag-limit 10 \
+  --policy-file flowsense-policy.json
+```
+
+Explicit policy options override values loaded from the file. Generate its JSON
+Schema with `flowsense schema --document policy`.
+
 `--fail-on` accepts `medium`, `high`, or `critical`. The report is always
 written before FlowSense exits: code `0` means the severity is below the
 threshold, code `2` means the threshold was reached, and code `1` remains
@@ -265,6 +293,8 @@ flowsense analyze <dag_id> --dag-run-id <dag_run_id>
 
 The MCP tool exposes the same option as `dag_run_id`. Analysis output includes
 `current_dag_run_id`; this field was introduced in output schema version `1.1`.
+Output schema version `1.2` adds drift `direction`, the regularized
+`effective_mad`, and the dispersion-floor policy values.
 
 The JSON document and MCP tool response share the same serialization contract
 and include a `schema_version` field. The serializer is also available from the
@@ -279,8 +309,13 @@ The same schema can be emitted without connecting to Airflow:
 
 ```bash
 flowsense schema > flowsense-analysis.schema.json
+flowsense schema --document batch > flowsense-batch.schema.json
 ```
 
+The default remains the single-DAG analysis schema. Use `--document batch` to
+emit the versioned batch envelope schema, including successful analyses and
+per-DAG failure records.
+
 This output is deterministic and can be used in CI contract checks, editor
 tooling, or client code generation.
 
@@ -332,6 +367,8 @@ policy = AnalysisPolicy(
     medium_threshold=2.5,
     high_threshold=4.0,
     critical_threshold=6.0,
+    minimum_relative_dispersion=0.01,
+    minimum_absolute_dispersion=0.001,
     mapped_task_aggregation=MappedTaskAggregation.MAX,
     change_point_minimum_segment_size=4,
     change_point_score_threshold=4.0,
@@ -344,6 +381,8 @@ policy = AnalysisPolicy(
 `baseline_window` limits the number of historical values used before the current
 run. Mapped task durations can be aggregated with `MAX`, `MEAN`, or `SUM`. The
 same policy options are available through the CLI and MCP tool.
+The dispersion floors prevent constant or nearly constant baselines from
+turning negligible timing noise into an automatic critical anomaly.
 Change-point and trend detection can also be disabled independently with
 `change_point_detection_enabled=False` or `trend_detection_enabled=False`.
 
@@ -383,7 +422,11 @@ information for a DAG.
 Run continuous analysis and expose the latest DAG and bounded task-level metrics:
 
 ```bash
-flowsense serve-metrics example_dag --host 0.0.0.0 --port 9108
+flowsense serve-metrics example_dag \
+  --host 0.0.0.0 \
+  --port 9108 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
 See [docs/prometheus.md](docs/prometheus.md) for the metric contract,

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/README.md

@@ -2,10 +2,12 @@
 
 Temporal drift, anomaly detection, and dependency-aware propagation analysis for Apache Airflow.
 
-Current package release: `0.4.0`. See [CHANGELOG.md](CHANGELOG.md) for release
+Current package release: `0.5.0`. See [CHANGELOG.md](CHANGELOG.md) for release
 notes, [docs/architecture.md](docs/architecture.md) for the system architecture,
 [docs/benchmarks.md](docs/benchmarks.md) for performance measurement, and
-[docs/migrating-to-0.3.md](docs/migrating-to-0.3.md) for migration guidance.
+[docs/production-quickstart.md](docs/production-quickstart.md) for a production
+walkthrough. See [docs/migrating-to-0.5.md](docs/migrating-to-0.5.md) for the
+current migration guidance.
 Maintainers can use [docs/releasing.md](docs/releasing.md) as the release checklist.
 
 FlowSense analyzes historical DAG executions to identify abnormal task behavior and trace how anomalies propagate through downstream dependencies.
@@ -118,6 +120,9 @@ Once published on PyPI, install the distribution with:
 pip install flowsense-engine
 ```
 
+For an end-to-end Airflow setup, follow the
+[production quickstart](docs/production-quickstart.md).
+
 The distribution name is `flowsense-engine`; Python imports and CLI commands
 remain `flowsense`.
 
@@ -183,9 +188,15 @@ export $(grep -v '^#' .env | xargs)
 Then run:
 
 ```bash
+flowsense doctor
+flowsense dags
 flowsense analyze <dag_id>
+flowsense analyze-batch <dag_id> [<dag_id> ...]
 ```
 
+Both single and batch analysis commands support `--fail-on` for CI severity
+gates.
+
 The CLI report includes a DAG summary and separate tables for task drift,
 handoff drift, change points, trends, propagation paths, and diagnostics.
 Results are ordered by severity or subject so repeated analyses remain easy to
@@ -195,14 +206,30 @@ For automation and CI/CD integrations, request the versioned JSON document:
 
 ```bash
 flowsense analyze <dag_id> --output json
+flowsense analyze <dag_id> --output-file flowsense-analysis.json
 ```
 
+`--output-file` always writes the versioned JSON contract and avoids relying on
+shell redirection in CI jobs.
+
 CI jobs can also fail when the analysis reaches a selected severity:
 
 ```bash
 flowsense analyze <dag_id> --output json --fail-on high
 ```
 
+Store reusable analysis settings in a versioned JSON policy file and share it
+between single and batch analysis:
+
+```bash
+flowsense analyze <dag_id> --policy-file flowsense-policy.json
+flowsense analyze-batch --all-dags --dag-limit 10 \
+  --policy-file flowsense-policy.json
+```
+
+Explicit policy options override values loaded from the file. Generate its JSON
+Schema with `flowsense schema --document policy`.
+
 `--fail-on` accepts `medium`, `high`, or `critical`. The report is always
 written before FlowSense exits: code `0` means the severity is below the
 threshold, code `2` means the threshold was reached, and code `1` remains
@@ -230,6 +257,8 @@ flowsense analyze <dag_id> --dag-run-id <dag_run_id>
 
 The MCP tool exposes the same option as `dag_run_id`. Analysis output includes
 `current_dag_run_id`; this field was introduced in output schema version `1.1`.
+Output schema version `1.2` adds drift `direction`, the regularized
+`effective_mad`, and the dispersion-floor policy values.
 
 The JSON document and MCP tool response share the same serialization contract
 and include a `schema_version` field. The serializer is also available from the
@@ -244,8 +273,13 @@ The same schema can be emitted without connecting to Airflow:
 
 ```bash
 flowsense schema > flowsense-analysis.schema.json
+flowsense schema --document batch > flowsense-batch.schema.json
 ```
 
+The default remains the single-DAG analysis schema. Use `--document batch` to
+emit the versioned batch envelope schema, including successful analyses and
+per-DAG failure records.
+
 This output is deterministic and can be used in CI contract checks, editor
 tooling, or client code generation.
 
@@ -297,6 +331,8 @@ policy = AnalysisPolicy(
     medium_threshold=2.5,
     high_threshold=4.0,
     critical_threshold=6.0,
+    minimum_relative_dispersion=0.01,
+    minimum_absolute_dispersion=0.001,
     mapped_task_aggregation=MappedTaskAggregation.MAX,
     change_point_minimum_segment_size=4,
     change_point_score_threshold=4.0,
@@ -309,6 +345,8 @@ policy = AnalysisPolicy(
 `baseline_window` limits the number of historical values used before the current
 run. Mapped task durations can be aggregated with `MAX`, `MEAN`, or `SUM`. The
 same policy options are available through the CLI and MCP tool.
+The dispersion floors prevent constant or nearly constant baselines from
+turning negligible timing noise into an automatic critical anomaly.
 Change-point and trend detection can also be disabled independently with
 `change_point_detection_enabled=False` or `trend_detection_enabled=False`.
 
@@ -348,7 +386,11 @@ information for a DAG.
 Run continuous analysis and expose the latest DAG and bounded task-level metrics:
 
 ```bash
-flowsense serve-metrics example_dag --host 0.0.0.0 --port 9108
+flowsense serve-metrics example_dag \
+  --host 0.0.0.0 \
+  --port 9108 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
 See [docs/prometheus.md](docs/prometheus.md) for the metric contract,

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/docs/architecture.md

@@ -81,6 +81,14 @@ Authentication is supplied through `AirflowAuthProvider`. Built-in providers
 cover Basic Auth, Airflow login-token exchange, static bearer tokens, and custom
 headers without coupling request collection to a deployment's identity system.
 
+Collection is bounded at the Airflow API boundary whenever the configured API
+supports it. Airflow 2 uses the stable batch POST endpoints, while Airflow 3
+uses filtered collection endpoints for successful DAG runs and task instances.
+Unsupported filter or batch responses fall back to the paginated per-run API,
+preserving compatibility with older Airflow 2 deployments. This optimization
+belongs exclusively to the infrastructure adapter and does not leak transport
+capabilities into application or domain contracts.
+
 ### Delivery adapters
 
 `flowsense.cli`, `flowsense.mcp`, and `flowsense.observability` translate user input into an

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/docs/authentication.md

@@ -16,8 +16,11 @@ AIRFLOW_USERNAME=airflow
 AIRFLOW_PASSWORD=secret
 ```
 
-FlowSense exchanges the credentials once and caches the returned bearer token
-for the lifetime of the client.
+FlowSense exchanges the credentials and caches the returned bearer token for
+the lifetime of the client. If an Airflow API request returns `401`, FlowSense
+invalidates a login-issued token, obtains a fresh token, and retries that API
+request once. A second `401` is returned as `AirflowApiError`; it is never
+retried indefinitely.
 
 ### Basic authentication
 
@@ -42,6 +45,9 @@ AIRFLOW_BEARER_TOKEN=secret-token
 Username and password are not required in bearer mode. Secrets are excluded
 from the `AirflowConfig` representation, but applications must still keep them
 out of logs, shell history, source control, and command-line arguments.
+Static bearer tokens are not automatically replaced after `401`; their
+lifecycle remains owned by the application or identity provider that issued
+them.
 
 ## Custom provider
 
@@ -74,4 +80,6 @@ provider = BearerTokenAuthProvider(load_current_token)
 
 Custom providers should return only authentication material and leave request
 execution, retry, timeout, payload validation, and error translation to
-`AirflowClient`.
+`AirflowClient`. A `401` from a custom provider is not interpreted as permission
+to rotate external credentials; the owning application decides how to refresh
+them.

{flowsense_engine-0.4.0 → flowsense_engine-0.5.0}/docs/grafana.md

@@ -12,9 +12,13 @@ Configure the required `AIRFLOW_*` environment variables, then run:
 flowsense serve-metrics example_dag another_dag \
   --host 0.0.0.0 \
   --port 9108 \
-  --interval-seconds 60
+  --interval-seconds 60 \
+  --history-run-limit 50 \
+  --policy-file flowsense-policy.json
 ```
 
+Omit `--policy-file` to use the default analysis policy.
+
 Binding to `0.0.0.0` makes the endpoint reachable from the local Prometheus
 container. Do not expose this port to untrusted networks without an appropriate
 network policy or reverse proxy.

flowsense_engine-0.5.0/docs/migrating-to-0.5.md

@@ -0,0 +1,63 @@
+# Migrating to FlowSense 0.5
+
+FlowSense 0.5 improves production-scale Airflow collection, authentication
+resilience, and statistical drift accuracy. Existing Python, CLI, and MCP entry
+points remain available, but JSON consumers must account for analysis output
+schema version `1.2`.
+
+## Analysis output schema 1.2
+
+Each task and handoff drift result now includes:
+
+- `direction`: `INCREASE`, `DECREASE`, or `UNCHANGED`;
+- `effective_mad`: the regularized dispersion used to calculate the robust
+  z-score.
+
+The embedded policy also includes `minimum_relative_dispersion` and
+`minimum_absolute_dispersion`. Consumers that validate JSON should regenerate
+their model or schema with:
+
+```bash
+flowsense schema --document analysis
+```
+
+The batch envelope remains at schema version `1.0`; each successful analysis
+inside it now uses schema version `1.2`.
+
+## Drift behavior
+
+Raw MAD is still reported without modification. Score calculation now uses the
+largest of raw MAD, the configured relative floor, and the configured absolute
+floor. This prevents negligible timing noise on constant baselines from being
+classified as an automatic critical anomaly.
+
+The defaults are:
+
+```json
+{
+  "minimum_relative_dispersion": 0.01,
+  "minimum_absolute_dispersion": 0.001
+}
+```
+
+Pin explicit values in a policy file if existing alert thresholds were tuned
+around the previous zero-MAD behavior.
+
+## Airflow collection and authentication
+
+Recent successful DAG runs and task instances use bounded or batch API calls
+when supported. Older Airflow deployments automatically fall back to the
+paginated per-run endpoints.
+
+Airflow login tokens obtained through `AIRFLOW_AUTH_MODE=token` are refreshed
+once after a `401` response. Basic, static bearer, and custom authentication
+providers retain application-owned credential lifecycles.
+
+## Validation checklist
+
+1. Regenerate and compare the analysis JSON Schema.
+2. Validate alerting against representative historical DAG runs.
+3. Confirm the configured Airflow identity can use batch endpoints or fallback
+   endpoints.
+4. Run `flowsense doctor` against the target Airflow environment.
+5. Perform a canary analysis before replacing a pinned 0.4 installation.

flowsense_engine-0.5.0/docs/production-quickstart.md

@@ -0,0 +1,235 @@
+# Production quickstart
+
+This guide runs FlowSense as an installed Python package against an existing
+Apache Airflow deployment. FlowSense remains an in-process library and CLI; it
+does not require a separate hosted service or database.
+
+## 1. Install an isolated version
+
+FlowSense requires Python 3.12 or 3.13.
+
+```bash
+python3.12 -m venv .venv
+source .venv/bin/activate
+python -m pip install "flowsense-engine==0.5.0"
+flowsense --version
+```
+
+The expected version is `0.5.0`. Pin the version in production dependency files
+and upgrade deliberately after reviewing the changelog.
+
+## 2. Configure Airflow access
+
+Keep credentials in a secret manager or injected environment variables rather
+than source control.
+
+For an Airflow 3 deployment using login-token exchange:
+
+```bash
+export AIRFLOW_BASE_URL="https://airflow.example.com"
+export AIRFLOW_API_VERSION="v2"
+export AIRFLOW_AUTH_MODE="token"
+export AIRFLOW_USERNAME="flowsense"
+export AIRFLOW_PASSWORD="replace-from-secret-manager"
+```
+
+For an Airflow 2 deployment using Basic Auth:
+
+```bash
+export AIRFLOW_BASE_URL="https://airflow.example.com"
+export AIRFLOW_API_VERSION="v1"
+export AIRFLOW_AUTH_MODE="basic"
+export AIRFLOW_USERNAME="flowsense"
+export AIRFLOW_PASSWORD="replace-from-secret-manager"
+```
+
+Static bearer tokens use `AIRFLOW_AUTH_MODE=bearer` and
+`AIRFLOW_BEARER_TOKEN`. Custom identity gateways can use the Python
+authentication provider API described in [authentication.md](authentication.md).
+
+Use a read-only Airflow identity with access to DAG definitions, DAG runs, and
+task instances. Configure conservative network and history bounds:
+
+```bash
+export AIRFLOW_CONNECT_TIMEOUT="10"
+export AIRFLOW_READ_TIMEOUT="30"
+export AIRFLOW_MAX_RETRIES="2"
+export AIRFLOW_RETRY_BACKOFF="0.5"
+export AIRFLOW_HISTORY_RUN_LIMIT="100"
+```
+
+## 3. Run a single-DAG smoke test
+
+Validate configuration, authentication, API routing, and DAG visibility without
+running an analysis:
+
+```bash
+flowsense doctor
+```
+
+For automation, `flowsense doctor --output json` emits structured checks and
+returns exit code `1` when a required check fails.
+
+List the active DAGs visible to the configured identity:
+
+```bash
+flowsense dags
+flowsense dags --output json
+```
+
+Paused DAGs are excluded unless `--include-paused` is provided.
+
+Analyze an explicit, bounded DAG set directly from the CLI:
+
+```bash
+flowsense analyze-batch orders payments inventory \
+  --history-run-limit 50 \
+  --max-concurrency 3 \
+  --fail-on high \
+  --output-file flowsense-batch.json
+```
+
+The command always writes the versioned batch JSON when analysis completes. It
+returns exit code `1` if any DAG has an expected failure, while preserving all
+successful results and failure records in the output file.
+
+Batch analysis accepts the same policy controls as single-DAG analysis,
+including history requirements, severity thresholds, mapped-task aggregation,
+and change-point or trend detector settings.
+
+For repeatable analysis across environments, create `flowsense-policy.json`:
+
+```json
+{
+  "schema_version": "1.0",
+  "minimum_history": 10,
+  "baseline_window": 30,
+  "medium_threshold": 2.0,
+  "high_threshold": 3.5,
+  "critical_threshold": 5.0,
+  "minimum_relative_dispersion": 0.01,
+  "minimum_absolute_dispersion": 0.001,
+  "mapped_task_aggregation": "MAX"
+}
+```
+
+Use the same file with either command:
+
+```bash
+flowsense analyze orders --policy-file flowsense-policy.json
+flowsense analyze-batch orders payments \
+  --policy-file flowsense-policy.json \
+  --minimum-history 15
+```
+
+Explicit CLI policy options override file values. Unknown fields, unsupported
+schema versions, malformed JSON, and invalid domain settings fail before an
+Airflow analysis begins. Export the policy schema with
+`flowsense schema --document policy`.
+
+To discover active DAGs automatically, keep the workload explicitly bounded:
+
+```bash
+flowsense analyze-batch \
+  --all-dags \
+  --dag-limit 10 \
+  --history-run-limit 50 \
+  --max-concurrency 3 \
+  --output-file flowsense-batch.json
+```
+
+Discovery excludes paused DAGs by default. Add `--include-paused` only when
+paused workflows are intentionally part of the analysis scope. Explicit DAG
+ids and `--all-dags` are mutually exclusive.
+
+With `--fail-on`, exit code `2` means at least one successful DAG analysis
+reached the selected severity. Operational DAG failures take precedence and
+retain exit code `1`.
+
+Export the matching JSON Schema for validation in downstream automation:
+
+```bash
+flowsense schema --document batch > flowsense-batch.schema.json
+```
+
+Start with a DAG that has several successful historical runs:
+
+```bash
+flowsense analyze example_dag --output json
+```
+
+Exit code `0` means analysis completed. Exit code `1` represents a configuration,
+Airflow, or analysis failure. When `--fail-on` is used, exit code `2` means the
+selected severity threshold was reached.
+
+For CI:
+
+```bash
+flowsense analyze example_dag \
+  --output-file analysis.json \
+  --fail-on high
+```
+
+## 4. Discover and analyze several DAGs
+
+The repository includes a bounded production example:
+
+```bash
+python examples/python/production_batch.py \
+  --dag-limit 10 \
+  --history-run-limit 50 \
+  --max-concurrency 3 \
+  --output flowsense-batch.json
+```
+
+The example:
+
+1. loads validated Airflow configuration;
+2. discovers active DAGs through the paginated Airflow API;
+3. limits the number of selected DAGs;
+4. runs independent analyses with bounded concurrency;
+5. writes the versioned batch output contract.
+
+Paused DAGs are excluded unless `--include-paused` is supplied. The command
+returns exit code `1` when one or more expected DAG-level failures are present,
+while still writing successful results and failure records.
+
+## 5. Size the workload
+
+Each DAG analysis reads DAG runs, task instances for the selected successful
+runs, and the current DAG structure. Start with:
+
+- `--dag-limit 5` or `10`;
+- `--history-run-limit 30` to `100`;
+- `--max-concurrency 1` to `3`.
+
+Increase these values only after observing Airflow API latency and rate limits.
+FlowSense retries transient `429`, `502`, `503`, and `504` responses, but
+concurrency should remain below the capacity of the Airflow webserver.
+
+## 6. Operate safely
+
+- Pin the FlowSense package version.
+- Inject secrets at runtime and never log them.
+- Use a read-only Airflow account.
+- Retain the output `schema_version` fields with stored JSON.
+- Treat batch `failures` separately from anomaly severity.
+- Alert when analysis stops succeeding, not only when severity increases.
+- Validate upgrades in a staging Airflow environment.
+
+Prometheus and Grafana are optional adapters. See [prometheus.md](prometheus.md),
+[grafana.md](grafana.md), and [alerting.md](alerting.md) when metrics-based
+operation is required.
+
+## Troubleshooting
+
+`401` or `403` responses usually indicate a mismatched authentication mode or
+insufficient Airflow permissions. Confirm the API version and auth backend.
+
+Insufficient-history diagnostics mean the DAG has fewer successful observations
+than the active analysis policy requires. Increase the available history or
+adjust the policy deliberately; do not treat missing history as a normal result.
+
+For timeouts or rate limiting, lower batch concurrency and history limits before
+increasing retry counts. See [authentication.md](authentication.md) and
+[python-api.md](python-api.md) for deeper integration options.