@1aboveio/skills 0.0.0-stage → 1.0.0

package/skills/app-dev/pyspark/assets/templates/utils/__init__.py

@@ -0,0 +1 @@
+"""Companion utilities for the bundled ETL template."""

package/skills/app-dev/pyspark/assets/templates/utils/hudi_metadata.py

@@ -0,0 +1,14 @@
+"""Safe default for repositories without a scoped Hudi metadata index.
+
+Replace this helper with repository-owned read/write metadata configuration when
+the target cluster maintains a Hudi metadata table. Returning no options keeps
+the ETL template from enabling an index that writers may not maintain.
+"""
+
+from typing import Dict
+
+
+def hudi_metadata_write_options(table: str) -> Dict[str, str]:
+    """Return no table-specific Hudi metadata overrides by default."""
+    del table
+    return {}

package/skills/app-dev/pyspark/references/diagnosis-and-profiling.md

@@ -0,0 +1,169 @@
+# Diagnosis And Profiling
+
+Use this reference to decide whether a Spark job is fast enough, identify the
+largest performance gap, and determine whether a canary is ready for write and
+reconciliation. Focus on evidence and outcome; choose environment-specific
+commands from the available runtime.
+
+## When Profiling Is Required
+
+Profiling is optional during production validation. It is **required** when
+**either**:
+
+1. The small-sample canary (typically one month) wall time exceeds **5 minutes**
+   **and** the run was **not** resource-constrained (queued, under-provisioned
+   executors, competing warehouse work).
+2. **Any** job — canary, authorized write, or a scheduled production run —
+   exceeds **30 minutes**. This is a hard gate. Resource contention does not
+   waive it.
+
+Otherwise skip. When required, parse the event log and compare an isolated
+baseline on the same sample and resource profile. Keep validate-only time
+separate from write time.
+
+```bash
+set -o pipefail
+hdfs dfs -cat "$EVENT_LOG" | \
+  python {baseDir}/scripts/spark_eventlog_summary.py > canary-summary.json
+```
+
+## Performance Target
+
+Use this default target unless the repository defines a stricter one:
+
+- A medium batch job with fewer than 10 million input records should complete
+  within 5 minutes under properly isolated profiling conditions.
+- Measure one complete scheduled job, not a selected fast stage.
+- Report compute/validation, publication, and end-to-end wall time separately.
+- A validate-only canary must meet the 5-minute compute/validation target, but it
+  does not prove publication or end-to-end write performance.
+- Define a separate explicit target for larger datasets, streaming jobs, or jobs
+  whose required behavior is inherently outside this workload class.
+
+Treat the target as failed until comparable evidence proves it passed.
+
+## Require An Isolated, Comparable Profile
+
+Accept a baseline or candidate profile only when all of these are fixed and
+recorded:
+
+- exact code hash and Spark, Python, Java, and table-format runtimes;
+- immutable source snapshot and input-row denominator;
+- equivalent job mode, history scope, validation, and write mode;
+- stable executor/resource configuration and YARN queue;
+- no competing Spark or warehouse workload that can materially consume the
+  same CPU, memory, network, disk, metastore, or source systems;
+- readable driver logs, executor/container logs, and retained Spark event log;
+- explicit statement of whether scheduler/queue wait and publication are
+  included in the reported time.
+
+If isolation or scope equivalence cannot be established, mark the comparison
+inconclusive instead of attributing the difference to code.
+
+## Improve Logic, Not Parameters
+
+Apply these rules to every performance change:
+
+- Reduce physical work at the same input scope, resource profile, and required
+  behavior.
+- Prefer fewer scans, pairs, rows, passes, shuffles, sorts, Python crossings,
+  state copies, and writes.
+- Fix unbounded joins, repeated computation, poor state shape, missing pruning,
+  hot-key algorithms, and write amplification in the job logic or data layout.
+- Do not claim a performance fix from changing shuffle partitions, executor
+  count/cores/memory, timeouts, broadcast thresholds or hints, AQE switches,
+  repartition counts, cache/persistence, or retry settings. Raising Spark or
+  Hudi shuffle parallelism also creates small files on the write path; leave
+  those settings unset.
+- Do not meet the target by dropping history, rows, columns, validations,
+  formulas, or publication work required by the job.
+- Classify a proven cluster, storage, or configuration defect as an operational
+  issue. Correct and measure it separately; do not present it as a logic
+  optimization.
+
+Accept a candidate only when its measured gain comes from a better computation
+shape rather than more resources or relaxed work.
+
+## Use The Evidence Source That Answers The Question
+
+| Evidence source | Use it to determine |
+|---|---|
+| YARN ResourceManager, application report, and container status | Queue/allocation delay, application attempts, executor/container loss, resource contention, and final lifecycle state |
+| Live Spark UI | Current jobs, stages, SQL executions, executors, task skew, shuffle, spill, GC, storage, and active bottlenecks |
+| Spark History Server | The same Spark execution evidence after completion when retained event logs are available |
+| Retained Spark event log and `scripts/spark_eventlog_summary.py` | Durable stage-attempt/task metrics, final adaptive plan-pattern counts, output evidence, and baseline/candidate comparison |
+| Spark History Server REST API | Programmatic job, stage, task-summary, executor, SQL-plan, and environment evidence |
+| YARN driver and executor/container logs | OOM, fetch failure, retries, executor loss, Python/JVM exceptions, writer behavior, and commit boundaries |
+| Logical/physical plans and SQL runtime statistics | Scans, joins, exchanges, sorts, windows, Python nodes, estimated/runtime cardinality, and optimizer decisions |
+| Source/table statistics, file/partition metadata, and table-format timeline | Input size, file shape, partition coverage, stale statistics, write amplification, and commit evidence |
+| Cluster metrics such as Prometheus, Grafana, JMX, or node telemetry | CPU, memory, GC, disk, network, and infrastructure saturation outside the Spark plan |
+| Code and test inspection | Unbounded joins, repeated actions, unnecessary scans, collect/explode growth, UDF boundaries, incorrect history scope, and missing plan-shape protection |
+
+Use code inference to form a hypothesis when runtime evidence is incomplete.
+Do not present inference alone as measured performance proof.
+
+## Identify The Dominant Gap
+
+Name the dominant gap before changing code. Classify it as one or more of:
+
+- queue, allocation, startup, or dependency delay;
+- source listing, scan, partition pruning, or small-file overhead;
+- intermediate cardinality explosion from join, explode, or repeated grouping;
+- shuffle, sort, exchange, or network transfer;
+- hot-key or partition skew and straggler tasks;
+- Python serialization, UDF, or whole-group memory overhead;
+- JVM/Python memory pressure, GC, spill, or executor loss;
+- driver collection, planning, or result-size pressure;
+- repeated computation across Spark actions;
+- table writer, file layout, commit, or downstream storage latency.
+
+Require an evidence chain:
+
+```text
+performance target gap
+-> application/job/stage/task evidence
+-> physical-plan or runtime operator
+-> responsible transformation or infrastructure constraint
+```
+
+Fix the earliest responsible cause in that chain. Do not tune unrelated Spark
+settings merely because the slow work becomes visible at a later action.
+
+## Resolution Workflow
+
+1. **Measure performance:** capture an isolated baseline and quantify the gap to
+   the 5-minute target using aggregate-only evidence.
+2. **Identify the gap:** locate the dominant job/stage/task and connect it to a
+   plan operator, code shape, data skew, writer, or infrastructure constraint.
+3. **Make the fix:** reduce the responsible logical or physical work without
+   tuning parameters, adding resources, or weakening output identity, ordering,
+   boundaries, arithmetic, history, validation, or write behavior.
+4. **Validate locally:** prove required behavior, boundary cases, and the
+   intended physical-plan change with focused tests. Do not treat local Spark as
+   cluster parity.
+5. **Production validation:** one-month (typical) sample: canary, optional
+   profile (5-minute unconstrained canary **or** any job over 30 minutes),
+   authorized write, then reconciliation.
+
+Repeat the loop when the target is still missed or the dominant gap moves.
+
+## Minimum Evidence
+
+- **Scope:** target, input-row denominator, snapshot/date range, and
+  validate-only or write-enabled mode.
+- **Run identity:** baseline/candidate application IDs and code hashes, plus
+  confirmation of the same runtime, resources, and isolation conditions.
+- **Performance:** wall time, dominant stage, maximum task, cardinality,
+  shuffle/spill, logic change, and pass/fail against the 5-minute target.
+- **Correctness:** local plan/boundary results, canary blocker/output status,
+  and cluster reconciliation (aggregates + bidirectional parity).
+- **Delivery:** cluster application IDs, published hash/commit, reconciliation
+  result when a write ran, and remaining limitations.
+
+Keep evidence aggregate-only. Never print credentials or entity-level rows.
+
+## Primary Spark References
+
+- [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html)
+- [Running Spark on YARN](https://spark.apache.org/docs/latest/running-on-yarn.html)
+- [Spark SQL Performance Tuning](https://spark.apache.org/docs/latest/sql-performance-tuning.html)

package/skills/app-dev/pyspark/references/etl-contract.md

@@ -0,0 +1,129 @@
+# ETL Contract
+
+Read this reference during Step 1 before creating or changing a Spark ETL job.
+Use [the complete ETL module](../assets/templates/etl.py) when the repository
+does not already provide an equivalent base module. Prefer and extend an
+established repository module instead of introducing a parallel framework.
+
+The bundled `etl.py` is derived from a complete production implementation. It
+retains every source adapter, session helper, and writer, with template-only
+fixes for empty repartition settings, recursive CLI flags, logging, and a safe
+Hudi metadata default.
+
+## Included ETL Modules
+
+| Module | Included behavior |
+|---|---|
+| Spark session helpers | Native Spark and Spark Connect creation, active-runtime detection, nested leases, and ownership-safe shutdown |
+| `Etl` | Date-scoped extraction, full/per-day execution, schema completion, partition preparation, Spark/Hudi publication, and CLI entry point |
+| `MongoDbEtl` | MongoDB connector extraction with optional declared schema |
+| `JdbcEtl` | Predicate-scoped JDBC extraction, fetch sizing, and partitioned reads when date bounds are available |
+| `MySqlEtl` | MySQL JDBC driver specialization |
+| `MsSqlEtl` | SQL Server quoting, URL-property handling, SSL defaults, and single-partition extraction for redirect-sensitive servers |
+| `JsonEtl` | Recursive JSON extraction with optional schema |
+| `CsvEtl` | Recursive/header-aware CSV extraction with optional schema |
+
+## Declare The Contract Before Transformation Logic
+
+| Surface | Required declarations | `etl.py` mapping |
+|---|---|---|
+| Input parameters | Names, defaults, validation, secrets by reference, and source scope | constructor and `run_from_cli`; `start_date`, `end_date`, `url`, `bulk` |
+| Runtime | Exact Python and PySpark versions | deployment/runtime files outside the class; verify before session creation |
+| Concurrency | Overlap policy, idempotency, lock ownership, and retry/restart behavior | `concurrency_mode`, `zookeeper`, Hudi lock key/options, scheduler policy |
+| Output table | Destination and `spark_table` versus `hudi_table` | `dst_db`, `dst_tbl`, `path`, `table_type` |
+| Output identity | Complete record key and schema | `id`, transformed DataFrame schema |
+| Output layout | Partition columns and Hudi conflict order | `par_cols`, `ts` precombine field |
+| Write behavior | Append, overwrite, upsert, bulk insert, or insert overwrite | `hudi_mode`, `hudi_mode_override`, `load_hudi`, `load_spark` |
+| Spark runtime | Native Spark versus Spark Connect and session owner | `create_spark_session`, `managed_spark_session`, `spark_remote` |
+| Processing scope | Full versus incremental/per-day | `bulk`, `filter_by`, `offset`, `start_date`, `end_date`, `process` |
+| Incremental state | Watermark, lookback/checkpoint, restart boundary, and late-data behavior | `filter_by`, `offset`, subclass/checkpoint implementation |
+
+Do not design the transformation until every applicable surface has an explicit
+value. Reject defaults that can silently change data scope, key semantics,
+concurrency, or publication behavior.
+
+## Adapt Repository-Specific Dependencies
+
+The complete module depends on its source warehouse. Resolve each dependency
+before reuse:
+
+- The bundled [`utils/hudi_metadata.py`](../assets/templates/utils/hudi_metadata.py)
+  returns no metadata overrides. Replace it only when the target repository
+  maintains a scoped Hudi metadata index on both reads and writes.
+- Pin compatible PySpark, pandas, python-dateutil, Typer, and typing-extensions
+  versions.
+- Provide the Hudi Spark bundle and Hive sync configuration expected by the
+  target cluster.
+- Provide MongoDB, MySQL, and SQL Server connector/driver artifacts only for the
+  adapters the job uses.
+- Replace Zookeeper lock endpoints, base path, port, and lock ownership with the
+  target repository's concurrency mechanism.
+- Review Hudi key generator, complex-key encoding, metadata, schema reconcile,
+  initial-load fallback, and write operation against the deployed Hudi version.
+- Route connection strings and credentials through the repository's secret
+  mechanism. Do not place secret values in class attributes, CLI examples, or
+  logs.
+
+Remove unused adapters only after confirming no job imports them. Keep the
+bundled asset as the reference and adapt a copy inside the target repository.
+
+## Close Contract Gaps Before Use
+
+The bundled module implements `extract -> transform -> load`. Before adopting
+it as a job base, require the target repository to add or confirm:
+
+- an aggregate validation boundary before every `load` path;
+- an in-job validate-only guard that logs publication was skipped;
+- complete output-identity and required-column checks;
+- explicit overlapping-run and idempotency behavior;
+- exact source snapshot/restart semantics for incremental runs;
+- focused tests for native/Connect, full/incremental, and selected adapters;
+- independent post-write reconciliation.
+
+Do not hide these gaps in wrapper scripts. Validation and no-write controls
+belong before the writer in the job process.
+
+## Contract Rules
+
+- Pin validation and publication to the same immutable source boundary.
+- Treat `ts` as Hudi conflict-resolution/precombine order, not physical row
+  sort order.
+- Declare partition columns independently from record keys.
+- Reject Hudi output without a complete record key and precombine field.
+- Reject incremental mode without a filter/watermark and exact restart boundary.
+- Reject optimistic concurrency without a working lock provider.
+- Do not rely on scheduler timing as the single-writer guarantee.
+- Do not set `spark.sql.shuffle.partitions` or Hudi
+  `hoodie.insert.shuffle.parallelism` / `hoodie.upsert.shuffle.parallelism` /
+  `hoodie.bulkinsert.shuffle.parallelism`. A raised shuffle partition count
+  becomes one small file per task on the write path. Size files with
+  `HudiLayout` (default one ~128MB file per partition), not shuffle parallelism.
+- Keep bulk inserts physically grouped and sorted by partition with
+  `hoodie.bulkinsert.sort.mode=PARTITION_PATH_REPARTITION_AND_SORT`; override it
+  through `extra_hudi_options` only when the job contract requires another layout.
+
+### Repair small files
+
+Existing tables written with high shuffle parallelism stay small-file until
+rewritten. Use `Etl.rewrite_hudi_layout` (same keys, partitions, locks, and
+`HudiLayout`). Restrict with a partition predicate when the table is large.
+Do not set shuffle parallelism on the rewrite either.
+
+```python
+job = MyEtl(start_date="2026-08-01", end_date="2026-08-02")
+job.rewrite_hudi_layout("dt = '2026-08-01'")  # one partition
+# job.rewrite_hudi_layout()                   # whole table
+```
+
+- Do not present `repartition`, JDBC partition count, fetch size, or writer
+  option changes as logic optimization; classify required configuration fixes
+  as separate operational work.
+
+## Minimum Review Evidence
+
+- Contract fields and runtime dependencies are explicit.
+- The selected adapter is tested against its real connector/runtime.
+- Writer options match table type, key, order, partitions, concurrency, and
+  write mode.
+- Validation and validate-only controls precede every writer.
+- Full/incremental and native/Connect paths used by the job are covered.

package/skills/app-dev/pyspark/references/production-validation.md

@@ -0,0 +1,131 @@
+# Production Validation
+
+Playbook for workflow step 5: **canary → (optional) profile → write →
+reconciliation** on the cluster. Use a **small sample**, typically **one month**
+of source data. Bind canary and write to the same pinned snapshot. Local tests
+do not substitute.
+
+## Fail-Closed Admission
+
+Do not launch until all applicable facts are recorded and valid:
+
+- exact commit/module hash;
+- production Spark, Python, Java, and table-format runtimes;
+- source snapshot or date boundary (the one-month sample);
+- scheduler pause/ownership state;
+- zero competing warehouse work;
+- explicit validate-only or write-enabled mode;
+- durable writable log and event-log destination;
+- restart/recovery plan for writes.
+
+Classify existing processes and working-tree changes before touching them. Do
+not kill or overwrite another operator's work.
+
+## 1. Canary
+
+Implement the validate-only guard **in the Spark job**, not only in a wrapper.
+Validate with one aggregate action, then return before any writer.
+
+```python
+import json
+
+from pyspark.sql import functions as F
+
+
+def validate_candidate(candidate):
+    # first() is a Spark action: it executes the lazy transform plan.
+    metrics = candidate.agg(
+        F.count("*").alias("row_count"),
+        F.countDistinct("event_id").alias("distinct_event_count"),
+        F.sum(
+            F.when(F.col("event_id").isNull(), 1).otherwise(0)
+        ).alias("null_event_id_count"),
+        F.sum(
+            F.when(F.col("event_count_1h") < 0, 1).otherwise(0)
+        ).alias("invalid_event_count_1h"),
+        F.count("amount_usd_1h").alias("non_null_amount_usd_1h_count"),
+    ).first().asDict()
+
+    blockers = []
+    if metrics["row_count"] != metrics["distinct_event_count"]:
+        blockers.append("event_id is not unique")
+    if metrics["null_event_id_count"]:
+        blockers.append("event_id contains nulls")
+    if metrics["invalid_event_count_1h"]:
+        blockers.append("event_count_1h contains negative values")
+    if blockers:
+        raise RuntimeError(f"validation blocked: {blockers}")
+    return metrics
+
+
+def run(validate_only, source_snapshot):
+    source = extract(source_snapshot)
+    candidate = transform(source)
+
+    metrics = validate_candidate(candidate)
+    print(json.dumps(metrics, sort_keys=True))
+
+    if validate_only:
+        print("validate-only: skipping publication")
+        return
+
+    publish(candidate)
+```
+
+Pin every source read to the same immutable snapshot. If the source cannot be
+pinned, the canary does not prove what the writer will publish. Reference every
+derived output in at least one validation aggregate so Catalyst cannot prune it.
+
+Launch through the production `spark-submit` path and runtimes:
+
+```bash
+export PYSPARK_DRIVER_PYTHON="$DRIVER_PYTHON"
+export PYSPARK_PYTHON="$EXECUTOR_PYTHON"
+
+spark-submit \
+  --name "$CANARY_NAME" \
+  --conf spark.pyspark.driver.python="$DRIVER_PYTHON" \
+  --conf spark.pyspark.python="$EXECUTOR_PYTHON" \
+  --conf spark.eventLog.enabled=true \
+  job.py --validate-only
+```
+
+Do not replace another operator's scheduler files in place.
+
+Prove the canary did not write:
+
+1. Validation completed with zero blockers.
+2. The terminal log says publication was skipped.
+3. The event log reports zero Spark output records and bytes.
+
+Do not add `.cache()` or `.persist()` to warm the canary.
+
+## 2. Profiling (optional)
+
+After the canary, profile when **either** the small-sample canary exceeds
+**5 minutes** without resource constraints, **or any job exceeds 30 minutes**
+(hard gate). Otherwise skip. When required, follow
+[Diagnosis And Profiling](diagnosis-and-profiling.md).
+
+## 3. Write
+
+Write-enabled only after a successful canary and **explicit authorization**.
+Re-check admission. Publish the same pinned sample. Do not present canary time
+as write performance.
+
+If a write is cancelled, prove from logs/source that publication was not
+reached. YARN `KILLED` alone does not prove no commit occurred.
+
+## 4. Reconciliation
+
+Independent cluster read of the published scope — not a reuse of canary
+aggregates. Follow [Reconciliation](reconciliation.md). Fail closed on mismatch.
+
+## Evidence And Ownership
+
+Safe evidence: counts, null rates, distinct counts, min/max dates, hashes,
+stage metrics, plan-node counts, commit IDs. Do not print credentials or
+entity-level rows.
+
+After the run: confirm processes are terminal, restore scheduler ownership,
+and report compute/validation, publication, and end-to-end time separately.

package/skills/app-dev/pyspark/references/reconciliation.md

@@ -0,0 +1,38 @@
+# Reconciliation
+
+Cluster check after an authorized write in production validation. Use the **same
+one-month sample** as the canary. Local fixtures do not prove this.
+
+Independently read the published table or Hudi commit. Do not reuse canary
+aggregates. Fail closed on mismatch.
+
+## What To Reconcile
+
+Against the canary metrics and the previous path/oracle:
+
+- row count and distinct identity count
+- schema and required columns
+- date/partition coverage of the sample
+- write mode, commit success, and downstream readability
+- validation blockers still zero on the published scope
+
+## Bidirectional Row Parity
+
+Counts alone miss wrong values, swapped identities, and duplicate multiplicity.
+Compare full rows both ways with `exceptAll`, not `subtract`:
+
+```python
+expected = old_build(source).select(*OUTPUT_COLUMNS)
+actual = published.select(*OUTPUT_COLUMNS)
+
+assert expected.exceptAll(actual).count() == 0
+assert actual.exceptAll(expected).count() == 0
+```
+
+Pin `source` to the same snapshot the writer used. Keep the previous
+implementation only as this oracle while parity is being proven.
+
+## Evidence
+
+Safe evidence: counts, distinct counts, null rates, min/max dates, hashes, and
+commit IDs. Do not print credentials or entity-level production rows.

package/skills/app-dev/pyspark/references/transformation-design.md

@@ -0,0 +1,190 @@
+# Transformation Design
+
+Use this playbook when implementing or restructuring Spark transformations.
+The snippets illustrate shapes; adapt names, schemas, and state to the job's
+required behavior.
+
+## Define Required Behavior Before Operators
+
+Record output identity/cardinality, event-time and timezone, deterministic
+ordering, time boundaries, null/tombstone behavior, arithmetic, and required
+history. Performance work preserves this behavior unless the user explicitly
+approves a change.
+
+## Choose The Shallowest Sufficient Mechanism
+
+Prefer, in order:
+
+1. Built-in column expressions and aggregations.
+2. Spark windows when state and partition size are bounded.
+3. Exact temporal bucketing with overlap correction.
+4. One sorted grouped sweep for stateful transitions.
+5. Python/pandas UDFs only when they remove substantially more distributed work
+   than they add.
+
+Measure rather than assuming a UDF, Arrow, or built-in rewrite is faster.
+
+## Keep One DataFrame Boundary
+
+```python
+def build_features(events: DataFrame, rules: DataFrame) -> DataFrame:
+    normalized = normalize_events(events, rules)
+    return add_stateful_features(normalized)
+```
+
+Put complex state transitions in pure functions with explicit input and output
+schemas so boundary semantics can be tested without a cluster.
+
+## Use Built-In Windows For Independent Aggregates
+
+When the feature is a standard rolling aggregate and all same-time peers should
+share the same frame, keep execution inside Catalyst:
+
+```python
+from pyspark.sql import Window, functions as F
+
+one_hour = (
+    Window.partitionBy("entity_id")
+    .orderBy(F.col("event_time").cast("long"))
+    .rangeBetween(-3600, 0)
+)
+
+features = (
+    events.withColumn("event_count_1h", F.count(F.lit(1)).over(one_hour))
+    .withColumn("amount_usd_1h", F.sum("amount_usd").over(one_hour))
+)
+```
+
+Check peer semantics carefully: a time-only range frame includes every row with
+the same timestamp. It is wrong when same-time visibility depends on stable
+secondary keys.
+
+## Keep Bounded State Native With Higher-Order Aggregation
+
+For bounded per-key arrays and group-level state, `functions.aggregate` keeps
+the fold in Catalyst and does not call Python per group:
+
+```python
+ordered = events.groupBy("entity_id").agg(
+    F.sort_array(
+        F.collect_list(
+            F.struct(
+                "event_time",
+                "event_id",
+                F.col("amount_usd").cast("double").alias("amount_usd"),
+            )
+        )
+    ).alias("events")
+)
+
+initial = F.struct(
+    F.lit(0.0).alias("balance"),
+    F.lit(0).alias("event_count"),
+)
+
+state = ordered.select(
+    "entity_id",
+    F.aggregate(
+        "events",
+        initial,
+        lambda acc, event: F.struct(
+            (acc["balance"] + event["amount_usd"]).alias("balance"),
+            (acc["event_count"] + 1).alias("event_count"),
+        ),
+    ).alias("state"),
+)
+```
+
+This example emits final group state. Carrying one output per event inside the
+accumulator grows an array and can become expensive, so use it only when group
+size and output state are bounded.
+
+## Choose The Stateful Execution API Deliberately
+
+Arbitrary per-row FIFO and temporal eviction have no general Catalyst-native
+PySpark batch operator. Choose based on the actual state shape:
+
+| State shape | Preferred API | Main constraint |
+|---|---|---|
+| Independent rolling aggregates | `Window` | Same-time peer semantics must fit the frame |
+| Bounded array fold or final group state | `functions.aggregate` | Collected array and accumulator must remain bounded |
+| Bounded batch group needing Python | [`applyInPandas`](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.GroupedData.applyInPandas.html) | Full shuffle; each group is loaded into memory |
+| Stateful streaming group | [`applyInPandasWithState`](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.GroupedData.applyInPandasWithState.html) | Streaming API, not a batch replacement |
+| Large or unbounded sequential batch state | Redesign with exact bucketing, Scala `mapGroups`, or a stateful engine | More implementation and operational complexity |
+
+Keep the transition function pure regardless of the execution API. If a Python
+grouped path replaces a much worse relation, benchmark it against the same
+snapshot and report Python serialization and maximum-group memory explicitly.
+
+## Preserve Exact Output Columns And Row Identity
+
+Follow these rules:
+
+1. Declare the complete output identity before writing the transformation.
+2. Carry every identity column through grouping, sorting, UDF, and explode
+   boundaries.
+3. Require one grouped result per identity before joining it to the base rows.
+4. Join on the complete identity. Do not substitute customer, timestamp, or
+   another non-unique lookup key.
+5. Declare the required output columns and their order. Never use `select("*")`
+   for the final candidate.
+6. Drop helper, duplicate, and intermediate columns at the final projection.
+7. Fail tests and production validation on null, missing, or duplicate
+   identities and on unexpected output columns.
+
+```python
+IDENTITY_COLUMNS = ["event_id"]
+BASE_OUTPUT_COLUMNS = [
+    "event_id",
+    "entity_id",
+    "event_time",
+    "amount_usd",
+]
+FEATURE_COLUMNS = ["event_count_1h", "amount_usd_1h"]
+OUTPUT_COLUMNS = BASE_OUTPUT_COLUMNS + FEATURE_COLUMNS
+
+feature_projection = features_by_event.select(
+    *IDENTITY_COLUMNS,
+    *FEATURE_COLUMNS,
+)
+
+candidate = (
+    events.select(*BASE_OUTPUT_COLUMNS)
+    .join(feature_projection, on=IDENTITY_COLUMNS, how="left")
+    .select(*OUTPUT_COLUMNS)
+)
+
+assert candidate.columns == OUTPUT_COLUMNS
+```
+
+For composite identity, list every component, for example
+`IDENTITY_COLUMNS = ["transaction_id", "transaction_leg_id"]`.
+
+## Local proof
+
+These checks belong in **local tests** (workflow step 4). Cluster row parity is
+[Reconciliation](reconciliation.md) after a write.
+
+Keep a bounded previous-path oracle on **synthetic** fixtures. Assert critical
+rules directly so oracle and candidate cannot share the same mistake:
+
+```python
+by_id = {row.event_id: row for row in actual.collect()}
+assert by_id["at-24h"].count_24h == 2       # inclusive lower boundary
+assert by_id["after-24h"].count_24h == 1    # one unit outside
+assert by_id["same-time-b"].count_1h == 2   # stable tie ordering
+```
+
+Cover empty and single-event groups; exact boundaries and one unit beyond;
+same-timestamp ties; null key/time/amount; open/close/eviction; floating-point
+residue; one dense hot key; duplicate and missing identities.
+
+Assert the expensive plan shape is absent without snapshotting the whole plan:
+
+```python
+plan = actual._jdf.queryExecution().executedPlan().toString().lower()
+assert not ("join" in plan and "event_time" in plan and "interval" in plan)
+```
+
+Mutation-prove high-risk tests (cutoff direction, tie-breaker, pair join,
+required column) so they fail when the rule is broken.