@1aboveio/skills 0.10.1 → 0.11.0

package/skills/backend/pyspark/assets/templates/utils/hudi_metadata.py

@@ -0,0 +1,14 @@
+"""Safe default for repositories without a scoped Hudi metadata index.
+
+Replace this helper with repository-owned read/write metadata configuration when
+the target cluster maintains a Hudi metadata table. Returning no options keeps
+the ETL template from enabling an index that writers may not maintain.
+"""
+
+from typing import Dict
+
+
+def hudi_metadata_write_options(table: str) -> Dict[str, str]:
+    """Return no table-specific Hudi metadata overrides by default."""
+    del table
+    return {}

package/skills/backend/pyspark/references/diagnosis-and-profiling.md

@@ -0,0 +1,145 @@
+# Diagnosis And Profiling
+
+Use this reference to decide whether a Spark job is fast enough, identify the
+largest performance gap, and determine whether a fix is ready to ship. Focus on
+the required evidence and outcome; choose environment-specific commands and
+APIs from the available runtime.
+
+## Performance Target
+
+Use this default target unless the repository defines a stricter one:
+
+- A medium batch job with fewer than 10 million input records should complete
+  within 5 minutes under properly isolated profiling conditions.
+- Measure one complete scheduled job, not a selected fast stage.
+- Report compute/validation, publication, and end-to-end wall time separately.
+- A validate-only canary must meet the 5-minute compute/validation target, but it
+  does not prove publication or end-to-end write performance.
+- Define a separate explicit target for larger datasets, streaming jobs, or jobs
+  whose required behavior is inherently outside this workload class.
+
+Treat the target as failed until comparable evidence proves it passed.
+
+## Require An Isolated, Comparable Profile
+
+Accept a baseline or candidate profile only when all of these are fixed and
+recorded:
+
+- exact code hash and Spark, Python, Java, and table-format runtimes;
+- immutable source snapshot and input-row denominator;
+- equivalent job mode, history scope, validation, and write mode;
+- stable executor/resource configuration and YARN queue;
+- no competing Spark or warehouse workload that can materially consume the
+  same CPU, memory, network, disk, metastore, or source systems;
+- readable driver logs, executor/container logs, and retained Spark event log;
+- explicit statement of whether scheduler/queue wait and publication are
+  included in the reported time.
+
+If isolation or scope equivalence cannot be established, mark the comparison
+inconclusive instead of attributing the difference to code.
+
+## Improve Logic, Not Parameters
+
+Apply these rules to every performance change:
+
+- Reduce physical work at the same input scope, resource profile, and required
+  behavior.
+- Prefer fewer scans, pairs, rows, passes, shuffles, sorts, Python crossings,
+  state copies, and writes.
+- Fix unbounded joins, repeated computation, poor state shape, missing pruning,
+  hot-key algorithms, and write amplification in the job logic or data layout.
+- Do not claim a performance fix from changing shuffle partitions, executor
+  count/cores/memory, timeouts, broadcast thresholds or hints, AQE switches,
+  repartition counts, cache/persistence, or retry settings.
+- Do not meet the target by dropping history, rows, columns, validations,
+  formulas, or publication work required by the job.
+- Classify a proven cluster, storage, or configuration defect as an operational
+  issue. Correct and measure it separately; do not present it as a logic
+  optimization.
+
+Accept a candidate only when its measured gain comes from a better computation
+shape rather than more resources or relaxed work.
+
+## Use The Evidence Source That Answers The Question
+
+| Evidence source | Use it to determine |
+|---|---|
+| YARN ResourceManager, application report, and container status | Queue/allocation delay, application attempts, executor/container loss, resource contention, and final lifecycle state |
+| Live Spark UI | Current jobs, stages, SQL executions, executors, task skew, shuffle, spill, GC, storage, and active bottlenecks |
+| Spark History Server | The same Spark execution evidence after completion when retained event logs are available |
+| Retained Spark event log and `scripts/spark_eventlog_summary.py` | Durable stage-attempt/task metrics, final adaptive plan-pattern counts, output evidence, and baseline/candidate comparison |
+| Spark History Server REST API | Programmatic job, stage, task-summary, executor, SQL-plan, and environment evidence |
+| YARN driver and executor/container logs | OOM, fetch failure, retries, executor loss, Python/JVM exceptions, writer behavior, and commit boundaries |
+| Logical/physical plans and SQL runtime statistics | Scans, joins, exchanges, sorts, windows, Python nodes, estimated/runtime cardinality, and optimizer decisions |
+| Source/table statistics, file/partition metadata, and table-format timeline | Input size, file shape, partition coverage, stale statistics, write amplification, and commit evidence |
+| Cluster metrics such as Prometheus, Grafana, JMX, or node telemetry | CPU, memory, GC, disk, network, and infrastructure saturation outside the Spark plan |
+| Code and test inspection | Unbounded joins, repeated actions, unnecessary scans, collect/explode growth, UDF boundaries, incorrect history scope, and missing plan-shape protection |
+
+Use code inference to form a hypothesis when runtime evidence is incomplete.
+Do not present inference alone as measured performance proof.
+
+## Identify The Dominant Gap
+
+Name the dominant gap before changing code. Classify it as one or more of:
+
+- queue, allocation, startup, or dependency delay;
+- source listing, scan, partition pruning, or small-file overhead;
+- intermediate cardinality explosion from join, explode, or repeated grouping;
+- shuffle, sort, exchange, or network transfer;
+- hot-key or partition skew and straggler tasks;
+- Python serialization, UDF, or whole-group memory overhead;
+- JVM/Python memory pressure, GC, spill, or executor loss;
+- driver collection, planning, or result-size pressure;
+- repeated computation across Spark actions;
+- table writer, file layout, commit, or downstream storage latency.
+
+Require an evidence chain:
+
+```text
+performance target gap
+-> application/job/stage/task evidence
+-> physical-plan or runtime operator
+-> responsible transformation or infrastructure constraint
+```
+
+Fix the earliest responsible cause in that chain. Do not tune unrelated Spark
+settings merely because the slow work becomes visible at a later action.
+
+## Resolution Workflow
+
+1. **Measure performance:** capture an isolated baseline and quantify the gap to
+   the 5-minute target using aggregate-only evidence.
+2. **Identify the gap:** locate the dominant job/stage/task and connect it to a
+   plan operator, code shape, data skew, writer, or infrastructure constraint.
+3. **Make the fix:** reduce the responsible logical or physical work without
+   tuning parameters, adding resources, or weakening output identity, ordering,
+   boundaries, arithmetic, history, validation, or write behavior.
+4. **Validate locally:** prove required behavior, bidirectional parity, boundary
+   cases, and the intended physical-plan change with focused tests.
+5. **Canary and profile in the cluster:** run the production entry point in
+   validate-only mode on the comparable isolated snapshot; retain logs, prove
+   zero writes, and compare the same metrics with the baseline.
+6. **Ship:** open and review the PR, pass CI, enqueue and merge, deploy the merged
+   commit, then run authorized production validation and reconciliation.
+
+Repeat the loop when the target is still missed or the dominant gap moves.
+
+## Minimum Evidence
+
+- **Scope:** target, input-row denominator, snapshot/date range, and
+  validate-only or write-enabled mode.
+- **Run identity:** baseline/candidate application IDs and code hashes, plus
+  confirmation of the same runtime, resources, and isolation conditions.
+- **Performance:** wall time, dominant stage, maximum task, cardinality,
+  shuffle/spill, logic change, and pass/fail against the 5-minute target.
+- **Correctness:** local parity/plan results and canary blocker/output status.
+- **Delivery:** deployed hash, publication/reconciliation result when applicable,
+  and remaining limitations.
+
+Keep evidence aggregate-only. Never print credentials or entity-level rows.
+
+## Primary Spark References
+
+- [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html)
+- [Running Spark on YARN](https://spark.apache.org/docs/latest/running-on-yarn.html)
+- [Spark SQL Performance Tuning](https://spark.apache.org/docs/latest/sql-performance-tuning.html)

package/skills/backend/pyspark/references/etl-contract.md

@@ -0,0 +1,107 @@
+# ETL Contract
+
+Read this reference during Step 1 before creating or changing a Spark ETL job.
+Use [the complete ETL module](../assets/templates/etl.py) when the repository
+does not already provide an equivalent base module. Prefer and extend an
+established repository module instead of introducing a parallel framework.
+
+The bundled `etl.py` is derived from a complete production implementation. It
+retains every source adapter, session helper, and writer, with template-only
+fixes for empty repartition settings, recursive CLI flags, logging, and a safe
+Hudi metadata default.
+
+## Included ETL Modules
+
+| Module | Included behavior |
+|---|---|
+| Spark session helpers | Native Spark and Spark Connect creation, active-runtime detection, nested leases, and ownership-safe shutdown |
+| `Etl` | Date-scoped extraction, full/per-day execution, schema completion, partition preparation, Spark/Hudi publication, and CLI entry point |
+| `MongoDbEtl` | MongoDB connector extraction with optional declared schema |
+| `JdbcEtl` | Predicate-scoped JDBC extraction, fetch sizing, and partitioned reads when date bounds are available |
+| `MySqlEtl` | MySQL JDBC driver specialization |
+| `MsSqlEtl` | SQL Server quoting, URL-property handling, SSL defaults, and single-partition extraction for redirect-sensitive servers |
+| `JsonEtl` | Recursive JSON extraction with optional schema |
+| `CsvEtl` | Recursive/header-aware CSV extraction with optional schema |
+
+## Declare The Contract Before Transformation Logic
+
+| Surface | Required declarations | `etl.py` mapping |
+|---|---|---|
+| Input parameters | Names, defaults, validation, secrets by reference, and source scope | constructor and `run_from_cli`; `start_date`, `end_date`, `url`, `bulk` |
+| Runtime | Exact Python and PySpark versions | deployment/runtime files outside the class; verify before session creation |
+| Concurrency | Overlap policy, idempotency, lock ownership, and retry/restart behavior | `concurrency_mode`, `zookeeper`, Hudi lock key/options, scheduler policy |
+| Output table | Destination and `spark_table` versus `hudi_table` | `dst_db`, `dst_tbl`, `path`, `table_type` |
+| Output identity | Complete record key and schema | `id`, transformed DataFrame schema |
+| Output layout | Partition columns and Hudi conflict order | `par_cols`, `ts` precombine field |
+| Write behavior | Append, overwrite, upsert, bulk insert, or insert overwrite | `hudi_mode`, `hudi_mode_override`, `load_hudi`, `load_spark` |
+| Spark runtime | Native Spark versus Spark Connect and session owner | `create_spark_session`, `managed_spark_session`, `spark_remote` |
+| Processing scope | Full versus incremental/per-day | `bulk`, `filter_by`, `offset`, `start_date`, `end_date`, `process` |
+| Incremental state | Watermark, lookback/checkpoint, restart boundary, and late-data behavior | `filter_by`, `offset`, subclass/checkpoint implementation |
+
+Do not design the transformation until every applicable surface has an explicit
+value. Reject defaults that can silently change data scope, key semantics,
+concurrency, or publication behavior.
+
+## Adapt Repository-Specific Dependencies
+
+The complete module depends on its source warehouse. Resolve each dependency
+before reuse:
+
+- The bundled [`utils/hudi_metadata.py`](../assets/templates/utils/hudi_metadata.py)
+  returns no metadata overrides. Replace it only when the target repository
+  maintains a scoped Hudi metadata index on both reads and writes.
+- Pin compatible PySpark, pandas, python-dateutil, Typer, and typing-extensions
+  versions.
+- Provide the Hudi Spark bundle and Hive sync configuration expected by the
+  target cluster.
+- Provide MongoDB, MySQL, and SQL Server connector/driver artifacts only for the
+  adapters the job uses.
+- Replace Zookeeper lock endpoints, base path, port, and lock ownership with the
+  target repository's concurrency mechanism.
+- Review Hudi key generator, complex-key encoding, metadata, schema reconcile,
+  initial-load fallback, and write operation against the deployed Hudi version.
+- Route connection strings and credentials through the repository's secret
+  mechanism. Do not place secret values in class attributes, CLI examples, or
+  logs.
+
+Remove unused adapters only after confirming no job imports them. Keep the
+bundled asset as the reference and adapt a copy inside the target repository.
+
+## Close Contract Gaps Before Use
+
+The bundled module implements `extract -> transform -> load`. Before adopting
+it as a job base, require the target repository to add or confirm:
+
+- an aggregate validation boundary before every `load` path;
+- an in-job validate-only guard that logs publication was skipped;
+- complete output-identity and required-column checks;
+- explicit overlapping-run and idempotency behavior;
+- exact source snapshot/restart semantics for incremental runs;
+- focused tests for native/Connect, full/incremental, and selected adapters;
+- independent post-write reconciliation.
+
+Do not hide these gaps in wrapper scripts. Validation and no-write controls
+belong before the writer in the job process.
+
+## Contract Rules
+
+- Pin validation and publication to the same immutable source boundary.
+- Treat `ts` as Hudi conflict-resolution/precombine order, not physical row
+  sort order.
+- Declare partition columns independently from record keys.
+- Reject Hudi output without a complete record key and precombine field.
+- Reject incremental mode without a filter/watermark and exact restart boundary.
+- Reject optimistic concurrency without a working lock provider.
+- Do not rely on scheduler timing as the single-writer guarantee.
+- Do not present `repartition`, JDBC partition count, fetch size, or writer
+  option changes as logic optimization; classify required configuration fixes
+  as separate operational work.
+
+## Minimum Review Evidence
+
+- Contract fields and runtime dependencies are explicit.
+- The selected adapter is tested against its real connector/runtime.
+- Writer options match table type, key, order, partitions, concurrency, and
+  write mode.
+- Validation and validate-only controls precede every writer.
+- Full/incremental and native/Connect paths used by the job are covered.

package/skills/backend/pyspark/references/parity-testing.md

@@ -0,0 +1,83 @@
+# Parity Testing
+
+Use this playbook for semantics-preserving Spark rewrites.
+
+## Keep The Old Path As A Bounded Oracle
+
+Retain the previous implementation only in tests while parity is being proven.
+Use small fixtures that are dense enough to expose the old complexity but
+bounded enough to run reliably.
+
+## Compare Full Rows In Both Directions
+
+```python
+expected = old_build(source).select(*OUTPUT_COLUMNS)
+actual = new_build(source).select(*OUTPUT_COLUMNS)
+
+assert expected.exceptAll(actual).count() == 0
+assert actual.exceptAll(expected).count() == 0
+```
+
+Counts alone cannot detect wrong values, swapped identities, or duplicate rows.
+Use `exceptAll`, not `subtract`, when duplicate multiplicity matters.
+
+## Add Literal Boundary Assertions
+
+Parity can be self-confirming if oracle and candidate share the same mistake.
+Assert critical rules directly:
+
+```python
+by_id = {row.event_id: row for row in actual.collect()}
+assert by_id["at-24h"].count_24h == 2       # inclusive lower boundary
+assert by_id["after-24h"].count_24h == 1    # one unit outside
+assert by_id["same-time-b"].count_1h == 2   # stable tie ordering
+```
+
+Keep collected fixtures synthetic and non-sensitive.
+
+## Minimum Fixture Matrix
+
+Cover:
+
+- empty and single-event groups;
+- exact lower/upper boundaries and one unit beyond;
+- same-timestamp events with every tie-breaker;
+- null key, event time, amount, FX, and counterparty;
+- both directions and relevant statuses;
+- repeated and distinct counterparties;
+- state open, partial close, full close, eviction, and re-entry;
+- floating-point add/remove residue;
+- one dense hot key;
+- duplicate and missing output identities.
+
+## Test The Physical Shape
+
+Assert the expensive relation is absent without snapshotting the whole plan:
+
+```python
+plan = actual._jdf.queryExecution().executedPlan().toString().lower()
+assert not ("join" in plan and "event_time" in plan and "interval" in plan)
+```
+
+Prefer a targeted helper that counts known pair-join nodes. Also assert the
+expected grouped-sweep or built-in operator is present when that guards the
+optimization.
+
+## Mutation-Prove Important Tests
+
+Temporarily introduce representative faults and confirm tests fail:
+
+- change `< cutoff` to `<= cutoff`;
+- remove a tie-breaker;
+- use ordinary rolling addition/subtraction;
+- reintroduce the pair join;
+- omit a downstream-required column.
+
+Record red/green evidence for high-risk money, date, and identity semantics.
+
+## Match Test Cost To Ownership
+
+Behavioral Spark tests belong on files that own formulas, source scope,
+history, or state transitions. Unrelated upstream changes may use a focused
+required-column check instead of rerunning an expensive full-history suite.
+Do not demote behavioral coverage for a semantic change.

package/skills/backend/pyspark/references/production-validation.md

@@ -0,0 +1,166 @@
+# Production Validation
+
+Use this playbook before production-scale profiling, canaries, backfills, or
+writes.
+
+## Fail-Closed Admission
+
+Do not launch until all applicable facts are recorded and valid:
+
+- exact commit/module hash;
+- production Spark, Python, Java, and table-format runtimes;
+- source snapshot or date boundary;
+- scheduler pause/ownership state;
+- zero competing warehouse work;
+- explicit validate-only or write-enabled mode;
+- durable writable log and event-log destination;
+- restart/recovery plan for writes.
+
+Classify existing processes and working-tree changes before touching them. Do
+not kill or overwrite another operator's work.
+
+## Run A Validate-Only Canary Before The Writer
+
+Implement the guard in the Spark job, not only in a wrapper. Keep each stage
+literal:
+
+```python
+import json
+
+from pyspark.sql import functions as F
+
+
+def validate_candidate(candidate):
+    # first() is a Spark action: it executes the lazy transform plan.
+    metrics = candidate.agg(
+        F.count("*").alias("row_count"),
+        F.countDistinct("event_id").alias("distinct_event_count"),
+        F.sum(
+            F.when(F.col("event_id").isNull(), 1).otherwise(0)
+        ).alias("null_event_id_count"),
+        F.sum(
+            F.when(F.col("event_count_1h") < 0, 1).otherwise(0)
+        ).alias("invalid_event_count_1h"),
+        F.count("amount_usd_1h").alias("non_null_amount_usd_1h_count"),
+    ).first().asDict()
+
+    blockers = []
+    if metrics["row_count"] != metrics["distinct_event_count"]:
+        blockers.append("event_id is not unique")
+    if metrics["null_event_id_count"]:
+        blockers.append("event_id contains nulls")
+    if metrics["invalid_event_count_1h"]:
+        blockers.append("event_count_1h contains negative values")
+    if blockers:
+        raise RuntimeError(f"validation blocked: {blockers}")
+    return metrics
+
+
+def run(validate_only, source_snapshot):
+    # Bind every action to the same immutable source version or date boundary.
+    source = extract(source_snapshot)
+    candidate = transform(source)      # build a lazy DataFrame plan
+
+    metrics = validate_candidate(candidate)  # execute validation aggregates
+    print(json.dumps(metrics, sort_keys=True))
+
+    if validate_only:                  # canary stops before any writer call
+        print("validate-only: skipping publication")
+        return
+
+    publish(candidate)                 # execute the writer from the same source
+    reconcile_published_scope(metrics) # independently read back aggregates
+```
+
+Validation and publication are separate Spark actions, so Spark may recompute
+the candidate for publication. Pin every source read to the same immutable
+snapshot, table version, or exact date boundary. If the source cannot be pinned,
+the validation result does not prove what the writer will publish; stop and
+redesign the run.
+
+Adapt identity and blocker checks to the table's required behavior. Reference
+every derived output in at least one validation aggregate; otherwise Catalyst
+may prune an unreferenced calculation and the canary will not exercise the full
+candidate. Validation must use aggregate evidence and fail before publication.
+The canary exercises the production-shaped extract, transform, and validation
+path without calling the writer.
+
+## Avoid Explicit Cache And Persistence
+
+Do not add `.cache()` or `.persist()` as a performance fix. They add storage,
+serialization, eviction, and cleanup concerns without improving the logical
+work. Prefer a better transformation shape and a pinned source snapshot.
+
+Treat existing persistence as separate operational behavior: do not expand it
+while optimizing the job, and remove it when repository tests and same-snapshot
+evidence show it is unnecessary.
+
+## Launch Through The Production Entry Point
+
+Use the same `spark-submit` path and driver/executor runtimes as production:
+
+```bash
+export PYSPARK_DRIVER_PYTHON="$DRIVER_PYTHON"
+export PYSPARK_PYTHON="$EXECUTOR_PYTHON"
+
+spark-submit \
+  --name "$CANARY_NAME" \
+  --conf spark.pyspark.driver.python="$DRIVER_PYTHON" \
+  --conf spark.pyspark.python="$EXECUTOR_PYTHON" \
+  --conf spark.eventLog.enabled=true \
+  job.py --validate-only
+```
+
+Do not replace deployed scheduler files with an unmerged staged module. Stage
+an isolated artifact or deploy merged code through the normal path.
+
+## Prove The Canary Did Not Write
+
+Require all three forms of evidence:
+
+1. Validation completed with zero blockers.
+2. The terminal log says publication was skipped.
+3. The event log reports zero Spark output records and bytes.
+
+Parse retained logs with upstream failures visible:
+
+```bash
+set -o pipefail
+hdfs dfs -cat "$EVENT_LOG" | \
+  python {baseDir}/scripts/spark_eventlog_summary.py > canary-summary.json
+```
+
+Record application ID, shell/event-log wall, stage/task metrics, plan nodes,
+shuffle, spill, output, code hash, and snapshot denominator.
+
+## Run Write Verification Separately
+
+Run write-enabled verification only after merge, deployment, successful
+validate-only evidence, and explicit authorization. Re-check admission just
+before launch.
+
+Verify row count, distinct identity count, date/partition coverage, commit
+success, schema, consumer compatibility, validation blockers, write mode,
+single-writer constraints, and downstream readability.
+
+Report compute/validation, publication, and total time separately. Never
+present validate-only time as end-to-end write performance.
+
+If a write run is cancelled, prove from logs/source that publication was not
+reached. YARN `KILLED` alone does not prove no commit occurred.
+
+## Keep Evidence Aggregate-Only
+
+Safe evidence includes counts, null rates, distinct counts, min/max dates,
+hashes, stage metrics, plan-node counts, and commit IDs. Do not print
+credentials, entity rows, account identifiers, or per-entity values.
+
+## Restore Ownership
+
+After the run:
+
+1. Confirm application and wrapper processes are terminal.
+2. Remove agent-owned watchers and temporary processes.
+3. Restore scheduler pause state or ownership.
+4. Separate unrelated downstream failures from the optimized job.
+5. Publish evidence with limitations and unmeasured surfaces.

package/skills/backend/pyspark/references/transformation-design.md

@@ -0,0 +1,162 @@
+# Transformation Design
+
+Use this playbook when implementing or restructuring Spark transformations.
+The snippets illustrate shapes; adapt names, schemas, and state to the job's
+required behavior.
+
+## Define Required Behavior Before Operators
+
+Record output identity/cardinality, event-time and timezone, deterministic
+ordering, time boundaries, null/tombstone behavior, arithmetic, and required
+history. Performance work preserves this behavior unless the user explicitly
+approves a change.
+
+## Choose The Shallowest Sufficient Mechanism
+
+Prefer, in order:
+
+1. Built-in column expressions and aggregations.
+2. Spark windows when state and partition size are bounded.
+3. Exact temporal bucketing with overlap correction.
+4. One sorted grouped sweep for stateful transitions.
+5. Python/pandas UDFs only when they remove substantially more distributed work
+   than they add.
+
+Measure rather than assuming a UDF, Arrow, or built-in rewrite is faster.
+
+## Keep One DataFrame Boundary
+
+```python
+def build_features(events: DataFrame, rules: DataFrame) -> DataFrame:
+    normalized = normalize_events(events, rules)
+    return add_stateful_features(normalized)
+```
+
+Put complex state transitions in pure functions with explicit input and output
+schemas so boundary semantics can be tested without a cluster.
+
+## Use Built-In Windows For Independent Aggregates
+
+When the feature is a standard rolling aggregate and all same-time peers should
+share the same frame, keep execution inside Catalyst:
+
+```python
+from pyspark.sql import Window, functions as F
+
+one_hour = (
+    Window.partitionBy("entity_id")
+    .orderBy(F.col("event_time").cast("long"))
+    .rangeBetween(-3600, 0)
+)
+
+features = (
+    events.withColumn("event_count_1h", F.count(F.lit(1)).over(one_hour))
+    .withColumn("amount_usd_1h", F.sum("amount_usd").over(one_hour))
+)
+```
+
+Check peer semantics carefully: a time-only range frame includes every row with
+the same timestamp. It is wrong when same-time visibility depends on stable
+secondary keys.
+
+## Keep Bounded State Native With Higher-Order Aggregation
+
+For bounded per-key arrays and group-level state, `functions.aggregate` keeps
+the fold in Catalyst and does not call Python per group:
+
+```python
+ordered = events.groupBy("entity_id").agg(
+    F.sort_array(
+        F.collect_list(
+            F.struct(
+                "event_time",
+                "event_id",
+                F.col("amount_usd").cast("double").alias("amount_usd"),
+            )
+        )
+    ).alias("events")
+)
+
+initial = F.struct(
+    F.lit(0.0).alias("balance"),
+    F.lit(0).alias("event_count"),
+)
+
+state = ordered.select(
+    "entity_id",
+    F.aggregate(
+        "events",
+        initial,
+        lambda acc, event: F.struct(
+            (acc["balance"] + event["amount_usd"]).alias("balance"),
+            (acc["event_count"] + 1).alias("event_count"),
+        ),
+    ).alias("state"),
+)
+```
+
+This example emits final group state. Carrying one output per event inside the
+accumulator grows an array and can become expensive, so use it only when group
+size and output state are bounded.
+
+## Choose The Stateful Execution API Deliberately
+
+Arbitrary per-row FIFO and temporal eviction have no general Catalyst-native
+PySpark batch operator. Choose based on the actual state shape:
+
+| State shape | Preferred API | Main constraint |
+|---|---|---|
+| Independent rolling aggregates | `Window` | Same-time peer semantics must fit the frame |
+| Bounded array fold or final group state | `functions.aggregate` | Collected array and accumulator must remain bounded |
+| Bounded batch group needing Python | [`applyInPandas`](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.GroupedData.applyInPandas.html) | Full shuffle; each group is loaded into memory |
+| Stateful streaming group | [`applyInPandasWithState`](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.GroupedData.applyInPandasWithState.html) | Streaming API, not a batch replacement |
+| Large or unbounded sequential batch state | Redesign with exact bucketing, Scala `mapGroups`, or a stateful engine | More implementation and operational complexity |
+
+Keep the transition function pure regardless of the execution API. If a Python
+grouped path replaces a much worse relation, benchmark it against the same
+snapshot and report Python serialization and maximum-group memory explicitly.
+
+## Preserve Exact Output Columns And Row Identity
+
+Follow these rules:
+
+1. Declare the complete output identity before writing the transformation.
+2. Carry every identity column through grouping, sorting, UDF, and explode
+   boundaries.
+3. Require one grouped result per identity before joining it to the base rows.
+4. Join on the complete identity. Do not substitute customer, timestamp, or
+   another non-unique lookup key.
+5. Declare the required output columns and their order. Never use `select("*")`
+   for the final candidate.
+6. Drop helper, duplicate, and intermediate columns at the final projection.
+7. Fail tests and production validation on null, missing, or duplicate
+   identities and on unexpected output columns.
+
+```python
+IDENTITY_COLUMNS = ["event_id"]
+BASE_OUTPUT_COLUMNS = [
+    "event_id",
+    "entity_id",
+    "event_time",
+    "amount_usd",
+]
+FEATURE_COLUMNS = ["event_count_1h", "amount_usd_1h"]
+OUTPUT_COLUMNS = BASE_OUTPUT_COLUMNS + FEATURE_COLUMNS
+
+feature_projection = features_by_event.select(
+    *IDENTITY_COLUMNS,
+    *FEATURE_COLUMNS,
+)
+
+candidate = (
+    events.select(*BASE_OUTPUT_COLUMNS)
+    .join(feature_projection, on=IDENTITY_COLUMNS, how="left")
+    .select(*OUTPUT_COLUMNS)
+)
+
+assert candidate.columns == OUTPUT_COLUMNS
+```
+
+For composite identity, list every component, for example
+`IDENTITY_COLUMNS = ["transaction_id", "transaction_leg_id"]`. Add the
+bidirectional identity and row-parity tests from [Parity Testing](parity-testing.md).