@1aboveio/skills 0.14.0 → 0.16.1

package/skills/backend/pyspark/references/parity-testing.md

@@ -1,83 +0,0 @@
-# Parity Testing
-
-Use this playbook for semantics-preserving Spark rewrites.
-
-## Keep The Old Path As A Bounded Oracle
-
-Retain the previous implementation only in tests while parity is being proven.
-Use small fixtures that are dense enough to expose the old complexity but
-bounded enough to run reliably.
-
-## Compare Full Rows In Both Directions
-
-```python
-expected = old_build(source).select(*OUTPUT_COLUMNS)
-actual = new_build(source).select(*OUTPUT_COLUMNS)
-
-assert expected.exceptAll(actual).count() == 0
-assert actual.exceptAll(expected).count() == 0
-```
-
-Counts alone cannot detect wrong values, swapped identities, or duplicate rows.
-Use `exceptAll`, not `subtract`, when duplicate multiplicity matters.
-
-## Add Literal Boundary Assertions
-
-Parity can be self-confirming if oracle and candidate share the same mistake.
-Assert critical rules directly:
-
-```python
-by_id = {row.event_id: row for row in actual.collect()}
-assert by_id["at-24h"].count_24h == 2       # inclusive lower boundary
-assert by_id["after-24h"].count_24h == 1    # one unit outside
-assert by_id["same-time-b"].count_1h == 2   # stable tie ordering
-```
-
-Keep collected fixtures synthetic and non-sensitive.
-
-## Minimum Fixture Matrix
-
-Cover:
-
-- empty and single-event groups;
-- exact lower/upper boundaries and one unit beyond;
-- same-timestamp events with every tie-breaker;
-- null key, event time, amount, FX, and counterparty;
-- both directions and relevant statuses;
-- repeated and distinct counterparties;
-- state open, partial close, full close, eviction, and re-entry;
-- floating-point add/remove residue;
-- one dense hot key;
-- duplicate and missing output identities.
-
-## Test The Physical Shape
-
-Assert the expensive relation is absent without snapshotting the whole plan:
-
-```python
-plan = actual._jdf.queryExecution().executedPlan().toString().lower()
-assert not ("join" in plan and "event_time" in plan and "interval" in plan)
-```
-
-Prefer a targeted helper that counts known pair-join nodes. Also assert the
-expected grouped-sweep or built-in operator is present when that guards the
-optimization.
-
-## Mutation-Prove Important Tests
-
-Temporarily introduce representative faults and confirm tests fail:
-
-- change `< cutoff` to `<= cutoff`;
-- remove a tie-breaker;
-- use ordinary rolling addition/subtraction;
-- reintroduce the pair join;
-- omit a downstream-required column.
-
-Record red/green evidence for high-risk money, date, and identity semantics.
-
-## Match Test Cost To Ownership
-
-Behavioral Spark tests belong on files that own formulas, source scope,
-history, or state transitions. Unrelated upstream changes may use a focused
-required-column check instead of rerunning an expensive full-history suite.
-Do not demote behavioral coverage for a semantic change.

package/skills/backend/pyspark/references/production-validation.md

@@ -1,166 +0,0 @@
-# Production Validation
-
-Use this playbook before production-scale profiling, canaries, backfills, or
-writes.
-
-## Fail-Closed Admission
-
-Do not launch until all applicable facts are recorded and valid:
-
-- exact commit/module hash;
-- production Spark, Python, Java, and table-format runtimes;
-- source snapshot or date boundary;
-- scheduler pause/ownership state;
-- zero competing warehouse work;
-- explicit validate-only or write-enabled mode;
-- durable writable log and event-log destination;
-- restart/recovery plan for writes.
-
-Classify existing processes and working-tree changes before touching them. Do
-not kill or overwrite another operator's work.
-
-## Run A Validate-Only Canary Before The Writer
-
-Implement the guard in the Spark job, not only in a wrapper. Keep each stage
-literal:
-
-```python
-import json
-
-from pyspark.sql import functions as F
-
-
-def validate_candidate(candidate):
-    # first() is a Spark action: it executes the lazy transform plan.
-    metrics = candidate.agg(
-        F.count("*").alias("row_count"),
-        F.countDistinct("event_id").alias("distinct_event_count"),
-        F.sum(
-            F.when(F.col("event_id").isNull(), 1).otherwise(0)
-        ).alias("null_event_id_count"),
-        F.sum(
-            F.when(F.col("event_count_1h") < 0, 1).otherwise(0)
-        ).alias("invalid_event_count_1h"),
-        F.count("amount_usd_1h").alias("non_null_amount_usd_1h_count"),
-    ).first().asDict()
-
-    blockers = []
-    if metrics["row_count"] != metrics["distinct_event_count"]:
-        blockers.append("event_id is not unique")
-    if metrics["null_event_id_count"]:
-        blockers.append("event_id contains nulls")
-    if metrics["invalid_event_count_1h"]:
-        blockers.append("event_count_1h contains negative values")
-    if blockers:
-        raise RuntimeError(f"validation blocked: {blockers}")
-    return metrics
-
-
-def run(validate_only, source_snapshot):
-    # Bind every action to the same immutable source version or date boundary.
-    source = extract(source_snapshot)
-    candidate = transform(source)      # build a lazy DataFrame plan
-
-    metrics = validate_candidate(candidate)  # execute validation aggregates
-    print(json.dumps(metrics, sort_keys=True))
-
-    if validate_only:                  # canary stops before any writer call
-        print("validate-only: skipping publication")
-        return
-
-    publish(candidate)                 # execute the writer from the same source
-    reconcile_published_scope(metrics) # independently read back aggregates
-```
-
-Validation and publication are separate Spark actions, so Spark may recompute
-the candidate for publication. Pin every source read to the same immutable
-snapshot, table version, or exact date boundary. If the source cannot be pinned,
-the validation result does not prove what the writer will publish; stop and
-redesign the run.
-
-Adapt identity and blocker checks to the table's required behavior. Reference
-every derived output in at least one validation aggregate; otherwise Catalyst
-may prune an unreferenced calculation and the canary will not exercise the full
-candidate. Validation must use aggregate evidence and fail before publication.
-The canary exercises the production-shaped extract, transform, and validation
-path without calling the writer.
-
-## Avoid Explicit Cache And Persistence
-
-Do not add `.cache()` or `.persist()` as a performance fix. They add storage,
-serialization, eviction, and cleanup concerns without improving the logical
-work. Prefer a better transformation shape and a pinned source snapshot.
-
-Treat existing persistence as separate operational behavior: do not expand it
-while optimizing the job, and remove it when repository tests and same-snapshot
-evidence show it is unnecessary.
-
-## Launch Through The Production Entry Point
-
-Use the same `spark-submit` path and driver/executor runtimes as production:
-
-```bash
-export PYSPARK_DRIVER_PYTHON="$DRIVER_PYTHON"
-export PYSPARK_PYTHON="$EXECUTOR_PYTHON"
-
-spark-submit \
-  --name "$CANARY_NAME" \
-  --conf spark.pyspark.driver.python="$DRIVER_PYTHON" \
-  --conf spark.pyspark.python="$EXECUTOR_PYTHON" \
-  --conf spark.eventLog.enabled=true \
-  job.py --validate-only
-```
-
-Do not replace deployed scheduler files with an unmerged staged module. Stage
-an isolated artifact or deploy merged code through the normal path.
-
-## Prove The Canary Did Not Write
-
-Require all three forms of evidence:
-
-1. Validation completed with zero blockers.
-2. The terminal log says publication was skipped.
-3. The event log reports zero Spark output records and bytes.
-
-Parse retained logs with upstream failures visible:
-
-```bash
-set -o pipefail
-hdfs dfs -cat "$EVENT_LOG" | \
-  python {baseDir}/scripts/spark_eventlog_summary.py > canary-summary.json
-```
-
-Record application ID, shell/event-log wall, stage/task metrics, plan nodes,
-shuffle, spill, output, code hash, and snapshot denominator.
-
-## Run Write Verification Separately
-
-Run write-enabled verification only after merge, deployment, successful
-validate-only evidence, and explicit authorization. Re-check admission just
-before launch.
-
-Verify row count, distinct identity count, date/partition coverage, commit
-success, schema, consumer compatibility, validation blockers, write mode,
-single-writer constraints, and downstream readability.
-
-Report compute/validation, publication, and total time separately. Never
-present validate-only time as end-to-end write performance.
-
-If a write run is cancelled, prove from logs/source that publication was not
-reached. YARN `KILLED` alone does not prove no commit occurred.
-
-## Keep Evidence Aggregate-Only
-
-Safe evidence includes counts, null rates, distinct counts, min/max dates,
-hashes, stage metrics, plan-node counts, and commit IDs. Do not print
-credentials, entity rows, account identifiers, or per-entity values.
-
-## Restore Ownership
-
-After the run:
-
-1. Confirm application and wrapper processes are terminal.
-2. Remove agent-owned watchers and temporary processes.
-3. Restore scheduler pause state or ownership.
-4. Separate unrelated downstream failures from the optimized job.
-5. Publish evidence with limitations and unmeasured surfaces.