@1aboveio/skills 0.15.0 → 0.17.0

package/skills/payment/payment-analysis/references/metrics.md

@@ -0,0 +1,62 @@
+# Metrics and denominators
+
+Compute with explicit denominators. Prefer count and amount side by side.
+Amounts are USD per [fx.md](fx.md) before any sum or share.
+
+## Authorization
+
+| KPI | Formula | Notes |
+|---|---|---|
+| Attempts | count of auth rows in window | |
+| Approvals | count where response = approval | |
+| Auth rate | approvals / attempts | Not capture rate |
+| Approved amount | sum(amount \| approval) | USD after [fx.md](fx.md) |
+| Declines | attempts − approvals | Or hard-only if stated |
+| Decline rate | declines / attempts | |
+| Reason share (of declines) | reason_count / declines | Default for mix tables |
+| Reason incidence (of attempts) | reason_count / attempts | Label clearly if used |
+| Country attempt share | country_attempts / attempts | |
+| Country amount share | country_amount / total_amount | Often diverges from attempt share |
+| Country auth rate | country_approvals / country_attempts | Require min-n (e.g. ≥30) for rankings |
+
+Trend grains: daily (`Timestamp` date) and weekly (state week rule, e.g. W-SUN).
+
+Card product: `Payment method` × `Card type` when both exist.
+
+## Settlement
+
+| KPI | Formula | Notes |
+|---|---|---|
+| Sale count / amount | Settlement type = Sale | |
+| Refund count / amount | Settlement type = Refund | Show sign or absolute + label |
+| Net settled amount | sales + refunds (refunds negative) | |
+| Refund rate (count) | refunds / sales | |
+| Refund rate (amount) | abs(refund amount) / sale amount | |
+| Status mix | by Settlement status | |
+
+Do not equalize settled sales to approved auth amount without a timing caveat.
+
+## Chargebacks
+
+Monthly **拒付率** by brand is [chargebacks.md](chargebacks.md): Visa
+same-month sales, Mastercard previous-month sales, skip empty brand-months,
+no blended Visa+Mastercard ratio. The table below is mix / match, not that
+programme rate.
+
+| KPI | Formula | Notes |
+|---|---|---|
+| CB count / dispute amount | as filed | Amounts USD after [fx.md](fx.md); use abs for magnitude if signed |
+| Reason mix | by Reason category / code | Fraud ≠ all CBs |
+| Match rate to auths | matched / CB rows | State join key |
+| Visa monthly 拒付率 | CB_Visa(M) / Sales_Visa(M) | Received month; not VAMP unless TC40 in file |
+| Mastercard monthly 拒付率 | CB_MC(M) / Sales_MC(M−1) | Skip M if M−1 sales missing |
+
+Lag: chargeback date is later than original auth. Programme rates use
+**received** month, not origination vintage.
+
+## Ranking and “Other”
+
+- Rank countries by attempts unless the user asks for amount.
+- Roll the long tail into `Other` after top 10–15; show Other share.
+- For decline-by-country matrices, use top countries by attempts and top reasons
+  globally; collapse remaining reasons to `Other declines`.

package/skills/payment/payment-analysis/references/report-template.md

@@ -0,0 +1,98 @@
+# Report template
+
+Save:
+
+```text
+reports/payment-analysis/<scope>_overview.md
+reports/payment-analysis/<scope>_overview.zh.md
+reports/payment-analysis/<scope>_overview.zh.html   # preferred when charts
+```
+
+Follow [visualization.md](visualization.md), [journey.md](journey.md),
+[terminology.md](terminology.md), and [fx.md](fx.md) for amount currency. Omit
+topic sections whose layer is missing.
+No Markdown bold in report body. After writing, grep banned strings in
+terminology.md.
+
+## Required structure
+
+English Markdown headings on the left; Chinese HTML / `.zh.md` on the right.
+
+```markdown
+# Payment Overview — <window>          # 支付业务概览 — <window>
+
+## 1. Executive summary                 # 1. 执行摘要
+## 2. Top statistics                    # 2. 核心指标
+## 3. Payment journey (Sankey)          # 3. 支付旅程（Sankey）
+## 4. Topics                            # 4. 专题
+### 4.1 Overall transaction volume      # 4.1 整体交易量
+### 4.2 Week-on-week trend              # 4.2 周趋势
+### 4.3 Authorization rate              # 4.3 授权成功率
+### 4.4 Contribution by BIN country     # 4.4 BIN 国家贡献
+### 4.5 Decline by reason code          # 4.5 授权拒绝原因贡献
+### 4.6 Decline by BIN country          # 4.6 各国授权拒绝结构
+### 4.7 Settlements (when present)      # 4.7 清算
+### 4.8 Chargebacks (optional)          # 4.8 拒付
+## 5. Operational impact                # 5. 运营影响
+## 6. Appendix                          # 6. 附录
+```
+
+HTML uses the Chinese headings. KPI labels, legends, and Sankey node names use
+the glossary in terminology.md (`授权尝试`, `授权成功`, `授权拒绝`,
+`发卡行国家`, `差错`, `产品服务质量`, `Do not honor`).
+
+## Chapter requirements
+
+### 1. Executive summary / 执行摘要
+
+≤5 numbered findings: volume, trend, auth rate, country concentration, main
+decline or settlement/CB driver.
+
+### 2. Top statistics / 核心指标
+
+KPI cards / table from auth + settlement: attempts, approvals, auth rate,
+attempt amount, approved amount, sale count/amount, refund count/amount, net
+settled, distinct PAN/BIN/countries, portfolio constants (MID, MCC, provider).
+
+Chinese labels: 授权尝试, 授权成功, 授权成功率, 授权成功金额, 清算销售, 退款,
+净清算, 发卡行国家.
+
+### 3. Payment journey (Sankey) / 支付旅程
+
+Count Sankey per [journey.md](journey.md). Caption lists skipped checkpoints
+(e.g. risk decision / 3DS when authentication fields are empty).
+
+Collapsed nodes when 3DS/risk are absent:
+
+`开始 → 授权 → 授权成功 / 授权拒绝 → 已清算销售 / 授权成功未清算`
+
+### 4. Topics / 专题
+
+Each topic: short lede + chart(s) + supporting table.
+
+| Topic | Minimum viz |
+|---|---|
+| 4.1 Volume | count + amount bars (dual axis) |
+| 4.2 WoW trend | weekly count + amount bars; auth-rate line |
+| 4.3 Auth rate | rate line by day and/or segment |
+| 4.4 BIN country | count + amount bars dual axis; rate line |
+| 4.5 Decline reason | count bars |
+| 4.6 Decline by country | stacked count bars |
+| 4.7 Settlements | sale/refund amount bars; country contribution dual axis |
+| 4.8 Chargebacks | reason-category bars; Visa / Mastercard monthly 拒付率 line per [chargebacks.md](chargebacks.md) (note under the rate chart; skip empty brand-months) |
+
+Use both authorization and settlement data when both files are present.
+
+### 5. Operational impact / 运营影响
+
+Numbered operational takeaways. Do not title this 运营含义. Do not invent
+supervised fraud rules. Choose from
+[auth-rate-actions.md](auth-rate-actions.md) only when the extract supports
+the trigger (SCA → 3DS; high 疑似欺诈 country → 风控拦截 or 3DS; 软性拒绝 →
+retry).
+
+### 6. Appendix / 附录
+
+Definitions (from terminology.md) and caveats (FX per [fx.md](fx.md),
+issuer country ≠ ship-to, auth rate ≠ capture, journey skips,
+chargeback rates are network-style estimates).

package/skills/payment/payment-analysis/references/terminology.md

@@ -0,0 +1,85 @@
+# Terminology
+
+Use these terms consistently in payment-analysis reports. Read this file before
+writing English or Chinese copy. After writing, run the proofread step in
+`SKILL.md`.
+
+## Canonical glossary
+
+| English | Chinese | Do not write |
+|---|---|---|
+| Attempt | 授权尝试 | 单独的「尝试」；交易；conversion |
+| Approval | 授权成功 | 批准；成功（易与清算混淆） |
+| Auth rate | 授权成功率 | 转化率；单独的「成功率」（首次出现时） |
+| Decline | 授权拒绝 | 全部叫欺诈 |
+| Hard / soft / reject | 硬性拒绝 / 软性拒绝 / 风控拦截 | hard 拦截（指拒绝类型时） |
+| Reason / reason code | 拒绝原因 / 原因码 | |
+| Insufficient funds | 余额不足 | 余额 |
+| Suspected fraud | 疑似欺诈 | 已确认欺诈；正文里写 D102（表内 Pazien 代码可保留） |
+| Invalid expiration date | 有效期无效 | |
+| Security code invalid | 安全码无效 | |
+| Policy reasons | 政策原因 | |
+| Do not honor | Do not honor | 拒绝承兑 |
+| BIN / IIN | BIN / IIN | |
+| BIN country | 发卡行国家 / BIN 国家 | 客户国家；收货国；发卡国 |
+| Settlement | 清算 | |
+| Sale / Refund | 销售 / 退款 | 把销售金额写成清算金额 |
+| Chargeback / dispute | 拒付 / 争议 | 与清算混用 |
+| Clerical | 差错 | 文书；文书错误 |
+| Quality | 产品服务质量 | 质量；质量争议 |
+| Incidence | 发生率（必须写出分母） | |
+| GMV (loose) | 销售金额合计（说明是否含退款） | |
+| Week-on-week trend | 周趋势 | 周环比（除非单元格是环比百分比） |
+| Operational impact | 运营影响 | 运营含义 |
+| Ticket size | 客单价 | |
+| Debit / credit | 借记卡 / 贷记卡 | 借记；贷记 |
+| Payment journey | 支付旅程 | 支付旅程 Sankey；Sankey 旅程（标题/结构行） |
+| This analysis window | 当前分析时段内 | 本窗口 |
+| SCA | SCA | 强客户认证（可在首次括注后只用 SCA） |
+| Authentication / 3DS | 鉴权 / 认证 / 3DS | 把 3DS 叫转化 |
+| Visa / Mastercard 拒付率 | Visa / Mastercard 拒付率 | VAMP ratio（无 TC40 时）；ECM（从本抽取断言商户在项目中） |
+| Amounts in USD | 金额均为 USD | 未做汇率折算（当全部已是 USD） |
+| Count-heavy / amount-light | 授权尝试笔数多但金额小 | 走廊；笔数负担型走廊 |
+
+Keep proper nouns in source form when useful: Visa, MasterCard, BIN, 3DS,
+Sankey (in chart captions only), Order ID, MCC, Worldpay, Pazien, MID.
+Do not use Pazien codes (e.g. D102) in prose; use 疑似欺诈 / suspected fraud.
+Tables may still show the code column.
+
+## Axis and UI exceptions
+
+- Rate axis may say `成功率 %` if the legend or lede already says 授权成功率.
+  Amount axes still use the full metric plus currency
+  ([visualization.md](visualization.md)).
+- Week-over-week *percentage* columns may say 笔数周环比 / 金额周环比.
+- Product rankings that exclude tiny samples: write
+  `在授权尝试超过 50 笔的卡种里` (or the n actually used), never
+  `有规模的最弱卡种` or `弱卡种`.
+- If every amount is USD, write `金额均为 USD`. Do not add `未做汇率折算`.
+  Only mention missing FX conversion when mixed currencies were actually summed
+  or dropped.
+- Chart amount fields append the currency: `授权拒绝金额 USD`.
+- HTML footer: data-source filenames only. No skill name, no Chart.js / ECharts.
+- Do not call issuer country “customer country”.
+- Do not call auth rate “conversion” unless contrasting checkout vs issuer
+  response, and then say so.
+
+## Banned strings (grep after writing)
+
+Chinese: `走廊`, `杠杆`, `弱转化`, `转化率`, `赋能`, `抓手`, `闭环`, `打法`,
+`运营含义`, `本窗口`, `弱卡种`, `有规模的最弱`, `强客户认证`, `拒绝承兑`,
+`文书错误`, `质量争议`, `发卡国`
+
+English: `auth drag`, `conversion problem`, `count-heavy`, `amount-light`,
+`Protect US amount`, `Unlike currencies`
+
+If a hit is a legitimate exception (e.g. 周环比 as a table header for WoW %,
+or D102 in a reason-code table), leave it and do not expand the ban.
+
+## Proofread
+
+1. Apply the glossary to headings, KPI labels, chart legends, Sankey skip
+   chips, and body copy.
+2. Grep the report files for the banned strings.
+3. Replace hits using this table. Do not change numbers, formulas, or file
+   paths.

package/skills/payment/payment-analysis/references/visualization.md

@@ -0,0 +1,47 @@
+# Visualization rules
+
+Preferred deliverable: self-contained HTML with Chart.js (KPI charts) and
+ECharts (Sankey). Markdown tables remain required beside charts.
+
+## Mark encoding
+
+| Metric kind | Chart mark | Notes |
+|---|---|---|
+| Count | Bar | Vertical or horizontal |
+| Amount | Bar | Never encode amount as a line |
+| Rate | Line | Percent on its own axis |
+
+## Multi-metric charts
+
+1. Count and amount in the same chart → both are **bars** on **different axes**
+   (left = count, right = amount).
+2. Rate with count and/or amount → rate is a **line** on a dedicated % axis.
+3. Rate alone → line chart (points allowed). No bars for a lone rate series.
+4. Composition of one total → doughnut or stacked bars; not for time trends.
+
+## Axis hygiene
+
+- Label every axis with the full metric name. Amount axes/legends include
+  currency (`授权尝试金额 USD`, `授权拒绝金额 USD`, `清算销售金额 USD`,
+  `拒付金额 USD`). Do not use bare `笔数` / `金额` / `%`.
+- Only one axis draws the main grid on dual-axis charts.
+- Stacked bars = mix within a group (e.g. decline reasons by country), never rate.
+- Sort categories by a stated order (usually attempts desc).
+
+## Required charts (HTML)
+
+1. Payment journey Sankey — see [journey.md](journey.md)
+2. Volume trend (WoW / daily): count bar + amount bar (dual axis) + auth-rate line
+3. Authorization rate by segment (card product and/or country): rate as line
+4. BIN-country contribution: count bar + amount bar (dual axis)
+5. Decline by reason: count bars (+ amount bars/dual axis if shown)
+6. Decline mix by BIN country: stacked count bars
+
+## Anti-patterns
+
+- Line for amount or count
+- Bar for a standalone rate series
+- Count and amount on one axis without dual-axis scaling
+- Mixing native EUR (etc.) with USD in one amount series; convert first ([fx.md](fx.md))
+- Plotting a 0% chargeback-rate point for a brand-month with no CBs ([chargebacks.md](chargebacks.md))
+- Inventing Sankey checkpoints not present in the data

package/skills/backend/pyspark/SKILL.md

@@ -1,116 +0,0 @@
----
-name: pyspark
-description: Implement, debug, review, and optimize Apache Spark or PySpark batch jobs, ETL pipelines, feature calculations, and Hudi/warehouse writes. Use whenever the task involves Spark DataFrames or Spark SQL, joins, windows, grouped state, UDFs, shuffle, skew, spill, slow stages, driver/executor OOM, Spark Connect, spark-submit, Hudi commits, backfills, bidirectional parity tests, validate-only canaries, or Spark event-log profiling — including when the user only mentions a warehouse job timeout, empty Hudi write, or performance regression on an EMR/Dataproc job.
----
-
-# PySpark
-
-Use this skill as the workflow and routing index. Read the referenced playbooks
-that match the task before editing code or running a production-scale job.
-
-## Workflow
-
-```text
-contract -> transform -> materialize -> test -> CI/CD
-```
-
-Follow all five steps in order.
-
-### 1. Contract
-
-- **Input:** declare all parameters, defaults, validation, source snapshot/date
-  scope, overlapping-run behavior, idempotency/single-writer rules, and exact
-  Python and PySpark versions.
-- **Output:** declare plain Spark versus Hudi table type, output identity/schema,
-  partition columns, record key, deterministic order or precombine field, and
-  append/upsert/overwrite write mode.
-- **Conventions:** declare native Spark versus Spark Connect and full rebuild
-  versus incremental processing, including the incremental watermark,
-  lookback/checkpoint, and restart boundary when applicable.
-
-Read [ETL Contract](references/etl-contract.md). When the repository has no ETL
-base module, adapt [`assets/templates/etl.py`](assets/templates/etl.py).
-
-### 2. Transform
-
-- Make `transform` accept DataFrames and return the complete candidate without
-  invoking validation or a writer.
-- Estimate rows after joins/explodes/groups and measure maximum rows per key.
-  Fix algorithmic expansion and hot keys before tuning Spark configuration.
-- Prefer built-in expressions and windows. Use exact bucketing or one sorted
-  grouped sweep only for state that built-ins cannot express cleanly.
-- Preserve complete row identity, ordering, output columns, and required
-  behavior.
-- Avoid explicit `.cache()` and `.persist()` by default.
-
-Read [Transformation Design](references/transformation-design.md). For temporal
-windows or FIFO paths, also read
-[Velocity Feature Calculation](references/velocity-feature-calculation.md).
-
-### 3. Materialize
-
-- Treat materialization as the Spark action that executes the lazy candidate,
-  not as caching.
-- Use one aggregate validation action before every writer. Reference every
-  derived output so Catalyst cannot prune untested calculations.
-- Check row count, distinct/null/duplicate identities, domain blockers, and
-  required date/partition scope. Fail closed on any blocker.
-- Bind validation and publication to the same immutable source snapshot because
-  Spark may recompute the candidate.
-- Keep the validate-only guard inside the job and before every writer. Do not
-  add `.count()`, `.cache()`, or `.persist()` to warm execution.
-
-Read [Production Validation](references/production-validation.md).
-
-### 4. Test
-
-1. **Local test:** run pure state, tiny real-Spark, plan-shape, focused repository,
-   and bidirectional parity tests. Cover boundaries, ties, nulls, arithmetic,
-   identity, and precedence.
-2. **Cluster canary test:** run the production entry point and source scope with
-   `--validate-only`. Require zero blockers, an explicit publication skip, zero
-   Spark output records/bytes, and readable driver/event logs.
-3. **Profiling:** measure an isolated baseline, parse the canary event log, and
-   compare both runs on the same source scope and resource profile. Record
-   wall/stage/task time, cardinality, shuffle, spill, skew, output, and target
-   plan patterns.
-
-```bash
-spark-submit <production-options> job.py --validate-only
-set -o pipefail
-hdfs dfs -cat "$EVENT_LOG" | \
-  python {baseDir}/scripts/spark_eventlog_summary.py > canary-summary.json
-```
-
-Reject missing evidence, incomparable scope, unexplained regressions, or a
-missed performance gate. Keep validate-only time separate from write time.
-
-Read [Parity Testing](references/parity-testing.md),
-[Diagnosis And Profiling](references/diagnosis-and-profiling.md), and
-[Production Validation](references/production-validation.md).
-
-### 5. CI/CD
-
-1. Open a PR with required-behavior, test, canary, profiling, and risk evidence.
-2. Run required CI and independent review; fix failures.
-3. Enqueue through the configured merge queue and merge without bypassing gates.
-4. Deploy only the merged commit through the normal deployment path.
-5. Run write-enabled production validation only with explicit authorization;
-   reconcile counts, identity, schema, coverage, blockers, readability, write
-   mode, and commit evidence independently.
-6. Restore scheduler/process ownership and report compute/validation,
-   publication, and end-to-end time separately.
-
-Read [Production Validation](references/production-validation.md).
-
-## Reference Index
-
-| Need | Read or run |
-|---|---|
-| Define or review an ETL job contract | [ETL Contract](references/etl-contract.md) and `assets/templates/etl.py` |
-| Diagnose or profile | [Diagnosis And Profiling](references/diagnosis-and-profiling.md) |
-| Design transformations | [Transformation Design](references/transformation-design.md) |
-| Optimize temporal/FIFO state | [Velocity Feature Calculation](references/velocity-feature-calculation.md) |
-| Prove parity | [Parity Testing](references/parity-testing.md) |
-| Run canaries or writes | [Production Validation](references/production-validation.md) |
-| Summarize an event log | `scripts/spark_eventlog_summary.py` |

package/skills/backend/pyspark/references/parity-testing.md

@@ -1,83 +0,0 @@
-# Parity Testing
-
-Use this playbook for semantics-preserving Spark rewrites.
-
-## Keep The Old Path As A Bounded Oracle
-
-Retain the previous implementation only in tests while parity is being proven.
-Use small fixtures that are dense enough to expose the old complexity but
-bounded enough to run reliably.
-
-## Compare Full Rows In Both Directions
-
-```python
-expected = old_build(source).select(*OUTPUT_COLUMNS)
-actual = new_build(source).select(*OUTPUT_COLUMNS)
-
-assert expected.exceptAll(actual).count() == 0
-assert actual.exceptAll(expected).count() == 0
-```
-
-Counts alone cannot detect wrong values, swapped identities, or duplicate rows.
-Use `exceptAll`, not `subtract`, when duplicate multiplicity matters.
-
-## Add Literal Boundary Assertions
-
-Parity can be self-confirming if oracle and candidate share the same mistake.
-Assert critical rules directly:
-
-```python
-by_id = {row.event_id: row for row in actual.collect()}
-assert by_id["at-24h"].count_24h == 2       # inclusive lower boundary
-assert by_id["after-24h"].count_24h == 1    # one unit outside
-assert by_id["same-time-b"].count_1h == 2   # stable tie ordering
-```
-
-Keep collected fixtures synthetic and non-sensitive.
-
-## Minimum Fixture Matrix
-
-Cover:
-
-- empty and single-event groups;
-- exact lower/upper boundaries and one unit beyond;
-- same-timestamp events with every tie-breaker;
-- null key, event time, amount, FX, and counterparty;
-- both directions and relevant statuses;
-- repeated and distinct counterparties;
-- state open, partial close, full close, eviction, and re-entry;
-- floating-point add/remove residue;
-- one dense hot key;
-- duplicate and missing output identities.
-
-## Test The Physical Shape
-
-Assert the expensive relation is absent without snapshotting the whole plan:
-
-```python
-plan = actual._jdf.queryExecution().executedPlan().toString().lower()
-assert not ("join" in plan and "event_time" in plan and "interval" in plan)
-```
-
-Prefer a targeted helper that counts known pair-join nodes. Also assert the
-expected grouped-sweep or built-in operator is present when that guards the
-optimization.
-
-## Mutation-Prove Important Tests
-
-Temporarily introduce representative faults and confirm tests fail:
-
-- change `< cutoff` to `<= cutoff`;
-- remove a tie-breaker;
-- use ordinary rolling addition/subtraction;
-- reintroduce the pair join;
-- omit a downstream-required column.
-
-Record red/green evidence for high-risk money, date, and identity semantics.
-
-## Match Test Cost To Ownership
-
-Behavioral Spark tests belong on files that own formulas, source scope,
-history, or state transitions. Unrelated upstream changes may use a focused
-required-column check instead of rerunning an expensive full-history suite.
-Do not demote behavioral coverage for a semantic change.

package/skills/backend/pyspark/references/production-validation.md

@@ -1,166 +0,0 @@
-# Production Validation
-
-Use this playbook before production-scale profiling, canaries, backfills, or
-writes.
-
-## Fail-Closed Admission
-
-Do not launch until all applicable facts are recorded and valid:
-
-- exact commit/module hash;
-- production Spark, Python, Java, and table-format runtimes;
-- source snapshot or date boundary;
-- scheduler pause/ownership state;
-- zero competing warehouse work;
-- explicit validate-only or write-enabled mode;
-- durable writable log and event-log destination;
-- restart/recovery plan for writes.
-
-Classify existing processes and working-tree changes before touching them. Do
-not kill or overwrite another operator's work.
-
-## Run A Validate-Only Canary Before The Writer
-
-Implement the guard in the Spark job, not only in a wrapper. Keep each stage
-literal:
-
-```python
-import json
-
-from pyspark.sql import functions as F
-
-
-def validate_candidate(candidate):
-    # first() is a Spark action: it executes the lazy transform plan.
-    metrics = candidate.agg(
-        F.count("*").alias("row_count"),
-        F.countDistinct("event_id").alias("distinct_event_count"),
-        F.sum(
-            F.when(F.col("event_id").isNull(), 1).otherwise(0)
-        ).alias("null_event_id_count"),
-        F.sum(
-            F.when(F.col("event_count_1h") < 0, 1).otherwise(0)
-        ).alias("invalid_event_count_1h"),
-        F.count("amount_usd_1h").alias("non_null_amount_usd_1h_count"),
-    ).first().asDict()
-
-    blockers = []
-    if metrics["row_count"] != metrics["distinct_event_count"]:
-        blockers.append("event_id is not unique")
-    if metrics["null_event_id_count"]:
-        blockers.append("event_id contains nulls")
-    if metrics["invalid_event_count_1h"]:
-        blockers.append("event_count_1h contains negative values")
-    if blockers:
-        raise RuntimeError(f"validation blocked: {blockers}")
-    return metrics
-
-
-def run(validate_only, source_snapshot):
-    # Bind every action to the same immutable source version or date boundary.
-    source = extract(source_snapshot)
-    candidate = transform(source)      # build a lazy DataFrame plan
-
-    metrics = validate_candidate(candidate)  # execute validation aggregates
-    print(json.dumps(metrics, sort_keys=True))
-
-    if validate_only:                  # canary stops before any writer call
-        print("validate-only: skipping publication")
-        return
-
-    publish(candidate)                 # execute the writer from the same source
-    reconcile_published_scope(metrics) # independently read back aggregates
-```
-
-Validation and publication are separate Spark actions, so Spark may recompute
-the candidate for publication. Pin every source read to the same immutable
-snapshot, table version, or exact date boundary. If the source cannot be pinned,
-the validation result does not prove what the writer will publish; stop and
-redesign the run.
-
-Adapt identity and blocker checks to the table's required behavior. Reference
-every derived output in at least one validation aggregate; otherwise Catalyst
-may prune an unreferenced calculation and the canary will not exercise the full
-candidate. Validation must use aggregate evidence and fail before publication.
-The canary exercises the production-shaped extract, transform, and validation
-path without calling the writer.
-
-## Avoid Explicit Cache And Persistence
-
-Do not add `.cache()` or `.persist()` as a performance fix. They add storage,
-serialization, eviction, and cleanup concerns without improving the logical
-work. Prefer a better transformation shape and a pinned source snapshot.
-
-Treat existing persistence as separate operational behavior: do not expand it
-while optimizing the job, and remove it when repository tests and same-snapshot
-evidence show it is unnecessary.
-
-## Launch Through The Production Entry Point
-
-Use the same `spark-submit` path and driver/executor runtimes as production:
-
-```bash
-export PYSPARK_DRIVER_PYTHON="$DRIVER_PYTHON"
-export PYSPARK_PYTHON="$EXECUTOR_PYTHON"
-
-spark-submit \
-  --name "$CANARY_NAME" \
-  --conf spark.pyspark.driver.python="$DRIVER_PYTHON" \
-  --conf spark.pyspark.python="$EXECUTOR_PYTHON" \
-  --conf spark.eventLog.enabled=true \
-  job.py --validate-only
-```
-
-Do not replace deployed scheduler files with an unmerged staged module. Stage
-an isolated artifact or deploy merged code through the normal path.
-
-## Prove The Canary Did Not Write
-
-Require all three forms of evidence:
-
-1. Validation completed with zero blockers.
-2. The terminal log says publication was skipped.
-3. The event log reports zero Spark output records and bytes.
-
-Parse retained logs with upstream failures visible:
-
-```bash
-set -o pipefail
-hdfs dfs -cat "$EVENT_LOG" | \
-  python {baseDir}/scripts/spark_eventlog_summary.py > canary-summary.json
-```
-
-Record application ID, shell/event-log wall, stage/task metrics, plan nodes,
-shuffle, spill, output, code hash, and snapshot denominator.
-
-## Run Write Verification Separately
-
-Run write-enabled verification only after merge, deployment, successful
-validate-only evidence, and explicit authorization. Re-check admission just
-before launch.
-
-Verify row count, distinct identity count, date/partition coverage, commit
-success, schema, consumer compatibility, validation blockers, write mode,
-single-writer constraints, and downstream readability.
-
-Report compute/validation, publication, and total time separately. Never
-present validate-only time as end-to-end write performance.
-
-If a write run is cancelled, prove from logs/source that publication was not
-reached. YARN `KILLED` alone does not prove no commit occurred.
-
-## Keep Evidence Aggregate-Only
-
-Safe evidence includes counts, null rates, distinct counts, min/max dates,
-hashes, stage metrics, plan-node counts, and commit IDs. Do not print
-credentials, entity rows, account identifiers, or per-entity values.
-
-## Restore Ownership
-
-After the run:
-
-1. Confirm application and wrapper processes are terminal.
-2. Remove agent-owned watchers and temporary processes.
-3. Restore scheduler pause state or ownership.
-4. Separate unrelated downstream failures from the optimized job.
-5. Publish evidence with limitations and unmeasured surfaces.