@1aboveio/skills 1.0.4 → 1.0.6

package/README.md

@@ -31,7 +31,7 @@ Each package also publishes on its own; install either this bundle or the packag
 | `engineering` | `@1aboveio/engineering-skills` | better-review, contributing, e2e-test, engineering-runtime, ensure-coverage, golden-path, harness-runtime, resolve-issues, resolve-release, rush-issues, rush-release, test-quality |
 | `cicd` | `@1aboveio/cicd-skills` | cloud-build, cloud-debug, cloud-deploy, docker, github-actions, github-flow, google-cloud, mergify, podman |
 | `app-dev` | `@1aboveio/app-dev-skills` | airflow, android-foundation, app-debug, better-auth, data-catalog, data-quality, ios-foundation, monorepo, nextjs-fullstack, prisma-setup, pyspark, python-backend, shadcn, zod-v4 |
-| `risk-compliance` | `@1aboveio/risk-compliance-skills` | compliance-audit, contract-compare, contract-review, doc-authenticity, fraud-analysis, payment-analysis, report-writing, similar-domain, tld-detection |
+| `risk-compliance` | `@1aboveio/risk-compliance-skills` | compliance-audit, contract-compare, contract-review, doc-authenticity, fraud-analysis, payment-analysis, similar-domain, tld-detection |
 | `content` | `@1aboveio/content-skills` | article-review, article-to-image, flowchart-svg, image-gen, mermaid, wechat-article-writer, wechat-publisher, wechat-search, xai-search |
 
 ## Third-party sources

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1aboveio/skills",
-  "version": "1.0.4",
+  "version": "1.0.6",
   "description": "Every 1aboveio skill package in one: install once, then choose which packages Pi loads with npx @1aboveio/skills select.",
   "license": "MIT",
   "repository": {
@@ -123,7 +123,6 @@
           "doc-authenticity",
           "fraud-analysis",
           "payment-analysis",
-          "report-writing",
           "similar-domain",
           "tld-detection"
         ]
@@ -252,7 +251,7 @@
         ]
       }
     ],
-    "digest": "41ecf581db6ac404ebca3cfac75779b62cb19ea98fa5ad8c9791b236fecc639e"
+    "digest": "efb93166fdfdb1405f9a596f79d068c34b10a09ff176dc8f51021d64bd80e83d"
   },
   "publishConfig": {
     "registry": "https://registry.npmjs.org/",

package/skills/app-dev/airflow/SKILL.md

@@ -2,7 +2,7 @@
 name: airflow
 description: Airflow DAG architecture and conventions. Use when designing, writing, reviewing, testing, or debugging Airflow DAGs and tasks, including TaskGroups, data intervals, timezone-aware date parameters, CLI validation, environment-based scheduling, and pytest integrity tests.
 user-invocable: true
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Airflow DAG Development Patterns

package/skills/app-dev/android-foundation/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: android-foundation
 description: "Prove an Expo / React Native app on a native Android emulator, and fix it when that proof fails. Use before you fix and re-run or re-dispatch a failed Android CI job (`gh workflow run android.yml`, `gh run rerun`): a re-run is not a debugger, so reproduce the failing journey checkpoint off CI first. Also use when a UiAutomator journey or its Python harness tests fail (a CHECK/FAIL line, a wait that timed out, a control not found, adb offline, a lost Metro bridge, a fixture failing on the runner), to build the debug APK and run the device journey, to set up a fresh Android workspace (SDK/NDK/JDK, Gradle caches, private AVD, disposable PostgreSQL), or to add the emulator job to CI. NOT for workflow YAML, CI speed or runner design (github-actions), browser tests (e2e-test), iOS (ios-foundation), plain Jest component tests, or signing and store release."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Android foundation

package/skills/app-dev/app-debug/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: app-debug
 description: "Diagnose runtime failures in deployed Cloud Run services — 5xx/422 errors, tracebacks, Next.js SSR/hydration or Server Action failures, DB/Secret Manager connection errors, cold-start timeouts, container OOM. NOT for build or deploy failures (cloud-debug)."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # App Debug

package/skills/app-dev/better-auth/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: better-auth
 description: "Add or debug Better Auth for Next.js App Router: login, sessions, protected routes, middleware, OAuth (including Feishu/Lark), cookies, Prisma, and Auth.js migration."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Better Auth for Next.js

package/skills/app-dev/data-catalog/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: data-catalog
 description: "Column definitions for warehouse tables, written inside the Spark/PySpark or SQL script that builds each table (CATALOG_TABLE / CATALOG_FIELDS, or a `-- @field` comment block), with a generated markdown catalog in docs/catalog/ and a CI check that fails a PR whose catalog is stale. Use when writing or changing a table script that adds, renames, removes, or changes the meaning of an output column or feature; when asked what a warehouse column or feature means; to set a repo up ('init the data catalog', 'generate the data dictionary', '字段字典', 'add catalog CI'); or to audit existing SQL or Spark scripts for undocumented columns. NOT for implementing or tuning the Spark job itself (pyspark), DAG wiring (airflow), or general workflow YAML (github-actions)."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Data catalog

package/skills/app-dev/data-quality/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: data-quality
 description: "Check a published warehouse table's data against its data-catalog definitions with read-only SELECTs: columns never filled or stuck at one value, counts below zero, ratios outside 0–1, declared values that never appear and undeclared values that do. Then classify each finding as a wrong definition, wrong code, or wrong data. Use when asked whether a table's data is right, to profile or sanity-check a dwd/ads table, to find dead or constant columns, to check a column's values against its documented list, after a pipeline change ships, or when a data-catalog definition and the data might disagree. NOT for validating a Spark change before it ships (pyspark canary and reconciliation), writing definitions (data-catalog), or monitoring dashboards."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Data quality

package/skills/app-dev/ios-foundation/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: ios-foundation
 description: "Set up and prove an Expo / React Native native iOS workspace from a clean checkout to a manual Login smoke on an owned iOS Simulator: isolated worktree and private caches, Xcode/runtime/CocoaPods preflight, frozen install with `pod install --deployment`, platform-scoped prebuild, an ad-hoc-signed Debug simulator build, a private simulator by exact UDID, Metro on a free port, a disposable PostgreSQL backend with owned cleanup, and the evidence record (setup proof only; feature work reports the smoke result). Use for 'set up iOS', 'build the iOS app', 'run the app on the iOS simulator', 'add iOS to the React Native app', 'prove the iOS build', or when a pod install, xcodebuild, simulator boot, Metro connection or Keychain session fails. NOT for Android (android-foundation), signing, TestFlight or App Store release, or general workflow YAML (github-actions)."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # iOS foundation

package/skills/app-dev/monorepo/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: monorepo
 description: Turborepo architecture for shared packages, pnpm workspaces, DAL boundaries, and pipelines.
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Turborepo Monorepo Architecture

package/skills/app-dev/nextjs-fullstack/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: nextjs-fullstack
 description: "Build or debug Next.js App Router features with React 19 — pages/layouts/route handlers, Server/Client Components, Server Actions, forms, middleware, DAL, strict TypeScript, Zod — plus our conventions: nuqs URL state, SWR cache, Skeleton/Progress + Sonner feedback, drag-and-drop upload, the portal app shell, list pages as filter/sort/paginate tables. shadcn component mechanics: see the shadcn skill."
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Next.js 16 Fullstack Development

package/skills/app-dev/prisma-setup/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: prisma-setup
 description: Use when adding or debugging Prisma with PostgreSQL, Next.js, or Turborepo, including schema.prisma, prisma.config.ts, client generation, migrations, seeds, relations, adapters, and DAL/database-layer setup. Triggers on phrases like "Prisma schema", "prisma migrate", "schema.prisma", "seed the database", "PrismaClient error", "relation issue", or "database layer".
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # Prisma v7 Setup Guide

package/skills/app-dev/pyspark/SKILL.md

@@ -4,15 +4,15 @@ description: >
   Implement, debug, review, and optimize Apache Spark or PySpark batch jobs,
   ETL pipelines, feature calculations, and Hudi/warehouse writes. Follow
   contract → transform → materialize → local test → production validation
-  (one-month canary, optional profile if that canary exceeds 5 minutes without
-  resource constraints or any job exceeds 30 minutes, authorized write, cluster
-  reconciliation). Use whenever
+  (small canary, measured adaptive batching for every backfill/rebuild, optional
+  profile if that canary exceeds 5 minutes without resource constraints or any
+  job exceeds 30 minutes, authorized write, cluster reconciliation). Use whenever
   the task involves Spark DataFrames or Spark SQL, joins, windows, grouped
   state, UDFs, shuffle, skew, spill, slow stages, driver/executor OOM, Spark
   Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
   cluster reconciliation, or Spark event-log profiling — including warehouse
   job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
-release-version: 0.41.0
+release-version: 0.42.1
 ---
 **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
 # PySpark
@@ -20,6 +20,24 @@ release-version: 0.41.0
 Use this skill as the workflow and routing index. Read the playbook for the
 step you are on before editing code or submitting to the cluster.
 
+## Companion Skills
+
+Read the matching companion skill before doing its work:
+
+| When | Skill and handoff |
+|---|---|
+| Adding, renaming, removing, or changing the meaning of an output column/feature; explaining a column; auditing missing definitions | [data-catalog](../data-catalog/SKILL.md): read existing definitions first. When editing output columns, update definitions in the table builder, regenerate the catalog, and run drift checks in the same change. Backfill/rebuild wrappers share the builder's definitions; do not create a second dictionary. |
+| Checking a published table against its definitions, including after a pipeline change ships; investigating empty/constant columns, invalid ranges, missing declared values, or observed undeclared values | [data-quality](../data-quality/SKILL.md): use bounded, aggregate-only, read-only checks. Read the builder and classify findings as definition, code, data, or expected behavior; report rather than automatically repair. |
+
+If a published-data check lacks definitions, follow data-catalog first, respecting
+any planning-only or no-repository-change instruction. These handoffs do not
+require catalog setup or warehouse scans for every Spark task. A performance-only
+rewrite that preserves column meaning does not require rewritten definitions.
+
+Pre-publication tests, aggregate candidate validation, validate-only canaries,
+and independent write reconciliation remain this skill's responsibility.
+Data-quality checks do not replace those gates or authorize a production write.
+
 ## Workflow
 
 ```text
@@ -39,9 +57,18 @@ Follow all five steps in order.
 - **Conventions:** declare native Spark versus Spark Connect and full rebuild
   versus incremental processing, including the incremental watermark,
   lookback/checkpoint, and restart boundary when applicable.
-
-Read [ETL Contract](references/etl-contract.md). When the repository has no ETL
-base module, adapt [`assets/templates/etl.py`](assets/templates/etl.py).
+- **Backfills/rebuilds:** adaptive batching is the default for every run, not
+  opt-in. Separate the small canary from the full execution plan. Inventory
+  aggregate volume, expansion, hot keys, bytes, runtime, and resources; select
+  the largest measured-safe year/quarter/month/smaller interval. Keep batching
+  budgets within the applicable performance target. Pin a resumable manifest
+  and deterministic half-open split/resume boundaries; never resubmit a split
+  parent over its children or reuse incompatible completed receipts.
+
+Read [ETL Contract](references/etl-contract.md). For every backfill/rebuild,
+also read [Adaptive Batching](references/adaptive-batching.md). When the
+repository has no ETL base module, adapt
+[`assets/templates/etl.py`](assets/templates/etl.py).
 
 ### 2. Transform
 
@@ -84,11 +111,19 @@ evidence.
 
 - Pure state, tiny real-Spark, plan-shape, and focused repository tests.
 - Cover boundaries, ties, nulls, arithmetic, identity, and precedence.
+- Before widening batches, prove the wider candidate equals the union of
+  smaller candidates on identical immutable inputs. Exercise the full call
+  path, including shared date guards, and preserve per-Activity-Date cutoffs.
 
 ### 5. Production validation
 
 Run the production entry point on the cluster against a **small sample**,
-typically **one month** of source data. Local tests do not substitute.
+typically **one month** of source data. Local tests do not substitute. This
+sample is not the execution batch size. For backfills/rebuilds, follow the
+required [adaptive plan](references/adaptive-batching.md), including a bounded
+wider validate-only canary before an authorized wider write. The small canary
+need not be published first; the first write may be the separately proven wider
+batch.
 
 1. **Canary:** `spark-submit` with `--validate-only` on that sample and the
    production runtime. Require zero blockers, an explicit publication skip,
@@ -101,9 +136,13 @@ typically **one month** of source data. Local tests do not substitute.
    Otherwise skip. When required, follow
    [Diagnosis And Profiling](references/diagnosis-and-profiling.md).
 3. **Write (authorized):** re-check admission, then publish the same pinned
-   sample. Do not present canary time as write performance.
-4. **Reconciliation:** independently prove the published sample. Follow
-   [Reconciliation](references/reconciliation.md).
+   sample or wider batch proven by its own canary. Verify complete partition
+   population, Hudi identity/keys/locks, and single-writer admission. Do not
+   present canary time as write performance.
+4. **Reconciliation:** independently prove every published scope, retaining
+   monthly or finer key/preserved-column checks inside wider writes. Follow
+   [Reconciliation](references/reconciliation.md). Record receipts; after an
+   interruption inspect Spark and Hudi commits before retrying.
 
 ```bash
 spark-submit <production-options> job.py --validate-only
@@ -119,9 +158,12 @@ canary evidence, and write rules.
 | Need | Read or run |
 |---|---|
 | Define or review an ETL job contract | [ETL Contract](references/etl-contract.md) and `assets/templates/etl.py` |
+| Define or change output-column meaning, explain columns, or audit missing definitions | [data-catalog](../data-catalog/SKILL.md) |
+| Check published data against column definitions | [data-quality](../data-quality/SKILL.md) |
+| Plan any backfill/rebuild, size batches, split/resume, or recover an interruption | [Adaptive Batching](references/adaptive-batching.md) |
 | Diagnose or profile | [Diagnosis And Profiling](references/diagnosis-and-profiling.md) |
 | Design transformations | [Transformation Design](references/transformation-design.md) |
 | Optimize temporal/FIFO state | [Velocity Feature Calculation](references/velocity-feature-calculation.md) |
-| Reconcile published sample vs previous path | [Reconciliation](references/reconciliation.md) |
+| Reconcile published scope vs previous path, including finer checks within wider writes | [Reconciliation](references/reconciliation.md) |
 | Run production validation (canary, optional profile, write, reconciliation) | [Production Validation](references/production-validation.md) |
 | Summarize an event log | `scripts/spark_eventlog_summary.py` |

package/skills/app-dev/pyspark/references/adaptive-batching.md

@@ -0,0 +1,294 @@
+# Adaptive Batching For Backfills And Rebuilds
+
+Use measured adaptive batching by default for **every backfill and rebuild**.
+This is a required planning step, not an optional speed optimization. A small
+scope may need one batch; dense history may still require monthly or smaller
+batches. A typically one-month canary does not imply month-by-month execution.
+
+Keep two separate artifacts: small-canary evidence and the full execution plan.
+Follow [Production Validation](production-validation.md) for admission,
+zero-output evidence, profiling gates, authorization, and publication. Batching
+does not authorize writes, concurrent Hudi writers, or shuffle tuning.
+
+## 1. Establish Scope And Publication Semantics
+
+Declare output scope as `[start, end)`: start included, end excluded, in the
+contract's timezone and date precision. Separate output dates from required
+source-history dates. Earlier history and boundary state must remain available
+to each smaller batch.
+
+Inspect the **whole call path**: controller, CLI, wrapper, shared/base
+constructor, extraction, transformation, validation, and every writer. Look for
+calendar-month checks, shared **31-day constructor guards**, default dates,
+per-day loops, history offsets, and post-transform filters. Changing wrapper
+dates alone does not establish wider-range support. Do not bypass a shared
+guard merely to accept a year. Establish its purpose and test affected callers.
+Route changes requiring a shared-contract design decision to Architect. Until
+resolved, retain supported safe intervals in the adaptive plan.
+
+Preserve each **Activity Date's cutoff**, source visibility, lookback, tie order,
+and late-data rules. Never replace per-Activity-Date cutoffs with a batch-end
+cutoff. Record semantic, schema, and source-availability boundaries as hard
+interval boundaries. Unknown coverage is a blocker, not sparse history. Resolve
+missing product semantics with Product Manager and shared design decisions
+with Architect before crossing a boundary.
+
+Before widening a Hudi write, verify actual table path/name and identity,
+table type/version, complete record key, key generator, precombine order,
+partition columns, operation, locks, and single-writer admission. Inspect
+initial-load fallbacks too. The candidate must contain the **complete population
+of every partition it replaces**, including unchanged entities and preserved
+columns. An overlay's changed rows alone are not a partition replacement.
+A partial first/last interval cannot overwrite a coarser partition unless its
+complete population is safely assembled and the write scope is authorized.
+Otherwise block; do not silently expand requested or already-reconciled scope.
+
+## 2. Inventory Volume Without Writing
+
+Run bounded, aggregate-only, read-only queries against pinned source snapshots.
+Inventory years, quarters, months, and denser subperiods as needed. Include
+partial edges and the heaviest known periods/entities. Record scope and source
+pins with each result. Do not print entity-level rows.
+
+| Measurement | What to establish |
+|---|---|
+| Output rows and distinct entities | Complete population, not only changed rows; recompute wider distinct counts rather than summing them |
+| Source-history rows and coverage | Required history including pre-start lookback/state; distinguish known zero from unavailable data |
+| Temporal expansion | Estimated/measured pairs or rows after joins, date expansion, explodes, and groups; record absolute counts and expansion denominator |
+| Maximum rows per entity/key | Largest source and expanded groups, hot-key state bytes, and straggler shape; averages hide risk |
+| Bytes | Source scan, intermediate/state, shuffle, and candidate/write bytes; label estimates and uncertainty |
+| Runtime and resources | Startup, scans, identity preparation, transform/validation, publication, reconciliation, peak JVM/Python memory, spill, GC, max-task time, executor allocation, queue wait, and competing work |
+
+Prefer aggregates or bounded probes over materializing an unbounded pair
+relation just to count it. Output-row count alone cannot establish safety.
+Tiny output can require huge history joins or one group exceeding task memory.
+Measure unknown metrics with a bounded candidate before admitting that shape.
+Splitting dates may not shrink a lifetime-history hot key. If the smallest
+supported interval still exceeds budget, stop for a bounded algorithm/design
+rather than retrying ever smaller dates indefinitely.
+
+## 3. Select The Largest Measured-Safe Interval
+
+Derive a vector of budgets from comparable successful measurements on the same
+runtime, resources, source pins, validation, and publication semantics. Include
+output rows, distinct entities, history, expanded rows, max-key rows/state bytes,
+scan/shuffle bytes, memory/spill, and runtime limits with explicit headroom.
+These budgets must also satisfy the applicable performance target; fitting a
+batch budget does not waive that target. Record each budget's evidence and
+uncertainty. There is **no universal row cap**. The proposed
+200,000 customer-day limit from the motivating workload is provisional and
+workload-specific, not skill policy.
+
+For each remaining contiguous scope, consider calendar years, then quarters,
+months, and smaller supported intervals. Clip to scope edges and hard
+boundaries. Prefer yearly batches for measured-safe sparse history, quarterly
+for medium volumes, and monthly or smaller for dense/skewed periods. Admit only
+intervals within **all** budgets and covered by local equivalence and a bounded
+wider production canary. Inventory nominates candidates; it does not replace
+runtime proof. Do not extrapolate a sparse-year canary to an unmeasured dense
+quarter or hot key. Reinventory and revise only pending work when observations
+exceed budgets or source/resource assumptions change.
+
+### Worked Example (Illustrative, Not Production Evidence)
+
+Suppose immutable-source measurements on one fixed resource profile establish
+these workload-specific budgets, with headroom already reserved: 500,000 output
+rows, 4 million history rows, 8 million expanded rows, 80,000 expanded rows per
+key, 64 MiB max-key state, 6 GiB scan, 3 GiB shuffle, and 4 minutes end-to-end
+per write, within the applicable five-minute complete-job target. Comparable
+probes also bound peak memory and spill; those limits remain part of admission
+although the table omits them.
+
+| Candidate | Output rows | Distinct entities | History rows | Expanded rows | Max-key rows/state | Scan/shuffle | Measured write time | Decision |
+|---|---:|---:|---:|---:|---|---|---|---|
+| Sparse year | 120,000 | 900 | 700,000 | 1.4 million | 12,000 / 10 MiB | 1 / 0.5 GiB | 3 min | Year fits |
+| Medium year | 900,000 | 2,400 | 6 million | 12 million | 60,000 / 48 MiB | 8 / 5 GiB | Not admitted | Split to quarters |
+| Heaviest medium quarter | 260,000 | 1,800 | 1.9 million | 3.8 million | 35,000 / 28 MiB | 3 / 1.5 GiB | 4 min | Quarter fits; check other quarters |
+| Dense month | 300,000 | 1,200 | 2 million | 10 million | 150,000 / 120 MiB | 4 / 4 GiB | Not admitted | Output fits, expansion/key budgets fail |
+| Heaviest dense week | 75,000 | 800 | 900,000 | 2.5 million | 40,000 / 32 MiB | 2 / 1 GiB | 2 min | Week fits; check remaining weeks/edges |
+
+Use the full metric vector, not rows per calendar unit. These write times
+assume already-authorized, reconciled measurements. A new wider shape still
+needs the proof below before its first write. Validate-only times cannot fill
+the measured-write column. Existing profiling gates still apply.
+
+## 4. Split And Resume Deterministically
+
+Persist the policy, timezone, minimum interval, and stable ordering. Split at
+calendar-aligned year -> quarter -> month -> declared smaller boundaries.
+Clip children to their parent's `[start, end)`. For smaller periods, record an
+exact rule such as seven-day boundaries anchored at a named date. Retries must
+not choose new boundaries from the current clock.
+
+Verify reconciliation receipts against the active campaign's exact source
+pins, code/module identity, output contract, and destination before subtracting
+completed scopes. A plan revision alone does not make older results compatible.
+Reuse a changed identity only with explicit equivalence evidence covering the
+completed scope. Without that proof, block campaign completion and identify all
+affected published scopes for an explicitly authorized rework plan; do not
+silently mix old and new semantics or relabel old receipts. Split at
+semantic/source boundaries and compatible completed-scope edges before sizing.
+Sort by start date. Prove children are nonempty,
+pairwise disjoint, and their union equals the parent's remaining scope. Across
+the campaign, reconciled plus pending scopes must equal exactly the authorized
+scope, with no gaps or overlaps. Never mark a parent complete because one child
+committed. Retain parent/child IDs and durably make the split parent
+non-runnable before scheduling children (for example, mark it superseded).
+Settle any unresolved parent publication state first. Restart must use the
+persisted child plan, never resubmit the original parent over completed children.
+
+For example, `[2021-02-10, 2022-05-07)` with reconciled work
+`[2021-07-01, 2021-10-01)` can split into these quarter-sized pending intervals:
+
+```text
+[2021-02-10, 2021-04-01)
+[2021-04-01, 2021-07-01)
+[2021-10-01, 2022-01-01)
+[2022-01-01, 2022-04-01)
+[2022-04-01, 2022-05-07)
+```
+
+Keep partial first and last periods. Exclude reconciled scope from output
+candidates, though required source history may overlap it. Check partition
+replacement completeness separately from logical interval coverage.
+
+## 5. Prove Wider Equals Smaller Before Writing
+
+In focused local tests, build the wider candidate and the union of contiguous
+smaller candidates using **the same immutable inputs**, including history and
+boundary state. Use `unionByName` without deduplication for the smaller results.
+Compare schema, complete keys, duplicate multiplicity, all required values,
+and preserved columns with bidirectional `exceptAll`. Counts
+alone are insufficient. Assert literal per-Activity-Date cutoffs so two paths
+cannot share the same batch-end bug.
+
+Cover year/quarter/month edges, partial endpoints, leap days and relevant
+timezone boundaries, hot entities, duplicate identities, missing source
+coverage, and already-reconciled holes. Exercise the actual entry point and
+shared constructor, including any 31-day guard. Transform-only tests cannot
+prove that the wider entry point accepts the interval or uses the right writer
+scope. Test deterministic split/resume and interruption after commit but before
+its receipt. Do not relax production guards to make tests pass.
+
+After local proof and the small canary, run a **bounded wider validate-only
+production canary** through the full production path before authorizing its
+wider write. Bound it by declared volume/resource budgets and an exact interval.
+Exercise the proposed width and relevant heavy/key shape; a month cannot prove
+a year. Cover each materially different shape or keep unproven periods at
+verified sizes. Require zero blockers, explicit publication skip, zero Spark
+output records/bytes, and retained logs. The small sample is validate-only;
+there is no mandatory sample write before the wider write. The first authorized
+publication may be the proven wider batch. If the sample was independently
+published, reconcile it and account for its completed scope before planning the
+wider write, including partition-replacement completeness.
+
+Keep [profiling gates](diagnosis-and-profiling.md#when-profiling-is-required)
+unchanged: a small-sample canary over 5 minutes when unconstrained, or any job
+over 30 minutes regardless of contention. Explicit write authorization and
+fresh admission follow the wider canary. Publish the same pinned candidate.
+Independently reconcile every monthly or finer scope inside the wider commit
+using [Reconciliation](reconciliation.md), including preserved columns and full
+keys, not only whole-batch totals.
+
+## 6. Keep A Resumable Manifest
+
+Persist the plan and append attempt history **before** launching work. This
+illustrative YAML shows one reconciled entry, not a new runner implementation.
+Replace example pins/receipts with exact durable identities and real evidence.
+
+```yaml
+campaign_id: rebuild-2021
+plan_revision: 2
+authorized_scope: {start: '2021-02-10', end_exclusive: '2022-05-07'}
+timezone: UTC
+split_policy: calendar-year-quarter-month-then-7d-from-2021-01-01
+minimum_interval: 1d
+hard_boundaries_receipt: receipts/source-and-semantic-boundaries.json
+source_pins: {activity: 'snapshot:example-a', history: 'snapshot:example-b'}
+code: {commit: 'example-full-sha', module: 'jobs.rebuild', module_sha256: 'example-sha256'}
+runtime_resources_receipt: receipts/runtime-resources.json
+table_contract_receipt: receipts/table-keys-partitions-operation-locks.json
+inventory_receipt: receipts/aggregate-inventory.json
+budgets_receipt: receipts/budget-vector-with-headroom-and-measurements.json
+batches:
+  - id: '2021-q2'
+    parent_id: '2021'
+    interval: {start: '2021-04-01', end_exclusive: '2021-07-01'}
+    publication_partitions_receipt: receipts/q2-complete-population.json
+    estimates_and_budget_checks: receipts/q2-sizing.json
+    local_equivalence_receipt: receipts/q2-local-parity.json
+    wider_canary_receipt: receipts/q2-validate-only.json
+    write_authorization_receipt: receipts/q2-authorization.json
+    status: reconciled
+    attempts:
+      - attempt: 1
+        controller_id: controller-example-1
+        application_id: application_example_1
+        application_attempt_id: appattempt_example_1
+        mode: write
+        status: reconciled
+        publication_instant: 'example-hudi-instant'
+        timeline_inspection_receipt: receipts/q2-timeline.json
+        reconciliation_receipts:
+          - receipts/2021-04-keys-and-preserved-columns.json
+          - receipts/2021-05-keys-and-preserved-columns.json
+          - receipts/2021-06-keys-and-preserved-columns.json
+```
+
+Budget receipts hold the full vector, measured/proposed distinction, headroom,
+and expansion/hot-key bounds. Record timings and resources per attempt. Use
+explicit states such as planned, validating, validated, writing,
+publication-unknown, committed-unreconciled, reconciled, blocked, failed, and
+superseded. A committed entry is not complete until finer reconciliation passes.
+Changes to pins, code, resources, or budgets create a plan revision. Recheck
+completed-receipt compatibility under the split/resume rules and invalidate
+affected pending validation or sizing proof. Resource-only changes may require
+new capacity evidence without changing reconciled values; source/code/output
+changes require exact identity or explicit equivalence before old receipts
+count toward completion. Keep original attempt identities and receipts
+append-only. Do not silently reuse canary evidence or overwrite attempt history.
+
+## 7. Recover An Interruption Before Retrying
+
+1. Stop/fence the old controller from launching more intervals. Confirm no
+   overlapping controller or scheduler can submit work. Record ownership.
+2. Inspect the existing Spark application and all attempts, driver logs, and
+   Hudi timeline for the exact destination. A stopped client or YARN `KILLED`
+   does **not** mean no write occurred. Do not retry while the old application
+   can still publish.
+3. Inspect completed, requested, and inflight instants, including commits that
+   completed before interruption but lack a manifest receipt. Map each instant
+   to its actual partitions, keys/scope, code, and source pins.
+4. Independently reconcile any committed scope at monthly or finer granularity.
+   Record instants and receipts before subtracting completed work. Mark uncertain
+   or mismatched scope blocked/publication-unknown, never unwritten.
+5. Resolve inflight/failed writes through the repository's supported Hudi
+   recovery procedure under the lock owner. Do not delete timeline entries or
+   force-unlock an application that might still write.
+6. Only after publication state is settled, replan the verified remainder,
+   preserve pins, increment the attempt, and re-check admission and authorization.
+   Resume with one controller and one writer; never blindly retry the parent.
+
+## 8. Estimate Completion From Measured Costs
+
+Estimate remaining **reconciled completion** time, not only transform time.
+Separate startup, history scans, identity preparation, validation, publication,
+and finer reconciliation. Include queue wait, contention, failed attempts and
+recovery, and uncertainty. Use comparable measured costs for each remaining
+shape. Do not turn a monthly rate into a promised yearly speedup or treat
+validate-only time as write time.
+
+Compare finishing verified remaining intervals against paying for guard/code
+changes, local tests, wider canaries, required profiling, and migration/recovery
+before wider execution. Show a range and assumptions for each plan. For an
+illustrative tight deadline, 12 verified monthly intervals at 3-4 minutes each
+imply 36-48 minutes remaining. If guard changes plus proof need 90-120 minutes
+before any wider write, there is no measured case that migration finishes
+sooner. Keep verified intervals in the adaptive plan unless evidence changes
+that comparison. If neither plan meets the deadline, report that limitation.
+Ask Product Manager about scope/deadline changes and Architect about necessary
+shared-contract changes.
+
+Reduce repeated fixed costs only where measured wider batching preserves all
+required work. Do not substitute shuffle tuning, concurrent writers, skipped
+checks, or speculative speedups for this plan.

package/skills/app-dev/pyspark/references/diagnosis-and-profiling.md

@@ -42,6 +42,23 @@ Use this default target unless the repository defines a stricter one:
 
 Treat the target as failed until comparable evidence proves it passed.
 
+## Backfill And Rebuild Batching Evidence
+
+Every backfill/rebuild follows [Adaptive Batching](adaptive-batching.md).
+Separate the typically one-month canary from execution sizing. Inventory rows,
+entities, required history, temporal expansion, max-key rows/state, and bytes;
+use comparable runtime/resource measurements to select the largest safe
+contiguous interval. These are workload budgets, not a universal row cap or
+permission to relax the performance target above.
+
+Measure repeated startup, scans, identity preparation, validation, Hudi commit,
+and reconciliation overhead separately from volume-dependent work. Report a
+remaining-plan ETA range with queue/contention uncertainty. Compare finishing
+verified pending intervals against code/guard migration, local proof, wider
+canary, profiling, and execution costs. Do not claim wider writes are faster
+without comparable authorized-write evidence. Tight deadlines do not waive
+proof, single-writer admission, or the unchanged profiling gates.
+
 ## Require An Isolated, Comparable Profile
 
 Accept a baseline or candidate profile only when all of these are fixed and
@@ -141,9 +158,13 @@ settings merely because the slow work becomes visible at a later action.
 4. **Validate locally:** prove required behavior, boundary cases, and the
    intended physical-plan change with focused tests. Do not treat local Spark as
    cluster parity.
-5. **Production validation:** one-month (typical) sample: canary, optional
-   profile (5-minute unconstrained canary **or** any job over 30 minutes),
-   authorized write, then reconciliation.
+5. **Production validation:** start with a small (typically one-month)
+   validate-only canary and applicable profiling (over 5 minutes when that
+   sample is unconstrained, or any job over 30 minutes). For backfills/rebuilds,
+   use the adaptive execution plan and wider-canary proof before wider writes.
+   The small sample need not be written first: the first authorized write may
+   be the separately proven wider batch. Reconcile whichever scope was
+   published, retaining finer checks within wider writes.
 
 Repeat the loop when the target is still missed or the dominant gap moves.
 

package/skills/app-dev/pyspark/references/etl-contract.md

@@ -36,12 +36,39 @@ Hudi metadata default.
 | Write behavior | Append, overwrite, upsert, bulk insert, or insert overwrite | `hudi_mode`, `hudi_mode_override`, `load_hudi`, `load_spark` |
 | Spark runtime | Native Spark versus Spark Connect and session owner | `create_spark_session`, `managed_spark_session`, `spark_remote` |
 | Processing scope | Full versus incremental/per-day | `bulk`, `filter_by`, `offset`, `start_date`, `end_date`, `process` |
+| Backfill/rebuild plan | Required adaptive sizing, exact half-open output/history scopes, hard boundaries, measured budgets, deterministic split/resume, and manifest | controller and all job/base-module date and publication paths; see [Adaptive Batching](adaptive-batching.md) |
 | Incremental state | Watermark, lookback/checkpoint, restart boundary, and late-data behavior | `filter_by`, `offset`, subclass/checkpoint implementation |
 
 Do not design the transformation until every applicable surface has an explicit
 value. Reject defaults that can silently change data scope, key semantics,
 concurrency, or publication behavior.
 
+## Backfill And Rebuild Contract
+
+Adaptive batching is the default for every backfill/rebuild. Declare the small
+canary separately from the full execution plan. Follow
+[Adaptive Batching](adaptive-batching.md) for aggregate inventory, measured
+budgets, interval selection, manifest, and interruption recovery.
+
+Inspect controller -> CLI -> wrapper -> shared constructor -> extraction ->
+transformation -> validation -> writer. Check calendar-month restrictions and
+any shared 31-day constructor guard; changing wrapper dates is insufficient.
+Do not bypass guards to admit larger dates. Preserve per-Activity-Date cutoffs
+and required pre-start history. Route unresolved product semantics to Product
+Manager and shared-contract design decisions to Architect.
+
+Pin exact `[start, end)` intervals (start included, end excluded), source/code/
+module identities, and semantic/source boundaries. Completed receipts must
+match the active campaign's source/code/output contract, or carry explicit
+equivalence evidence for completed scope, before their exact intervals are
+subtracted. A plan revision alone is not compatibility. Preserve original
+receipts and block incompatible completed scope pending an authorized rework
+plan. Before widening, confirm actual Hudi table identity, full keys,
+precombine, overwrite partitions, complete population,
+locks, and single-writer admission. Partial edges do not authorize incomplete
+partition replacement. Require wider-versus-smaller local proof and a bounded
+wider validate-only production canary before an explicitly authorized write.
+
 ## Adapt Repository-Specific Dependencies
 
 The complete module depends on its source warehouse. Resolve each dependency