@1aboveio/app-dev-skills 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -2
- package/package.json +34 -4
- package/skills/airflow/LICENSE +3 -0
- package/skills/airflow/SKILL.md +199 -0
- package/skills/airflow/release-check.mjs +93 -0
- package/skills/android-foundation/LICENSE +3 -0
- package/skills/android-foundation/SKILL.md +192 -0
- package/skills/android-foundation/assets/android-ci-job.yml +217 -0
- package/skills/android-foundation/references/ci.md +87 -0
- package/skills/android-foundation/references/gotchas.md +221 -0
- package/skills/android-foundation/references/setup.md +164 -0
- package/skills/android-foundation/release-check.mjs +93 -0
- package/skills/android-foundation/scripts/device-run.sh +176 -0
- package/skills/android-foundation/scripts/device-run.test.mjs +172 -0
- package/skills/app-debug/LICENSE +3 -0
- package/skills/app-debug/SKILL.md +110 -0
- package/skills/app-debug/references/common-errors.md +128 -0
- package/skills/app-debug/release-check.mjs +93 -0
- package/skills/better-auth/LICENSE +3 -0
- package/skills/better-auth/SKILL.md +598 -0
- package/skills/better-auth/references/feishu-api.md +270 -0
- package/skills/better-auth/release-check.mjs +93 -0
- package/skills/data-catalog/LICENSE +3 -0
- package/skills/data-catalog/SKILL.md +207 -0
- package/skills/data-catalog/agents/openai.yaml +3 -0
- package/skills/data-catalog/references/data-catalog-conventions.md +76 -0
- package/skills/data-catalog/references/field-definitions.md +123 -0
- package/skills/data-catalog/references/format.md +130 -0
- package/skills/data-catalog/release-check.mjs +93 -0
- package/skills/data-catalog/scripts/data-catalog.test.mjs +36 -0
- package/skills/data-catalog/scripts/data_catalog.py +1435 -0
- package/skills/data-catalog/scripts/persist-conventions-core.mjs +117 -0
- package/skills/data-catalog/scripts/persist-conventions.mjs +17 -0
- package/skills/data-catalog/scripts/test_data_catalog.py +603 -0
- package/skills/ios-foundation/LICENSE +3 -0
- package/skills/ios-foundation/SKILL.md +184 -0
- package/skills/ios-foundation/references/setup.md +243 -0
- package/skills/ios-foundation/release-check.mjs +93 -0
- package/skills/monorepo/LICENSE +3 -0
- package/skills/monorepo/SKILL.md +462 -0
- package/skills/monorepo/release-check.mjs +93 -0
- package/skills/nextjs-fullstack/LICENSE +3 -0
- package/skills/nextjs-fullstack/SKILL.md +211 -0
- package/skills/nextjs-fullstack/conventions.md +318 -0
- package/skills/nextjs-fullstack/frontend-conventions.md +61 -0
- package/skills/nextjs-fullstack/nextjs16.md +287 -0
- package/skills/nextjs-fullstack/release-check.mjs +93 -0
- package/skills/nextjs-fullstack/server-actions.md +409 -0
- package/skills/prisma-setup/LICENSE +3 -0
- package/skills/prisma-setup/SKILL.md +181 -0
- package/skills/prisma-setup/nextjs.md +258 -0
- package/skills/prisma-setup/release-check.mjs +93 -0
- package/skills/prisma-setup/turborepo.md +301 -0
- package/skills/pyspark/LICENSE +3 -0
- package/skills/pyspark/SKILL.md +127 -0
- package/skills/pyspark/assets/templates/etl.py +748 -0
- package/skills/pyspark/assets/templates/utils/__init__.py +1 -0
- package/skills/pyspark/assets/templates/utils/hudi_metadata.py +14 -0
- package/skills/pyspark/references/diagnosis-and-profiling.md +169 -0
- package/skills/pyspark/references/etl-contract.md +129 -0
- package/skills/pyspark/references/production-validation.md +131 -0
- package/skills/pyspark/references/reconciliation.md +38 -0
- package/skills/pyspark/references/transformation-design.md +190 -0
- package/skills/pyspark/references/velocity-feature-calculation.md +193 -0
- package/skills/pyspark/release-check.mjs +93 -0
- package/skills/pyspark/scripts/spark_eventlog_summary.py +223 -0
- package/skills/python-backend/LICENSE +3 -0
- package/skills/python-backend/SKILL.md +325 -0
- package/skills/python-backend/assets/uv-install.sh +48 -0
- package/skills/python-backend/references/uv-installs.md +83 -0
- package/skills/python-backend/release-check.mjs +93 -0
- package/skills/shadcn/LICENSE +3 -0
- package/skills/shadcn/SKILL.md +120 -0
- package/skills/shadcn/assets/shadcn-small.png +0 -0
- package/skills/shadcn/assets/shadcn.png +0 -0
- package/skills/shadcn/cli.md +411 -0
- package/skills/shadcn/customization.md +224 -0
- package/skills/shadcn/mcp.md +101 -0
- package/skills/shadcn/release-check.mjs +93 -0
- package/skills/shadcn/rules/base-vs-radix.md +323 -0
- package/skills/shadcn/rules/component-selection.md +67 -0
- package/skills/shadcn/rules/composition.md +195 -0
- package/skills/shadcn/rules/data-table.md +201 -0
- package/skills/shadcn/rules/forms.md +255 -0
- package/skills/shadcn/rules/icons.md +103 -0
- package/skills/shadcn/rules/styling.md +167 -0
- package/skills/zod-v4/LICENSE +3 -0
- package/skills/zod-v4/SKILL.md +286 -0
- package/skills/zod-v4/release-check.mjs +93 -0
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: pyspark
|
|
3
|
+
description: >
|
|
4
|
+
Implement, debug, review, and optimize Apache Spark or PySpark batch jobs,
|
|
5
|
+
ETL pipelines, feature calculations, and Hudi/warehouse writes. Follow
|
|
6
|
+
contract → transform → materialize → local test → production validation
|
|
7
|
+
(one-month canary, optional profile if that canary exceeds 5 minutes without
|
|
8
|
+
resource constraints or any job exceeds 30 minutes, authorized write, cluster
|
|
9
|
+
reconciliation). Use whenever
|
|
10
|
+
the task involves Spark DataFrames or Spark SQL, joins, windows, grouped
|
|
11
|
+
state, UDFs, shuffle, skew, spill, slow stages, driver/executor OOM, Spark
|
|
12
|
+
Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
|
|
13
|
+
cluster reconciliation, or Spark event-log profiling — including warehouse
|
|
14
|
+
job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
|
|
15
|
+
release-version: 0.38.0
|
|
16
|
+
---
|
|
17
|
+
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
18
|
+
# PySpark
|
|
19
|
+
|
|
20
|
+
Use this skill as the workflow and routing index. Read the playbook for the
|
|
21
|
+
step you are on before editing code or submitting to the cluster.
|
|
22
|
+
|
|
23
|
+
## Workflow
|
|
24
|
+
|
|
25
|
+
```text
|
|
26
|
+
contract -> transform -> materialize -> test -> production validation
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Follow all five steps in order.
|
|
30
|
+
|
|
31
|
+
### 1. Contract
|
|
32
|
+
|
|
33
|
+
- **Input:** declare all parameters, defaults, validation, source snapshot/date
|
|
34
|
+
scope, overlapping-run behavior, idempotency/single-writer rules, and exact
|
|
35
|
+
Python and PySpark versions.
|
|
36
|
+
- **Output:** declare plain Spark versus Hudi table type, output identity/schema,
|
|
37
|
+
partition columns, record key, deterministic order or precombine field, and
|
|
38
|
+
append/upsert/overwrite write mode.
|
|
39
|
+
- **Conventions:** declare native Spark versus Spark Connect and full rebuild
|
|
40
|
+
versus incremental processing, including the incremental watermark,
|
|
41
|
+
lookback/checkpoint, and restart boundary when applicable.
|
|
42
|
+
|
|
43
|
+
Read [ETL Contract](references/etl-contract.md). When the repository has no ETL
|
|
44
|
+
base module, adapt [`assets/templates/etl.py`](assets/templates/etl.py).
|
|
45
|
+
|
|
46
|
+
### 2. Transform
|
|
47
|
+
|
|
48
|
+
- Make `transform` accept DataFrames and return the complete candidate without
|
|
49
|
+
invoking validation or a writer.
|
|
50
|
+
- Estimate rows after joins/explodes/groups and measure maximum rows per key.
|
|
51
|
+
Fix algorithmic expansion and hot keys before tuning Spark configuration.
|
|
52
|
+
- Prefer built-in expressions and windows. Use exact bucketing or one sorted
|
|
53
|
+
grouped sweep only for state that built-ins cannot express cleanly.
|
|
54
|
+
- Preserve complete row identity, ordering, output columns, and required
|
|
55
|
+
behavior.
|
|
56
|
+
- Avoid explicit `.cache()` and `.persist()` by default.
|
|
57
|
+
|
|
58
|
+
Read [Transformation Design](references/transformation-design.md). For temporal
|
|
59
|
+
windows or FIFO paths, also read
|
|
60
|
+
[Velocity Feature Calculation](references/velocity-feature-calculation.md).
|
|
61
|
+
|
|
62
|
+
### 3. Materialize
|
|
63
|
+
|
|
64
|
+
- Treat materialization as the Spark action that executes the lazy candidate,
|
|
65
|
+
not as caching.
|
|
66
|
+
- Use one aggregate validation action before every writer. Reference every
|
|
67
|
+
derived output so Catalyst cannot prune untested calculations.
|
|
68
|
+
- Check row count, distinct/null/duplicate identities, domain blockers, and
|
|
69
|
+
required date/partition scope. Fail closed on any blocker.
|
|
70
|
+
- Bind validation and publication to the same immutable source snapshot because
|
|
71
|
+
Spark may recompute the candidate.
|
|
72
|
+
- Keep the validate-only guard inside the job and before every writer. Do not
|
|
73
|
+
add `.count()`, `.cache()`, or `.persist()` to warm execution.
|
|
74
|
+
- Do not set Spark or Hudi shuffle parallelism (`spark.sql.shuffle.partitions`,
|
|
75
|
+
`hoodie.*.shuffle.parallelism`). Those knobs split the write into many tiny
|
|
76
|
+
files. Leave Spark/Hudi defaults; size writes with `HudiLayout`. Repair an
|
|
77
|
+
existing small-file table with `Etl.rewrite_hudi_layout` (see
|
|
78
|
+
[ETL Contract](references/etl-contract.md)).
|
|
79
|
+
|
|
80
|
+
### 4. Test
|
|
81
|
+
|
|
82
|
+
Run **local** tests only. Do not treat a laptop Spark session as cluster
|
|
83
|
+
evidence.
|
|
84
|
+
|
|
85
|
+
- Pure state, tiny real-Spark, plan-shape, and focused repository tests.
|
|
86
|
+
- Cover boundaries, ties, nulls, arithmetic, identity, and precedence.
|
|
87
|
+
|
|
88
|
+
### 5. Production validation
|
|
89
|
+
|
|
90
|
+
Run the production entry point on the cluster against a **small sample**,
|
|
91
|
+
typically **one month** of source data. Local tests do not substitute.
|
|
92
|
+
|
|
93
|
+
1. **Canary:** `spark-submit` with `--validate-only` on that sample and the
|
|
94
|
+
production runtime. Require zero blockers, an explicit publication skip,
|
|
95
|
+
zero Spark output records/bytes, and readable driver/event logs.
|
|
96
|
+
2. **Profiling (optional):** required when **either** (a) the small-sample
|
|
97
|
+
canary wall time exceeds **5 minutes** **and** the run was not
|
|
98
|
+
resource-constrained (queued, under-provisioned executors, competing
|
|
99
|
+
warehouse work), **or** (b) **any** job (canary, write, or scheduled run)
|
|
100
|
+
exceeds **30 minutes** — a hard gate, even if the cluster was constrained.
|
|
101
|
+
Otherwise skip. When required, follow
|
|
102
|
+
[Diagnosis And Profiling](references/diagnosis-and-profiling.md).
|
|
103
|
+
3. **Write (authorized):** re-check admission, then publish the same pinned
|
|
104
|
+
sample. Do not present canary time as write performance.
|
|
105
|
+
4. **Reconciliation:** independently prove the published sample. Follow
|
|
106
|
+
[Reconciliation](references/reconciliation.md).
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
spark-submit <production-options> job.py --validate-only
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Restore scheduler/process ownership after the run.
|
|
113
|
+
|
|
114
|
+
Read [Production Validation](references/production-validation.md) for admission,
|
|
115
|
+
canary evidence, and write rules.
|
|
116
|
+
|
|
117
|
+
## Reference Index
|
|
118
|
+
|
|
119
|
+
| Need | Read or run |
|
|
120
|
+
|---|---|
|
|
121
|
+
| Define or review an ETL job contract | [ETL Contract](references/etl-contract.md) and `assets/templates/etl.py` |
|
|
122
|
+
| Diagnose or profile | [Diagnosis And Profiling](references/diagnosis-and-profiling.md) |
|
|
123
|
+
| Design transformations | [Transformation Design](references/transformation-design.md) |
|
|
124
|
+
| Optimize temporal/FIFO state | [Velocity Feature Calculation](references/velocity-feature-calculation.md) |
|
|
125
|
+
| Reconcile published sample vs previous path | [Reconciliation](references/reconciliation.md) |
|
|
126
|
+
| Run production validation (canary, optional profile, write, reconciliation) | [Production Validation](references/production-validation.md) |
|
|
127
|
+
| Summarize an event log | `scripts/spark_eventlog_summary.py` |
|