@1aboveio/app-dev-skills 0.0.0-stage → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (89) hide show
  1. package/README.md +30 -2
  2. package/package.json +34 -4
  3. package/skills/airflow/LICENSE +3 -0
  4. package/skills/airflow/SKILL.md +199 -0
  5. package/skills/airflow/release-check.mjs +93 -0
  6. package/skills/android-foundation/LICENSE +3 -0
  7. package/skills/android-foundation/SKILL.md +192 -0
  8. package/skills/android-foundation/assets/android-ci-job.yml +217 -0
  9. package/skills/android-foundation/references/ci.md +87 -0
  10. package/skills/android-foundation/references/gotchas.md +221 -0
  11. package/skills/android-foundation/references/setup.md +164 -0
  12. package/skills/android-foundation/release-check.mjs +93 -0
  13. package/skills/android-foundation/scripts/device-run.sh +176 -0
  14. package/skills/android-foundation/scripts/device-run.test.mjs +172 -0
  15. package/skills/app-debug/LICENSE +3 -0
  16. package/skills/app-debug/SKILL.md +110 -0
  17. package/skills/app-debug/references/common-errors.md +128 -0
  18. package/skills/app-debug/release-check.mjs +93 -0
  19. package/skills/better-auth/LICENSE +3 -0
  20. package/skills/better-auth/SKILL.md +598 -0
  21. package/skills/better-auth/references/feishu-api.md +270 -0
  22. package/skills/better-auth/release-check.mjs +93 -0
  23. package/skills/data-catalog/LICENSE +3 -0
  24. package/skills/data-catalog/SKILL.md +207 -0
  25. package/skills/data-catalog/agents/openai.yaml +3 -0
  26. package/skills/data-catalog/references/data-catalog-conventions.md +76 -0
  27. package/skills/data-catalog/references/field-definitions.md +123 -0
  28. package/skills/data-catalog/references/format.md +130 -0
  29. package/skills/data-catalog/release-check.mjs +93 -0
  30. package/skills/data-catalog/scripts/data-catalog.test.mjs +36 -0
  31. package/skills/data-catalog/scripts/data_catalog.py +1435 -0
  32. package/skills/data-catalog/scripts/persist-conventions-core.mjs +117 -0
  33. package/skills/data-catalog/scripts/persist-conventions.mjs +17 -0
  34. package/skills/data-catalog/scripts/test_data_catalog.py +603 -0
  35. package/skills/ios-foundation/LICENSE +3 -0
  36. package/skills/ios-foundation/SKILL.md +184 -0
  37. package/skills/ios-foundation/references/setup.md +243 -0
  38. package/skills/ios-foundation/release-check.mjs +93 -0
  39. package/skills/monorepo/LICENSE +3 -0
  40. package/skills/monorepo/SKILL.md +462 -0
  41. package/skills/monorepo/release-check.mjs +93 -0
  42. package/skills/nextjs-fullstack/LICENSE +3 -0
  43. package/skills/nextjs-fullstack/SKILL.md +211 -0
  44. package/skills/nextjs-fullstack/conventions.md +318 -0
  45. package/skills/nextjs-fullstack/frontend-conventions.md +61 -0
  46. package/skills/nextjs-fullstack/nextjs16.md +287 -0
  47. package/skills/nextjs-fullstack/release-check.mjs +93 -0
  48. package/skills/nextjs-fullstack/server-actions.md +409 -0
  49. package/skills/prisma-setup/LICENSE +3 -0
  50. package/skills/prisma-setup/SKILL.md +181 -0
  51. package/skills/prisma-setup/nextjs.md +258 -0
  52. package/skills/prisma-setup/release-check.mjs +93 -0
  53. package/skills/prisma-setup/turborepo.md +301 -0
  54. package/skills/pyspark/LICENSE +3 -0
  55. package/skills/pyspark/SKILL.md +127 -0
  56. package/skills/pyspark/assets/templates/etl.py +748 -0
  57. package/skills/pyspark/assets/templates/utils/__init__.py +1 -0
  58. package/skills/pyspark/assets/templates/utils/hudi_metadata.py +14 -0
  59. package/skills/pyspark/references/diagnosis-and-profiling.md +169 -0
  60. package/skills/pyspark/references/etl-contract.md +129 -0
  61. package/skills/pyspark/references/production-validation.md +131 -0
  62. package/skills/pyspark/references/reconciliation.md +38 -0
  63. package/skills/pyspark/references/transformation-design.md +190 -0
  64. package/skills/pyspark/references/velocity-feature-calculation.md +193 -0
  65. package/skills/pyspark/release-check.mjs +93 -0
  66. package/skills/pyspark/scripts/spark_eventlog_summary.py +223 -0
  67. package/skills/python-backend/LICENSE +3 -0
  68. package/skills/python-backend/SKILL.md +325 -0
  69. package/skills/python-backend/assets/uv-install.sh +48 -0
  70. package/skills/python-backend/references/uv-installs.md +83 -0
  71. package/skills/python-backend/release-check.mjs +93 -0
  72. package/skills/shadcn/LICENSE +3 -0
  73. package/skills/shadcn/SKILL.md +120 -0
  74. package/skills/shadcn/assets/shadcn-small.png +0 -0
  75. package/skills/shadcn/assets/shadcn.png +0 -0
  76. package/skills/shadcn/cli.md +411 -0
  77. package/skills/shadcn/customization.md +224 -0
  78. package/skills/shadcn/mcp.md +101 -0
  79. package/skills/shadcn/release-check.mjs +93 -0
  80. package/skills/shadcn/rules/base-vs-radix.md +323 -0
  81. package/skills/shadcn/rules/component-selection.md +67 -0
  82. package/skills/shadcn/rules/composition.md +195 -0
  83. package/skills/shadcn/rules/data-table.md +201 -0
  84. package/skills/shadcn/rules/forms.md +255 -0
  85. package/skills/shadcn/rules/icons.md +103 -0
  86. package/skills/shadcn/rules/styling.md +167 -0
  87. package/skills/zod-v4/LICENSE +3 -0
  88. package/skills/zod-v4/SKILL.md +286 -0
  89. package/skills/zod-v4/release-check.mjs +93 -0
@@ -0,0 +1,127 @@
1
+ ---
2
+ name: pyspark
3
+ description: >
4
+ Implement, debug, review, and optimize Apache Spark or PySpark batch jobs,
5
+ ETL pipelines, feature calculations, and Hudi/warehouse writes. Follow
6
+ contract → transform → materialize → local test → production validation
7
+ (one-month canary, optional profile if that canary exceeds 5 minutes without
8
+ resource constraints or any job exceeds 30 minutes, authorized write, cluster
9
+ reconciliation). Use whenever
10
+ the task involves Spark DataFrames or Spark SQL, joins, windows, grouped
11
+ state, UDFs, shuffle, skew, spill, slow stages, driver/executor OOM, Spark
12
+ Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
13
+ cluster reconciliation, or Spark event-log profiling — including warehouse
14
+ job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
15
+ release-version: 0.38.0
16
+ ---
17
+ **Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
18
+ # PySpark
19
+
20
+ Use this skill as the workflow and routing index. Read the playbook for the
21
+ step you are on before editing code or submitting to the cluster.
22
+
23
+ ## Workflow
24
+
25
+ ```text
26
+ contract -> transform -> materialize -> test -> production validation
27
+ ```
28
+
29
+ Follow all five steps in order.
30
+
31
+ ### 1. Contract
32
+
33
+ - **Input:** declare all parameters, defaults, validation, source snapshot/date
34
+ scope, overlapping-run behavior, idempotency/single-writer rules, and exact
35
+ Python and PySpark versions.
36
+ - **Output:** declare plain Spark versus Hudi table type, output identity/schema,
37
+ partition columns, record key, deterministic order or precombine field, and
38
+ append/upsert/overwrite write mode.
39
+ - **Conventions:** declare native Spark versus Spark Connect and full rebuild
40
+ versus incremental processing, including the incremental watermark,
41
+ lookback/checkpoint, and restart boundary when applicable.
42
+
43
+ Read [ETL Contract](references/etl-contract.md). When the repository has no ETL
44
+ base module, adapt [`assets/templates/etl.py`](assets/templates/etl.py).
45
+
46
+ ### 2. Transform
47
+
48
+ - Make `transform` accept DataFrames and return the complete candidate without
49
+ invoking validation or a writer.
50
+ - Estimate rows after joins/explodes/groups and measure maximum rows per key.
51
+ Fix algorithmic expansion and hot keys before tuning Spark configuration.
52
+ - Prefer built-in expressions and windows. Use exact bucketing or one sorted
53
+ grouped sweep only for state that built-ins cannot express cleanly.
54
+ - Preserve complete row identity, ordering, output columns, and required
55
+ behavior.
56
+ - Avoid explicit `.cache()` and `.persist()` by default.
57
+
58
+ Read [Transformation Design](references/transformation-design.md). For temporal
59
+ windows or FIFO paths, also read
60
+ [Velocity Feature Calculation](references/velocity-feature-calculation.md).
61
+
62
+ ### 3. Materialize
63
+
64
+ - Treat materialization as the Spark action that executes the lazy candidate,
65
+ not as caching.
66
+ - Use one aggregate validation action before every writer. Reference every
67
+ derived output so Catalyst cannot prune untested calculations.
68
+ - Check row count, distinct/null/duplicate identities, domain blockers, and
69
+ required date/partition scope. Fail closed on any blocker.
70
+ - Bind validation and publication to the same immutable source snapshot because
71
+ Spark may recompute the candidate.
72
+ - Keep the validate-only guard inside the job and before every writer. Do not
73
+ add `.count()`, `.cache()`, or `.persist()` to warm execution.
74
+ - Do not set Spark or Hudi shuffle parallelism (`spark.sql.shuffle.partitions`,
75
+ `hoodie.*.shuffle.parallelism`). Those knobs split the write into many tiny
76
+ files. Leave Spark/Hudi defaults; size writes with `HudiLayout`. Repair an
77
+ existing small-file table with `Etl.rewrite_hudi_layout` (see
78
+ [ETL Contract](references/etl-contract.md)).
79
+
80
+ ### 4. Test
81
+
82
+ Run **local** tests only. Do not treat a laptop Spark session as cluster
83
+ evidence.
84
+
85
+ - Pure state, tiny real-Spark, plan-shape, and focused repository tests.
86
+ - Cover boundaries, ties, nulls, arithmetic, identity, and precedence.
87
+
88
+ ### 5. Production validation
89
+
90
+ Run the production entry point on the cluster against a **small sample**,
91
+ typically **one month** of source data. Local tests do not substitute.
92
+
93
+ 1. **Canary:** `spark-submit` with `--validate-only` on that sample and the
94
+ production runtime. Require zero blockers, an explicit publication skip,
95
+ zero Spark output records/bytes, and readable driver/event logs.
96
+ 2. **Profiling (optional):** required when **either** (a) the small-sample
97
+ canary wall time exceeds **5 minutes** **and** the run was not
98
+ resource-constrained (queued, under-provisioned executors, competing
99
+ warehouse work), **or** (b) **any** job (canary, write, or scheduled run)
100
+ exceeds **30 minutes** — a hard gate, even if the cluster was constrained.
101
+ Otherwise skip. When required, follow
102
+ [Diagnosis And Profiling](references/diagnosis-and-profiling.md).
103
+ 3. **Write (authorized):** re-check admission, then publish the same pinned
104
+ sample. Do not present canary time as write performance.
105
+ 4. **Reconciliation:** independently prove the published sample. Follow
106
+ [Reconciliation](references/reconciliation.md).
107
+
108
+ ```bash
109
+ spark-submit <production-options> job.py --validate-only
110
+ ```
111
+
112
+ Restore scheduler/process ownership after the run.
113
+
114
+ Read [Production Validation](references/production-validation.md) for admission,
115
+ canary evidence, and write rules.
116
+
117
+ ## Reference Index
118
+
119
+ | Need | Read or run |
120
+ |---|---|
121
+ | Define or review an ETL job contract | [ETL Contract](references/etl-contract.md) and `assets/templates/etl.py` |
122
+ | Diagnose or profile | [Diagnosis And Profiling](references/diagnosis-and-profiling.md) |
123
+ | Design transformations | [Transformation Design](references/transformation-design.md) |
124
+ | Optimize temporal/FIFO state | [Velocity Feature Calculation](references/velocity-feature-calculation.md) |
125
+ | Reconcile published sample vs previous path | [Reconciliation](references/reconciliation.md) |
126
+ | Run production validation (canary, optional profile, write, reconciliation) | [Production Validation](references/production-validation.md) |
127
+ | Summarize an event log | `scripts/spark_eventlog_summary.py` |