@1aboveio/skills 0.15.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -2
- package/package.json +1 -1
- package/runtime/skills/distribution/generated/recipes.json +178 -34
- package/runtime/skills/distribution/scripts/bundles.mjs +11 -3
- package/runtime/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/data-science/pyspark/SKILL.md +126 -0
- package/skills/{backend → data-science}/pyspark/assets/templates/etl.py +51 -0
- package/skills/{backend → data-science}/pyspark/references/diagnosis-and-profiling.md +38 -14
- package/skills/{backend → data-science}/pyspark/references/etl-contract.md +19 -0
- package/skills/data-science/pyspark/references/production-validation.md +131 -0
- package/skills/data-science/pyspark/references/reconciliation.md +38 -0
- package/skills/{backend → data-science}/pyspark/references/transformation-design.md +30 -2
- package/skills/engineering/engineering-runtime/coherence/workflow.json +14 -14
- package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/engineering/resolve-issues/generated/workflow-repair-policy.json +11 -11
- package/skills/engineering/resolve-issues/scripts/run-state.mjs +1 -1
- package/skills/payment/fraud-analysis/LICENSE +3 -0
- package/skills/payment/fraud-analysis/SKILL.md +113 -0
- package/skills/payment/fraud-analysis/evals/evals.json +40 -0
- package/skills/payment/fraud-analysis/references/archetypes/authorized-payment-scam.md +41 -0
- package/skills/payment/fraud-analysis/references/archetypes/first-party-fraud.md +44 -0
- package/skills/payment/fraud-analysis/references/archetypes/third-party-fraud.md +27 -0
- package/skills/payment/fraud-analysis/references/contexts/bank-transfer.md +24 -0
- package/skills/payment/fraud-analysis/references/contexts/card-payment.md +30 -0
- package/skills/payment/fraud-analysis/references/contexts/payment-collection.md +20 -0
- package/skills/payment/fraud-analysis/references/contexts/payout.md +20 -0
- package/skills/payment/fraud-analysis/references/feature-engineering.md +158 -0
- package/skills/payment/fraud-analysis/references/mechanisms/account-takeover.md +36 -0
- package/skills/payment/fraud-analysis/references/report-rationale.md +45 -0
- package/skills/payment/fraud-analysis/references/report-template.md +190 -0
- package/skills/payment/fraud-analysis/references/review-checklist.md +175 -0
- package/skills/payment/fraud-analysis/references/taxonomy.md +79 -0
- package/skills/payment/fraud-analysis/references/terminology.md +108 -0
- package/skills/payment/fraud-analysis/references/workflow.md +175 -0
- package/skills/payment/payment-analysis/LICENSE +3 -0
- package/skills/payment/payment-analysis/SKILL.md +127 -0
- package/skills/payment/payment-analysis/references/auth-rate-actions.md +30 -0
- package/skills/payment/payment-analysis/references/chargebacks.md +88 -0
- package/skills/payment/payment-analysis/references/event-layers.md +79 -0
- package/skills/payment/payment-analysis/references/fx.md +59 -0
- package/skills/payment/payment-analysis/references/journey.md +78 -0
- package/skills/payment/payment-analysis/references/metrics.md +62 -0
- package/skills/payment/payment-analysis/references/report-template.md +98 -0
- package/skills/payment/payment-analysis/references/terminology.md +85 -0
- package/skills/payment/payment-analysis/references/visualization.md +47 -0
- package/skills/backend/pyspark/SKILL.md +0 -116
- package/skills/backend/pyspark/references/parity-testing.md +0 -83
- package/skills/backend/pyspark/references/production-validation.md +0 -166
- /package/skills/{backend → data-science}/airflow-dag-develop/LICENSE +0 -0
- /package/skills/{backend → data-science}/airflow-dag-develop/SKILL.md +0 -0
- /package/skills/{backend → data-science}/pyspark/LICENSE +0 -0
- /package/skills/{backend → data-science}/pyspark/assets/templates/utils/__init__.py +0 -0
- /package/skills/{backend → data-science}/pyspark/assets/templates/utils/hudi_metadata.py +0 -0
- /package/skills/{backend → data-science}/pyspark/references/velocity-feature-calculation.md +0 -0
- /package/skills/{backend → data-science}/pyspark/scripts/spark_eventlog_summary.py +0 -0
package/README.md
CHANGED
|
@@ -67,7 +67,9 @@ planned same-named skill. The coordinator still owns group and new-member consen
|
|
|
67
67
|
|
|
68
68
|
- `engineering-workflow` — Engineering Workflow: e2e-test, ensure-coverage, implement-and-pr, resolve-issues, resolve-release, review-pr, smoke, engineering-runtime
|
|
69
69
|
- `delivery-infrastructure` — Delivery Infrastructure: cloud-build, cloud-debug, cloud-deploy, docker, google-cloud, mergify, podman
|
|
70
|
-
- `backend` — Backend:
|
|
70
|
+
- `backend` — Backend: app-debug, python-backend
|
|
71
|
+
- `data-science` — Data Science: airflow-dag-develop, pyspark
|
|
72
|
+
- `payment` — Payment: fraud-analysis, payment-analysis
|
|
71
73
|
- `fullstack` — Fullstack: better-auth, monorepo, nextjs-fullstack, prisma-setup, shadcn, zod-v4
|
|
72
74
|
|
|
73
75
|
Internal dependencies, installed with the groups above and not selectable on their own: `harness-runtime`.
|
|
@@ -119,7 +121,11 @@ npx skills@1.5.22 remove --global --agent universal claude-code --skill e2e-test
|
|
|
119
121
|
# Delivery Infrastructure
|
|
120
122
|
npx skills@1.5.22 remove --global --agent universal claude-code --skill cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman
|
|
121
123
|
# Backend
|
|
122
|
-
npx skills@1.5.22 remove --global --agent universal claude-code --skill
|
|
124
|
+
npx skills@1.5.22 remove --global --agent universal claude-code --skill app-debug python-backend
|
|
125
|
+
# Data Science
|
|
126
|
+
npx skills@1.5.22 remove --global --agent universal claude-code --skill airflow-dag-develop pyspark
|
|
127
|
+
# Payment
|
|
128
|
+
npx skills@1.5.22 remove --global --agent universal claude-code --skill fraud-analysis payment-analysis
|
|
123
129
|
# Fullstack
|
|
124
130
|
npx skills@1.5.22 remove --global --agent universal claude-code --skill better-auth monorepo nextjs-fullstack prisma-setup shadcn zod-v4
|
|
125
131
|
```
|
package/package.json
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
"id": "first-party",
|
|
9
9
|
"type": "first-party",
|
|
10
10
|
"package": "@1aboveio/skills",
|
|
11
|
-
"version": "0.
|
|
11
|
+
"version": "0.17.0",
|
|
12
12
|
"updatePolicy": "pinned-npm-version",
|
|
13
13
|
"members": [
|
|
14
14
|
{
|
|
@@ -102,26 +102,38 @@
|
|
|
102
102
|
"licenseResponsibility": "1aboveio-mit"
|
|
103
103
|
},
|
|
104
104
|
{
|
|
105
|
-
"installName": "
|
|
106
|
-
"path": "skills/backend/
|
|
105
|
+
"installName": "app-debug",
|
|
106
|
+
"path": "skills/backend/app-debug",
|
|
107
107
|
"role": "skill",
|
|
108
108
|
"licenseResponsibility": "1aboveio-mit"
|
|
109
109
|
},
|
|
110
110
|
{
|
|
111
|
-
"installName": "
|
|
112
|
-
"path": "skills/backend/
|
|
111
|
+
"installName": "python-backend",
|
|
112
|
+
"path": "skills/backend/python-backend",
|
|
113
|
+
"role": "skill",
|
|
114
|
+
"licenseResponsibility": "1aboveio-mit"
|
|
115
|
+
},
|
|
116
|
+
{
|
|
117
|
+
"installName": "airflow-dag-develop",
|
|
118
|
+
"path": "skills/data-science/airflow-dag-develop",
|
|
113
119
|
"role": "skill",
|
|
114
120
|
"licenseResponsibility": "1aboveio-mit"
|
|
115
121
|
},
|
|
116
122
|
{
|
|
117
123
|
"installName": "pyspark",
|
|
118
|
-
"path": "skills/
|
|
124
|
+
"path": "skills/data-science/pyspark",
|
|
119
125
|
"role": "skill",
|
|
120
126
|
"licenseResponsibility": "1aboveio-mit"
|
|
121
127
|
},
|
|
122
128
|
{
|
|
123
|
-
"installName": "
|
|
124
|
-
"path": "skills/
|
|
129
|
+
"installName": "fraud-analysis",
|
|
130
|
+
"path": "skills/payment/fraud-analysis",
|
|
131
|
+
"role": "skill",
|
|
132
|
+
"licenseResponsibility": "1aboveio-mit"
|
|
133
|
+
},
|
|
134
|
+
{
|
|
135
|
+
"installName": "payment-analysis",
|
|
136
|
+
"path": "skills/payment/payment-analysis",
|
|
125
137
|
"role": "skill",
|
|
126
138
|
"licenseResponsibility": "1aboveio-mit"
|
|
127
139
|
},
|
|
@@ -184,7 +196,7 @@
|
|
|
184
196
|
"sourceId": "first-party",
|
|
185
197
|
"sourceType": "first-party",
|
|
186
198
|
"package": "@1aboveio/skills",
|
|
187
|
-
"version": "0.
|
|
199
|
+
"version": "0.17.0",
|
|
188
200
|
"installPath": null,
|
|
189
201
|
"members": [
|
|
190
202
|
"harness-runtime",
|
|
@@ -218,7 +230,7 @@
|
|
|
218
230
|
"sourceId": "first-party",
|
|
219
231
|
"sourceType": "first-party",
|
|
220
232
|
"package": "@1aboveio/skills",
|
|
221
|
-
"version": "0.
|
|
233
|
+
"version": "0.17.0",
|
|
222
234
|
"installPath": null,
|
|
223
235
|
"members": [
|
|
224
236
|
"harness-runtime",
|
|
@@ -262,7 +274,7 @@
|
|
|
262
274
|
"sourceId": "first-party",
|
|
263
275
|
"sourceType": "first-party",
|
|
264
276
|
"package": "@1aboveio/skills",
|
|
265
|
-
"version": "0.
|
|
277
|
+
"version": "0.17.0",
|
|
266
278
|
"installPath": null,
|
|
267
279
|
"members": [
|
|
268
280
|
"harness-runtime",
|
|
@@ -295,7 +307,7 @@
|
|
|
295
307
|
"sourceId": "first-party",
|
|
296
308
|
"sourceType": "first-party",
|
|
297
309
|
"package": "@1aboveio/skills",
|
|
298
|
-
"version": "0.
|
|
310
|
+
"version": "0.17.0",
|
|
299
311
|
"installPath": null,
|
|
300
312
|
"members": [
|
|
301
313
|
"harness-runtime",
|
|
@@ -338,19 +350,17 @@
|
|
|
338
350
|
"sourceId": "first-party",
|
|
339
351
|
"sourceType": "first-party",
|
|
340
352
|
"package": "@1aboveio/skills",
|
|
341
|
-
"version": "0.
|
|
353
|
+
"version": "0.17.0",
|
|
342
354
|
"installPath": null,
|
|
343
355
|
"members": [
|
|
344
356
|
"harness-runtime",
|
|
345
|
-
"airflow-dag-develop",
|
|
346
357
|
"app-debug",
|
|
347
|
-
"pyspark",
|
|
348
358
|
"python-backend"
|
|
349
359
|
],
|
|
350
360
|
"commands": [
|
|
351
361
|
{
|
|
352
362
|
"transport": "local",
|
|
353
|
-
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime
|
|
363
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime app-debug python-backend"
|
|
354
364
|
}
|
|
355
365
|
],
|
|
356
366
|
"onFailure": {
|
|
@@ -368,19 +378,149 @@
|
|
|
368
378
|
"sourceId": "first-party",
|
|
369
379
|
"sourceType": "first-party",
|
|
370
380
|
"package": "@1aboveio/skills",
|
|
371
|
-
"version": "0.
|
|
381
|
+
"version": "0.17.0",
|
|
372
382
|
"installPath": null,
|
|
373
383
|
"members": [
|
|
374
384
|
"harness-runtime",
|
|
375
|
-
"airflow-dag-develop",
|
|
376
385
|
"app-debug",
|
|
377
|
-
"pyspark",
|
|
378
386
|
"python-backend"
|
|
379
387
|
],
|
|
380
388
|
"commands": [
|
|
381
389
|
{
|
|
382
390
|
"transport": "local",
|
|
383
|
-
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime
|
|
391
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime app-debug python-backend"
|
|
392
|
+
}
|
|
393
|
+
],
|
|
394
|
+
"onFailure": {
|
|
395
|
+
"action": "stop",
|
|
396
|
+
"message": "Source first-party fetch failed; stop this recipe before running another source."
|
|
397
|
+
}
|
|
398
|
+
}
|
|
399
|
+
]
|
|
400
|
+
},
|
|
401
|
+
"uninstall": {
|
|
402
|
+
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill app-debug python-backend",
|
|
403
|
+
"removesDependencies": false
|
|
404
|
+
}
|
|
405
|
+
},
|
|
406
|
+
{
|
|
407
|
+
"id": "data-science",
|
|
408
|
+
"displayName": "Data Science",
|
|
409
|
+
"dependsOn": [
|
|
410
|
+
"harness-runtime"
|
|
411
|
+
],
|
|
412
|
+
"install": {
|
|
413
|
+
"stopOnFailure": true,
|
|
414
|
+
"steps": [
|
|
415
|
+
{
|
|
416
|
+
"sourceId": "first-party",
|
|
417
|
+
"sourceType": "first-party",
|
|
418
|
+
"package": "@1aboveio/skills",
|
|
419
|
+
"version": "0.17.0",
|
|
420
|
+
"installPath": null,
|
|
421
|
+
"members": [
|
|
422
|
+
"harness-runtime",
|
|
423
|
+
"airflow-dag-develop",
|
|
424
|
+
"pyspark"
|
|
425
|
+
],
|
|
426
|
+
"commands": [
|
|
427
|
+
{
|
|
428
|
+
"transport": "local",
|
|
429
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime airflow-dag-develop pyspark"
|
|
430
|
+
}
|
|
431
|
+
],
|
|
432
|
+
"onFailure": {
|
|
433
|
+
"action": "stop",
|
|
434
|
+
"message": "Source first-party fetch failed; stop this recipe before running another source."
|
|
435
|
+
}
|
|
436
|
+
}
|
|
437
|
+
]
|
|
438
|
+
},
|
|
439
|
+
"update": {
|
|
440
|
+
"mode": "rerun-install-recipe",
|
|
441
|
+
"stopOnFailure": true,
|
|
442
|
+
"steps": [
|
|
443
|
+
{
|
|
444
|
+
"sourceId": "first-party",
|
|
445
|
+
"sourceType": "first-party",
|
|
446
|
+
"package": "@1aboveio/skills",
|
|
447
|
+
"version": "0.17.0",
|
|
448
|
+
"installPath": null,
|
|
449
|
+
"members": [
|
|
450
|
+
"harness-runtime",
|
|
451
|
+
"airflow-dag-develop",
|
|
452
|
+
"pyspark"
|
|
453
|
+
],
|
|
454
|
+
"commands": [
|
|
455
|
+
{
|
|
456
|
+
"transport": "local",
|
|
457
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime airflow-dag-develop pyspark"
|
|
458
|
+
}
|
|
459
|
+
],
|
|
460
|
+
"onFailure": {
|
|
461
|
+
"action": "stop",
|
|
462
|
+
"message": "Source first-party fetch failed; stop this recipe before running another source."
|
|
463
|
+
}
|
|
464
|
+
}
|
|
465
|
+
]
|
|
466
|
+
},
|
|
467
|
+
"uninstall": {
|
|
468
|
+
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill airflow-dag-develop pyspark",
|
|
469
|
+
"removesDependencies": false
|
|
470
|
+
}
|
|
471
|
+
},
|
|
472
|
+
{
|
|
473
|
+
"id": "payment",
|
|
474
|
+
"displayName": "Payment",
|
|
475
|
+
"dependsOn": [
|
|
476
|
+
"harness-runtime"
|
|
477
|
+
],
|
|
478
|
+
"install": {
|
|
479
|
+
"stopOnFailure": true,
|
|
480
|
+
"steps": [
|
|
481
|
+
{
|
|
482
|
+
"sourceId": "first-party",
|
|
483
|
+
"sourceType": "first-party",
|
|
484
|
+
"package": "@1aboveio/skills",
|
|
485
|
+
"version": "0.17.0",
|
|
486
|
+
"installPath": null,
|
|
487
|
+
"members": [
|
|
488
|
+
"harness-runtime",
|
|
489
|
+
"fraud-analysis",
|
|
490
|
+
"payment-analysis"
|
|
491
|
+
],
|
|
492
|
+
"commands": [
|
|
493
|
+
{
|
|
494
|
+
"transport": "local",
|
|
495
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime fraud-analysis payment-analysis"
|
|
496
|
+
}
|
|
497
|
+
],
|
|
498
|
+
"onFailure": {
|
|
499
|
+
"action": "stop",
|
|
500
|
+
"message": "Source first-party fetch failed; stop this recipe before running another source."
|
|
501
|
+
}
|
|
502
|
+
}
|
|
503
|
+
]
|
|
504
|
+
},
|
|
505
|
+
"update": {
|
|
506
|
+
"mode": "rerun-install-recipe",
|
|
507
|
+
"stopOnFailure": true,
|
|
508
|
+
"steps": [
|
|
509
|
+
{
|
|
510
|
+
"sourceId": "first-party",
|
|
511
|
+
"sourceType": "first-party",
|
|
512
|
+
"package": "@1aboveio/skills",
|
|
513
|
+
"version": "0.17.0",
|
|
514
|
+
"installPath": null,
|
|
515
|
+
"members": [
|
|
516
|
+
"harness-runtime",
|
|
517
|
+
"fraud-analysis",
|
|
518
|
+
"payment-analysis"
|
|
519
|
+
],
|
|
520
|
+
"commands": [
|
|
521
|
+
{
|
|
522
|
+
"transport": "local",
|
|
523
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime fraud-analysis payment-analysis"
|
|
384
524
|
}
|
|
385
525
|
],
|
|
386
526
|
"onFailure": {
|
|
@@ -391,7 +531,7 @@
|
|
|
391
531
|
]
|
|
392
532
|
},
|
|
393
533
|
"uninstall": {
|
|
394
|
-
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill
|
|
534
|
+
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill fraud-analysis payment-analysis",
|
|
395
535
|
"removesDependencies": false
|
|
396
536
|
}
|
|
397
537
|
},
|
|
@@ -408,7 +548,7 @@
|
|
|
408
548
|
"sourceId": "first-party",
|
|
409
549
|
"sourceType": "first-party",
|
|
410
550
|
"package": "@1aboveio/skills",
|
|
411
|
-
"version": "0.
|
|
551
|
+
"version": "0.17.0",
|
|
412
552
|
"installPath": null,
|
|
413
553
|
"members": [
|
|
414
554
|
"harness-runtime",
|
|
@@ -440,7 +580,7 @@
|
|
|
440
580
|
"sourceId": "first-party",
|
|
441
581
|
"sourceType": "first-party",
|
|
442
582
|
"package": "@1aboveio/skills",
|
|
443
|
-
"version": "0.
|
|
583
|
+
"version": "0.17.0",
|
|
444
584
|
"installPath": null,
|
|
445
585
|
"members": [
|
|
446
586
|
"harness-runtime",
|
|
@@ -480,7 +620,7 @@
|
|
|
480
620
|
"sourceId": "first-party",
|
|
481
621
|
"sourceType": "first-party",
|
|
482
622
|
"package": "@1aboveio/skills",
|
|
483
|
-
"version": "0.
|
|
623
|
+
"version": "0.17.0",
|
|
484
624
|
"installPath": null,
|
|
485
625
|
"members": [
|
|
486
626
|
"harness-runtime"
|
|
@@ -506,7 +646,7 @@
|
|
|
506
646
|
"sourceId": "first-party",
|
|
507
647
|
"sourceType": "first-party",
|
|
508
648
|
"package": "@1aboveio/skills",
|
|
509
|
-
"version": "0.
|
|
649
|
+
"version": "0.17.0",
|
|
510
650
|
"installPath": null,
|
|
511
651
|
"members": [
|
|
512
652
|
"harness-runtime"
|
|
@@ -538,7 +678,7 @@
|
|
|
538
678
|
"sourceId": "first-party",
|
|
539
679
|
"sourceType": "first-party",
|
|
540
680
|
"package": "@1aboveio/skills",
|
|
541
|
-
"version": "0.
|
|
681
|
+
"version": "0.17.0",
|
|
542
682
|
"installPath": null,
|
|
543
683
|
"members": [
|
|
544
684
|
"harness-runtime",
|
|
@@ -557,10 +697,12 @@
|
|
|
557
697
|
"google-cloud",
|
|
558
698
|
"mergify",
|
|
559
699
|
"podman",
|
|
560
|
-
"airflow-dag-develop",
|
|
561
700
|
"app-debug",
|
|
562
|
-
"pyspark",
|
|
563
701
|
"python-backend",
|
|
702
|
+
"airflow-dag-develop",
|
|
703
|
+
"pyspark",
|
|
704
|
+
"fraud-analysis",
|
|
705
|
+
"payment-analysis",
|
|
564
706
|
"better-auth",
|
|
565
707
|
"monorepo",
|
|
566
708
|
"nextjs-fullstack",
|
|
@@ -571,7 +713,7 @@
|
|
|
571
713
|
"commands": [
|
|
572
714
|
{
|
|
573
715
|
"transport": "local",
|
|
574
|
-
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman airflow-dag-develop
|
|
716
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman app-debug python-backend airflow-dag-develop pyspark fraud-analysis payment-analysis better-auth monorepo nextjs-fullstack prisma-setup shadcn zod-v4 && npm --prefix \"$(realpath \"$HOME/.agents/skills/engineering-runtime/scripts\")\" ci --ignore-scripts"
|
|
575
717
|
}
|
|
576
718
|
],
|
|
577
719
|
"onFailure": {
|
|
@@ -589,7 +731,7 @@
|
|
|
589
731
|
"sourceId": "first-party",
|
|
590
732
|
"sourceType": "first-party",
|
|
591
733
|
"package": "@1aboveio/skills",
|
|
592
|
-
"version": "0.
|
|
734
|
+
"version": "0.17.0",
|
|
593
735
|
"installPath": null,
|
|
594
736
|
"members": [
|
|
595
737
|
"harness-runtime",
|
|
@@ -608,10 +750,12 @@
|
|
|
608
750
|
"google-cloud",
|
|
609
751
|
"mergify",
|
|
610
752
|
"podman",
|
|
611
|
-
"airflow-dag-develop",
|
|
612
753
|
"app-debug",
|
|
613
|
-
"pyspark",
|
|
614
754
|
"python-backend",
|
|
755
|
+
"airflow-dag-develop",
|
|
756
|
+
"pyspark",
|
|
757
|
+
"fraud-analysis",
|
|
758
|
+
"payment-analysis",
|
|
615
759
|
"better-auth",
|
|
616
760
|
"monorepo",
|
|
617
761
|
"nextjs-fullstack",
|
|
@@ -622,7 +766,7 @@
|
|
|
622
766
|
"commands": [
|
|
623
767
|
{
|
|
624
768
|
"transport": "local",
|
|
625
|
-
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman airflow-dag-develop
|
|
769
|
+
"command": "npx skills@1.5.22 add . --global --agent universal claude-code --skill harness-runtime e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman app-debug python-backend airflow-dag-develop pyspark fraud-analysis payment-analysis better-auth monorepo nextjs-fullstack prisma-setup shadcn zod-v4 && npm --prefix \"$(realpath \"$HOME/.agents/skills/engineering-runtime/scripts\")\" ci --ignore-scripts"
|
|
626
770
|
}
|
|
627
771
|
],
|
|
628
772
|
"onFailure": {
|
|
@@ -633,7 +777,7 @@
|
|
|
633
777
|
]
|
|
634
778
|
},
|
|
635
779
|
"uninstall": {
|
|
636
|
-
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman airflow-dag-develop
|
|
780
|
+
"command": "npx skills@1.5.22 remove --global --agent universal claude-code --skill e2e-test ensure-coverage implement-and-pr resolve-issues resolve-release review-pr smoke engineering-runtime cloud-build cloud-debug cloud-deploy docker google-cloud mergify podman app-debug python-backend airflow-dag-develop pyspark fraud-analysis payment-analysis better-auth monorepo nextjs-fullstack prisma-setup shadcn zod-v4 harness-runtime"
|
|
637
781
|
}
|
|
638
782
|
}
|
|
639
783
|
}
|
|
@@ -38,6 +38,10 @@ const COMMAND_TRANSPORTS = ['local', 'https'];
|
|
|
38
38
|
// is never presented as something a user picks. The picker, `list`, the plan and the generated
|
|
39
39
|
// package README all derive their answer from here rather than each re-listing the ids.
|
|
40
40
|
export const SUPPORT_GROUPS = new Set(['harness-runtime']);
|
|
41
|
+
// External toolkits stay in the manifest for standalone docs/canary, but the coordinator — checkout
|
|
42
|
+
// and the public package alike — does not offer or install them.
|
|
43
|
+
export const EXTERNAL_ONLY_GROUPS = new Set(['matt-pocock-toolkit']);
|
|
44
|
+
const HIDDEN_GROUPS = new Set([...SUPPORT_GROUPS, ...EXTERNAL_ONLY_GROUPS]);
|
|
41
45
|
const ACTIONS = new Set(['install', 'update']);
|
|
42
46
|
|
|
43
47
|
// Selection memory (ADR 0002/0004 amendment, epic #1124): the remembered user-facing group
|
|
@@ -113,7 +117,10 @@ function writeSelectionState(state, env = process.env) {
|
|
|
113
117
|
// user-selected) are dropped with a warning; memory never resurrects them and never falls back to
|
|
114
118
|
// filesystem inference. Nothing left is an error, not an empty plan.
|
|
115
119
|
function rememberedGroupIds(state, catalog, stderr) {
|
|
116
|
-
const known = new Set(
|
|
120
|
+
const known = new Set([
|
|
121
|
+
...catalog.map((bundle) => bundle.id),
|
|
122
|
+
...EXTERNAL_ONLY_GROUPS,
|
|
123
|
+
]);
|
|
117
124
|
const kept = [];
|
|
118
125
|
const dropped = [];
|
|
119
126
|
for (const group of state.groups) {
|
|
@@ -256,7 +263,7 @@ function assertLocalRoot(localRoot) {
|
|
|
256
263
|
|
|
257
264
|
export function bundleCatalog(transport = {}) {
|
|
258
265
|
return readRecipes(resolveBundleTransport(transport).recipesPath).groups
|
|
259
|
-
.filter((group) => !
|
|
266
|
+
.filter((group) => !HIDDEN_GROUPS.has(group.id))
|
|
260
267
|
.map((group) => ({
|
|
261
268
|
id: group.id,
|
|
262
269
|
displayName: group.displayName,
|
|
@@ -744,7 +751,8 @@ async function consentNewMembers(candidates, { yes, stdin, stdout, stderr }) {
|
|
|
744
751
|
|
|
745
752
|
function selectionIncludesTdd(groupIds, catalog) {
|
|
746
753
|
const selected = new Set(groupIds);
|
|
747
|
-
return catalog.some((bundle) => selected.has(bundle.id) && bundle.members.includes('tdd'))
|
|
754
|
+
return catalog.some((bundle) => selected.has(bundle.id) && bundle.members.includes('tdd'))
|
|
755
|
+
|| groupIds.includes('matt-pocock-toolkit');
|
|
748
756
|
}
|
|
749
757
|
|
|
750
758
|
// Dependencies are injectable so the coordinator seam can be tested black-box: a stubbed native
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: pyspark
|
|
3
|
+
description: >
|
|
4
|
+
Implement, debug, review, and optimize Apache Spark or PySpark batch jobs,
|
|
5
|
+
ETL pipelines, feature calculations, and Hudi/warehouse writes. Follow
|
|
6
|
+
contract → transform → materialize → local test → production validation
|
|
7
|
+
(one-month canary, optional profile if that canary exceeds 5 minutes without
|
|
8
|
+
resource constraints or any job exceeds 30 minutes, authorized write, cluster
|
|
9
|
+
reconciliation). Use whenever
|
|
10
|
+
the task involves Spark DataFrames or Spark SQL, joins, windows, grouped
|
|
11
|
+
state, UDFs, shuffle, skew, spill, slow stages, driver/executor OOM, Spark
|
|
12
|
+
Connect, spark-submit, Hudi commits, backfills, validate-only canaries,
|
|
13
|
+
cluster reconciliation, or Spark event-log profiling — including warehouse
|
|
14
|
+
job timeouts, empty Hudi writes, or EMR/Dataproc performance regressions.
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# PySpark
|
|
18
|
+
|
|
19
|
+
Use this skill as the workflow and routing index. Read the playbook for the
|
|
20
|
+
step you are on before editing code or submitting to the cluster.
|
|
21
|
+
|
|
22
|
+
## Workflow
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
contract -> transform -> materialize -> test -> production validation
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Follow all five steps in order.
|
|
29
|
+
|
|
30
|
+
### 1. Contract
|
|
31
|
+
|
|
32
|
+
- **Input:** declare all parameters, defaults, validation, source snapshot/date
|
|
33
|
+
scope, overlapping-run behavior, idempotency/single-writer rules, and exact
|
|
34
|
+
Python and PySpark versions.
|
|
35
|
+
- **Output:** declare plain Spark versus Hudi table type, output identity/schema,
|
|
36
|
+
partition columns, record key, deterministic order or precombine field, and
|
|
37
|
+
append/upsert/overwrite write mode.
|
|
38
|
+
- **Conventions:** declare native Spark versus Spark Connect and full rebuild
|
|
39
|
+
versus incremental processing, including the incremental watermark,
|
|
40
|
+
lookback/checkpoint, and restart boundary when applicable.
|
|
41
|
+
|
|
42
|
+
Read [ETL Contract](references/etl-contract.md). When the repository has no ETL
|
|
43
|
+
base module, adapt [`assets/templates/etl.py`](assets/templates/etl.py).
|
|
44
|
+
|
|
45
|
+
### 2. Transform
|
|
46
|
+
|
|
47
|
+
- Make `transform` accept DataFrames and return the complete candidate without
|
|
48
|
+
invoking validation or a writer.
|
|
49
|
+
- Estimate rows after joins/explodes/groups and measure maximum rows per key.
|
|
50
|
+
Fix algorithmic expansion and hot keys before tuning Spark configuration.
|
|
51
|
+
- Prefer built-in expressions and windows. Use exact bucketing or one sorted
|
|
52
|
+
grouped sweep only for state that built-ins cannot express cleanly.
|
|
53
|
+
- Preserve complete row identity, ordering, output columns, and required
|
|
54
|
+
behavior.
|
|
55
|
+
- Avoid explicit `.cache()` and `.persist()` by default.
|
|
56
|
+
|
|
57
|
+
Read [Transformation Design](references/transformation-design.md). For temporal
|
|
58
|
+
windows or FIFO paths, also read
|
|
59
|
+
[Velocity Feature Calculation](references/velocity-feature-calculation.md).
|
|
60
|
+
|
|
61
|
+
### 3. Materialize
|
|
62
|
+
|
|
63
|
+
- Treat materialization as the Spark action that executes the lazy candidate,
|
|
64
|
+
not as caching.
|
|
65
|
+
- Use one aggregate validation action before every writer. Reference every
|
|
66
|
+
derived output so Catalyst cannot prune untested calculations.
|
|
67
|
+
- Check row count, distinct/null/duplicate identities, domain blockers, and
|
|
68
|
+
required date/partition scope. Fail closed on any blocker.
|
|
69
|
+
- Bind validation and publication to the same immutable source snapshot because
|
|
70
|
+
Spark may recompute the candidate.
|
|
71
|
+
- Keep the validate-only guard inside the job and before every writer. Do not
|
|
72
|
+
add `.count()`, `.cache()`, or `.persist()` to warm execution.
|
|
73
|
+
- Do not set Spark or Hudi shuffle parallelism (`spark.sql.shuffle.partitions`,
|
|
74
|
+
`hoodie.*.shuffle.parallelism`). Those knobs split the write into many tiny
|
|
75
|
+
files. Leave Spark/Hudi defaults; size writes with `HudiLayout`. Repair an
|
|
76
|
+
existing small-file table with `Etl.rewrite_hudi_layout` (see
|
|
77
|
+
[ETL Contract](references/etl-contract.md)).
|
|
78
|
+
|
|
79
|
+
### 4. Test
|
|
80
|
+
|
|
81
|
+
Run **local** tests only. Do not treat a laptop Spark session as cluster
|
|
82
|
+
evidence.
|
|
83
|
+
|
|
84
|
+
- Pure state, tiny real-Spark, plan-shape, and focused repository tests.
|
|
85
|
+
- Cover boundaries, ties, nulls, arithmetic, identity, and precedence.
|
|
86
|
+
|
|
87
|
+
### 5. Production validation
|
|
88
|
+
|
|
89
|
+
Run the production entry point on the cluster against a **small sample**,
|
|
90
|
+
typically **one month** of source data. Local tests do not substitute.
|
|
91
|
+
|
|
92
|
+
1. **Canary:** `spark-submit` with `--validate-only` on that sample and the
|
|
93
|
+
production runtime. Require zero blockers, an explicit publication skip,
|
|
94
|
+
zero Spark output records/bytes, and readable driver/event logs.
|
|
95
|
+
2. **Profiling (optional):** required when **either** (a) the small-sample
|
|
96
|
+
canary wall time exceeds **5 minutes** **and** the run was not
|
|
97
|
+
resource-constrained (queued, under-provisioned executors, competing
|
|
98
|
+
warehouse work), **or** (b) **any** job (canary, write, or scheduled run)
|
|
99
|
+
exceeds **30 minutes** — a hard gate, even if the cluster was constrained.
|
|
100
|
+
Otherwise skip. When required, follow
|
|
101
|
+
[Diagnosis And Profiling](references/diagnosis-and-profiling.md).
|
|
102
|
+
3. **Write (authorized):** re-check admission, then publish the same pinned
|
|
103
|
+
sample. Do not present canary time as write performance.
|
|
104
|
+
4. **Reconciliation:** independently prove the published sample. Follow
|
|
105
|
+
[Reconciliation](references/reconciliation.md).
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
spark-submit <production-options> job.py --validate-only
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Restore scheduler/process ownership after the run.
|
|
112
|
+
|
|
113
|
+
Read [Production Validation](references/production-validation.md) for admission,
|
|
114
|
+
canary evidence, and write rules.
|
|
115
|
+
|
|
116
|
+
## Reference Index
|
|
117
|
+
|
|
118
|
+
| Need | Read or run |
|
|
119
|
+
|---|---|
|
|
120
|
+
| Define or review an ETL job contract | [ETL Contract](references/etl-contract.md) and `assets/templates/etl.py` |
|
|
121
|
+
| Diagnose or profile | [Diagnosis And Profiling](references/diagnosis-and-profiling.md) |
|
|
122
|
+
| Design transformations | [Transformation Design](references/transformation-design.md) |
|
|
123
|
+
| Optimize temporal/FIFO state | [Velocity Feature Calculation](references/velocity-feature-calculation.md) |
|
|
124
|
+
| Reconcile published sample vs previous path | [Reconciliation](references/reconciliation.md) |
|
|
125
|
+
| Run production validation (canary, optional profile, write, reconciliation) | [Production Validation](references/production-validation.md) |
|
|
126
|
+
| Summarize an event log | `scripts/spark_eventlog_summary.py` |
|