@1aboveio/skills 0.10.1 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/package.json +1 -1
- package/runtime/skills/distribution/generated/recipes.json +29 -19
- package/skills/backend/pyspark/LICENSE +3 -0
- package/skills/backend/pyspark/SKILL.md +116 -0
- package/skills/backend/pyspark/assets/templates/etl.py +696 -0
- package/skills/backend/pyspark/assets/templates/utils/__init__.py +1 -0
- package/skills/backend/pyspark/assets/templates/utils/hudi_metadata.py +14 -0
- package/skills/backend/pyspark/references/diagnosis-and-profiling.md +145 -0
- package/skills/backend/pyspark/references/etl-contract.md +107 -0
- package/skills/backend/pyspark/references/parity-testing.md +83 -0
- package/skills/backend/pyspark/references/production-validation.md +166 -0
- package/skills/backend/pyspark/references/transformation-design.md +162 -0
- package/skills/backend/pyspark/references/velocity-feature-calculation.md +193 -0
- package/skills/backend/pyspark/scripts/spark_eventlog_summary.py +223 -0
- package/skills/cicd-pipeline/mergify/SKILL.md +1 -0
- package/skills/cicd-pipeline/mergify/references/configuration.md +12 -6
- package/skills/cicd-pipeline/mergify/references/traps.md +28 -0
- package/skills/engineering/engineering-runtime/coherence/workflow.json +17 -17
- package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/engineering/implement-and-pr/SKILL.md +4 -3
- package/skills/engineering/implement-and-pr/agents/openai.yaml +9 -0
- package/skills/engineering/resolve-issues/SKILL.md +4 -3
- package/skills/engineering/resolve-issues/agents/openai.yaml +9 -0
- package/skills/engineering/resolve-issues/generated/workflow-repair-policy.json +11 -11
- package/skills/engineering/resolve-issues/references/pre-flight-model-slots.md +2 -2
- package/skills/engineering/resolve-issues/references/pre-flight-recording-and-checkout.md +1 -1
- package/skills/engineering/resolve-issues/references/pre-flight.md +1 -1
- package/skills/engineering/resolve-issues/scripts/preflight-questions.mjs +32 -9
- package/skills/engineering/resolve-issues/scripts/run-state.mjs +1 -1
- package/skills/engineering/resolve-release/references/preflight.md +3 -2
- package/skills/engineering/resolve-release/scripts/preflight-probes.mjs +13 -1
- package/skills/engineering/review-pr/SKILL.md +2 -1
- package/skills/engineering/review-pr/agents/openai.yaml +9 -0
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# Velocity Feature Calculation Best Practices
|
|
2
|
+
|
|
3
|
+
Use these practices for rolling counts and sums, distinct counterparties,
|
|
4
|
+
temporal velocity features, and FIFO/path calculations over event histories.
|
|
5
|
+
The snippets illustrate the design; adapt schemas and semantics to the job's
|
|
6
|
+
required behavior.
|
|
7
|
+
|
|
8
|
+
## Use Built-Ins When Velocity Metrics Are Independent
|
|
9
|
+
|
|
10
|
+
For ordinary rolling counts and sums where same-time peers share one frame,
|
|
11
|
+
prefer a Spark range window:
|
|
12
|
+
|
|
13
|
+
```python
|
|
14
|
+
from pyspark.sql import Window, functions as F
|
|
15
|
+
|
|
16
|
+
last_24h = (
|
|
17
|
+
Window.partitionBy("entity_id")
|
|
18
|
+
.orderBy(F.col("event_time").cast("long"))
|
|
19
|
+
.rangeBetween(-24 * 60 * 60, 0)
|
|
20
|
+
)
|
|
21
|
+
|
|
22
|
+
velocity = (
|
|
23
|
+
events.withColumn("event_count_24h", F.count(F.lit(1)).over(last_24h))
|
|
24
|
+
.withColumn("amount_usd_24h", F.sum("amount_usd").over(last_24h))
|
|
25
|
+
)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
This stays inside Catalyst and avoids Python serialization. It is not sufficient
|
|
29
|
+
when same-time visibility uses secondary keys, when distinct state requires
|
|
30
|
+
reference-counted eviction, or when velocity and FIFO transitions must share
|
|
31
|
+
one exact sequence.
|
|
32
|
+
|
|
33
|
+
## Keep Coupled State In One Pure Sweep
|
|
34
|
+
|
|
35
|
+
When rolling windows and FIFO/path transitions share one exact sequence,
|
|
36
|
+
isolate that sequence as a pure algorithm before choosing how Spark executes
|
|
37
|
+
it:
|
|
38
|
+
|
|
39
|
+
```python
|
|
40
|
+
from datetime import timedelta
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def sweep_events(events):
|
|
44
|
+
ordered = sorted(
|
|
45
|
+
events,
|
|
46
|
+
key=lambda event: (
|
|
47
|
+
event.event_time,
|
|
48
|
+
event.transaction_id,
|
|
49
|
+
event.event_id,
|
|
50
|
+
),
|
|
51
|
+
)
|
|
52
|
+
windows = {
|
|
53
|
+
"1h": RollingWindow(timedelta(hours=1)),
|
|
54
|
+
"24h": RollingWindow(timedelta(hours=24)),
|
|
55
|
+
}
|
|
56
|
+
fifo = FifoState()
|
|
57
|
+
|
|
58
|
+
for event in ordered:
|
|
59
|
+
for window in windows.values():
|
|
60
|
+
window.append(event)
|
|
61
|
+
path = fifo.advance(event)
|
|
62
|
+
yield build_result(event, windows, path)
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
This is the semantic core, not a recommendation to wrap it in `F.udf`. Select
|
|
66
|
+
the Spark execution API using the decision table in
|
|
67
|
+
[Transformation Design](transformation-design.md): built-in windows first,
|
|
68
|
+
then bounded higher-order aggregation, grouped pandas only for bounded groups,
|
|
69
|
+
and a different engine/design for large sequential state.
|
|
70
|
+
|
|
71
|
+
Whichever bridge is chosen, emit one result per unique output identity and
|
|
72
|
+
measure it against the pair relation or repeated grouping it replaces.
|
|
73
|
+
|
|
74
|
+
## Make Window Boundaries Literal
|
|
75
|
+
|
|
76
|
+
For a current-inclusive horizon `H`, evict only events strictly older than the
|
|
77
|
+
cutoff:
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
from collections import Counter, deque
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
class RollingWindow:
|
|
84
|
+
def __init__(self, horizon):
|
|
85
|
+
self.horizon = horizon
|
|
86
|
+
self.events = deque()
|
|
87
|
+
self.count = 0
|
|
88
|
+
self.counterparties = Counter()
|
|
89
|
+
|
|
90
|
+
def append(self, event):
|
|
91
|
+
cutoff = event.event_time - self.horizon
|
|
92
|
+
while self.events and self.events[0].event_time < cutoff:
|
|
93
|
+
self.remove(self.events.popleft())
|
|
94
|
+
|
|
95
|
+
self.events.append(event) # current event is visible
|
|
96
|
+
self.count += 1
|
|
97
|
+
if event.counterparty_key is not None:
|
|
98
|
+
self.counterparties[event.counterparty_key] += 1
|
|
99
|
+
|
|
100
|
+
def remove(self, event):
|
|
101
|
+
self.count -= 1
|
|
102
|
+
key = event.counterparty_key
|
|
103
|
+
if key is not None:
|
|
104
|
+
self.counterparties[key] -= 1
|
|
105
|
+
if self.counterparties[key] == 0:
|
|
106
|
+
del self.counterparties[key]
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Using `<= cutoff` makes the lower boundary exclusive. Test the exact boundary
|
|
110
|
+
and one smallest supported time unit beyond it.
|
|
111
|
+
|
|
112
|
+
A reference-counted map is required for distinct counterparties because a set
|
|
113
|
+
cannot represent repeated counterparties during eviction.
|
|
114
|
+
|
|
115
|
+
## Define Same-Timestamp Visibility
|
|
116
|
+
|
|
117
|
+
Use the complete stable order, for example:
|
|
118
|
+
|
|
119
|
+
```text
|
|
120
|
+
(event_time, transaction_id, event_id)
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
If same-time events see earlier tie-ordered events, update and emit in that
|
|
124
|
+
order. If they should all see the same batch, emit the timestamp batch before
|
|
125
|
+
updating state. Never let incidental Spark input order define visibility.
|
|
126
|
+
|
|
127
|
+
## Use Numerically Stable Rolling State
|
|
128
|
+
|
|
129
|
+
Repeated floating addition and eviction can leave residue. Use compensated
|
|
130
|
+
addition/removal, or exact decimal state when the domain requires exact money:
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
class CompensatedSum:
|
|
134
|
+
def __init__(self):
|
|
135
|
+
self.total = 0.0
|
|
136
|
+
self.correction = 0.0
|
|
137
|
+
|
|
138
|
+
def add(self, value):
|
|
139
|
+
updated = self.total + value
|
|
140
|
+
if abs(self.total) >= abs(value):
|
|
141
|
+
self.correction += (self.total - updated) + value
|
|
142
|
+
else:
|
|
143
|
+
self.correction += (value - updated) + self.total
|
|
144
|
+
self.total = updated
|
|
145
|
+
|
|
146
|
+
def remove(self, value):
|
|
147
|
+
self.add(-value)
|
|
148
|
+
|
|
149
|
+
def value(self):
|
|
150
|
+
return self.total + self.correction
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Test values with different magnitudes, evict them, and assert that the previous
|
|
154
|
+
state is restored.
|
|
155
|
+
|
|
156
|
+
## Combine Related State
|
|
157
|
+
|
|
158
|
+
Within the same sweep:
|
|
159
|
+
|
|
160
|
+
1. Evict expired events from each horizon.
|
|
161
|
+
2. Add the current event according to visibility semantics.
|
|
162
|
+
3. Emit counts, sums, missing-value counts, directional metrics, and distinct
|
|
163
|
+
counterparties.
|
|
164
|
+
4. Advance FIFO/path state using the same stable order.
|
|
165
|
+
5. Emit one result keyed by output identity.
|
|
166
|
+
|
|
167
|
+
This removes the pair relation and avoids grouping and sorting the same history
|
|
168
|
+
for each feature family.
|
|
169
|
+
|
|
170
|
+
## Prove Parity And Plan Shape
|
|
171
|
+
|
|
172
|
+
Keep the old self-join only as a bounded test oracle. Cover exact 1h/24h
|
|
173
|
+
boundaries, one unit outside, same-time ties, nulls, both directions,
|
|
174
|
+
missing/present FX, repeated counterparties, path transitions, and floating
|
|
175
|
+
residue.
|
|
176
|
+
|
|
177
|
+
Compare full rows in both directions, add literal boundary assertions, and
|
|
178
|
+
assert the candidate physical plan has no customer/time interval join.
|
|
179
|
+
|
|
180
|
+
## Measure Residual Per-Key Memory
|
|
181
|
+
|
|
182
|
+
`groupBy + collect_list` changes quadratic distributed work into linear per-key
|
|
183
|
+
work, but one task still holds the entire largest key. Measure and report
|
|
184
|
+
`max_rows_per_key`, then production-canary the largest known shape. Use bounded
|
|
185
|
+
state or exact temporal bucketing if one key no longer fits.
|
|
186
|
+
|
|
187
|
+
## Avoid These False Fixes
|
|
188
|
+
|
|
189
|
+
- Do not rely on repartitioning to split one hot grouping key.
|
|
190
|
+
- Do not drop history, events, or feature formulas to gain speed.
|
|
191
|
+
- Do not present Arrow transport, broadcasting, or explicit persistence as the
|
|
192
|
+
performance fix; reduce the logical work instead.
|
|
193
|
+
- Do not report validate-only runtime as end-to-end write performance.
|
|
@@ -0,0 +1,223 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Summarize stage attempts and final SQL plans from a Spark JSON event log."""
|
|
3
|
+
|
|
4
|
+
from __future__ import annotations
|
|
5
|
+
|
|
6
|
+
import argparse
|
|
7
|
+
import json
|
|
8
|
+
import re
|
|
9
|
+
import sys
|
|
10
|
+
from collections.abc import Iterable
|
|
11
|
+
from typing import Any
|
|
12
|
+
|
|
13
|
+
METRICS = (
|
|
14
|
+
"input_bytes",
|
|
15
|
+
"input_records",
|
|
16
|
+
"output_bytes",
|
|
17
|
+
"output_records",
|
|
18
|
+
"shuffle_read_bytes",
|
|
19
|
+
"shuffle_read_records",
|
|
20
|
+
"shuffle_write_bytes",
|
|
21
|
+
"shuffle_write_records",
|
|
22
|
+
"memory_spill_bytes",
|
|
23
|
+
"disk_spill_bytes",
|
|
24
|
+
)
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def _patterns(values: Iterable[str]) -> dict[str, re.Pattern[str]]:
|
|
28
|
+
result: dict[str, re.Pattern[str]] = {}
|
|
29
|
+
for value in values:
|
|
30
|
+
name, separator, pattern = value.partition("=")
|
|
31
|
+
if not separator or not name or not pattern:
|
|
32
|
+
raise ValueError(
|
|
33
|
+
f"invalid --plan-pattern {value!r}; expected NAME=REGEX"
|
|
34
|
+
)
|
|
35
|
+
result[name] = re.compile(pattern, re.IGNORECASE)
|
|
36
|
+
return result
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def _add(stage: dict[str, Any], name: str, value: Any) -> None:
|
|
40
|
+
stage[name] = stage.get(name, 0) + (value or 0)
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def summarize(
|
|
44
|
+
lines: Iterable[str], plan_patterns: dict[str, re.Pattern[str]]
|
|
45
|
+
) -> dict[str, Any]:
|
|
46
|
+
app: dict[str, Any] = {}
|
|
47
|
+
stages: dict[tuple[int, int], dict[str, Any]] = {}
|
|
48
|
+
sql_execution_ids: set[Any] = set()
|
|
49
|
+
sql_plans: dict[Any, str] = {}
|
|
50
|
+
anonymous_sql_id = 0
|
|
51
|
+
|
|
52
|
+
for raw in lines:
|
|
53
|
+
try:
|
|
54
|
+
event = json.loads(raw)
|
|
55
|
+
except (json.JSONDecodeError, TypeError):
|
|
56
|
+
continue
|
|
57
|
+
|
|
58
|
+
kind = event.get("Event")
|
|
59
|
+
if kind == "SparkListenerApplicationStart":
|
|
60
|
+
app.update(
|
|
61
|
+
{
|
|
62
|
+
"app_id": event.get("App ID"),
|
|
63
|
+
"app_name": event.get("App Name"),
|
|
64
|
+
"start_time_ms": event.get("Timestamp"),
|
|
65
|
+
}
|
|
66
|
+
)
|
|
67
|
+
elif kind == "SparkListenerApplicationEnd":
|
|
68
|
+
app["end_time_ms"] = event.get("Timestamp")
|
|
69
|
+
elif kind in {
|
|
70
|
+
"org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
|
|
71
|
+
"org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate",
|
|
72
|
+
}:
|
|
73
|
+
execution_id = event.get("executionId")
|
|
74
|
+
if execution_id is None:
|
|
75
|
+
execution_id = f"anonymous-{anonymous_sql_id}"
|
|
76
|
+
anonymous_sql_id += 1
|
|
77
|
+
sql_execution_ids.add(execution_id)
|
|
78
|
+
sql_plans[execution_id] = event.get("physicalPlanDescription") or ""
|
|
79
|
+
elif kind == "SparkListenerStageCompleted":
|
|
80
|
+
info = event.get("Stage Info") or {}
|
|
81
|
+
stage_id = info.get("Stage ID")
|
|
82
|
+
if stage_id is None:
|
|
83
|
+
continue
|
|
84
|
+
attempt_id = info.get("Stage Attempt ID", 0)
|
|
85
|
+
key = (stage_id, attempt_id)
|
|
86
|
+
stage = stages.setdefault(
|
|
87
|
+
key,
|
|
88
|
+
{"stage_id": stage_id, "stage_attempt_id": attempt_id},
|
|
89
|
+
)
|
|
90
|
+
stage.update(
|
|
91
|
+
{
|
|
92
|
+
"name": info.get("Stage Name"),
|
|
93
|
+
"task_count": info.get("Number of Tasks", 0),
|
|
94
|
+
"submission_time_ms": info.get("Submission Time"),
|
|
95
|
+
"completion_time_ms": info.get("Completion Time"),
|
|
96
|
+
}
|
|
97
|
+
)
|
|
98
|
+
if (
|
|
99
|
+
stage["submission_time_ms"] is not None
|
|
100
|
+
and stage["completion_time_ms"] is not None
|
|
101
|
+
):
|
|
102
|
+
stage["wall_ms"] = (
|
|
103
|
+
stage["completion_time_ms"] - stage["submission_time_ms"]
|
|
104
|
+
)
|
|
105
|
+
elif kind == "SparkListenerTaskEnd":
|
|
106
|
+
stage_id = event.get("Stage ID")
|
|
107
|
+
if stage_id is None:
|
|
108
|
+
continue
|
|
109
|
+
attempt_id = event.get("Stage Attempt ID", 0)
|
|
110
|
+
key = (stage_id, attempt_id)
|
|
111
|
+
stage = stages.setdefault(
|
|
112
|
+
key,
|
|
113
|
+
{"stage_id": stage_id, "stage_attempt_id": attempt_id},
|
|
114
|
+
)
|
|
115
|
+
info = event.get("Task Info") or {}
|
|
116
|
+
metrics = event.get("Task Metrics") or {}
|
|
117
|
+
launch = info.get("Launch Time")
|
|
118
|
+
finish = info.get("Finish Time")
|
|
119
|
+
duration = finish - launch if launch is not None and finish is not None else 0
|
|
120
|
+
stage["max_task_ms"] = max(stage.get("max_task_ms", 0), duration)
|
|
121
|
+
_add(stage, "task_end_count", 1)
|
|
122
|
+
_add(stage, "memory_spill_bytes", metrics.get("Memory Bytes Spilled"))
|
|
123
|
+
_add(stage, "disk_spill_bytes", metrics.get("Disk Bytes Spilled"))
|
|
124
|
+
|
|
125
|
+
input_metrics = metrics.get("Input Metrics") or {}
|
|
126
|
+
output_metrics = metrics.get("Output Metrics") or {}
|
|
127
|
+
shuffle_read = metrics.get("Shuffle Read Metrics") or {}
|
|
128
|
+
shuffle_write = metrics.get("Shuffle Write Metrics") or {}
|
|
129
|
+
_add(stage, "input_bytes", input_metrics.get("Bytes Read"))
|
|
130
|
+
_add(stage, "input_records", input_metrics.get("Records Read"))
|
|
131
|
+
_add(stage, "output_bytes", output_metrics.get("Bytes Written"))
|
|
132
|
+
_add(stage, "output_records", output_metrics.get("Records Written"))
|
|
133
|
+
_add(
|
|
134
|
+
stage,
|
|
135
|
+
"shuffle_read_bytes",
|
|
136
|
+
(shuffle_read.get("Remote Bytes Read") or 0)
|
|
137
|
+
+ (shuffle_read.get("Local Bytes Read") or 0),
|
|
138
|
+
)
|
|
139
|
+
_add(
|
|
140
|
+
stage,
|
|
141
|
+
"shuffle_read_records",
|
|
142
|
+
shuffle_read.get("Total Records Read"),
|
|
143
|
+
)
|
|
144
|
+
_add(
|
|
145
|
+
stage,
|
|
146
|
+
"shuffle_write_bytes",
|
|
147
|
+
shuffle_write.get("Shuffle Bytes Written"),
|
|
148
|
+
)
|
|
149
|
+
_add(
|
|
150
|
+
stage,
|
|
151
|
+
"shuffle_write_records",
|
|
152
|
+
shuffle_write.get("Shuffle Records Written"),
|
|
153
|
+
)
|
|
154
|
+
|
|
155
|
+
completed = sorted(
|
|
156
|
+
(stage for stage in stages.values() if "wall_ms" in stage),
|
|
157
|
+
key=lambda stage: (stage["stage_id"], stage["stage_attempt_id"]),
|
|
158
|
+
)
|
|
159
|
+
totals = {
|
|
160
|
+
name: sum(stage.get(name, 0) for stage in completed) for name in METRICS
|
|
161
|
+
}
|
|
162
|
+
sql = {
|
|
163
|
+
"execution_count": len(sql_execution_ids),
|
|
164
|
+
"plan_pattern_counts": {
|
|
165
|
+
name: sum(len(pattern.findall(plan)) for plan in sql_plans.values())
|
|
166
|
+
for name, pattern in plan_patterns.items()
|
|
167
|
+
},
|
|
168
|
+
}
|
|
169
|
+
if app.get("start_time_ms") is not None and app.get("end_time_ms") is not None:
|
|
170
|
+
app["eventlog_wall_ms"] = app["end_time_ms"] - app["start_time_ms"]
|
|
171
|
+
|
|
172
|
+
def largest(metric: str) -> dict[str, Any]:
|
|
173
|
+
return max(completed, key=lambda stage: stage.get(metric, 0), default={})
|
|
174
|
+
|
|
175
|
+
return {
|
|
176
|
+
"application": app,
|
|
177
|
+
"sql": sql,
|
|
178
|
+
"completed_stage_count": len({stage["stage_id"] for stage in completed}),
|
|
179
|
+
"completed_stage_attempt_count": len(completed),
|
|
180
|
+
"totals": totals,
|
|
181
|
+
"dominant_stage": largest("wall_ms"),
|
|
182
|
+
"max_task_stage": largest("max_task_ms"),
|
|
183
|
+
"max_shuffle_write_records_stage": largest("shuffle_write_records"),
|
|
184
|
+
"stages": completed,
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
def main() -> int:
|
|
189
|
+
parser = argparse.ArgumentParser(description=__doc__)
|
|
190
|
+
parser.add_argument(
|
|
191
|
+
"--plan-pattern",
|
|
192
|
+
action="append",
|
|
193
|
+
default=[],
|
|
194
|
+
metavar="NAME=REGEX",
|
|
195
|
+
help="count matching physical-plan fragments; may be repeated",
|
|
196
|
+
)
|
|
197
|
+
parser.add_argument(
|
|
198
|
+
"--include-stages",
|
|
199
|
+
action="store_true",
|
|
200
|
+
help="include every completed stage instead of only aggregate/top stages",
|
|
201
|
+
)
|
|
202
|
+
args = parser.parse_args()
|
|
203
|
+
|
|
204
|
+
try:
|
|
205
|
+
patterns = _patterns(args.plan_pattern)
|
|
206
|
+
except (ValueError, re.error) as error:
|
|
207
|
+
parser.error(str(error))
|
|
208
|
+
|
|
209
|
+
result = summarize(sys.stdin, patterns)
|
|
210
|
+
if not result["application"].get("app_id"):
|
|
211
|
+
parser.error(
|
|
212
|
+
"no SparkListenerApplicationStart record found; verify the event-log "
|
|
213
|
+
"path and use shell pipefail so upstream read failures remain visible"
|
|
214
|
+
)
|
|
215
|
+
if not args.include_stages:
|
|
216
|
+
result.pop("stages", None)
|
|
217
|
+
json.dump(result, sys.stdout, indent=2, sort_keys=True)
|
|
218
|
+
sys.stdout.write("\n")
|
|
219
|
+
return 0
|
|
220
|
+
|
|
221
|
+
|
|
222
|
+
if __name__ == "__main__":
|
|
223
|
+
raise SystemExit(main())
|
|
@@ -27,6 +27,7 @@ page alone — it deliberately does not contain enough to act on.
|
|
|
27
27
|
|---|---|---|
|
|
28
28
|
| **Get an open or just-pushed PR merged** | [`references/watch-contract.md`](references/watch-contract.md#enqueue-before-you-watch) | Enqueue, *then* watch — the watch never enqueues, so a watch started on a PR nothing will enqueue can only end `STALLED`. A push to an already-queued PR dequeues it (`pull-request-updated`): the verdict is stale, re-validate before re-enqueuing. Several ready at once? Enqueue them together — next row. |
|
|
29
29
|
| **Deliver several ready PRs at once (a stack, an epic's units, a burst)** | [`references/watch-contract.md`](references/watch-contract.md#batch-delivery--enqueue-the-whole-ready-set-together) | Enqueue the whole ready set **together**, then watch: PRs riding one batch cost one CI run, and a red batch bisects to the culprit while the rest carry on. "If possible" is the point — a lone PR, or stragglers still waiting on review/CI/conflict, never hold the batch window open. |
|
|
30
|
+
| Stacked PR dequeued: "could not be retargeted onto … `main`" | [`references/traps.md`](references/traps.md) | Not a `batch_size` bug — enable GitHub `delete_branch_on_merge` (auto-delete head branches). Then `mergify stack push --force-rebase` for the stuck tip. |
|
|
30
31
|
| Author or change `.mergify.yml` | [`references/configuration.md`](references/configuration.md) | [Rule 1](#rule-1--fetch-the-docs-before-writing-config): never write config from memory; finish with `validate` **and** `simulate`. |
|
|
31
32
|
| **Watch a PR through the queue** | [`references/watch-contract.md`](references/watch-contract.md) | **[Rule 2](#rule-2--never-hand-roll-a-watcher): run the bundled watcher, never hand-roll one.** A watch observes a *progression*, not one state. |
|
|
32
33
|
| Check status — "where is my PR?", "why is it stuck?" | [`references/diagnosis.md`](references/diagnosis.md#querying-pr-and-queue-status) | `mergify queue show <PR> -v` names the single blocking condition; nothing else in the toolchain does. `gh pr checks` is CI truth, never queue truth — and a one-off read of the `queued`/`dequeued` labels is neither. |
|
|
@@ -161,23 +161,29 @@ prefer on-demand unless you have a reason.
|
|
|
161
161
|
"fine." `--json` for machine output; exits non-zero on any warn (`--allow-warn` to
|
|
162
162
|
downgrade). Runs on the committed `.mergify.yml`, so it fits a pre-merge check.
|
|
163
163
|
7. Confirm Mergify's app id: `gh api /apps/mergify --jq .id`.
|
|
164
|
-
8.
|
|
164
|
+
8. **Enable GitHub "Automatically delete head branches"** before relying on
|
|
165
|
+
stacked PRs in the merge queue:
|
|
166
|
+
`gh api repos/<owner>/<repo> -X PATCH -f delete_branch_on_merge=true`.
|
|
167
|
+
With this off, survivors of a stack dequeue with "could not be retargeted
|
|
168
|
+
onto the queue base branch" after a parent merges — see
|
|
169
|
+
[`traps.md`](traps.md). Confirm: `gh api repos/<owner>/<repo> --jq .delete_branch_on_merge`.
|
|
170
|
+
9. `gh ruleset list` — if a ruleset already covers this branch, fold these rules
|
|
165
171
|
into it (**PUT**) rather than stacking a second one. Overlapping rulesets are
|
|
166
172
|
additive and the most restrictive wins, which is a confusing way to be blocked.
|
|
167
|
-
|
|
173
|
+
10. Inventory anything that pushes to the protected branch — the `update` rule
|
|
168
174
|
will block it. Add each as a bypass actor (GitHub Actions = `15368`):
|
|
169
175
|
`grep -rnE 'git push|create-pull-request|github-script' .github/workflows/`
|
|
170
|
-
|
|
176
|
+
11. Apply `assets/templates/ruleset.json`. Dry-run with
|
|
171
177
|
`"enforcement": "evaluate"` first if you want to see what it would block —
|
|
172
178
|
read the results from `rule-suites`.
|
|
173
|
-
|
|
179
|
+
12. Prove the allowed path **before** the blocked one: enqueue a PR with
|
|
174
180
|
`@mergifyio queue`, confirm the 👍, and watch `mergify queue show <PR> -v`
|
|
175
181
|
through to merge. A ruleset strict enough to block humans can also block
|
|
176
182
|
Mergify if a bypass id is wrong, and you would rather learn that while you
|
|
177
183
|
can still merge the fix.
|
|
178
|
-
|
|
184
|
+
13. Now prove the blocked path: open a throwaway PR and try to merge it without
|
|
179
185
|
enqueuing, as a non-bypass user. GitHub must refuse, naming the ruleset.
|
|
180
|
-
|
|
186
|
+
14. Decide the escape hatches: keep `priority_rules` (jump the line), and either
|
|
181
187
|
keep or delete the `emergency-merge` rule (skips CI, cannot be limited to
|
|
182
188
|
admins). Keep the `OrganizationAdmin` bypass either way — it is the only way
|
|
183
189
|
back in if Mergify has an outage.
|
|
@@ -76,3 +76,31 @@ The default (`queue`) injects ruleset requirements into entry conditions too, an
|
|
|
76
76
|
a Mergify check that deliberately fails pre-queue then blocks the enqueue that
|
|
77
77
|
would make it pass.
|
|
78
78
|
|
|
79
|
+
**Stacked PRs dequeue with "could not be retargeted onto the queue base
|
|
80
|
+
branch" when head branches survive merge.** This is not a `.mergify.yml`
|
|
81
|
+
`batch_size` / `merge-batch` bug. Mergify *can* land a stack when members are
|
|
82
|
+
queued together (stack-aware base + batching), but after a lower member merges
|
|
83
|
+
GitHub must restack the survivors onto `main`. That retarget happens when the
|
|
84
|
+
merged PR's **head branch is deleted**. If the repository has
|
|
85
|
+
`delete_branch_on_merge: false`, parent stack refs stay on the remote, GitHub
|
|
86
|
+
never moves the next PR's base from `parent-head` → `main`, and Mergify dequeues
|
|
87
|
+
with `base-ref-alignment-timeout` / "stacked on another whose head branch still
|
|
88
|
+
exists after merging" — even while the tip was already in the queue and draft
|
|
89
|
+
batch CI was green. Hit on skyee-ai-risk ADR 0044 stack (#2108 after #2107;
|
|
90
|
+
#2110 after #2108/#2109). Fix the repo setting, not the queue rule:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
gh api repos/<owner>/<repo> -X PATCH -f delete_branch_on_merge=true \
|
|
94
|
+
--jq .delete_branch_on_merge
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Verify with `gh api repos/<owner>/<repo> --jq .delete_branch_on_merge` (must be
|
|
98
|
+
`true`). If a tip is already stuck, `mergify stack push --force-rebase` (or a
|
|
99
|
+
manual base edit onto `main`) unblocks that incident; auto-delete prevents the
|
|
100
|
+
next one. Keep the full `Depends-On: #N` chain on every stack member — without
|
|
101
|
+
it Mergify may not treat the chain as a stack and keeps literal bases, which
|
|
102
|
+
collides with `queue_conditions: base = main`. Do not merge unrelated `main` PRs
|
|
103
|
+
through the middle of an open stack unless you expect another rebase cycle.
|
|
104
|
+
For GitHub-native stacks, Mergify's ruleset bypass should be **`exempt`** (not
|
|
105
|
+
only `always`); see [Stacked Pull Requests](https://docs.mergify.com/merge-queue/stacks/).
|
|
106
|
+
|
|
@@ -360,7 +360,7 @@
|
|
|
360
360
|
"id": "first-party",
|
|
361
361
|
"type": "first-party",
|
|
362
362
|
"package": "@1aboveio/skills",
|
|
363
|
-
"version": "0.
|
|
363
|
+
"version": "0.12.0"
|
|
364
364
|
},
|
|
365
365
|
"contentDigest": "eff6c7b5931bce5b2265a619bccddd371a89df6ce7bc2edc74f11d00060a71dd",
|
|
366
366
|
"digestExcludes": []
|
|
@@ -373,7 +373,7 @@
|
|
|
373
373
|
"id": "first-party",
|
|
374
374
|
"type": "first-party",
|
|
375
375
|
"package": "@1aboveio/skills",
|
|
376
|
-
"version": "0.
|
|
376
|
+
"version": "0.12.0"
|
|
377
377
|
},
|
|
378
378
|
"contentDigest": "08230dc57a53d6526692b50138038abbd80abce1a067e01c52ad6acc182caf7e",
|
|
379
379
|
"digestExcludes": []
|
|
@@ -386,7 +386,7 @@
|
|
|
386
386
|
"id": "first-party",
|
|
387
387
|
"type": "first-party",
|
|
388
388
|
"package": "@1aboveio/skills",
|
|
389
|
-
"version": "0.
|
|
389
|
+
"version": "0.12.0"
|
|
390
390
|
},
|
|
391
391
|
"contentDigest": "9eea7bfba348ddaea1b934c9a7fabdd9b101df8a146e1c93e947da4d335f1d98",
|
|
392
392
|
"digestExcludes": []
|
|
@@ -399,9 +399,9 @@
|
|
|
399
399
|
"id": "first-party",
|
|
400
400
|
"type": "first-party",
|
|
401
401
|
"package": "@1aboveio/skills",
|
|
402
|
-
"version": "0.
|
|
402
|
+
"version": "0.12.0"
|
|
403
403
|
},
|
|
404
|
-
"contentDigest": "
|
|
404
|
+
"contentDigest": "58b0556228a9271cf9e33727a05fc08a4fdfd29512ec0936b169f0a55d60e6ef",
|
|
405
405
|
"digestExcludes": []
|
|
406
406
|
},
|
|
407
407
|
{
|
|
@@ -412,9 +412,9 @@
|
|
|
412
412
|
"id": "first-party",
|
|
413
413
|
"type": "first-party",
|
|
414
414
|
"package": "@1aboveio/skills",
|
|
415
|
-
"version": "0.
|
|
415
|
+
"version": "0.12.0"
|
|
416
416
|
},
|
|
417
|
-
"contentDigest": "
|
|
417
|
+
"contentDigest": "5351b88d73d56c27b25ff09c72ab6029fe0805bf7c459648424747108433c175",
|
|
418
418
|
"digestExcludes": []
|
|
419
419
|
},
|
|
420
420
|
{
|
|
@@ -425,9 +425,9 @@
|
|
|
425
425
|
"id": "first-party",
|
|
426
426
|
"type": "first-party",
|
|
427
427
|
"package": "@1aboveio/skills",
|
|
428
|
-
"version": "0.
|
|
428
|
+
"version": "0.12.0"
|
|
429
429
|
},
|
|
430
|
-
"contentDigest": "
|
|
430
|
+
"contentDigest": "b5af4dccf703362d4f41cac4fdff48305f652a00338d85975a2e5c35ec6bc61c",
|
|
431
431
|
"digestExcludes": []
|
|
432
432
|
},
|
|
433
433
|
{
|
|
@@ -438,9 +438,9 @@
|
|
|
438
438
|
"id": "first-party",
|
|
439
439
|
"type": "first-party",
|
|
440
440
|
"package": "@1aboveio/skills",
|
|
441
|
-
"version": "0.
|
|
441
|
+
"version": "0.12.0"
|
|
442
442
|
},
|
|
443
|
-
"contentDigest": "
|
|
443
|
+
"contentDigest": "f048fd00c69f2dc666fc7a3096cfeee6dfba73933bfb69602f3f0e26159798cc",
|
|
444
444
|
"digestExcludes": []
|
|
445
445
|
},
|
|
446
446
|
{
|
|
@@ -451,7 +451,7 @@
|
|
|
451
451
|
"id": "first-party",
|
|
452
452
|
"type": "first-party",
|
|
453
453
|
"package": "@1aboveio/skills",
|
|
454
|
-
"version": "0.
|
|
454
|
+
"version": "0.12.0"
|
|
455
455
|
},
|
|
456
456
|
"contentDigest": "90a7c4e1ad6da1e632ea2c5259e4967ffebb84353adccbca8dfc5d6bc50b60c1",
|
|
457
457
|
"digestExcludes": []
|
|
@@ -464,15 +464,15 @@
|
|
|
464
464
|
"id": "first-party",
|
|
465
465
|
"type": "first-party",
|
|
466
466
|
"package": "@1aboveio/skills",
|
|
467
|
-
"version": "0.
|
|
467
|
+
"version": "0.12.0"
|
|
468
468
|
},
|
|
469
|
-
"contentDigest": "
|
|
469
|
+
"contentDigest": "e9bfbdfcbca59efce2c530a8a68f66466b0333dcb046760aab4eaee53ec7e559",
|
|
470
470
|
"digestExcludes": [
|
|
471
471
|
"coherence/workflow.json"
|
|
472
472
|
]
|
|
473
473
|
}
|
|
474
474
|
],
|
|
475
|
-
"releaseIdentity": "
|
|
475
|
+
"releaseIdentity": "559141ff07a7aa2b7e20884c7a3a636c4b23207993d25b7ba00b8a872def0517",
|
|
476
476
|
"lifecycleAuthority": "native-skills-cli",
|
|
477
477
|
"repairRecipe": {
|
|
478
478
|
"id": "engineering-workflow-dependency-first",
|
|
@@ -524,7 +524,7 @@
|
|
|
524
524
|
"sourceId": "first-party",
|
|
525
525
|
"sourceType": "first-party",
|
|
526
526
|
"package": "@1aboveio/skills",
|
|
527
|
-
"version": "0.
|
|
527
|
+
"version": "0.12.0",
|
|
528
528
|
"installPath": null,
|
|
529
529
|
"members": [
|
|
530
530
|
"harness-runtime",
|
|
@@ -540,7 +540,7 @@
|
|
|
540
540
|
"commands": [
|
|
541
541
|
{
|
|
542
542
|
"transport": "npm",
|
|
543
|
-
"command": "npx @1aboveio/skills@0.
|
|
543
|
+
"command": "npx @1aboveio/skills@0.12.0 install --group engineering-workflow"
|
|
544
544
|
}
|
|
545
545
|
],
|
|
546
546
|
"onFailure": {
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: implement-and-pr
|
|
3
|
-
description: "
|
|
3
|
+
description: "Slash-command only (/implement-and-pr). Produce step: implement an approved issue/PRD/plan with TDD + tests + evidence and open a PR with a Review Contract. Do not auto-select — only on explicit user/orchestrator invoke. NOT the full issue→merge-ready loop (resolve-issues / rush-issues)."
|
|
4
|
+
disable-model-invocation: true
|
|
4
5
|
dependencies:
|
|
5
6
|
- ensure-coverage
|
|
6
7
|
- tdd
|
|
@@ -9,13 +10,13 @@ dependencies:
|
|
|
9
10
|
|
|
10
11
|
# Implement and PR
|
|
11
12
|
|
|
12
|
-
|
|
13
|
+
Turn an approved source work item into a tested pull request with clear review evidence.
|
|
13
14
|
|
|
14
15
|
## Related Skills
|
|
15
16
|
|
|
16
17
|
This is the *produce* step in the PR chain; it consumes the skills below by name rather than re-deriving their doctrine:
|
|
17
18
|
|
|
18
|
-
- `resolve-issues` —
|
|
19
|
+
- `resolve-issues` / `rush-issues` — orchestrators that call this as the implement step; use standalone only for a one-off "implement and open a PR, no review loop" request.
|
|
19
20
|
- `ensure-coverage` — coverage doctrine you implement against: its create-mode criteria are your input (approved **first/upstream**), its `coverage-ledger.mjs` gate is your self-check; source of truth for required levels, the Surface Baseline Contract, and mock/depth rules.
|
|
20
21
|
- `tdd` — the red-green-refactor loop used in TDD Mode.
|
|
21
22
|
- `e2e-test` — Playwright structure + route coverage for browser/journey tests; its render-health tier + presentation sweep is step 6's per-surface render check (distinct from the curated golden-path `smoke` gate step 6 runs when a critical journey is touched).
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
interface:
|
|
2
|
+
display_name: "Implement and PR"
|
|
3
|
+
short_description: "Slash/explicit-only produce step: implement + open a PR with a Review Contract"
|
|
4
|
+
|
|
5
|
+
policy:
|
|
6
|
+
# Codex counterpart to SKILL.md disable-model-invocation: true
|
|
7
|
+
# (Claude Code / Pi). Keeps $implement-and-pr / explicit invoke; blocks
|
|
8
|
+
# description-based auto-selection.
|
|
9
|
+
allow_implicit_invocation: false
|