@1aboveio/skills 0.10.1 → 0.11.0

package/skills/backend/pyspark/references/velocity-feature-calculation.md

@@ -0,0 +1,193 @@
+# Velocity Feature Calculation Best Practices
+
+Use these practices for rolling counts and sums, distinct counterparties,
+temporal velocity features, and FIFO/path calculations over event histories.
+The snippets illustrate the design; adapt schemas and semantics to the job's
+required behavior.
+
+## Use Built-Ins When Velocity Metrics Are Independent
+
+For ordinary rolling counts and sums where same-time peers share one frame,
+prefer a Spark range window:
+
+```python
+from pyspark.sql import Window, functions as F
+
+last_24h = (
+    Window.partitionBy("entity_id")
+    .orderBy(F.col("event_time").cast("long"))
+    .rangeBetween(-24 * 60 * 60, 0)
+)
+
+velocity = (
+    events.withColumn("event_count_24h", F.count(F.lit(1)).over(last_24h))
+    .withColumn("amount_usd_24h", F.sum("amount_usd").over(last_24h))
+)
+```
+
+This stays inside Catalyst and avoids Python serialization. It is not sufficient
+when same-time visibility uses secondary keys, when distinct state requires
+reference-counted eviction, or when velocity and FIFO transitions must share
+one exact sequence.
+
+## Keep Coupled State In One Pure Sweep
+
+When rolling windows and FIFO/path transitions share one exact sequence,
+isolate that sequence as a pure algorithm before choosing how Spark executes
+it:
+
+```python
+from datetime import timedelta
+
+
+def sweep_events(events):
+    ordered = sorted(
+        events,
+        key=lambda event: (
+            event.event_time,
+            event.transaction_id,
+            event.event_id,
+        ),
+    )
+    windows = {
+        "1h": RollingWindow(timedelta(hours=1)),
+        "24h": RollingWindow(timedelta(hours=24)),
+    }
+    fifo = FifoState()
+
+    for event in ordered:
+        for window in windows.values():
+            window.append(event)
+        path = fifo.advance(event)
+        yield build_result(event, windows, path)
+```
+
+This is the semantic core, not a recommendation to wrap it in `F.udf`. Select
+the Spark execution API using the decision table in
+[Transformation Design](transformation-design.md): built-in windows first,
+then bounded higher-order aggregation, grouped pandas only for bounded groups,
+and a different engine/design for large sequential state.
+
+Whichever bridge is chosen, emit one result per unique output identity and
+measure it against the pair relation or repeated grouping it replaces.
+
+## Make Window Boundaries Literal
+
+For a current-inclusive horizon `H`, evict only events strictly older than the
+cutoff:
+
+```python
+from collections import Counter, deque
+
+
+class RollingWindow:
+    def __init__(self, horizon):
+        self.horizon = horizon
+        self.events = deque()
+        self.count = 0
+        self.counterparties = Counter()
+
+    def append(self, event):
+        cutoff = event.event_time - self.horizon
+        while self.events and self.events[0].event_time < cutoff:
+            self.remove(self.events.popleft())
+
+        self.events.append(event)  # current event is visible
+        self.count += 1
+        if event.counterparty_key is not None:
+            self.counterparties[event.counterparty_key] += 1
+
+    def remove(self, event):
+        self.count -= 1
+        key = event.counterparty_key
+        if key is not None:
+            self.counterparties[key] -= 1
+            if self.counterparties[key] == 0:
+                del self.counterparties[key]
+```
+
+Using `<= cutoff` makes the lower boundary exclusive. Test the exact boundary
+and one smallest supported time unit beyond it.
+
+A reference-counted map is required for distinct counterparties because a set
+cannot represent repeated counterparties during eviction.
+
+## Define Same-Timestamp Visibility
+
+Use the complete stable order, for example:
+
+```text
+(event_time, transaction_id, event_id)
+```
+
+If same-time events see earlier tie-ordered events, update and emit in that
+order. If they should all see the same batch, emit the timestamp batch before
+updating state. Never let incidental Spark input order define visibility.
+
+## Use Numerically Stable Rolling State
+
+Repeated floating addition and eviction can leave residue. Use compensated
+addition/removal, or exact decimal state when the domain requires exact money:
+
+```python
+class CompensatedSum:
+    def __init__(self):
+        self.total = 0.0
+        self.correction = 0.0
+
+    def add(self, value):
+        updated = self.total + value
+        if abs(self.total) >= abs(value):
+            self.correction += (self.total - updated) + value
+        else:
+            self.correction += (value - updated) + self.total
+        self.total = updated
+
+    def remove(self, value):
+        self.add(-value)
+
+    def value(self):
+        return self.total + self.correction
+```
+
+Test values with different magnitudes, evict them, and assert that the previous
+state is restored.
+
+## Combine Related State
+
+Within the same sweep:
+
+1. Evict expired events from each horizon.
+2. Add the current event according to visibility semantics.
+3. Emit counts, sums, missing-value counts, directional metrics, and distinct
+   counterparties.
+4. Advance FIFO/path state using the same stable order.
+5. Emit one result keyed by output identity.
+
+This removes the pair relation and avoids grouping and sorting the same history
+for each feature family.
+
+## Prove Parity And Plan Shape
+
+Keep the old self-join only as a bounded test oracle. Cover exact 1h/24h
+boundaries, one unit outside, same-time ties, nulls, both directions,
+missing/present FX, repeated counterparties, path transitions, and floating
+residue.
+
+Compare full rows in both directions, add literal boundary assertions, and
+assert the candidate physical plan has no customer/time interval join.
+
+## Measure Residual Per-Key Memory
+
+`groupBy + collect_list` changes quadratic distributed work into linear per-key
+work, but one task still holds the entire largest key. Measure and report
+`max_rows_per_key`, then production-canary the largest known shape. Use bounded
+state or exact temporal bucketing if one key no longer fits.
+
+## Avoid These False Fixes
+
+- Do not rely on repartitioning to split one hot grouping key.
+- Do not drop history, events, or feature formulas to gain speed.
+- Do not present Arrow transport, broadcasting, or explicit persistence as the
+  performance fix; reduce the logical work instead.
+- Do not report validate-only runtime as end-to-end write performance.

package/skills/backend/pyspark/scripts/spark_eventlog_summary.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""Summarize stage attempts and final SQL plans from a Spark JSON event log."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from collections.abc import Iterable
+from typing import Any
+
+METRICS = (
+    "input_bytes",
+    "input_records",
+    "output_bytes",
+    "output_records",
+    "shuffle_read_bytes",
+    "shuffle_read_records",
+    "shuffle_write_bytes",
+    "shuffle_write_records",
+    "memory_spill_bytes",
+    "disk_spill_bytes",
+)
+
+
+def _patterns(values: Iterable[str]) -> dict[str, re.Pattern[str]]:
+    result: dict[str, re.Pattern[str]] = {}
+    for value in values:
+        name, separator, pattern = value.partition("=")
+        if not separator or not name or not pattern:
+            raise ValueError(
+                f"invalid --plan-pattern {value!r}; expected NAME=REGEX"
+            )
+        result[name] = re.compile(pattern, re.IGNORECASE)
+    return result
+
+
+def _add(stage: dict[str, Any], name: str, value: Any) -> None:
+    stage[name] = stage.get(name, 0) + (value or 0)
+
+
+def summarize(
+    lines: Iterable[str], plan_patterns: dict[str, re.Pattern[str]]
+) -> dict[str, Any]:
+    app: dict[str, Any] = {}
+    stages: dict[tuple[int, int], dict[str, Any]] = {}
+    sql_execution_ids: set[Any] = set()
+    sql_plans: dict[Any, str] = {}
+    anonymous_sql_id = 0
+
+    for raw in lines:
+        try:
+            event = json.loads(raw)
+        except (json.JSONDecodeError, TypeError):
+            continue
+
+        kind = event.get("Event")
+        if kind == "SparkListenerApplicationStart":
+            app.update(
+                {
+                    "app_id": event.get("App ID"),
+                    "app_name": event.get("App Name"),
+                    "start_time_ms": event.get("Timestamp"),
+                }
+            )
+        elif kind == "SparkListenerApplicationEnd":
+            app["end_time_ms"] = event.get("Timestamp")
+        elif kind in {
+            "org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
+            "org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate",
+        }:
+            execution_id = event.get("executionId")
+            if execution_id is None:
+                execution_id = f"anonymous-{anonymous_sql_id}"
+                anonymous_sql_id += 1
+            sql_execution_ids.add(execution_id)
+            sql_plans[execution_id] = event.get("physicalPlanDescription") or ""
+        elif kind == "SparkListenerStageCompleted":
+            info = event.get("Stage Info") or {}
+            stage_id = info.get("Stage ID")
+            if stage_id is None:
+                continue
+            attempt_id = info.get("Stage Attempt ID", 0)
+            key = (stage_id, attempt_id)
+            stage = stages.setdefault(
+                key,
+                {"stage_id": stage_id, "stage_attempt_id": attempt_id},
+            )
+            stage.update(
+                {
+                    "name": info.get("Stage Name"),
+                    "task_count": info.get("Number of Tasks", 0),
+                    "submission_time_ms": info.get("Submission Time"),
+                    "completion_time_ms": info.get("Completion Time"),
+                }
+            )
+            if (
+                stage["submission_time_ms"] is not None
+                and stage["completion_time_ms"] is not None
+            ):
+                stage["wall_ms"] = (
+                    stage["completion_time_ms"] - stage["submission_time_ms"]
+                )
+        elif kind == "SparkListenerTaskEnd":
+            stage_id = event.get("Stage ID")
+            if stage_id is None:
+                continue
+            attempt_id = event.get("Stage Attempt ID", 0)
+            key = (stage_id, attempt_id)
+            stage = stages.setdefault(
+                key,
+                {"stage_id": stage_id, "stage_attempt_id": attempt_id},
+            )
+            info = event.get("Task Info") or {}
+            metrics = event.get("Task Metrics") or {}
+            launch = info.get("Launch Time")
+            finish = info.get("Finish Time")
+            duration = finish - launch if launch is not None and finish is not None else 0
+            stage["max_task_ms"] = max(stage.get("max_task_ms", 0), duration)
+            _add(stage, "task_end_count", 1)
+            _add(stage, "memory_spill_bytes", metrics.get("Memory Bytes Spilled"))
+            _add(stage, "disk_spill_bytes", metrics.get("Disk Bytes Spilled"))
+
+            input_metrics = metrics.get("Input Metrics") or {}
+            output_metrics = metrics.get("Output Metrics") or {}
+            shuffle_read = metrics.get("Shuffle Read Metrics") or {}
+            shuffle_write = metrics.get("Shuffle Write Metrics") or {}
+            _add(stage, "input_bytes", input_metrics.get("Bytes Read"))
+            _add(stage, "input_records", input_metrics.get("Records Read"))
+            _add(stage, "output_bytes", output_metrics.get("Bytes Written"))
+            _add(stage, "output_records", output_metrics.get("Records Written"))
+            _add(
+                stage,
+                "shuffle_read_bytes",
+                (shuffle_read.get("Remote Bytes Read") or 0)
+                + (shuffle_read.get("Local Bytes Read") or 0),
+            )
+            _add(
+                stage,
+                "shuffle_read_records",
+                shuffle_read.get("Total Records Read"),
+            )
+            _add(
+                stage,
+                "shuffle_write_bytes",
+                shuffle_write.get("Shuffle Bytes Written"),
+            )
+            _add(
+                stage,
+                "shuffle_write_records",
+                shuffle_write.get("Shuffle Records Written"),
+            )
+
+    completed = sorted(
+        (stage for stage in stages.values() if "wall_ms" in stage),
+        key=lambda stage: (stage["stage_id"], stage["stage_attempt_id"]),
+    )
+    totals = {
+        name: sum(stage.get(name, 0) for stage in completed) for name in METRICS
+    }
+    sql = {
+        "execution_count": len(sql_execution_ids),
+        "plan_pattern_counts": {
+            name: sum(len(pattern.findall(plan)) for plan in sql_plans.values())
+            for name, pattern in plan_patterns.items()
+        },
+    }
+    if app.get("start_time_ms") is not None and app.get("end_time_ms") is not None:
+        app["eventlog_wall_ms"] = app["end_time_ms"] - app["start_time_ms"]
+
+    def largest(metric: str) -> dict[str, Any]:
+        return max(completed, key=lambda stage: stage.get(metric, 0), default={})
+
+    return {
+        "application": app,
+        "sql": sql,
+        "completed_stage_count": len({stage["stage_id"] for stage in completed}),
+        "completed_stage_attempt_count": len(completed),
+        "totals": totals,
+        "dominant_stage": largest("wall_ms"),
+        "max_task_stage": largest("max_task_ms"),
+        "max_shuffle_write_records_stage": largest("shuffle_write_records"),
+        "stages": completed,
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--plan-pattern",
+        action="append",
+        default=[],
+        metavar="NAME=REGEX",
+        help="count matching physical-plan fragments; may be repeated",
+    )
+    parser.add_argument(
+        "--include-stages",
+        action="store_true",
+        help="include every completed stage instead of only aggregate/top stages",
+    )
+    args = parser.parse_args()
+
+    try:
+        patterns = _patterns(args.plan_pattern)
+    except (ValueError, re.error) as error:
+        parser.error(str(error))
+
+    result = summarize(sys.stdin, patterns)
+    if not result["application"].get("app_id"):
+        parser.error(
+            "no SparkListenerApplicationStart record found; verify the event-log "
+            "path and use shell pipefail so upstream read failures remain visible"
+        )
+    if not args.include_stages:
+        result.pop("stages", None)
+    json.dump(result, sys.stdout, indent=2, sort_keys=True)
+    sys.stdout.write("\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/skills/engineering/engineering-runtime/coherence/workflow.json

@@ -360,7 +360,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "eff6c7b5931bce5b2265a619bccddd371a89df6ce7bc2edc74f11d00060a71dd",
       "digestExcludes": []
@@ -373,7 +373,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "08230dc57a53d6526692b50138038abbd80abce1a067e01c52ad6acc182caf7e",
       "digestExcludes": []
@@ -386,7 +386,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "9eea7bfba348ddaea1b934c9a7fabdd9b101df8a146e1c93e947da4d335f1d98",
       "digestExcludes": []
@@ -399,7 +399,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "8e379be5d9412d5784c1286a55fc612af4b6479c2247bc1d53f4fb0183fecd98",
       "digestExcludes": []
@@ -412,9 +412,9 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
-      "contentDigest": "8c56881270957ad781e21881ab872a1b7b1b2cfad325cd04738b2a8d1cafa89b",
+      "contentDigest": "430b29b6b95e5cba5b06d365254ae5abd680684b7f053b1ca1694319c1b7f9fb",
       "digestExcludes": []
     },
     {
@@ -425,7 +425,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "b9b918b357b78f3c5d8f5110b66418b3b62a94825dc7f1905dcc5691a9476e86",
       "digestExcludes": []
@@ -438,7 +438,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "9cc85e87f5bfbb5570770692f14a95959aac6ade2d5364e827fc39b08871d346",
       "digestExcludes": []
@@ -451,7 +451,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
       "contentDigest": "90a7c4e1ad6da1e632ea2c5259e4967ffebb84353adccbca8dfc5d6bc50b60c1",
       "digestExcludes": []
@@ -464,15 +464,15 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       },
-      "contentDigest": "785b998edafac2458b8d3cab6ed8af78220c61518e58be1afa6cf75fd8e895e6",
+      "contentDigest": "0b24d6220d6d2577f2c838ef2c4fa165384594cf4c626960d2e707fd7cd0a228",
       "digestExcludes": [
         "coherence/workflow.json"
       ]
     }
   ],
-  "releaseIdentity": "1a553e637bd6cc46d59819d07b5ae19d7a3adc2d263f613d7b997f4e3fd1737b",
+  "releaseIdentity": "2e943acd0d8493f82146b6bd7e2dc769a4f876f2587155913f314b0a5efc120c",
   "lifecycleAuthority": "native-skills-cli",
   "repairRecipe": {
     "id": "engineering-workflow-dependency-first",
@@ -524,7 +524,7 @@
         "sourceId": "first-party",
         "sourceType": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1",
+        "version": "0.11.0",
         "installPath": null,
         "members": [
           "harness-runtime",
@@ -540,7 +540,7 @@
         "commands": [
           {
             "transport": "npm",
-            "command": "npx @1aboveio/skills@0.10.1 install --group engineering-workflow"
+            "command": "npx @1aboveio/skills@0.11.0 install --group engineering-workflow"
           }
         ],
         "onFailure": {

package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs

@@ -93,7 +93,7 @@ export const WORKFLOW_TRUSTED_SOURCES = deepFreeze({
     id: 'first-party',
     type: 'first-party',
     package: '@1aboveio/skills',
-    version: '0.10.1',
+    version: '0.11.0',
   },
   matt: {
     id: 'matt-pocock',

package/skills/engineering/resolve-issues/generated/workflow-repair-policy.json

@@ -236,7 +236,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -247,7 +247,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -258,7 +258,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -269,7 +269,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -280,7 +280,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -291,7 +291,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -302,7 +302,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -313,7 +313,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     },
     {
@@ -324,7 +324,7 @@
         "id": "first-party",
         "type": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1"
+        "version": "0.11.0"
       }
     }
   ],
@@ -419,7 +419,7 @@
         "sourceId": "first-party",
         "sourceType": "first-party",
         "package": "@1aboveio/skills",
-        "version": "0.10.1",
+        "version": "0.11.0",
         "installPath": null,
         "members": [
           "harness-runtime",
@@ -435,7 +435,7 @@
         "commands": [
           {
             "transport": "npm",
-            "command": "npx @1aboveio/skills@0.10.1 install --group engineering-workflow"
+            "command": "npx @1aboveio/skills@0.11.0 install --group engineering-workflow"
           }
         ],
         "onFailure": {

package/skills/engineering/resolve-issues/scripts/run-state.mjs

@@ -3412,7 +3412,7 @@ export const WORKFLOW_PREFLIGHT_COMMANDS = Object.freeze([
 
 const WORKFLOW_VERIFIER_URL = new URL('../../engineering-runtime/scripts/workflow-coherence.mjs', import.meta.url)
 const WORKFLOW_FALLBACK_POLICY_URL = new URL('../generated/workflow-repair-policy.json', import.meta.url)
-export const WORKFLOW_TRUSTED_FALLBACK_POLICY_SHA256 = '6135740bbbe780364e78b49f2a2ef6ccc3736b9f8b47d04dcc53d927a6fb7153'
+export const WORKFLOW_TRUSTED_FALLBACK_POLICY_SHA256 = 'b8b3b30d3509ec8c74c4d42b88e53a5958fc17e0066fc18b07fa4b9cd7eb343a'
 const WORKFLOW_REPAIR_RECIPE_REFERENCE = Object.freeze({
   id: 'engineering-workflow-dependency-first',
   generatedFrom: 'skills/distribution/generated/recipes.json',