@1aboveio/skills 0.0.0-stage → 1.0.0

package/skills/app-dev/pyspark/references/velocity-feature-calculation.md

@@ -0,0 +1,193 @@
+# Velocity Feature Calculation Best Practices
+
+Use these practices for rolling counts and sums, distinct counterparties,
+temporal velocity features, and FIFO/path calculations over event histories.
+The snippets illustrate the design; adapt schemas and semantics to the job's
+required behavior.
+
+## Use Built-Ins When Velocity Metrics Are Independent
+
+For ordinary rolling counts and sums where same-time peers share one frame,
+prefer a Spark range window:
+
+```python
+from pyspark.sql import Window, functions as F
+
+last_24h = (
+    Window.partitionBy("entity_id")
+    .orderBy(F.col("event_time").cast("long"))
+    .rangeBetween(-24 * 60 * 60, 0)
+)
+
+velocity = (
+    events.withColumn("event_count_24h", F.count(F.lit(1)).over(last_24h))
+    .withColumn("amount_usd_24h", F.sum("amount_usd").over(last_24h))
+)
+```
+
+This stays inside Catalyst and avoids Python serialization. It is not sufficient
+when same-time visibility uses secondary keys, when distinct state requires
+reference-counted eviction, or when velocity and FIFO transitions must share
+one exact sequence.
+
+## Keep Coupled State In One Pure Sweep
+
+When rolling windows and FIFO/path transitions share one exact sequence,
+isolate that sequence as a pure algorithm before choosing how Spark executes
+it:
+
+```python
+from datetime import timedelta
+
+
+def sweep_events(events):
+    ordered = sorted(
+        events,
+        key=lambda event: (
+            event.event_time,
+            event.transaction_id,
+            event.event_id,
+        ),
+    )
+    windows = {
+        "1h": RollingWindow(timedelta(hours=1)),
+        "24h": RollingWindow(timedelta(hours=24)),
+    }
+    fifo = FifoState()
+
+    for event in ordered:
+        for window in windows.values():
+            window.append(event)
+        path = fifo.advance(event)
+        yield build_result(event, windows, path)
+```
+
+This is the semantic core, not a recommendation to wrap it in `F.udf`. Select
+the Spark execution API using the decision table in
+[Transformation Design](transformation-design.md): built-in windows first,
+then bounded higher-order aggregation, grouped pandas only for bounded groups,
+and a different engine/design for large sequential state.
+
+Whichever bridge is chosen, emit one result per unique output identity and
+measure it against the pair relation or repeated grouping it replaces.
+
+## Make Window Boundaries Literal
+
+For a current-inclusive horizon `H`, evict only events strictly older than the
+cutoff:
+
+```python
+from collections import Counter, deque
+
+
+class RollingWindow:
+    def __init__(self, horizon):
+        self.horizon = horizon
+        self.events = deque()
+        self.count = 0
+        self.counterparties = Counter()
+
+    def append(self, event):
+        cutoff = event.event_time - self.horizon
+        while self.events and self.events[0].event_time < cutoff:
+            self.remove(self.events.popleft())
+
+        self.events.append(event)  # current event is visible
+        self.count += 1
+        if event.counterparty_key is not None:
+            self.counterparties[event.counterparty_key] += 1
+
+    def remove(self, event):
+        self.count -= 1
+        key = event.counterparty_key
+        if key is not None:
+            self.counterparties[key] -= 1
+            if self.counterparties[key] == 0:
+                del self.counterparties[key]
+```
+
+Using `<= cutoff` makes the lower boundary exclusive. Test the exact boundary
+and one smallest supported time unit beyond it.
+
+A reference-counted map is required for distinct counterparties because a set
+cannot represent repeated counterparties during eviction.
+
+## Define Same-Timestamp Visibility
+
+Use the complete stable order, for example:
+
+```text
+(event_time, transaction_id, event_id)
+```
+
+If same-time events see earlier tie-ordered events, update and emit in that
+order. If they should all see the same batch, emit the timestamp batch before
+updating state. Never let incidental Spark input order define visibility.
+
+## Use Numerically Stable Rolling State
+
+Repeated floating addition and eviction can leave residue. Use compensated
+addition/removal, or exact decimal state when the domain requires exact money:
+
+```python
+class CompensatedSum:
+    def __init__(self):
+        self.total = 0.0
+        self.correction = 0.0
+
+    def add(self, value):
+        updated = self.total + value
+        if abs(self.total) >= abs(value):
+            self.correction += (self.total - updated) + value
+        else:
+            self.correction += (value - updated) + self.total
+        self.total = updated
+
+    def remove(self, value):
+        self.add(-value)
+
+    def value(self):
+        return self.total + self.correction
+```
+
+Test values with different magnitudes, evict them, and assert that the previous
+state is restored.
+
+## Combine Related State
+
+Within the same sweep:
+
+1. Evict expired events from each horizon.
+2. Add the current event according to visibility semantics.
+3. Emit counts, sums, missing-value counts, directional metrics, and distinct
+   counterparties.
+4. Advance FIFO/path state using the same stable order.
+5. Emit one result keyed by output identity.
+
+This removes the pair relation and avoids grouping and sorting the same history
+for each feature family.
+
+## Prove Parity And Plan Shape
+
+Keep the old self-join only as a bounded test oracle. Cover exact 1h/24h
+boundaries, one unit outside, same-time ties, nulls, both directions,
+missing/present FX, repeated counterparties, path transitions, and floating
+residue.
+
+Compare full rows in both directions, add literal boundary assertions, and
+assert the candidate physical plan has no customer/time interval join.
+
+## Measure Residual Per-Key Memory
+
+`groupBy + collect_list` changes quadratic distributed work into linear per-key
+work, but one task still holds the entire largest key. Measure and report
+`max_rows_per_key`, then production-canary the largest known shape. Use bounded
+state or exact temporal bucketing if one key no longer fits.
+
+## Avoid These False Fixes
+
+- Do not rely on repartitioning to split one hot grouping key.
+- Do not drop history, events, or feature formulas to gain speed.
+- Do not present Arrow transport, broadcasting, or explicit persistence as the
+  performance fix; reduce the logical work instead.
+- Do not report validate-only runtime as end-to-end write performance.

package/skills/app-dev/pyspark/release-check.mjs

@@ -0,0 +1,93 @@
+#!/usr/bin/env node
+import { spawnSync } from 'node:child_process';
+import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { homedir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+const WEEK_MS = 7 * DAY_MS;
+const here = dirname(fileURLToPath(import.meta.url));
+
+function readJson(path, fallback = {}) {
+  try { return JSON.parse(readFileSync(path, 'utf8')); } catch { return fallback; }
+}
+
+function frontmatterValue(text, key) {
+  const frontmatter = /^---\n([\s\S]*?)\n---/.exec(text)?.[1] ?? '';
+  const match = new RegExp('^' + key + ':\\s*([^\\n]+)$', 'm').exec(frontmatter);
+  return match?.[1].trim().replace(/^['"]|['"]$/g, '') ?? null;
+}
+
+function repositoryOf(source) {
+  const value = String(source ?? '').trim();
+  const match = /^(?:git@github\.com:|https:\/\/github\.com\/)?([A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+?)(?:\.git)?$/.exec(value);
+  return match?.[1] ?? null;
+}
+
+function stableVersion(value) {
+  const match = /^v?(\d+)\.(\d+)\.(\d+)$/.exec(String(value ?? '').trim());
+  return match ? match.slice(1).map(Number) : null;
+}
+
+function newer(left, right) {
+  for (let index = 0; index < 3; index += 1) {
+    if (left[index] !== right[index]) return left[index] > right[index];
+  }
+  return false;
+}
+
+function writeCache(path, value) {
+  try {
+    mkdirSync(dirname(path), { recursive: true });
+    writeFileSync(path, JSON.stringify(value));
+  } catch { /* A read-only cache must not block skill use. */ }
+}
+
+function main(now = Date.now()) {
+  if (process.env.ONEABOVEIO_NO_UPDATE_CHECK) return;
+  const skill = readFileSync(join(here, 'SKILL.md'), 'utf8');
+  const name = frontmatterValue(skill, 'name');
+  const localVersion = stableVersion(frontmatterValue(skill, 'release-version'));
+  if (!name || !localVersion) return;
+
+  const lock = readJson(join(process.env.HOME || homedir(), '.agents', '.skill-lock.json'));
+  const entry = lock.skills?.[name];
+  const repository = repositoryOf(entry?.sourceUrl || entry?.source);
+  if (!repository) return;
+
+  const cacheRoot = process.env.XDG_CACHE_HOME || join(process.env.HOME || homedir(), '.cache');
+  const cachePath = join(cacheRoot, '1aboveio-skills', 'release-check.json');
+  const all = readJson(cachePath);
+  const cache = all[repository] && typeof all[repository] === 'object' ? all[repository] : {};
+
+  if (!cache.lastCheck || now - cache.lastCheck >= DAY_MS) {
+    const result = spawnSync('gh', ['api', 'repos/' + repository + '/releases/latest', '--jq', '.tag_name'], {
+      encoding: 'utf8',
+      stdio: ['ignore', 'pipe', 'ignore'],
+      timeout: 1500,
+    });
+    cache.lastCheck = now;
+    if (result.status === 0 && stableVersion(result.stdout)) cache.latestTag = result.stdout.trim();
+    else delete cache.latestTag;
+    all[repository] = cache;
+    writeCache(cachePath, all);
+  }
+
+  const latestVersion = stableVersion(cache.latestTag);
+  if (!latestVersion || !newer(latestVersion, localVersion)) return;
+  if (cache.notifiedTag === cache.latestTag && cache.notifiedAt && now - cache.notifiedAt < WEEK_MS) return;
+
+  cache.notifiedTag = cache.latestTag;
+  cache.notifiedAt = now;
+  all[repository] = cache;
+  writeCache(cachePath, all);
+  const command = `npx skills@1.5.23 add '${repository}#${cache.latestTag}' --global --agent universal claude-code --skill ${name} --yes`;
+  process.stdout.write(
+    `UPDATE_AVAILABLE: A newer 1AboveIO skills release (${cache.latestTag}) is available; this skill is `
+      + `v${localVersion.join('.')}. Ask once whether to refresh ${name}. On approval run \`${command}\`. `
+      + 'The refreshed instructions apply next session. Either way, continue the current task and do not ask again this session.\n',
+  );
+}
+
+try { main(); } catch { /* Update checks are advisory and always fail silent. */ }

package/skills/app-dev/pyspark/scripts/spark_eventlog_summary.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""Summarize stage attempts and final SQL plans from a Spark JSON event log."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from collections.abc import Iterable
+from typing import Any
+
+METRICS = (
+    "input_bytes",
+    "input_records",
+    "output_bytes",
+    "output_records",
+    "shuffle_read_bytes",
+    "shuffle_read_records",
+    "shuffle_write_bytes",
+    "shuffle_write_records",
+    "memory_spill_bytes",
+    "disk_spill_bytes",
+)
+
+
+def _patterns(values: Iterable[str]) -> dict[str, re.Pattern[str]]:
+    result: dict[str, re.Pattern[str]] = {}
+    for value in values:
+        name, separator, pattern = value.partition("=")
+        if not separator or not name or not pattern:
+            raise ValueError(
+                f"invalid --plan-pattern {value!r}; expected NAME=REGEX"
+            )
+        result[name] = re.compile(pattern, re.IGNORECASE)
+    return result
+
+
+def _add(stage: dict[str, Any], name: str, value: Any) -> None:
+    stage[name] = stage.get(name, 0) + (value or 0)
+
+
+def summarize(
+    lines: Iterable[str], plan_patterns: dict[str, re.Pattern[str]]
+) -> dict[str, Any]:
+    app: dict[str, Any] = {}
+    stages: dict[tuple[int, int], dict[str, Any]] = {}
+    sql_execution_ids: set[Any] = set()
+    sql_plans: dict[Any, str] = {}
+    anonymous_sql_id = 0
+
+    for raw in lines:
+        try:
+            event = json.loads(raw)
+        except (json.JSONDecodeError, TypeError):
+            continue
+
+        kind = event.get("Event")
+        if kind == "SparkListenerApplicationStart":
+            app.update(
+                {
+                    "app_id": event.get("App ID"),
+                    "app_name": event.get("App Name"),
+                    "start_time_ms": event.get("Timestamp"),
+                }
+            )
+        elif kind == "SparkListenerApplicationEnd":
+            app["end_time_ms"] = event.get("Timestamp")
+        elif kind in {
+            "org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart",
+            "org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate",
+        }:
+            execution_id = event.get("executionId")
+            if execution_id is None:
+                execution_id = f"anonymous-{anonymous_sql_id}"
+                anonymous_sql_id += 1
+            sql_execution_ids.add(execution_id)
+            sql_plans[execution_id] = event.get("physicalPlanDescription") or ""
+        elif kind == "SparkListenerStageCompleted":
+            info = event.get("Stage Info") or {}
+            stage_id = info.get("Stage ID")
+            if stage_id is None:
+                continue
+            attempt_id = info.get("Stage Attempt ID", 0)
+            key = (stage_id, attempt_id)
+            stage = stages.setdefault(
+                key,
+                {"stage_id": stage_id, "stage_attempt_id": attempt_id},
+            )
+            stage.update(
+                {
+                    "name": info.get("Stage Name"),
+                    "task_count": info.get("Number of Tasks", 0),
+                    "submission_time_ms": info.get("Submission Time"),
+                    "completion_time_ms": info.get("Completion Time"),
+                }
+            )
+            if (
+                stage["submission_time_ms"] is not None
+                and stage["completion_time_ms"] is not None
+            ):
+                stage["wall_ms"] = (
+                    stage["completion_time_ms"] - stage["submission_time_ms"]
+                )
+        elif kind == "SparkListenerTaskEnd":
+            stage_id = event.get("Stage ID")
+            if stage_id is None:
+                continue
+            attempt_id = event.get("Stage Attempt ID", 0)
+            key = (stage_id, attempt_id)
+            stage = stages.setdefault(
+                key,
+                {"stage_id": stage_id, "stage_attempt_id": attempt_id},
+            )
+            info = event.get("Task Info") or {}
+            metrics = event.get("Task Metrics") or {}
+            launch = info.get("Launch Time")
+            finish = info.get("Finish Time")
+            duration = finish - launch if launch is not None and finish is not None else 0
+            stage["max_task_ms"] = max(stage.get("max_task_ms", 0), duration)
+            _add(stage, "task_end_count", 1)
+            _add(stage, "memory_spill_bytes", metrics.get("Memory Bytes Spilled"))
+            _add(stage, "disk_spill_bytes", metrics.get("Disk Bytes Spilled"))
+
+            input_metrics = metrics.get("Input Metrics") or {}
+            output_metrics = metrics.get("Output Metrics") or {}
+            shuffle_read = metrics.get("Shuffle Read Metrics") or {}
+            shuffle_write = metrics.get("Shuffle Write Metrics") or {}
+            _add(stage, "input_bytes", input_metrics.get("Bytes Read"))
+            _add(stage, "input_records", input_metrics.get("Records Read"))
+            _add(stage, "output_bytes", output_metrics.get("Bytes Written"))
+            _add(stage, "output_records", output_metrics.get("Records Written"))
+            _add(
+                stage,
+                "shuffle_read_bytes",
+                (shuffle_read.get("Remote Bytes Read") or 0)
+                + (shuffle_read.get("Local Bytes Read") or 0),
+            )
+            _add(
+                stage,
+                "shuffle_read_records",
+                shuffle_read.get("Total Records Read"),
+            )
+            _add(
+                stage,
+                "shuffle_write_bytes",
+                shuffle_write.get("Shuffle Bytes Written"),
+            )
+            _add(
+                stage,
+                "shuffle_write_records",
+                shuffle_write.get("Shuffle Records Written"),
+            )
+
+    completed = sorted(
+        (stage for stage in stages.values() if "wall_ms" in stage),
+        key=lambda stage: (stage["stage_id"], stage["stage_attempt_id"]),
+    )
+    totals = {
+        name: sum(stage.get(name, 0) for stage in completed) for name in METRICS
+    }
+    sql = {
+        "execution_count": len(sql_execution_ids),
+        "plan_pattern_counts": {
+            name: sum(len(pattern.findall(plan)) for plan in sql_plans.values())
+            for name, pattern in plan_patterns.items()
+        },
+    }
+    if app.get("start_time_ms") is not None and app.get("end_time_ms") is not None:
+        app["eventlog_wall_ms"] = app["end_time_ms"] - app["start_time_ms"]
+
+    def largest(metric: str) -> dict[str, Any]:
+        return max(completed, key=lambda stage: stage.get(metric, 0), default={})
+
+    return {
+        "application": app,
+        "sql": sql,
+        "completed_stage_count": len({stage["stage_id"] for stage in completed}),
+        "completed_stage_attempt_count": len(completed),
+        "totals": totals,
+        "dominant_stage": largest("wall_ms"),
+        "max_task_stage": largest("max_task_ms"),
+        "max_shuffle_write_records_stage": largest("shuffle_write_records"),
+        "stages": completed,
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--plan-pattern",
+        action="append",
+        default=[],
+        metavar="NAME=REGEX",
+        help="count matching physical-plan fragments; may be repeated",
+    )
+    parser.add_argument(
+        "--include-stages",
+        action="store_true",
+        help="include every completed stage instead of only aggregate/top stages",
+    )
+    args = parser.parse_args()
+
+    try:
+        patterns = _patterns(args.plan_pattern)
+    except (ValueError, re.error) as error:
+        parser.error(str(error))
+
+    result = summarize(sys.stdin, patterns)
+    if not result["application"].get("app_id"):
+        parser.error(
+            "no SparkListenerApplicationStart record found; verify the event-log "
+            "path and use shell pipefail so upstream read failures remain visible"
+        )
+    if not args.include_stages:
+        result.pop("stages", None)
+    json.dump(result, sys.stdout, indent=2, sort_keys=True)
+    sys.stdout.write("\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/skills/app-dev/python-backend/LICENSE

@@ -0,0 +1,3 @@
+MIT License
+
+Copyright (c) 2026 1AboveIO