sgnmon 0.1.0__tar.gz

sgnmon-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+*.egg-info
+build/
+dist/
+site/
+__pycache__
+_version.py
+.coverage
+.vscode/
+.idea/
+*.patch
+.claude/settings.local.json

sgnmon-0.1.0/.gitlab-ci.yml

@@ -0,0 +1,67 @@
+include:
+  # -- python
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/sdist@1
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/wheel@1
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/code-quality@1
+    inputs:
+      analyzer: "ruff"
+      requirements: "-r requirements-lint.txt"
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/dependency-scanning@1
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/type-checking@1
+    inputs:
+      project_dir: "src"
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/test@1
+    inputs:
+      install_extra: test
+      python_versions:
+        - "3.11"
+        - "3.12"
+        - "3.13"
+  # test against sgn and sgn-ts from git (main)
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/python/test@1
+    inputs:
+      job_name: "python_test_latest"
+      install_extra: test
+      python_versions:
+        - "3.11"
+        - "3.12"
+        - "3.13"
+  # -- docs
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/mkdocs/build@1
+    inputs:
+      requirements: "-r requirements-docs.txt"
+  - component: $CI_SERVER_FQDN/computing/gitlab/components/mkdocs/pages@1
+    inputs:
+      pages_when: "default"
+
+# -- customizations
+
+# install latest sgn and sgn-ts from git (main) to catch any
+# prerelease changes before they ship
+python_test_latest:
+  before_script:
+    - pip install git+https://git.ligo.org/greg/sgn.git@main
+    - pip install git+https://git.ligo.org/greg/sgn-ts.git@main
+
+ruff:
+  needs:
+    - requirements
+
+mkdocs:
+  needs:
+    - requirements
+
+# -- requirements
+requirements:
+  stage: build
+  image: python:3.12
+  script:
+    - python -m pip install pipx
+    - pipx ensurepath
+    - source ~/.bashrc
+    - pipx install hatch
+    - hatch dep show requirements --feature docs > requirements-docs.txt
+    - hatch dep show requirements --feature lint > requirements-lint.txt
+  artifacts:
+    paths:
+      - requirements-*.txt

sgnmon-0.1.0/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.1.0] - 2026-07-23
+
+### Added
+
+- Initial release: monitoring for SGN pipelines — non-invasive taps and
+  in-graph elements feeding Prometheus metrics (data flow, latency,
+  per-pad execution time, freshness, EOS; opt-in process metrics), a
+  declarative custom-metrics API, per-event latency for event streams,
+  self-pacing health checks, and a monitoring server with a live
+  dashboard, `/metrics`, `/health`, `/status`, and server-sent events,
+  plus a CLI for zero-code attach (`sgnmon run`), demoing, and health
+  checks with plugin-friendly exit codes
+
+[unreleased]: https://git.ligo.org/greg/sgnmon/-/compare/0.1.0...main
+[0.1.0]: https://git.ligo.org/greg/sgnmon/-/tags/0.1.0

sgnmon-0.1.0/CONSOLIDATION-TRACKING.md

@@ -0,0 +1,180 @@
+# Consolidation tracking
+
+Live tracker for the remaining decisions and work in the monitoring
+consolidation ([CONSOLIDATION.md](CONSOLIDATION.md)). Analysis and rationale
+live there; this file only tracks state. Last updated 2026-07-22.
+
+## Done
+
+- [x] Comparative analysis of sgnmon / sgn-skig / sgn testpoints (CONSOLIDATION.md §1–§5)
+- [x] sgnmon gap-closing, four commits on `main` (§6):
+  - [x] `0b61ebe` pad execution-time histogram (`sgnmon_exec_seconds`)
+  - [x] `f97881e` opt-in process metrics (`Monitor(process_metrics=True)`)
+  - [x] `048f98e` custom metrics API (`MetricSpec`/`MetricsMixin`, `Monitor` factories)
+  - [x] `dbb144e` `EventLatency` + `PadProbe.record_latency`
+- [x] Storage decision (§8): Prometheus stays the only query layer; single-node
+      VictoriaMetrics as long-term backend for both ops history (remote_write)
+      and sgnl scientific results (line-protocol push); InfluxDB, scald retention
+      policies, ginterval, and the Grafana exporter all retire
+- [x] Health-check integration decision: monitor `/health` with stock Nagios
+      `check_http` (200 → OK, 503/refused → CRITICAL). No igwn-monitoring-plugins
+      packaging needed — it would put a Python sgnmon install on monitoring hosts
+      just to enrich alert text. Revisit only if on-call wants failing-check
+      names in the page itself; `sgnmon check` remains a dev/debug convenience.
+
+## Decisions open
+
+- [x] **`IGWNAlertSource` home**: moved to **sgn-llai** (decided 2026-07-23,
+      new `sgnllai/sources/` package) — alert-driven low-latency work is
+      sgn-llai's domain, and repo-wide grep found zero current consumers
+      anywhere (verified twice: exact class name across all ligo repos, and
+      case-insensitive igwn-alert terms across sgnl/sgn-cal/sgn-ligo/
+      sgn-llai src+tests+bin+pyproject). Removed from sgn-ligo along with
+      its `igwn-alert` dependency — sgn-ligo's dependency delta from the
+      transport move is now just `igwn-auth-utils` + `jsonschema`;
+      `igwn-alert` is declared by sgn-llai instead.
+
+- [x] **Home for sgn-skig's transport elements**: **sgn-ligo** (decided
+      2026-07-22). Dependency delta is small — sgn-ligo already has
+      `confluent-kafka`, `sgn >= 0.8`, and `sgn-ts`; net-new declarations are
+      `igwn-alert` (lazy-imported by `IGWNAlertSource`), `igwn-auth-utils`
+      (SciToken discovery in `ScitokenKafkaSource`; likely already transitive
+      via gwpy, declare explicitly anyway), and `jsonschema` (`json_frames.py`,
+      which the Kafka elements use). skig's `pyyaml`/`requests` are used only
+      by the Grafana/pipeline layer and stay behind. Migration caveat:
+      `KafkaSource`/`KafkaSink`/`IGWNAlertSource`/`json_frames` are
+      metrics-free and move as-is, but `DelayBuffer`/`RoundRobinDistributor`
+      mix in skig's `MetricsCollectorMixin` — port their metrics to
+      `sgnmon.MetricsMixin` (adds a sgnmon dep wherever they land) or strip
+      them; `EventLatency` is NOT moved — already reimplemented in sgnmon.
+      Merge plan (decided 2026-07-22, supersedes the earlier rename idea):
+      - **Sources — merge.** `ScitokenKafkaSource` folds into `KafkaSource`
+        as an auth option (OAUTHBEARER token callback via lazy-imported
+        `igwn-auth-utils`); it only overrides consumer config. sgn-llai's
+        `run_superevent.py` already auto-picks between the two classes by
+        broker hostname — that logic becomes an `auth="auto"` default.
+      - **Sinks — merge on one base.** scald's kafka `Client.write()` is
+        just `json.dumps` + `produce(key=".".join(tags))` (~15 lines), so
+        both sinks are confluent-kafka JSON producers differing only in
+        payload schema and batching: sgn-ligo's writes scald-consumer format
+        (`{time: [...], data: [...]}`, interval batching; used by sgnl
+        `inspiral.py` + `ll_dq.py`), skig's writes generic per-frame JSON
+        (per-topic pads, lz4, delivery modes; used by sgn-llai). Plan
+        (2026-07-22, Olivia OK'd sgnl adapting): a single `KafkaSink` class,
+        confluent-kafka direct, generic JSON by default with a scald-format
+        payload mode (`{topic: {time, data}}` accumulation, interval flush,
+        tags-as-dotted-key) — no compatibility front-ends; sgnl's two call
+        sites update natively with the migration. Drops `ligo.scald` from
+        the kafka sink.
+      - Consumer census: sgnl uses sgnligo's scald-format sink; sgn-llai uses
+        skig's generic pair; **sgn-cal uses no Kafka at all**.
+      - Discovered: `sgnligo/sinks/influx_sink.py` is a *second* scald→Influx
+        writer outside skig (sgn-ligo keeps `ligo-scald` for it regardless).
+        Out of scope for the transport move, but it is another Influx client
+        to fold into the VM/Prometheus story eventually — added to
+        follow-ups.
+- [ ] **Shared Grafana dashboard ownership**: one static dashboard with
+      `element`/`pad`/`job` variables replaces generated per-pipeline JSON.
+      Who builds/owns it, and where dashboards-as-code live in IGWN Grafana
+      provisioning. sgn-llai's custom-metric panels get hand-built once.
+- [ ] **Manifest question**: does sgn-llai CI depend on skig's
+      `write_metrics_manifest()` YAML export? Dropped from the port; if needed,
+      it is a small addition to the sgnmon mixin. Ask sgnl folks before the
+      migration starts.
+
+## Work queue (in order)
+
+- [ ] **VM prototype** — gates the §8 storage plan. Sysadmin handoff plan:
+      [VM-PILOT-PLAN.md](VM-PILOT-PLAN.md). PromQL/Grafana aggregation notes
+      (how query-time reduction replaces scald's write-time tiers):
+      [PROMQL-GRAFANA-NOTES.md](PROMQL-GRAFANA-NOTES.md). Push a few weeks of one
+      representative sgnl result series (include the sparsest one) into a
+      scratch VictoriaMetrics via InfluxDB line protocol; rebuild the Grafana
+      panel beside the Influx original. Verify: (a) GPS-vs-Unix timestamp
+      convention the scald dashboards assume (VM sample timestamps must be Unix;
+      GPS as value or companion series); (b) sparse series query shape
+      (`max_over_time(x[$__interval])` / points panels vs the ~5m gauge
+      staleness window); (c) whether any Influx data is event *records* rather
+      than time series — those belong in a proper service (e.g. GraceDB).
+- [ ] **Thin results writer** for sgnl (~50 lines, HTTP line-protocol POST to
+      VM). Lives in sgnl or a small dedicated package — not sgnmon.
+- [ ] **sgn-llai migration, metrics first**: `MetricsCollectorMixin` →
+      `sgnmon.MetricsMixin`; `MetricsPipeline` → plain `Pipeline` +
+      `monitor.tap()`. DONE 2026-07-23: `DelayBuffer` +
+      `RoundRobinDistributor` moved into `sgnllai/transforms/` with metrics
+      ported to `sgnmon.MetricsMixin` — Prometheus-shaped families
+      (`delay_buffer_events_total{state=buffered|released|flushed_eos}`,
+      `delay_buffer_size`, `events_distributed_total{worker=N}`), skig's
+      auto-elapsed timing dropped (superseded by `sgnmon_exec_seconds`
+      taps); `metrics_enabled` kwarg kept for bin compatibility, optional
+      `monitor=` for test isolation; sgnmon added to sgn-llai deps; bins'
+      imports updated (EventLatency etc. still from sgneskig until the
+      metrics migration); 29 ported tests + full suite green (test_make_dag
+      collection error is a pre-existing missing `ezdag` dep), flake8/isort
+      clean, all uncommitted. With this, EVERY skig element needing a home
+      has one; what remains of skig is superseded (metrics layer,
+      MetricsPipeline, EventLatency) or dies with the Influx retirement
+      (writer, ScaldMetricsSink, influxdb/grafana modules, Grafana exporter,
+      all five bin scripts). Remaining here: swap sgn-llai's own
+      MetricsCollectorMixin/EventLatency/MetricsPipeline usage to sgnmon. Gotchas: sgnmon `EventLatency` defaults
+      `time_field="time"` (skig: `"gpstime"`) — pass explicitly; push/flusher
+      model becomes scrape — each service needs a `MonitorServer` port and a
+      Prometheus scrape-config entry; GPS point timestamps and
+      `storage_aggregate` have no equivalent — affected dashboards get rebuilt
+      against PromQL.
+- [ ] **Ops wiring**: scrape configs for pipeline services on IGWN Prometheus;
+      `remote_write` → VM; `check_http` `/health` checks in IGWN monitoring.
+- [x] **Transport element move** — COMMITTED 2026-07-23 on feature branches:
+      sgn-ligo `kafka-transport` (`fd94c82` unified KafkaSink w/ provenance
+      + derivation from Yun-Jing Huang's implementation in the message,
+      `35e921b` KafkaSource + json_frames), sgn-llai `skig-elements`
+      (`8cc4d8c` DelayBuffer/RoundRobinDistributor on sgnmon metrics,
+      `a7a1719` IGWNAlertSource), sgnl `kafka-sink-args` (`8c9ea6f` sink
+      kwargs). Module docstrings carry no port notes per Olivia — commit
+      messages hold provenance. Not pushed; MRs when Olivia is ready.
+      Original notes: Done: merged `KafkaSource` (auth="none|scitoken|auto", helpers +
+      lazy igwn-auth-utils) in `sgnligo/sources/kafka_source.py`;
+      `IGWNAlertSource` and `json_frames.py` copied; unified `KafkaSink`
+      (payload_format="json"|"scald", stdout dry-run mode, LIGOTimeGPS-safe
+      encoding on all paths) replacing the scald-client sink; three deps added
+      to sgn-ligo pyproject; sgnl call sites (`inspiral.py`, `ll_dq.py`)
+      updated to `bootstrap_servers=` + `payload_format="scald"`; smoke tests
+      pass. Note: sgn-ligo checkout moved from `compat` branch to `main`
+      (compat's sgnts.compat layer isn't in the installed sgnts).
+      2026-07-23: test port complete — tests/test_kafka_source.py (71),
+      test_kafka_sink.py (39, incl. new scald-mode coverage: accumulation,
+      trigger flush, interval batching, LIGOTimeGPS, stdout mode),
+      test_igwn_alert_source.py (13), test_json_frames.py (ported, minus
+      skig's version-fallback test); sgn-ligo's OWN bins also migrated
+      (`sgnligo/bin/ll_dq.py`, `gwistat.py` used the old sink kwargs — found
+      via test failure); one pre-existing test assertion updated
+      (test_ll_dq test_no_kafka_server, asserts new kwargs). mypy + flake8
+      clean, 182 tests green across affected files, full suite passes (gwosc
+      live-data tests are network-flaky, unrelated). All uncommitted.
+      Remaining: sgn-llai import updates (deferred to the metrics-first
+      sgn-llai migration), commit/MR when Olivia is ready.
+      COMMIT-MESSAGE NOTE (Olivia, 2026-07-23): module docstrings carry no
+      port/provenance notes — the sgn-ligo commit message must say: ported
+      from sgn-skig; KafkaSource merges sgneskig's KafkaSource +
+      ScitokenKafkaSource (auth folded into an option); KafkaSink unifies
+      sgnligo's scald-format sink (scald wire format now produced directly,
+      dropping ligo.scald.io.kafka) with sgneskig's generic JSON sink;
+      IGWNAlertSource and json_frames ported as-is.
+- [ ] **Archive sgn-skig** with a README pointer — only after the migration is
+      *deployed*, not merely merged.
+
+## Follow-ups (no urgency)
+
+- [ ] Upstream conversation with sgn maintainers: sgnmon consuming sgn's
+      built-in `record_frame`/`record_exec` hooks instead of wrapping
+      `pad.call` — would remove the third parallel interception mechanism
+      (CONSOLIDATION.md §3).
+- [x] Document the exec-metric coverage limitation in the sgnmon user docs —
+      done 2026-07-23 (metrics.md + attaching.md: `src`/`snk` taps only, not
+      `adp` wraps or in-graph elements; absence means "not measured").
+- [ ] `sgnligo/sinks/influx_sink.py` (scald→Influx, separate from skig):
+      grep across sgnl/sgn-cal/sgn-llai/sgn-ligo/gstlal found **zero
+      consumers** (only the `sgnligo.sinks` re-export) — likely deletable
+      rather than migratable. Confirm no external/site users, then remove;
+      that clears the last scald→Influx writer outside skig without needing
+      the VM push path for it.

sgnmon-0.1.0/CONSOLIDATION.md

@@ -0,0 +1,273 @@
+# Monitoring consolidation: sgnmon, sgn-skig, and sgn testpoints
+
+Comparative analysis of the three monitoring systems in the sgn ecosystem, with a
+consolidation plan to deprecate sgn-skig and rally behind sgnmon as the monitoring
+library. Written 2026-07-22.
+
+## TL;DR
+
+The three systems barely overlap in what they *collect* — they overlap in *mission*.
+sgnmon is already the strongest general-purpose monitoring library of the three: it
+collects far more out of the box, has health checks, a live dashboard, and a
+Prometheus-native model. sgn-skig's real value is not its metrics (it ships almost
+none) but its **custom-metrics ergonomics** and its **InfluxDB/scald/Grafana ops
+story**; both have clean Prometheus-world equivalents. sgn's testpoints are a
+terminal debugging tool, not a monitoring system — they stay, they're complementary.
+The one hard prerequisite for deprecating sgn-skig: **sgn-llai is its only downstream
+consumer**, and it uses everything — including the Kafka transport elements, which
+are not monitoring code and need a new home before sgn-skig can be archived.
+
+## 1. What each system is
+
+**sgnmon** observes frames at pad granularity via two attachment styles —
+non-invasive taps that wrap pad callables (including post-audioadapter taps), and
+in-graph `MonitorTransform`/`MonitorSink` elements. It exposes ten Prometheus metric
+families plus windowed gauges over an in-memory two-resolution bucket store (1 s
+buckets for windows up to 300 s, 60 s buckets up to 6 h), six health checks (three on
+by default, self-calibrating), a Starlette server with `/metrics`, `/health`,
+`/status`, `/graph`, and `/events` SSE, a live SVG dashboard, and a Typer CLI with
+zero-code attach (`sgnmon run -- <cmd>`) and a Nagios-style `sgnmon check`.
+
+**sgn-skig** (Scald–Kafka–InfluxDB–Grafana) is a metrics *pipeline to storage*, not a
+metrics *collector*. Elements mix in `MetricsCollectorMixin`, declare
+`MetricDeclaration` schemas (timing/counter/gauge + tags), and record via
+`time_operation()` / `increment_counter()` / `record_metric()`. A shared
+background-flushing `MetricsWriter` pushes to InfluxDB through ligo.scald, which owns
+the powers-of-10 reduction. A `GrafanaExporter` generates dashboards from the
+declared schemas, and a small Flask "ginterval" service picks the right retention
+policy for a Grafana time range. It also carries transport elements (KafkaSource,
+KafkaSink, ScitokenKafkaSource, IGWNAlertSource) and utility transforms
+(DelayBuffer, RoundRobinDistributor, EventLatency) that are unrelated to monitoring.
+
+**sgn testpoints** (`src/sgn/testpoint.py` in sgn) are an env-var-activated
+(`TESTPOINT=frame|exec`), in-process debugging facility that renders a live `rich`
+terminal table. Frame mode keeps only the *last* frame per pad; exec mode keeps a
+10-sample sliding window of per-pad execution time (mean/stdev). No HTTP, no export,
+no retention. Its hooks are compiled into `SourcePad.__call__`/`SinkPad.__call__` and
+the graph executor, so it sees things no external library can: actual frame objects
+at every pad boundary and true per-node compute time inside the scheduler.
+Separately, sgn also ships `StatsSource` (psutil process/system metrics emitted *as
+frames*, i.e. data-plane), memory profiling, graphviz pipeline visualization, and a
+bottle-based HTTP control plane.
+
+## 2. Metrics collected — side by side
+
+| What | sgnmon | sgn-skig | sgn testpoints |
+|---|---|---|---|
+| Frame/throughput counts | ✅ `frames`, `gap_frames`, `data/gap_seconds`, `samples` per pad | ❌ | ❌ (last frame snapshot only) |
+| Data latency (GPS now − frame end) | ✅ histogram + last-value gauge per pad, plus `EventLatency` for event streams | ⚠️ only via the `EventLatency` element you must splice in | ❌ |
+| Per-element/pad **execution time** | ✅ `sgnmon_exec_seconds` histogram from `src`/`snk` taps | ✅ auto elapsed-time per pull→new cycle | ✅ exec mode, 10-sample window, terminal only |
+| Gap/liveness/EOS | ✅ gap ratio windows, EOS gauge, freshness timestamps | ❌ | ⚠️ `has gap` column of last frame |
+| Process/system stats (CPU, RSS, IO) | ✅ opt-in `Monitor(process_metrics=True)` (Linux) | ❌ | ⚠️ `StatsSource` (as pipeline frames, not monitoring) |
+| App-defined custom metrics | ✅ declarative `MetricSpec`/`MetricsMixin` + `Monitor.counter/gauge/histogram` | ✅ declarative schema + recording API | ❌ |
+| Windowed rollups in-process | ✅ 5 s–6 h trailing windows | ❌ (delegated to scald/Influx) | ⚠️ 10-sample exec deque |
+| Health checks | ✅ 6 checks, adaptive baselines, HTTP 503 semantics | ❌ | ❌ |
+
+The striking asymmetry: sgn-skig ships only three metric-emitting transforms
+(`events_buffered/released/flushed_eos`, `buffer_size`, `events_distributed`,
+`event_latency`) plus the automatic per-cycle elapsed timer. Everything sgnmon
+measures out of the box, sgn-skig simply doesn't measure. The two things sgnmon
+originally did **not** measure — per-pad execution time and process resource
+usage — existed only in the other two projects; both landed in sgnmon on
+2026-07-22 (§6).
+
+## 3. Feature overlap map
+
+Genuine overlap across all three is narrow:
+
+- **Pipeline topology visualization exists three times**: sgnmon's live SVG dashboard
+  + `/graph` JSON, sgn-skig's Mermaid panel in generated Grafana dashboards, and
+  sgn's graphviz `Pipeline.visualize()`. sgnmon's `/graph` is the only
+  machine-readable, live one; the Mermaid generator can die with sgn-skig, and
+  graphviz stays as a static docs/debug tool.
+- **Per-pad observation hooks exist three times**: sgnmon taps wrap `pad.call`;
+  testpoints have hooks compiled into sgn's `base.py`/`apps.py`; sgn-skig
+  monkey-wraps `pull()`/`new()` via `__init_subclass__`. Three interception
+  mechanisms doing the same job at three code layers. Longer-term, sgn's built-in
+  `record_frame`/`record_exec` hook points are the architecturally cleanest
+  interception layer — sgnmon could optionally consume them instead of wrapping pads
+  itself, but that is an optimization, not a blocker.
+- **Timing**: sgn-skig's elapsed-time auto-tracking ≈ testpoints' exec mode ≈ nothing
+  in sgnmon. This is the one real *metric* overlap between skig and testpoints, and
+  the one gap in sgnmon.
+
+Everything else is disjoint: health checks, SSE dashboard, and Prometheus exposition
+exist only in sgnmon; durable storage, Grafana generation, and multi-resolution
+retention exist only in sgn-skig; frame-content inspection exists only in testpoints.
+
+## 4. Custom metrics in sgnmon
+
+`Monitor` already owns a standard `prometheus_client.CollectorRegistry`, so custom
+metrics are already *possible* — just undocumented and unergonomic. What is worth
+porting from sgn-skig is the ergonomics, not the machinery:
+
+1. **A small declarative layer on `Monitor`**: `monitor.counter(name, description,
+   labels)`, `monitor.gauge(...)`, `monitor.histogram(...)` returning
+   prometheus_client families registered in the monitor's registry.
+2. **A mixin for custom elements** (the `MetricsCollectorMixin` replacement)
+   providing sgn-skig's recording verbs backed by those families:
+   `increment_counter(name, amount, tags)`, `set_gauge(...)`, and a
+   `time_operation(name, tags)` context manager backed by a Histogram. Keeping the
+   verb names close to sgn-skig's makes the sgn-llai migration mostly find-and-replace.
+3. **Type mapping**: skig `counter` → Counter (its `storage_aggregate: sum` concern
+   evaporates — Prometheus counters are cumulative, `increase()` is exact at any
+   resolution); skig `timing` → Histogram, optionally with a last-value gauge
+   (histograms preserve tail behavior at any query resolution, which is what skig's
+   `max` aggregate was hand-rolling); skig `gauge` → Gauge.
+
+Deliberately not carried over: `MetricsPipeline` (tapping means no Pipeline subclass
+is needed), the writer/flusher thread (scraping replaces pushing), schema
+auto-registration against Influx, and `storage_aggregate`.
+
+## 5. Powers-of-10 aggregation — what it solves, and the alternatives
+
+What the scald scheme is *for*: bounded query cost (~300–1000 points) over any
+Grafana time range, achieved by write-side reduction every 300 s into six InfluxDB
+retention policies (1 s → 100 000 s), with per-type aggregate functions (sum for
+counters so counts don't inflate, max for timings so spikes survive), plus
+sgn-skig's Flask ginterval service doing read-side resolution selection via a hidden
+Grafana template variable.
+
+In the Prometheus model sgnmon commits to, **most of that problem does not exist**,
+and the rest is an ops decision outside the library:
+
+- **Correctness needs no pre-aggregation.** Counters are cumulative, so
+  `rate()`/`increase()` are exact at any step size; Grafana auto-scales the query
+  step with the time range, so point counts are bounded by construction. Latency
+  tails survive via the histogram (quantiles compose over any range) and
+  `max_over_time` on the gauges. Both aggregate-correctness rationales documented in
+  skig's `concepts.md` are native PromQL semantics.
+- **What remains is long-range query cost and retention** — a TSDB deployment
+  choice, in rough order of preference:
+  1. **Plain Prometheus with long retention.** sgnmon's cardinality is tiny (10
+     families × element × pad × a few jobs). A single Prometheus with 1–2 y
+     retention handles this scale trivially; query-time stepping does the
+     "resolution selection" for free. Start here; escalate only if range queries
+     actually get slow.
+  2. **Recording rules** for the handful of expensive range queries (e.g. 5 m
+     rollups of `rate(sgnmon_data_seconds[...])`). The moral equivalent of scald's
+     write-side reduction, expressed in ~20 lines of ops YAML instead of a library
+     feature plus a bespoke Flask service.
+  3. **Thanos**, if years of history in object storage are needed: its compactor
+     downsamples to 5 m and 1 h resolutions and its query layer auto-selects
+     resolution per time range — the maintained-infrastructure version of
+     powers-of-10 + ginterval. Mimir/Cortex are the scale-out alternatives but do
+     *not* downsample; VictoriaMetrics is very cheap for long raw retention but
+     gates downsampling behind its enterprise edition.
+  4. **Bridge option** if ops is institutionally committed to InfluxDB/Grafana:
+     scrape `/metrics` with Telegraf into InfluxDB and keep continuous queries
+     there. Retains the storage stack with zero sgn-skig code.
+- **The genuinely open ops question is push vs. pull**, not aggregation: sgn-skig
+  pushes, which works from ephemeral Condor jobs; Prometheus pulls, which requires
+  the scraper to find the job. Standard answers: a Grafana Alloy/vmagent sidecar
+  that scrapes localhost and `remote_write`s out, or file-based service discovery
+  fed from job submission. This is the one item to settle with ops *before*
+  declaring sgn-skig deprecated — the only thing skig's architecture handles that
+  sgnmon does not yet have a designated answer for.
+
+## 6. Gaps sgnmon should close — CLOSED 2026-07-22
+
+1. **Per-pad execution time** — DONE (`0b61ebe`): taps time the original pad
+   callable into the `sgnmon_exec_seconds` histogram (buckets 100µs–5s,
+   `Monitor(exec_buckets=...)`). Known limitation: recorded on `src`/`snk` taps
+   only — not on `adp` internal-pad wraps (one wrap serves several probes) and
+   not by the in-graph elements — so absence of exec data for those probes means
+   "not measured", not "zero cost".
+2. **Process metrics** — DONE (`f97881e`): `Monitor(process_metrics=True)` /
+   `sgnmon run --process-metrics` registers the prometheus_client process
+   collector (`/proc`-based, Linux only). sgn's `StatsSource` keeps its
+   data-plane role.
+3. **Custom metrics API** — DONE (`048f98e`): `sgnmon.metrics` with
+   `MetricSpec`/`MetricsMixin` (skig-compatible verbs) plus
+   `Monitor.counter/gauge/histogram` factories.
+4. **Event-frame latency** — DONE (`dbb144e`): `EventLatency` element records
+   per-event GPS latency through the new `PadProbe.record_latency` without
+   inflating frame counts. Note the default `time_field` is `"time"` (skig
+   defaulted to `"gpstime"`) — migrations must pass it explicitly.
+
+## 7. Deprecation path for sgn-skig
+
+Exactly one downstream consumer exists: **sgn-llai**, and it imports the full
+surface — `MetricsCollectorMixin`/`MetricDeclaration` (heavily, for custom metrics),
+`MetricsPipeline`, `EventLatency`, `DelayBuffer`, `RoundRobinDistributor`, and
+`KafkaSource`/`KafkaSink`/`ScitokenKafkaSource`. The plan splits in three:
+
+- **Monitoring surface → sgnmon** (§4/§6). After that, sgn-llai's `MetricsPipeline`
+  becomes plain `Pipeline` + `monitor.tap()`, and its mixin usage becomes the sgnmon
+  mixin.
+- **Transport elements → a new home.** KafkaSource/KafkaSink/ScitokenKafkaSource/
+  IGWNAlertSource are ingestion, not monitoring — they cannot land in sgnmon without
+  recreating the scope creep that made skig awkward. Options: a new `sgn-kafka`
+  package, or fold into sgn-ligo; the SciToken/IGWN-alert pieces argue for the
+  LIGO-flavored home. `DelayBuffer`/`RoundRobinDistributor` go with them or move
+  into sgn-llai itself if no one else needs them.
+- **Ops surface → dropped or replaced by config.** Grafana dashboard generation
+  becomes largely unnecessary: because sgnmon's families and labels are uniform
+  across every pipeline, one static, hand-maintained Grafana dashboard with
+  `element`/`pad`/`job` variables serves all deployments — strictly better than
+  per-pipeline generated JSON. The ginterval service, provisioning CLIs, and
+  netrc/Influx user tooling are replaced by whatever TSDB choice comes out of §5.
+
+Sequencing: (1) land §6 items in sgnmon — DONE 2026-07-22; (2) settle the
+push-vs-pull + TSDB question with ops — decided, see §8; (3) migrate sgn-llai
+(metrics first, transports once they have a home); (4) archive sgn-skig with a
+README pointer.
+
+Two items discovered after the original analysis, tracked in
+[CONSOLIDATION-TRACKING.md](CONSOLIDATION-TRACKING.md) with the rest of the open
+work: whether sgn-llai's CI depends on skig's `write_metrics_manifest()` YAML
+export (dropped from the port — confirm before migrating), and health-check
+integration with IGWN monitoring (resolved: a stock Nagios `check_http` against
+`/health` suffices; no igwn-monitoring-plugins packaging needed).
+
+**Testpoints**: unchanged. They are a zero-config terminal debugger with access to
+live frame *contents* — a different job than fleet monitoring, and the only
+user-facing thing sgnmon could subsume (exec timing) is §6 item 1.
+
+## 8. Storage decision (resolved 2026-07-22)
+
+Deployment facts: production pipelines are long-lived services; IGWN ops already
+runs Grafana and Prometheus; InfluxDB is maintained only for the sgn-skig stack.
+Two workloads must be covered: operational metrics (weeks-long queries) and sgnl's
+scientific results (GPS-stamped, sparse/backfilled series currently stored via
+scald→InfluxDB and browsed in Grafana).
+
+**Decision: Prometheus stays the only query layer; a single-node VictoriaMetrics
+instance becomes the long-term backend; InfluxDB is retired.**
+
+- Ops metrics: IGWN Prometheus scrapes each service's `/metrics` and
+  `remote_write`s to VictoriaMetrics for multi-year raw retention. No
+  downsampling needed at sgnmon's cardinality; recording rules are the escape
+  hatch if a specific range query ever gets slow.
+- How this preserves scald's bounded-query property without pre-aggregation:
+  Grafana range queries carry a step ≈ range / max-data-points (~1000), so any
+  panel returns ~1000 points per series regardless of time span — the read-side
+  resolution selection ginterval provided, but automatic and per zoom level
+  (and zooming in reaches raw data, which scald expired after minutes). Scald's
+  write-time aggregate choice becomes a query-time choice via
+  `increase(x[$__interval])` / `max_over_time(x[$__interval])` etc. Server-side
+  scan cost of raw data is negligible at this scale (an 8-week, 20-series panel
+  scans ~10M samples; VM processes tens of millions per second per core), and
+  raw retention is single-digit GB/year. Caveat: sparse science series should
+  be queried with range functions or points panels — plain gauge queries only
+  fill forward within the ~5m staleness window; verify during the prototype.
+- Scientific results: plain Prometheus scraping is the wrong tool (wall-clock
+  scrape timestamps, no backfill, no per-event fidelity). VictoriaMetrics accepts
+  pushed data with arbitrary timestamps via the InfluxDB v1 line-protocol `/write`
+  API and a timestamped Prometheus-text import endpoint, and serves the
+  Prometheus query API, so it appears in Grafana as just another Prometheus
+  datasource. sgnl pushes results directly; a thin line-protocol writer (~50
+  lines) should live in sgnl or a small dedicated package — not in sgnmon.
+- "Current state" science values (latest range, live thresholds) can instead be
+  sgnmon custom gauges and ride the scrape path for free.
+- Thanos was rejected (object-storage retention only, no push/backfill story —
+  would still require a second system for science data); Mimir likewise, plus
+  heavier operations.
+- Migration checks: (a) prototype one representative sgnl series end to end
+  (line-protocol push of a few weeks of GPS-stamped values into a scratch VM
+  instance, rebuild the Grafana panel, compare against the Influx original);
+  (b) confirm the GPS-vs-Unix timestamp convention the existing scald dashboards
+  assume — VM/Grafana sample timestamps must be Unix time, with GPS kept as the
+  value or a companion series; (c) audit whether anything in sgnl's Influx data
+  is event *records* rather than time series — those belong in a proper service
+  (e.g. GraceDB), not any TSDB.