@bull-board/metrics 8.4.0 → 8.6.0

package/README.md

@@ -39,8 +39,20 @@ recorder downtime, for example:
 
 On shutdown, call `recorder.stop()` and `provider.disconnect()`. Both only close the Redis connection if the recorder/provider opened it internally, so it's a safe no-op if you passed in your own `Redis` instance.
 
+`connection` may be ioredis options or a `Redis` instance you created. `ioredis` is a peer dependency (v5 or v6): resolve a single copy in your app, and if you reuse an existing client, pass one built from that same `ioredis` — a client from a different install (for example one created internally by a BullMQ pinned to a different ioredis major) is not recognized as a `Redis` instance and would be misread as options.
+
 Timestamps and buckets are UTC.
 
+## Job latency
+
+Alongside the completed/failed counters, the recorder tracks two histograms per queue: wait time (`processedOn - timestamp`, how long a job sat before a worker picked it up) and run time (`finishedOn - processedOn`, how long the handler took). They diagnose different problems, so they're kept separate rather than combined into one number.
+
+Both are collected by scanning the completed and failed sorted sets (BullMQ scores them by finish time via `moveToFinished`'s `ZADD`) past a watermark on the recorder's existing tick, so no worker changes are needed and there's no precondition on `queue.getMetrics()`.
+
+The wait histogram only sees jobs that finished, so it goes quiet exactly when a queue is backed up and jobs stop finishing. A queue-age gauge (oldest job still waiting) is recorded alongside it for that reason, and the UI overlays it on the wait chart. Retries are excluded from wait time only, since `timestamp` is a job's creation but `processedOn` is its latest attempt. Percentiles are estimates bounded by bucket width; the bucket layout is fixed, not configurable, because two ranges with different layouts can't be merged into one percentile.
+
+`removeOnComplete: true` deletes jobs the instant they finish, so there's nothing left to scan; that queue will never show latency data. Latency sampling is on by default; set `latency: false` on `MetricsRecorder` to turn it off.
+
 ## Storage
 
 Every key lives under `bull-board:metrics:`. Each snapshot is written at three resolutions at once, each with its own retention, because they cost very different amounts:
@@ -61,6 +73,16 @@ At the defaults that's roughly 1.1 MB for a queue busy every minute of every day
 
 The minute window is the one worth tuning: it holds essentially all the bytes, and it doubles as the recorder's catch-up window after downtime. `retentionDays: N` still works and sets the hourly and daily windows, leaving the minute window at its default.
 
+Latency histograms and the queue-age gauge use a separate packed format and are measured separately, at 90 day retention:
+
+| Scenario | Measured |
+| --- | --- |
+| One queue, both histograms plus the queue-age gauge, realistic distribution | 254.5 KB |
+| Same, pathological: all 18 buckets populated heavily every hour | 574.6 KB |
+| Shared `__global__` cross-queue rollup | ~224 KB once, for the whole board |
+
+That rollup is a single shared cost, not multiplied per queue: 200 queues at typical traffic is roughly 200 × 254.5 KB, about 50 MB, plus the one shared 224 KB rollup. `sample()` takes about 9.11 ms per tick at 1000 finished jobs and about 12 Redis round trips per queue per tick, flat in job count; a subsampling cap above `maxLatencySamplesPerTick` keeps that bounded even at 10,000 jobs a tick.
+
 Retention is enforced by Redis. Day-scoped keys expire on their own TTL; the daily totals hashes are trimmed to the window as each new day rolls in.
 
 ## Inspecting and clearing history

package/dist/HistoryAdmin.d.ts

@@ -91,6 +91,12 @@ export declare class MetricsHistoryAdmin {
      * Deletes recorded history. Purging a single queue also subtracts that queue's minutes
      * from the global rollup, so the cross-queue chart stays correct instead of keeping the
      * removed queue's throughput folded into it forever.
+     *
+     * That correction covers the counter metrics only. The global runtime, waittime and
+     * queueage rollups keep the purged queue's contribution until their own retention drops
+     * it: a packed bucket vector cannot be decremented field by field, and a max gauge has no
+     * record of which queue produced the maximum, so there is nothing to subtract. The
+     * per-queue keys are still deleted either way. See SUMMABLE_METRICS.
      */
     purge(opts?: PurgeOptions): Promise<PurgeResult>;
     /**
@@ -101,12 +107,17 @@ export declare class MetricsHistoryAdmin {
      * Fields that drain to zero are dropped: the recorder never writes a zero bucket, so a
      * leftover zero would read as recorded-but-idle instead of not recorded.
      * Returns the number of global keys it deleted.
+     *
+     * Only the summable metrics are touched; the latency ones are skipped outright rather than
+     * silently producing a no-op subtraction of their packed values. See SUMMABLE_METRICS.
      */
     private subtractDayFromGlobal;
     /**
      * Same idea for the daily rollup: the global totals hash is the sum of the per-queue
      * totals hashes, so it is corrected from those rather than re-derived from day hashes,
      * which may already have expired. Returns the number of global fields it removed.
+     *
+     * Skips the latency metrics for the same reason as subtractDayFromGlobal.
      */
     private subtractTotalsFromGlobal;
     /**

package/dist/HistoryAdmin.js

@@ -9,7 +9,14 @@ const BATCH = 256;
 /** Keys per pipelined `stats()` batch. Each key costs two commands. */
 const MEASURE_BATCH = 250;
 const DAY_PATTERN = /^\d{4}-\d{2}-\d{2}$/;
-const METRICS = ['completed', 'failed'];
+const METRICS = ['completed', 'failed', 'runtime', 'waittime', 'queueage'];
+/**
+ * Metrics whose global rollup is a plain summable counter, so one queue's share can be taken
+ * back out of it with HINCRBY. The latency metrics are deliberately absent: runtime and
+ * waittime pack a whole bucket vector into a single field and queueage holds a max gauge,
+ * and neither can be corrected by a scalar decrement. See `purge`.
+ */
+const SUMMABLE_METRICS = ['completed', 'failed'];
 /**
  * Parses the three key shapes from the right, because queue names may themselves contain
  * colons:
@@ -86,7 +93,10 @@ class MetricsHistoryAdmin {
             this.ownsRedis = false;
         }
         else {
-            this.redis = new ioredis_1.Redis(opts.connection);
+            // Default to RESP2 (ioredis v6 enables RESP3 by default). Keeps exact v5 wire parity and
+            // support for Redis < 6.0, whose `HELLO 3` handshake fails. Spread order lets an explicit
+            // caller `protocol` win. Only on the options path -- an injected client's protocol is its own.
+            this.redis = new ioredis_1.Redis({ protocol: 2, ...opts.connection });
             this.ownsRedis = true;
         }
     }
@@ -189,6 +199,12 @@ class MetricsHistoryAdmin {
      * Deletes recorded history. Purging a single queue also subtracts that queue's minutes
      * from the global rollup, so the cross-queue chart stays correct instead of keeping the
      * removed queue's throughput folded into it forever.
+     *
+     * That correction covers the counter metrics only. The global runtime, waittime and
+     * queueage rollups keep the purged queue's contribution until their own retention drops
+     * it: a packed bucket vector cannot be decremented field by field, and a max gauge has no
+     * record of which queue produced the maximum, so there is nothing to subtract. The
+     * per-queue keys are still deleted either way. See SUMMABLE_METRICS.
      */
     async purge(opts = {}) {
         const before = opts.before === undefined ? null : toDay(opts.before);
@@ -246,9 +262,12 @@ class MetricsHistoryAdmin {
      * Fields that drain to zero are dropped: the recorder never writes a zero bucket, so a
      * leftover zero would read as recorded-but-idle instead of not recorded.
      * Returns the number of global keys it deleted.
+     *
+     * Only the summable metrics are touched; the latency ones are skipped outright rather than
+     * silently producing a no-op subtraction of their packed values. See SUMMABLE_METRICS.
      */
     async subtractDayFromGlobal(key, parsed) {
-        if (parsed.day === null) {
+        if (parsed.day === null || !SUMMABLE_METRICS.includes(parsed.metric)) {
             return 0;
         }
         const minutes = await this.redis.hgetall(key);
@@ -277,8 +296,13 @@ class MetricsHistoryAdmin {
      * Same idea for the daily rollup: the global totals hash is the sum of the per-queue
      * totals hashes, so it is corrected from those rather than re-derived from day hashes,
      * which may already have expired. Returns the number of global fields it removed.
+     *
+     * Skips the latency metrics for the same reason as subtractDayFromGlobal.
      */
     async subtractTotalsFromGlobal(metric, days, values) {
+        if (!SUMMABLE_METRICS.includes(metric)) {
+            return 0;
+        }
         const globalTotals = (0, keys_1.totalsHashKey)(keys_1.GLOBAL_QUEUE, metric);
         const pipeline = this.redis.multi();
         const touched = [];

package/dist/HistoryAdmin.js.map

@@ -1 +1 @@
-{"version":3,"file":"HistoryAdmin.js","sourceRoot":"","sources":["../src/HistoryAdmin.ts"],"names":[],"mappings":";;;AA8EA,0CA0BC;AAxGD,qCAAmD;AACnD,iCAAoG;AAEpG,MAAM,UAAU,GAAG,GAAG,CAAC;AACvB,MAAM,KAAK,GAAG,GAAG,CAAC;AAClB,uEAAuE;AACvE,MAAM,aAAa,GAAG,GAAG,CAAC;AAC1B,MAAM,WAAW,GAAG,qBAAqB,CAAC;AAC1C,MAAM,OAAO,GAAG,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;AA0DxC;;;;;;;;;;;GAWG;AACH,SAAgB,eAAe,CAAC,GAAW;IACzC,MAAM,MAAM,GAAG,GAAG,gBAAS,GAAG,CAAC;IAC/B,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClD,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAErC,IAAI,IAAI,KAAK,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;QACnE,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC3F,CAAC;IACD,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,KAAK,CAAC,MAAM,IAAI,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,KAAK,gBAAS,EAAE,CAAC;QAC/D,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;QACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;AACjG,CAAC;AAED,SAAS,UAAU;IACjB,OAAO;QACL,MAAM,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC3B,GAAG,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;KAC3B,CAAC;AACJ,CAAC;AAED,+EAA+E;AAC/E,SAAS,YAAY,CAAC,MAAiB;IACrC,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;QACxB,OAAO,IAAA,oBAAa,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC;IACpD,CAAC;IACD,OAAO,MAAM,CAAC,IAAI,KAAK,MAAM;QAC3B,CAAC,CAAC,IAAA,kBAAW,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC;QACtD,CAAC,CAAC,IAAA,iBAAU,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC;AAC1D,CAAC;AAED,SAAS,KAAK,CAAC,KAAoB;IACjC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YAC7B,MAAM,IAAI,KAAK,CAAC,6CAA6C,KAAK,GAAG,CAAC,CAAC;QACzE,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC;IACD,OAAO,KAAK,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AAC1C,CAAC;AAED;;;;;GAKG;AACH,MAAa,mBAAmB;IAI9B,YAAY,IAAgC;QAC1C,IAAI,IAAI,CAAC,UAAU,YAAY,eAAK,EAAE,CAAC;YACrC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,UAAU,CAAC;YAC7B,IAAI,CAAC,SAAS,GAAG,KAAK,CAAC;QACzB,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,KAAK,GAAG,IAAI,eAAK,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;YACxC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC;QACxB,CAAC;IACH,CAAC;IAED,UAAU;QACR,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,CAAC;QAC1B,CAAC;IACH,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,KAAK;;QACT,MAAM,OAAO,GAAG,IAAI,GAAG,EAA6B,CAAC;QACrD,MAAM,KAAK,GAAG,UAAU,EAAE,CAAC;QAC3B,IAAI,IAAI,GAAG,CAAC,CAAC;QACb,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,IAAI,SAAS,GAAkB,IAAI,CAAC;QACpC,IAAI,SAAS,GAAkB,IAAI,CAAC;QAEpC,MAAM,KAAK,GAAyC,EAAE,CAAC;QACvD,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC;YACpC,IAAI,MAAM,EAAE,CAAC;gBACX,KAAK,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAC9B,CAAC;QACH,CAAC;QACD,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;QAEvE,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;YAClD,MAAM,EAAE,IAAI,EAAE,GAAG,EAAE,GAAG,YAAY,CAAC,KAAK,CAAC,CAAC;YAE1C,MAAM,KAAK,GAAG,MAAA,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,mCAAI;gBACzC,KAAK,EAAE,MAAM,CAAC,KAAK;gBACnB,IAAI,EAAE,CAAC;gBACP,KAAK,EAAE,CAAC;gBACR,OAAO,EAAE,CAAC;gBACV,IAAI,EAAE,EAAE;gBACR,KAAK,EAAE,UAAU,EAAE;aACpB,CAAC;YACF,KAAK,CAAC,IAAI,IAAI,CAAC,CAAC;YAChB,KAAK,CAAC,KAAK,IAAI,IAAI,CAAC;YACpB,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YACnC,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YACvC,IAAI,IAAI,CAAC,CAAC;YACV,KAAK,IAAI,IAAI,CAAC;YACd,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YAC7B,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YAEjC,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC7B,KAAK,CAAC,OAAO,IAAI,GAAG,CAAC;gBACrB,OAAO,IAAI,GAAG,CAAC;YACjB,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,EAAE,CAAC;gBACf,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBAC5B,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;gBACD,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;YACH,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;QACnC,CAAC;QAED,MAAM,MAAM,GAAG,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvE,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;YAC3B,KAAK,CAAC,IAAI,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAC/C,CAAC;QACD,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC;IACvE,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,OAAO,CAAC,IAAc;;QAClC,MAAM,GAAG,GAAoC,EAAE,CAAC;QAChD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,aAAa,EAAE,CAAC;YACpD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,aAAa,CAAC,CAAC;YAC/C,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvC,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;gBACxB,QAAQ,CAAC,MAAM,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;gBAC9B,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACrB,CAAC;YACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;gBACtC,GAAG,CAAC,IAAI,CAAC;oBACP,IAAI,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;oBACzC,GAAG,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;iBAC7C,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QACD,OAAO,GAAG,CAAC;IACb,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,KAAK,CAAC,OAAqB,EAAE;QACjC,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACrE,MAAM,MAAM,GAAgB,EAAE,WAAW,EAAE,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,CAAC;QACjE,MAAM,OAAO,GAAyC,EAAE,CAAC;QACzD,MAAM,UAAU,GAAyC,EAAE,CAAC;QAE5D,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC;YACpC,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,SAAS;YACX,CAAC;YACD,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,MAAM,CAAC,KAAK,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC;gBAC5D,SAAS;YACX,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;gBACxB,UAAU,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YACnC,CAAC;iBAAM,IAAI,MAAM,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,MAAM,EAAE,CAAC;gBAClD,OAAO,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAChC,CAAC;QACH,CAAC;QAED,qFAAqF;QACrF,2EAA2E;QAC3E,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,IAAI,CAAC,KAAK,KAAK,mBAAY,CAAC;QAE7E,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,OAAO,EAAE,CAAC;YACtC,IAAI,YAAY,EAAE,CAAC;gBACjB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,qBAAqB,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YACtE,CAAC;YACD,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACrD,CAAC;QAED,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,UAAU,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,MAAM,KAAK,IAAI,IAAI,GAAG,GAAG,MAAM,CAAC,CAAC;YAC7F,IAAI,YAAY,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBACrC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,CAAC;gBACrD,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,wBAAwB,CAAC,MAAM,CAAC,MAAM,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC;YAC5F,CAAC;YACD,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;gBACpB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBACnD,SAAS;YACX,CAAC;YACD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;gBAC7C,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;YACnF,CAAC;YACD,IAAI,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC;gBACvC,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;YACrD,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;;;;OAQG;IACK,KAAK,CAAC,qBAAqB,CAAC,GAAW,EAAE,MAAiB;QAChE,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;YACxB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC9C,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC;YAC1C,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,SAAS;YACX,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACpB,QAAQ,CAAC,OAAO,CAAC,SAAS,EAAE,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC;QAC7C,CAAC;QACD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACnE,CAAC;QACD,OAAO,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3F,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,wBAAwB,CACpC,MAAc,EACd,IAAc,EACd,MAAyB;QAEzB,MAAM,YAAY,GAAG,IAAA,oBAAa,EAAC,mBAAY,EAAE,MAAM,CAAC,CAAC;QACzD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,IAAI,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE;YACtB,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;YACrC,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,OAAO;YACT,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YAClB,QAAQ,CAAC,OAAO,CAAC,YAAY,EAAE,GAAG,EAAE,CAAC,KAAK,CAAC,CAAC;QAC9C,CAAC,CAAC,CAAC;QACH,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,OAAO,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,YAAY,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACjF,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,CAAC,IAAI;QACjB,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;QAC/B,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,GAAG,CAAC;YACF,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CACzC,MAAM,EACN,OAAO,EACP,GAAG,gBAAS,IAAI,EAChB,OAAO,EACP,UAAU,CACX,CAAC;YACF,MAAM,GAAG,IAAI,CAAC;YACd,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;gBACxB,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;oBACnB,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;oBACd,MAAM,GAAG,CAAC;gBACZ,CAAC;YACH,CAAC;QACH,CAAC,QAAQ,MAAM,KAAK,GAAG,EAAE;IAC3B,CAAC;CACF;AA3QD,kDA2QC"}
+{"version":3,"file":"HistoryAdmin.js","sourceRoot":"","sources":["../src/HistoryAdmin.ts"],"names":[],"mappings":";;;AAqFA,0CA0BC;AA/GD,qCAAmD;AACnD,iCAAoG;AAEpG,MAAM,UAAU,GAAG,GAAG,CAAC;AACvB,MAAM,KAAK,GAAG,GAAG,CAAC;AAClB,uEAAuE;AACvE,MAAM,aAAa,GAAG,GAAG,CAAC;AAC1B,MAAM,WAAW,GAAG,qBAAqB,CAAC;AAC1C,MAAM,OAAO,GAAG,CAAC,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAE,UAAU,CAAC,CAAC;AAC3E;;;;;GAKG;AACH,MAAM,gBAAgB,GAAG,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;AA0DjD;;;;;;;;;;;GAWG;AACH,SAAgB,eAAe,CAAC,GAAW;IACzC,MAAM,MAAM,GAAG,GAAG,gBAAS,GAAG,CAAC;IAC/B,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClD,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAErC,IAAI,IAAI,KAAK,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;QACnE,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC3F,CAAC;IACD,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,KAAK,CAAC,MAAM,IAAI,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,KAAK,gBAAS,EAAE,CAAC;QAC/D,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;QACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;AACjG,CAAC;AAED,SAAS,UAAU;IACjB,OAAO;QACL,MAAM,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC3B,GAAG,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;KAC3B,CAAC;AACJ,CAAC;AAED,+EAA+E;AAC/E,SAAS,YAAY,CAAC,MAAiB;IACrC,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;QACxB,OAAO,IAAA,oBAAa,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC;IACpD,CAAC;IACD,OAAO,MAAM,CAAC,IAAI,KAAK,MAAM;QAC3B,CAAC,CAAC,IAAA,kBAAW,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC;QACtD,CAAC,CAAC,IAAA,iBAAU,EAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC;AAC1D,CAAC;AAED,SAAS,KAAK,CAAC,KAAoB;IACjC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YAC7B,MAAM,IAAI,KAAK,CAAC,6CAA6C,KAAK,GAAG,CAAC,CAAC;QACzE,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC;IACD,OAAO,KAAK,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AAC1C,CAAC;AAED;;;;;GAKG;AACH,MAAa,mBAAmB;IAI9B,YAAY,IAAgC;QAC1C,IAAI,IAAI,CAAC,UAAU,YAAY,eAAK,EAAE,CAAC;YACrC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,UAAU,CAAC;YAC7B,IAAI,CAAC,SAAS,GAAG,KAAK,CAAC;QACzB,CAAC;aAAM,CAAC;YACN,yFAAyF;YACzF,0FAA0F;YAC1F,+FAA+F;YAC/F,IAAI,CAAC,KAAK,GAAG,IAAI,eAAK,CAAC,EAAE,QAAQ,EAAE,CAAC,EAAE,GAAG,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;YAC5D,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC;QACxB,CAAC;IACH,CAAC;IAED,UAAU;QACR,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,CAAC;QAC1B,CAAC;IACH,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,KAAK;;QACT,MAAM,OAAO,GAAG,IAAI,GAAG,EAA6B,CAAC;QACrD,MAAM,KAAK,GAAG,UAAU,EAAE,CAAC;QAC3B,IAAI,IAAI,GAAG,CAAC,CAAC;QACb,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,IAAI,SAAS,GAAkB,IAAI,CAAC;QACpC,IAAI,SAAS,GAAkB,IAAI,CAAC;QAEpC,MAAM,KAAK,GAAyC,EAAE,CAAC;QACvD,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC;YACpC,IAAI,MAAM,EAAE,CAAC;gBACX,KAAK,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAC9B,CAAC;QACH,CAAC;QACD,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;QAEvE,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;YAClD,MAAM,EAAE,IAAI,EAAE,GAAG,EAAE,GAAG,YAAY,CAAC,KAAK,CAAC,CAAC;YAE1C,MAAM,KAAK,GAAG,MAAA,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,mCAAI;gBACzC,KAAK,EAAE,MAAM,CAAC,KAAK;gBACnB,IAAI,EAAE,CAAC;gBACP,KAAK,EAAE,CAAC;gBACR,OAAO,EAAE,CAAC;gBACV,IAAI,EAAE,EAAE;gBACR,KAAK,EAAE,UAAU,EAAE;aACpB,CAAC;YACF,KAAK,CAAC,IAAI,IAAI,CAAC,CAAC;YAChB,KAAK,CAAC,KAAK,IAAI,IAAI,CAAC;YACpB,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YACnC,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YACvC,IAAI,IAAI,CAAC,CAAC;YACV,KAAK,IAAI,IAAI,CAAC;YACd,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YAC7B,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YAEjC,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC7B,KAAK,CAAC,OAAO,IAAI,GAAG,CAAC;gBACrB,OAAO,IAAI,GAAG,CAAC;YACjB,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,EAAE,CAAC;gBACf,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBAC5B,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;gBACD,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;YACH,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;QACnC,CAAC;QAED,MAAM,MAAM,GAAG,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvE,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;YAC3B,KAAK,CAAC,IAAI,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAC/C,CAAC;QACD,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC;IACvE,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,OAAO,CAAC,IAAc;;QAClC,MAAM,GAAG,GAAoC,EAAE,CAAC;QAChD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,aAAa,EAAE,CAAC;YACpD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,aAAa,CAAC,CAAC;YAC/C,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvC,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;gBACxB,QAAQ,CAAC,MAAM,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;gBAC9B,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACrB,CAAC;YACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;gBACtC,GAAG,CAAC,IAAI,CAAC;oBACP,IAAI,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;oBACzC,GAAG,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;iBAC7C,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QACD,OAAO,GAAG,CAAC;IACb,CAAC;IAED;;;;;;;;;;OAUG;IACH,KAAK,CAAC,KAAK,CAAC,OAAqB,EAAE;QACjC,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACrE,MAAM,MAAM,GAAgB,EAAE,WAAW,EAAE,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,CAAC;QACjE,MAAM,OAAO,GAAyC,EAAE,CAAC;QACzD,MAAM,UAAU,GAAyC,EAAE,CAAC;QAE5D,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC;YACpC,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,SAAS;YACX,CAAC;YACD,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,MAAM,CAAC,KAAK,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC;gBAC5D,SAAS;YACX,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;gBACxB,UAAU,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YACnC,CAAC;iBAAM,IAAI,MAAM,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,MAAM,EAAE,CAAC;gBAClD,OAAO,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAChC,CAAC;QACH,CAAC;QAED,qFAAqF;QACrF,2EAA2E;QAC3E,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,IAAI,CAAC,KAAK,KAAK,mBAAY,CAAC;QAE7E,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,OAAO,EAAE,CAAC;YACtC,IAAI,YAAY,EAAE,CAAC;gBACjB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,qBAAqB,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YACtE,CAAC;YACD,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACrD,CAAC;QAED,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,UAAU,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,MAAM,KAAK,IAAI,IAAI,GAAG,GAAG,MAAM,CAAC,CAAC;YAC7F,IAAI,YAAY,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBACrC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,CAAC;gBACrD,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,wBAAwB,CAAC,MAAM,CAAC,MAAM,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC;YAC5F,CAAC;YACD,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;gBACpB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBACnD,SAAS;YACX,CAAC;YACD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;gBAC7C,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;YACnF,CAAC;YACD,IAAI,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC;gBACvC,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;YACrD,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;;;;;;;OAWG;IACK,KAAK,CAAC,qBAAqB,CAAC,GAAW,EAAE,MAAiB;QAChE,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,IAAI,CAAC,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC;YACrE,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC9C,MAAM,SAAS,GAAG,YAAY,CAAC,MAAM,CAAC,CAAC;QACvC,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC;YAC1C,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,SAAS;YACX,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACpB,QAAQ,CAAC,OAAO,CAAC,SAAS,EAAE,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC;QAC7C,CAAC;QACD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACnE,CAAC;QACD,OAAO,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3F,CAAC;IAED;;;;;;OAMG;IACK,KAAK,CAAC,wBAAwB,CACpC,MAAc,EACd,IAAc,EACd,MAAyB;QAEzB,IAAI,CAAC,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YACvC,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,YAAY,GAAG,IAAA,oBAAa,EAAC,mBAAY,EAAE,MAAM,CAAC,CAAC;QACzD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,IAAI,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE;YACtB,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;YACrC,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,OAAO;YACT,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YAClB,QAAQ,CAAC,OAAO,CAAC,YAAY,EAAE,GAAG,EAAE,CAAC,KAAK,CAAC,CAAC;QAC9C,CAAC,CAAC,CAAC;QACH,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,OAAO,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,YAAY,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACjF,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,CAAC,IAAI;QACjB,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;QAC/B,IAAI,MAAM,GAAG,GAAG,CAAC;QACjB,GAAG,CAAC;YACF,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CACzC,MAAM,EACN,OAAO,EACP,GAAG,gBAAS,IAAI,EAChB,OAAO,EACP,UAAU,CACX,CAAC;YACF,MAAM,GAAG,IAAI,CAAC;YACd,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;gBACxB,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;oBACnB,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;oBACd,MAAM,GAAG,CAAC;gBACZ,CAAC;YACH,CAAC;QACH,CAAC,QAAQ,MAAM,KAAK,GAAG,EAAE;IAC3B,CAAC;CACF;AA5RD,kDA4RC"}

package/dist/LatencySampler.d.ts

@@ -0,0 +1,84 @@
+import type { BaseAdapter } from '@bull-board/api/baseAdapter';
+import type { Redis } from 'ioredis';
+import type { LatencyStore } from './LatencyStore';
+export interface LatencySamplerOptions {
+    redis: Redis;
+    store: LatencyStore;
+    /** Recorder tick, used to size the lease and to bound a cold start. */
+    tickMs: number;
+    /** Above this, the tick subsamples uniformly rather than fetching every job. */
+    maxSamplesPerTick?: number;
+    /**
+     * How far back from now a scan stops. Defaults to SAFETY_MARGIN_MS. Injectable so tests
+     * can set it to 0 and sample jobs that just finished, rather than sleeping past the
+     * margin in every case.
+     */
+    safetyMarginMs?: number;
+    /**
+     * Called with anything `sample()` swallows. The default is silent, which keeps a broken
+     * latency scan from taking the counter snapshot down with it, but also makes a collector
+     * that is failing every tick look exactly like an idle board. Supply this to tell the two
+     * apart. Errors thrown by the hook itself are ignored, since a throwing reporter would
+     * undo the containment it was added to observe.
+     */
+    onError?: (error: unknown, queueName: string) => void;
+}
+/**
+ * Copies job durations out of BullMQ's finished sets on the recorder's tick.
+ *
+ * BullMQ's moveToFinished does `ZADD targetSet, timestamp, jobId` and writes the same value
+ * to finishedOn, so the completed and failed sets are sorted sets scored by finish time.
+ * Scanning past a stored watermark therefore returns exactly the jobs finished since the
+ * last tick, with no gaps and no bias. The only loss is a queue whose removeOnComplete
+ * trims faster than the tick runs.
+ */
+export declare class LatencySampler {
+    private readonly redis;
+    private readonly store;
+    private readonly tickMs;
+    private readonly maxSamples;
+    private readonly safetyMarginMs;
+    private readonly onError?;
+    private readonly id;
+    constructor(opts: LatencySamplerOptions);
+    static supports(adapter: BaseAdapter): boolean;
+    /**
+     * One queue, one tick. Swallows its own errors: the counter snapshot is the more
+     * important metric and must not fail as collateral damage from a latency scan. Pass
+     * `onError` to see what was swallowed; without it a collector failing every tick is
+     * indistinguishable from a queue with nothing to sample.
+     */
+    sample(adapter: BaseAdapter): Promise<void>;
+    private leaseKey;
+    /**
+     * Increments are not idempotent the way the counter upsert is, so two recorders scanning
+     * the same range would double every histogram. Only the lease holder scans.
+     *
+     * The TTL is a crash ceiling, not the normal lifetime: a process that dies mid-scan must
+     * not lock the queue out forever. The normal path releases in a finally, because a lease
+     * outliving its scan would make the next tick no-op and halve the sampling rate.
+     */
+    private acquireLease;
+    /** Compare and delete, so a lease that already expired and was retaken is left alone. */
+    private releaseLease;
+    private watermarkKey;
+    /**
+     * Bounded rather than eternal: an unexpiring watermark would leave one key behind per
+     * queue forever once that queue is purged or decommissioned, which is exactly the
+     * unbounded-storage failure this package exists to avoid. The day retention is already the
+     * horizon everything else here is bounded by. Losing the watermark just means the next
+     * tick cold starts, which is already a supported path.
+     */
+    private watermarkTtlSeconds;
+    private sampleDurations;
+    private flush;
+    /**
+     * The backlog is not all in one place. `wait` holds it while the queue is running, but
+     * pausing RENAMEs that list to `paused` and routes new jobs there, and anything added with
+     * a priority goes to the `prioritized` sorted set instead, which can leave `wait`
+     * permanently empty. Reading only `wait` reports a healthy zero for a queue that is badly
+     * backed up, which is the opposite of what this gauge is for, so all three are consulted
+     * and the worst age wins.
+     */
+    private sampleQueueAge;
+}

package/dist/LatencySampler.js

@@ -0,0 +1,273 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.LatencySampler = void 0;
+const histogram_1 = require("./histogram");
+const keys_1 = require("./keys");
+const MS_PER_HOUR = 3600000;
+const SECONDS_PER_DAY = 86400;
+const DEFAULT_MAX_SAMPLES = 5000;
+/**
+ * ZADD into the finished set and this scan are not atomic with respect to each other, so a
+ * job finishing microseconds before the scan can be absent from it. Advancing the watermark
+ * to the highest score seen would skip that job forever. Scanning only up to a slightly
+ * stale bound, and advancing the watermark to that bound rather than to what was observed,
+ * means anything finishing inside the margin is picked up by the next tick instead.
+ * Costs a few seconds of freshness in data that is bucketed hourly.
+ *
+ * The score is `Date.now()` read in the worker process and passed into BullMQ's Lua, not a
+ * Redis-side clock, so the margin only holds while the worker and this recorder agree on
+ * the time: the effective margin is `margin - skew`. A worker running far enough ahead of
+ * the recorder can still land jobs below an already-advanced watermark and lose them.
+ */
+const SAFETY_MARGIN_MS = 5000;
+/**
+ * Copies job durations out of BullMQ's finished sets on the recorder's tick.
+ *
+ * BullMQ's moveToFinished does `ZADD targetSet, timestamp, jobId` and writes the same value
+ * to finishedOn, so the completed and failed sets are sorted sets scored by finish time.
+ * Scanning past a stored watermark therefore returns exactly the jobs finished since the
+ * last tick, with no gaps and no bias. The only loss is a queue whose removeOnComplete
+ * trims faster than the tick runs.
+ */
+class LatencySampler {
+    constructor(opts) {
+        var _a, _b;
+        this.id = `${process.pid}-${Math.random().toString(36).slice(2)}`;
+        this.redis = opts.redis;
+        this.store = opts.store;
+        this.tickMs = opts.tickMs;
+        this.maxSamples = (_a = opts.maxSamplesPerTick) !== null && _a !== void 0 ? _a : DEFAULT_MAX_SAMPLES;
+        this.safetyMarginMs = (_b = opts.safetyMarginMs) !== null && _b !== void 0 ? _b : SAFETY_MARGIN_MS;
+        this.onError = opts.onError;
+    }
+    static supports(adapter) {
+        return typeof adapter.getQueueKey === 'function';
+    }
+    /**
+     * One queue, one tick. Swallows its own errors: the counter snapshot is the more
+     * important metric and must not fail as collateral damage from a latency scan. Pass
+     * `onError` to see what was swallowed; without it a collector failing every tick is
+     * indistinguishable from a queue with nothing to sample.
+     */
+    async sample(adapter) {
+        var _a;
+        if (!LatencySampler.supports(adapter)) {
+            return;
+        }
+        const name = adapter.getName();
+        try {
+            if (!(await this.acquireLease(name))) {
+                return;
+            }
+            try {
+                await this.sampleDurations(adapter, name);
+                await this.sampleQueueAge(adapter, name);
+            }
+            finally {
+                await this.releaseLease(name);
+            }
+        }
+        catch (error) {
+            // Intentionally swallowed, see the method comment.
+            try {
+                (_a = this.onError) === null || _a === void 0 ? void 0 : _a.call(this, error, name);
+            }
+            catch {
+                // A reporter that throws must not resurrect the failure this catch contains.
+            }
+        }
+    }
+    leaseKey(name) {
+        return `${keys_1.NAMESPACE}:${name}:latency:lease`;
+    }
+    /**
+     * Increments are not idempotent the way the counter upsert is, so two recorders scanning
+     * the same range would double every histogram. Only the lease holder scans.
+     *
+     * The TTL is a crash ceiling, not the normal lifetime: a process that dies mid-scan must
+     * not lock the queue out forever. The normal path releases in a finally, because a lease
+     * outliving its scan would make the next tick no-op and halve the sampling rate.
+     */
+    async acquireLease(name) {
+        const held = await this.redis.set(this.leaseKey(name), this.id, 'PX', this.tickMs * 2, 'NX');
+        return held === 'OK';
+    }
+    /** Compare and delete, so a lease that already expired and was retaken is left alone. */
+    async releaseLease(name) {
+        await this.redis.eval(`if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end
+       return 0`, 1, this.leaseKey(name), this.id);
+    }
+    watermarkKey(name) {
+        return `${keys_1.NAMESPACE}:${name}:latency:watermark`;
+    }
+    /**
+     * Bounded rather than eternal: an unexpiring watermark would leave one key behind per
+     * queue forever once that queue is purged or decommissioned, which is exactly the
+     * unbounded-storage failure this package exists to avoid. The day retention is already the
+     * horizon everything else here is bounded by. Losing the watermark just means the next
+     * tick cold starts, which is already a supported path.
+     */
+    watermarkTtlSeconds() {
+        return Math.max(1, Math.floor(this.store.retention.days * SECONDS_PER_DAY));
+    }
+    async sampleDurations(adapter, name) {
+        var _a;
+        const watermarkRaw = await this.redis.get(this.watermarkKey(name));
+        // Cold start begins at the previous tick rather than backfilling, since a first run
+        // against a large completed set would be a surprise fetch storm.
+        const watermark = watermarkRaw ? Number(watermarkRaw) : Date.now() - this.tickMs;
+        const upperBound = Date.now() - this.safetyMarginMs;
+        if (upperBound <= watermark) {
+            return; // ticks closer together than the margin; next tick covers this range
+        }
+        const ids = [];
+        for (const set of ['completed', 'failed']) {
+            const found = await this.redis.zrangebyscore(adapter.getQueueKey(set), `(${watermark}`, upperBound);
+            ids.push(...found);
+        }
+        if (ids.length === 0) {
+            await this.redis.set(this.watermarkKey(name), String(upperBound), 'EX', this.watermarkTtlSeconds());
+            return;
+        }
+        // The id list is one cheap round trip; the HMGETs are the real cost. Above the cap take
+        // a uniform subset rather than the first N, which would bias towards the tick's start.
+        const selected = ids.length > this.maxSamples ? sampleUniformly(ids, this.maxSamples) : ids;
+        // Counts are scaled back up by this ratio, so a subsampled hour reads as an estimate
+        // with the same shape rather than a dip. The fact that it was subsampled is currently
+        // invisible to clients: marking it would mean persisting a flag alongside the packed
+        // vector, which is a storage-format change. Known follow-up.
+        const ratio = ids.length / selected.length;
+        const pipeline = this.redis.pipeline();
+        for (const id of selected) {
+            pipeline.hmget(adapter.getQueueKey(String(id)), 'timestamp', 'processedOn', 'finishedOn', 
+            // BullMQ 5 counts attempts in `atm`; `attemptsMade` is the pre-v5 name and is read
+            // as a fallback exactly the way Job.fromJSON does.
+            'atm', 'attemptsMade');
+        }
+        const rows = await pipeline.exec();
+        const runByHour = new Map();
+        const waitByHour = new Map();
+        for (const row of rows !== null && rows !== void 0 ? rows : []) {
+            const values = row === null || row === void 0 ? void 0 : row[1];
+            if (!values) {
+                continue;
+            }
+            const [timestamp, processedOn, finishedOn, atm, attemptsMade] = values;
+            if (!processedOn || !finishedOn) {
+                continue;
+            }
+            const finished = Number(finishedOn);
+            const processed = Number(processedOn);
+            const hour = Math.floor(finished / MS_PER_HOUR);
+            observe(runByHour, hour, finished - processed, ratio);
+            // A retried job's timestamp is its creation, but processedOn is the latest attempt,
+            // so wait would absorb every prior attempt and backoff. Run time is unaffected.
+            const attempts = Number((_a = atm !== null && atm !== void 0 ? atm : attemptsMade) !== null && _a !== void 0 ? _a : 0);
+            if (timestamp && attempts <= 1) {
+                // Every one of these is a Date.now() taken in some client process, not a Redis-side
+                // clock: timestamp in the producer, processedOn in the worker. Skew between the two
+                // can make the difference negative, hence the clamp.
+                observe(waitByHour, hour, Math.max(0, processed - Number(timestamp)), ratio);
+            }
+        }
+        await this.flush(name, 'runtime', runByHour);
+        await this.flush(name, 'waittime', waitByHour);
+        // The bound, not the highest score observed. See SAFETY_MARGIN_MS.
+        await this.redis.set(this.watermarkKey(name), String(upperBound), 'EX', this.watermarkTtlSeconds());
+    }
+    async flush(name, metric, byHour) {
+        for (const [hour, vector] of byHour) {
+            // Scaled counts are fractional. Left unrounded, join(',') would write seventeen
+            // significant digits per bucket and blow up the packed value the storage design
+            // depends on staying small.
+            await this.store.addSamples(name, metric, hour, vector.map((v) => Math.round(v)));
+        }
+    }
+    /**
+     * The backlog is not all in one place. `wait` holds it while the queue is running, but
+     * pausing RENAMEs that list to `paused` and routes new jobs there, and anything added with
+     * a priority goes to the `prioritized` sorted set instead, which can leave `wait`
+     * permanently empty. Reading only `wait` reports a healthy zero for a queue that is badly
+     * backed up, which is the opposite of what this gauge is for, so all three are consulted
+     * and the worst age wins.
+     */
+    async sampleQueueAge(adapter, name) {
+        var _a;
+        const hour = Math.floor(Date.now() / MS_PER_HOUR);
+        const candidates = await this.redis
+            .pipeline()
+            // BullMQ LPUSHes to the wait list and workers RPOPLPUSH from it, so the tail is oldest.
+            .lrange(adapter.getQueueKey('wait'), -1, -1)
+            .lrange(adapter.getQueueKey('paused'), -1, -1)
+            // The prioritized set is scored by priority, not by time, so neither end is guaranteed
+            // to hold the oldest job. Both ends are an approximation, and a cheap one: the true
+            // oldest would mean fetching the whole set every tick.
+            // String indices: ioredis v6 types `zrange`'s stop arg as string-only (Redis coerces
+            // either way), so numeric literals no longer typecheck. '0'/'-1' are the first/last members.
+            .zrange(adapter.getQueueKey('prioritized'), '0', '0')
+            .zrange(adapter.getQueueKey('prioritized'), '-1', '-1')
+            .exec();
+        // A pipeline reports failures per command: a Redis error arrives in the entry's error
+        // slot next to a null result, rather than as a throw. Reading only the result slot turns
+        // a failed read into an empty backlog and records an age of 0, which is the most
+        // reassuring number this gauge can produce, at the moment it is least entitled to. A gap
+        // in the series reads as "not measured" and is recoverable; a zero is a wrong answer that
+        // looks like a healthy one, so a tick that could not read cleanly records nothing.
+        if (!candidates || candidates.some(([error]) => error)) {
+            return;
+        }
+        const ids = new Set();
+        for (const entry of candidates) {
+            for (const id of (_a = entry[1]) !== null && _a !== void 0 ? _a : []) {
+                ids.add(String(id));
+            }
+        }
+        if (ids.size === 0) {
+            await this.store.recordQueueAge(name, hour, 0);
+            return;
+        }
+        const stamps = this.redis.pipeline();
+        for (const id of ids) {
+            stamps.hget(adapter.getQueueKey(id), 'timestamp');
+        }
+        const rows = await stamps.exec();
+        // Same reasoning: a failed timestamp read does not zero the gauge, it understates it,
+        // which is the same wrong-but-reassuring answer in a quieter form.
+        if (!rows || rows.some(([error]) => error)) {
+            return;
+        }
+        const now = Date.now();
+        let oldest = 0;
+        for (const row of rows) {
+            const raw = row[1];
+            if (!raw) {
+                continue;
+            }
+            const enqueuedAt = Number(raw);
+            if (!Number.isFinite(enqueuedAt)) {
+                continue;
+            }
+            oldest = Math.max(oldest, now - enqueuedAt);
+        }
+        await this.store.recordQueueAge(name, hour, Math.max(0, oldest));
+    }
+}
+exports.LatencySampler = LatencySampler;
+function observe(byHour, hour, durationMs, ratio) {
+    let vector = byHour.get(hour);
+    if (!vector) {
+        vector = (0, histogram_1.emptyVector)();
+        byHour.set(hour, vector);
+    }
+    vector[(0, histogram_1.bucketIndex)(durationMs)] += ratio;
+}
+/** Evenly spaced pick across the list, which preserves the distribution's shape. */
+function sampleUniformly(ids, target) {
+    const stride = ids.length / target;
+    const out = [];
+    for (let i = 0; i < target; i++) {
+        out.push(ids[Math.floor(i * stride)]);
+    }
+    return out;
+}
+//# sourceMappingURL=LatencySampler.js.map

package/dist/LatencySampler.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"LatencySampler.js","sourceRoot":"","sources":["../src/LatencySampler.ts"],"names":[],"mappings":";;;AAEA,2CAAuD;AACvD,iCAAmC;AAGnC,MAAM,WAAW,GAAG,OAAO,CAAC;AAC5B,MAAM,eAAe,GAAG,KAAK,CAAC;AAC9B,MAAM,mBAAmB,GAAG,IAAI,CAAC;AACjC;;;;;;;;;;;;GAYG;AACH,MAAM,gBAAgB,GAAG,IAAI,CAAC;AA6B9B;;;;;;;;GAQG;AACH,MAAa,cAAc;IASzB,YAAY,IAA2B;;QAFtB,OAAE,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC;QAG5E,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;QACxB,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC;QACxB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,IAAI,CAAC,UAAU,GAAG,MAAA,IAAI,CAAC,iBAAiB,mCAAI,mBAAmB,CAAC;QAChE,IAAI,CAAC,cAAc,GAAG,MAAA,IAAI,CAAC,cAAc,mCAAI,gBAAgB,CAAC;QAC9D,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAC9B,CAAC;IAED,MAAM,CAAC,QAAQ,CAAC,OAAoB;QAClC,OAAO,OAAQ,OAAoC,CAAC,WAAW,KAAK,UAAU,CAAC;IACjF,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,MAAM,CAAC,OAAoB;;QAC/B,IAAI,CAAC,cAAc,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;YACtC,OAAO;QACT,CAAC;QACD,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC;QAC/B,IAAI,CAAC;YACH,IAAI,CAAC,CAAC,MAAM,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;gBACrC,OAAO;YACT,CAAC;YACD,IAAI,CAAC;gBACH,MAAM,IAAI,CAAC,eAAe,CAAC,OAA0B,EAAE,IAAI,CAAC,CAAC;gBAC7D,MAAM,IAAI,CAAC,cAAc,CAAC,OAA0B,EAAE,IAAI,CAAC,CAAC;YAC9D,CAAC;oBAAS,CAAC;gBACT,MAAM,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;YAChC,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,mDAAmD;YACnD,IAAI,CAAC;gBACH,MAAA,IAAI,CAAC,OAAO,qDAAG,KAAK,EAAE,IAAI,CAAC,CAAC;YAC9B,CAAC;YAAC,MAAM,CAAC;gBACP,6EAA6E;YAC/E,CAAC;QACH,CAAC;IACH,CAAC;IAEO,QAAQ,CAAC,IAAY;QAC3B,OAAO,GAAG,gBAAS,IAAI,IAAI,gBAAgB,CAAC;IAC9C,CAAC;IAED;;;;;;;OAOG;IACK,KAAK,CAAC,YAAY,CAAC,IAAY;QACrC,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,EAAE,IAAI,CAAC,EAAE,EAAE,IAAI,EAAE,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,IAAI,CAAC,CAAC;QAC7F,OAAO,IAAI,KAAK,IAAI,CAAC;IACvB,CAAC;IAED,yFAAyF;IACjF,KAAK,CAAC,YAAY,CAAC,IAAY;QACrC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CACnB;gBACU,EACV,CAAC,EACD,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,EACnB,IAAI,CAAC,EAAE,CACR,CAAC;IACJ,CAAC;IAEO,YAAY,CAAC,IAAY;QAC/B,OAAO,GAAG,gBAAS,IAAI,IAAI,oBAAoB,CAAC;IAClD,CAAC;IAED;;;;;;OAMG;IACK,mBAAmB;QACzB,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,GAAG,eAAe,CAAC,CAAC,CAAC;IAC9E,CAAC;IAEO,KAAK,CAAC,eAAe,CAAC,OAAwB,EAAE,IAAY;;QAClE,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC,CAAC;QACnE,oFAAoF;QACpF,iEAAiE;QACjE,MAAM,SAAS,GAAG,YAAY,CAAC,CAAC,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,MAAM,CAAC;QAEjF,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,cAAc,CAAC;QACpD,IAAI,UAAU,IAAI,SAAS,EAAE,CAAC;YAC5B,OAAO,CAAC,qEAAqE;QAC/E,CAAC;QAED,MAAM,GAAG,GAAa,EAAE,CAAC;QACzB,KAAK,MAAM,GAAG,IAAI,CAAC,WAAW,EAAE,QAAQ,CAAC,EAAE,CAAC;YAC1C,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,aAAa,CAC1C,OAAO,CAAC,WAAW,CAAC,GAAG,CAAC,EACxB,IAAI,SAAS,EAAE,EACf,UAAU,CACX,CAAC;YACF,GAAG,CAAC,IAAI,CAAC,GAAG,KAAK,CAAC,CAAC;QACrB,CAAC;QACD,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACrB,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAClB,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,EACvB,MAAM,CAAC,UAAU,CAAC,EAClB,IAAI,EACJ,IAAI,CAAC,mBAAmB,EAAE,CAC3B,CAAC;YACF,OAAO;QACT,CAAC;QAED,wFAAwF;QACxF,uFAAuF;QACvF,MAAM,QAAQ,GAAG,GAAG,CAAC,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC;QAC5F,qFAAqF;QACrF,sFAAsF;QACtF,qFAAqF;QACrF,6DAA6D;QAC7D,MAAM,KAAK,GAAG,GAAG,CAAC,MAAM,GAAG,QAAQ,CAAC,MAAM,CAAC;QAE3C,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC;QACvC,KAAK,MAAM,EAAE,IAAI,QAAQ,EAAE,CAAC;YAC1B,QAAQ,CAAC,KAAK,CACZ,OAAO,CAAC,WAAW,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,EAC/B,WAAW,EACX,aAAa,EACb,YAAY;YACZ,mFAAmF;YACnF,mDAAmD;YACnD,KAAK,EACL,cAAc,CACf,CAAC;QACJ,CAAC;QACD,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAEnC,MAAM,SAAS,GAAG,IAAI,GAAG,EAAoB,CAAC;QAC9C,MAAM,UAAU,GAAG,IAAI,GAAG,EAAoB,CAAC;QAE/C,KAAK,MAAM,GAAG,IAAI,IAAI,aAAJ,IAAI,cAAJ,IAAI,GAAI,EAAE,EAAE,CAAC;YAC7B,MAAM,MAAM,GAAG,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAkC,CAAC;YACzD,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,SAAS;YACX,CAAC;YACD,MAAM,CAAC,SAAS,EAAE,WAAW,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,CAAC,GAAG,MAAM,CAAC;YACvE,IAAI,CAAC,WAAW,IAAI,CAAC,UAAU,EAAE,CAAC;gBAChC,SAAS;YACX,CAAC;YACD,MAAM,QAAQ,GAAG,MAAM,CAAC,UAAU,CAAC,CAAC;YACpC,MAAM,SAAS,GAAG,MAAM,CAAC,WAAW,CAAC,CAAC;YACtC,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,GAAG,WAAW,CAAC,CAAC;YAEhD,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,QAAQ,GAAG,SAAS,EAAE,KAAK,CAAC,CAAC;YAEtD,oFAAoF;YACpF,gFAAgF;YAChF,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAA,GAAG,aAAH,GAAG,cAAH,GAAG,GAAI,YAAY,mCAAI,CAAC,CAAC,CAAC;YAClD,IAAI,SAAS,IAAI,QAAQ,IAAI,CAAC,EAAE,CAAC;gBAC/B,oFAAoF;gBACpF,oFAAoF;gBACpF,qDAAqD;gBACrD,OAAO,CAAC,UAAU,EAAE,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,SAAS,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC;YAC/E,CAAC;QACH,CAAC;QAED,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC;QAC7C,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,UAAU,EAAE,UAAU,CAAC,CAAC;QAC/C,mEAAmE;QACnE,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAClB,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,EACvB,MAAM,CAAC,UAAU,CAAC,EAClB,IAAI,EACJ,IAAI,CAAC,mBAAmB,EAAE,CAC3B,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,KAAK,CACjB,IAAY,EACZ,MAAqB,EACrB,MAA6B;QAE7B,KAAK,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,IAAI,MAAM,EAAE,CAAC;YACpC,gFAAgF;YAChF,gFAAgF;YAChF,4BAA4B;YAC5B,MAAM,IAAI,CAAC,KAAK,CAAC,UAAU,CACzB,IAAI,EACJ,MAAM,EACN,IAAI,EACJ,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CACjC,CAAC;QACJ,CAAC;IACH,CAAC;IAED;;;;;;;OAOG;IACK,KAAK,CAAC,cAAc,CAAC,OAAwB,EAAE,IAAY;;QACjE,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,CAAC,CAAC;QAClD,MAAM,UAAU,GAAG,MAAM,IAAI,CAAC,KAAK;aAChC,QAAQ,EAAE;YACX,wFAAwF;aACvF,MAAM,CAAC,OAAO,CAAC,WAAW,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;aAC3C,MAAM,CAAC,OAAO,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;YAC9C,uFAAuF;YACvF,oFAAoF;YACpF,uDAAuD;YACvD,qFAAqF;YACrF,6FAA6F;aAC5F,MAAM,CAAC,OAAO,CAAC,WAAW,CAAC,aAAa,CAAC,EAAE,GAAG,EAAE,GAAG,CAAC;aACpD,MAAM,CAAC,OAAO,CAAC,WAAW,CAAC,aAAa,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC;aACtD,IAAI,EAAE,CAAC;QAEV,sFAAsF;QACtF,yFAAyF;QACzF,iFAAiF;QACjF,yFAAyF;QACzF,0FAA0F;QAC1F,mFAAmF;QACnF,IAAI,CAAC,UAAU,IAAI,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,EAAE,EAAE,CAAC,KAAK,CAAC,EAAE,CAAC;YACvD,OAAO;QACT,CAAC;QAED,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;QAC9B,KAAK,MAAM,KAAK,IAAI,UAAU,EAAE,CAAC;YAC/B,KAAK,MAAM,EAAE,IAAI,MAAC,KAAK,CAAC,CAAC,CAAqB,mCAAI,EAAE,EAAE,CAAC;gBACrD,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;YACtB,CAAC;QACH,CAAC;QACD,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;YACnB,MAAM,IAAI,CAAC,KAAK,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;YAC/C,OAAO;QACT,CAAC;QAED,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC;QACrC,KAAK,MAAM,EAAE,IAAI,GAAG,EAAE,CAAC;YACrB,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,WAAW,CAAC,EAAE,CAAC,EAAE,WAAW,CAAC,CAAC;QACpD,CAAC;QACD,MAAM,IAAI,GAAG,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;QACjC,sFAAsF;QACtF,mEAAmE;QACnE,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,EAAE,EAAE,CAAC,KAAK,CAAC,EAAE,CAAC;YAC3C,OAAO;QACT,CAAC;QAED,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,IAAI,MAAM,GAAG,CAAC,CAAC;QACf,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;YACvB,MAAM,GAAG,GAAG,GAAG,CAAC,CAAC,CAA8B,CAAC;YAChD,IAAI,CAAC,GAAG,EAAE,CAAC;gBACT,SAAS;YACX,CAAC;YACD,MAAM,UAAU,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC;YAC/B,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,UAAU,CAAC,EAAE,CAAC;gBACjC,SAAS;YACX,CAAC;YACD,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,GAAG,GAAG,UAAU,CAAC,CAAC;QAC9C,CAAC;QACD,MAAM,IAAI,CAAC,KAAK,CAAC,cAAc,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC;IACnE,CAAC;CACF;AAvRD,wCAuRC;AAED,SAAS,OAAO,CACd,MAA6B,EAC7B,IAAY,EACZ,UAAkB,EAClB,KAAa;IAEb,IAAI,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC9B,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,GAAG,IAAA,uBAAW,GAAE,CAAC;QACvB,MAAM,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAC3B,CAAC;IACD,MAAM,CAAC,IAAA,uBAAW,EAAC,UAAU,CAAC,CAAC,IAAI,KAAK,CAAC;AAC3C,CAAC;AAED,oFAAoF;AACpF,SAAS,eAAe,CAAC,GAAa,EAAE,MAAc;IACpD,MAAM,MAAM,GAAG,GAAG,CAAC,MAAM,GAAG,MAAM,CAAC;IACnC,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAChC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IACxC,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}

package/dist/LatencyStore.d.ts

@@ -0,0 +1,18 @@
+import type { Redis } from 'ioredis';
+import type { Retention } from './HistoryStore';
+export type LatencyMetric = 'runtime' | 'waittime';
+export declare const QUEUE_AGE_METRIC = "queueage";
+export declare class LatencyStore {
+    private readonly redis;
+    readonly retention: Retention;
+    constructor(opts: {
+        redis: Redis;
+        retention: Retention;
+    });
+    addSamples(queue: string, metric: LatencyMetric, hour: number, vector: number[]): Promise<void>;
+    recordQueueAge(queue: string, hour: number, ms: number): Promise<void>;
+    readRange(queue: string, metric: LatencyMetric, granularity: 'hour' | 'day', days: string[]): Promise<Record<string, number[]>>;
+    readQueueAge(queue: string, granularity: 'hour' | 'day', days: string[]): Promise<Record<string, number>>;
+    /** Epoch ms for an absolute hour index, for building response timestamps. */
+    static hourToMs(hour: number): number;
+}