@worker-manager/metrics 1.0.0

package/README.md

@@ -0,0 +1,136 @@
+# @worker-manager/metrics
+
+> Status: Beta. The API and Redis storage layout may still change in a minor release while the feature settles. It is safe to run (opt-in, and it only writes its own namespaced keys), but pin an exact version if you depend on the storage format.
+
+Opt-in long-retention historical job metrics for [Worker Manager](https://github.com/naldomadeira/worker-manager).
+
+Snapshots native BullMQ per-minute metrics into long-retention Redis buckets and exposes a
+`MetricsHistoryProvider` that feeds bull-board's history charts. Everything is opt-in: the core
+`@worker-manager/api` stays stateless.
+
+## Precondition
+
+Your BullMQ workers must have native metrics enabled, with a window large enough to survive any
+recorder downtime, for example:
+
+    new Worker(name, processor, {
+      connection,
+      metrics: { maxDataPoints: MetricsTime.ONE_WEEK },
+    });
+
+## Usage
+
+    import { MetricsRecorder, RedisMetricsHistoryProvider } from '@worker-manager/metrics';
+
+    // In your always-on worker/app process:
+    const recorder = new MetricsRecorder({
+      queues: [new BullMQAdapter(queue)],
+      connection,
+      retentionDays: 90,
+    });
+    recorder.start();
+
+    // Where you build the board:
+    createBullBoard({
+      queues,
+      serverAdapter,
+      options: { historyProvider: new RedisMetricsHistoryProvider({ connection }) },
+    });
+
+`queues` also accepts a function, resolved on every tick instead of once, which is what you want when the queue set changes while the recorder runs.
+
+Not embedding bull-board in an app of your own? This package ships inside [`@worker-manager/cli`](https://www.npmjs.com/package/@worker-manager/cli) and the `ghcr.io/naldomadeira/worker-manager` image, where `--history` registers the provider and starts a recorder in the same process. See the [CLI guide](https://naldomadeira.github.io/worker-manager/guide/cli#historical-metrics).
+
+On shutdown, call `recorder.stop()` and `provider.disconnect()`. Both only close the Redis connection if the recorder/provider opened it internally, so it's a safe no-op if you passed in your own `Redis` instance.
+
+`connection` may be ioredis options, or a `Redis` or `Cluster` instance you created. `ioredis` is a peer dependency (v5 or v6): resolve a single copy in your app, and if you reuse an existing client, pass one built from that same `ioredis`. A client from a different install (for example one created internally by a BullMQ pinned to a different ioredis major) is not recognized as a client and would be misread as options.
+
+Timestamps and buckets are UTC.
+
+## Key namespace
+
+Every key the recorder writes lives under `bull-board:metrics:`. Pass `prefix` to move it, which is how two boards share one Redis without their histories running together:
+
+    const recorder = new MetricsRecorder({ queues, connection, prefix: 'staging:metrics' });
+    const provider = new RedisMetricsHistoryProvider({ connection, prefix: 'staging:metrics' });
+
+The provider, the recorder and any `MetricsHistoryAdmin` must all be given the same prefix. A provider reading a namespace nothing writes to reports empty history rather than an error, the same way a mismatched retention quietly shortens the window.
+
+## Redis Cluster
+
+Pass a `Cluster` as `connection` and it works, with one thing worth knowing about the key layout.
+
+Each snapshot writes a queue's three tiers and the three `__global__` rollup tiers in a single `EVAL`, which is what makes the write idempotent across all resolutions at once. Redis Cluster rejects a multi-key command whose keys land in different slots, so the whole namespace has to hash to one slot. It is given a [hash tag](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags) for that: `bull-board:metrics` becomes `{bull-board:metrics}`, and a `prefix` of your own is wrapped the same way unless it already carries a `{...}` tag, in which case yours is used and you choose the slot.
+
+One slot means one master holds the history for the whole board. That is the trade for keeping the rollup consistent on write rather than recomputing it on read, and the volume is the same as the storage section below: roughly 50 MB at 200 busy queues, and one `EVAL` per queue per metric per minute.
+
+Standalone keys are untagged and unchanged, so nothing moves for an existing deployment. Nothing carries over from a standalone Redis to a cluster, since the key names differ.
+
+Latency sampling reads BullMQ's own keys through the same connection, so your queues need the hash-tagged prefix BullMQ already asks for in cluster mode (`new Queue(name, { prefix: '{bull}' })`). Without it the sampler's pipelines span slots; it swallows that error, so pass `onLatencyError` to see it.
+
+The CLI and the Docker image reach a cluster with `--cluster`, where `--history` works the same way.
+
+## Job latency
+
+Alongside the completed/failed counters, the recorder tracks two histograms per queue: wait time (`processedOn - timestamp`, how long a job sat before a worker picked it up) and run time (`finishedOn - processedOn`, how long the handler took). They diagnose different problems, so they're kept separate rather than combined into one number.
+
+Both are collected by scanning the completed and failed sorted sets (BullMQ scores them by finish time via `moveToFinished`'s `ZADD`) past a watermark on the recorder's existing tick, so no worker changes are needed and there's no precondition on `queue.getMetrics()`.
+
+The wait histogram only sees jobs that finished, so it goes quiet exactly when a queue is backed up and jobs stop finishing. A queue-age gauge (oldest job still waiting) is recorded alongside it for that reason, and the UI overlays it on the wait chart. Retries are excluded from wait time only, since `timestamp` is a job's creation but `processedOn` is its latest attempt. Percentiles are estimates bounded by bucket width; the bucket layout is fixed, not configurable, because two ranges with different layouts can't be merged into one percentile.
+
+`removeOnComplete: true` deletes jobs the instant they finish, so there's nothing left to scan; that queue will never show latency data. Deleting, cleaning, or retrying jobs by hand does the same to whatever finished since the last tick. The counter charts are unaffected, since BullMQ counts a job as it finishes and never decrements when it is removed. Latency sampling is on by default; set `latency: false` on `MetricsRecorder` to turn it off.
+
+## PostgreSQL-backed queues
+
+A BullMQ 6 queue backed by PostgreSQL records no history. Its `getMetrics()` reports `prevTS` as 0, which leaves the per-minute buffer undatable, so the counters are dropped rather than dated from the recorder's clock; the field is tracked in the backend's schema, so this may resolve upstream. Latency sampling needs BullMQ's Redis keys, which such a queue does not have, so it is skipped rather than recorded as an empty backlog. Redis-backed queues on the same board are unaffected.
+
+## Storage
+
+Each snapshot is written at three resolutions at once, each with its own retention, because they cost very different amounts:
+
+| Tier | Default retention | Size per busy day, per queue and metric |
+| --- | --- | --- |
+| Minute | 7 days | ~72 KB |
+| Hour | 90 days | ~0.3 KB |
+| Day | 90 days | ~15 bytes |
+
+At the defaults that's roughly 1.1 MB for a queue busy every minute of every day across both metrics, and far less for a bursty one. Minutes with no activity are never written, so the footprint follows how busy a queue is, not how long it has been recording.
+
+    const recorder = new MetricsRecorder({
+      queues,
+      connection,
+      retention: { minutes: 7, hours: 90, days: 90 },
+    });
+
+The minute window is the one worth tuning: it holds essentially all the bytes, and it doubles as the recorder's catch-up window after downtime. `retentionDays: N` still works and sets the hourly and daily windows, leaving the minute window at its default.
+
+Latency histograms and the queue-age gauge use a separate packed format and are measured separately, at 90 day retention:
+
+| Scenario | Measured |
+| --- | --- |
+| One queue, both histograms plus the queue-age gauge, realistic distribution | 254.5 KB |
+| Same, pathological: all 18 buckets populated heavily every hour | 574.6 KB |
+| Shared `__global__` cross-queue rollup | ~224 KB once, for the whole board |
+
+That rollup is a single shared cost, not multiplied per queue: 200 queues at typical traffic is roughly 200 × 254.5 KB, about 50 MB, plus the one shared 224 KB rollup. `sample()` takes about 9.11 ms per tick at 1000 finished jobs and about 12 Redis round trips per queue per tick, flat in job count; a subsampling cap above `maxLatencySamplesPerTick` keeps that bounded even at 10,000 jobs a tick.
+
+Retention is enforced by Redis. Day-scoped keys expire on their own TTL; the daily totals hashes are trimmed to the window as each new day rolls in.
+
+## Inspecting and clearing history
+
+    import { MetricsHistoryAdmin } from '@worker-manager/metrics';
+
+    const admin = new MetricsHistoryAdmin({ connection });        // add `prefix` if the recorder has one
+
+    await admin.stats();                          // bytes per tier and per queue, day range
+    await admin.purge();                          // delete everything
+    await admin.purge({ queue: 'mailer' });       // delete one queue
+    await admin.purge({ before: '2026-06-01' });  // delete anything older than a day
+
+Both are `SCAN`-driven and confined to this package's namespace, so they never block Redis and never touch BullMQ's own keys. On a cluster they scan every master, since `SCAN` carries no key for the client to route by. Purging a single queue also subtracts it from the cross-queue rollup. Call `admin.disconnect()` when done.
+
+`RedisMetricsHistoryProvider` exposes the same two operations to the board, which turns them into a storage panel on the Metrics history page with a confirmation before anything is deleted.
+
+## Scope
+
+The shipped Worker Manager UI reads daily rollups. `getHistory` also supports hourly granularity for custom consumers (via the core's `/api/metrics/history` endpoint), though the built-in charts don't use it.

package/dist/HistoryAdmin.d.ts

@@ -0,0 +1,134 @@
+import { type MetricsConnection } from './connection';
+export interface TierStats {
+    keys: number;
+    /** Sum of `MEMORY USAGE` over this tier's keys, in bytes. */
+    bytes: number;
+}
+export interface HistoryQueueStats {
+    /** Queue name, or `__global__` for the cross-queue rollup. */
+    queue: string;
+    keys: number;
+    bytes: number;
+    /** Recorded minute buckets, the tier that drives storage size. */
+    minutes: number;
+    /** Days covered by a minute or hour hash, ascending. */
+    days: string[];
+    /** Where this queue's bytes actually sit, so a footprint can be diagnosed. */
+    tiers: Record<HistoryTier, TierStats>;
+}
+export interface HistoryStats {
+    keys: number;
+    bytes: number;
+    minutes: number;
+    oldestDay: string | null;
+    newestDay: string | null;
+    tiers: Record<HistoryTier, TierStats>;
+    queues: HistoryQueueStats[];
+}
+export interface PurgeOptions {
+    /** Limit the purge to one queue. Omit to purge every queue plus the global rollup. */
+    queue?: string;
+    /** Only drop days strictly before this date (UTC). Omit to drop everything in scope. */
+    before?: Date | string;
+}
+export interface PurgeResult {
+    keysDeleted: number;
+    /** Day fields removed from totals hashes. */
+    fieldsDeleted: number;
+}
+export interface MetricsHistoryAdminOptions {
+    connection: MetricsConnection;
+    /** Must match the recorder's. See `MetricsRecorderOptions.prefix`. */
+    prefix?: string;
+}
+export type HistoryTier = 'minute' | 'hour' | 'day';
+interface ParsedKey {
+    queue: string;
+    metric: string;
+    tier: HistoryTier;
+    /** ISO day the key covers, `null` for the daily totals hash. */
+    day: string | null;
+}
+/**
+ * Parses the three key shapes from the right, because queue names may themselves contain
+ * colons:
+ *
+ *   <ns>:<queue>:<metric>:<day>         minute buckets
+ *   <ns>:<queue>:<metric>:hour:<day>    hourly rollup
+ *   <ns>:<queue>:<metric>:totals        daily totals
+ *
+ * The metric segment is checked against the known set, so a queue named `hour` or one
+ * ending in `:completed` still resolves correctly. Anything that doesn't fit returns null
+ * and is then reported but never deleted, so a stray key can't be destroyed by accident.
+ */
+export declare function parseHistoryKey(key: string, namespace: string): ParsedKey | null;
+/**
+ * Inspection and cleanup for the Redis keys written by `MetricsRecorder`.
+ *
+ * Every operation is confined to the recorder's namespace and driven by SCAN, so it never
+ * blocks Redis and never touches BullMQ's own keys. Deletes use UNLINK.
+ */
+export declare class MetricsHistoryAdmin {
+    private readonly redis;
+    private readonly keys;
+    private readonly ownsRedis;
+    constructor(opts: MetricsHistoryAdminOptions);
+    disconnect(): void;
+    /**
+     * Per-queue footprint of the stored history.
+     *
+     * Every key has to be measured individually, since only `MEMORY USAGE` knows what a hash
+     * really costs. Issuing those one at a time would mean a round trip per key, which at a
+     * 90-day retention across a dozen queues runs into the thousands, so the measurements go
+     * out in pipelined batches instead. Still an ops-scale call rather than a hot path: it
+     * reads the whole namespace, so it belongs behind a debug endpoint, not a poll.
+     */
+    stats(): Promise<HistoryStats>;
+    /**
+     * Size and entry count for each key, in pipelined batches so the cost is a handful of
+     * round trips rather than one per key. A key that expires between the scan and the
+     * measurement simply reads as zero rather than failing the whole call.
+     */
+    private measure;
+    /**
+     * Deletes recorded history. Purging a single queue also subtracts that queue's minutes
+     * from the global rollup, so the cross-queue chart stays correct instead of keeping the
+     * removed queue's throughput folded into it forever.
+     *
+     * That correction covers the counter metrics only. The global runtime, waittime and
+     * queueage rollups keep the purged queue's contribution until their own retention drops
+     * it: a packed bucket vector cannot be decremented field by field, and a max gauge has no
+     * record of which queue produced the maximum, so there is nothing to subtract. The
+     * per-queue keys are still deleted either way. See SUMMABLE_METRICS.
+     */
+    purge(opts?: PurgeOptions): Promise<PurgeResult>;
+    /**
+     * Removes one queue's buckets from the matching global key, so the cross-queue series
+     * reflects the queues that are left rather than keeping the removed queue folded in.
+     * Each tier is corrected from its own source key, because the tiers have independent
+     * retention and the minute hash may already be gone while the hourly one survives.
+     * Fields that drain to zero are dropped: the recorder never writes a zero bucket, so a
+     * leftover zero would read as recorded-but-idle instead of not recorded.
+     * Returns the number of global keys it deleted.
+     *
+     * Only the summable metrics are touched; the latency ones are skipped outright rather than
+     * silently producing a no-op subtraction of their packed values. See SUMMABLE_METRICS.
+     */
+    private subtractDayFromGlobal;
+    /**
+     * Same idea for the daily rollup: the global totals hash is the sum of the per-queue
+     * totals hashes, so it is corrected from those rather than re-derived from day hashes,
+     * which may already have expired. Returns the number of global fields it removed.
+     *
+     * Skips the latency metrics for the same reason as subtractDayFromGlobal.
+     */
+    private subtractTotalsFromGlobal;
+    /**
+     * SCAN over the namespace, once per master: SCAN carries no key, so a cluster client has
+     * no slot to route by and would answer from one arbitrary node. SCAN may also hand back the
+     * same key on more than one cursor iteration, which would double-count in `stats()`, so
+     * emissions are de-duped here. The set is bounded by queues x metrics x retention days.
+     */
+    private scan;
+}
+export {};

package/dist/HistoryAdmin.js

@@ -0,0 +1,345 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MetricsHistoryAdmin = void 0;
+exports.parseHistoryKey = parseHistoryKey;
+const connection_1 = require("./connection");
+const keys_1 = require("./keys");
+const SCAN_COUNT = 500;
+const BATCH = 256;
+/** Keys per pipelined `stats()` batch. Each key costs two commands. */
+const MEASURE_BATCH = 250;
+const DAY_PATTERN = /^\d{4}-\d{2}-\d{2}$/;
+const METRICS = ['completed', 'failed', 'runtime', 'waittime', 'queueage'];
+/**
+ * Metrics whose global rollup is a plain summable counter, so one queue's share can be taken
+ * back out of it with HINCRBY. The latency metrics are deliberately absent: runtime and
+ * waittime pack a whole bucket vector into a single field and queueage holds a max gauge,
+ * and neither can be corrected by a scalar decrement. See `purge`.
+ */
+const SUMMABLE_METRICS = ['completed', 'failed'];
+/**
+ * Parses the three key shapes from the right, because queue names may themselves contain
+ * colons:
+ *
+ *   <ns>:<queue>:<metric>:<day>         minute buckets
+ *   <ns>:<queue>:<metric>:hour:<day>    hourly rollup
+ *   <ns>:<queue>:<metric>:totals        daily totals
+ *
+ * The metric segment is checked against the known set, so a queue named `hour` or one
+ * ending in `:completed` still resolves correctly. Anything that doesn't fit returns null
+ * and is then reported but never deleted, so a stray key can't be destroyed by accident.
+ */
+function parseHistoryKey(key, namespace) {
+    const prefix = `${namespace}:`;
+    if (!key.startsWith(prefix)) {
+        return null;
+    }
+    const parts = key.slice(prefix.length).split(':');
+    if (parts.length < 3) {
+        return null;
+    }
+    const last = parts[parts.length - 1];
+    if (last === 'totals' && METRICS.includes(parts[parts.length - 2])) {
+        const queue = parts.slice(0, -2).join(':');
+        return queue ? { queue, metric: parts[parts.length - 2], tier: 'day', day: null } : null;
+    }
+    if (!DAY_PATTERN.test(last)) {
+        return null;
+    }
+    if (parts.length >= 4 && parts[parts.length - 2] === keys_1.HOUR_TIER) {
+        const metric = parts[parts.length - 3];
+        const queue = parts.slice(0, -3).join(':');
+        return queue && METRICS.includes(metric) ? { queue, metric, tier: 'hour', day: last } : null;
+    }
+    const metric = parts[parts.length - 2];
+    const queue = parts.slice(0, -2).join(':');
+    return queue && METRICS.includes(metric) ? { queue, metric, tier: 'minute', day: last } : null;
+}
+function emptyTiers() {
+    return {
+        minute: { keys: 0, bytes: 0 },
+        hour: { keys: 0, bytes: 0 },
+        day: { keys: 0, bytes: 0 },
+    };
+}
+/** The global rollup key mirroring a per-queue key, same tier and same day. */
+function globalKeyFor(keys, parsed) {
+    if (parsed.day === null) {
+        return keys.totals(keys_1.GLOBAL_QUEUE, parsed.metric);
+    }
+    return parsed.tier === 'hour'
+        ? keys.hour(keys_1.GLOBAL_QUEUE, parsed.metric, parsed.day)
+        : keys.day(keys_1.GLOBAL_QUEUE, parsed.metric, parsed.day);
+}
+function toDay(value) {
+    if (typeof value === 'string') {
+        if (!DAY_PATTERN.test(value)) {
+            throw new Error(`Expected a YYYY-MM-DD day or a Date, got "${value}"`);
+        }
+        return value;
+    }
+    return value.toISOString().slice(0, 10);
+}
+/**
+ * Inspection and cleanup for the Redis keys written by `MetricsRecorder`.
+ *
+ * Every operation is confined to the recorder's namespace and driven by SCAN, so it never
+ * blocks Redis and never touches BullMQ's own keys. Deletes use UNLINK.
+ */
+class MetricsHistoryAdmin {
+    constructor(opts) {
+        const { client, owned } = (0, connection_1.resolveClient)(opts.connection);
+        this.redis = client;
+        this.ownsRedis = owned;
+        this.keys = (0, keys_1.metricsKeys)((0, keys_1.resolveNamespace)(opts.prefix, (0, connection_1.isCluster)(client)));
+    }
+    disconnect() {
+        if (this.ownsRedis) {
+            this.redis.disconnect();
+        }
+    }
+    /**
+     * Per-queue footprint of the stored history.
+     *
+     * Every key has to be measured individually, since only `MEMORY USAGE` knows what a hash
+     * really costs. Issuing those one at a time would mean a round trip per key, which at a
+     * 90-day retention across a dozen queues runs into the thousands, so the measurements go
+     * out in pipelined batches instead. Still an ops-scale call rather than a hot path: it
+     * reads the whole namespace, so it belongs behind a debug endpoint, not a poll.
+     */
+    async stats() {
+        var _a;
+        const byQueue = new Map();
+        const tiers = emptyTiers();
+        let keys = 0;
+        let bytes = 0;
+        let minutes = 0;
+        let oldestDay = null;
+        let newestDay = null;
+        const found = [];
+        for await (const key of this.scan()) {
+            const parsed = parseHistoryKey(key, this.keys.namespace);
+            if (parsed) {
+                found.push({ key, parsed });
+            }
+        }
+        const measurements = await this.measure(found.map((item) => item.key));
+        for (const [index, { parsed }] of found.entries()) {
+            const { size, len } = measurements[index];
+            const entry = (_a = byQueue.get(parsed.queue)) !== null && _a !== void 0 ? _a : {
+                queue: parsed.queue,
+                keys: 0,
+                bytes: 0,
+                minutes: 0,
+                days: [],
+                tiers: emptyTiers(),
+            };
+            entry.keys += 1;
+            entry.bytes += size;
+            entry.tiers[parsed.tier].keys += 1;
+            entry.tiers[parsed.tier].bytes += size;
+            keys += 1;
+            bytes += size;
+            tiers[parsed.tier].keys += 1;
+            tiers[parsed.tier].bytes += size;
+            if (parsed.tier === 'minute') {
+                entry.minutes += len;
+                minutes += len;
+            }
+            if (parsed.day) {
+                entry.days.push(parsed.day);
+                if (oldestDay === null || parsed.day < oldestDay) {
+                    oldestDay = parsed.day;
+                }
+                if (newestDay === null || parsed.day > newestDay) {
+                    newestDay = parsed.day;
+                }
+            }
+            byQueue.set(parsed.queue, entry);
+        }
+        const queues = [...byQueue.values()].sort((a, b) => b.bytes - a.bytes);
+        for (const queue of queues) {
+            queue.days = [...new Set(queue.days)].sort();
+        }
+        return { keys, bytes, minutes, oldestDay, newestDay, tiers, queues };
+    }
+    /**
+     * Size and entry count for each key, in pipelined batches so the cost is a handful of
+     * round trips rather than one per key. A key that expires between the scan and the
+     * measurement simply reads as zero rather than failing the whole call.
+     */
+    async measure(keys) {
+        var _a, _b, _c, _d;
+        const out = [];
+        for (let i = 0; i < keys.length; i += MEASURE_BATCH) {
+            const chunk = keys.slice(i, i + MEASURE_BATCH);
+            const pipeline = this.redis.pipeline();
+            for (const key of chunk) {
+                pipeline.memory('USAGE', key);
+                pipeline.hlen(key);
+            }
+            const res = await pipeline.exec();
+            for (let j = 0; j < chunk.length; j++) {
+                out.push({
+                    size: Number((_b = (_a = res === null || res === void 0 ? void 0 : res[j * 2]) === null || _a === void 0 ? void 0 : _a[1]) !== null && _b !== void 0 ? _b : 0) || 0,
+                    len: Number((_d = (_c = res === null || res === void 0 ? void 0 : res[j * 2 + 1]) === null || _c === void 0 ? void 0 : _c[1]) !== null && _d !== void 0 ? _d : 0) || 0,
+                });
+            }
+        }
+        return out;
+    }
+    /**
+     * Deletes recorded history. Purging a single queue also subtracts that queue's minutes
+     * from the global rollup, so the cross-queue chart stays correct instead of keeping the
+     * removed queue's throughput folded into it forever.
+     *
+     * That correction covers the counter metrics only. The global runtime, waittime and
+     * queueage rollups keep the purged queue's contribution until their own retention drops
+     * it: a packed bucket vector cannot be decremented field by field, and a max gauge has no
+     * record of which queue produced the maximum, so there is nothing to subtract. The
+     * per-queue keys are still deleted either way. See SUMMABLE_METRICS.
+     */
+    async purge(opts = {}) {
+        const before = opts.before === undefined ? null : toDay(opts.before);
+        const result = { keysDeleted: 0, fieldsDeleted: 0 };
+        const dayKeys = [];
+        const totalsKeys = [];
+        for await (const key of this.scan()) {
+            const parsed = parseHistoryKey(key, this.keys.namespace);
+            if (!parsed) {
+                continue;
+            }
+            if (opts.queue !== undefined && parsed.queue !== opts.queue) {
+                continue;
+            }
+            if (parsed.day === null) {
+                totalsKeys.push({ key, parsed });
+            }
+            else if (before === null || parsed.day < before) {
+                dayKeys.push({ key, parsed });
+            }
+        }
+        // Rewriting the global rollup only makes sense when a single queue is being removed:
+        // a full purge drops the global keys outright, along with everything else.
+        const adjustGlobal = opts.queue !== undefined && opts.queue !== keys_1.GLOBAL_QUEUE;
+        for (const { key, parsed } of dayKeys) {
+            if (adjustGlobal) {
+                result.keysDeleted += await this.subtractDayFromGlobal(key, parsed);
+            }
+            result.keysDeleted += await this.redis.unlink(key);
+        }
+        for (const { key, parsed } of totalsKeys) {
+            const stale = (await this.redis.hkeys(key)).filter((day) => before === null || day < before);
+            if (adjustGlobal && stale.length > 0) {
+                const totals = await this.redis.hmget(key, ...stale);
+                result.fieldsDeleted += await this.subtractTotalsFromGlobal(parsed.metric, stale, totals);
+            }
+            if (before === null) {
+                result.keysDeleted += await this.redis.unlink(key);
+                continue;
+            }
+            for (let i = 0; i < stale.length; i += BATCH) {
+                result.fieldsDeleted += await this.redis.hdel(key, ...stale.slice(i, i + BATCH));
+            }
+            if ((await this.redis.hlen(key)) === 0) {
+                result.keysDeleted += await this.redis.unlink(key);
+            }
+        }
+        return result;
+    }
+    /**
+     * Removes one queue's buckets from the matching global key, so the cross-queue series
+     * reflects the queues that are left rather than keeping the removed queue folded in.
+     * Each tier is corrected from its own source key, because the tiers have independent
+     * retention and the minute hash may already be gone while the hourly one survives.
+     * Fields that drain to zero are dropped: the recorder never writes a zero bucket, so a
+     * leftover zero would read as recorded-but-idle instead of not recorded.
+     * Returns the number of global keys it deleted.
+     *
+     * Only the summable metrics are touched; the latency ones are skipped outright rather than
+     * silently producing a no-op subtraction of their packed values. See SUMMABLE_METRICS.
+     */
+    async subtractDayFromGlobal(key, parsed) {
+        if (parsed.day === null || !SUMMABLE_METRICS.includes(parsed.metric)) {
+            return 0;
+        }
+        const minutes = await this.redis.hgetall(key);
+        const globalDay = globalKeyFor(this.keys, parsed);
+        const pipeline = this.redis.multi();
+        const touched = [];
+        for (const field of Object.keys(minutes)) {
+            const value = Number(minutes[field]) || 0;
+            if (value === 0) {
+                continue;
+            }
+            touched.push(field);
+            pipeline.hincrby(globalDay, field, -value);
+        }
+        if (touched.length === 0) {
+            return 0;
+        }
+        const res = await pipeline.exec();
+        const drained = touched.filter((_, i) => { var _a, _b; return Number((_b = (_a = res === null || res === void 0 ? void 0 : res[i]) === null || _a === void 0 ? void 0 : _a[1]) !== null && _b !== void 0 ? _b : 0) <= 0; });
+        for (let i = 0; i < drained.length; i += BATCH) {
+            await this.redis.hdel(globalDay, ...drained.slice(i, i + BATCH));
+        }
+        return (await this.redis.hlen(globalDay)) === 0 ? await this.redis.unlink(globalDay) : 0;
+    }
+    /**
+     * Same idea for the daily rollup: the global totals hash is the sum of the per-queue
+     * totals hashes, so it is corrected from those rather than re-derived from day hashes,
+     * which may already have expired. Returns the number of global fields it removed.
+     *
+     * Skips the latency metrics for the same reason as subtractDayFromGlobal.
+     */
+    async subtractTotalsFromGlobal(metric, days, values) {
+        if (!SUMMABLE_METRICS.includes(metric)) {
+            return 0;
+        }
+        const globalTotals = this.keys.totals(keys_1.GLOBAL_QUEUE, metric);
+        const pipeline = this.redis.multi();
+        const touched = [];
+        days.forEach((day, i) => {
+            const value = Number(values[i]) || 0;
+            if (value === 0) {
+                return;
+            }
+            touched.push(day);
+            pipeline.hincrby(globalTotals, day, -value);
+        });
+        if (touched.length === 0) {
+            return 0;
+        }
+        const res = await pipeline.exec();
+        const drained = touched.filter((_, i) => { var _a, _b; return Number((_b = (_a = res === null || res === void 0 ? void 0 : res[i]) === null || _a === void 0 ? void 0 : _a[1]) !== null && _b !== void 0 ? _b : 0) <= 0; });
+        let removed = 0;
+        for (let i = 0; i < drained.length; i += BATCH) {
+            removed += await this.redis.hdel(globalTotals, ...drained.slice(i, i + BATCH));
+        }
+        return removed;
+    }
+    /**
+     * SCAN over the namespace, once per master: SCAN carries no key, so a cluster client has
+     * no slot to route by and would answer from one arbitrary node. SCAN may also hand back the
+     * same key on more than one cursor iteration, which would double-count in `stats()`, so
+     * emissions are de-duped here. The set is bounded by queues x metrics x retention days.
+     */
+    async *scan() {
+        const seen = new Set();
+        for (const target of (0, connection_1.scanTargets)(this.redis)) {
+            let cursor = '0';
+            do {
+                const [next, batch] = await target.scan(cursor, 'MATCH', this.keys.scanPattern, 'COUNT', SCAN_COUNT);
+                cursor = next;
+                for (const key of batch) {
+                    if (!seen.has(key)) {
+                        seen.add(key);
+                        yield key;
+                    }
+                }
+            } while (cursor !== '0');
+        }
+    }
+}
+exports.MetricsHistoryAdmin = MetricsHistoryAdmin;
+//# sourceMappingURL=HistoryAdmin.js.map

package/dist/HistoryAdmin.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"HistoryAdmin.js","sourceRoot":"","sources":["../src/HistoryAdmin.ts"],"names":[],"mappings":";;;AA6FA,0CA0BC;AAvHD,6CAMsB;AACtB,iCAAkG;AAElG,MAAM,UAAU,GAAG,GAAG,CAAC;AACvB,MAAM,KAAK,GAAG,GAAG,CAAC;AAClB,uEAAuE;AACvE,MAAM,aAAa,GAAG,GAAG,CAAC;AAC1B,MAAM,WAAW,GAAG,qBAAqB,CAAC;AAC1C,MAAM,OAAO,GAAG,CAAC,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAE,UAAU,CAAC,CAAC;AAC3E;;;;;GAKG;AACH,MAAM,gBAAgB,GAAG,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;AA4DjD;;;;;;;;;;;GAWG;AACH,SAAgB,eAAe,CAAC,GAAW,EAAE,SAAiB;IAC5D,MAAM,MAAM,GAAG,GAAG,SAAS,GAAG,CAAC;IAC/B,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,MAAM,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClD,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,OAAO,IAAI,CAAC;IACd,CAAC;IACD,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IAErC,IAAI,IAAI,KAAK,QAAQ,IAAI,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;QACnE,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC3F,CAAC;IACD,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAC5B,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,KAAK,CAAC,MAAM,IAAI,CAAC,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,KAAK,gBAAS,EAAE,CAAC;QAC/D,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;QACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;IAC/F,CAAC;IACD,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,KAAK,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;AACjG,CAAC;AAED,SAAS,UAAU;IACjB,OAAO;QACL,MAAM,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC7B,IAAI,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;QAC3B,GAAG,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,CAAC,EAAE;KAC3B,CAAC;AACJ,CAAC;AAED,+EAA+E;AAC/E,SAAS,YAAY,CAAC,IAAiB,EAAE,MAAiB;IACxD,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;QACxB,OAAO,IAAI,CAAC,MAAM,CAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC;IAClD,CAAC;IACD,OAAO,MAAM,CAAC,IAAI,KAAK,MAAM;QAC3B,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC;QACpD,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,mBAAY,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC;AACxD,CAAC;AAED,SAAS,KAAK,CAAC,KAAoB;IACjC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;QAC9B,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YAC7B,MAAM,IAAI,KAAK,CAAC,6CAA6C,KAAK,GAAG,CAAC,CAAC;QACzE,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC;IACD,OAAO,KAAK,CAAC,WAAW,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AAC1C,CAAC;AAED;;;;;GAKG;AACH,MAAa,mBAAmB;IAK9B,YAAY,IAAgC;QAC1C,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,IAAA,0BAAa,EAAC,IAAI,CAAC,UAAU,CAAC,CAAC;QACzD,IAAI,CAAC,KAAK,GAAG,MAAM,CAAC;QACpB,IAAI,CAAC,SAAS,GAAG,KAAK,CAAC;QACvB,IAAI,CAAC,IAAI,GAAG,IAAA,kBAAW,EAAC,IAAA,uBAAgB,EAAC,IAAI,CAAC,MAAM,EAAE,IAAA,sBAAS,EAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IAC5E,CAAC;IAED,UAAU;QACR,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,CAAC;QAC1B,CAAC;IACH,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,KAAK;;QACT,MAAM,OAAO,GAAG,IAAI,GAAG,EAA6B,CAAC;QACrD,MAAM,KAAK,GAAG,UAAU,EAAE,CAAC;QAC3B,IAAI,IAAI,GAAG,CAAC,CAAC;QACb,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,IAAI,SAAS,GAAkB,IAAI,CAAC;QACpC,IAAI,SAAS,GAAkB,IAAI,CAAC;QAEpC,MAAM,KAAK,GAAyC,EAAE,CAAC;QACvD,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;YACzD,IAAI,MAAM,EAAE,CAAC;gBACX,KAAK,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAC9B,CAAC;QACH,CAAC;QACD,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;QAEvE,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,MAAM,EAAE,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;YAClD,MAAM,EAAE,IAAI,EAAE,GAAG,EAAE,GAAG,YAAY,CAAC,KAAK,CAAC,CAAC;YAE1C,MAAM,KAAK,GAAG,MAAA,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,mCAAI;gBACzC,KAAK,EAAE,MAAM,CAAC,KAAK;gBACnB,IAAI,EAAE,CAAC;gBACP,KAAK,EAAE,CAAC;gBACR,OAAO,EAAE,CAAC;gBACV,IAAI,EAAE,EAAE;gBACR,KAAK,EAAE,UAAU,EAAE;aACpB,CAAC;YACF,KAAK,CAAC,IAAI,IAAI,CAAC,CAAC;YAChB,KAAK,CAAC,KAAK,IAAI,IAAI,CAAC;YACpB,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YACnC,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YACvC,IAAI,IAAI,CAAC,CAAC;YACV,KAAK,IAAI,IAAI,CAAC;YACd,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC;YAC7B,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC;YAEjC,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC7B,KAAK,CAAC,OAAO,IAAI,GAAG,CAAC;gBACrB,OAAO,IAAI,GAAG,CAAC;YACjB,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,EAAE,CAAC;gBACf,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBAC5B,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;gBACD,IAAI,SAAS,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,SAAS,EAAE,CAAC;oBACjD,SAAS,GAAG,MAAM,CAAC,GAAG,CAAC;gBACzB,CAAC;YACH,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;QACnC,CAAC;QAED,MAAM,MAAM,GAAG,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACvE,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;YAC3B,KAAK,CAAC,IAAI,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAC/C,CAAC;QACD,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC;IACvE,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,OAAO,CAAC,IAAc;;QAClC,MAAM,GAAG,GAAoC,EAAE,CAAC;QAChD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,IAAI,aAAa,EAAE,CAAC;YACpD,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,aAAa,CAAC,CAAC;YAC/C,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvC,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;gBACxB,QAAQ,CAAC,MAAM,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;gBAC9B,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACrB,CAAC;YACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;YAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;gBACtC,GAAG,CAAC,IAAI,CAAC;oBACP,IAAI,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;oBACzC,GAAG,EAAE,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC;iBAC7C,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QACD,OAAO,GAAG,CAAC;IACb,CAAC;IAED;;;;;;;;;;OAUG;IACH,KAAK,CAAC,KAAK,CAAC,OAAqB,EAAE;QACjC,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QACrE,MAAM,MAAM,GAAgB,EAAE,WAAW,EAAE,CAAC,EAAE,aAAa,EAAE,CAAC,EAAE,CAAC;QACjE,MAAM,OAAO,GAAyC,EAAE,CAAC;QACzD,MAAM,UAAU,GAAyC,EAAE,CAAC;QAE5D,IAAI,KAAK,EAAE,MAAM,GAAG,IAAI,IAAI,CAAC,IAAI,EAAE,EAAE,CAAC;YACpC,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,EAAE,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;YACzD,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,SAAS;YACX,CAAC;YACD,IAAI,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,MAAM,CAAC,KAAK,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC;gBAC5D,SAAS;YACX,CAAC;YACD,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,EAAE,CAAC;gBACxB,UAAU,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YACnC,CAAC;iBAAM,IAAI,MAAM,KAAK,IAAI,IAAI,MAAM,CAAC,GAAG,GAAG,MAAM,EAAE,CAAC;gBAClD,OAAO,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;YAChC,CAAC;QACH,CAAC;QAED,qFAAqF;QACrF,2EAA2E;QAC3E,MAAM,YAAY,GAAG,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,IAAI,CAAC,KAAK,KAAK,mBAAY,CAAC;QAE7E,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,OAAO,EAAE,CAAC;YACtC,IAAI,YAAY,EAAE,CAAC;gBACjB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,qBAAqB,CAAC,GAAG,EAAE,MAAM,CAAC,CAAC;YACtE,CAAC;YACD,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACrD,CAAC;QAED,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,UAAU,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,MAAM,KAAK,IAAI,IAAI,GAAG,GAAG,MAAM,CAAC,CAAC;YAC7F,IAAI,YAAY,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBACrC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,CAAC;gBACrD,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,wBAAwB,CAAC,MAAM,CAAC,MAAM,EAAE,KAAK,EAAE,MAAM,CAAC,CAAC;YAC5F,CAAC;YACD,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;gBACpB,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;gBACnD,SAAS;YACX,CAAC;YACD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;gBAC7C,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;YACnF,CAAC;YACD,IAAI,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,EAAE,CAAC;gBACvC,MAAM,CAAC,WAAW,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;YACrD,CAAC;QACH,CAAC;QAED,OAAO,MAAM,CAAC;IAChB,CAAC;IAED;;;;;;;;;;;OAWG;IACK,KAAK,CAAC,qBAAqB,CAAC,GAAW,EAAE,MAAiB;QAChE,IAAI,MAAM,CAAC,GAAG,KAAK,IAAI,IAAI,CAAC,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC;YACrE,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC9C,MAAM,SAAS,GAAG,YAAY,CAAC,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QAClD,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;YACzC,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC;YAC1C,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,SAAS;YACX,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACpB,QAAQ,CAAC,OAAO,CAAC,SAAS,EAAE,KAAK,EAAE,CAAC,KAAK,CAAC,CAAC;QAC7C,CAAC;QACD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACnE,CAAC;QACD,OAAO,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3F,CAAC;IAED;;;;;;OAMG;IACK,KAAK,CAAC,wBAAwB,CACpC,MAAc,EACd,IAAc,EACd,MAAyB;QAEzB,IAAI,CAAC,gBAAgB,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YACvC,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,YAAY,GAAG,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,mBAAY,EAAE,MAAM,CAAC,CAAC;QAC5D,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACpC,MAAM,OAAO,GAAa,EAAE,CAAC;QAC7B,IAAI,CAAC,OAAO,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE;YACtB,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;YACrC,IAAI,KAAK,KAAK,CAAC,EAAE,CAAC;gBAChB,OAAO;YACT,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YAClB,QAAQ,CAAC,OAAO,CAAC,YAAY,EAAE,GAAG,EAAE,CAAC,KAAK,CAAC,CAAC;QAC9C,CAAC,CAAC,CAAC;QACH,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,GAAG,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QAClC,MAAM,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,eAAC,OAAA,MAAM,CAAC,MAAA,MAAA,GAAG,aAAH,GAAG,uBAAH,GAAG,CAAG,CAAC,CAAC,0CAAG,CAAC,CAAC,mCAAI,CAAC,CAAC,IAAI,CAAC,CAAA,EAAA,CAAC,CAAC;QAC1E,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,IAAI,KAAK,EAAE,CAAC;YAC/C,OAAO,IAAI,MAAM,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,YAAY,EAAE,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC;QACjF,CAAC;QACD,OAAO,OAAO,CAAC;IACjB,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,CAAC,IAAI;QACjB,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;QAC/B,KAAK,MAAM,MAAM,IAAI,IAAA,wBAAW,EAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YAC7C,IAAI,MAAM,GAAG,GAAG,CAAC;YACjB,GAAG,CAAC;gBACF,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,GAAG,MAAM,MAAM,CAAC,IAAI,CACrC,MAAM,EACN,OAAO,EACP,IAAI,CAAC,IAAI,CAAC,WAAW,EACrB,OAAO,EACP,UAAU,CACX,CAAC;gBACF,MAAM,GAAG,IAAI,CAAC;gBACd,KAAK,MAAM,GAAG,IAAI,KAAK,EAAE,CAAC;oBACxB,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;wBACnB,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;wBACd,MAAM,GAAG,CAAC;oBACZ,CAAC;gBACH,CAAC;YACH,CAAC,QAAQ,MAAM,KAAK,GAAG,EAAE;QAC3B,CAAC;IACH,CAAC;CACF;AA1RD,kDA0RC"}