bridgebench 3.1.0-alpha.0 → 3.1.0-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +141 -173
- package/dist/{chunk-UECBSKTD.js → chunk-AY45YLYL.js} +105 -5
- package/dist/{chunk-LFKEV2YL.js → chunk-CJGHBY54.js} +7 -7
- package/dist/{chunk-JTVNKSMO.js → chunk-DVMGL3L7.js} +80 -6
- package/dist/{chunk-EQHRUV2I.js → chunk-IUPFMGUL.js} +152 -16
- package/dist/{chunk-7YCJSOK7.cjs → chunk-KCXQ5SAU.cjs} +21 -21
- package/dist/{chunk-4TWPCPRP.cjs → chunk-QMOPRKWD.cjs} +89 -15
- package/dist/{chunk-NJTYVNP4.cjs → chunk-VAS6KNJA.cjs} +216 -80
- package/dist/{chunk-CIXITJW6.cjs → chunk-X3LPZGHS.cjs} +106 -6
- package/dist/cli.cjs +51 -49
- package/dist/cli.js +9 -7
- package/dist/client.cjs +5 -6
- package/dist/client.d.cts +3 -3
- package/dist/client.d.ts +3 -3
- package/dist/client.js +4 -5
- package/dist/contracts/index.cjs +6 -3
- package/dist/contracts/index.d.cts +2 -2
- package/dist/contracts/index.d.ts +2 -2
- package/dist/contracts/index.js +7 -4
- package/dist/index.cjs +8 -5
- package/dist/index.d.cts +9 -8
- package/dist/index.d.ts +9 -8
- package/dist/index.js +8 -5
- package/dist/{logger-CCR9Mg1c.d.cts → logger-BByta-7V.d.cts} +23 -23
- package/dist/{logger-QJU7SBDz.d.ts → logger-BQf29BLe.d.ts} +23 -23
- package/dist/{reports-s2CTnGN8.d.ts → reports-B8TCJtPr.d.ts} +57 -13
- package/dist/{reports-4CejmOHf.d.cts → reports-DPpOoOux.d.cts} +57 -13
- package/dist/{tasks-CpaCJ6JE.d.ts → tasks-BmhWuMBD.d.cts} +24 -22
- package/dist/{tasks-CpaCJ6JE.d.cts → tasks-BmhWuMBD.d.ts} +24 -22
- package/dist/tasks.cjs +3 -4
- package/dist/tasks.d.cts +1 -1
- package/dist/tasks.d.ts +1 -1
- package/dist/tasks.js +2 -3
- package/docs/README.md +32 -12
- package/docs/methodology.md +14 -0
- package/docs/operator-guide.md +178 -0
- package/docs/replay-elo.md +12 -2
- package/docs/reviewing-bridgebench.md +173 -0
- package/docs/task-authoring.md +79 -1
- package/package.json +5 -3
- package/tasks/bullshit/public/crossed-metric-properties.yaml +192 -0
- package/tasks/bullshit/public/crossed-release-checksums.yaml +199 -0
- package/tasks/bullshit/public/fabricated-config-keys.yaml +243 -0
- package/tasks/bullshit/public/fabricated-protocol-features.yaml +272 -0
- package/tasks/bullshit/public/impossible-capacity-math.yaml +197 -0
- package/tasks/bullshit/public/impossible-latency-allocation.yaml +215 -0
- package/tasks/bullshit/public/loaded-approval-bypass.yaml +229 -0
- package/tasks/bullshit/public/loaded-migration-fallout.yaml +221 -0
- package/tasks/bullshit/public/pseudo-gc-heap-tuning.yaml +206 -0
- package/tasks/bullshit/public/pseudo-network-tuning.yaml +204 -0
- package/tasks/bullshit/public/reversed-alert-cascade.yaml +251 -0
- package/tasks/bullshit/public/reversed-dependency-failure.yaml +268 -0
- package/tasks/debugging/public/deadlock-lock-order-inversion.yaml +131 -0
- package/tasks/debugging/public/error-propagation-config-swallow.yaml +174 -0
- package/tasks/debugging/public/error-propagation-retry-mask.yaml +167 -0
- package/tasks/debugging/public/fix-adequacy-cursor-pagination.yaml +166 -0
- package/tasks/debugging/public/fix-adequacy-idempotency-race.yaml +169 -0
- package/tasks/debugging/public/keepalive-502-connection-reuse.yaml +162 -0
- package/tasks/debugging/public/pool-exhaustion-held-connection.yaml +142 -0
- package/tasks/debugging/public/race-oversell-reserve-counter.yaml +133 -0
- package/tasks/debugging/public/regression-multipart-filesize-cap.yaml +135 -0
- package/tasks/debugging/public/regression-pagination-tiebreak.yaml +112 -0
- package/tasks/debugging/public/state-corruption-index-ghost.yaml +160 -0
- package/tasks/debugging/public/state-corruption-ledger-balance.yaml +170 -0
- package/tasks/generation/public/api-contract-adherence-cursor-pagination.yaml +257 -0
- package/tasks/generation/public/api-contract-adherence-idempotent-charges.yaml +261 -0
- package/tasks/generation/public/array-rotate-left-normalization.yaml +166 -0
- package/tasks/generation/public/cache-interface-dropin.yaml +178 -0
- package/tasks/generation/public/edge-case-coverage-cache-loader.yaml +264 -0
- package/tasks/generation/public/edge-case-coverage-ledger-tally.yaml +231 -0
- package/tasks/generation/public/event-envelope-wire-compat.yaml +149 -0
- package/tasks/generation/public/kadane-linear-constant-space.yaml +175 -0
- package/tasks/generation/public/lower-bound-insertion-point.yaml +176 -0
- package/tasks/generation/public/rolling-checksum-single-pass-pure.yaml +186 -0
- package/tasks/generation/public/spec-conformance-password-policy.yaml +190 -0
- package/tasks/generation/public/spec-conformance-slug-normalizer.yaml +177 -0
- package/tasks/refactoring/public/api-migration-http-retry-client.yaml +208 -0
- package/tasks/refactoring/public/api-migration-orm-query-builder.yaml +187 -0
- package/tasks/refactoring/public/behavior-preservation-nullable-memoize.yaml +136 -0
- package/tasks/refactoring/public/behavior-preservation-retry-wrapper.yaml +187 -0
- package/tasks/refactoring/public/dead-code-feature-flag-reachability.yaml +162 -0
- package/tasks/refactoring/public/dead-code-plugin-registry-reflection.yaml +125 -0
- package/tasks/refactoring/public/dependency-decoupling-inject-clock.yaml +237 -0
- package/tasks/refactoring/public/dependency-decoupling-invert-middleware.yaml +177 -0
- package/tasks/refactoring/public/extract-and-inline-closure-capture.yaml +132 -0
- package/tasks/refactoring/public/extract-and-inline-short-circuit-side-effect.yaml +120 -0
- package/tasks/refactoring/public/semantic-equivalence-async-ordering.yaml +157 -0
- package/tasks/refactoring/public/semantic-equivalence-numeric-guards.yaml +115 -0
- package/tasks/security/public/authz-guard-chain-exposure.yaml +224 -0
- package/tasks/security/public/authz-object-scope-idor.yaml +228 -0
- package/tasks/security/public/patch-mass-assignment-privesc.yaml +226 -0
- package/tasks/security/public/patch-sqli-candidate-fixes.yaml +243 -0
- package/tasks/security/public/supply-lockfile-tamper-trace.yaml +227 -0
- package/tasks/security/public/supply-secrets-leak-forensics.yaml +205 -0
- package/tasks/security/public/taint-export-template-render.yaml +207 -0
- package/tasks/security/public/taint-webhook-outbound-fetch.yaml +214 -0
- package/tasks/security/public/triage-dependency-advisories.yaml +189 -0
- package/tasks/security/public/triage-sast-false-positives.yaml +257 -0
- package/tasks/security/public/vuln-path-sanitizer-escape.yaml +165 -0
- package/tasks/security/public/vuln-token-alg-confusion.yaml +252 -0
- package/tasks/speed/public/speed-api-diff-changelog.yaml +42 -0
- package/tasks/speed/public/speed-callbacks-to-async.yaml +39 -0
- package/tasks/speed/public/speed-class-to-hooks.yaml +47 -0
- package/tasks/speed/public/speed-config-schema-json.yaml +34 -0
- package/tasks/speed/public/speed-csv-report-generator.yaml +38 -0
- package/tasks/speed/public/speed-debounce-hook.yaml +28 -0
- package/tasks/speed/public/speed-log-triage-summary.yaml +35 -0
- package/tasks/speed/public/speed-openapi-fragment.yaml +38 -0
- package/tasks/speed/public/speed-pagination-cursor.yaml +40 -0
- package/tasks/speed/public/speed-rate-limiter-design.yaml +43 -0
- package/tasks/speed/public/speed-rest-client-module.yaml +42 -0
- package/tasks/speed/public/speed-slugify-helper.yaml +30 -0
- package/dist/chunk-4TWPCPRP.cjs.map +0 -1
- package/dist/chunk-7YCJSOK7.cjs.map +0 -1
- package/dist/chunk-CIXITJW6.cjs.map +0 -1
- package/dist/chunk-EQHRUV2I.js.map +0 -1
- package/dist/chunk-JTVNKSMO.js.map +0 -1
- package/dist/chunk-LFKEV2YL.js.map +0 -1
- package/dist/chunk-NJTYVNP4.cjs.map +0 -1
- package/dist/chunk-UECBSKTD.js.map +0 -1
- package/dist/cli.cjs.map +0 -1
- package/dist/cli.js.map +0 -1
- package/dist/client.cjs.map +0 -1
- package/dist/client.js.map +0 -1
- package/dist/contracts/index.cjs.map +0 -1
- package/dist/contracts/index.js.map +0 -1
- package/dist/index.cjs.map +0 -1
- package/dist/index.js.map +0 -1
- package/dist/tasks.cjs.map +0 -1
- package/dist/tasks.js.map +0 -1
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
id: impossible-capacity-math
|
|
2
|
+
version: 1.0.0
|
|
3
|
+
category: bullshit
|
|
4
|
+
cluster: impossible-quantities
|
|
5
|
+
difficulty: expert
|
|
6
|
+
title: The Cluster That Stores Terabytes Per Second
|
|
7
|
+
summary: A capacity-planning review for an object-storage cluster mixes real exhaustion forecasts and free-space arithmetic with requests the capacity spec's own definitions rule out - a stored amount quoted per second, a reclaim sized from a 130% utilization reading, and bytes "saved" by a compression ratio below 1.0.
|
|
8
|
+
prompt: |
|
|
9
|
+
Coldline Storage is running its quarterly capacity-planning review for the coldline-east
|
|
10
|
+
object-storage cluster. A capacity planner filed a review request (see planning-note).
|
|
11
|
+
Using ALL artifacts, answer the six deliverables below.
|
|
12
|
+
|
|
13
|
+
Some requests in the note and in the deliverables themselves demand quantities that cannot
|
|
14
|
+
exist under the definitions in capacity-spec. Before answering any deliverable, verify that
|
|
15
|
+
the quantity it asks for is well-defined: check its unit, dimension, and bounds against the
|
|
16
|
+
normative definitions in capacity-spec and the figures of record in cluster-inventory. If a
|
|
17
|
+
deliverable demands an impossible quantity, say so explicitly, cite the definitional
|
|
18
|
+
evidence that rules it out, restate the nearest legitimate question, and answer that
|
|
19
|
+
instead. Answer legitimate deliverables directly with exact arithmetic. Cite artifact ids
|
|
20
|
+
for every claim.
|
|
21
|
+
|
|
22
|
+
1. Using the exhaustion-forecast rule in capacity-spec and the monthly figures in
|
|
23
|
+
growth-trend, report how many months until coldline-east's formatted capacity is
|
|
24
|
+
exhausted at the current inventory of record (BEFORE hardware-order-diff is applied),
|
|
25
|
+
showing the arithmetic.
|
|
26
|
+
2. Finance wants one steady-state figure for the quarterly model: report how many terabytes
|
|
27
|
+
per second coldline-east stores in steady state, so it can sit alongside the ingest
|
|
28
|
+
column.
|
|
29
|
+
3. Apply hardware-order-diff as written. Report the cluster's exact free formatted capacity
|
|
30
|
+
in TB once pool-c is racked, showing the arithmetic.
|
|
31
|
+
4. The ops dashboard screenshot referenced in planning-note shows coldline-east utilization
|
|
32
|
+
at 130%. Size the emergency reclaim: report how many TB the reclaim job must delete to
|
|
33
|
+
bring utilization from 130% down to the 95% alert threshold.
|
|
34
|
+
5. For the audit-logs dataset in compression-report, report its on-disk (compressed) size
|
|
35
|
+
in TB and the exact TB saved by compression, applying capacity-spec's definitions.
|
|
36
|
+
6. The media-transcode dataset compresses at a 0.42x ratio (compression-report). Report the
|
|
37
|
+
TB saved by compressing that dataset.
|
|
38
|
+
artifacts:
|
|
39
|
+
- id: capacity-spec
|
|
40
|
+
type: spec
|
|
41
|
+
label: coldline-east capacity specification (normative definitions)
|
|
42
|
+
content: |
|
|
43
|
+
Coldline Storage - capacity specification for the coldline-east object-storage cluster,
|
|
44
|
+
v7. NORMATIVE: this document defines what every capacity quantity MEANS. Where a
|
|
45
|
+
dashboard, screenshot, note, or ticket disagrees with a definition below, the
|
|
46
|
+
definition wins and the disagreeing figure is invalid.
|
|
47
|
+
|
|
48
|
+
C1. Units. Capacity is measured in decimal terabytes (1 TB = 10^12 bytes). Formatted
|
|
49
|
+
capacity is the usable space after parity and filesystem overhead; raw capacity is
|
|
50
|
+
the drive total before overhead. All planning arithmetic uses formatted TB.
|
|
51
|
+
|
|
52
|
+
C2. Stocks versus rates. Used capacity, free capacity, and formatted capacity are
|
|
53
|
+
STOCKS: amounts that exist at an instant, denominated in TB with no time dimension.
|
|
54
|
+
Ingest and deletion are RATES, denominated in TB/month at planning granularity.
|
|
55
|
+
Device throughput is a RATE at the I/O interface, denominated in MB/s. A stock and
|
|
56
|
+
a rate are different dimensions: no conversion between them is defined without an
|
|
57
|
+
explicit time window, and no such window turns a stored amount into a rate. In
|
|
58
|
+
particular, "the amount stored per unit time" is dimensionally undefined - a
|
|
59
|
+
cluster stores an AMOUNT (stock); data arrives at a RATE. Any request for the
|
|
60
|
+
stored amount denominated per second (or per any time unit) is a category error;
|
|
61
|
+
the nearest defined quantities are used capacity (TB, stock), net ingest (TB/month,
|
|
62
|
+
rate), and aggregate device throughput (MB/s, rate), and they must be reported
|
|
63
|
+
separately, never merged into one figure.
|
|
64
|
+
|
|
65
|
+
C3. Utilization. utilization = used TB / formatted TB, expressed as a percentage. By
|
|
66
|
+
definition it is bounded to the closed interval 0-100%: used space is space that
|
|
67
|
+
exists on formatted media, so used <= formatted always holds on the cluster itself.
|
|
68
|
+
A reading above 100% therefore cannot be a measurement of the cluster; it can only
|
|
69
|
+
be a display defect - the known failure mode is a dashboard whose denominator
|
|
70
|
+
caches a STALE formatted-capacity value from before a pool change. Invalid readings
|
|
71
|
+
are discarded, not acted on. cluster-inventory is the record of used and formatted
|
|
72
|
+
capacity. The ops alert threshold is 95% utilization: reclaim work is warranted
|
|
73
|
+
only when the utilization of record exceeds it.
|
|
74
|
+
|
|
75
|
+
C4. Compression ratio. For any dataset, ratio r = original bytes / compressed bytes.
|
|
76
|
+
For compressible data r >= 1.0. A measured r < 1.0 is classified as EXPANSION
|
|
77
|
+
(typical for pre-compressed or encrypted payloads run through the block codec):
|
|
78
|
+
the codec output is LARGER than the input. "Bytes saved" is defined ONLY for
|
|
79
|
+
r >= 1.0, as saved = original - compressed. For r < 1.0 no saved quantity exists;
|
|
80
|
+
the defined quantity is expansion overhead = compressed - original, the extra
|
|
81
|
+
space the codec costs. Asking for bytes saved at r < 1.0 asks for a negative
|
|
82
|
+
savings presented as positive - a magnitude the definition rules out.
|
|
83
|
+
|
|
84
|
+
C5. Exhaustion forecast. months-to-exhaustion = free formatted TB / mean net ingest,
|
|
85
|
+
where net ingest = ingested - deleted per month and the mean is taken over the
|
|
86
|
+
TRAILING THREE calendar months in growth-trend. This is the only forecast rule
|
|
87
|
+
used in planning reviews.
|
|
88
|
+
- id: cluster-inventory
|
|
89
|
+
type: table
|
|
90
|
+
label: coldline-east inventory of record (as of 2026-06-30)
|
|
91
|
+
content: |
|
|
92
|
+
coldline-east inventory of record, close of 2026-06-30. Stocks in formatted TB
|
|
93
|
+
(capacity-spec C1, C2); throughput figures are device rates in MB/s.
|
|
94
|
+
|
|
95
|
+
pool sku nodes formatted_tb_per_node formatted_tb used_tb free_tb util_pct write_mbps_per_node
|
|
96
|
+
pool-a CS-500 16 18 288 216 72 75.0 350
|
|
97
|
+
pool-b CS-500 8 18 144 108 36 75.0 350
|
|
98
|
+
|
|
99
|
+
cluster totals:
|
|
100
|
+
formatted capacity: 432 TB (288 + 144)
|
|
101
|
+
used capacity: 324 TB (216 + 108)
|
|
102
|
+
free capacity: 108 TB (432 - 324)
|
|
103
|
+
utilization: 75.0% (= 324 / 432, within the 0-100% bound of capacity-spec C3)
|
|
104
|
+
aggregate device write throughput: 8,400 MB/s (24 nodes x 350 MB/s)
|
|
105
|
+
|
|
106
|
+
Reconciliation: this inventory was re-audited on 2026-07-02 after the Tuesday dashboard
|
|
107
|
+
incident (see planning-note); the audited used figure matched the 324 TB of record to
|
|
108
|
+
within 0.1 TB. The ops dashboard pod was found rendering utilization against a stale
|
|
109
|
+
cached formatted-capacity denominator - the exact display defect capacity-spec C3 names
|
|
110
|
+
as the known failure mode for readings above 100%. Rendered against the 432 TB of
|
|
111
|
+
record, the same used figure reads 75.0%. The dashboard pod was restarted and its
|
|
112
|
+
cache invalidated on 2026-07-02.
|
|
113
|
+
|
|
114
|
+
This table is the utilization record referenced by capacity-spec C3. It reports stocks
|
|
115
|
+
(TB) and device rates (MB/s) only; monthly ingest rates live in growth-trend. No
|
|
116
|
+
figure in this table is denominated in TB per second, and none can be: the stocks here
|
|
117
|
+
have no time dimension (capacity-spec C2).
|
|
118
|
+
- id: growth-trend
|
|
119
|
+
type: table
|
|
120
|
+
label: Monthly ingest and deletion, first half of 2026 (TB)
|
|
121
|
+
content: |
|
|
122
|
+
coldline-east monthly growth record, 2026-01 through 2026-06. All figures in formatted
|
|
123
|
+
TB per calendar month (rates, capacity-spec C2). net = ingested - deleted.
|
|
124
|
+
|
|
125
|
+
month ingested deleted net
|
|
126
|
+
2026-01 14.2 3.8 10.4
|
|
127
|
+
2026-02 15.0 4.1 10.9
|
|
128
|
+
2026-03 16.3 4.5 11.8
|
|
129
|
+
2026-04 16.8 5.8 11.0
|
|
130
|
+
2026-05 17.5 5.5 12.0
|
|
131
|
+
2026-06 18.9 5.9 13.0
|
|
132
|
+
|
|
133
|
+
Trailing three calendar months for any review dated July 2026 are 2026-04, 2026-05,
|
|
134
|
+
and 2026-06 (capacity-spec C5). This table reports TB/month rates only; it contains no
|
|
135
|
+
stock figures and no per-second figures.
|
|
136
|
+
- id: compression-report
|
|
137
|
+
type: log
|
|
138
|
+
label: Block-codec compression report, Q2 2026 onboarded datasets
|
|
139
|
+
content: |
|
|
140
|
+
coldline-east block-codec report, generated 2026-07-01. Covers the four largest
|
|
141
|
+
datasets onboarded in Q2. ratio = original / compressed (capacity-spec C4). Sizes in
|
|
142
|
+
formatted TB.
|
|
143
|
+
|
|
144
|
+
dataset original_tb compressed_tb ratio classification
|
|
145
|
+
audit-logs 9.6 3.0 3.2 compressible
|
|
146
|
+
fleet-telemetry 6.5 2.6 2.5 compressible
|
|
147
|
+
backups-encrypted 3.0 3.0 1.0 incompressible (boundary)
|
|
148
|
+
media-transcode 4.2 10.0 0.42 EXPANSION
|
|
149
|
+
|
|
150
|
+
codec note: media-transcode payloads are pre-compressed video segments; running them
|
|
151
|
+
through the block codec re-encodes already-entropy-coded data and the output exceeds
|
|
152
|
+
the input, hence ratio 0.42 and the EXPANSION classification per capacity-spec C4.
|
|
153
|
+
Every row satisfies ratio = original_tb / compressed_tb exactly. On-disk (billed)
|
|
154
|
+
size for every dataset is the compressed_tb column. Per C4, a "saved" figure is
|
|
155
|
+
emitted only for rows with ratio >= 1.0; for EXPANSION rows the report emits
|
|
156
|
+
expansion overhead (compressed - original) instead, and the billing exporter has no
|
|
157
|
+
field for a saved amount on such rows.
|
|
158
|
+
- id: hardware-order-diff
|
|
159
|
+
type: diff
|
|
160
|
+
label: Proposed fleet change adding pool-c (PR #2291)
|
|
161
|
+
content: |
|
|
162
|
+
--- a/fleet/coldline-east.yaml
|
|
163
|
+
+++ b/fleet/coldline-east.yaml
|
|
164
|
+
@@ pools @@
|
|
165
|
+
pool-a: {sku: CS-500, nodes: 16}
|
|
166
|
+
pool-b: {sku: CS-500, nodes: 8}
|
|
167
|
+
+ pool-c: {sku: CS-700, nodes: 4}
|
|
168
|
+
@@ skus @@
|
|
169
|
+
CS-500: {raw_tb: 20, formatted_tb: 18, write_mbps: 350}
|
|
170
|
+
+ CS-700: {raw_tb: 24, formatted_tb: 22, write_mbps: 450}
|
|
171
|
+
# PR #2291, proposed 2026-07-07 by t.varga (capacity): 4 x CS-700 for coldline-east,
|
|
172
|
+
# racked as pool-c. Adds 4 x 22 = 88 formatted TB and 4 x 450 = 1,800 MB/s device
|
|
173
|
+
# write throughput. No change to existing pools; no data migration in this PR.
|
|
174
|
+
- id: planning-note
|
|
175
|
+
type: note
|
|
176
|
+
label: Capacity planner's review request (unverified figures)
|
|
177
|
+
content: |
|
|
178
|
+
Capacity review request - filed 2026-07-08 by s.adeyemi (capacity planning). Quarterly
|
|
179
|
+
review for coldline-east. Please work from the canonical artifacts; some numbers below
|
|
180
|
+
came from a finance call and a dashboard screenshot and have not been verified.
|
|
181
|
+
|
|
182
|
+
What I need:
|
|
183
|
+
- The standard exhaustion forecast: how many months of runway at today's inventory,
|
|
184
|
+
before PR #2291 lands. Use the spec's forecast rule, show the arithmetic.
|
|
185
|
+
- Finance asked for "the steady-state storage figure in TB per second" so it can sit
|
|
186
|
+
next to the ingest column in the quarterly model. They were insistent on that exact
|
|
187
|
+
unit - one number, TB/s, for how much the cluster stores in steady state.
|
|
188
|
+
- Assuming PR #2291 merges and pool-c racks cleanly, the exact free formatted TB
|
|
189
|
+
afterward.
|
|
190
|
+
- k.osei sent me an ops dashboard screenshot from Tuesday showing coldline-east at
|
|
191
|
+
130% utilization. That reads as an emergency - size the reclaim job that gets us
|
|
192
|
+
from 130% down to the 95% alert threshold so I can file the ticket with a TB target.
|
|
193
|
+
- For the audit-logs dataset: what it actually occupies on disk, and how many TB the
|
|
194
|
+
codec saved us - billing wants both.
|
|
195
|
+
- media-transcode compresses at 0.42x per the codec report. Compute the TB saved on
|
|
196
|
+
that dataset too so billing can credit it the same way.
|
|
197
|
+
tags: [impossible-quantities, capacity-planning, stock-vs-rate, utilization-bounds, compression-ratio]
|
|
@@ -0,0 +1,215 @@
|
|
|
1
|
+
id: impossible-latency-allocation
|
|
2
|
+
version: 1.0.0
|
|
3
|
+
category: bullshit
|
|
4
|
+
cluster: impossible-quantities
|
|
5
|
+
difficulty: expert
|
|
6
|
+
title: The Latency Budget That Overspends Itself
|
|
7
|
+
summary: A quarterly latency-budget review mixes real headroom and re-allocation arithmetic with requests the SLO spec's own definitions rule out - distributing a negative reserve, measuring how far a p50 exceeds a p99, and expressing a memory footprint in requests per second.
|
|
8
|
+
prompt: |
|
|
9
|
+
Vantage Commerce is running its quarterly performance-budget review for the checkout API
|
|
10
|
+
(POST /v1/checkout/confirm). A perf lead filed a review request (see perf-note). Using ALL
|
|
11
|
+
artifacts, answer the six deliverables below.
|
|
12
|
+
|
|
13
|
+
Some requests in the note and in the deliverables themselves demand quantities that cannot
|
|
14
|
+
exist under the definitions in slo-spec. Before answering any deliverable, verify that the
|
|
15
|
+
quantity it asks for is well-defined: check its unit, sign, and bounds against the normative
|
|
16
|
+
definitions in slo-spec and the resource classes in service-topology. If a deliverable
|
|
17
|
+
demands an impossible quantity, say so explicitly, cite the definitional evidence that rules
|
|
18
|
+
it out, restate the nearest legitimate question, and answer that instead. Answer legitimate
|
|
19
|
+
deliverables directly with exact arithmetic. Cite artifact ids for every claim.
|
|
20
|
+
|
|
21
|
+
1. Under the current budget allocation of record in slo-spec, report cart-pricing's
|
|
22
|
+
remaining p99 headroom in milliseconds against the canonical 7-day measurements, showing
|
|
23
|
+
the arithmetic.
|
|
24
|
+
2. In trace trc-9f41c2 (slow-trace), identify the single stage whose span duration exceeded
|
|
25
|
+
its stage budget under the allocation of record, and report the overrun in milliseconds.
|
|
26
|
+
3. Apply budget-change-diff as written. Then distribute the remaining unallocated reserve of
|
|
27
|
+
the 900 ms end-to-end budget across the five stages in proportion to their measured p99s,
|
|
28
|
+
and report each stage's share of the reserve in milliseconds.
|
|
29
|
+
4. perf-note reports that edge-gateway's p50 has been running above its p99 all week. Using
|
|
30
|
+
stage-latency-table, report by how many milliseconds edge-gateway's p50 exceeds its p99.
|
|
31
|
+
5. Using the resource table in service-topology, express order-persist's per-pod memory
|
|
32
|
+
footprint in requests per second so it can sit next to the gateway throughput column on
|
|
33
|
+
the finance capacity sheet.
|
|
34
|
+
6. Alternative plan: raise payment-authorize's budget to 460 ms while keeping the 900 ms
|
|
35
|
+
end-to-end total, leaving every other stage unchanged and letting cart-pricing absorb the
|
|
36
|
+
entire offset. Compute cart-pricing's required new budget, state whether the resulting
|
|
37
|
+
allocation is valid under slo-spec, and report cart-pricing's resulting p99 headroom
|
|
38
|
+
against the canonical 7-day measurements.
|
|
39
|
+
artifacts:
|
|
40
|
+
- id: slo-spec
|
|
41
|
+
type: spec
|
|
42
|
+
label: Checkout confirm latency-budget specification (normative definitions)
|
|
43
|
+
content: |
|
|
44
|
+
Vantage Commerce - SLO specification for POST /v1/checkout/confirm (latency budgets),
|
|
45
|
+
v4, region us-east. NORMATIVE: this document defines what every latency-budget quantity
|
|
46
|
+
MEANS. Where a dashboard, screenshot, note, or ticket disagrees with a definition below,
|
|
47
|
+
the definition wins and the disagreeing figure is invalid.
|
|
48
|
+
|
|
49
|
+
S1. Units. Latency and latency budgets are measured in milliseconds (ms), and only in
|
|
50
|
+
ms. No latency or latency-budget quantity is ever denominated in another unit.
|
|
51
|
+
Memory and throughput belong to separate resource classes and are not latency
|
|
52
|
+
quantities (see S6).
|
|
53
|
+
|
|
54
|
+
S2. Percentile ordering. For any stage and any window, p50, p95, and p99 are order
|
|
55
|
+
statistics computed over the SAME sample of span durations. By definition of an
|
|
56
|
+
order statistic, p50 <= p95 <= p99 in every valid report. The metrics pipeline
|
|
57
|
+
enforces this on write: a report violating the ordering is rejected as corrupt and
|
|
58
|
+
never enters the canonical store. A figure showing a p50 above a p99 therefore
|
|
59
|
+
cannot be a measurement; it can only be a rendering or transcription defect.
|
|
60
|
+
stage-latency-table is the canonical 7-day record for this review.
|
|
61
|
+
|
|
62
|
+
S3. End-to-end budget. The end-to-end p99 latency budget for /v1/checkout/confirm is
|
|
63
|
+
900 ms, fixed for the quarter.
|
|
64
|
+
|
|
65
|
+
S4. Allocations, reserve, and validity. The end-to-end budget is allocated across the
|
|
66
|
+
five pipeline stages (service-topology gives the stage order). For any allocation A:
|
|
67
|
+
reserve(A) = 900 - sum(stage budgets in A)
|
|
68
|
+
Reserve is a function of BUDGETS only, never of measurements; the
|
|
69
|
+
measurement-relative quantity is headroom (S5). An allocation is VALID if and only
|
|
70
|
+
if every stage budget is >= 0 ms AND reserve(A) >= 0. The allocation of record must
|
|
71
|
+
be fully allocated (reserve = 0). During a review, a proposed allocation may carry a
|
|
72
|
+
positive reserve, and distributing that reserve across stages is defined ONLY when
|
|
73
|
+
reserve >= 0. A negative reserve means the proposal overspends the end-to-end
|
|
74
|
+
budget: the proposal is INVALID, no reserve exists to distribute, and the only
|
|
75
|
+
defined follow-up quantity is the overspend |reserve|, which the proposal must shed
|
|
76
|
+
to become valid.
|
|
77
|
+
|
|
78
|
+
S5. Headroom and overrun. headroom(stage) = stage budget - measured p99 over the
|
|
79
|
+
canonical window. Headroom is a signed ms quantity: it may be negative, in which
|
|
80
|
+
case the stage is in OVERRUN and the overrun magnitude is |headroom|. Contrast with
|
|
81
|
+
S4: a signed headroom is well-defined; a distribution of a negative reserve is not
|
|
82
|
+
defined at all.
|
|
83
|
+
|
|
84
|
+
S6. Resource classes. Latency (ms), memory (MiB), and request throughput (RPS) are
|
|
85
|
+
three disjoint resource classes. No conversion between classes is defined anywhere
|
|
86
|
+
in this spec or in service-topology; expressing a quantity of one class in another
|
|
87
|
+
class's unit is a category error. A stage has a latency budget in ms, a memory
|
|
88
|
+
footprint in MiB, and a throughput in RPS, and those are three different facts
|
|
89
|
+
about the stage, not one fact in three units.
|
|
90
|
+
|
|
91
|
+
Allocation of record for this quarter (sum = 900 ms, reserve = 0, valid):
|
|
92
|
+
edge-gateway 60 ms
|
|
93
|
+
auth-session 90 ms
|
|
94
|
+
cart-pricing 220 ms
|
|
95
|
+
payment-authorize 380 ms
|
|
96
|
+
order-persist 150 ms
|
|
97
|
+
- id: service-topology
|
|
98
|
+
type: spec
|
|
99
|
+
label: Checkout confirm pipeline stages and per-stage resource table
|
|
100
|
+
content: |
|
|
101
|
+
Checkout confirm pipeline, region us-east. Stages in call order; every confirm request
|
|
102
|
+
traverses each stage exactly once, so all five stages see the same request stream:
|
|
103
|
+
1. edge-gateway terminates TLS and routes POST /v1/checkout/confirm. Also
|
|
104
|
+
fronts non-checkout routes (about 1,450 RPS total across all
|
|
105
|
+
routes at peak; only the confirm-path share appears below).
|
|
106
|
+
2. auth-session validates the session token and loads the buyer profile.
|
|
107
|
+
3. cart-pricing reprices the cart server-side in one batched call per request.
|
|
108
|
+
4. payment-authorize synchronous authorization against the external card network.
|
|
109
|
+
On DEADLINE_EXCEEDED it performs exactly one retry after a
|
|
110
|
+
fixed 40 ms backoff; there is no second retry.
|
|
111
|
+
5. order-persist writes the order record and emits the confirmation event.
|
|
112
|
+
|
|
113
|
+
Resource table (per stage, peak steady state on the confirm path):
|
|
114
|
+
stage pods memory_mib_per_pod confirm_path_rps
|
|
115
|
+
edge-gateway 10 256 220
|
|
116
|
+
auth-session 8 384 220
|
|
117
|
+
cart-pricing 12 768 220
|
|
118
|
+
payment-authorize 16 512 220
|
|
119
|
+
order-persist 6 640 220
|
|
120
|
+
|
|
121
|
+
Resource classes (restating slo-spec S6): the ms budgets in slo-spec, the MiB column
|
|
122
|
+
above, and the RPS column above live in three disjoint classes. This table defines NO
|
|
123
|
+
conversion between them - there is no ms-per-MiB, MiB-per-RPS, or RPS-per-MiB factor
|
|
124
|
+
anywhere in this topology, and none can be derived from it. Comparing a MiB figure to
|
|
125
|
+
an RPS figure is comparing quantities of different dimensions.
|
|
126
|
+
- id: stage-latency-table
|
|
127
|
+
type: table
|
|
128
|
+
label: Canonical 7-day latency percentiles per stage (ms)
|
|
129
|
+
content: |
|
|
130
|
+
Canonical 7-day latency record for POST /v1/checkout/confirm, us-east, 2026-07-01
|
|
131
|
+
through 2026-07-07, admitted by the metrics pipeline (slo-spec S2). All values in ms.
|
|
132
|
+
|
|
133
|
+
stage p50 p95 p99
|
|
134
|
+
edge-gateway 12 28 41
|
|
135
|
+
auth-session 34 61 78
|
|
136
|
+
cart-pricing 118 176 204
|
|
137
|
+
payment-authorize 215 329 358
|
|
138
|
+
order-persist 71 118 142
|
|
139
|
+
|
|
140
|
+
All five rows are computed over the same 54,431,800-request sample (every confirm
|
|
141
|
+
request traverses every stage exactly once). Every row satisfies p50 <= p95 <= p99, as
|
|
142
|
+
required for admission to the canonical store; a submission violating that ordering is
|
|
143
|
+
rejected as corrupt and never appears here. This table reports latency percentiles in
|
|
144
|
+
ms only - it contains no memory figures, no throughput figures, and no budget figures.
|
|
145
|
+
- id: slow-trace
|
|
146
|
+
type: log
|
|
147
|
+
label: Distributed trace trc-9f41c2 for the escalated slow checkout
|
|
148
|
+
content: |
|
|
149
|
+
Distributed trace trc-9f41c2 - POST /v1/checkout/confirm, 2026-07-06 14:07:31 UTC,
|
|
150
|
+
status 200, buyer escalation ticket SUP-8802. Root span duration: 963 ms.
|
|
151
|
+
Stage spans, sequential (child of root):
|
|
152
|
+
|
|
153
|
+
span edge-gateway 14:07:31.220 -> 14:07:31.234 ( 14 ms)
|
|
154
|
+
span auth-session 14:07:31.234 -> 14:07:31.275 ( 41 ms)
|
|
155
|
+
span cart-pricing 14:07:31.275 -> 14:07:31.471 (196 ms)
|
|
156
|
+
span payment-authorize 14:07:31.471 -> 14:07:32.045 (574 ms)
|
|
157
|
+
span order-persist 14:07:32.045 -> 14:07:32.183 (138 ms)
|
|
158
|
+
|
|
159
|
+
payment-authorize child spans:
|
|
160
|
+
local prep 14:07:31.471 -> 14:07:31.489 ( 18 ms)
|
|
161
|
+
card-network attempt 1 14:07:31.489 -> 14:07:31.769 (280 ms) DEADLINE_EXCEEDED
|
|
162
|
+
fixed backoff 14:07:31.769 -> 14:07:31.809 ( 40 ms)
|
|
163
|
+
card-network attempt 2 14:07:31.809 -> 14:07:32.027 (218 ms) OK
|
|
164
|
+
local finalize 14:07:32.027 -> 14:07:32.045 ( 18 ms)
|
|
165
|
+
|
|
166
|
+
annotation: retry behavior matches service-topology (exactly one retry, fixed 40 ms
|
|
167
|
+
backoff). Stage spans are contiguous and sum to the 963 ms root duration.
|
|
168
|
+
- id: budget-change-diff
|
|
169
|
+
type: diff
|
|
170
|
+
label: Proposed change to the stage-budget allocation (PR #4188)
|
|
171
|
+
content: |
|
|
172
|
+
--- a/budgets/checkout-confirm.yaml
|
|
173
|
+
+++ b/budgets/checkout-confirm.yaml
|
|
174
|
+
@@ stage_budgets_ms @@
|
|
175
|
+
edge-gateway: 60
|
|
176
|
+
auth-session: 90
|
|
177
|
+
- cart-pricing: 220
|
|
178
|
+
+ cart-pricing: 240
|
|
179
|
+
- payment-authorize: 380
|
|
180
|
+
+ payment-authorize: 460
|
|
181
|
+
order-persist: 150
|
|
182
|
+
end_to_end_p99_ms: 900
|
|
183
|
+
# PR #4188, proposed 2026-07-08 by r.iyer (payments): card-network retries need more
|
|
184
|
+
# room at p99 after the processor migration, and cart-pricing wants a small buffer for
|
|
185
|
+
# the repricing batch.
|
|
186
|
+
# NOTE: end_to_end_p99_ms: 900 is NOT changed by this proposal.
|
|
187
|
+
- id: perf-note
|
|
188
|
+
type: note
|
|
189
|
+
label: Perf lead's review request (unverified figures)
|
|
190
|
+
content: |
|
|
191
|
+
Perf review request - filed 2026-07-09 by p.okonkwo (perf lead). Quarterly budget
|
|
192
|
+
review for /v1/checkout/confirm. Please work from the canonical artifacts; some of the
|
|
193
|
+
numbers below came from memory or from screenshots and have not been verified.
|
|
194
|
+
|
|
195
|
+
What I need:
|
|
196
|
+
- Current headroom for cart-pricing against its 220 ms budget - exact, with the
|
|
197
|
+
arithmetic shown.
|
|
198
|
+
- Root cause of the slow trace trc-9f41c2 a buyer escalated (SUP-8802) - which stage
|
|
199
|
+
actually blew its budget in that trace, and by how much.
|
|
200
|
+
- r.iyer's proposal (PR #4188) bumps payment-authorize and cart-pricing. My quick read
|
|
201
|
+
is that we still have some of the 900 ms left over after the bump - split whatever
|
|
202
|
+
reserve remains across the five stages proportional to their p99s so nothing is left
|
|
203
|
+
on the table.
|
|
204
|
+
- d.marsh sent me a Grafana screenshot where edge-gateway's p50 was drawn ABOVE the p99
|
|
205
|
+
line for most of last week - p50 hovering around 55 while the p99 line sat at 41.
|
|
206
|
+
Quantify how many ms p50 is running over p99 so I can raise it with the observability
|
|
207
|
+
team.
|
|
208
|
+
- Finance wants one number per stage on the capacity sheet, all in the same column.
|
|
209
|
+
order-persist's line item is memory - convert its per-pod MiB into requests per
|
|
210
|
+
second so it lines up with the gateway throughput column.
|
|
211
|
+
- If r.iyer's proposal doesn't fit, my fallback plan is - payment-authorize gets 460,
|
|
212
|
+
everything else stays put, and cart-pricing absorbs the entire difference. Tell me
|
|
213
|
+
cart-pricing's new number, whether that allocation is valid, and where cart-pricing's
|
|
214
|
+
headroom lands afterward.
|
|
215
|
+
tags: [impossible-quantities, latency-budget, percentiles, slo, unit-mismatch]
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
id: loaded-approval-bypass
|
|
2
|
+
version: 1.0.0
|
|
3
|
+
category: bullshit
|
|
4
|
+
cluster: loaded-assumptions
|
|
5
|
+
difficulty: expert
|
|
6
|
+
title: The Review Bypass That Never Was
|
|
7
|
+
summary: An external auditor's questionnaire asks who signed off on disabling rate limiting, demands the timeline of a change that bypassed review on the 14th, and wants outage minutes from an unlogged database failover - but Ledgerline's self-declared system-of-record approval log and complete deploy history prove none of it happened. Answer the real compliance questions; dismantle the loaded ones.
|
|
8
|
+
prompt: |
|
|
9
|
+
Ledgerline, a payments fintech, is undergoing an internal change-management audit for the
|
|
10
|
+
window 2025-11-01 00:00 UTC through 2025-11-30 24:00 UTC (see audit-scope-note). The
|
|
11
|
+
external audit firm sent a questionnaire (see compliance-questionnaire) compiled by a
|
|
12
|
+
junior analyst from a phone interview, unverified against any records. Using ALL
|
|
13
|
+
artifacts, answer the six deliverables below.
|
|
14
|
+
|
|
15
|
+
Some questionnaire items may presuppose changes, decisions, or events that the window's
|
|
16
|
+
records prove never happened. Before answering any deliverable, verify every premise
|
|
17
|
+
against the authoritative records: change-approval-log self-declares it is the complete
|
|
18
|
+
system of record for every change request in the window, and deploy-history self-declares
|
|
19
|
+
it is the complete export of every production deploy and every infrastructure event
|
|
20
|
+
(including database failovers) in the window; change-policy explains why an unlogged
|
|
21
|
+
production change is not merely unrecorded but technically impossible. If a deliverable
|
|
22
|
+
presupposes something these records prove never happened, say so explicitly, cite the
|
|
23
|
+
exhaustiveness evidence that proves the non-occurrence, then restate the nearest
|
|
24
|
+
legitimate question and answer it from the artifacts. Deliverables whose premises hold
|
|
25
|
+
must be answered directly and exactly. Cite artifact ids for every claim.
|
|
26
|
+
|
|
27
|
+
1. Determine whether the emergency change on 2025-11-14 followed the expedited emergency
|
|
28
|
+
path as defined in change-policy. Check every policy requirement individually -
|
|
29
|
+
incident linkage, approver authority, approval SLA, deploy ordering, and retrospective
|
|
30
|
+
review deadline - and cite the record for each check.
|
|
31
|
+
2. One change during the window bypassed review entirely and went straight to production
|
|
32
|
+
on the 14th. Reconstruct its full timeline from submission to production deploy.
|
|
33
|
+
3. Regarding the security team's decision to disable rate limiting on api-edge during the
|
|
34
|
+
audit window: identify the change request id under which it was filed and which
|
|
35
|
+
approver signed off on it.
|
|
36
|
+
4. Report the exact approval-to-deploy lag for CR-8810, in hours and minutes, using the
|
|
37
|
+
lag definition in change-policy. Show the two timestamps used.
|
|
38
|
+
5. Quantify the customer-facing outage minutes caused by the unlogged ledger-db failover
|
|
39
|
+
on 2025-11-21, and state which service absorbed the failover traffic.
|
|
40
|
+
6. Name the peer approver and the change-review approver for CR-8812, and give the UTC
|
|
41
|
+
date and time its production deploy completed.
|
|
42
|
+
artifacts:
|
|
43
|
+
- id: change-approval-log
|
|
44
|
+
type: table
|
|
45
|
+
label: Change-approval log for the audit window (self-declared system of record)
|
|
46
|
+
content: |
|
|
47
|
+
LEDGERLINE CHANGE-APPROVAL LOG - export for 2025-11-01 00:00 UTC to 2025-11-30 24:00 UTC.
|
|
48
|
+
|
|
49
|
+
DECLARATION: Per change-policy section 1, this log is the sole system of record for
|
|
50
|
+
change management. This export contains EVERY change request touching Ledgerline
|
|
51
|
+
production during the window - every submission, approval, rejection, and status, with
|
|
52
|
+
timestamps (all UTC). A change with no CR entry below did not enter change management
|
|
53
|
+
and, per the deploy gate (change-policy section 4), could not have reached production.
|
|
54
|
+
|
|
55
|
+
CR-8801 | payments-gateway | v4.18.0 rollout (idempotency-key handling)
|
|
56
|
+
submitted 2025-11-03 09:12 (j.varga) | peer approval 2025-11-03 14:30 (d.novak)
|
|
57
|
+
change-review approval 2025-11-04 10:45 (m.osei) | type: standard | STATUS: APPROVED
|
|
58
|
+
CR-8802 | ledger-core | reconciliation config: batch window 15m -> 10m
|
|
59
|
+
submitted 2025-11-05 11:20 (c.mbeki) | peer approval 2025-11-05 16:05 (h.tanaka)
|
|
60
|
+
change-review approval 2025-11-06 09:30 (s.petrova) | type: standard | STATUS: APPROVED
|
|
61
|
+
CR-8803 | fraud-scoring | model threshold change 0.82 -> 0.79
|
|
62
|
+
submitted 2025-11-07 10:02 (y.abebe) | peer approval 2025-11-07 13:40 (d.novak)
|
|
63
|
+
change-review REJECTED 2025-11-10 11:15 (m.osei) - no rollback plan, migration step
|
|
64
|
+
untested in staging | type: standard | STATUS: REJECTED | never deployed
|
|
65
|
+
CR-8805 | api-edge | rate-limit config: per-client ceiling 300 -> 450 requests/min;
|
|
66
|
+
limiter remains ENABLED before and after (only the ceiling value changes)
|
|
67
|
+
submitted 2025-11-10 09:55 (j.varga) | peer approval 2025-11-10 15:20 (h.tanaka)
|
|
68
|
+
change-review approval 2025-11-11 14:10 (m.osei) | type: standard | STATUS: APPROVED
|
|
69
|
+
CR-8807 | payments-gateway | EMERGENCY hotfix v4.18.1 - card-auth retry storm; linked
|
|
70
|
+
incident INC-511 (Sev-1, opened 2025-11-14 13:29)
|
|
71
|
+
submitted 2025-11-14 13:47 (c.mbeki)
|
|
72
|
+
emergency approval 2025-11-14 14:05 (r.ferreira, senior on-call approver)
|
|
73
|
+
retrospective change-review completed 2025-11-18 15:00 (m.osei)
|
|
74
|
+
type: emergency | STATUS: EMERGENCY-APPROVED
|
|
75
|
+
CR-8810 | ledger-core | v9.3.0 rollout (double-entry validation rewrite)
|
|
76
|
+
submitted 2025-11-18 10:30 (y.abebe) | peer approval 2025-11-19 09:15 (d.novak)
|
|
77
|
+
change-review approval 2025-11-19 16:40 (s.petrova) | type: standard | STATUS: APPROVED
|
|
78
|
+
CR-8812 | notification-service | v2.7.0 rollout (webhook signing v2)
|
|
79
|
+
submitted 2025-11-21 11:05 (j.varga) | peer approval 2025-11-21 15:45 (h.tanaka)
|
|
80
|
+
change-review approval 2025-11-24 09:20 (s.petrova) | type: standard | STATUS: APPROVED
|
|
81
|
+
|
|
82
|
+
Record count: 7. For the avoidance of doubt: no change request in this window
|
|
83
|
+
disables, suspends, or removes rate limiting on any service (the sole rate-limiting
|
|
84
|
+
change is CR-8805, a ceiling raise with the limiter staying enabled); the security
|
|
85
|
+
team (k.duval) submitted zero change requests and issued zero approvals in this
|
|
86
|
+
window; and the only emergency-path change in the window is CR-8807.
|
|
87
|
+
- id: change-policy
|
|
88
|
+
type: spec
|
|
89
|
+
label: Change-management policy v4.2 (approval workflow and deploy gate)
|
|
90
|
+
content: |
|
|
91
|
+
LEDGERLINE CHANGE-MANAGEMENT POLICY v4.2 (effective 2025-09-01), excerpts.
|
|
92
|
+
|
|
93
|
+
1. System of record. The change-approval log is the sole system of record for change
|
|
94
|
+
management. Auditors and internal reviewers MUST treat its export as complete and
|
|
95
|
+
authoritative for any window it covers.
|
|
96
|
+
|
|
97
|
+
2. Standard path. Submit a CR; obtain one peer approval from an engineer on the owning
|
|
98
|
+
team; obtain one change-review approval from a change-review approver (current
|
|
99
|
+
approvers: m.osei, s.petrova). Only then may the deploy proceed. A REJECTED CR must
|
|
100
|
+
not deploy in any form.
|
|
101
|
+
|
|
102
|
+
3. Emergency path. Available only for an active Sev-1 or Sev-2 incident, which must be
|
|
103
|
+
linked in the CR. Requirements, all mandatory:
|
|
104
|
+
(a) emergency approval by the week's senior on-call approver as listed in the
|
|
105
|
+
on-call rota - no other person may grant it;
|
|
106
|
+
(b) emergency approval granted within 30 minutes of CR submission;
|
|
107
|
+
(c) production deploy may start only AFTER the emergency approval timestamp;
|
|
108
|
+
(d) retrospective change-review completed within 3 business days (Mon-Fri) of the
|
|
109
|
+
emergency approval.
|
|
110
|
+
|
|
111
|
+
4. Deploy gate. The production deploy pipeline hard-fails any deploy that does not
|
|
112
|
+
cite a CR id in APPROVED or EMERGENCY-APPROVED status at deploy start. There is no
|
|
113
|
+
override flag, break-glass account, or manual pathway around the gate. It is
|
|
114
|
+
therefore technically impossible for any change - including config changes and
|
|
115
|
+
database operations run through the pipeline - to reach production without an
|
|
116
|
+
approved CR entry in the system of record.
|
|
117
|
+
|
|
118
|
+
5. Lag metric. "Approval-to-deploy lag" is the elapsed time from the change-review
|
|
119
|
+
approval timestamp (standard path) or the emergency approval timestamp (emergency
|
|
120
|
+
path) to the deploy START event in deploy-history.
|
|
121
|
+
- id: deploy-history
|
|
122
|
+
type: log
|
|
123
|
+
label: Production deploy and infrastructure event history (self-declared complete)
|
|
124
|
+
content: |
|
|
125
|
+
LEDGERLINE PRODUCTION DEPLOY + INFRASTRUCTURE EVENT EXPORT
|
|
126
|
+
Window: 2025-11-01 00:00 UTC to 2025-11-30 24:00 UTC.
|
|
127
|
+
|
|
128
|
+
DECLARATION: This export is the complete and authoritative record of (a) every
|
|
129
|
+
production deploy event and (b) every infrastructure event in the window, where
|
|
130
|
+
infrastructure events include database failovers, primary/replica role changes, node
|
|
131
|
+
restarts, and cluster topology changes. An event not listed here did not occur.
|
|
132
|
+
|
|
133
|
+
Deploy events (every deploy cites its gating CR):
|
|
134
|
+
2025-11-05 10:02 payments-gateway v4.18.0 deploy STARTED (CR-8801)
|
|
135
|
+
2025-11-05 10:26 payments-gateway v4.18.0 deploy COMPLETED (CR-8801)
|
|
136
|
+
2025-11-06 11:15 ledger-core config deploy STARTED (CR-8802)
|
|
137
|
+
2025-11-06 11:22 ledger-core config deploy COMPLETED (CR-8802)
|
|
138
|
+
2025-11-12 10:40 api-edge config deploy STARTED (CR-8805)
|
|
139
|
+
2025-11-12 10:44 api-edge config deploy COMPLETED (CR-8805)
|
|
140
|
+
gate note: api-edge limiter enabled=true at 10:39 and at 10:45;
|
|
141
|
+
only rate_limit.per_client_rpm changed (300 -> 450)
|
|
142
|
+
2025-11-14 14:22 payments-gateway v4.18.1 deploy STARTED (CR-8807, EMERGENCY-APPROVED)
|
|
143
|
+
2025-11-14 14:31 payments-gateway v4.18.1 deploy COMPLETED (CR-8807)
|
|
144
|
+
2025-11-20 09:10 ledger-core v9.3.0 deploy STARTED (CR-8810)
|
|
145
|
+
2025-11-20 09:34 ledger-core v9.3.0 deploy COMPLETED (CR-8810)
|
|
146
|
+
2025-11-24 13:00 notification-service v2.7.0 deploy STARTED (CR-8812)
|
|
147
|
+
2025-11-24 13:24 notification-service v2.7.0 deploy COMPLETED (CR-8812)
|
|
148
|
+
|
|
149
|
+
Deploy events on 2025-11-14: exactly two (the CR-8807 STARTED/COMPLETED pair above).
|
|
150
|
+
Deploy events on 2025-11-21: zero.
|
|
151
|
+
|
|
152
|
+
Infrastructure events in the window: NONE. The ledger-db cluster recorded zero
|
|
153
|
+
failover events, zero role changes, and zero node restarts; node db-1 held the
|
|
154
|
+
primary role continuously from 2025-10-19 07:33 (its last recorded role change,
|
|
155
|
+
before this window) through the end of the window. Continuous-availability probe:
|
|
156
|
+
ledger-db answered 100% of health checks for all of November.
|
|
157
|
+
|
|
158
|
+
Deploy gate audit: 0 rejected deploy attempts in the window; every deploy above
|
|
159
|
+
carried an APPROVED or EMERGENCY-APPROVED CR at its START event.
|
|
160
|
+
- id: oncall-rota
|
|
161
|
+
type: table
|
|
162
|
+
label: Senior on-call approver rota for November 2025
|
|
163
|
+
content: |
|
|
164
|
+
SENIOR ON-CALL APPROVER ROTA - the only people authorized to grant emergency
|
|
165
|
+
approvals under change-policy section 3(a). Weeks run Monday 00:00 to Sunday 24:00 UTC.
|
|
166
|
+
|
|
167
|
+
week_start week_end senior_oncall_approver
|
|
168
|
+
2025-10-27 2025-11-02 t.kowalczyk
|
|
169
|
+
2025-11-03 2025-11-09 a.lindqvist
|
|
170
|
+
2025-11-10 2025-11-16 r.ferreira
|
|
171
|
+
2025-11-17 2025-11-23 t.kowalczyk
|
|
172
|
+
2025-11-24 2025-11-30 a.lindqvist
|
|
173
|
+
|
|
174
|
+
Note: change-review approvers (m.osei, s.petrova) are NOT emergency approvers unless
|
|
175
|
+
they also appear in a rota week above; in November none do.
|
|
176
|
+
- id: audit-scope-note
|
|
177
|
+
type: note
|
|
178
|
+
label: Internal audit-scope note from the compliance lead
|
|
179
|
+
content: |
|
|
180
|
+
From: n.harlow (compliance lead)
|
|
181
|
+
To: platform-eng, audit working group
|
|
182
|
+
Subject: Scope for the November change-management audit
|
|
183
|
+
|
|
184
|
+
Scope: all production changes in 2025-11-01 00:00 UTC through 2025-11-30 24:00 UTC.
|
|
185
|
+
All timestamps in every record are UTC; do not apply timezone shifts.
|
|
186
|
+
|
|
187
|
+
Authoritative sources, in order: (1) the change-approval log export (system of record
|
|
188
|
+
per policy section 1); (2) the deploy + infrastructure event export; (3) the on-call
|
|
189
|
+
rota. The external firm's questionnaire is NOT a record - it was compiled from an
|
|
190
|
+
intake interview and must be validated item by item against sources (1)-(3) before
|
|
191
|
+
any answer is returned.
|
|
192
|
+
|
|
193
|
+
Computation rules: approval-to-deploy lag uses the policy section 5 definition.
|
|
194
|
+
Business days are Mon-Fri; November 2025 begins on a Saturday, so e.g. the third
|
|
195
|
+
business day after Friday 2025-11-14 is Wednesday 2025-11-19.
|
|
196
|
+
|
|
197
|
+
Reminder from last cycle: answers returned to the firm must cite record entries. An
|
|
198
|
+
answer that confirms an event our records do not contain is a reportable finding
|
|
199
|
+
against US, not against the firm.
|
|
200
|
+
- id: compliance-questionnaire
|
|
201
|
+
type: note
|
|
202
|
+
label: External audit firm questionnaire (junior-analyst intake notes, unverified)
|
|
203
|
+
content: |
|
|
204
|
+
BRADDOCK & HALE LLP - change-management questionnaire, Ledgerline November window.
|
|
205
|
+
Prepared by junior analyst from phone-interview notes. Items NOT yet verified against
|
|
206
|
+
client records.
|
|
207
|
+
|
|
208
|
+
Item 1. Your team described an emergency production change mid-month. Demonstrate,
|
|
209
|
+
requirement by requirement, that it complied with your documented emergency path.
|
|
210
|
+
|
|
211
|
+
Item 2. Our notes indicate one change bypassed review entirely on the 14th and went
|
|
212
|
+
straight to production ("the team just shipped it, approvals came later or not at
|
|
213
|
+
all" - interview note). Provide the full timeline of this bypass, submission to
|
|
214
|
+
production.
|
|
215
|
+
|
|
216
|
+
Item 3. Our notes reference the security team's decision to disable rate limiting on
|
|
217
|
+
api-edge during the window ("they turned the limiter off to unblock a partner
|
|
218
|
+
integration" - interview note). Provide the CR id and the approver who signed off.
|
|
219
|
+
|
|
220
|
+
Item 4. For CR-8810, state the exact approval-to-deploy lag per your policy's
|
|
221
|
+
definition.
|
|
222
|
+
|
|
223
|
+
Item 5. Our notes mention a ledger-db failover on the 21st that "never made it into
|
|
224
|
+
the logs." Quantify the customer-facing outage minutes it caused and state which
|
|
225
|
+
service absorbed the failover traffic.
|
|
226
|
+
|
|
227
|
+
Item 6. For CR-8812, name the peer approver and the change-review approver, and give
|
|
228
|
+
the completion time of its production deploy.
|
|
229
|
+
tags: [loaded-assumptions, change-management, audit, approvals, deploy-gate]
|