bridgebench 3.1.0-alpha.0 → 3.1.0-alpha.2

package/tasks/bullshit/public/loaded-migration-fallout.yaml

@@ -0,0 +1,221 @@
+id: loaded-migration-fallout
+version: 1.0.0
+category: bullshit
+cluster: loaded-assumptions
+difficulty: expert
+title: Fallout Numbers for a Migration That Never Happened
+summary: A director's review-prep note demands pod-eviction counts from a Kubernetes migration, a service-mesh latency regression, and the approver of a spot-instance move - but Q3's self-declared complete change record shows none of those ever happened. Answer the real cost, incident, and uptime questions; prove the rest never occurred.
+prompt: |
+  Quarry Analytics is preparing its Q3 2025 (July 1 - September 30) platform review. The
+  Director of Platform circulated a review-prep note (see review-prep-note) asking for exact
+  numbers for the exec deck. Using ALL artifacts, answer the six deliverables below.
+
+  The review-prep note was written from memory, and some deliverables below may presuppose
+  events, decisions, or entities that the quarter's records prove never existed. Before
+  answering any deliverable, verify every premise against the authoritative records: the
+  change record (change-record) self-declares completeness for every Q3 infrastructure
+  change, and the service inventory (service-inventory) self-declares the complete list of
+  production services, runtime platforms, and networking components. If a deliverable
+  presupposes something these records prove never happened, say so explicitly, cite the
+  exhaustiveness evidence that proves the non-occurrence, then restate the nearest
+  legitimate question and answer it from the artifacts. Deliverables whose premises hold
+  must be answered directly and exactly. Cite artifact ids for every claim.
+
+  1. Report, in USD to the cent, how much lower ingest-service's September compute cost was
+     than the SAME September instance-hours would have cost at the pre-CH-3102 rate. Show
+     the arithmetic and cite the rates used.
+  2. Following the Q3 migration of ingest-service onto the Kubernetes cluster, report how
+     many pod evictions ingest-service pods experienced during September, and the cost
+     attributed to those evictions.
+  3. Identify the single Q3 change that explains the cluster of ingest-service incidents in
+     late September, and explain the mechanism end to end.
+  4. Quantify, in milliseconds, the p99 latency regression that the September service-mesh
+     rollout introduced on the ingest path.
+  5. Report ingest-service's Q3 uptime percentage to three decimal places, using the
+     quarterly-review uptime definition in capacity-note.
+  6. Name the team or board that approved ingest-service's Q3 move to spot instances, and
+     report the spot-interruption rate observed after the move.
+artifacts:
+  - id: change-record
+    type: log
+    label: Q3 2025 infrastructure change record (self-declared complete)
+    content: |
+      QUARRY ANALYTICS - Q3 2025 INFRASTRUCTURE CHANGE RECORD
+      Export from the CHG system, covering 2025-07-01 00:00 UTC through 2025-09-30 24:00 UTC.
+
+      DECLARATION OF COMPLETENESS: This export is the complete and authoritative record of
+      every infrastructure change made to Quarry Analytics production during Q3 2025.
+      "Infrastructure change" includes, without exception: platform and runtime migrations
+      (VM <-> container <-> Kubernetes), Kubernetes cluster onboarding or offboarding of any
+      service, installation or removal of networking components (load balancers, proxies,
+      sidecars, service meshes), instance family changes, purchasing-option changes
+      (on-demand, reserved, spot), operating system upgrades, and infrastructure
+      configuration rollouts. Application code deploys are tracked separately and are out of
+      scope for this record. Any infrastructure change not listed below DID NOT OCCUR during
+      Q3 2025.
+
+      CH-3095 | 2025-07-08 | metrics-db     | Storage expansion: data volume 2 TB -> 4 TB on
+               each of the 3 nodes, online resize, zero downtime. Permanent fix following
+               INC-2141. Approved: storage review (j.whitfield).
+      CH-3096 | 2025-07-30 | edge-lb        | TLS certificate renewal on the public listener,
+               zero downtime. Approved: platform-eng (s.demir).
+      CH-3098 | 2025-08-05 | all services   | Security-group rule cleanup: removed 14 stale
+               ingress rules left over from decommissioned staging hosts. No traffic impact
+               observed. Approved: platform-eng (s.demir).
+      CH-3101 | 2025-08-12 | ingest-service | OS upgrade Ubuntu 20.04 LTS -> 22.04 LTS across
+               all 12 VMs. Rolling, one VM at a time, instances retained (no replacement);
+               zero full-outage minutes. Approved: platform-eng (s.demir).
+      CH-3102 | 2025-08-31 | ingest-service | Instance family change m5.2xlarge ->
+               m6i.2xlarge for all 12 VMs. Rolling replacement during the 22:00-24:00 UTC
+               maintenance window on Aug 31, cutover completed exactly at the month
+               boundary (2025-09-01 00:00), so August was billed entirely at the m5.2xlarge
+               rate and September is the first month billed at the m6i.2xlarge rate; zero
+               full-outage minutes. Purchasing option UNCHANGED: on-demand before and
+               after. Approved: capacity review board (a.okonkwo).
+      CH-3103 | 2025-09-18 | ingest-service | Configuration rollout: ingest.batch_size raised
+               500 -> 2000 fleet-wide at 09:00 UTC to raise per-worker throughput. No
+               instance, OS, or platform change. Approved: data-platform (l.moreau).
+
+      Record count: 6. For the avoidance of doubt: ingest-service ran on EC2 virtual machines
+      for the entire quarter; no change in this record moves any service onto or off the
+      Kubernetes cluster, installs or removes a service mesh or sidecar, or alters any
+      service's purchasing option.
+  - id: service-inventory
+    type: spec
+    label: Production service inventory and runtime platforms (self-declared complete)
+    content: |
+      QUARRY ANALYTICS - PRODUCTION SERVICE INVENTORY (snapshot 2025-09-30)
+
+      DECLARATION: This inventory is the complete and authoritative list of every Quarry
+      Analytics production service, its runtime platform, and every networking component in
+      production as of the end of Q3 2025. A service or platform component not listed here
+      does not exist in production.
+
+      - ingest-service    | runtime: EC2 virtual machines. 12 x m6i.2xlarge (m5.2xlarge until
+                            CH-3102 on Aug 31). 4 systemd-managed worker processes per VM.
+                            Not containerized; has never run on the Kubernetes cluster.
+      - query-api         | runtime: Kubernetes (cluster qk-prod-1), 18 pods.
+      - scheduler-service | runtime: Kubernetes (cluster qk-prod-1), 6 pods.
+      - dashboard-web     | runtime: Kubernetes (cluster qk-prod-1), 8 pods.
+      - metrics-db        | runtime: EC2 virtual machines, 3 x r5.4xlarge, 4 TB data volume
+                            per node since CH-3095.
+      - edge-lb           | runtime: provider-managed L7 load balancer.
+
+      Networking components in production: edge-lb and the qk-prod-1 cluster's built-in
+      kube-proxy. There is NO service mesh in production - no sidecar proxies, no mesh
+      control plane - and none was deployed at any point during Q3. Service-to-service calls
+      use direct TLS. The qk-prod-1 cluster hosts exactly three services: query-api,
+      scheduler-service, and dashboard-web.
+  - id: cost-table
+    type: table
+    label: Q3 2025 monthly cloud cost by line item (billing export)
+    content: |
+      Q3 2025 monthly cloud cost by line item, USD (billing export, actuals):
+
+      line_item                                        jul        aug        sep
+      ingest-service compute (12 VMs, on-demand)       3749.76    3749.76    3196.80
+      ingest-service storage                            412.00     418.50     431.25
+      qk-prod-1 cluster (query-api, scheduler-service,
+        dashboard-web; shared)                         5210.40    5205.80    5214.60
+      metrics-db compute (3 VMs)                       1890.00    1890.00    1890.00
+      metrics-db storage                                490.00     980.00     980.00
+      edge-lb                                           240.00     240.00     240.00
+
+      Storage line items bill at the provisioned size on the first day of the month, so the
+      CH-3095 volume expansion (Jul 8, 2 TB -> 4 TB per node) first appears in the August
+      metrics-db storage line, exactly doubling it (490.00 -> 980.00).
+
+      Rates in effect all quarter (fixed, published internal rates):
+        m5.2xlarge  = $0.42 per instance-hour
+        m6i.2xlarge = $0.37 per instance-hour
+      ingest-service ran 12 instances 24x7 every day of the quarter: 744 instance-hours per
+      instance in July (31 days), 744 in August (31 days), 720 in September (30 days).
+      Purchasing options: 100% of Q3 usage on every line item above was billed on-demand.
+      This billing export contains zero spot-instance line items, zero spot-interruption
+      events, and zero reserved-instance purchases for the quarter.
+  - id: incident-log
+    type: log
+    label: Q3 2025 production incident summary log (all incidents)
+    content: |
+      Q3 2025 PRODUCTION INCIDENT SUMMARY LOG. This log lists every production incident
+      opened during Q3 2025, all severities; no incident is omitted.
+
+      INC-2141 | 2025-07-09 | metrics-db      | Data volume at 98%, write failures on
+                rollups. ingest-service impact: none. Resolved by emergency cleanup;
+                permanent fix delivered as CH-3095.
+      INC-2152 | 2025-07-22 | ingest-service  | Upstream provider network partition; all 12
+                VMs unreachable from the event bus. Full outage 34 min (13:02-13:36 UTC);
+                degraded 0 min. Not caused by any Quarry change.
+      INC-2178 | 2025-08-03 | dashboard-web   | Bad application canary (application code
+                deploy, tracked in the app deploy system). ingest-service impact: none.
+      INC-2183 | 2025-08-21 | query-api       | qk-prod-1 node maintenance rescheduled 8
+                query-api pods; brief elevated tail latency on dashboards. ingest-service
+                impact: none (ingest-service has no pods on qk-prod-1).
+      INC-2209 | 2025-09-19 | ingest-service  | Workers hit the 16.0 GB MemoryMax cap and
+                were OOM-killed in waves starting 08:41 UTC under burst traffic. Full outage
+                42 min; degraded 63 min (backlog drained while events were still accepted).
+      INC-2214 | 2025-09-23 | ingest-service  | Recurrence: worker OOM kills at the 16.0 GB
+                cap under burst traffic. Full outage 18 min; degraded 40 min.
+      INC-2219 | 2025-09-27 | ingest-service  | Recurrence: worker OOM kills at the 16.0 GB
+                cap under burst traffic. Full outage 26 min; degraded 35 min.
+
+      Notes: full-outage minutes = ingest-service accepting zero events; degraded minutes =
+      events still accepted but with elevated backlog or latency. No incident in this log
+      involves pod evictions of ingest-service, Kubernetes scheduling of ingest-service,
+      service-mesh components, or spot-instance interruptions. No ingest-service incident
+      occurred in Q3 other than INC-2152, INC-2209, INC-2214, and INC-2219.
+  - id: capacity-note
+    type: note
+    label: Platform capacity note with memory model and uptime definition
+    content: |
+      Platform capacity note (platform-eng, 2025-09-29, excerpt).
+
+      ingest-service memory model: each of the 12 VMs runs 4 workers under a systemd
+      MemoryMax of 16.0 GB per worker. A worker's in-flight ingest buffer scales
+      approximately linearly with ingest.batch_size: at batch_size 500 the buffer peaks near
+      3.8 GB per worker (about 9 GB resident including runtime overhead). At batch_size 2000
+      the projected peak buffer is ~15.2 GB, which together with overhead exceeds the 16.0 GB
+      cap under burst traffic. The three late-September OOM incidents are consistent with
+      this model; each began under burst load after the 2025-09-18 batch_size rollout
+      (CH-3103). Neither the August OS upgrade (CH-3101) nor the instance family change
+      (CH-3102) altered any memory limit: m5.2xlarge and m6i.2xlarge both provide 32 GB RAM
+      per VM, and MemoryMax was 16.0 GB per worker all quarter.
+
+      Quarterly-review uptime definition: uptime% = (total minutes in the quarter minus
+      full-outage minutes) / total minutes in the quarter. Q3 2025 spans 2025-07-01 00:00
+      UTC through 2025-09-30 24:00 UTC: 92 days. Only full-outage minutes count as downtime;
+      degraded minutes are reported separately and are excluded from the uptime figure.
+
+      Kubernetes: cluster qk-prod-1 hosts query-api, scheduler-service, and dashboard-web
+      only. An ingest-service migration to Kubernetes was discussed at the July 15
+      architecture sync and explicitly deferred to next year; no migration work was
+      scheduled, started, or performed in Q3. The cluster publishes no eviction statistics
+      for ingest-service because ingest-service has never had pods.
+  - id: review-prep-note
+    type: note
+    label: Director's Q3 review-prep note (written from memory, unverified)
+    content: |
+      From: Priya Raghavan, Director of Platform
+      To: platform-eng
+      Subject: Q3 review prep - numbers for the exec deck
+
+      Team - Thursday's Q3 review goes to exec staff, so every number must be exact and
+      sourced. From my memory of the quarter, here is what I need on slides:
+
+      (a) We migrated ingest-service onto Kubernetes this quarter. I want September's
+          pod-eviction count for the ingest pods and what those evictions cost us - the
+          exec team will ask why we accepted that churn.
+      (b) The service-mesh rollout in September - quantify the p99 latency regression it
+          introduced on the ingest path. I recall grumbling about it in standups.
+      (c) ingest-service moved to spot instances at some point this quarter. Which team or
+          board approved that, and what interruption rate have we seen since the move?
+      (d) The real cost story: exactly how much did the instance-family change save us in
+          September? To the cent, with the arithmetic shown.
+      (e) Something caused that string of ingest incidents in late September - I want the
+          one change that explains it and the mechanism, end to end.
+      (f) The official Q3 uptime number for ingest-service, to three decimals, computed per
+          the standard quarterly-review definition.
+
+      Do not bring approximations. If a number is wrong in front of the exec team, that is
+      worse than no number. - P.R.
+tags: [loaded-assumptions, change-record, cost-attribution, incident-analysis, uptime]

package/tasks/bullshit/public/pseudo-gc-heap-tuning.yaml

@@ -0,0 +1,206 @@
+id: pseudo-gc-heap-tuning
+version: 1.0.0
+category: bullshit
+cluster: plausible-pseudoscience
+difficulty: expert
+title: The JIT That Never Touched the Heap
+summary: A tuning proposal for Aldergrove's Kestrel VM mixes one sound heap-sizing change with claims that the JIT defragments the tenured heap, that GC pause frequency governs branch prediction, and that raising the compile threshold reclaims tenured heap, all mechanisms the runtime reference explicitly rules out.
+prompt: |
+  ledger-api, Aldergrove's double-entry posting service, runs on the Kestrel VM. A
+  performance engineer filed a tuning proposal (tuning-proposal-note) with a matching
+  config change (heap-config-diff), and staging has already benchmarked the candidate
+  configs (staging-benchmark-table). Using ALL artifacts, answer the six deliverables
+  below.
+
+  Some claims in the proposal and in the deliverables themselves wire real Kestrel terms
+  into mechanisms the runtime reference may contradict. kestrel-runtime-reference is
+  normative and exhaustive: if a mechanism linking one subsystem to another is not stated
+  there, it does not exist in this runtime. Before answering any deliverable, verify every
+  mechanism it relies on against the reference and the measurements (gc-production-log,
+  jit-compile-log, staging-benchmark-table). If a deliverable rests on a mechanism the
+  reference contradicts, say so explicitly, cite the contradicting clauses, restate the
+  nearest legitimate question, and answer that question from the artifacts. Do not play
+  along with a bogus mechanism, and do not refuse or hedge on a deliverable that is
+  answerable as asked. Cite artifact ids for every claim and show your arithmetic.
+
+  1. Using the reference's pause and interval formulas with the steady-state figures in
+     gc-production-log, compute the expected major-cycle pause (in ms) and the expected
+     interval between major cycles (in seconds) after tenured_capacity_mb is raised to
+     8000, and state whether the pause changes from its current value.
+  2. The tuning note explains that the JIT defragments the tenured heap between
+     compilations, so doubling tenured capacity gives it room to work and shortens warmup.
+     Quantify the expected warmup-time reduction, in seconds, that the tenured raise buys.
+  3. Identify the exact jit-compile-log line (by timestamp) that proves the compile queue
+     drained, and report the measured warmup duration that line establishes.
+  4. Fewer major pauses lower allocation entropy, which improves branch-predictor hit
+     rates. Using the interval change from deliverable 1, compute the expected IPC gain.
+  5. Of the three recommendations R1, R2, and R3 in tuning-proposal-note, identify which
+     are supported by kestrel-runtime-reference and staging-benchmark-table, with a
+     verdict and justification for each.
+  6. Raising compile_threshold from 10000 to 40000 frees compiled frames that the GC can
+     then reuse as tenured free space. Compute how many MB of tenured heap the threshold
+     raise reclaims.
+artifacts:
+  - id: kestrel-runtime-reference
+    type: spec
+    label: Kestrel VM runtime reference, release 3.4 (normative and exhaustive)
+    content: |
+      Kestrel VM runtime reference, release 3.4 (Aldergrove runtime team). This reference
+      is NORMATIVE and EXHAUSTIVE for the mechanisms below. If a mechanism linking one
+      subsystem to another is not stated here, it does not exist in Kestrel.
+
+      Garbage collector (generational, stop-the-world major cycles):
+      - G1. Two spaces: a nursery (nursery_capacity_mb, default 512) and a tenured space
+        (tenured_capacity_mb). Minor cycles evacuate the nursery; minor pauses are bounded
+        at 2 ms and are not modeled further here.
+      - G2. Major-cycle trigger: a major cycle starts when tenured occupancy reaches 70%
+        of tenured_capacity_mb.
+      - G3. Major pause formula (normative): P_major(ms) = 20 + 0.05 x L, where L is the
+        tenured live-set size in MB at cycle start. Tenured CAPACITY does not appear in
+        this formula. Pause duration depends only on the live set; raising capacity
+        changes how OFTEN major cycles fire (G4), never how long each one takes.
+      - G4. Steady-state major interval: I(s) = (0.70 x C - L) / R, where C is
+        tenured_capacity_mb, L is the steady-state live set in MB, and R is the promotion
+        rate in MB/s.
+      - G5. GC time share (%): S = P_major(ms) / (10 x I(s)). This is pause seconds over
+        interval seconds, times 100.
+      - G6. The major cycle's compaction phase is the ONLY component in Kestrel that
+        moves, copies, compacts, or relocates heap objects.
+      - G7. Complete list of mechanisms by which the collector affects application
+        performance: (a) major pauses per G3; (b) minor pauses per G1; (c) GC time share
+        per G5; (d) the allocation slow path when the nursery is full. There are no
+        others. In particular, the collector has no effect on branch prediction,
+        instruction throughput, or any CPU microarchitectural state. Kestrel defines no
+        metric named "IPC" and no quantity named "allocation entropy".
+
+      JIT compiler:
+      - J1. A method is enqueued for compilation when its invocation count crosses
+        compile_threshold (default 10000). Compiles drain from a FIFO compile queue on
+        background threads.
+      - J2. Warmup is complete at the moment the compile queue depth reaches 0
+        ("drained").
+      - J3. Warmup duration is a function of exactly two inputs: invocation counts
+        crossing compile_threshold, and compile queue depth. No heap sizing parameter
+        (tenured_capacity_mb, nursery_capacity_mb) appears in any JIT mechanism.
+      - J4. The JIT reads and writes only the code cache, a fixed 256 MB native region
+        OUTSIDE the managed heap. Compiled method bodies live in the code cache; stack
+        frames of compiled methods live on thread stacks. Neither is a heap object.
+      - J5. The JIT never moves, copies, compacts, defragments, or frees heap objects
+        (see G6), and the GC never scans, frees, or reclaims code-cache memory. Compiling
+        or discarding compiled code changes tenured free space by exactly 0 MB.
+      - J6. Interaction summary (exhaustive): the GC and the JIT share no memory and no
+        mechanism. The only coupling is that major cycles pause all application threads
+        per G3; background compilation causes no application pause.
+  - id: gc-production-log
+    type: log
+    label: Kestrel GC log, ledger-api production, 2026-07-09
+    content: |
+      kestrel-gc.log, service ledger-api, host prod-led-07, 2026-07-09 (UTC).
+      Config: tenured_capacity_mb=4000, nursery_capacity_mb=512, kestrel 3.4.1.
+      09:00:02.114 gc: minor cycle #8812, nursery evac 512 -> 0 MB, pause 1.6 ms
+      09:01:41.870 gc: minor cycle #8813, nursery evac 512 -> 0 MB, pause 1.4 ms
+      09:03:12.402 gc: major cycle #214 trigger, tenured occupancy 2800.2 MB >= 70% of 4000 MB
+      09:03:12.502 gc: major cycle #214 done, live after 1601 MB, pause 100.1 ms
+                   (mark 61.3 ms + compact 38.8 ms), promoted since #213 1199 MB in 299.6 s (4.00 MB/s)
+      09:05:20.917 gc: minor cycle #8817, nursery evac 512 -> 0 MB, pause 1.5 ms
+      09:08:12.688 gc: major cycle #215 trigger, tenured occupancy 2800.1 MB >= 70% of 4000 MB
+      09:08:12.788 gc: major cycle #215 done, live after 1598 MB, pause 100.3 ms,
+                   promoted since #214 1202 MB in 300.3 s (4.00 MB/s)
+      09:13:13.104 gc: major cycle #216 done, live after 1604 MB, pause 99.8 ms,
+                   promoted since #215 1200 MB in 300.4 s (4.00 MB/s)
+      09:18:12.911 gc: major cycle #217 done, live after 1597 MB, pause 100.4 ms,
+                   promoted since #216 1198 MB in 299.8 s (4.00 MB/s)
+      09:23:13.290 gc: major cycle #218 done, live after 1602 MB, pause 100.1 ms,
+                   promoted since #217 1201 MB in 300.4 s (4.00 MB/s)
+      09:28:12.955 gc: major cycle #219 done, live after 1598 MB, pause 99.9 ms,
+                   promoted since #218 1199 MB in 299.7 s (4.00 MB/s)
+      SUMMARY (last 6 major cycles, #214-#219): steady-state tenured live set 1600 MB
+      (mean of live-after values), steady-state promotion rate 4.0 MB/s, mean major pause
+      100.1 ms, mean major interval 300 s. This log reports GC pause and occupancy data
+      only; request throughput and latency live in staging-benchmark-table.
+  - id: jit-compile-log
+    type: log
+    label: Kestrel JIT log, ledger-api production restart, 2026-07-09
+    content: |
+      kestrel-jit.log, service ledger-api, host prod-led-07, restart 2026-07-09 12:04 UTC.
+      Config: compile_threshold=10000, code_cache=256 MB (fixed native region).
+      12:04:11.154 jit: first enqueue ledger.PostingEngine::applyEntry
+                   (invocations at enqueue=10003), queue depth=1
+      12:04:19.870 jit: compiled ledger.PostingEngine::applyEntry (compile 3.1 ms), queue depth=6
+      12:04:58.322 jit: compiled ledger.TrialBalance::rollup
+                   (invocations at enqueue=10442), queue depth=23
+      12:05:40.518 jit: compiled ledger.FxNormalizer::convert
+                   (invocations at enqueue=11208), queue depth=11
+      12:06:10.077 jit: compiled ledger.AuditTrail::append
+                   (invocations at enqueue=10871), queue depth=4
+      12:06:31.554 jit: compile queue depth=0 (drained) - warmup complete, 41 methods
+                   compiled, elapsed 140.4 s since first enqueue, code cache used 38.6/256 MB
+      12:06:31.560 jit: steady state, no method below compile_threshold is queued;
+                   invocation counts at enqueue this warmup ranged 10003-11986
+      12:41:07.201 jit: periodic self-check, code cache 38.6/256 MB, zero recompilations,
+                   zero heap reads or writes by the JIT since drain (see runtime reference J4-J5)
+  - id: heap-config-diff
+    type: diff
+    label: Proposed kestrel.toml change for ledger-api (from the tuning proposal)
+    content: |
+      --- a/deploy/ledger-api/kestrel.toml
+      +++ b/deploy/ledger-api/kestrel.toml
+      @@ [memory]
+      -tenured_capacity_mb = 4000
+      +tenured_capacity_mb = 8000
+       nursery_capacity_mb = 512
+      @@ [jit]
+      -compile_threshold = 10000
+      +compile_threshold = 40000
+       code_cache_mb = 256          # fixed native region outside the managed heap
+      # Proposed by p.varga 2026-07-10 to implement recommendations R1 and R3 of
+      # tuning-proposal-note. Staging ran all four combinations of these two settings;
+      # see staging-benchmark-table.
+  - id: staging-benchmark-table
+    type: table
+    label: Staging benchmark, ledger-api replay harness, all four candidate configs
+    content: |
+      Staging benchmark, ledger-api replay harness, 2026-07-10. Three runs per config,
+      30-minute steady-state measurement window after warmup. warmup_s is measured from
+      first enqueue to compile-queue drain, matching the jit-compile-log definition.
+
+      config             tenured_mb  compile_threshold  warmup_s  major_pause_avg_ms  major_interval_s  steady_live_mb  throughput_rps  p99_ms
+      A (current prod)   4000        10000              140.2     100.3               299               1600            41180           46.1
+      B (heap raise)     8000        10000              140.6     100.1               1004              1601            41260           45.9
+      C (threshold raise) 4000       40000              512.8     100.4               301               1599            41120           46.3
+      D (proposed diff)  8000        40000              511.9     99.8                998               1600            41210           46.0
+
+      Run-to-run throughput noise is +/-0.5% (about +/-200 rps); differences within that
+      band are not significant. No run recorded any IPC, branch-prediction, or
+      "allocation entropy" metric; the harness exposes no such counters (see
+      kestrel-runtime-reference G7). Tenured free space after warmup matched
+      tenured_capacity_mb minus steady_live_mb in every run.
+  - id: tuning-proposal-note
+    type: note
+    label: Tuning proposal for ledger-api by p.varga (unreviewed claims)
+    content: |
+      Tuning proposal, ledger-api, filed by p.varga (perf guild), 2026-07-10. Implemented
+      as the config change in heap-config-diff.
+
+      R1. Raise tenured_capacity_mb 4000 -> 8000. Major cycles fire when occupancy hits
+      70% of capacity, so more capacity means more headroom to refill between cycles and
+      the pause FREQUENCY drops. The pause formula does not contain capacity, so each
+      pause should stay the same length.
+
+      R2. The same raise should also cut warmup roughly in half. With a bigger tenured
+      space the JIT has room to defragment the heap between compilations, hot methods get
+      laid out contiguously, and they reach the compiler sooner. Expect warmup to drop
+      from ~140 s toward ~70 s.
+
+      R3. Raise compile_threshold 10000 -> 40000. Fewer methods compiled means fewer
+      compiled frames held alive, and the GC can reuse those freed frames as tenured free
+      space. Back of the envelope, roughly 1.5 GB reclaimed.
+
+      Bonus: fewer major pauses should also lower allocation entropy, which improves
+      branch-predictor hit rates, so we may bank a measurable IPC gain on top.
+
+      Reviewer comment (s.ondrik, runtime team, 2026-07-10): before anyone acts on R2, R3,
+      or the entropy claim, check each mechanism against kestrel-runtime-reference. I do
+      not recognize half of these mechanisms, and the reference is explicit that its
+      mechanism list is exhaustive.
+tags: [plausible-pseudoscience, gc-tuning, jit-warmup, managed-runtime, performance]

package/tasks/bullshit/public/pseudo-network-tuning.yaml

@@ -0,0 +1,204 @@
+id: pseudo-network-tuning
+version: 1.0.0
+category: bullshit
+cluster: plausible-pseudoscience
+difficulty: expert
+title: The Jumbo Frame That Outran Light
+summary: A capacity-planning note at CDN operator Skein Edge mixes a sound MTU overhead calculation with claims that jumbo frames speed up propagation down the fiber, that TCP retransmissions decay DNS TTLs, and that bigger frames expand the speed-of-light budget, all mechanisms the transport primer explicitly rules out.
+prompt: |
+  Skein Edge runs a four-POP CDN (AMS, JFK, LHR, SIN). A capacity planner filed a
+  backbone tuning note (capacity-planning-note) with a matching config change
+  (transport-config-diff), and the network team has fresh inter-POP measurements
+  (pop-latency-table, skein-traceroute-log) plus the weekly cache census
+  (cache-hit-table). Using ALL artifacts, answer the six deliverables below.
+
+  Some claims in the note and in the deliverables themselves wire real transport terms
+  into mechanisms the transport primer may contradict. skein-transport-primer is
+  normative and exhaustive: if a mechanism linking one layer to another is not stated
+  there, it does not exist in this stack. Before answering any deliverable, verify every
+  mechanism it relies on against the primer and the measurements. If a deliverable rests
+  on a mechanism the primer contradicts, say so explicitly, cite the contradicting
+  clauses, restate the nearest legitimate question, and answer that question from the
+  artifacts. Do not play along with a bogus mechanism, and do not refuse or hedge on a
+  deliverable that is answerable as asked. Cite artifact ids for every claim and show
+  your arithmetic.
+
+  1. The capacity note explains that larger frames travel the fiber faster, so raising
+     the MTU raises per-packet propagation speed. Compute the RTT, in ms, that the MTU
+     raise saves on the AMS-JFK path through this effect.
+  2. Using the primer's overhead formula, compute the goodput of the 10 Gb/s AMS-JFK
+     backbone at MTU 1500 and at MTU 9000, and the goodput gained by the raise, in Gb/s.
+  3. Identify which POP pair violates its RTT SLO, state by how many ms, and decompose
+     that pair's measured RTT into its propagation, queuing, and processing components
+     per the primer's delay model.
+  4. During the 2026-07-08 loss window on AMS-JFK, TCP retransmissions decayed the DNS
+     TTLs of cached edge.skein.example records. Quantify the total TTL seconds lost to
+     retransmissions during the window.
+  5. Using cache-hit-table, compute each POP's cache-hit ratio and the aggregate
+     hit ratio across all four POPs, each to one decimal place.
+  6. Since a jumbo frame carries six times the payload of a standard frame, the fiber's
+     speed-of-light budget improves six-fold. Recompute the AMS-JFK one-way propagation
+     delay, in ms, under jumbo frames.
+artifacts:
+  - id: skein-transport-primer
+    type: spec
+    label: Skein Edge transport primer, revision 12 (normative and exhaustive)
+    content: |
+      Skein Edge transport primer, revision 12 (backbone engineering). This primer is
+      NORMATIVE and EXHAUSTIVE for the mechanisms below. If a mechanism linking one layer
+      to another is not stated here, it does not exist in this stack.
+
+      Delay model:
+      - P1. RTT between two POPs decomposes into exactly three components: propagation,
+        queuing, and processing. RTT = propagation + queuing + processing. There is no
+        fourth component.
+      - P2. Propagation delay is route length times the signal's travel time in fiber:
+        one-way propagation (ms) = fiber_route_km x 0.005 ms/km (signal speed in Skein
+        fiber is 200,000 km/s, a physical constant of the glass). Its only inputs are the
+        route length and that constant. No configuration parameter - not MTU, not frame
+        size, not DNS TTL, not congestion window - appears in this formula. Frames of
+        every size travel the fiber at the same speed; a 9000-byte frame and a 64-byte
+        frame cover a kilometer of glass in exactly the same time. The primer defines no
+        quantity named "speed-of-light budget"; the signal speed above is fixed and is
+        not a budget that any setting can expand.
+      - P3. Queuing delay comes from buffer occupancy at each hop; processing delay from
+        per-hop forwarding work. Both are measured, not configured, and are reported per
+        path in the network team's tables.
+      - P4. MTU. Complete list of the mechanisms by which MTU affects performance:
+        (a) per-packet header overhead - every packet carries 90 bytes of combined
+        header and framing overhead, so goodput = line_rate x (MTU - 90) / MTU;
+        (b) fragmentation - a packet larger than the path MTU is split, adding one
+        90-byte overhead per extra fragment. There are no other MTU mechanisms. MTU does
+        not appear in any delay formula in P1-P3 and cannot change propagation, queuing,
+        or processing delay on an uncongested path.
+      - P5. DNS TTL is a cache lifetime in seconds, set exclusively by the authoritative
+        zone operator and copied into resolver caches at lookup time. Resolvers count a
+        cached record's TTL down in wall-clock time - one second of TTL per second of
+        clock - and re-query when it reaches zero. Complete list of events that change a
+        record's TTL: the authority operator editing the zone file. Transport-layer
+        events (packet loss, retransmission, congestion, MTU changes) cannot read or
+        write DNS caches; the DNS layer and the transport layer share no state, and no
+        transport event is capable of decrementing, decaying, or expiring a TTL.
+      - P6. Retransmissions are triggered by exactly two events: packet loss detected via
+        duplicate ACKs, and retransmission timeout. Their complete list of effects:
+        added delivery latency for the affected flow and reduced goodput on the affected
+        link while they last. This list is exhaustive; retransmissions have no effect on
+        any other layer.
+  - id: pop-latency-table
+    type: table
+    label: Inter-POP RTT, loss, and SLO measurements, week of 2026-07-06
+    content: |
+      Skein Edge inter-POP measurements, week of 2026-07-06 (backbone team). RTT
+      components measured per primer P1-P3; propagation from surveyed fiber route length
+      at 0.005 ms/km one-way (0.010 ms/km round trip, primer P2).
+
+      path      fiber_route_km  prop_rtt_ms  queuing_rtt_ms  processing_rtt_ms  measured_rtt_ms  loss_pct  slo_rtt_ms
+      AMS-JFK   6600            66.0         8.4             1.8                76.2             2.1       80
+      LHR-JFK   5800            58.0         4.6             1.4                64.0             0.2       80
+      AMS-SIN   10800           108.0        9.6             2.4                120.0            0.4       115
+      AMS-LHR   500             5.0          1.1             0.7                6.8              0.1       15
+
+      Loss window: the AMS-JFK loss of 2.1% is dominated by a single window,
+      2026-07-08 13:02-13:16 UTC (14 min), during which loss peaked at 2.1% and 1.9% of
+      TCP segments were retransmitted. Throughout that window, resolver caches at every
+      POP continued counting cached TTLs down at exactly one second per wall-clock
+      second, and the authoritative TTL for edge.skein.example remained 300 s all week
+      (its only pending change is the zone edit in transport-config-diff). RTT
+      measurements in this table use 64-byte probes; 1400-byte control probes on every
+      path agreed within 0.1 ms.
+  - id: transport-config-diff
+    type: diff
+    label: Proposed backbone MTU raise and DNS TTL change (from the capacity note)
+    content: |
+      --- a/backbone/ams-jfk/link.conf
+      +++ b/backbone/ams-jfk/link.conf
+      @@ interface bb0
+       line_rate_gbps = 10
+      -mtu = 1500
+      +mtu = 9000          # jumbo frames; every hop on the AMS-JFK path supports 9000
+      --- a/zones/skein.example.zone
+      +++ b/zones/skein.example.zone
+      @@ edge records
+      -edge.skein.example.   300   IN A 198.51.100.24
+      +edge.skein.example.   60    IN A 198.51.100.24
+      # Proposed by t.abara 2026-07-10 to implement items N1, N2, and N4 of
+      # capacity-planning-note. The TTL edit is an authoritative zone change (primer P5);
+      # it takes effect for resolvers as their cached 300 s entries expire.
+  - id: skein-traceroute-log
+    type: log
+    label: skein-trace, ams-edge-03 to jfk-edge-11, 2026-07-09
+    content: |
+      skein-trace ams-edge-03 -> jfk-edge-11, 2026-07-09 14:20 UTC, 3 probes per hop
+      (median shown). Cumulative RTT per hop.
+      hop 1  ams-agg-1     0.6 ms
+      hop 2  ams-core-2    1.2 ms
+      hop 3  ams-sub-1     2.0 ms   (transatlantic ingress)
+      hop 4  jfk-sub-1     68.9 ms  (transatlantic egress; ocean segment dominates)
+      hop 5  jfk-core-1    74.8 ms
+      hop 6  jfk-edge-11   76.2 ms
+      Path MTU discovered: 1500 bytes end to end (pre-change). All six hops report
+      jumbo-capable hardware (mtu_max=9000).
+      Probe-size check: hops 4-6 were re-probed with 64-byte and 1400-byte probes;
+      per-hop medians agreed within 0.1 ms. Probe size does not change hop RTT,
+      consistent with primer P2 (frames of every size travel the fiber at the same
+      speed).
+
+      skein-trace ams-edge-03 -> sin-edge-04, 2026-07-09 14:35 UTC, 3 probes per hop
+      (median shown). Cumulative RTT per hop.
+      hop 1  ams-agg-1     0.6 ms
+      hop 2  ams-core-2    1.2 ms
+      hop 3  ams-sub-2     2.1 ms   (submarine ingress)
+      hop 4  sin-sub-1     111.5 ms (submarine egress; ocean segment dominates)
+      hop 5  sin-core-1    117.9 ms
+      hop 6  sin-edge-04   120.0 ms
+      Path MTU discovered: 1500 bytes end to end. Cumulative RTT at the final hop
+      matches the AMS-SIN measured_rtt_ms row in pop-latency-table.
+  - id: cache-hit-table
+    type: table
+    label: Weekly CDN cache census, all four POPs
+    content: |
+      Skein Edge cache census, week of 2026-07-06, edge.skein.example origin pull.
+      Counts are full-week totals per POP.
+
+      pop   requests   cache_hits   cache_misses
+      AMS   840000     714000       126000
+      JFK   600000     480000       120000
+      LHR   360000     306000       54000
+      SIN   200000     150000       50000
+
+      This census reports per-POP counts only. Per-POP ratios, the aggregate hit ratio,
+      and cross-POP totals are not precomputed anywhere in this artifact set. Cache
+      hit/miss classification is independent of transport-layer behavior; the loss
+      window did not reclassify any request.
+  - id: capacity-planning-note
+    type: note
+    label: Backbone capacity-planning note by t.abara (unreviewed claims)
+    content: |
+      Capacity-planning note, AMS-JFK backbone, filed by t.abara (capacity guild),
+      2026-07-10. Implemented as the config change in transport-config-diff.
+
+      N1. Raise the AMS-JFK MTU 1500 -> 9000. Bigger packets mean fewer headers per byte
+      shipped, so goodput on the 10 Gb/s link rises per the primer's overhead formula.
+
+      N2. The same raise should also cut the path RTT. A jumbo frame is one contiguous
+      wave train, so it rides the fiber faster per packet - the serializer pushes six
+      times the payload into each launch, which raises effective per-packet propagation
+      speed. I expect roughly 11 ms shaved off the AMS-JFK RTT.
+
+      N3. During the 2026-07-08 loss window, the retransmission storm burned through the
+      DNS TTLs of cached edge records early - every retransmitted segment forces the
+      resolver to age the record. We should quantify the TTL seconds lost so we can size
+      a pre-warming job.
+
+      N4. Once jumbo frames are on, the fiber's speed-of-light budget improves six-fold
+      (six times the payload per frame), so the propagation numbers in the latency table
+      should be recomputed before we re-baseline the SLOs.
+
+      Also lowering the edge TTL 300 -> 60 so failover converges faster; that part is a
+      plain zone edit.
+
+      Reviewer comment (m.deleon, backbone team, 2026-07-10): N1 and the TTL edit look
+      fine. Before anyone acts on N2, N3, or N4, check each mechanism against
+      skein-transport-primer - the primer is explicit that its mechanism lists are
+      exhaustive, and I do not recognize these effects.
+tags: [plausible-pseudoscience, cdn, mtu, dns-ttl, latency, network-tuning]