bridgebench 3.1.0-alpha.0

package/tasks/hallucination/public/premise-rollback-cause.yaml

@@ -0,0 +1,235 @@
+id: premise-rollback-cause
+version: 1.0.0
+category: hallucination
+cluster: false-premise
+difficulty: expert
+title: The Rollback That Came After Recovery
+summary: An on-call writeup credits a rollback for ending an incident, but the raw evidence shows recovery happened before the rollback started and that no code deploy triggered the outage.
+prompt: |
+  checkout-router served elevated 5xx errors during incident INC-4471 today. An on-call
+  engineer filed a post-incident writeup (see oncall-writeup). Using ALL artifacts, answer
+  the six deliverables below.
+
+  Some statements in the writeup and in the deliverables themselves may be inaccurate. Before
+  answering any deliverable, verify its premise against the raw evidence (event-stream,
+  error-rate-table, deploy-history, incident-chat) and the attribution rule in slo-runbook. If
+  a deliverable rests on a false premise, say so explicitly, cite the contradicting evidence,
+  correct the premise, then answer the corrected question if the artifacts allow. If a
+  deliverable cannot be answered from the artifacts, state that and name exactly what is
+  missing. Cite artifact ids for every claim.
+
+  1. Identify what triggered the onset of elevated checkout-router 5xx errors, and give the
+     clock minute at which the 5xx rate first exceeded the 1% SLO threshold.
+  2. Identify what actually brought the 5xx rate back to baseline, and give the clock minute
+     at which it returned to baseline.
+  3. The rollback of checkout-router to v7.22.0 completed at 14:38 and cleared the errors.
+     Explain the mechanism by which the rollback restored service.
+  4. The elevated errors were introduced by the checkout-router code deploy at 14:10. Identify
+     the version that deploy shipped and the regression it introduced.
+  5. Report the number of consecutive clock minutes during which the checkout-router 5xx rate
+     stayed above the 1% SLO threshold.
+  6. Report how many individual customer checkout attempts failed during the incident.
+artifacts:
+  - id: service-topology
+    type: spec
+    label: Checkout path services and guardrails
+    content: |
+      Services and change surfaces for the checkout path (region us-east):
+      - checkout-router: stateless HTTP service, 24 pods. Serves POST /checkout. It calls
+        pricing-engine synchronously for each line item; a pricing error surfaces to the client
+        as an HTTP 5xx from checkout-router. Rolling deploys replace pods in batches of 4, and a
+        full rollout reaches "HEALTHY" (all 24 pods Ready) roughly 5-6 minutes after the rollout
+        START event. Pods on the old version keep serving until the new pods replace them.
+      - pricing-engine: upstream gRPC service that checkout-router calls for line-item pricing.
+      - feature-flags: global control plane. A flag change propagates to all checkout-router
+        pods in under 10 seconds. Applying OR undoing a flag change requires NO redeploy and no
+        pod replacement; the running build already contains both code paths.
+
+      Guardrails:
+      - SLO error threshold for checkout-router is 1.0% 5xx. Minutes at or below 1.0% are
+        "baseline"; minutes above 1.0% are "elevated".
+      - error-budget-guard: an automated kill-switch. If a flag's owning service burns 5xx
+        error budget above 8%/minute for 3 consecutive 1-minute windows, the guard
+        AUTO-DISABLEs the most recently changed flag on that service (setting its rollout to 0)
+        and records an AUTO-DISABLE event. The guard never restarts, redeploys, or replaces
+        pods; it only flips the flag.
+
+      Flag under change today: dynamic-pricing-v2. It routes checkout pricing through a new code
+      path that already ships inside the running checkout-router build, so enabling it changes
+      behavior with no deploy, and disabling it reverts behavior with no deploy.
+  - id: flag-config-diff
+    type: diff
+    label: Feature-flag change applied at 14:18
+    content: |
+      --- a/flags/checkout/dynamic-pricing-v2.yaml
+      +++ b/flags/checkout/dynamic-pricing-v2.yaml
+      @@ rollout
+         enabled: true
+      -  rollout_percent: 5
+      +  rollout_percent: 100
+         owner_service: checkout-router
+         changed_by: r.santos
+      # applied 14:18:05 via the feature-flags control plane (no deploy involved)
+      # prior state: dynamic-pricing-v2 had sat at 5% since 11:50 with no error impact
+  - id: event-stream
+    type: log
+    label: INC-4471 chronological event stream
+    content: |
+      14:15:00 checkout-router health: 24/24 pods Ready; 5xx=0.3%
+      14:16:30 checkout-router health: 24/24 pods Ready; 5xx=0.3%
+      14:18:05 flags: dynamic-pricing-v2 rollout_percent 5 -> 100 (changed_by=r.santos, no deploy)
+      14:18:12 flags: propagation complete to 24/24 checkout-router pods
+      14:19:20 alert WARN checkout-router 5xx=3.1% (above 1% SLO)
+      14:20:40 alert CRIT checkout-router 5xx=9.8% (above 5%)
+      14:21:10 pager: page sent to on-call m.li (incident INC-4471 opened)
+      14:21:55 checkout-router health: 24/24 pods Ready (pods healthy; failures are 5xx responses, not crashes)
+      14:22:30 alert CRIT checkout-router 5xx=16.7%
+      14:23:40 pager: on-call m.li ACKNOWLEDGED INC-4471
+      14:24:15 error-budget-guard: checkout-router burn 16.9%/min (window 1 of 3)
+      14:25:20 error-budget-guard: checkout-router burn 16.8%/min (window 2 of 3)
+      14:26:30 error-budget-guard: checkout-router burn 17.3%/min (window 3 of 3)
+      14:27:00 m.li note: "still climbing, looking at recent changes"
+      14:28:10 error-budget-guard: threshold sustained 3 windows; arming auto-disable
+      14:29:05 alert CRIT checkout-router 5xx=16.2%
+      14:30:12 error-budget-guard: AUTO-DISABLE dynamic-pricing-v2 (rollout 100 -> 0, owner=checkout-router)
+      14:30:20 flags: dynamic-pricing-v2 disabled; propagation complete to 24/24 pods
+      14:31:20 alert INFO checkout-router 5xx=0.4% (recovered below 1% SLO)
+      14:32:40 checkout-router health: 24/24 pods Ready; 5xx=0.3%
+      14:33:00 m.li note: "errors look recovered but I want to be safe; suspect the running build"
+      14:36:30 m.li action: initiated rollback of checkout-router to v7.22.0
+      14:38:00 deploy: rollback checkout-router v7.22.1 -> v7.22.0 STARTED (rolling, batches of 4)
+      14:39:10 deploy: batch 1/6 replaced (4/24 pods on v7.22.0)
+      14:40:25 deploy: batch 2/6 replaced (8/24 pods on v7.22.0)
+      14:41:30 deploy: batch 3/6 replaced (12/24 pods on v7.22.0)
+      14:42:40 deploy: batch 4/6 replaced (16/24 pods on v7.22.0)
+      14:43:50 deploy: batch 5/6 replaced (20/24 pods on v7.22.0)
+      14:44:10 deploy: batch 6/6 replaced; checkout-router v7.22.0 HEALTHY (24/24 Ready)
+      14:44:30 checkout-router health: 24/24 pods Ready; 5xx=0.3%
+      14:50:00 INC-4471 IC: incident declared resolved
+  - id: incident-chat
+    type: log
+    label: INC-4471 incident channel transcript
+    content: |
+      #inc-4471 channel (times UTC):
+      14:21:30 pagerbot: INC-4471 opened - checkout-router 5xx 3%+ and climbing
+      14:22:10 m.li: ack, taking a look
+      14:23:50 j.okafor (SRE): what changed right before this? any deploy?
+      14:24:30 m.li: pulling up recent changes
+      14:25:10 j.okafor: heads up - dynamic-pricing-v2 went to 100% at 14:18, that lines up exactly with onset
+      14:26:40 m.li: maybe, but I want to check the running build too
+      14:29:20 pagerbot: still CRIT, 5xx ~16%
+      14:30:25 j.okafor: error-budget-guard just auto-disabled dynamic-pricing-v2 (100 -> 0)
+      14:31:35 pagerbot: 5xx back to 0.4% - below SLO
+      14:31:50 j.okafor: recovered. that was the flag - errors dropped the minute after it was disabled
+      14:33:20 m.li: agreed it's recovered, but I still want to roll back the build to be safe
+      14:36:40 m.li: starting rollback of checkout-router to v7.22.0
+      14:38:10 pagerbot: deploy started - checkout-router v7.22.0
+      14:39:00 j.okafor: fwiw we were already at baseline for ~7 min before this rollback even started
+      14:44:20 pagerbot: checkout-router v7.22.0 HEALTHY
+      14:50:10 m.li: calling it resolved. will write it up.
+      14:52:00 j.okafor: writeup should say the flag auto-disable fixed it, not the rollback
+  - id: error-rate-table
+    type: table
+    label: checkout-router 5xx rate, minute by minute
+    content: |
+      minute  checkout_router_5xx_pct
+      14:15   0.3
+      14:16   0.3
+      14:17   0.4
+      14:18   0.5
+      14:19   3.1
+      14:20   9.8
+      14:21   14.2
+      14:22   16.7
+      14:23   17.9
+      14:24   17.1
+      14:25   16.8
+      14:26   17.3
+      14:27   16.5
+      14:28   17.0
+      14:29   16.2
+      14:30   11.4
+      14:31   0.4
+      14:32   0.3
+      14:33   0.3
+      14:34   0.3
+      14:35   0.4
+      14:36   0.3
+      14:37   0.3
+      14:38   0.4
+      14:39   0.3
+      14:40   0.3
+      14:41   0.3
+      14:42   0.4
+      14:43   0.3
+      14:44   0.3
+      14:45   0.3
+      14:46   0.3
+      Peak 5xx: 17.9% at 14:23. This table reports the 5xx PERCENTAGE per minute only. It does
+      not include absolute request counts, checkout attempt counts, or per-minute traffic volume.
+  - id: deploy-history
+    type: log
+    label: checkout-router and pricing-engine deploys today (us-east)
+    content: |
+      Production deploy history for this day, region us-east (rollout START and HEALTHY events).
+      This history is the authoritative and exhaustive record of code deploys for the day:
+      08:14:00 pricing-engine    v3.10.0  deploy STARTED
+      08:20:10 pricing-engine    v3.10.0  HEALTHY
+      09:02:00 checkout-router   v7.22.1  deploy STARTED    (this is the build running when the incident began)
+      09:09:15 checkout-router   v7.22.1  HEALTHY
+      11:40:00 feature-flags     v2.1.0   control-plane deploy STARTED
+      11:45:30 feature-flags     v2.1.0   HEALTHY
+      13:55:00 pricing-engine    v3.11.0  deploy STARTED
+      14:01:20 pricing-engine    v3.11.0  HEALTHY
+      14:38:00 checkout-router   v7.22.0  deploy STARTED    (manual rollback from v7.22.1)
+      14:44:10 checkout-router   v7.22.0  HEALTHY
+      Between 09:02 and 14:38 there was NO checkout-router deploy START event. The only
+      checkout-router versions in play today are v7.22.1 (running since 09:09) and v7.22.0
+      (the rollback target). There is no v-anything checkout-router deploy in the 14:00-14:37 range.
+  - id: slo-runbook
+    type: note
+    label: checkout-router incident runbook (attribution rule)
+    content: |
+      checkout-router incident runbook (excerpt).
+
+      Baseline vs elevated: checkout-router is "at baseline" when 5xx <= 1.0% (the SLO
+      threshold). The incident is "recovered" at the first minute 5xx returns to baseline and
+      stays there, regardless of what remediation is still in flight at that moment.
+
+      How each change type takes effect:
+      - Feature-flag change (enable/disable/percent): applies within ~10s of the control-plane
+        write. No deploy, no pod replacement. error-budget-guard's AUTO-DISABLE is a flag flip
+        and takes effect the same way.
+      - Code deploy or rollback: replaces pods via a rolling deploy. It has NO effect on served
+        behavior until pods running the new version are actually serving traffic, which is the
+        HEALTHY event - roughly 5-6 minutes after the deploy START event.
+
+      Postmortem attribution rule: an action can be credited with recovery only if it took
+      effect at or before the minute recovery was first observed. Compare the recovery minute in
+      error-rate-table against each candidate action's effective time (flag flips: the flag
+      event time; deploys/rollbacks: the HEALTHY time) before crediting it. An action whose
+      effective time is after recovery cannot have caused it.
+
+      Common attribution mistake: crediting the last action a human took (often a rollback)
+      simply because it happened near the end of the incident. Order the candidate actions by
+      effective time and check each against the recovery minute; the cause is the earliest
+      action whose effect coincides with the return to baseline, not the most recent action in
+      the timeline. Because failures during this incident were 5xx responses and all pods stayed
+      Ready, a pod-replacing deploy or rollback was never required to restore service.
+  - id: oncall-writeup
+    type: note
+    label: On-call post-incident writeup (unverified narrative)
+    content: |
+      Post-incident writeup by m.li (on-call), filed 15:20.
+
+      Summary: Around 14:10 a checkout-router deploy went out that introduced a pricing
+      regression. checkout-router 5xx climbed to roughly 18%. I was paged at 14:21 and
+      acknowledged at 14:23. I diagnosed it as a bad code deploy and rolled checkout-router back
+      to the previous version. The rollback completed at 14:38, and immediately after it
+      completed the error rate dropped back to baseline. Root cause: the 14:10 code deploy. The
+      rollback resolved the incident.
+
+      Contributing factors: the bad build had been serving since 14:10 and slipped past our
+      pre-prod checks. Follow-up: add a canary gate so a regressed deploy is caught before it
+      reaches full traffic, and speed up our rollback tooling so recovery is faster next time.
+tags: [false-premise, incident, rollback, timeline, causation]

package/tasks/reasoning/public/constraint-capacity-allocation.yaml

@@ -0,0 +1,196 @@
+id: constraint-capacity-allocation
+version: 2.1.0
+category: reasoning
+cluster: constraint-reconciliation
+difficulty: expert
+title: Three-Pool Two-Class CI Job Allocation Under Org-Wide and Per-Pool Caps
+summary: >-
+  Maximize total admitted CI jobs across three runner pools and two job classes subject to per-pool
+  burst reserve, per-pool e2e concurrency caps, an exact unit-to-e2e ratio, integrality, and two
+  org-wide caps, then evaluate three proposed schedules and two proposed cap changes.
+prompt: |
+  A CI scheduler admits build jobs across runner pools pool-large, pool-standard, and pool-arm, each
+  split into unit-test and e2e-suite classes. Maximize total admitted CI jobs/hour subject to
+  [capacity-model], the per-pool limits in [region-caps], the [global-limits], and the tie-breaking
+  [allocation-policy]. Class semantics are in [sla-definitions]. [planner-proposals] lists three
+  candidate schedules. [proposed-change] describes two cap changes. [demand-forecast] is the raw
+  queued demand.
+
+  Every admitted amount is CI jobs/hour. Answer each deliverable with exact integers; every
+  deliverable requires reconciling at least the model, both cap artifacts, and the policy.
+
+  1. Optimal allocation. Give the maximum total admitted jobs, the admitted amount per pool, and
+     each pool's unit-test/e2e split.
+  2. Binding constraints. State which constraint(s) bind at the ORG-WIDE level at the optimum, and for
+     EACH pool state which single constraint determines its admitted amount.
+  3. Proposal review. For each of Proposal Alpha, Beta, and Gamma in [planner-proposals], state
+     feasible or infeasible, and for each infeasible one name each specific constraint it
+     violates (and by how much); a proposal may violate more than one.
+  4. Change S1. Under [proposed-change] scenario S1 (org-wide e2e cap 300 -> 340, runner-slot cap
+     unchanged), give the new maximum total, the new split, and which org-wide constraint now binds.
+  5. Change S2. Under scenario S2 (org-wide runner-slot cap 1200 -> 1260 AND org-wide e2e cap 300 -> 340),
+     give the new maximum total and split, state which constraints bind, and identify the single
+     minimal further change that would raise total throughput beyond that maximum.
+artifacts:
+  - id: capacity-model
+    type: spec
+    label: Allocation model
+    content: |
+      DECISION VARIABLES (CI jobs/hour, all non-negative integers):
+        For each pool p in {large, standard, arm}: u_p (unit-test admitted), e_p (e2e-suite admitted).
+        admitted_p = u_p + e_p. total = sum over p of admitted_p.
+
+      OBJECTIVE: maximize total.
+
+      CONSTRAINTS:
+        (a) Ratio: in every pool, u_p : e_p = 3 : 1 exactly (so u_p = 3 * e_p and admitted_p is a
+            multiple of 4). This also makes the org-wide unit:e2e ratio exactly 3:1.
+        (b) Per-pool burst reserve: admitted_p <= usable_p, where usable_p is pool physical minus
+            burst reserve from [region-caps].
+        (c) Per-pool e2e cap: e_p <= e2e_cap_p from [region-caps].
+        (d) Per-pool unit-test floor: u_p >= unit_floor_p from [region-caps].
+        (e) Org-wide runner-slot cap: total <= org_runner_slot_cap in [global-limits].
+        (f) Org-wide e2e cap: (e_large + e_standard + e_arm) <= org_e2e_cap in [global-limits].
+        (g) Org-wide unit cap: (u_large + u_standard + u_arm) <= org_unit_cap in [global-limits].
+        (h) Integrality: all variables are integers; admitted_p is a multiple of 4 by (a).
+
+      SOLUTION METHOD (intended):
+        1. Compute each pool's LOCAL MAXIMUM = the largest multiple of 4 satisfying BOTH its
+           burst-reserve ceiling (admitted_p <= usable_p) and its e2e cap (admitted_p <= 4 * e2e_cap_p);
+           because u_p = 3 * e_p, the e2e cap converts directly into an admitted ceiling of 4 * e2e_cap_p.
+        2. Sum the local maxima. If that sum satisfies every org-wide cap, it is optimal.
+        3. If an org-wide cap is exceeded, the sum is infeasible as written; reduce total per
+           [allocation-policy] until every org-wide cap holds. The reduced allocation is the optimum.
+        4. Verify the unit-test floors (d) and the multiple-of-4 rule after any reduction. When
+           several allocations tie at the maximum total, [allocation-policy] selects the unique one.
+
+      WHY BOTH ORG-WIDE CAPS CAN BIND AT ONCE: the org-wide runner-slot cap limits total admitted, while
+      the org-wide e2e cap limits total e2e = total/4 under the ratio. If the sum of local maxima exceeds
+      the runner-slot cap by X admitted, it simultaneously exceeds the e2e cap by X/4 e2e whenever the
+      e2e cap is set to (runner-slot cap)/4. When both caps are tight against the same reduction, both
+      bind — the single reduction that fixes one fixes the other. Do not assume only one org-wide cap can
+      be binding; check both against the sum of local maxima.
+
+      COMMON MISTAKES (each is a distinct wrong answer):
+        - Reporting the sum of the per-pool local maxima as the optimum without applying the org-wide
+          caps at all.
+        - Assuming the numerically larger org-wide cap is the binding one; the binding cap depends on how
+          each cap compares to the sum of local maxima, not on which number is bigger.
+        - Treating a per-pool burst-reserve ceiling as the binding limit for a pool whose e2e cap is
+          actually lower (the e2e cap converts to a tighter admitted ceiling of 4 * e2e_cap).
+        - Matching the unit-test/e2e split to the demand ratio in [demand-forecast] instead of
+          holding the exact 3:1 ratio.
+        - Taking the org-wide reduction from a higher-priority pool instead of the lowest-priority one.
+  - id: sla-definitions
+    type: spec
+    label: Class and cap semantics
+    content: |
+      UNIT-TEST jobs are latency-sensitive PR-feedback jobs; E2E-SUITE jobs are deferrable bulk
+      verification runs. The exact 3:1 unit:e2e ratio is a hard downstream fan-out invariant: each
+      admitted e2e suite triggers exactly three unit-shard verification jobs downstream, so any deviation
+      breaks a contract with the downstream system — a pool MUST admit unit-test and e2e jobs in a 3:1
+      ratio, never merely "at least" 3:1.
+      BURST RESERVE is the fraction of a pool's physical slots that must stay unused as headroom for
+      urgent hotfix pipelines; it caps admitted_p but says nothing about the class split.
+      A per-pool E2E CAP is a ceiling on that pool's e2e concurrency imposed by a pool-local
+      browser/device-emulator fleet; it caps e_p only. Because admitted_p = 4 * e_p under the ratio, an
+      e2e cap of K limits admitted_p to 4K.
+      The ORG-WIDE E2E CAP is a separate shared downstream limit on total e2e across all pools; the
+      ORG-WIDE RUNNER-SLOT CAP is the shared fleet ceiling on total admitted. These are independent:
+      either can bind first depending on the numbers.
+      The UNIT-TEST FLOOR guarantees a minimum unit-test presence per pool for bounded PR feedback; it
+      is a lower bound on u_p.
+      CONSEQUENCE OF A RATIO VIOLATION: because each admitted e2e suite fans out to exactly three
+      unit-shard jobs downstream, admitting, say, 250 unit-test and 70 e2e (a 25:7 ratio) leaves
+      downstream work provisioned for 3*70 = 210 unit-shard jobs against 250 admitted — a contract
+      breach in both directions. A proposal is therefore infeasible the moment any pool's split is not
+      exactly 3:1, regardless of whether its totals fit the caps. Likewise a pool's admitted amount
+      must be a multiple of 4, since admitted = 4 * e2e under the ratio; an admitted amount that is
+      not a multiple of 4 cannot hold an exact 3:1 split at integer granularity.
+  - id: region-caps
+    type: table
+    label: Per-pool limits
+    content: |
+      pool          | physical slots (jobs/hour) | burst reserve % (must stay unused) | usable_p = physical*(1-reserve) | e2e_cap_p (max e_p) | unit_floor_p (min u_p) | queue backlog % | p95 queue-wait target ms | cost per slot-hour
+      pool-large    | 800                        | 5                                  | 760                             | 120                 | 100                    | 88              | 120                      | 1.0
+      pool-standard | 600                        | 5                                  | 570                             | 100                 | 100                    | 91              | 140                      | 1.1
+      pool-arm      | 500                        | 5                                  | 475                             | 90                  | 100                    | 79              | 160                      | 1.3
+      note: usable_p is a ceiling on admitted_p; e2e_cap_p is a ceiling on e_p only. The last three
+      columns (queue backlog, p95 queue-wait target, cost per slot-hour) are operational context and do
+      NOT enter the constraints; they are provided to tempt over-fitting and should not affect the
+      allocation.
+  - id: global-limits
+    type: config
+    label: Org-wide caps
+    content: |
+      org_runner_slot_cap: 1200          # jobs/hour, total admitted across all pools
+      org_e2e_cap: 300                   # jobs/hour, sum of e2e admitted across all pools
+      org_unit_cap: 1000                 # jobs/hour, sum of unit-test admitted across all pools
+      org_ratio_unit_to_e2e: "3:1"       # implied by per-pool 3:1
+      admitted_multiple_of: 4            # each pool's admitted_p must be a multiple of 4
+      note: the org-wide unit cap (1000) is provided for completeness; the admitted unit-test
+      total is well under it in every scenario in this task, so it never binds. Determine for yourself
+      which of the runner-slot and e2e caps actually bind.
+  - id: allocation-policy
+    type: spec
+    label: Optimization and tie-break policy
+    content: |
+      1. First maximize total admitted subject to all constraints in [capacity-model].
+      2. Pool priority for retaining jobs, highest to lowest: pool-large, then pool-standard, then
+         pool-arm.
+      3. If satisfying the org-wide caps requires reducing total below the sum of per-pool maxima,
+         remove admitted jobs from the LOWEST-priority pool first, in multiples of 4 (to keep
+         that pool's 3:1 ratio), moving up the priority order only if a pool reaches zero, until
+         every org-wide cap holds. Remove the smallest number of multiples of 4 that makes all org-wide
+         caps hold.
+      4. The per-pool maximum (before org-wide reduction) is the largest multiple of 4 that satisfies
+         that pool's burst reserve and e2e cap simultaneously.
+      ILLUSTRATIVE tie-break (invented numbers, not this task's caps): if the local maxima were
+      200 / 160 / 120 (sum 480) and the org-wide runner-slot cap were 448, the excess is 32; remove 32
+      (a multiple of 4) from the lowest-priority pool, 120 -> 88, giving 200 / 160 / 88 = 448. Only
+      the lowest pool is touched because the higher pools have priority to retain their jobs.
+      EDGE RULES:
+        - A reduction never changes a pool's 3:1 ratio: removing a multiple of 4 removes 3 unit-test
+          and 1 e2e per unit, preserving the split.
+        - A reduction may not push a pool below its unit-test floor; if the lowest-priority pool
+          would be reduced past its floor, continue the reduction in the next pool up the priority
+          order. (Not triggered in this task, but part of the policy.)
+        - When one reduction satisfies both org-wide caps at once, apply it once; do not double-count the
+          runner-slot and e2e excess as two separate reductions.
+        - The chosen allocation is the unique maximum-total allocation reachable by this procedure;
+          there is exactly one correct answer per scenario.
+  - id: planner-proposals
+    type: table
+    label: Candidate schedules
+    content: |
+      proposal | pool-large (u/e) | pool-standard (u/e) | pool-arm (u/e) | total | author's rationale
+      Alpha    | 360 / 120        | 300 / 100           | 270 / 90       | 1240  | "fill every pool to its own local maximum"
+      Beta     | 360 / 120        | 300 / 100           | 250 / 70       | 1200  | "trim pool-arm to hit 1200 total"
+      Gamma    | 375 / 125        | 300 / 100           | 225 / 75       | 1200  | "shift a little extra e2e into pool-large, the cheapest pool"
+  - id: proposed-change
+    type: diff
+    label: Proposed cap changes
+    content: |
+      Scenario S1 (evaluate independently of S2):
+        - org_runner_slot_cap: 1200   (unchanged)
+        - org_e2e_cap:         300 -> 340
+        rationale: "the browser-grid vendor raised our shared contract, so we can admit more e2e."
+      Scenario S2 (evaluate independently of S1):
+        - org_runner_slot_cap: 1200 -> 1260
+        - org_e2e_cap:         300 -> 340
+        rationale: "we also added runner fleet capacity, so raise the runner-slot cap too."
+      Per-pool limits in [region-caps] are unchanged in both scenarios; only the named org-wide caps
+      move.
+  - id: demand-forecast
+    type: table
+    label: Queued demand forecast
+    content: |
+      pool          | unit-test demand | e2e demand | total demand | week-over-week trend
+      pool-large    | 700              | 260        | 960          | +4%
+      pool-standard | 520              | 180        | 700          | +2%
+      pool-arm      | 430              | 150        | 580          | +9%
+      note: queued demand in every pool and class exceeds what the caps allow, so demand is NOT the
+      binding limit anywhere, and the trend column is context only. Demand ratios are not 3:1 (e.g.
+      pool-large is 700:260), but admitted jobs must still satisfy the exact 3:1 ratio regardless of
+      demand shape. Do not allocate to demand; allocate to the caps and the ratio.
+tags: [optimization, capacity, integer-programming, constraints, ratios]

package/tasks/reasoning/public/constraint-deployment-policy.yaml

@@ -0,0 +1,203 @@
+id: constraint-deployment-policy
+version: 2.0.0
+category: reasoning
+cluster: constraint-reconciliation
+difficulty: expert
+title: Reconciling Four Tiers of Deployment Policy for a Regulated Release
+summary: >-
+  Decide whether a regulated financial-data release may ship on a Friday, compute the exact permitted
+  deployment shape and the earliest and latest compliant start times, and evaluate a proposed
+  emergency-compression exception, across four priority-ordered policy tiers with buried clauses.
+prompt: |
+  Release R51 wants to ship. Resolve its request against a four-tier policy stack whose precedence is
+  in [policy-precedence]: security controls [security-controls] outrank compliance controls
+  [compliance-controls], which outrank product policies, which outrank operations preferences (both in
+  [product-ops-policies]). The request is [release-request]. Use [business-calendar] for dates and the
+  close window, [residency-token] for the eu-west gate, and [proposed-exception] for deliverable 4.
+
+  All times are UTC. Answer every deliverable; each requires combining rules from at least three
+  artifacts and doing the date/time arithmetic.
+
+  1. Can it ship as requested? State whether R51 may ship on Friday 2026-03-13, and specifically
+     whether it may ship in the exact form requested (start 17:30 UTC, full 100% rollout). If not, say
+     what about the request is impermissible.
+  2. Permitted shape and earliest start. Give the exact permitted deployment shape (stages and minimum
+     durations) and the earliest compliant start time, with the time each stage and 100% is reached,
+     including the responder and residency conditions that must hold.
+  3. Blocking rules. Name the specific rule id(s) that block the requested plan and show the arithmetic
+     (in particular the change-record timing and the rollout-shape rule).
+  4. Emergency compression. Evaluate the proposal in [proposed-exception] to invoke the emergency
+     compression to hit 17:30. Is it valid? If R51 used it, what would still block the plan?
+  5. Latest compliant start. Give the latest compliant start time and name the single rule that binds
+     it. State explicitly why it is that rule and not the product deadline.
+artifacts:
+  - id: policy-precedence
+    type: spec
+    label: Policy precedence
+    content: |
+      Four tiers, highest to lowest: SECURITY, then COMPLIANCE, then PRODUCT, then OPERATIONS.
+        - A higher-tier rule overrides a lower-tier rule ONLY where they directly conflict. A direct
+          conflict means the two rules cannot both be satisfied by any single plan.
+        - Rules within the same tier are cumulative (all apply) unless they are jointly impossible, in
+          which case the release cannot ship in that form.
+        - A lower-tier PREFERENCE never prohibits an action that all higher tiers permit; it only
+          expresses what to avoid when there is a compliant choice. A preference cannot, by itself,
+          make a plan non-compliant.
+      A release is permitted iff it satisfies every SECURITY and COMPLIANCE control and the binding
+      PRODUCT requirements; OPERATIONS items are honored only when they do not force a higher-tier
+      requirement to be missed.
+      WORKED READING: if OPERATIONS says "avoid X" but a binding PRODUCT requirement can only be met by
+      doing X, then X is permitted and the operations item is noted-but-overridden, not a blocker. If
+      two SECURITY controls cannot both hold, no plan ships. When several compliant start times exist,
+      choose per the deliverable asked (earliest vs latest); do not treat a preference as narrowing the
+      compliant window.
+      DEFINITIONS:
+        - A "binding PRODUCT requirement" is a PRODUCT rule stated as a requirement (P1), not a
+          preference (P2). Binding requirements can override lower-tier preferences; preferences at any
+          tier never override anything.
+        - "Directly conflict" is strict: two rules conflict only if no single plan satisfies both. Two
+          rules that merely point in different directions but admit a common plan do NOT conflict, and
+          both must be satisfied.
+        - Where a lower-tier preference happens to agree with the compliant plan, it is satisfied
+          incidentally; where it disagrees, it is overridden. Either way it never changes whether the
+          release is permitted.
+      ORDER OF EVALUATION: establish the SECURITY-required shape, then intersect the COMPLIANCE timing
+      windows, then confirm the binding PRODUCT deadline is met, and only then note which OPERATIONS
+      preferences are honored or overridden. A plan is compliant iff a non-empty start-time window
+      survives the SECURITY and COMPLIANCE steps and meets P1.
+  - id: security-controls
+    type: spec
+    label: Security controls (highest tier)
+    content: |
+      S1. A service handling regulated financial data must deploy as a staged canary: 5% held for at
+          least 60 minutes, then 25% held for at least 60 minutes, then 100%. The only permitted stage
+          percentages are 5, 25, and 100. "Held for at least N minutes" means the stage's traffic share
+          is maintained continuously for >= N minutes before advancing; advancing between stages is
+          instantaneous. A direct full (100%) rollout in one step is prohibited for such services, as
+          is any reordering or omission of the 5% and 25% stages.
+      S2. A staffed security responder — a named on-call engineer actively monitoring the rollout —
+          must be online for the entire rollout window (from the 5% start until 100% is reached) AND
+          for 1 hour after 100% is reached.
+      S3. Any deployment to eu-west requires a data-residency review token that is valid at the deploy
+          start instant (the 5% start). An expired or not-yet-valid token blocks the eu-west portion.
+      S4. The S1 staging MAY be compressed to a single 10% step held for at least 30 minutes, but ONLY
+          for a release that the on-call security lead has classified, in writing and before the
+          deploy start, as an emergency-hotfix. This exception applies to nothing else and does not
+          alter any COMPLIANCE control.
+      S5. A validated rollback plan that can be executed within 5 minutes at any stage must be attached
+          before start. (A capability requirement, not a timing constraint.)
+      S6. Every stage transition of a regulated deploy must emit a signed audit event. (Satisfied by
+          the deploy tooling automatically; not a timing constraint.)
+      NOTE ON S1 MECHANICS: the three stages are sequential and non-overlapping. The minimum elapsed
+      time from the 5% start to the 100% start is therefore 60 + 60 = 120 minutes, achieved only if
+      each hold runs exactly its minimum. A slower rollout (longer holds) is allowed by S1 but pushes
+      the 100% time later, which interacts with the COMPLIANCE deadline in C2. The responder in S2 is a
+      single named engineer for the whole window; no responder handoff is scheduled on 2026-03-13
+      outside the O2 coverage.
+  - id: compliance-controls
+    type: spec
+    label: Compliance controls (second tier)
+    content: |
+      C1. A regulated service may not deploy during the monthly close window. The close window is the
+          last two BUSINESS days of a month plus the first business day of the following month. A
+          business day is Monday-Friday excluding holidays (there are no holidays in the relevant
+          range). A deploy is placed in a window by its START date.
+      C2. Any eu-west deployment must reach 100% no later than 21:00 UTC (inclusive) on a business day.
+          "Reach 100%" is the instant the 100% stage begins.
+      C3. A change record must be filed at least 24 wall-clock hours before the deploy start time. The
+          change record's filed timestamp is authoritative; 24 wall-clock hours is measured on the
+          clock, not in business hours.
+      C4. A rollback plan must be attached to the change record. (Satisfied when S5's plan is filed
+          with the record.)
+      C5. The change record must name an approver from the change-advisory board. (Satisfied by the
+          board approval on the record.)
+      CLOSE-WINDOW COMPUTATION (C1): identify the last two business days of the month from
+      [business-calendar] (skip weekend days), then add the first business day of the following month.
+      The window is those three dates. A start date outside all three is not in the close window. Note
+      that the close window is anchored to month boundaries, not to the calendar midpoint, so a
+      mid-month Friday is not automatically inside or outside it — you must compute the actual dates.
+      DEADLINE ARITHMETIC (C3): earliest permitted start = change-record filed timestamp + exactly 24
+      hours. A start even one minute earlier than that is non-compliant.
+  - id: product-ops-policies
+    type: spec
+    label: Product requirements and operations preferences
+    content: |
+      PRODUCT (third tier, binding requirements):
+        P1. Contractual availability requires R51 to BEGIN rollout (the 5% start) no later than Friday
+            2026-03-13 20:00 UTC.
+        P2. Prefer to reach 100% on the same calendar day as the start. (A product preference.)
+      OPERATIONS (fourth tier, preferences only):
+        O1. Avoid production changes on Fridays after 16:00 UTC.
+        O2. Security responder on-call coverage on Friday 2026-03-13 is 16:00-22:30 UTC (responder
+            "SecOncall-3"). No responder is scheduled outside that window that day.
+        O3. EU change-control staffing is available 08:00-21:00 UTC on weekdays.
+        O4. Prefer deploys during business hours 09:00-17:00 UTC when possible.
+        O5. A batch-analytics freeze runs 22:00-02:00 UTC; avoid overlapping deploys with it.
+      Note: O1, O4, and O5 are preferences and cannot by themselves prohibit a release; O2 and O3 are
+      staffing facts that constrain feasibility only insofar as a higher-tier rule requires staffing
+      that must fall inside them.
+  - id: release-request
+    type: note
+    label: Release request R51
+    content: |
+      Release: R51, service "ledger-api" (classification: regulated financial data).
+      Target regions: us-east and eu-west (single coordinated rollout; both regions advance together).
+      Desired start: Friday 2026-03-13, 17:30 UTC.
+      Desired shape: full 100% rollout in one step.
+      Change record: CR-8841, filed 2026-03-12, 18:00 UTC.
+      Change-advisory-board approval: granted 2026-03-12, 19:10 UTC.
+      Data-residency review token presented: eu-res-2231.
+      Emergency-hotfix classification by on-call security lead: none requested, none on file.
+      Rollback plan: validated, executable in ~3 minutes at every stage, attached to CR-8841.
+      Build artifact hash: 9f4c-ledgerapi-r51. Prior deploy attempts for R51: none.
+      Deploy tooling supports exactly two shapes: the 5/25/100 staged canary, or a single-step 100%.
+  - id: business-calendar
+    type: table
+    label: March-April 2026 calendar (this scenario)
+    content: |
+      In this scenario the weekends of March 2026 fall on the 7th-8th, 14th-15th, 21st-22nd, and
+      28th-29th; every other day is a business day, and there are no holidays in March or early April.
+      date        | weekday   | business day?
+      2026-03-12  | Thursday  | yes
+      2026-03-13  | Friday    | yes
+      2026-03-14  | Saturday  | no
+      2026-03-15  | Sunday    | no
+      2026-03-16  | Monday    | yes
+      2026-03-27  | Friday    | yes
+      2026-03-28  | Saturday  | no
+      2026-03-29  | Sunday    | no
+      2026-03-30  | Monday    | yes
+      2026-03-31  | Tuesday   | yes
+      2026-04-01  | Wednesday | yes
+      2026-04-02  | Thursday  | yes
+      Compute the close window from C1 using this calendar. EU change-control staffing (O3) is the same
+      08:00-21:00 UTC on every business day shown.
+  - id: residency-token
+    type: config
+    label: Data-residency review token
+    content: |
+      token_id: eu-res-2231
+      scope: eu-west (data-residency review for regulated financial data)
+      issued_by: data-governance-office
+      approved_by: dpo-review-board
+      valid_from: 2026-03-01 00:00 UTC
+      valid_until: 2026-03-31 23:59 UTC
+      status: approved
+      check_point: evaluated at the deploy start instant (the 5% start); must be within [valid_from,
+      valid_until] at that instant.
+      conditions: none beyond the validity window.
+  - id: proposed-exception
+    type: note
+    label: Proposed emergency compression
+    content: |
+      Proposal from the release team, attached to R51:
+        "To hit the 17:30 UTC start, invoke S4 and deploy R51 as a single 10% step held 30 minutes,
+        then go to 100%. This lets us start earlier and finish faster, and it satisfies security
+        because 10% is still a canary."
+      The team argues the release is 'urgent enough' to count as an emergency. However, no emergency-
+      hotfix classification has been requested from or granted by the on-call security lead, and R51 is
+      a scheduled feature release, not a hotfix. The team also floated 'just start the eu-west portion
+      after 21:00 to buy time' and 'skip the residency token since it is basically always approved' as
+      fallbacks. Evaluate the emergency-compression proposal on its merits against [security-controls]
+      and [compliance-controls].
+tags: [policy, deployment, precedence, scheduling, compliance, canary]