skilldrop-cli 0.16.19 → 0.16.20

package/packs/sre-oncall/skills/capacity-cost-model/evals/evals.json

@@ -14,6 +14,34 @@
         "Outputs are low/expected/high ranges with a lean, and the cost-per-tenant trend at scale is checked and flagged if it rises",
         "Does not invent demand figures or services beyond those in the prompt; unknown inputs are stated as assumptions"
       ]
+    },
+    {
+      "id": 2,
+      "held_out": "2026-10-18",
+      "prompt": "I'm on the platform team at Brackwater Water. The programme manager wants a cloud cost estimate for the smart meter data platform for the finance committee. Her request is in brief.md, field ops' rollout plan is in rollout-plan.md, and our provider's prices are in price-sheet.md. Can you put the estimate together?",
+      "assertions": [
+        "Sizes ingestion and database writes for the synchronised upload window (about 4,800 readings per second at 120,000 meters), not for the daily average (about 130 per second)",
+        "Points out that the brief's hourly readings and 'about 1 million readings a day' do not match the rollout plan's 15-minute readings, which give about 11.5 million readings a day at 120,000 meters",
+        "Names the database Standard tier's 5,000 writes-per-second limit as a cliff: the year-2 peak sits just under it with little headroom, and the 3x scenario crosses it into the $2,400-a-month Enterprise tier",
+        "Keeps the regulator's 7-year retention as a fixed requirement and does not propose shortening it to cut cost",
+        "Does not give a payback period or savings figure, and says the material has no benefit or old-contract figures to compute one from",
+        "Uses only prices from the price sheet; any cost not on it is marked as an assumption, not presented as a Nimbus price",
+        "Fits in about two pages, as the committee asked"
+      ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "hi, i run a bakery (one shop right now) and my nephew set up our online ordering website. the hosting bill was $45 last month and we had roughly 1,200 orders. we're opening two more shops next spring. how much will the website cost us then? also christmas week we get about 4x the normal orders. just a few bullet points please, i'm not technical",
+      "assertions": [
+        "Gives an estimated monthly cost or range for after the new shops open, rather than only asking questions",
+        "Bases the estimate on order volume and states the assumption about how orders grow with two more shops",
+        "Does not simply triple the $45 without saying that part of a hosting bill is often a fixed plan fee that does not grow with orders",
+        "Flags the Christmas peak (about 4x normal orders, so roughly 12x today's volume if orders triple) as the thing to check against the hosting plan's limits before December",
+        "Suggests what to ask the nephew or the hosting company, such as what the $45 plan covers and what its limits are",
+        "Does not invent a hosting company's prices, plan names or limits",
+        "Stays at a few plain-language bullet points with no unexplained jargon"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/capacity-cost-model/evals/files/2/brief.md

@@ -0,0 +1,22 @@
+# Smart metering platform: cost estimate request
+
+From: Programme Manager, Smart Metering Programme, Brackwater Water
+To: Platform team
+
+We need a cloud cost estimate for the meter data platform before the finance committee meets on 4 November.
+
+What we know:
+
+- We are replacing 120,000 household water meters with smart meters.
+- Each meter sends a reading every hour, so we're looking at about 1 million readings a day.
+- Readings land in the time-series database and a daily file of per-meter totals goes to our on-premises billing system.
+- The regulator requires us to keep reading-level data for 7 years.
+- Finance has pencilled in $1,500 a month for running costs.
+
+What the committee wants:
+
+1. Monthly running cost in year 1 and year 2.
+2. What happens if the three neighbouring utilities join our platform. They are each roughly our size, so call it 3x.
+3. The payback period in months, so they can compare it with the old meter-reading contract.
+
+Please keep it to two pages at most. The committee will not read more than that.

package/packs/sre-oncall/skills/capacity-cost-model/evals/files/2/price-sheet.md

@@ -0,0 +1,13 @@
+# Nimbus Cloud price sheet (excerpt, USD, region: north-1)
+
+| Item | Price | Notes |
+|---|---|---|
+| Ingestion queue | $0.40 per million messages | |
+| Ingestion worker (2 vCPU) | $70 per worker per month | Each worker processes up to 500 readings per second. Autoscaling adds workers with about a 3-minute delay. |
+| Time-series DB, Standard tier | $400 per month flat | Up to 5,000 writes per second sustained. One reading = one write. |
+| Time-series DB, Enterprise tier | $2,400 per month flat | Up to 50,000 writes per second sustained. |
+| Time-series DB storage (either tier) | $0.12 per GB-month | A stored reading takes about 64 bytes after compression. |
+| Archive storage | $0.01 per GB-month | Retrieval takes up to 12 hours. |
+| Log ingestion | $0.50 per GB | Ingestion workers log one line of about 300 bytes per reading at the default level. |
+| Data transfer out (to internet or on-premises) | $0.09 per GB | |
+| NAT gateway | $32 per gateway per month + $0.045 per GB processed | |

package/packs/sre-oncall/skills/capacity-cost-model/evals/files/2/rollout-plan.md

@@ -0,0 +1,27 @@
+# Field operations: smart meter rollout plan (v3)
+
+Prepared by the Field Operations lead.
+
+## Install ramp
+
+| Month | Meters installed (cumulative) |
+|---|---|
+| 3 | 15,000 |
+| 6 | 40,000 |
+| 9 | 65,000 |
+| 12 | 85,000 |
+| 18 | 105,000 |
+| 24 | 120,000 |
+
+## Meter behaviour (from the meter supplier's firmware spec)
+
+- Meters take a reading every 15 minutes.
+- Meters do not send each reading as it is taken. They upload a batch four times a day, at 00:00, 06:00, 12:00 and 18:00, carrying the last 6 hours of readings (24 readings per upload).
+- Each meter picks a random start time within a 10-minute window after the scheduled time. There is no other spreading.
+- The upload schedule is fixed in firmware for the whole 2027 contract term. The supplier will not change it before then.
+- One upload is one message on the ingestion queue.
+
+## Other environments
+
+- A test rig of 500 meters on the bench at the depot, uploading on the same schedule.
+- A staging copy of the full pipeline, which the platform team say they need for testing firmware updates.

package/packs/sre-oncall/skills/cloud-cost-review/evals/evals.json

@@ -24,6 +24,34 @@
         "Treats credits as netted into cost and says so",
         "Does not invent a prior month or savings percentages"
       ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "Can you review our September cloud spend and tell the trustees where we can save? The billing export is cost-export-2026-09.csv, per-machine utilisation is in utilisation-2026-09.csv, and my notes are in notes.md.",
+      "assertions": [
+        "Attributes the September rise to the one-off archive copy to the university (about $5,200 of outbound data transfer), and does not count it as a recurring saving or use it to size a commitment",
+        "Flags that the export totals about $16,900, not the $21,400 in the notes, and that August's $11,400 was after a $2,000 credit, so the jump is overstated",
+        "Does not recommend removing or downsizing the DR standby database despite its 3% CPU, because the insurer requires it",
+        "Lists the idle notebook instance (about $553 a month) and the two unattached volumes ($300 a month together) as savings, with amounts taken from the export",
+        "Advises against a 3-year commitment covering everything: leaves out the render machine that goes in February and the one-off transfer, and limits any commitment to the steady baseline",
+        "Does not invent utilisation, resources or prices that are not in the files: it gives no rightsizing saving for the reporting server, which has no utilisation data, and does not count two savings on the same machine twice",
+        "Fits on one page for the committee papers"
+      ]
+    },
+    {
+      "id": 4,
+      "held_out": "2026-10-18",
+      "prompt": "I'm treasurer for the Elmstead Community Choir. Our volunteer IT person moved away and nobody understands the Google Cloud bill. It was about $420 last month and all we have is a website and some concert photos. The bill page shows:\n\nCompute Engine  $212.40\nCloud SQL  $118.25\nNetworking  $41.20\nCloud Storage  $8.15\n\nWhat can we cut? Just a short list please.",
+      "assertions": [
+        "Gives a short, prioritised list of things to look at in the reply, not only questions",
+        "Notes that the four lines add up to $380.00, not about $420, and suggests checking for a missing line, tax or a different month",
+        "Does not tell them to delete or switch off the server or the database outright before someone confirms whether the website depends on it",
+        "Does not state exact savings that four service totals cannot support; any figure is labelled as an estimate and tied to the lines given",
+        "Gives at least one safe first step a non-technical treasurer can take or ask for, such as a budget alert or a list of what is actually running",
+        "Does not invent machines, sizes, usage or prices beyond the four lines given",
+        "Keeps to a short list in plain language"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/cloud-cost-review/evals/files/3/cost-export-2026-09.csv

@@ -0,0 +1,25 @@
+billing_period,account,region,service,resource_id,resource_name,usage_type,usage_quantity,unit,cost_usd,tag_team,tag_env
+2026-09,kh-prod,eu-west-1,EC2,i-0a01,web-1,BoxUsage:m5.2xlarge,720,Hrs,276.48,web,prod
+2026-09,kh-prod,eu-west-1,EC2,i-0a02,web-2,BoxUsage:m5.2xlarge,720,Hrs,276.48,web,prod
+2026-09,kh-prod,eu-west-1,EC2,i-0a03,collections-db,BoxUsage:r5.4xlarge,720,Hrs,725.76,collections,prod
+2026-09,kh-prod,eu-west-2,EC2,i-0a04,collections-db-dr,BoxUsage:r5.4xlarge,720,Hrs,725.76,collections,dr
+2026-09,kh-prod,eu-west-1,EC2,i-0a05,image-render,BoxUsage:c5.4xlarge,720,Hrs,489.6,digitisation,prod
+2026-09,kh-dev,eu-west-1,EC2,i-0a06,notebook-temp,BoxUsage:m5.4xlarge,720,Hrs,552.96,,
+2026-09,kh-prod,eu-west-1,EC2,i-0a07,bastion,BoxUsage:t3.large,720,Hrs,59.9,web,prod
+2026-09,kh-prod,eu-west-1,EC2,i-0a08,reporting,BoxUsage:m5.xlarge,720,Hrs,138.24,finance,prod
+2026-09,kh-prod,eu-west-1,EBS,vol-01,collections-db-data,VolumeUsage.gp2,500,GB-Mo,50.0,collections,prod
+2026-09,kh-dev,eu-west-1,EBS,vol-02,,VolumeUsage.gp2,2000,GB-Mo,200.0,,
+2026-09,kh-prod,eu-west-1,EBS,vol-03,render-scratch,VolumeUsage.gp2,1000,GB-Mo,100.0,digitisation,prod
+2026-09,kh-prod,eu-west-2,EBS,vol-04,collections-db-dr-data,VolumeUsage.gp2,500,GB-Mo,50.0,collections,dr
+2026-09,kh-prod,eu-west-1,EBS,snapshots,,SnapshotUsage,18000,GB-Mo,900.0,,
+2026-09,kh-prod,eu-west-1,S3,kh-archive-masters,,TimedStorage-ByteHrs,85000,GB-Mo,1955.0,digitisation,prod
+2026-09,kh-prod,eu-west-1,S3,kh-archive-masters,,Requests-Tier2,31000,Requests,12.4,digitisation,prod
+2026-09,kh-prod,eu-west-1,S3,kh-web-assets,,TimedStorage-ByteHrs,400,GB-Mo,9.2,web,prod
+2026-09,kh-prod,eu-west-1,VPC,nat-01,,NatGateway-Hours,1440,Hrs,69.12,,
+2026-09,kh-prod,eu-west-1,VPC,nat-01,,NatGateway-Bytes,42000,GB,2016.0,,
+2026-09,kh-prod,eu-west-1,DataTransfer,kh-archive-masters,,DataTransfer-Out-Bytes,61500,GB,5227.5,digitisation,prod
+2026-09,kh-prod,eu-west-1,DataTransfer,,,DataTransfer-Regional-Bytes,9000,GB,180.0,,
+2026-09,kh-prod,eu-west-1,DataTransfer,i-0a03,collections-db,DataTransfer-InterRegion-Out,3000,GB,60.0,collections,prod
+2026-09,kh-prod,eu-west-1,CloudWatch,,,DataProcessing-Bytes (logs ingest),2900,GB,1653.0,,
+2026-09,kh-prod,eu-west-1,CloudWatch,,,CW:MetricMonitorUsage,280,Metrics,84.0,,
+2026-09,kh-prod,eu-west-1,Support,,,Business support,1,Month,1100.0,,

package/packs/sre-oncall/skills/cloud-cost-review/evals/files/3/notes.md

@@ -0,0 +1,11 @@
+# Notes from the Finance Officer, Kestrel Hall Museum Trust
+
+- The September cloud bill is $21,400 against August's $11,400. That is nearly double and the trustees want to know why, and where we can save.
+- August's figure is after a $2,000 service credit the provider gave us for the July outage.
+- In September we sent a full copy of the image archive to the Northfield University consortium for their research mirror. That was a one-off; they won't need it again.
+- Our insurer requires a warm standby of the collections database in a second region, the same size as production. It is a condition of our cover.
+- Image rendering moves to the university's shared computing cluster in February 2027, so that machine goes away then.
+- The trustees would like to sign a 3-year savings plan covering everything before year end, because the account manager said it saves "up to 72%".
+- The archive masters are read maybe twice a year, when we do reprints. The web team say they never touch that bucket.
+- We don't have monitoring on the reporting server yet; the agent was never installed.
+- Please keep it to one page. It goes in the finance committee papers.

package/packs/sre-oncall/skills/cloud-cost-review/evals/files/3/utilisation-2026-09.csv

@@ -0,0 +1,12 @@
+resource_id,resource_name,cpu_p95_pct,mem_p95_pct,state,last_activity
+i-0a01,web-1,41,58,running,2026-09-30
+i-0a02,web-2,39,55,running,2026-09-30
+i-0a03,collections-db,62,81,running,2026-09-30
+i-0a04,collections-db-dr,3,22,running (replica),2026-09-30
+i-0a05,image-render,9,14,running,2026-09-30
+i-0a06,notebook-temp,1,4,running,2026-06-30
+i-0a07,bastion,2,10,running,2026-09-29
+vol-01,collections-db-data,,,in-use,2026-09-30
+vol-02,,,,available (unattached since 2026-03-14),2026-03-14
+vol-03,render-scratch,,,available (unattached since 2026-05-02),2026-05-02
+vol-04,collections-db-dr-data,,,in-use,2026-09-30

package/packs/sre-oncall/skills/delivery-metrics-report/evals/evals.json

@@ -23,6 +23,33 @@
         "Offers an alternative: each team's own trend over the same window and a shared definition of deployment and failure",
         "Does not invent metric values for any team"
       ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "I manage the fares team at Tarnmoor Rail. The steering group wants a one-page delivery report on fares-api for August and September. Our Head of Digital has been telling them we deployed about 60 times in those two months with a 5% change failure rate. The exports are deploys.csv, incidents.csv and prs.csv. While you're at it, can you rank the engineers by cycle time? I need it for performance reviews.",
+      "assertions": [
+        "Counts only fares-api production deployments (32 in the window), excluding staging and journey-planner rows, and says this does not match the 'about 60' figure",
+        "Gives a change failure rate of about 9% (3 of 32), leaving out the payment-provider outage and counting the duplicate incident once, and says this differs from the 5% claimed",
+        "Treats the week of 7 September, which has no rows for any service, as a likely gap in the export to confirm, not as a week with zero deployments",
+        "Reports recovery time with its tiny sample and flags the still-unresolved incident, rather than presenting a confident average",
+        "Does not rank individual engineers and explains why, offering a team-level view instead",
+        "Does not invent deployments, incidents or figures that are not in the exports",
+        "Fits on about one page for the steering group"
+      ]
+    },
+    {
+      "id": 4,
+      "held_out": "2026-10-18",
+      "prompt": "I run marketing for Fernhill Garden Centres. Our web agency sends us this every month and my boss wants to know if they're getting better or worse:\n\nJuly: 3 releases, 1 broke something (fixed in 2 days)\nAugust: 5 releases, 0 broke anything\nSeptember: 2 releases, 2 broke something (one fixed in 4 hours, one in 3 days)\n\nCan you give me a couple of sentences and a small table I can paste into an email?",
+      "assertions": [
+        "Delivers a couple of sentences and a small table ready to paste, not a long report",
+        "Computes the failure share correctly: 1 of 3 in July, 0 of 5 in August, 2 of 2 in September, 3 of 10 overall",
+        "Says that ten releases over three months are too few to call a trend with confidence, while noting September looks worse",
+        "Does not report lead time or any other measure the monthly lines do not contain; any mention of fix time uses the three fix times as given",
+        "Suggests one or two things to ask the agency to record each month so the question can be answered properly",
+        "Does not invent numbers beyond the three monthly lines"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/delivery-metrics-report/evals/files/3/deploys.csv

@@ -0,0 +1,68 @@
+deploy_id,service,environment,deployed_at,version
+D-1003,fares-api,staging,2026-08-03T07:05:00Z,fares-b170b
+D-1036,fares-api,production,2026-08-03T09:00:00Z,fares-f883a
+D-1005,journey-planner,production,2026-08-03T11:10:00Z,journ-1becd
+D-1037,fares-api,production,2026-08-04T11:00:00Z,fares-3ac34
+D-1038,fares-api,production,2026-08-05T13:15:00Z,fares-6b0ee
+D-1001,fares-api,staging,2026-08-06T07:05:00Z,fares-9d116
+D-1039,fares-api,production,2026-08-06T09:00:00Z,fares-d59db
+D-1002,fares-api,staging,2026-08-07T08:05:00Z,fares-e3ac9
+D-1004,journey-planner,production,2026-08-07T15:50:00Z,journ-758cd
+D-1007,fares-api,staging,2026-08-10T08:40:00Z,fares-981ed
+D-1040,fares-api,production,2026-08-10T13:00:00Z,fares-4908f
+D-1041,fares-api,production,2026-08-11T13:30:00Z,fares-9857f
+D-1008,fares-api,staging,2026-08-12T08:20:00Z,fares-8731a
+D-1042,fares-api,production,2026-08-12T13:15:00Z,fares-9aa42
+D-1043,fares-api,production,2026-08-13T09:15:00Z,fares-d7702
+D-1010,journey-planner,production,2026-08-13T12:50:00Z,journ-d7a2e
+D-1006,fares-api,staging,2026-08-14T07:40:00Z,fares-44b9b
+D-1044,fares-api,production,2026-08-14T11:00:00Z,fares-90b0c
+D-1009,journey-planner,production,2026-08-14T13:25:00Z,journ-4f98e
+D-1013,fares-api,staging,2026-08-17T08:20:00Z,fares-c1fee
+D-1045,fares-api,production,2026-08-17T13:45:00Z,fares-6464e
+D-1011,fares-api,staging,2026-08-18T08:20:00Z,fares-1a097
+D-1046,fares-api,production,2026-08-18T11:45:00Z,fares-b2edd
+D-1012,fares-api,staging,2026-08-19T07:40:00Z,fares-a2b1d
+D-1047,fares-api,production,2026-08-19T11:00:00Z,fares-49194
+D-1014,journey-planner,production,2026-08-19T15:25:00Z,journ-219a7
+D-1015,journey-planner,production,2026-08-20T11:25:00Z,journ-895e8
+D-1048,fares-api,production,2026-08-21T09:45:00Z,fares-acfc8
+D-1017,fares-api,staging,2026-08-24T07:20:00Z,fares-74154
+D-1049,fares-api,production,2026-08-24T11:45:00Z,fares-dfbf3
+D-1050,fares-api,production,2026-08-25T14:30:00Z,fares-d9d48
+D-1051,fares-api,production,2026-08-26T09:45:00Z,fares-d2216
+D-1016,fares-api,staging,2026-08-27T08:05:00Z,fares-cd056
+D-1052,fares-api,production,2026-08-27T11:15:00Z,fares-ea45e
+D-1019,journey-planner,production,2026-08-27T15:25:00Z,journ-f2257
+D-1018,fares-api,staging,2026-08-28T08:05:00Z,fares-3a96f
+D-1020,journey-planner,production,2026-08-28T12:25:00Z,journ-ed2e1
+D-1053,fares-api,production,2026-08-28T13:00:00Z,fares-41f51
+D-1022,fares-api,staging,2026-08-31T07:05:00Z,fares-7b401
+D-1021,fares-api,staging,2026-09-01T07:20:00Z,fares-582c9
+D-1054,fares-api,production,2026-09-01T11:45:00Z,fares-de5b2
+D-1025,journey-planner,production,2026-09-01T16:50:00Z,journ-cd628
+D-1055,fares-api,production,2026-09-02T10:15:00Z,fares-4d488
+D-1024,journey-planner,production,2026-09-02T16:50:00Z,journ-ae1a8
+D-1056,fares-api,production,2026-09-03T09:15:00Z,fares-e17e4
+D-1023,fares-api,staging,2026-09-04T08:40:00Z,fares-a0fbb
+D-1057,fares-api,production,2026-09-04T10:15:00Z,fares-76934
+D-1027,fares-api,staging,2026-09-14T07:20:00Z,fares-ad1de
+D-1058,fares-api,production,2026-09-14T12:00:00Z,fares-cd685
+D-1059,fares-api,production,2026-09-15T12:45:00Z,fares-dda6c
+D-1029,journey-planner,production,2026-09-15T12:50:00Z,journ-5093f
+D-1060,fares-api,production,2026-09-16T09:15:00Z,fares-4a0b9
+D-1028,fares-api,staging,2026-09-17T07:05:00Z,fares-efd43
+D-1061,fares-api,production,2026-09-17T09:15:00Z,fares-432dd
+D-1062,fares-api,production,2026-09-17T12:15:00Z,fares-9483f
+D-1030,journey-planner,production,2026-09-17T13:50:00Z,journ-6d39d
+D-1026,fares-api,staging,2026-09-18T07:40:00Z,fares-36b94
+D-1063,fares-api,production,2026-09-18T09:30:00Z,fares-8e26f
+D-1064,fares-api,production,2026-09-21T12:45:00Z,fares-6b062
+D-1034,journey-planner,production,2026-09-21T16:10:00Z,journ-c239f
+D-1065,fares-api,production,2026-09-22T12:30:00Z,fares-cb231
+D-1031,fares-api,staging,2026-09-23T07:40:00Z,fares-15e99
+D-1035,journey-planner,production,2026-09-23T13:50:00Z,journ-6de00
+D-1032,fares-api,staging,2026-09-24T07:40:00Z,fares-6c9bc
+D-1066,fares-api,production,2026-09-24T09:15:00Z,fares-176b3
+D-1033,fares-api,staging,2026-09-25T07:40:00Z,fares-9b0d5
+D-1067,fares-api,production,2026-09-25T09:30:00Z,fares-1726e

package/packs/sre-oncall/skills/delivery-metrics-report/evals/files/3/incidents.csv

@@ -0,0 +1,7 @@
+incident_id,service,opened_at,resolved_at,severity,caused_by_deploy,summary
+INC-201,fares-api,2026-08-11T14:10:00Z,2026-08-11T17:10:00Z,SEV2,D-1041,Child fares calculated at adult rate after release
+INC-202,fares-api,2026-08-19T11:25:00Z,2026-08-19T12:15:00Z,SEV2,D-1047,Fare lookups timing out after connection pool change
+INC-205,fares-api,2026-08-19T11:31:00Z,2026-08-19T12:20:00Z,SEV2,D-1047,Duplicate of INC-202 raised by the service desk; closed as duplicate
+INC-204,journey-planner,2026-09-01T17:20:00Z,2026-09-01T18:00:00Z,SEV3,D-1025,Platform numbers missing from journey results
+INC-203,fares-api,2026-09-02T13:10:00Z,2026-09-02T15:05:00Z,SEV2,,"Card payments failing: payment provider outage (third party), no change on our side"
+INC-206,fares-api,2026-09-17T10:20:00Z,,SEV3,D-1061,Railcard discount codes rejected for some users; mitigation in progress

package/packs/sre-oncall/skills/delivery-metrics-report/evals/files/3/prs.csv

@@ -0,0 +1,46 @@
+pr_id,service,author,first_commit_at,opened_at,merged_at,deploy_id
+PR-501,fares-api,eng-06,2026-08-01T04:00:00Z,2026-08-01T08:00:00Z,2026-08-02T17:00:00Z,D-1036
+PR-502,fares-api,eng-03,2026-08-01T10:00:00Z,2026-08-01T16:00:00Z,2026-08-03T23:00:00Z,D-1037
+PR-503,fares-api,eng-02,2026-08-03T00:00:00Z,2026-08-03T02:00:00Z,2026-08-04T08:00:00Z,D-1037
+PR-506,fares-api,eng-07,2026-08-01T12:00:00Z,2026-08-01T18:00:00Z,2026-08-05T03:00:00Z,D-1039
+PR-505,fares-api,eng-04,2026-08-01T18:00:00Z,2026-08-01T19:00:00Z,2026-08-05T04:00:00Z,D-1039
+PR-504,fares-api,eng-04,2026-08-03T07:15:00Z,2026-08-03T11:15:00Z,2026-08-05T06:15:00Z,D-1038
+PR-508,fares-api,eng-04,2026-08-05T12:00:00Z,2026-08-05T16:00:00Z,2026-08-09T11:00:00Z,D-1040
+PR-507,fares-api,eng-04,2026-08-05T20:00:00Z,2026-08-05T22:00:00Z,2026-08-10T10:00:00Z,D-1040
+PR-509,fares-api,eng-01,2026-08-06T14:30:00Z,2026-08-06T20:30:00Z,2026-08-10T23:30:00Z,D-1041
+PR-510,fares-api,eng-06,2026-08-09T09:15:00Z,2026-08-09T16:15:00Z,2026-08-12T00:15:00Z,D-1042
+PR-511,fares-api,eng-02,2026-08-08T10:15:00Z,2026-08-08T13:15:00Z,2026-08-12T10:15:00Z,D-1042
+PR-514,fares-api,eng-05,2026-08-11T05:00:00Z,2026-08-11T08:00:00Z,2026-08-13T05:00:00Z,D-1044
+PR-512,fares-api,eng-07,2026-08-12T09:15:00Z,2026-08-12T17:15:00Z,2026-08-13T08:15:00Z,D-1043
+PR-513,fares-api,eng-06,2026-08-10T20:00:00Z,2026-08-11T04:00:00Z,2026-08-14T06:00:00Z,D-1044
+PR-516,fares-api,eng-05,2026-08-12T22:45:00Z,2026-08-13T00:45:00Z,2026-08-16T13:45:00Z,D-1045
+PR-517,fares-api,eng-07,2026-08-16T08:45:00Z,2026-08-16T15:45:00Z,2026-08-17T05:45:00Z,D-1046
+PR-515,fares-api,eng-07,2026-08-17T02:45:00Z,2026-08-17T03:45:00Z,2026-08-17T08:45:00Z,D-1045
+PR-518,fares-api,eng-01,2026-08-13T15:45:00Z,2026-08-13T19:45:00Z,2026-08-18T04:45:00Z,D-1046
+PR-519,fares-api,eng-07,2026-08-17T11:00:00Z,2026-08-17T15:00:00Z,2026-08-19T04:00:00Z,D-1047
+PR-520,fares-api,eng-07,2026-08-19T09:45:00Z,2026-08-19T16:45:00Z,2026-08-20T22:45:00Z,D-1048
+PR-521,fares-api,eng-04,2026-08-20T17:45:00Z,2026-08-20T23:45:00Z,2026-08-21T04:45:00Z,D-1048
+PR-523,fares-api,eng-05,2026-08-18T09:45:00Z,2026-08-18T12:45:00Z,2026-08-23T05:45:00Z,D-1049
+PR-522,fares-api,eng-07,2026-08-19T04:45:00Z,2026-08-19T11:45:00Z,2026-08-23T16:45:00Z,D-1049
+PR-524,fares-api,eng-07,2026-08-22T00:30:00Z,2026-08-22T01:30:00Z,2026-08-24T21:30:00Z,D-1050
+PR-525,fares-api,eng-07,2026-08-24T05:45:00Z,2026-08-24T06:45:00Z,2026-08-25T08:45:00Z,D-1051
+PR-526,fares-api,eng-05,2026-08-26T07:15:00Z,2026-08-26T15:15:00Z,2026-08-27T05:15:00Z,D-1052
+PR-528,fares-api,eng-07,2026-08-24T17:00:00Z,2026-08-25T01:00:00Z,2026-08-27T15:00:00Z,D-1053
+PR-527,fares-api,eng-03,2026-08-25T06:00:00Z,2026-08-25T07:00:00Z,2026-08-28T09:00:00Z,D-1053
+PR-529,fares-api,eng-02,2026-08-28T03:45:00Z,2026-08-28T04:45:00Z,2026-08-31T06:45:00Z,D-1054
+PR-530,fares-api,eng-05,2026-09-01T16:15:00Z,2026-09-01T18:15:00Z,2026-09-02T01:15:00Z,D-1055
+PR-531,fares-api,eng-04,2026-09-02T08:15:00Z,2026-09-02T10:15:00Z,2026-09-02T15:15:00Z,D-1056
+PR-532,fares-api,eng-06,2026-08-31T18:15:00Z,2026-08-31T22:15:00Z,2026-09-03T14:15:00Z,D-1057
+PR-533,fares-api,eng-05,2026-09-11T00:00:00Z,2026-09-11T08:00:00Z,2026-09-13T21:00:00Z,D-1058
+PR-534,fares-api,eng-05,2026-09-11T15:45:00Z,2026-09-11T20:45:00Z,2026-09-14T13:45:00Z,D-1059
+PR-535,fares-api,eng-04,2026-09-12T17:15:00Z,2026-09-12T20:15:00Z,2026-09-15T06:15:00Z,D-1060
+PR-538,fares-api,eng-07,2026-09-14T20:15:00Z,2026-09-14T22:15:00Z,2026-09-16T14:15:00Z,D-1062
+PR-536,fares-api,eng-01,2026-09-14T08:15:00Z,2026-09-14T14:15:00Z,2026-09-16T20:15:00Z,D-1061
+PR-537,fares-api,eng-02,2026-09-14T18:15:00Z,2026-09-14T20:15:00Z,2026-09-17T04:15:00Z,D-1062
+PR-539,fares-api,eng-02,2026-09-13T20:30:00Z,2026-09-14T02:30:00Z,2026-09-17T10:30:00Z,D-1063
+PR-540,fares-api,eng-02,2026-09-19T10:45:00Z,2026-09-19T18:45:00Z,2026-09-20T07:45:00Z,D-1064
+PR-542,fares-api,eng-02,2026-09-17T00:30:00Z,2026-09-17T04:30:00Z,2026-09-21T14:30:00Z,D-1065
+PR-541,fares-api,eng-02,2026-09-20T02:30:00Z,2026-09-20T10:30:00Z,2026-09-22T08:30:00Z,D-1065
+PR-543,fares-api,eng-03,2026-09-20T22:15:00Z,2026-09-21T05:15:00Z,2026-09-23T19:15:00Z,D-1066
+PR-544,fares-api,eng-03,2026-09-22T14:15:00Z,2026-09-22T20:15:00Z,2026-09-23T19:15:00Z,D-1066
+PR-545,fares-api,eng-03,2026-09-22T07:30:00Z,2026-09-22T08:30:00Z,2026-09-24T09:30:00Z,D-1067

package/packs/sre-oncall/skills/incident-comms/evals/evals.json

@@ -11,6 +11,33 @@
         "The three audiences (customer status page, internal stakeholders, exec) get genuinely different messages rather than one text reused",
         "Nothing is asserted beyond what the operator supplied"
       ]
+    },
+    {
+      "id": 2,
+      "held_out": "2026-10-18",
+      "prompt": "I'm the duty comms officer for Corrin Valley Transit and we're mid-incident with TallyCard top-ups. It's 14:40 now. The incident channel is in incident-channel.txt, the last status page post is in last-status-update.md and our comms rules are in comms-rules.md. Please draft what I need to send now.",
+      "assertions": [
+        "The customer post describes the impact accurately: card payments at ticket machines failing, app top-ups partly failing, cash at machines working, and does not say all top-ups are down or that the problem is fixed",
+        "No customer-facing message gives the database failover theory or the '30 minutes' estimate, and the CEO note says there is no ETA yet instead of inventing one",
+        "Tells riders that inspectors will accept people who could not top up, until the end of service today",
+        "Describes the 'charged twice' reports as temporary holds that should drop off within 3 to 5 working days, without promising refunds or saying riders were charged twice",
+        "Each message states when the next update will come, and no public message names the payment processor or uses the word 'breach'",
+        "The status page post is no more than 600 characters",
+        "Does not invent figures, times, causes or actions that are not in the channel log"
+      ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "school office here. the online system parents use to pay for school lunches is down since this morning and parents are emailing us in a panic. the company that runs it just says they're 'working on it'. the head says children will still get lunch today and parents can pay later. can you write me a text message to parents (has to fit one SMS, 160 characters) and a slightly longer email?",
+      "assertions": [
+        "Delivers both a text message and an email ready to send, without stopping to ask questions first",
+        "The text message is 160 characters or fewer",
+        "Both say children will still get lunch today and parents can pay later",
+        "Neither gives a cause or promises when the system will be fixed",
+        "Tells parents when or how the school will update them next",
+        "Does not invent details that were not given, such as a payment deadline, refunds, a supplier name or a fix time"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/incident-comms/evals/files/2/comms-rules.md

@@ -0,0 +1,6 @@
+# Corrin Valley Transit: incident communications rules (extract)
+
+- The status page tool accepts at most 600 characters per post, including spaces.
+- Customer posts are written for riders, not engineers.
+- Internal staff updates go to station staff and the contact centre by the staff messaging app. Keep them short enough to read on a phone.
+- Exec notes go to the CEO's office by email.

package/packs/sre-oncall/skills/incident-comms/evals/files/2/incident-channel.txt

@@ -0,0 +1,17 @@
+#inc-0412-topup  (Corrin Valley Transit, TallyCard top-ups)  Thursday 22 October 2026
+
+13:05 [on-call eng] Paging: top-up error rate climbing on the payments gateway
+13:09 [incident lead] Declaring SEV1. I'm IC. Comms owner is the duty comms officer.
+13:14 [support lead] Phones are lit up. Customers saying ALL top-ups are down, app and machines
+13:18 [on-call eng] Dashboard: app top-ups 38% failing. Ticket machines: card payments 100% failing. Cash at machines is fine, I can see cash top-ups going through normally
+13:22 [incident lead] So not everything. Machines take cash fine, card at machines is dead, app is partial
+13:31 [duty comms] First status page post went out at 13:40, said next update 14:30
+13:52 [support lead] Several riders on social saying they've been "charged twice" for one top-up
+13:58 [payments eng] Checked: those are duplicate authorisation holds, not captured payments. We only capture once. Holds normally drop off in 3 to 5 working days depending on the rider's bank. We are NOT issuing refunds because nothing was taken
+14:06 [on-call eng] My guess is the database failover at 12:58 left the gateway pointing at the old primary. Maybe 30 min to fix if that's it
+14:08 [incident lead] Unconfirmed. Don't put the failover theory anywhere outside this channel until we know
+14:15 [ops control] Decision from Head of Operations: revenue inspectors are told to accept riders who could not top up. Valid until end of service today (last trains around 00:30). Inspectors briefed at 14:12
+14:21 [legal] Reminder: don't name our payment processor in anything public, and don't use the word "breach". No card data is involved as far as anyone knows
+14:29 [CEO's office] CEO needs an ETA for the board call at 15:00
+14:33 [incident lead] No ETA. We are testing the failover theory now
+14:38 [incident lead] Comms: we owe the status page an update (promised 14:30). Need status page post, internal update for station and contact-centre staff, and a note for the CEO. Next updates every 30 minutes

package/packs/sre-oncall/skills/incident-comms/evals/files/2/last-status-update.md

@@ -0,0 +1,5 @@
+# Status page post, 13:40
+
+**Investigating: problems topping up TallyCards**
+
+Some customers are having trouble topping up their TallyCard. We're investigating and will post an update by 14:30.

package/packs/sre-oncall/skills/observability-plan/evals/evals.json

@@ -11,6 +11,33 @@
         "Pages are symptom-based, actionable, and runbook-linked; anything not actionable is a ticket, not a page",
         "The plan makes a deliberate metrics-vs-logs-vs-traces choice rather than proposing everything"
       ]
+    },
+    {
+      "id": 2,
+      "held_out": "2026-10-18",
+      "prompt": "We need an observability plan for Larkspur Pharmacy's repeat prescriptions service. The service is described in service-overview.md, our current alerts and how often they fire are in current-alerts.csv, and the constraints are in constraints.md. Can you write the plan?",
+      "assertions": [
+        "Does not adopt the 99.999% target: explains that the dispensary system's 99.9% commitment, with no offline mode, caps what the refill journey can honestly promise, and sets a target with that in mind",
+        "Refuses patient ID and prescription ID as metric labels, giving both the cardinality cost and the health-data rule, and offers another way to trace individual complaints that keeps identifiers out of logs, labels and traces",
+        "Stops paging on causes that were rarely actioned (CPU, memory, pod restarts, any single 5xx) and makes pages depend on what patients experience",
+        "Handles the overnight low-traffic hours so that a handful of failed requests while branches are closed does not page anyone",
+        "Addresses the $1,500 cap against the current $2,100 and says what is cut or reduced to get there, with log ingestion as the main lever",
+        "Notes that the traffic figures conflict (40,000 a week in the overview, 40,000 a day in the constraints) and does not invent other traffic, alert or cost figures",
+        "Keeps the plan to about three pages"
+      ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "I sell yarn online, about 30 orders a day, it's just me. My web guy wants to set up 'alerts' and showed me a list of 25 things he can monitor (server load, memory, disk, and so on). I just want to know if customers can't buy stuff. What should actually text me at night? Few bullet points please.",
+      "assertions": [
+        "Recommends only one or a few alerts that would text her at night, each about customers being unable to reach the shop, check out or pay",
+        "Says most of the 25 items should not text her, and can go to email or an occasional check instead",
+        "Allows for the low order volume, for example by not alerting on 'no orders in the last hour' overnight or on one failed payment",
+        "Says what she or the web guy should do when each night-time alert fires",
+        "Does not invent details of her shop platform, tools or traffic beyond what she said",
+        "Delivers a few plain-language bullet points without stopping to ask questions first"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/observability-plan/evals/files/2/constraints.md

@@ -0,0 +1,9 @@
+# Constraints for the observability plan
+
+From the Head of Engineering and the Information Governance lead.
+
+- **Budget**: our observability vendor bill is currently about $2,100 a month. Finance has capped it at $1,500 a month from January. The two biggest lines are log ingestion (about $1,300) and custom metrics (about $500); traces are about $300.
+- **Health data**: patient identifiers, prescription identifiers and medication names are health data and must not be written to logs, metric labels or trace attributes. This is an Information Governance rule, not a preference.
+- **On-call**: two engineers share a weekly rotation. They have asked for no more than two pages a week on average.
+- **Volume for costing**: plan for about 40,000 refill requests a day.
+- **Review**: we'll go through the plan in a 30-minute meeting, so keep it to about three pages.

package/packs/sre-oncall/skills/observability-plan/evals/files/2/current-alerts.csv

@@ -0,0 +1,10 @@
+alert,condition,routes_to,fired_last_30d,actioned_last_30d
+RefillApiCpuHigh,CPU > 80% for 5m on any pod,page,140,2
+RefillApiMemoryHigh,memory > 85% for 10m,page,61,0
+RefillApi5xx,any 5xx in 1m,page,212,9
+DmsLatencyHigh,DMS p99 > 2s for 5m,page,18,6
+NotifierQueueDepth,SMS queue > 500 for 10m,ticket,4,3
+DiskUsage,disk > 90% on any node,page,7,7
+PodRestart,any pod restart,page,95,1
+CertExpiry,TLS cert expires in < 14 days,ticket,1,1
+SyntheticRefillCheck,synthetic refill request fails 3 times in a row,email,0,0

package/packs/sre-oncall/skills/observability-plan/evals/files/2/service-overview.md

@@ -0,0 +1,27 @@
+# Repeat Prescriptions service: overview
+
+Owner: Digital Pharmacy team, Larkspur Pharmacy (84 branches)
+
+## What it does
+
+Patients request a repeat prescription in the app or on the website. The request goes to the patient's branch, where it is checked against the Dispensary Management System (DMS) and dispensed. When the order is ready, the patient gets an SMS saying it can be collected.
+
+## Journey and components
+
+1. **Request**: app/web front end calls the Refill API.
+2. **Check and queue**: Refill API looks up the patient's repeat list in the DMS and puts the order on the branch's dispensing queue.
+3. **Dispense**: branch staff work through the queue in the DMS.
+4. **Notify**: when staff mark the order ready, the Notifier sends an SMS through our SMS provider.
+
+## Dependencies
+
+- **DMS** is run by an outside supplier. Their contract promises 99.9% monthly availability, and they report it monthly. The Refill API cannot take a request without it; there is no offline mode.
+- **SMS provider**: messages are usually delivered within a minute; the provider gives delivery receipts.
+
+## Traffic
+
+About 40,000 refill requests a week, almost all between 07:00 and 22:00. Branches are closed 22:00 to 07:00 and the app shows "requests will be processed in the morning", but requests can still be submitted overnight (typically 0 to 20 an hour).
+
+## What the product owner wants
+
+"I want 99.999% availability on refills. And please add patient ID and prescription ID as labels on every metric so we can debug individual complaints."

package/packs/sre-oncall/skills/runbook-generator/evals/evals.json

@@ -12,6 +12,33 @@
         "Escalation names the path and the trigger for escalating, not just a team name",
         "Facts not supplied (dashboard links, SLO values) are marked missing rather than invented"
       ]
+    },
+    {
+      "id": 2,
+      "held_out": "2026-10-18",
+      "prompt": "I've just taken over TideFeed at Saltholm Harbour Authority and the previous engineer leaves on Friday. Can you write the runbook from service-notes.md, deploy.sh, handover-chat.txt and standing-order.md? Save it as runbook.md. The control room keeps a printed copy, so it has to fit on three pages.",
+      "assertions": [
+        "Rollback is done by redeploying the previous release tag with deploy.sh, which takes only a tag; it flags that the script has no --rollback option despite the notes",
+        "Warns that migration 0042 is forward-only, so the app cannot be rolled back to a version before 2.6.0 such as v2.5.4",
+        "The first action when the display data is more than 15 minutes old or missing is to tell pilots to use the printed tide tables, as the standing order requires, before any troubleshooting",
+        "Does not list the gauge hardware supplier as a working escalation route, and flags that the support contract lapsed in August",
+        "Leaves SLOs and dashboards as open gaps to be filled, rather than inventing targets or dashboard links",
+        "Does not invent commands, hostnames, phone numbers or contacts that are not in the files",
+        "The runbook is saved as runbook.md and fits on about three printed pages"
+      ]
+    },
+    {
+      "id": 3,
+      "held_out": "2026-10-18",
+      "prompt": "our community radio station's internet stream keeps dropping and the volunteers on the overnight shift have no idea what to do. the stream goes from the studio PC through an encoder box to our streaming host. the station manager says only phone her if it's been down more than 30 minutes. can you write a one-page 'what to do if the stream stops' sheet we can stick on the wall?",
+      "assertions": [
+        "Delivers a one-page sheet in plain language a non-technical volunteer can follow",
+        "Orders the checks from the simplest and most likely (is the studio PC playing, is the encoder box on and connected, is the internet up) before anything harder",
+        "Says to phone the station manager only after the stream has been down for more than 30 minutes",
+        "Does not invent phone numbers, passwords, equipment models or host settings; leaves clear blanks for the station to fill in",
+        "Tells volunteers how to confirm the stream is back and to note what happened for the manager",
+        "Delivers the sheet without stopping to ask questions first"
+      ]
     }
   ]
 }

package/packs/sre-oncall/skills/runbook-generator/evals/files/2/deploy.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# deploy.sh: deploy a tagged release of TideFeed to tidefeed-app-01
+# Usage: ./deploy.sh <release-tag>      e.g. ./deploy.sh v2.6.1
+set -euo pipefail
+
+TAG="${1:?usage: ./deploy.sh <release-tag>}"
+HOST="tidefeed-app-01"
+
+echo "Deploying TideFeed ${TAG} to ${HOST}"
+ssh "${HOST}" "cd /opt/tidefeed && git fetch --tags && git checkout ${TAG}"
+ssh "${HOST}" "cd /opt/tidefeed && ./manage.py migrate"
+ssh "${HOST}" "sudo systemctl restart gauge-collector tidefeed-api"
+ssh "${HOST}" "curl -fsS http://localhost:8080/health"
+echo "Done. Deployed ${TAG}."

package/packs/sre-oncall/skills/runbook-generator/evals/files/2/handover-chat.txt

@@ -0,0 +1,8 @@
+[departing engineer] ok, last bits before I go on Friday
+[departing engineer] current release is v2.6.1. v2.6.0 was the one with migration 0042. v2.5.4 was the last before it
+[departing engineer] if a gauge stops reporting, first check the radio link light on the gauge cabinet panel in the control room. if it's red, it's the link not us
+[departing engineer] if the collector hangs, `sudo systemctl restart gauge-collector` on tidefeed-app-01 sorts it 9 times out of 10
+[departing engineer] oh, and the Meridian support contract ran out in August. procurement are "looking at it". so there's nobody to call about gauge hardware right now
+[departing engineer] we never wrote down SLOs or set up proper dashboards. there's the /health endpoint and that's it
+[new owner] what about the --rollback thing in the notes?
+[departing engineer] honestly not sure it still works, I always just redeployed the old tag

package/packs/sre-oncall/skills/runbook-generator/evals/files/2/service-notes.md

@@ -0,0 +1,26 @@
+# TideFeed: service notes
+
+Owner: Port Systems team, Saltholm Harbour Authority
+
+## What it does
+
+TideFeed collects water-level readings from the four tide gauges in the harbour (North Pier, Inner Basin, Lock Gate, Outer Mark) every minute and publishes the current level and the next 6 hours of predicted levels to the pilots' berthing display and to harbour control.
+
+## Components
+
+- **gauge-collector**: polls the four gauges over the harbour radio link. Runs on the VM `tidefeed-app-01`.
+- **tidefeed-api**: serves levels to the berthing display. Same VM.
+- **PostgreSQL** on `tidefeed-db-01`: stores readings.
+- Both VMs sit in the harbour authority's own server room.
+
+## Deploying
+
+Deploys use `deploy.sh` from the ops laptop. To roll back, run `./deploy.sh --rollback`.
+
+Database migrations run automatically on deploy. Migration 0042 (October 2026) split the readings table by gauge and is forward-only: there is no down migration, and app versions before 2.6.0 do not work against the new schema.
+
+## Contacts
+
+- Harbour IT service desk: ext. 4410, staffed 08:00 to 18:00.
+- Out of hours: the Port Systems on-call engineer, via the harbour control room.
+- Gauge hardware: Meridian Gauges support line (contract ref MG-2291).

package/packs/sre-oncall/skills/runbook-generator/evals/files/2/standing-order.md

@@ -0,0 +1,3 @@
+# Harbour Master's Standing Order 14 (extract)
+
+If the berthing display shows tide data that is more than 15 minutes old, or shows no data, harbour control must immediately tell all pilots on duty to use the printed tide tables and the manual tide boards, and log the time. This applies whatever the cause and before any technical investigation.