@ketecode/loom 0.0.0-stage → 0.1.0

package/skills/autonomous-loops/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: autonomous-loops
+description: Runs an agent unattended over many iterations without it drifting or doing damage. Defines a measurable goal and stop conditions up front, caps iterations, time and spend, verifies each iteration with a real check, commits checkpoints so any step can be rolled back, keeps a progress log, and stops to escalate on repeated failure or risky actions. Use when the user wants the agent to keep going on its own ("loop until the tests pass", "work through this backlog overnight", "fix every lint error"), or when designing a headless or scheduled agent run.
+license: MIT
+compatibility: Any Agent Skills host. Needs git 2.23 or later (for git switch and git restore) and the project's check commands.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: meta
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "git"
+  kete-loom/owner: "Kete-org"
+---
+
+# Autonomous loops
+
+An unattended loop is a small program with an agent inside. It needs what any long-running job needs: a definition of done, limits, a health check each step, a log, and a way out. Agree all of it with the user **before** the first iteration, because nobody will be watching after.
+
+## 1. Write the loop contract
+
+Fill this in and show it to the user. **Do not start until they confirm it.**
+
+```
+Goal:          <measurable end state, e.g. "npm test exits 0" or "0 eslint errors in src/">
+Check:         <command that measures progress, and the number it reports>
+Scope:         <paths the loop may edit>; everything else is read-only
+Per iteration: <one unit of work, e.g. "fix one failing test file">
+Stop when:     goal met | <N> iterations | <T> minutes | <budget> spent |
+               no progress for 3 iterations | same error 3 times
+Escalate if:   <actions needing a human: see section 4>
+Branch:        <work branch, never main>
+Log:           <path, e.g. .handoff/loop-log.md>
+```
+
+If the goal can't be checked by a command, it is not ready for a loop. Help the user turn it into one, or run it attended.
+
+## 2. Prepare
+
+1. Create a work branch: `git switch -c loop/<topic>`. Never loop on the default branch.
+2. Make sure the tree is clean (`git status --short`). If it isn't, ask the user what to do with the changes.
+3. Run the check once and record the baseline in the log. If the baseline is already at the goal, stop.
+
+## 3. Each iteration
+
+1. **Read the log** (last few entries) so you don't repeat a failed approach.
+2. **Pick one unit** of work, the smallest that moves the check.
+3. **Make the change** inside the agreed scope.
+4. **Verify**: run the check, plus the project's fast tests or lint if the check alone could hide regressions.
+5. **Decide**:
+   - Better and nothing else broke: commit with a message naming the unit (`loop: fix parser.test.ts timeouts`).
+   - Not better, or something else broke: revert the change: `git restore --staged --worktree -- <tracked files you modified>`, then delete the files this iteration created (git restore rejects untracked paths), and note why. Only ever revert this iteration's own uncommitted work.
+6. **Log** one entry:
+
+   ```
+   ## Iteration 7 - 14:32
+   Did: raised timeout handling in src/parser.ts
+   Check: 12 failing -> 9 failing (kept, commit a1b2c3d)
+   Next: auth.test.ts fixtures
+   ```
+
+7. **Check stop conditions** before starting the next iteration.
+
+## 4. Escalate instead of acting
+
+Stop the loop and report to the user when the next step would:
+
+- Edit files outside the agreed scope, delete files, or change CI, secrets or deployment config.
+- Delete, skip or weaken a test, or lower a lint or coverage threshold to make the check pass.
+- Install, upgrade or remove dependencies (unless the contract allows it).
+- Push, open a PR, deploy, run migrations, or call paid or production services.
+- Repeat an approach the log already shows failed.
+
+Also stop if the check gets worse twice in a row, or the same error shows three times. That's a sign the loop is stuck, not that it needs more tries.
+
+## 5. Budgets
+
+- Keep a counter of iterations and elapsed time in the log header and compare against the contract each iteration.
+- If the host reports cost or token usage, track it the same way. If it doesn't, the iteration cap is the budget.
+- For long runs, checkpoint the context too: after every few iterations, make sure the log alone is enough to continue (see the `session-handoff` and `context-budget` skills).
+
+## 6. Finish
+
+When any stop condition hits, end with this report:
+
+```
+Stopped because: <goal met | cap reached | stuck | escalation>
+Check: <baseline> -> <final>
+Iterations: <n> (<kept> kept, <reverted> reverted)
+Branch: loop/<topic> at <sha>
+Needs a human: <escalations and open questions, or "nothing">
+```
+
+Leave the branch for review. Don't merge or push it unless the user asked for that in the contract.
+
+## Never
+
+- Never run a loop without a check command and a hard iteration cap.
+- Never game the check (skip tests, add ignores, hard-code expected output, special-case the test).
+- Never keep a change that made the check worse.
+- Never let the loop widen its own scope or extend its own budget.

package/skills/autonomous-loops/evals/triggers.json

@@ -0,0 +1,14 @@
+{
+  "should_trigger": [
+    "Keep fixing the failing tests on your own until the whole suite is green, I'll check back tomorrow",
+    "Set up a headless agent run that works through the lint errors in src/ one file at a time overnight",
+    "Loop on this: try a fix, run the benchmark, keep it if it's faster, otherwise revert",
+    "What guardrails do I need before letting the agent run unattended on our backlog"
+  ],
+  "should_not_trigger": [
+    "Run every check on my branch once and tell me if it's ready for a pull request",
+    "Fan this refactor out to four subagents and merge their results",
+    "Run this cleanup command every morning at 9 using a cron job on the server",
+    "Write a failing test first, then implement the date parser"
+  ]
+}

package/skills/aws-patterns/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: aws-patterns
+description: Designs and reviews workloads on Amazon Web Services, covering least-privilege IAM roles and policies, picking Lambda versus ECS Fargate versus EC2, S3 bucket hardening, RDS and Aurora setup, VPC subnets and security groups, Secrets Manager and Parameter Store, and the bill surprises that come from NAT gateways, public IPv4 and idle resources. Use when building or changing something on AWS, writing an IAM policy, choosing an AWS compute service, or asking why an AWS account costs more than expected.
+license: MIT
+compatibility: Any Agent Skills host. Live checks need the AWS CLI with credentials for the target account. Infrastructure code may be Terraform, CloudFormation, CDK or SAM.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: devops
+  kete-loom/risk: network
+  kete-loom/requires: "aws"
+  kete-loom/owner: "Kete-org"
+---
+
+# AWS patterns
+
+Build on AWS so that every principal has only the access it needs, nothing is public by accident, secrets never sit in code, and the monthly bill holds no surprises. Terraform syntax itself belongs to `terraform-patterns`; container image detail to `docker-patterns`; CI wiring to `github-actions-ci`.
+
+## Ground rules
+
+1. **Everything is code.** Create resources through the project's IaC (Terraform, CloudFormation, CDK or SAM). Console changes are for exploring only; if you find drift, report it, don't "fix" it by clicking.
+2. **Roles, not users.** Workloads get IAM roles (Lambda execution role, ECS task role, EC2 instance profile). People sign in through IAM Identity Center. Never create long-lived access keys for an app or a CI system; CI uses OIDC federation (see `github-actions-ci`).
+3. **Least privilege, scoped by resource.** No `"Action": "*"` or `"Resource": "*"` on write actions. Name the ARNs. Use conditions (`aws:SourceArn`, `aws:PrincipalOrgID`, `s3:prefix`) when a service or cross-account principal is involved.
+4. **Private by default.** S3 Block Public Access stays on at account and bucket level. Databases live in private subnets with no public endpoint. Serve public content through CloudFront with origin access control, not a public bucket.
+5. **Secrets in Secrets Manager or SSM Parameter Store (`SecureString`).** Never in environment variables committed to the repo, task definitions in git, or user data. Grant `secretsmanager:GetSecretValue` on that one secret ARN only.
+6. **Encrypt and log.** Encryption at rest on S3, EBS, RDS (default keys are fine unless compliance asks for customer-managed KMS keys). CloudTrail on in every region for the account.
+7. **Tag everything** with at least owner, environment and cost centre so cost reports mean something.
+
+## Choosing compute
+
+| Need | Pick | Watch out for |
+|---|---|---|
+| Event-driven, spiky, short tasks (API, queue consumer, cron) | Lambda | 15-minute maximum run time; cold starts; database connection storms (use RDS Proxy) |
+| Long-running containers without managing servers | ECS on Fargate | Pay per vCPU/memory even when idle; right-size tasks |
+| Many services, team already runs Kubernetes | EKS | Control-plane fee per cluster plus node cost; operational load. See `kubernetes-patterns` |
+| GPUs, special kernels, licensed software, steady high load | EC2 (Auto Scaling group) | You patch the OS; use SSM Session Manager instead of SSH keys and open port 22 |
+| Static site | S3 + CloudFront | Bucket stays private behind origin access control |
+
+Decision rule: start with the simplest option that meets the run-time and scaling need. If the user wants EKS for one or two services, ask whether they have Kubernetes experience before committing.
+
+## Networking basics
+
+1. One VPC per environment, spread across at least two Availability Zones.
+2. Public subnets hold only load balancers and NAT gateways. App and database subnets are private.
+3. Security groups reference other security groups, not CIDR ranges: the database SG allows port 5432 from the app SG only. Never `0.0.0.0/0` on anything but 80/443 on a load balancer.
+4. Add a gateway VPC endpoint for S3 (and DynamoDB if used): it costs nothing and keeps that traffic off the NAT gateway.
+5. Use interface endpoints for other heavy AWS API traffic (ECR, Secrets Manager, CloudWatch Logs) when NAT data charges are high.
+
+## Data
+
+- **S3:** versioning on for anything you can't regenerate; lifecycle rules to move or expire old objects and abort incomplete multipart uploads; bucket policy that denies non-TLS requests (`aws:SecureTransport` = false).
+- **RDS / Aurora:** Multi-AZ for production, automated backups with a retention that matches the recovery need, deletion protection on, storage encrypted, credentials managed by Secrets Manager (RDS can manage and rotate the master password). Connect with IAM auth or a secret, never a password in config.
+- **DynamoDB:** design keys around access patterns first; on-demand capacity until traffic is predictable.
+
+## Cost traps to check every time
+
+- NAT gateway hourly charge plus per-GB processing; one per AZ multiplies it. Traffic to S3 through NAT is the classic waste.
+- Public IPv4 addresses are billed per hour, including ones attached to running instances and idle Elastic IPs.
+- Cross-AZ and internet data transfer.
+- Unattached EBS volumes, old snapshots, idle load balancers, forgotten dev RDS instances.
+- CloudWatch Logs with no retention set (the default is never expire).
+- Lambda with oversized memory or a retry loop on a failing event source.
+
+Set an AWS Budgets alert on every account before building anything else.
+
+## Steps for a change
+
+1. Read the existing IaC and identify the account, region and environment. If you can't tell which account a change targets, **stop and ask**.
+2. Write the change in IaC following the ground rules.
+3. Validate it: `terraform plan`, `cdk diff`, `sam validate` or `aws cloudformation validate-template`. For IAM, run `aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://policy.json` and fix every error and security warning.
+4. Show the user the plan. **Stop and ask before** any apply that deletes or replaces a stateful resource (database, bucket, volume), widens a security group to the internet, or touches production.
+5. After apply, verify: `aws sts get-caller-identity` for identity, the service's health check, and that no new resource is public.
+
+## Never
+
+- Commit access keys, or put them in `.env` files that reach git.
+- Use the root user for anything except the few tasks that require it; root has MFA and no access keys.
+- Attach `AdministratorAccess` to a workload role "to get it working".
+- Run destructive CLI commands (`aws s3 rm --recursive`, `delete-db-instance`) without the user confirming the exact target.
+
+## Review output
+
+When asked to review AWS setup, report:
+
+```
+AWS review: <account/env>
+Critical: <public data, admin roles, leaked keys>
+High: <missing encryption, open security groups, no backups>
+Cost: <each trap found, rough monthly impact if known>
+Suggested IaC changes: <file: change>
+Not checked: <what you had no access to>
+```

package/skills/aws-patterns/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Write a least-privilege IAM policy so our Lambda can only read one S3 bucket and one secret",
+    "Should this background worker run on Lambda, ECS Fargate or plain EC2?",
+    "Our AWS bill doubled and most of it is NAT gateway data processing, what are we doing wrong?",
+    "Set up a VPC with private subnets for RDS Postgres and security groups that only let the app in",
+    "Where should our app's database password live on AWS instead of the task definition env vars?"
+  ],
+  "should_not_trigger": [
+    "Configure GitHub Actions to assume an AWS role with OIDC instead of access keys",
+    "Refactor our Terraform modules to use for_each instead of count",
+    "Deploy the service to Google Cloud Run with its own service account",
+    "Plan a canary rollout for the new checkout service with automatic rollback",
+    "Write a Kubernetes Deployment manifest with resource limits and probes"
+  ]
+}

package/skills/azure-patterns/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: azure-patterns
+description: Sets up and audits Microsoft Azure workloads with Entra ID and managed identities instead of stored credentials, Azure RBAC at resource scope, a reasoned choice among App Service, Container Apps, Azure Functions and AKS, Azure SQL and Cosmos DB configuration, Key Vault references, virtual networks with private endpoints and NSGs, and Azure spending pitfalls like idle App Service plans and over-provisioned RUs. Use when building on Azure, wiring a managed identity, deciding where an Azure app should run, or explaining an Azure invoice.
+license: MIT
+compatibility: Any Agent Skills host. Live checks need the Azure CLI signed in to the target subscription. Infrastructure code may be Bicep, ARM or Terraform.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: devops
+  kete-loom/risk: network
+  kete-loom/requires: "az"
+  kete-loom/owner: "Kete-org"
+---
+
+# Azure patterns
+
+Build on Azure so apps authenticate with managed identities, secrets live in Key Vault, data services sit behind private endpoints, and costs are predictable. Terraform detail is in `terraform-patterns`, Kubernetes manifests in `kubernetes-patterns`, CI login in `github-actions-ci`, .NET code in `aspnet-core-patterns`.
+
+## Ground rules
+
+1. **Structure:** management groups, then one subscription per environment (or at least per prod / non-prod), then resource groups per application and environment. Resource groups share a lifecycle: everything in one is deployed and deleted together.
+2. **Managed identities for workloads.** Use a system-assigned identity when one resource needs access; a user-assigned identity when several resources share it or it must exist before the resource. In code use `DefaultAzureCredential` (or the language's equivalent) so the same code works locally with a developer sign-in and in Azure with the identity.
+3. **No client secrets in apps.** If an app registration secret seems needed, **stop and ask** whether a managed identity or workload identity federation (for CI and for AKS) works instead. If a secret is unavoidable, store it in Key Vault and set an expiry reminder.
+4. **Azure RBAC, narrowest scope, data-plane roles.** Grant `Key Vault Secrets User` on the vault, `Storage Blob Data Reader` on the storage account or container. Never Owner or Contributor on a subscription for a workload.
+5. **Key Vault for secrets, keys and certificates.** Use the RBAC permission model, soft delete and purge protection. App Service and Functions read secrets through Key Vault references in app settings; Container Apps through secret references to Key Vault.
+6. **Everything in IaC** (Bicep, ARM or Terraform). Preview with `az deployment group what-if` or `terraform plan`.
+
+## Choosing compute
+
+| Need | Pick | Watch out for |
+|---|---|---|
+| Web app or API, code or container, simple ops | App Service | The App Service plan bills while running even with zero traffic; deployment slots for zero-downtime swaps |
+| Containerised microservices, scale to zero, Dapr or KEDA scaling | Container Apps | Set min replicas deliberately; workload profiles for dedicated capacity |
+| Event-driven functions (queues, timers, HTTP) | Azure Functions, Flex Consumption plan for new serverless apps | Linux Consumption is legacy and scheduled for retirement; check timeouts per plan |
+| Full Kubernetes control, many teams, custom operators | AKS | You own upgrades and node pools; use workload identity, not pod-managed identity |
+| Lift-and-shift VM software | Virtual Machines / scale sets | Patch management; Bastion instead of public RDP/SSH |
+
+Decision rule: App Service or Container Apps first. Choose AKS only when the user names a requirement those can't meet.
+
+## Networking basics
+
+1. One virtual network per environment (or hub-and-spoke for larger estates), with subnets per tier.
+2. Data services (Azure SQL, Cosmos DB, Storage, Key Vault) get **private endpoints** and public network access disabled; add the matching Private DNS zones so names resolve to the private IP.
+3. App Service and Functions reach the VNet through VNet integration; Container Apps run in a VNet-integrated environment.
+4. NSGs on subnets allow only the ports each tier needs, by service tag or application security group. Never open RDP (3389) or SSH (22) to the internet; use Azure Bastion.
+5. Public HTTP goes through Application Gateway or Front Door, with WAF if the user needs it.
+
+## Data
+
+- **Azure SQL Database:** Entra ID authentication with the app's managed identity (no SQL passwords in connection strings), Entra-only auth where possible; geo-redundant backups and point-in-time restore configured; serverless tier with auto-pause for dev.
+- **Cosmos DB:** choose the partition key from the dominant query; serverless or autoscale throughput rather than high fixed RU/s; use data-plane RBAC with the managed identity instead of account keys, and disable key-based auth when nothing needs it.
+- **Storage accounts:** disable anonymous blob access and shared key auth where clients support Entra; soft delete for blobs and containers; lifecycle management to cool or archive tiers.
+
+## Cost traps
+
+- App Service plans and Premium Functions plans that keep billing for apps nobody uses; several small plans that could share one.
+- Cosmos DB provisioned RU/s far above actual use; one container per tenant with minimum RU each.
+- AKS node pools that never scale down; forgotten dev clusters.
+- Log Analytics ingestion with no daily cap or retention policy.
+- Unattached managed disks, public IPs, idle Application Gateways, VMs stopped but not deallocated.
+
+Create a budget with alerts on each subscription first.
+
+## Steps for a change
+
+1. Confirm the subscription and resource group: `az account show`. If it is not the one the user means, **stop and ask**.
+2. Write the change in IaC with identities, RBAC assignments and private endpoints in the same change.
+3. Preview: `az deployment group what-if` or `terraform plan`. Read every delete and replace line.
+4. **Stop and ask before** any deployment in `Complete` mode, any delete or replace of a database, storage account or vault, any role assignment at subscription scope, or any production change.
+5. After deploying, verify the app's health endpoint, that the identity has exactly the intended role assignments (`az role assignment list --assignee <principalId>`), and that public network access is off where intended.
+
+## Never
+
+- Put storage account keys, Cosmos keys or SQL passwords in app settings or source.
+- Purge a soft-deleted Key Vault or delete a resource group without the user confirming the exact name.
+- Grant Owner to a service principal used by CI "to save time".
+
+## Review output
+
+```
+Azure review: <subscription/env>
+Critical: <secrets in config, public data endpoints, Owner on workloads>
+High: <no managed identity, missing backups, open NSG rules>
+Cost: <each trap found>
+Suggested IaC changes: <file: change>
+Not checked: <what you had no access to>
+```

package/skills/azure-patterns/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Give our App Service a managed identity so it can read secrets from Key Vault without a client secret",
+    "Should we host these microservices on Azure Container Apps or AKS?",
+    "Lock down Azure SQL and the storage account with private endpoints and disable public access",
+    "Our Cosmos DB costs are huge, how do we cut the provisioned RU/s spend?",
+    "We still run Azure Functions on the Linux Consumption plan, what should we move to?"
+  ],
+  "should_not_trigger": [
+    "Use azure/login in a GitHub Actions workflow with a federated credential",
+    "Set up IAM roles and a VPC for an ECS service on AWS",
+    "Configure Entity Framework Core migrations for our ASP.NET Core app",
+    "Pick between Cloud Run and GKE for a new service on Google Cloud",
+    "Write a blue-green deployment plan for the billing API"
+  ]
+}

package/skills/benchmark-methodology/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: benchmark-methodology
+description: Designs and runs performance benchmarks whose numbers hold up, with warmup, enough iterations, medians and p95/p99 with spread, a quiet and pinned machine, interleaved before/after comparison with a significance check, and guards against dead-code elimination and other microbenchmark traps in Go, JMH, BenchmarkDotNet, Criterion, pytest-benchmark and JS. Use when measuring whether a change made code faster or slower, writing a microbenchmark, or judging a performance claim.
+license: MIT
+compatibility: Any Agent Skills host. Needs the project's language toolchain; hyperfine is optional for command-level timing.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: testing
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# Benchmark methodology
+
+A benchmark is an experiment. Its job is to answer one question ("is B faster than A for this workload, by how much, and are we sure?") with numbers someone else can reproduce. Most bad benchmarks measure the wrong thing, too few times, on a noisy machine.
+
+## 1. Write down the question
+
+Before any code, state:
+
+1. **What is measured**: a function, an endpoint, a CLI command, a whole job.
+2. **The metric**: latency per operation, throughput, allocations, peak memory, startup time.
+3. **The workload**: realistic input sizes and shapes. Use production-like data, not `n = 10`. If you don't know the real sizes, **ask the user**.
+4. **The decision**: what result would change what we do ("ship if p95 improves by 10 % or more with no regression in allocations").
+
+If the user wants to know why the app is slow rather than compare versions, profile first (CPU profiler, flame graph) and benchmark the hot spot after.
+
+## 2. Pick the level and the tool
+
+- **Microbenchmark** (a function, nanoseconds to milliseconds): use the language harness, never a hand-rolled timer loop. See `references/language-harnesses.md`.
+- **Command or process** (a CLI, a build step): `hyperfine --warmup 3 --runs 20 'old-cmd' 'new-cmd'`; add `--prepare` to reset caches or state between runs and `--export-json` to keep raw data.
+- **Service** (an HTTP endpoint): a load tool at a fixed request rate, measuring latency percentiles at that rate. Measuring at maximum throughput hides latency problems.
+
+## 3. Control the environment
+
+- Same machine, same build flags (release / optimised), same runtime version, same input for A and B.
+- Close other heavy work. On laptops, plug in power and disable low-power modes. In CI, shared runners are noisy: use a dedicated runner, or only trust large differences.
+- Where possible, disable CPU frequency boost and pin to cores (Linux: `taskset -c 2 <cmd>`). Record if you could not.
+- Warm up: JIT compilation, caches, connection pools and lazy initialisation make first runs slow. Discard warmup iterations, or measure cold start separately on purpose.
+- Note the hardware, OS, runtime version and commit for every run.
+
+## 4. Run enough and keep the raw data
+
+- At least 10 samples per variant for microbenchmarks the harness doesn't already repeat (Go: `-count=10`); 20 or more for noisy ones.
+- **Interleave** A and B (A, B, A, B...) or alternate whole runs so thermal drift and background jobs hit both equally. Never run all of A in the morning and all of B after lunch.
+- Keep the raw samples, not only the mean.
+
+## 5. Summarise honestly
+
+- Report the **median** and **p95 / p99** for latency; the mean hides tail latency and is pulled by outliers.
+- Report spread: standard deviation, median absolute deviation or a confidence interval.
+- Compare with a significance test, not by eye. Go: `benchstat old.txt new.txt`. JMH and BenchmarkDotNet print error margins; differences inside the error bars are noise. For raw samples, a Mann-Whitney U test is a safe default.
+- Say "no measurable difference" when that is the result. A 2 % change with 5 % noise is not a win.
+- Check memory too: allocations per op (`-benchmem` in Go, `[MemoryDiagnoser]` in BenchmarkDotNet, `-prof gc` in JMH). A faster version that allocates far more may be slower in production.
+
+## 6. Microbenchmark traps
+
+Check each before trusting a number:
+
+1. **Dead-code elimination**: the compiler removes work whose result is unused. Consume results (`Blackhole` in JMH, `std::hint::black_box` in Rust, return the value from a BenchmarkDotNet method, `b.Loop()` in Go 1.24+).
+2. **Constant folding**: inputs known at compile time get precomputed. Load inputs from a variable or setup step.
+3. **Measuring setup**: allocating the input inside the timed region. Do setup outside it.
+4. **Caches too warm**: the same small input every iteration stays in L1 and branch predictors learn it. Use realistic sizes and varied data.
+5. **Timer resolution**: timing a 20 ns operation once is noise; harnesses batch iterations for this reason.
+6. **GC and JIT pauses** landing in one variant only. Run long enough and check variance.
+7. **Debug builds**: `cargo bench` and `go test -bench` optimise by default; for C and C++ check optimisation flags (`-O2`, CMake `Release`), for .NET build with `-c Release`; on the JVM, use JMH so the JIT has warmed up.
+
+Language-specific details and example code are in `references/language-harnesses.md`.
+
+## Rules
+
+- Never claim a speed-up from a single run, a debug build, or a different input.
+- Never tune the benchmark until it shows the result you hoped for; fix the method, then accept the answer.
+- Never commit benchmark code that makes network calls or depends on wall-clock time of day.
+- If results contradict across runs, **stop and report the noise** instead of picking the best run.
+
+## Report back
+
+```
+Question: <what was compared, workload>
+Environment: <CPU, OS, runtime version, commit A, commit B, noise controls>
+Method: <tool, warmup, samples per variant, interleaved yes/no>
+Results:
+| Variant | median | p95 | spread | allocs/op |
+Verdict: <B is X% faster (p < 0.05) | no measurable difference | B regressed>
+Caveats: <what could still skew it>
+Commands: <exact commands to reproduce>
+```

package/skills/benchmark-methodology/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Write a Go benchmark comparing the old and new JSON decoder and tell me if the new one is actually faster",
+    "My JMH numbers swing 30% between runs, how do I get results I can trust?",
+    "Is my refactor of the tokenizer a performance regression? Measure it properly before and after",
+    "Set up BenchmarkDotNet for our hashing code with allocation stats",
+    "Someone claims the Rust rewrite is 3x faster, check whether their benchmark is valid"
+  ],
+  "should_not_trigger": [
+    "Profile the API and fix whatever is making the orders endpoint slow",
+    "Our Lighthouse score dropped, improve the page load time",
+    "Raise the unit test coverage of the parser package to 80%",
+    "Write Playwright tests for the search page",
+    "Run the test suite and lint before I push"
+  ]
+}

package/skills/benchmark-methodology/references/language-harnesses.md

@@ -0,0 +1,130 @@
+# Benchmark harnesses by language
+
+Read this when writing a microbenchmark. Use the harness the project already has; add a new one only with the user's agreement.
+
+## Go (`testing`)
+
+```go
+func BenchmarkParse(b *testing.B) {
+    input := loadFixture(b) // setup is outside the timed loop with b.Loop
+    for b.Loop() {
+        Parse(input)
+    }
+}
+```
+
+- `b.Loop()` (Go 1.24+) times only the loop body and stops the compiler from optimising the body away. On older Go, use `for i := 0; i < b.N; i++`, call `b.ResetTimer()` after setup, and assign results to a package-level sink variable.
+- Run: `go test -run '^$' -bench BenchmarkParse -benchmem -count=10 ./pkg/parse > new.txt`.
+- Compare: `go install golang.org/x/perf/cmd/benchstat@latest`, then `benchstat old.txt new.txt`. It prints the change with a p-value; "~" means no significant difference.
+- Pitfalls: `-run '^$'` stops unit tests from running alongside; `-cpu 1,4` shows scaling.
+
+## Java and Kotlin (JMH)
+
+```java
+@State(Scope.Benchmark)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.NANOSECONDS)
+@Warmup(iterations = 5) @Measurement(iterations = 10) @Fork(3)
+public class ParseBench {
+    String input;
+    @Setup public void setup() { input = Fixtures.load(); }
+    @Benchmark public void parse(Blackhole bh) { bh.consume(Parser.parse(input)); }
+}
+```
+
+- Never benchmark the JVM with `System.nanoTime()` loops: JIT tiers, on-stack replacement and dead-code elimination make them meaningless.
+- `@Fork` of 2 or more runs separate JVMs so one JIT profile doesn't decide the result.
+- `-prof gc` reports allocation rate; `-prof async` (async-profiler) shows where time goes.
+- Use `@Param` for input sizes so you see how cost scales.
+
+## C# / .NET (BenchmarkDotNet)
+
+```csharp
+[MemoryDiagnoser]
+public class ParseBench {
+    private string _input = default!;
+    [GlobalSetup] public void Setup() => _input = Fixtures.Load();
+    [Benchmark(Baseline = true)] public Result Old() => OldParser.Parse(_input);
+    [Benchmark] public Result New() => NewParser.Parse(_input);
+}
+// Program.cs: BenchmarkRunner.Run<ParseBench>();
+```
+
+- Run in Release: `dotnet run -c Release`. BenchmarkDotNet warns when it detects a debug build or an attached debugger.
+- Return the result from the method so it isn't eliminated.
+- `Baseline = true` adds a Ratio column for before/after.
+
+## Rust (Criterion or `cargo bench`)
+
+```rust
+use criterion::{criterion_group, criterion_main, Criterion};
+use std::hint::black_box;
+
+fn bench(c: &mut Criterion) {
+    let input = load_fixture();
+    c.bench_function("parse", |b| b.iter(|| parse(black_box(&input))));
+}
+criterion_group!(benches, bench);
+criterion_main!(benches);
+```
+
+- Needs `[[bench]] name = "parse" harness = false` in `Cargo.toml`.
+- Criterion saves a baseline and reports change and significance against the previous run; `cargo bench -- --save-baseline before` then `--baseline before` compares named baselines.
+- `black_box` on inputs prevents constant folding; Criterion handles the output.
+
+## Python (pytest-benchmark, pyperf, timeit)
+
+```python
+def test_parse(benchmark):
+    data = load_fixture()
+    result = benchmark(parse, data)
+    assert result.ok
+```
+
+- Run: `pytest --benchmark-only --benchmark-autosave`, then compare saved runs with `pytest-benchmark compare`.
+- For stand-alone scripts, `pyperf` handles warmup, multiple processes and system tuning advice (`python -m pyperf system tune` on Linux).
+- `python -m timeit -s 'setup' 'stmt'` is fine for a quick check, not for a decision.
+- Interpreter version matters a lot between releases; record it.
+
+## JavaScript and TypeScript (tinybench, mitata, Vitest bench)
+
+```ts
+import { Bench } from 'tinybench';
+const bench = new Bench({ time: 1000 }); // warmup option names differ between tinybench majors; check the installed version's docs
+const input = loadFixture();
+bench.add('old', () => oldParse(input)).add('new', () => newParse(input));
+await bench.run();
+console.table(bench.table());
+```
+
+- V8 optimises and deoptimises based on observed types; feed realistic, varied inputs or one shape will look unrealistically fast.
+- Keep the result reachable (return it, or push into an array checked after the run).
+- Vitest has `bench()` in `*.bench.ts` files, run with `npx vitest bench`.
+- Node version and flags change results; record `node --version`.
+
+## C and C++ (Google Benchmark)
+
+```cpp
+static void BM_Parse(benchmark::State& state) {
+  auto input = LoadFixture();
+  for (auto _ : state) benchmark::DoNotOptimize(Parse(input));
+}
+BENCHMARK(BM_Parse);
+BENCHMARK_MAIN();
+```
+
+- Build with optimisations (`-O2` or `-DCMAKE_BUILD_TYPE=Release`); the library warns if it was built as debug.
+- `--benchmark_repetitions=10 --benchmark_report_aggregates_only=true` gives mean, median and stddev.
+- `benchmark::ClobberMemory()` forces pending writes to be treated as observable.
+
+## Commands and processes (hyperfine)
+
+```sh
+# build both versions first (e.g. ./old-bin from main, ./new-bin from the branch)
+hyperfine --warmup 3 --runs 30 --export-json bench.json \
+  './old-bin input.txt' './new-bin input.txt'
+```
+
+- `--prepare 'cmd'` runs before each timing run (for example, clearing a cache directory).
+- Never put state-changing commands (`git stash`, `git checkout`) inside a timed command; build each variant once, then time the binaries.
+- hyperfine reports mean, standard deviation, min/max and a relative comparison; it warns about statistical outliers.