switchroom 0.5.0

package/skills/buildkite-secure-delivery/SKILL.md

@@ -0,0 +1,168 @@
+---
+name: buildkite-secure-delivery
+description: >
+  This skill should be used when the user asks to "publish to package registry",
+  "push a Docker image", "set up OIDC authentication", "request an OIDC token",
+  "authenticate without static credentials", "set up SLSA provenance",
+  "generate attestation", "sign pipelines", "verify pipeline signatures",
+  or "secure the supply chain".
+  Also use when the user mentions OIDC, SLSA, provenance, attestation, cosign,
+  JWKS, pipeline signing, pipeline verification, packages.buildkite.com,
+  Package Registry, artifact signing, or asks about credential-free publishing,
+  supply chain security, or secure delivery in Buildkite.
+---
+
+# Buildkite Secure Delivery
+
+Secure delivery covers the end-to-end flow of publishing artifacts with zero static credentials and proving supply chain integrity. This skill teaches OIDC-based authentication, Package Registry publishing, SLSA provenance attestation, and pipeline signing with JWKS.
+
+## Quick Start
+
+Build, authenticate via OIDC, and push a Docker image — no static credentials:
+
+```yaml
+steps:
+  - key: "docker-publish"
+    label: ":docker: Build & Push"
+    commands:
+      - docker build --tag packages.buildkite.com/my-org/my-registry/my-app:latest .
+      - buildkite-agent oidc request-token --audience "https://packages.buildkite.com/my-org/my-registry" --lifetime 300 | docker login packages.buildkite.com/my-org/my-registry --username buildkite --password-stdin
+      - docker push packages.buildkite.com/my-org/my-registry/my-app:latest
+    plugins:
+      - generate-provenance-attestation#v1.1.0:
+          artifacts: "my-app:latest"
+          attestation_name: "docker-attestation.json"
+```
+
+This single step authenticates with a short-lived OIDC token (5 minutes), pushes the image, and generates a SLSA provenance attestation. No API keys or registry passwords stored anywhere.
+
+> For `buildkite-agent oidc request-token` flag details, see the **buildkite-agent-runtime** skill.
+
+## OIDC Authentication
+
+Each job calls `buildkite-agent oidc request-token` to get a short-lived JWT (default 5 minutes) from the Buildkite backend. External services validate the JWT against Buildkite's OIDC provider (`https://agent.buildkite.com`) and grant access if claims match the trust policy.
+
+### Token Claims
+
+The `sub` claim encodes the full context: `organization:{org}:pipeline:{slug}:ref:{ref}:commit:{sha}:step:{key}`. Key claims for trust policies: `organization_slug`, `pipeline_slug`, `build_branch`, `step_key`, `runner_environment`.
+
+### OIDC with Buildkite Package Registry
+
+The `--audience` must exactly match the registry URL: `https://packages.buildkite.com/{org-slug}/{registry-slug}`. The username is always `buildkite`. The OIDC token acts as the password. See Quick Start for the full pattern.
+
+### OIDC with Cloud Providers
+
+OIDC tokens work with AWS (via `aws-assume-role-with-web-identity` plugin or `sts:AssumeRoleWithWebIdentity`), GCP (via Workload Identity Federation), and Azure (via federated credentials). Each provider validates the JWT against Buildkite's OIDC issuer and grants access based on claim conditions.
+
+> For cloud provider OIDC setup including IAM trust policies, Workload Identity Pools, and Azure app registration, see `references/oidc-cloud-providers.md`.
+
+### Scoping OIDC Policies
+
+Always restrict trust policies to minimum scope. Scope `sub` to pipeline and branch (`organization:acme-inc:pipeline:deploy-prod:ref:refs/heads/main:*`), never to the entire org (`organization:acme-inc:*`). Use `pipeline_slug` and `build_branch` conditions. For production deployments, require `build_branch: main`.
+
+## Package Registry
+
+Buildkite Package Registry hosts packages across multiple ecosystems with OIDC-native authentication. Registries are scoped to an organization and accessed at `packages.buildkite.com/{org-slug}/{registry-slug}`.
+
+### Supported Ecosystems
+
+| Ecosystem | Auth method |
+|-----------|-------------|
+| Docker / OCI | `docker login` with OIDC token |
+| npm | `.npmrc` with OIDC token |
+| Helm (OCI) | `helm registry login` with OIDC token |
+| Python, Ruby, Terraform, Debian, Alpine, RPM, Generic | `curl` with Bearer token header |
+
+### Docker / OCI Publishing
+
+The most common pattern -- build, authenticate via OIDC, push. Tag images with `${BUILDKITE_BUILD_NUMBER}` or git SHA for traceability. Avoid `latest` tags in production -- they are not immutable. See Quick Start for the full pipeline step.
+
+> For npm, Helm, Python, Ruby, Terraform, and generic publishing patterns, see `references/package-publishing.md`.
+
+## SLSA Provenance
+
+SLSA provenance records what was built, when, by whom, and from which source. Add `generate-provenance-attestation` to build steps to create signed attestations, then use `publish-to-packages` to upload artifacts with attestations attached.
+
+### Generating Attestations
+
+```yaml
+plugins:
+  - generate-provenance-attestation#v1.1.0:
+      artifacts: "my-library-*.gem"     # Glob pattern matching artifacts to attest
+      attestation_name: "build-attestation.json"  # Output attestation filename
+```
+
+The attestation captures builder identity, source (repo, branch, commit), build metadata, and material digests (SHA-256).
+
+### Publishing with Attestations
+
+Use `publish-to-packages` with `artifacts` (glob pattern), `registry` (`{org-slug}/{registry-slug}`), and `attestations` (list of attestation files). The `generate-provenance-attestation` plugin satisfies SLSA Build Level 1. For Level 2, combine with Hosted Agents. For Level 3, add pipeline signing.
+
+## Pipeline Signing (JWKS)
+
+Pipeline signing ensures the steps an agent runs are exactly the steps uploaded. Uploading agents sign pipeline definitions with a private JWKS key; executing agents verify signatures with the public key before running jobs.
+
+### Setup
+
+Generate keys with `buildkite-agent tool keygen --alg EdDSA --key-id my-signing-key`. This creates `EdDSA-my-signing-key-private.json` (signing) and `EdDSA-my-signing-key-public.json` (verification). Store the private key securely.
+
+Configure `buildkite-agent.cfg`:
+- **Uploading agents**: set `signing-jwks-file` (private key) + `signing-jwks-key-id`
+- **Executing agents**: set `verification-jwks-file` (public key)
+- **Both roles**: set all three config keys
+
+### Rollout
+
+1. Deploy signing on uploading agents
+2. Deploy verification in warn mode (`verification-failure-behavior=warn`)
+3. Monitor and fix unsigned pipeline warnings in agent logs
+4. Switch to block (remove `verification-failure-behavior=warn` -- default is `block`)
+
+Steps defined in the Buildkite UI are not agent-signed. Sign them manually with `buildkite-agent tool sign --update`. For Kubernetes, mount JWKS keys as secrets and configure `signingJWKSVolume`/`verificationJWKSVolume`.
+
+> For `buildkite-agent.cfg` configuration details, see the **buildkite-agent-infrastructure** skill.
+
+## End-to-End Secure Publish Flow
+
+A complete pipeline combines: (1) **Build & Attest** -- build image, generate provenance with `generate-provenance-attestation`; (2) **Publish** -- authenticate via OIDC, push image, attach attestation via `publish-to-packages` (using `depends_on`); (3) **Deploy** -- authenticate to cloud via OIDC (e.g., `aws-assume-role-with-web-identity`). See Quick Start for the single-step pattern.
+
+> For a non-Docker example (Ruby gem secure publish flow), see `references/package-publishing.md`.
+
+## Security Checklist
+
+- [ ] All authentication uses OIDC tokens -- no static API keys or passwords
+- [ ] OIDC `--audience` matches exact target service URL
+- [ ] `--lifetime 300` (5 minutes) or less
+- [ ] Trust policies scoped to `pipeline_slug` + `build_branch` + `organization_slug`
+- [ ] Published artifacts include SLSA attestation
+- [ ] Pipeline signing enabled with `verification-failure-behavior=block` in production
+- [ ] Dynamically-retrieved secrets added to the log redactor
+
+> For `buildkite-agent redactor add` syntax, see the **buildkite-agent-runtime** skill.
+> For `secrets:` pipeline YAML syntax, see the **buildkite-pipelines** skill.
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| OIDC `--audience` doesn't match registry URL exactly | Use `https://packages.buildkite.com/{org}/{registry}` with exact slugs |
+| Using static API keys instead of OIDC | Replace `docker login -p $API_KEY` with the OIDC pipe pattern |
+| `--lifetime` too long (e.g., 3600) | Set `--lifetime 300` (5 minutes) |
+| Skipping `warn` phase during signing rollout | Deploy with `verification-failure-behavior=warn` first, then switch to `block` |
+| OIDC trust policy too broad (`sub: organization:*`) | Scope to specific `pipeline_slug` and `build_branch` |
+| Forgetting to sign Pipeline Settings steps | Run `buildkite-agent tool sign --update` for pipelines with UI steps |
+| Unversioned plugins | Pin to exact version (`plugin#v1.2.0`) |
+
+## Additional Resources
+
+### Reference Files
+- **`references/oidc-cloud-providers.md`** -- OIDC setup for AWS (IAM trust policies, session tags), GCP (Workload Identity Federation), and Azure (federated credentials)
+- **`references/package-publishing.md`** -- Publishing patterns for npm, Helm, Python, Ruby, Terraform, and generic artifacts, plus installing from Package Registry
+
+## Further Reading
+
+- [OIDC with Buildkite Pipelines](https://buildkite.com/docs/pipelines/security/oidc.md) -- OIDC configuration and trust policies
+- [SLSA Provenance](https://buildkite.com/docs/package-registries/security/slsa-provenance.md) -- attestation generation and verification
+- [Signed Pipelines](https://buildkite.com/docs/agent/v3/signed-pipelines.md) -- JWKS key generation, agent configuration, and rollout
+- [Package Registries Overview](https://buildkite.com/docs/package-registries.md) -- supported ecosystems and registry management
+- [Buildkite Docs for LLMs](https://buildkite.com/docs/llms.txt)

package/skills/buildkite-secure-delivery/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "Buildkite Secure Delivery"
+  short_description: "OIDC authentication, Package Registry, SLSA provenance, and pipeline signing"
+  icon_small: "./assets/buildkite-icon-small.png"
+  icon_large: "./assets/buildkite-icon-large.png"
+  brand_color: "#00D974"

package/skills/buildkite-secure-delivery/references/oidc-cloud-providers.md

@@ -0,0 +1,83 @@
+# OIDC Cloud Provider Integration
+
+## OIDC with AWS
+
+Request an OIDC token for AWS, then assume an IAM role using web identity federation:
+
+```bash
+buildkite-agent oidc request-token --audience sts.amazonaws.com
+```
+
+Use the `aws-assume-role-with-web-identity` plugin for a streamlined pipeline step:
+
+```yaml
+steps:
+  - label: ":aws: Deploy"
+    command: ./scripts/deploy.sh
+    env:
+      AWS_DEFAULT_REGION: us-east-1
+      AWS_REGION: us-east-1
+    plugins:
+      - aws-assume-role-with-web-identity#v1.2.0:
+          role-arn: arn:aws:iam::012345678910:role/my-deploy-role
+          session-tags:
+            - organization_slug
+            - pipeline_slug
+```
+
+AWS IAM trust policy requirements:
+- Set the OIDC provider to `https://agent.buildkite.com`
+- Set the audience to `sts.amazonaws.com`
+- Add conditions on `sub` or individual claims (`organization_slug`, `pipeline_slug`, `build_branch`) to restrict which pipelines can assume the role
+
+To include AWS session tags in the token, use `--aws-session-tag`:
+
+```bash
+buildkite-agent oidc request-token \
+  --audience sts.amazonaws.com \
+  --aws-session-tag "organization_slug,organization_id"
+```
+
+This adds an `https://aws.amazon.com/tags` claim with `principal_tags` for use in tag-based IAM policies.
+
+## OIDC with GCP
+
+Use GCP Workload Identity Federation to exchange Buildkite OIDC tokens for GCP credentials:
+
+1. Create a Workload Identity Pool and OIDC Provider:
+
+```bash
+gcloud iam workload-identity-pools create buildkite-pool \
+  --display-name "Buildkite Pool"
+
+gcloud iam workload-identity-pools providers create-oidc buildkite-provider \
+  --workload-identity-pool buildkite-pool \
+  --issuer-uri "https://agent.buildkite.com" \
+  --attribute-mapping "google.subject=assertion.sub,attribute.pipeline_slug=assertion.pipeline_slug,attribute.organization_slug=assertion.organization_slug"
+```
+
+2. Grant the pool's service account the necessary IAM roles
+3. Request a token in the pipeline step with the pool's audience:
+
+```bash
+buildkite-agent oidc request-token \
+  --audience "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/buildkite-pool/providers/buildkite-provider"
+```
+
+Use attribute conditions on `pipeline_slug` or `organization_slug` to restrict which pipelines can authenticate.
+
+## OIDC with Azure
+
+Azure supports Workload Identity Federation with Buildkite OIDC:
+
+1. Register an app in Azure AD with federated credentials
+2. Set the issuer to `https://agent.buildkite.com`
+3. Set the subject to the `sub` claim pattern for the target pipeline
+4. Request a token in the pipeline step:
+
+```bash
+buildkite-agent oidc request-token \
+  --audience "api://AzureADTokenExchange"
+```
+
+Use the Azure CLI or SDK to exchange the token for an access token.

package/skills/buildkite-secure-delivery/references/package-publishing.md

@@ -0,0 +1,100 @@
+# Package Registry Publishing Guide
+
+## npm Publishing
+
+Configure `.npmrc` with the registry URL and authenticate with an OIDC token:
+
+```yaml
+steps:
+  - label: ":npm: Publish Package"
+    commands:
+      - export OIDC_TOKEN=$(buildkite-agent oidc request-token --audience "https://packages.buildkite.com/acme-inc/npm-packages" --lifetime 300)
+      - |
+        cat > .npmrc << EOF
+        //packages.buildkite.com/acme-inc/npm-packages/:_authToken=${OIDC_TOKEN}
+        registry=https://packages.buildkite.com/acme-inc/npm-packages/
+        EOF
+      - npm publish
+```
+
+## Helm Chart Publishing (OCI)
+
+Push Helm charts to a Buildkite Helm OCI registry:
+
+```yaml
+steps:
+  - label: ":helm: Publish Chart"
+    commands:
+      - helm package ./chart
+      - buildkite-agent oidc request-token --audience "https://packages.buildkite.com/acme-inc/helm-charts" --lifetime 300 | helm registry login packages.buildkite.com/acme-inc/helm-charts --username buildkite --password-stdin
+      - helm push my-chart-1.0.0.tgz oci://packages.buildkite.com/acme-inc/helm-charts
+```
+
+## Python / Ruby / Generic Publishing
+
+For ecosystems that use direct HTTP upload, request an OIDC token and pass it as a Bearer token:
+
+```yaml
+steps:
+  - label: ":python: Publish Package"
+    commands:
+      - export OIDC_TOKEN=$(buildkite-agent oidc request-token --audience "https://packages.buildkite.com/acme-inc/python-packages" --lifetime 300)
+      - python -m build
+      - curl -X POST "https://api.buildkite.com/v2/packages/organizations/acme-inc/registries/python-packages/packages" -H "Authorization: Bearer ${OIDC_TOKEN}" -F "file=@dist/my-package-1.0.0.tar.gz"
+```
+
+The same pattern applies to Ruby gems, Debian packages, RPMs, Alpine packages, Terraform modules, and generic artifacts -- change the file path and registry slug.
+
+## Terraform Module Publishing
+
+Terraform module filenames must follow the naming convention `terraform-{provider}-{module}-{major.minor.patch}.tgz`:
+
+```yaml
+steps:
+  - label: ":terraform: Publish Module"
+    commands:
+      - export OIDC_TOKEN=$(buildkite-agent oidc request-token --audience "https://packages.buildkite.com/acme-inc/terraform-modules" --lifetime 300)
+      - tar czf terraform-buildkite-pipeline-1.0.0.tgz -C modules .
+      - curl -X POST "https://api.buildkite.com/v2/packages/organizations/acme-inc/registries/terraform-modules/packages" -H "Authorization: Bearer ${OIDC_TOKEN}" -F "file=@terraform-buildkite-pipeline-1.0.0.tgz"
+```
+
+## Installing from Package Registry
+
+Pull packages using the same OIDC pattern. For Docker images:
+
+```bash
+buildkite-agent oidc request-token \
+  --audience "https://packages.buildkite.com/acme-inc/docker-images" \
+  --lifetime 300 \
+  | docker login packages.buildkite.com/acme-inc/docker-images \
+      --username buildkite --password-stdin
+
+docker pull packages.buildkite.com/acme-inc/docker-images/web-app:42
+```
+
+For npm, configure `.npmrc` with the read token. For Helm, use `helm registry login` then `helm pull`.
+
+## Ruby Gem Secure Publish Flow
+
+A non-Docker example using the `publish-to-packages` plugin directly:
+
+```yaml
+steps:
+  - label: ":ruby: Build Gem"
+    key: "build-gem"
+    command: "gem build my-library.gemspec"
+    artifact_paths: "my-library-*.gem"
+    plugins:
+      - generate-provenance-attestation#v1.1.0:
+          artifacts: "my-library-*.gem"
+          attestation_name: "gem-attestation.json"
+
+  - label: ":package: Publish Gem"
+    depends_on: "build-gem"
+    plugins:
+      - publish-to-packages#v2.2.0:
+          artifacts: "my-library-*.gem"
+          registry: "acme-inc/ruby-gems"
+          attestations:
+            - "gem-attestation.json"
+```

package/skills/buildkite-test-engine/SKILL.md

@@ -0,0 +1,239 @@
+---
+name: buildkite-test-engine
+description: >
+  This skill should be used when the user asks to "split tests across machines",
+  "set up test splitting", "parallelize test suite", "detect flaky tests",
+  "quarantine flaky tests", "configure test collectors", "speed up tests",
+  "set up bktec", "configure test engine", or "reduce flaky test failures".
+  Also use when the user mentions bktec, Test Engine, test suites,
+  BUILDKITE_TEST_ENGINE_* environment variables, BUILDKITE_ANALYTICS_TOKEN,
+  test-collector plugin, test reliability scores, test timing data,
+  or asks about Buildkite test splitting and flaky test management.
+---
+
+# Buildkite Test Engine
+
+Test Engine splits test suites across parallel machines and identifies flaky tests. Two components: **test collectors** gather timing data from runs, and **bktec** (the CLI) uses that data for intelligent splitting and automatic flaky test management.
+
+## Quick Start
+
+Three steps: create a suite, add a collector, run bktec with parallelism.
+
+**1. Create a test suite** in the Buildkite dashboard: Test Suites > New test suite > Set up suite. Copy the suite API token.
+
+**2. Add the test-collector plugin** to start gathering timing data:
+
+```yaml
+steps:
+  - label: ":rspec: Tests"
+    command: "bundle exec rspec"
+    plugins:
+      - test-collector#v2.0.0:
+          files: "tmp/rspec-*.xml"
+          format: "junit"
+    env:
+      BUILDKITE_ANALYTICS_TOKEN: "your-suite-api-token"
+```
+
+**3. After ~1-2 weeks of data**, switch to bktec for intelligent splitting:
+
+```yaml
+steps:
+  - label: ":rspec: Tests %n"
+    command: bktec
+    parallelism: 10
+    env:
+      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: "your-api-access-token"
+      BUILDKITE_TEST_ENGINE_SUITE_SLUG: "my-suite"
+      BUILDKITE_TEST_ENGINE_TEST_RUNNER: "rspec"
+      BUILDKITE_TEST_ENGINE_RESULT_PATH: "tmp/rspec-result.json"
+```
+
+> For `parallelism:` YAML syntax and pipeline structure, see the **buildkite-pipelines** skill.
+
+## How Test Engine Works
+
+Test Engine is a two-phase system:
+
+**Phase 1 — Data collection:** Test collectors send execution timing and pass/fail results to the Test Engine API after every run, building a historical profile for each test.
+
+**Phase 2 — Smart splitting + flaky management:** bktec reads historical data to partition tests across parallel agents by runtime, and to identify and quarantine flaky tests.
+
+### The Two Token Types
+
+Test Engine uses two different tokens:
+
+| Token | Environment Variable | Purpose | Where to get it |
+|-------|---------------------|---------|-----------------|
+| **Suite API token** | `BUILDKITE_ANALYTICS_TOKEN` | Collectors use this to send test data to a specific suite | Test suite settings page |
+| **API access token** | `BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN` | bktec uses this to fetch timing data and test plans | Personal Settings > API Access Tokens (requires `read_suites` scope) |
+
+These are not interchangeable.
+
+## Creating Test Suites
+
+Create via the dashboard: **Test Suites > New test suite > Set up suite**. The suite settings page shows the **suite API token** (for collectors) and the **suite slug** (for bktec).
+
+> For creating suites via REST API, see the **buildkite-api** skill.
+
+### Suites and pipelines
+
+Pipelines and suites do not need a one-to-one relationship. Multiple pipelines can report to the same suite (monorepos), and one pipeline can report to multiple suites (separate unit/integration suites).
+
+## Test Collectors
+
+Install the collector for the test framework, set `BUILDKITE_ANALYTICS_TOKEN`, and run tests normally. Collectors must be configured before bktec splitting works.
+
+- **Ruby** — `buildkite-test_collector` gem (RSpec, Minitest)
+- **JavaScript** — `buildkite-test-collector` npm package (Jest, Playwright, Cypress)
+- **Python** — bktec handles collection directly when runner is `pytest`
+- **Go / other languages** — JUnit XML upload via the analytics API
+- **Any framework** — `test-collector` Buildkite plugin for file-based upload
+
+> For per-framework setup instructions and configuration examples, see **`references/collectors.md`**.
+
+## bktec CLI
+
+bktec is the CLI that replaces the test runner command in pipeline steps. It fetches a test plan from the Test Engine API balanced by historical runtime, runs the assigned subset, uploads results, and optionally retries failed tests.
+
+### Installation
+
+Pre-installed on Buildkite hosted agents. For self-hosted agents:
+
+```bash
+curl -sL https://github.com/buildkite/test-engine-client/releases/latest/download/bktec-linux-amd64 -o /usr/local/bin/bktec
+chmod +x /usr/local/bin/bktec
+```
+
+### Supported test runners
+
+`rspec` (supports split-by-example), `jest`, `playwright`, `cypress`, `pytest`, `pytest-pants`, `go`, `cucumber` — all split by file/spec/package except RSpec which also supports split-by-example.
+
+If bktec cannot reach the API or no timing data exists, it falls back to file-count splitting. Enable `BUILDKITE_TEST_ENGINE_DEBUG_ENABLED` to verify the splitting strategy.
+
+## bktec Environment Variables
+
+### Required variables
+
+| Variable | Description |
+|----------|-------------|
+| `BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN` | API access token for authenticating with Test Engine (requires `read_suites` scope) |
+| `BUILDKITE_TEST_ENGINE_SUITE_SLUG` | Slug of the test suite to fetch timing data from |
+| `BUILDKITE_TEST_ENGINE_TEST_RUNNER` | Test runner to use: `rspec`, `jest`, `playwright`, `cypress`, `pytest`, `pytest-pants`, `go`, `cucumber` |
+| `BUILDKITE_TEST_ENGINE_RESULT_PATH` | Path where bktec writes test results (e.g., `tmp/rspec-result.json`) |
+
+### Optional variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BUILDKITE_TEST_ENGINE_RETRY_COUNT` | `0` | Number of times to retry failed tests. Set to `2` for flaky detection |
+| `BUILDKITE_TEST_ENGINE_SPLIT_BY_EXAMPLE` | `false` | Split by individual test example instead of by file (RSpec only) |
+| `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN` | Runner default | Glob pattern to select test files (e.g., `spec/**/*_spec.rb`) |
+| `BUILDKITE_TEST_ENGINE_DEBUG_ENABLED` | `false` | Enable debug logging to see splitting strategy and file assignments |
+
+Standard Buildkite environment variables (`BUILDKITE_BUILD_ID`, `BUILDKITE_PARALLEL_JOB`, `BUILDKITE_PARALLEL_JOB_COUNT`, etc.) are used automatically by bktec. When running in Docker, expose them explicitly via the plugin `environment` list.
+
+## Test Splitting
+
+### Timing-based splitting
+
+With historical timing data, bktec assigns files to agents based on cumulative runtime so each agent finishes in approximately the same time. The test plan updates every build, so splitting improves continuously as more data accumulates.
+
+Pipeline configuration is shown in Quick Start step 3. Use `%n` in the label to show the parallel job index.
+
+> For complete per-framework pipeline examples (RSpec, Jest, pytest, Go), split-by-example vs split-by-file guidance, parallelism tuning table, and custom test command configuration, see **`references/splitting-examples.md`**.
+
+## Flaky Test Detection
+
+Test Engine flags a test as flaky when the same test on the same commit produces both pass and fail results.
+
+### Automatic retry for flaky detection
+
+Set `BUILDKITE_TEST_ENGINE_RETRY_COUNT: "2"` to retry failed tests. If a test fails then passes on retry, it is flagged as flaky. The build passes if all tests eventually pass.
+
+> For listing flaky tests via the REST API, see the **buildkite-api** skill.
+
+### MCP tools for flaky test investigation
+
+When the Buildkite MCP server is available: `list_test_runs` (pass/fail trends), `get_test_run` (run details), `get_failed_executions` (error messages and stack traces), `get_build_test_engine_runs` (runs for a build).
+
+Triage: identify flaky tests via MCP tools or the dashboard, quarantine confirmed flakes, fix root causes, then remove from quarantine. Target < 3% flaky rate.
+
+## Test States and Quarantine
+
+### Test states
+
+| State | Meaning | bktec behavior |
+|-------|---------|----------------|
+| **Active** | Test runs normally | Included in test runs |
+| **Muted** | Test runs but failures are suppressed | Included in test runs; failures do not fail the build |
+| **Quarantined** | Test is excluded from runs | Excluded from test runs entirely |
+
+### How quarantine works
+
+- bktec automatically excludes quarantined tests from runs, so they cannot fail the build
+- Supported for RSpec, Jest, and Playwright runners
+
+No special pipeline configuration is needed — bktec reads test states automatically. Change states through the dashboard. Fix root causes, then move tests back to Active.
+
+## Docker and Containerized Environments
+
+When running tests inside Docker, expose both `BUILDKITE_TEST_ENGINE_*` and standard Buildkite variables via the Docker plugin `environment` list:
+
+```yaml
+steps:
+  - label: ":docker: Tests %n"
+    command: bktec
+    parallelism: 10
+    plugins:
+      - docker#v5.12.0:
+          image: "myapp:test"
+          environment:
+            - BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN
+            - BUILDKITE_TEST_ENGINE_SUITE_SLUG
+            - BUILDKITE_TEST_ENGINE_TEST_RUNNER
+            - BUILDKITE_TEST_ENGINE_RESULT_PATH
+            - BUILDKITE_BUILD_ID
+            - BUILDKITE_PARALLEL_JOB
+            - BUILDKITE_PARALLEL_JOB_COUNT
+    env:
+      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: "your-api-access-token"
+      BUILDKITE_TEST_ENGINE_SUITE_SLUG: "my-suite"
+      BUILDKITE_TEST_ENGINE_TEST_RUNNER: "rspec"
+      BUILDKITE_TEST_ENGINE_RESULT_PATH: "tmp/rspec-result.json"
+```
+
+## End-to-End Setup Walkthrough
+
+- **Week 1:** Run the collector pipeline (Quick Start step 2) for ~1-2 weeks to accumulate timing data.
+- **Week 2+:** Switch to bktec (Quick Start step 3). Add `BUILDKITE_TEST_ENGINE_RETRY_COUNT: "2"`.
+- **Week 3+:** Review the dashboard for flaky tests. Quarantine confirmed flakes. Target < 3% flaky rate.
+- **Ongoing:** Verify parallel agents finish within ~10% of each other. Review quarantined tests weekly. Monitor suite trends for regressions.
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Confusing the two tokens | `BUILDKITE_ANALYTICS_TOKEN` is for collectors (suite-scoped). `BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN` is for bktec (user-scoped, `read_suites`). |
+| Running bktec before enough data | Run collectors for 1-2 weeks first. Enable `DEBUG_ENABLED` to verify timing-based splitting. |
+| Missing env vars in Docker | Pass `BUILDKITE_PARALLEL_JOB`, `BUILDKITE_PARALLEL_JOB_COUNT`, and other vars via Docker plugin `environment` list. |
+| `parallelism` > test file count | Agents receive zero tests and waste compute. Set at most the number of test files. |
+| Omitting `RESULT_PATH` | bktec cannot write results back. Always set it. |
+| Wrong `TEST_RUNNER` value | Use exact values: `rspec`, `jest`, `playwright`, `cypress`, `pytest`, `pytest-pants`, `go`, `cucumber`. |
+| Not pinning `test-collector` version | Always pin: `test-collector#v2.0.0`, not `test-collector#v2`. |
+| `RETRY_COUNT` above 3 | Use `2`. Higher values waste compute and mask genuine failures. |
+
+## Additional Resources
+
+- **`references/collectors.md`** — Per-framework collector setup (Ruby, JavaScript, Python, Go, JUnit XML, test-collector plugin)
+- **`references/splitting-examples.md`** — Per-framework bktec examples, split-by-example, parallelism tuning
+- **`examples/collector-pipeline.yml`** — Minimal collector pipeline
+- **`examples/bktec-splitting.yml`** — bktec with parallelism, retry, and result path
+
+## Further Reading
+
+- [Test Engine overview](https://buildkite.com/docs/test-engine.md)
+- [Configuring bktec](https://buildkite.com/docs/test-engine/bktec/configuring.md)
+- [Test collection](https://buildkite.com/docs/test-engine/test-collection.md)
+- [Test states and quarantine](https://buildkite.com/docs/test-engine/test-suites/test-state-and-quarantine.md)
+- [bktec source (GitHub)](https://github.com/buildkite/test-engine-client)

package/skills/buildkite-test-engine/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "Buildkite Test Engine"
+  short_description: "Test splitting, flaky detection, quarantine, and bktec CLI"
+  icon_small: "./assets/buildkite-icon-small.png"
+  icon_large: "./assets/buildkite-icon-large.png"
+  brand_color: "#00D974"

package/skills/buildkite-test-engine/examples/bktec-splitting.yml

@@ -0,0 +1,16 @@
+# Pipeline using bktec for intelligent test splitting with parallelism,
+# automatic retry for flaky detection, and result path for data feedback
+steps:
+  - label: ":rspec: Tests %n"
+    command: bktec
+    parallelism: 10
+    retry:
+      automatic:
+        - exit_status: "*"
+          limit: 2
+    env:
+      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: "your-api-access-token"
+      BUILDKITE_TEST_ENGINE_SUITE_SLUG: "my-suite"
+      BUILDKITE_TEST_ENGINE_TEST_RUNNER: "rspec"
+      BUILDKITE_TEST_ENGINE_RESULT_PATH: "tmp/rspec-result.json"
+      BUILDKITE_TEST_ENGINE_RETRY_COUNT: "2"

package/skills/buildkite-test-engine/examples/collector-pipeline.yml

@@ -0,0 +1,11 @@
+# Minimal pipeline using the test-collector plugin with RSpec
+# Collects test timing data and sends it to Buildkite Test Engine
+steps:
+  - label: ":rspec: Tests"
+    command: "bundle exec rspec --format progress --format RspecJunitFormatter --out tmp/rspec.xml"
+    plugins:
+      - test-collector#v2.0.0:
+          files: "tmp/rspec.xml"
+          format: "junit"
+    env:
+      BUILDKITE_ANALYTICS_TOKEN: "your-suite-api-token"