@htekdev/actions-debugger 1.0.117 → 1.0.119

package/errors/permissions-auth/permissions-auth-069.yml

@@ -0,0 +1,161 @@
+id: permissions-auth-069
+title: 'OIDC trust policy silently fails for repos missing required custom property claim — repository_property:* absent when property unset'
+category: permissions-auth
+severity: error
+tags:
+  - oidc
+  - custom-properties
+  - trust-policy
+  - aws
+  - azure
+  - gcp
+  - repository-property
+  - april-2026
+patterns:
+  - regex: 'Not authorized to perform sts:AssumeRoleWithWebIdentity'
+    flags: 'i'
+  - regex: 'AccessDenied.*AssumeRoleWithWebIdentity|WebIdentityErr.*AccessDenied'
+    flags: 'i'
+  - regex: 'Couldn''t retrieve OIDC token.*403|OIDC token.*invalid.*claim'
+    flags: 'i'
+  - regex: 'Error: Credentials could not be loaded.*OIDC'
+    flags: 'i'
+error_messages:
+  - 'Error: Not authorized to perform sts:AssumeRoleWithWebIdentity'
+  - 'AccessDenied: User: arn:aws:sts::... is not authorized to perform: sts:AssumeRoleWithWebIdentity'
+  - 'Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers'
+  - 'google.auth.exceptions.DefaultCredentialsError: OIDC token condition not satisfied'
+root_cause: |
+  GitHub Actions OIDC tokens now include `repository_property:{name}` claims for each
+  custom property set on the repository (generally available from April 2026). This
+  feature lets organizations create finer-grained cloud trust policies — for example,
+  only allowing OIDC authentication for repos whose `deploy_tier` custom property is
+  set to `production`.
+
+  However, the claim is **absent from the OIDC token when the property is not set on
+  the repository**. Cloud providers (AWS IAM, Azure AD, Google Cloud) evaluate a
+  missing claim as a condition failure:
+
+  - **AWS IAM** `Condition: { StringEquals: { "...repository_property:deploy_tier": "production" } }`
+    → `AccessDenied` if the repo has no `deploy_tier` property set
+  - **GCP** workload identity attribute conditions on `attribute.repository_property_*`
+    → condition evaluates false, token exchange rejected
+
+  Common scenarios that trigger this:
+  1. **Org-wide trust policy** uses a custom property claim, but individual repos have
+     not been tagged with the required property.
+  2. **Property renamed or deleted** — the trust policy still references the old
+     property name; the token no longer includes the old claim.
+  3. **Fork PRs** — forked repositories do not inherit the parent org's custom
+     properties; OIDC tokens from fork CI lack the expected claims.
+  4. **New repo** — a repository was added to the org after the trust policy was
+     configured; the property has not yet been applied to it.
+
+  The error message (`Not authorized to perform sts:AssumeRoleWithWebIdentity`) is
+  identical to other OIDC failures (wrong `sub`, wrong `aud`, expired token) and gives
+  no indication that a missing custom property claim is the cause.
+fix: |
+  1. **Verify the claim is present in the token**: Use the GitHub OIDC debugger or
+     print the decoded token payload in a workflow step to confirm the
+     `repository_property:{name}` claim exists and has the expected value.
+
+  2. **Ensure the custom property is set on all target repos**: In the org settings,
+     verify that every repository expected to use the trust policy has the required
+     property configured. Newly added repos will not have it by default.
+
+  3. **Make the condition optional (if the property may not always be set)**:
+     In AWS IAM, use `StringLike` with a wildcard or remove the custom-property
+     condition from the trust policy; use a separate, more permissive role for repos
+     without the property.
+
+  4. **For fork PRs**: custom properties on the upstream org do not flow to forks.
+     Avoid trust policies that require custom property claims in workflows triggered
+     by `pull_request` from external forks.
+fix_code:
+  - language: yaml
+    label: 'Debug step — print OIDC token claims to diagnose missing custom property'
+    code: |
+      jobs:
+        debug-oidc:
+          runs-on: ubuntu-latest
+          permissions:
+            id-token: write
+          steps:
+            - name: Fetch OIDC token and decode payload
+              run: |
+                TOKEN=$(curl -sH "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
+                  "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value')
+                # Decode payload (second segment of JWT, base64url-encoded)
+                echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+' \
+                  | base64 -d 2>/dev/null | jq .
+              # Look for "repository_property:your_property_name" in the output.
+              # If the claim is missing, the repo does not have that property set.
+
+  - language: yaml
+    label: 'AWS IAM trust policy — correct use of repository_property claim'
+    code: |
+      # AWS IAM role trust policy (JSON, not YAML — shown here for reference)
+      # Only allow OIDC from repos where custom property "deploy_tier" = "production"
+      {
+        "Version": "2012-10-17",
+        "Statement": [{
+          "Effect": "Allow",
+          "Principal": { "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/token.actions.githubusercontent.com" },
+          "Action": "sts:AssumeRoleWithWebIdentity",
+          "Condition": {
+            "StringEquals": {
+              "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
+              "token.actions.githubusercontent.com:repository_property:deploy_tier": "production"
+            }
+          }
+        }]
+      }
+      # IMPORTANT: Every repo that runs this workflow MUST have the "deploy_tier"
+      # custom property set to "production" in org settings. If the property is
+      # unset or absent, the token will not include the claim and the assume-role
+      # call will return AccessDenied.
+
+  - language: yaml
+    label: 'Workflow — ensure the custom property claim is available before assuming role'
+    code: |
+      jobs:
+        deploy:
+          runs-on: ubuntu-latest
+          permissions:
+            id-token: write
+            contents: read
+          steps:
+            - uses: actions/checkout@v4
+
+            # Verify the custom property claim is present before assuming the role
+            - name: Validate OIDC custom property claim
+              run: |
+                TOKEN=$(curl -sH "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
+                  "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value')
+                PAYLOAD=$(echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null)
+                TIER=$(echo "$PAYLOAD" | jq -r '."repository_property:deploy_tier" // "MISSING"')
+                echo "deploy_tier claim: $TIER"
+                if [[ "$TIER" != "production" ]]; then
+                  echo "::error::Repository custom property 'deploy_tier' is not set to 'production'. Set it in org settings before running this workflow."
+                  exit 1
+                fi
+
+            - name: Configure AWS credentials via OIDC
+              uses: aws-actions/configure-aws-credentials@v4
+              with:
+                role-to-assume: arn:aws:iam::123456789012:role/my-production-deploy-role
+                aws-region: us-east-1
+
+prevention:
+  - 'Maintain a registry of which repositories have each custom property set — before applying a trust policy that requires a custom property claim, verify all target repos have the property configured.'
+  - 'When adding a new repo to an org that uses OIDC custom property trust policies, immediately apply the required custom properties before running any workflows that assume cloud roles.'
+  - 'Do not use repository custom property claims in OIDC trust policies for workflows triggered by external fork pull requests — forks do not inherit the upstream org''s custom properties.'
+  - 'Add a preflight validation step to workflows that assume cloud roles — verify the expected repository_property:* claim is present in the OIDC token before calling the cloud provider.'
+  - 'If a custom property is renamed or removed, update all OIDC trust policies before the change takes effect to avoid sudden AccessDenied failures.'
+docs:
+  - url: 'https://github.blog/changelog/2026-04-02-github-actions-early-april-2026-updates/#actions-oidc-tokens-now-support-repository-custom-properties'
+    label: 'GitHub Changelog: OIDC tokens now support repository custom properties (April 2026)'
+  - url: 'https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect#customizing-the-token-claims'
+    label: 'GitHub Docs: Customizing OIDC token claims — repository custom properties'
+  - url: 'https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization'
+    label: 'GitHub Docs: Managing custom properties for repositories in your organization'

package/errors/runner-environment/arc-autoscalinglistener-ephemeralrunnerset-stale-after-upgrade.yml

@@ -0,0 +1,134 @@
+id: runner-environment-211
+title: 'ARC Controller Upgrade Leaves Stale AutoscalingListener and EphemeralRunnerSet — Manual Intervention Required'
+category: runner-environment
+severity: error
+tags:
+  - arc
+  - actions-runner-controller
+  - kubernetes
+  - upgrade
+  - autoscaling
+  - helm
+  - stale-controller
+patterns:
+  - regex: 'AutoscalingListener.*spec\.image.*old.*version|spec\.image.*ghcr\.io/actions.*stale'
+    flags: 'i'
+  - regex: 'app\.kubernetes\.io/version.*mismatch|helm\.sh/chart.*stale.*controller'
+    flags: 'i'
+  - regex: 'RunnerScaleSet.*stale.*image|EphemeralRunnerSet.*old.*version'
+    flags: 'i'
+error_messages:
+  - 'AutoscalingListener CRs retain stale controller image after controller upgrade'
+  - 'EphemeralRunnerSet retains stale version labels after controller upgrade'
+  - 'spec.image still points to old controller version after helm upgrade'
+root_cause: |
+  When upgrading the `gha-runner-scale-set-controller` Helm chart, two
+  controller-managed objects are NOT updated to reflect the new version:
+
+  1. **AutoscalingListener CRs** — retain the old controller image in `spec.image`
+  2. **EphemeralRunnerSet objects** — retain old version labels
+     (`app.kubernetes.io/version`, `helm.sh/chart`)
+
+  The root cause is that the controller gates reconciliation on a **spec hash**.
+  A controller-only upgrade does not change any `AutoscalingRunnerSet` spec, so
+  the hash is identical and the controller skips reconciliation of both objects
+  entirely. The `updateStrategy` flag only governs spec-change rollout, not
+  controller version upgrades.
+
+  **Object hierarchy affected:**
+  ```
+  AutoscalingRunnerSet   (Helm-managed)
+  ├── AutoscalingListener   ← stale image after controller upgrade
+  └── EphemeralRunnerSet    ← stale labels/spec after controller upgrade
+          └── EphemeralRunner
+  ```
+
+  For minor version bumps (e.g. 0.14.1 → 0.14.2) the staleness may appear
+  cosmetic. For **major upgrades** where the `EphemeralRunnerSet` or
+  `EphemeralRunner` spec has breaking changes (new required fields, removed
+  fields, changed defaults), stale objects under a new controller can cause
+  runtime failures — jobs queued but never dispatched, runner pods using the old
+  image's entrypoint, or scale-set reporting incorrect capacity.
+
+  **Version confirmed affected:** controller 0.14.2 / scale-set 0.14.2,
+  Kubernetes RKE2 (reproducible on any Kubernetes distribution).
+fix: |
+  Two separate manual steps are required after every controller upgrade where
+  AutoscalingListener or EphemeralRunnerSet spec has changed:
+
+  **Step 1 — Delete all AutoscalingListener CRs** (the controller recreates them
+  immediately with the new image; runner pods are unaffected):
+
+  ```bash
+  kubectl delete autoscalinglisteners -A --all
+  ```
+
+  Verify the new version is used after recreation:
+  ```bash
+  kubectl get autoscalinglisteners -A \
+    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.image}{"\n"}{end}'
+  ```
+
+  **Step 2 — Trigger EphemeralRunnerSet reconciliation** via a dummy annotation
+  change to `spec.template.spec` (a change to `minRunners` alone is NOT
+  sufficient — it only affects the AutoscalingListener hash, not the
+  EphemeralRunnerSet hash):
+
+  ```yaml
+  spec:
+    template:
+      metadata:
+        annotations:
+          upgrade-trigger: "0.14.2"   # bump on each controller upgrade
+  ```
+
+  This triggers a graceful transition: the old EphemeralRunnerSet drains
+  (in-progress jobs complete) while the new one starts accepting jobs
+  immediately. Remove the annotation in a follow-up commit.
+
+  **Verify EphemeralRunnerSet version labels are updated:**
+  ```bash
+  kubectl get ephemeralrunnersets -A \
+    -o custom-columns='NAME:.metadata.name,VERSION:.metadata.labels.app\.kubernetes\.io/version'
+  ```
+fix_code:
+  - language: yaml
+    label: 'Trigger EphemeralRunnerSet reconciliation via dummy annotation (per scale set)'
+    code: |
+      # In your AutoscalingRunnerSet HelmRelease or values.yaml:
+      # Add a dummy annotation to spec.template.spec to force EphemeralRunnerSet
+      # reconciliation after a controller upgrade.
+      # Remove the annotation in a follow-up commit once migration is confirmed.
+      spec:
+        template:
+          metadata:
+            annotations:
+              upgrade-trigger: "0.14.2"   # bump to new controller version
+  - language: yaml
+    label: 'Post-upgrade runbook as a one-off Job'
+    code: |
+      # After upgrading the controller chart, run this in CI or manually:
+      # Step 1: delete stale AutoscalingListener CRs (controller recreates immediately)
+      # kubectl delete autoscalinglisteners -A --all
+      #
+      # Step 2: patch each AutoscalingRunnerSet with a dummy annotation to force
+      # EphemeralRunnerSet reconciliation:
+      # kubectl annotate autoscalingrunnersets -A --all \
+      #   upgrade-trigger=$(date +%s) --overwrite
+      #
+      # Note: kubectl annotate updates metadata.annotations, not spec.template.spec,
+      # so it does NOT trigger EphemeralRunnerSet reconciliation. Use the values
+      # approach above (spec.template.metadata.annotations) instead.
+prevention:
+  - 'After every ARC controller upgrade, check AutoscalingListener images and EphemeralRunnerSet labels before routing production traffic.'
+  - 'Add a post-upgrade step to your CI/CD pipeline that deletes AutoscalingListener CRs and adds a dummy upgrade-trigger annotation.'
+  - 'Pin a dummy annotation like `upgrade-trigger: "<version>"` in your HelmRelease values and bump it with each controller upgrade.'
+  - 'Subscribe to actions/actions-runner-controller releases and review EphemeralRunnerSet spec changes before upgrading.'
+  - 'Track the upstream issue at actions/actions-runner-controller#4513 for a platform-side fix.'
+docs:
+  - url: 'https://github.com/actions/actions-runner-controller/issues/4513'
+    label: 'ARC #4513 — AutoscalingListener and EphemeralRunnerSet retain stale controller image/labels after upgrade (open Jun 2026)'
+  - url: 'https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md'
+    label: 'ARC Troubleshooting Guide'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller'
+    label: 'About Actions Runner Controller'

package/errors/runner-environment/broker-server-socket-exception-nat-timeout-linux.yml

@@ -0,0 +1,114 @@
+id: runner-environment-209
+title: 'Self-hosted runner BrokerServer TaskCanceledException / SocketException — runner stuck in Busy, jobs delayed'
+category: runner-environment
+severity: error
+tags:
+  - self-hosted
+  - broker
+  - TaskCanceledException
+  - SocketException
+  - NAT
+  - kubernetes
+  - ARC
+  - busy-state
+patterns:
+  - regex: 'BrokerServer.*TaskCanceledException'
+    flags: 'i'
+  - regex: 'SocketException \(125\): Operation canceled'
+    flags: 'i'
+  - regex: 'GET request to https://broker\.actions\.githubusercontent\.com.*has been cancelled'
+    flags: 'i'
+error_messages:
+  - '[ERR BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.'
+  - '[ERR BrokerServer] System.IO.IOException: Unable to read data from the transport connection: Operation canceled.'
+  - '[ERR BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled'
+  - '[WARN GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=...&status=Busy&runnerVersion=... has been cancelled.'
+  - '[WARN BrokerServer] Back off 6.934 seconds before next retry. 4 attempt left.'
+root_cause: |
+  The GitHub Actions runner (on Linux, macOS, Kubernetes, and ARC) maintains a
+  persistent long-poll HTTPS connection to `broker.actions.githubusercontent.com`
+  to receive job dispatch messages. This connection is kept open by a blocking
+  GET request that the server holds for up to 90 seconds before responding.
+
+  When the runner operates behind a **NAT gateway, stateful firewall, or cloud
+  provider network** (common in Kubernetes/ARC deployments on EKS, GKE, AKS, or
+  on-premise k8s), the network layer's connection tracking table can expire the
+  idle TLS socket before the server responds. Most cloud NAT tables have a
+  default idle timeout of 30–60 seconds — shorter than the runner's 90-second
+  poll interval.
+
+  When the NAT table entry expires:
+  1. The next packet the runner sends receives an RST from the network (or is
+     silently dropped), causing the underlying `SslStream.ReadAsyncInternal`
+     to throw `SocketException (125): Operation canceled`
+  2. The exception propagates as `TaskCanceledException` through the
+     `BrokerHttpClient.GetRunnerMessageAsync` call chain
+  3. The runner logs `ERR BrokerServer` and backs off exponentially (6–60 s)
+  4. During the back-off, the runner remains in **Busy** status from the broker's
+     perspective, preventing new jobs from being dispatched
+
+  The back-off recovers automatically but delays job pickup by minutes. Under
+  high-frequency job dispatch (CI matrix builds), this causes jobs to queue
+  while the runner is technically idle.
+
+  **Distinct from re-199** (Windows V2 broker listener stops polling after the
+  first job — Windows-specific software bug): this issue affects Linux/macOS/K8s
+  and recovers automatically; re-199 causes a permanent stall requiring service
+  restart.
+fix: |
+  **1. Enable TCP keepalive on the runner host (most effective):**
+
+  Configure the OS to send TCP keepalive probes before the NAT table expires:
+  ```bash
+  # Linux — reduce keepalive idle time from default 7200s to 30s
+  sudo sysctl -w net.ipv4.tcp_keepalive_time=30
+  sudo sysctl -w net.ipv4.tcp_keepalive_intvl=10
+  sudo sysctl -w net.ipv4.tcp_keepalive_probes=3
+  # Make persistent:
+  echo "net.ipv4.tcp_keepalive_time=30" | sudo tee -a /etc/sysctl.conf
+  ```
+
+  **2. Increase NAT idle timeout (infrastructure change):**
+
+  - **AWS EKS:** Set `--conntrack-tcp-timeout-established=300` on kube-proxy,
+    or add a NAT gateway connection tracking timeout of 350 s
+  - **GKE:** Use Cloud NAT with `--nat-tcp-established-idle-timeout=350`
+  - **Azure AKS:** Set `--load-balancer-idle-timeout-in-minutes=10` (default is 4 min)
+  - **On-premise k8s:** Increase `conntrack` timeout or set up a keepalive proxy
+
+  **3. Use a runner proxy with keepalive support:**
+
+  Route runner outbound traffic through an application-level proxy that
+  maintains the connection, preventing the NAT table from expiring the socket.
+
+  **4. Upgrade runner version:**
+
+  Runner v2.326.0+ includes improved broker reconnect logic that reduces the
+  window where the runner stays in Busy state after a socket reset.
+fix_code:
+  - language: yaml
+    label: 'Runner DaemonSet init container — set TCP keepalive before runner starts'
+    code: |
+      # In your ARC runner DaemonSet / Pod spec
+      initContainers:
+        - name: set-sysctl
+          image: busybox
+          securityContext:
+            privileged: true
+          command:
+            - sh
+            - -c
+            - |
+              sysctl -w net.ipv4.tcp_keepalive_time=30
+              sysctl -w net.ipv4.tcp_keepalive_intvl=10
+              sysctl -w net.ipv4.tcp_keepalive_probes=3
+prevention:
+  - 'Set tcp_keepalive_time to 30 s on all Linux self-hosted runner hosts, especially those in Kubernetes'
+  - 'For ARC scale sets in EKS/GKE/AKS, explicitly configure NAT idle timeout to at least 350 seconds'
+  - 'Monitor runner diagnostic logs (Runner_<date>-utc.log) for repeated BrokerServer ERR lines — they indicate this issue'
+  - 'Upgrade runner to v2.326.0+ which has improved back-off and reconnect behavior'
+docs:
+  - url: 'https://github.com/actions/runner/issues/3904'
+    label: 'actions/runner#3904 — Runner fails to connect to broker, TaskCanceledException / SocketException (17 reactions)'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github'
+    label: 'GitHub Docs — Self-hosted runner communication requirements'

package/errors/runner-environment/runner-environment-210.yml

@@ -0,0 +1,105 @@
+id: runner-environment-210
+title: 'Runner step-log and summary uploads silently stall behind egress-only firewall — .NET BlobClient ignores HTTPS_PROXY'
+category: runner-environment
+severity: silent-failure
+tags:
+  - proxy
+  - https-proxy
+  - blob-storage
+  - self-hosted
+  - egress-firewall
+  - logs-missing
+  - azure-blob
+patterns:
+  - regex: 'productionresultssa\d+\.blob\.core\.windows\.net'
+    flags: 'i'
+  - regex: 'ua=azsdk-net-Storage\.Blobs.*latency=[0-9]{2,3}\.'
+    flags: 'i'
+  - regex: 'step summary.*not.*available|summary.*upload.*failed|diagnostic.*log.*missing'
+    flags: 'i'
+error_messages:
+  - 'Step logs not visible in Actions UI — log upload stalled silently'
+  - 'Job completed but step summary is blank or missing'
+  - 'latency=74.999634s  ua=azsdk-net-Storage.Blobs/12.27.0 (.NET 8.0)'
+  - 'host=productionresultssa6.blob.core.windows.net:443  latency=74.99s'
+root_cause: |
+  The GitHub Actions runner process (v2.333.1 and earlier) creates Azure SDK
+  `BlobClient` instances for uploading step logs, workflow summaries, and diagnostic
+  logs without configuring an `HttpClientTransport` that honours the `HTTPS_PROXY`
+  environment variable.
+
+  In `ResultsHttpClient.cs`, the `GetBlobClient()` and `GetAppendBlobClient()` methods
+  pass only retry/timeout options to `BlobClientOptions` — no `Transport`. The Azure SDK
+  therefore falls back to its internal default HTTP pipeline, which does **not** inherit
+  the runner's `RunnerWebProxy` configuration.
+
+  The impact differs depending on network topology:
+
+  - **Without an egress firewall**: The runner's BlobClient connects directly to
+    `*.blob.core.windows.net` and `*.actions.githubusercontent.com`, bypassing the
+    proxy entirely. Direct egress works, so no error is visible.
+
+  - **With a deny-all egress firewall (proxy-only)**:  The BlobClient attempts a direct
+    connection that the firewall silently drops. Each attempt stalls for ~75 seconds
+    (the TCP connect timeout), then times out. Since log uploads are non-fatal, the job
+    eventually completes — but step logs are absent from the Actions UI and the job
+    takes 5–15 minutes longer than expected.
+
+  This is separate from the `upload-artifact@v6` proxy CONNECT-headers regression
+  (re-208), which affects the Node.js artifact upload path.
+fix: |
+  Add the Azure Blob Storage and Actions results endpoints to the `NO_PROXY` (or
+  `no_proxy`) environment variable so the runner bypasses the proxy for those hosts
+  and connects to them directly.
+
+  This requires that the runner's egress firewall allows direct connections to
+  `*.blob.core.windows.net` and `results-receiver.actions.githubusercontent.com`.
+  If only proxy egress is available, the workaround is to configure the proxy to
+  pass through those hosts without TLS inspection.
+
+  A proper fix (runner-side, not yet released): the `BlobClientOptions.Transport` in
+  `ResultsHttpClient.cs` should be configured with an `HttpClientTransport` wrapping
+  the runner's `RunnerWebProxy` — tracked in actions/runner#4351.
+fix_code:
+  - language: yaml
+    label: 'Self-hosted runner — set NO_PROXY to bypass proxy for Azure Blob endpoints'
+    code: |
+      # Set at the OS level or in the runner's .env file before starting the runner service.
+      # This allows the BlobClient to connect directly while other traffic goes through the proxy.
+      #
+      # On Linux/macOS (add to /etc/environment or runner startup script):
+      # NO_PROXY=.blob.core.windows.net,.actions.githubusercontent.com,results-receiver.actions.githubusercontent.com
+      #
+      # In a workflow (if runner is ephemeral and you can set env per job):
+      jobs:
+        build:
+          runs-on: self-hosted
+          env:
+            HTTPS_PROXY: http://proxy.corp.example.com:3128
+            NO_PROXY: '.blob.core.windows.net,.actions.githubusercontent.com,results-receiver.actions.githubusercontent.com'
+          steps:
+            - uses: actions/checkout@v4
+            - run: echo "Logs will upload correctly now"
+  - language: yaml
+    label: 'ARC / Kubernetes — set NO_PROXY in RunnerDeployment or RunnerScaleSet'
+    code: |
+      apiVersion: actions.summerwind.dev/v1alpha1
+      kind: RunnerDeployment
+      spec:
+        template:
+          spec:
+            env:
+              - name: HTTPS_PROXY
+                value: http://proxy.corp.example.com:3128
+              - name: NO_PROXY
+                value: '.blob.core.windows.net,.actions.githubusercontent.com,results-receiver.actions.githubusercontent.com'
+prevention:
+  - 'When deploying self-hosted runners behind a forward proxy with deny-all egress, always set NO_PROXY to include Azure Blob Storage endpoints — the runner BlobClient does not inherit HTTPS_PROXY.'
+  - 'Monitor Actions step log visibility alongside job exit codes — missing logs with a successful exit often indicate a proxy or network configuration issue, not a code failure.'
+  - 'Run a proxy connectivity diagnostic on a new runner host: confirm .blob.core.windows.net is reachable (directly or via proxy) before routing real workloads to it.'
+  - 'Track actions/runner#4351 for a first-party fix that configures the BlobClient transport to use the runner proxy settings.'
+docs:
+  - url: 'https://github.com/actions/runner/issues/4351'
+    label: 'actions/runner #4351 — BlobClient uploads stall through HTTPS proxy (Apr 2026)'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-a-proxy-server-with-self-hosted-runners'
+    label: 'GitHub Docs: Using a proxy server with self-hosted runners'

package/errors/runner-environment/runner-environment-213.yml

@@ -0,0 +1,142 @@
+id: runner-environment-213
+title: 'Self-Hosted Runner Stuck in Active State Indefinitely When Job Process Hangs Without Exiting'
+category: runner-environment
+severity: error
+tags:
+  - self-hosted
+  - runner
+  - active-state
+  - hung-process
+  - child-process
+  - stuck
+  - service-restart
+  - timeout-minutes
+patterns:
+  - regex: 'Waiting for a runner to pick up this job'
+    flags: 'i'
+  - regex: 'Runner\.Worker.*hung|Worker.*process.*running|active.*runner.*blocking'
+    flags: 'i'
+  - regex: 'sudo systemctl restart actions\.runner'
+    flags: 'i'
+error_messages:
+  - 'Waiting for a runner to pick up this job...'
+  - 'Runner shows as Active in GitHub UI; new jobs remain Queued indefinitely'
+root_cause: |
+  On self-hosted runners, the Runner.Worker process tracks the lifecycle of the running
+  job. When a job step spawns a child process (e.g., a test runner like `vitest --coverage`,
+  a long network operation, or a background daemon) that does not exit cleanly, the
+  Runner.Worker stays alive waiting for the child to terminate.
+
+  While Runner.Worker is alive, the parent Runner.Listener considers the runner slot
+  occupied (busy=true) and does not accept new job messages from the broker. The runner
+  appears as "Active" in the GitHub UI and all queued jobs for that runner remain
+  in "Waiting for a runner to pick up this job..." state indefinitely.
+
+  This differs from:
+  - GitHub-hosted runners: these have a hard 6-hour job timeout enforced by the platform;
+    the job is cancelled and the slot freed automatically
+  - Self-hosted runners WITH timeout-minutes set: once timeout-minutes elapses the
+    job is cancelled and the runner sends SIGTERM to the worker — but if the child
+    process ignores SIGTERM (common with some test runners), the worker still hangs
+
+  Common triggers:
+  - vitest --coverage, jest --forceExit not used, pytest hanging due to unclosed resources
+  - npm/yarn scripts that spawn background processes not tied to the shell session
+  - Docker commands (docker run without --rm) that keep running after the step exits
+  - Network calls blocked by firewall with no connection timeout
+  - Interactive prompts waiting for stdin input in a CI non-interactive context
+
+  Automatic recovery does NOT occur. The runner stays Active until either:
+  1. The hung child process eventually exits on its own
+  2. An operator manually restarts the runner service
+  3. A watchdog script kills the orphaned worker process
+fix: |
+  Immediate recovery: restart the runner service to free the stuck slot.
+    sudo systemctl restart actions.runner.<scope>.<name>.service   # Linux systemd
+    launchctl unload ~/Library/LaunchAgents/actions.runner.*.plist  # macOS
+    .\svc.sh stop && .\svc.sh start                                 # Windows
+
+  Prevention (preferred):
+  1. Add timeout-minutes to ALL jobs on self-hosted runners to cap maximum runtime.
+     Even if the worker hangs, the platform cancels the job and sends SIGTERM after
+     the timeout. Pair with process group kill to catch SIGTERM-resistant children.
+
+  2. Ensure test commands force-exit when done:
+     - vitest: add --forceExit flag
+     - jest: use jest --forceExit or --detectOpenHandles to identify hanging handles
+     - pytest: add timeout fixtures via pytest-timeout plugin
+
+  3. Use process groups (setsid / start new session) so SIGTERM cascades to children:
+     run: |
+       setsid bash -c 'npm test' &
+       CHILD_PID=$!
+       wait $CHILD_PID
+
+  4. Deploy a runner watchdog that monitors Worker processes with no active child
+     CPU activity for > N minutes and kills them:
+     - Check elapsed time + zero CPU descendants
+     - SIGKILL stale Worker processes
+     - Trigger runner service restart via systemd or equivalent
+fix_code:
+  - language: yaml
+    label: 'Add timeout-minutes to prevent indefinite runner lock'
+    code: |
+      jobs:
+        test:
+          runs-on: self-hosted
+          timeout-minutes: 30   # Always set on self-hosted runners
+          steps:
+            - uses: actions/checkout@v4
+            - name: Run tests
+              run: npm test
+  - language: yaml
+    label: 'Force-exit test runner so worker process completes cleanly'
+    code: |
+      jobs:
+        test:
+          runs-on: self-hosted
+          timeout-minutes: 30
+          steps:
+            - name: Run Vitest tests
+              run: npx vitest run --forceExit
+
+            - name: Run Jest tests
+              run: npx jest --forceExit
+
+            - name: Run pytest with timeout
+              run: pytest --timeout=300
+  - language: yaml
+    label: 'Watchdog step — kill orphaned background processes after main step'
+    code: |
+      jobs:
+        test:
+          runs-on: self-hosted
+          timeout-minutes: 30
+          steps:
+            - name: Run tests
+              run: npm test
+              continue-on-error: true
+            - name: Kill orphaned processes
+              if: always()
+              run: |
+                # Kill any remaining node processes owned by this runner user
+                pkill -u "$(whoami)" -f "vitest|jest|mocha" || true
+prevention:
+  - 'Always set timeout-minutes on self-hosted runner jobs — without it there is no
+    platform-enforced maximum and a hung process can block the runner indefinitely'
+  - 'Use --forceExit with Jest/Vitest; use --timeout with pytest; audit any test suite
+    that takes longer than expected for open handles (jest --detectOpenHandles)'
+  - 'Avoid spawning background daemons in run: steps without explicit cleanup in an
+    if: always() cleanup step'
+  - 'Consider running self-hosted runners as ephemeral (ephemeral: true with ARC or
+    JIT tokens) — an ephemeral runner terminates after one job, so a hung runner
+    does not affect other jobs (a new runner pod is provisioned for each job)'
+  - 'Monitor runner Active state duration via GitHub REST API (GET /repos/{owner}/{repo}/actions/runners)
+    and alert when busy: true persists beyond expected max job duration'
+docs:
+  - url: 'https://github.com/actions/runner/issues/4312'
+    label: 'actions/runner#4312 — Self-hosted runner gets stuck in active state, blocking queued jobs'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/monitoring-and-troubleshooting-self-hosted-runners'
+    label: 'GitHub Docs — Monitoring and troubleshooting self-hosted runners'
+  - url: 'https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes'
+    label: 'GitHub Docs — timeout-minutes syntax reference'