@htekdev/actions-debugger 1.0.35 → 1.0.36

package/errors/known-unsolved/checkout-eu-runner-timeout-regional-degradation.yml

@@ -0,0 +1,97 @@
+id: known-unsolved-033
+title: 'actions/checkout Hangs or Times Out From EU GitHub-Hosted Runners (Regional Degradation)'
+category: known-unsolved
+severity: error
+tags:
+  - checkout
+  - performance
+  - eu-runners
+  - timeout
+  - regional
+  - infrastructure
+patterns:
+  - regex: 'fatal: unable to access.*https://github\.com.*timed out'
+    flags: 'i'
+  - regex: 'fatal: unable to access.*https://github\.com.*Could not resolve host'
+    flags: 'i'
+  - regex: 'Error: Process completed with exit code 128'
+    flags: 'i'
+error_messages:
+  - "fatal: unable to access 'https://github.com/owner/repo/': Operation timed out"
+  - "Error: Process completed with exit code 128."
+  - 'actions/checkout step hanging for 5-30 minutes with no output'
+root_cause: |
+  Starting May 19, 2026, workflows running on GitHub-hosted runners in European
+  data centers began experiencing severely degraded actions/checkout performance.
+  The fetch/clone phase hangs silently for 5-30 minutes before either completing
+  slowly or timing out, regardless of repository size. Workflows that previously
+  completed checkout in 10-30 seconds are affected.
+
+  The root cause is an infrastructure-level degradation on GitHub's side
+  affecting the European runner subnet's connectivity to GitHub's Smart HTTP
+  server or CDN endpoints. This is distinct from general large-repo slowness:
+  even tiny repositories with shallow clones exhibit the hang. GitHub has not
+  published a root cause analysis or resolution timeline as of June 2026.
+
+  Notably, runners in US regions (us-east-1, us-west-2) are not affected —
+  the issue is specific to EU runner region routing. The error manifests as
+  either a silent hang (no log output during the fetch phase) or an eventual
+  "Operation timed out" exit code 128.
+
+  Source: actions/checkout#2441 (52 reactions, opened May 24, 2026, still open).
+fix: |
+  No upstream fix available — this is a GitHub infrastructure issue with no
+  workaround that completely eliminates the problem. Mitigations to reduce
+  impact:
+
+  1. Add timeout-minutes to checkout steps to prevent indefinite hangs and
+     fail fast with a clear error rather than a silent stuck pipeline.
+
+  2. Use fetch-depth: 1 (shallow clone) to reduce transfer size, which may
+     reduce hang duration even if it does not eliminate it.
+
+  3. Use sparse-checkout to limit the files transferred from the CDN.
+
+  4. For critical pipelines, consider temporarily switching to ubuntu-latest
+     with an explicit us-east-1 runner label if your GitHub plan supports
+     regional runner selection.
+
+  5. Subscribe to GitHub Status (githubstatus.com) for EU infrastructure
+     degradation notices — incidents affecting this region are tracked there.
+fix_code:
+  - language: yaml
+    label: 'Shallow clone with timeout to fail fast during EU degradation'
+    code: |
+      steps:
+        - name: Checkout
+          uses: actions/checkout@v4
+          timeout-minutes: 5      # fail fast instead of hanging for 30+ minutes
+          with:
+            fetch-depth: 1        # shallow clone reduces CDN transfer size
+  - language: yaml
+    label: 'Sparse checkout to minimize data fetched during regional degradation'
+    code: |
+      steps:
+        - name: Sparse checkout
+          uses: actions/checkout@v4
+          timeout-minutes: 5
+          with:
+            fetch-depth: 1
+            sparse-checkout: |
+              src/
+              tests/
+              package.json
+              go.mod
+prevention:
+  - 'Always specify fetch-depth: 1 for workflows that do not require full commit history'
+  - 'Add timeout-minutes to every checkout step to prevent indefinite pipeline hangs'
+  - 'Monitor p99 checkout duration from EU runners as a CI health SLI'
+  - 'Subscribe to GitHub Status page (githubstatus.com) for EU infrastructure degradation notices'
+  - 'Use sparse-checkout in large monorepos to reduce CDN dependency during fetch'
+docs:
+  - url: 'https://github.com/actions/checkout/issues/2441'
+    label: 'actions/checkout #2441: Checkouts extremely slow or timing out from EU (52 reactions, May 2026)'
+  - url: 'https://www.githubstatus.com/'
+    label: 'GitHub Status page for infrastructure incidents'
+  - url: 'https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/evaluate-expressions-in-workflows-and-actions'
+    label: 'GitHub Docs: sparse-checkout in actions/checkout'

package/errors/permissions-auth/checkout-v6-includif-symlinked-work-credential-failure.yml

@@ -0,0 +1,93 @@
+id: permissions-auth-032
+title: 'checkout@v6 Credential Injection Fails on Self-Hosted Runners With Symlinked _work Directory'
+category: permissions-auth
+severity: error
+tags:
+  - checkout-v6
+  - self-hosted
+  - symlink
+  - credentials
+  - includif
+  - macos
+patterns:
+  - regex: 'fatal: could not read Username for.*terminal prompts disabled'
+    flags: 'i'
+  - regex: 'includeIf.*gitdir.*_work'
+    flags: 'i'
+  - regex: 'fatal: repository.*not found'
+    flags: 'i'
+error_messages:
+  - "fatal: could not read Username for 'https://github.com': terminal prompts disabled"
+  - 'Error: fatal: repository not found'
+  - 'Authentication failed'
+root_cause: |
+  actions/checkout@v6 changed credential injection from writing directly into
+  the repository configuration file as http.https://github.com/.extraheader
+  (the v5 approach) to using includeIf "gitdir:..." directives that reference
+  a temporary credentials file stored in _work/_temp/.
+
+  v6 writes the includeIf path using the symlink path of the runner _work
+  directory. However, the version control system evaluates gitdir: conditions
+  against the resolved (real) absolute path — it follows symlinks when
+  determining the current repository's directory.
+
+  When the runner _work directory is a symlink to an external volume (a common
+  setup for macOS Apple Silicon runners using external SSD storage), the
+  includeIf path written by v6 uses the symlink path
+  (e.g., /Users/runner/actions-runner-N/_work/repo/.git) but the actual
+  resolved path is different
+  (e.g., /Volumes/External/actions-runner-N-work/repo/.git).
+  These never match, so the credentials config file is never loaded and the
+  fetch step fails with "terminal prompts disabled."
+
+  v5 is unaffected because it injects credentials directly into the repository
+  configuration file rather than using conditional includes.
+  Source: actions/checkout#2393 (open March 2026, macOS Apple Silicon).
+fix: |
+  Option 1 (recommended): Pin to actions/checkout@v5 for workflows running on
+  self-hosted runners with symlinked _work directories. v5 injects credentials
+  directly and is not affected by this symlink resolution issue.
+
+  Option 2: Reconfigure the runner to use the real volume path directly.
+  Remove the symlink from _work and mount the external volume at the actual
+  runner work path location. This eliminates the symlink entirely.
+
+  Option 3: Use persist-credentials: false with a separate authentication
+  step that does not rely on the includeIf mechanism.
+fix_code:
+  - language: yaml
+    label: 'Pin to v5 as workaround for symlinked _work runners (checkout#2393)'
+    code: |
+      steps:
+        - name: Checkout
+          # Pinned to v5 — v6 includeIf credential injection fails when runner
+          # _work directory is a symlink to an external volume (checkout#2393)
+          uses: actions/checkout@v5
+          with:
+            token: ${{ secrets.GITHUB_TOKEN }}
+  - language: yaml
+    label: 'Use persist-credentials false with explicit token for subsequent steps'
+    code: |
+      steps:
+        - name: Checkout without credential persistence
+          uses: actions/checkout@v6
+          with:
+            persist-credentials: false
+
+        - name: Subsequent steps using explicit token
+          env:
+            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          run: |
+            echo "Use GITHUB_TOKEN env var in subsequent authenticated operations"
+prevention:
+  - 'Audit self-hosted runner _work paths for symlinks before upgrading from checkout@v5 to v6'
+  - 'Avoid symlinking the runner _work directory — use bind mounts or configure the real path'
+  - 'Test checkout behavior on self-hosted runners in a canary workflow before rolling out v6'
+  - 'Check the resolved path differs from the symlink path when debugging "terminal prompts disabled" errors'
+docs:
+  - url: 'https://github.com/actions/checkout/issues/2393'
+    label: 'actions/checkout #2393: v6 includeIf credential matching fails on symlinked _work (open March 2026)'
+  - url: 'https://github.com/actions/checkout/issues/2313'
+    label: 'actions/checkout #2313: v6 breaks Docker actions using credential auth (related, closed)'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/configuring-the-self-hosted-runner-application-as-a-service'
+    label: 'GitHub Docs: Configuring the self-hosted runner as a service'

package/errors/runner-environment/ephemeral-runner-not-found-after-register.yml

@@ -0,0 +1,115 @@
+id: runner-environment-090
+title: 'Ephemeral Self-Hosted Runner Fails Immediately With "An error occurred: Runner not found"'
+category: runner-environment
+severity: error
+tags:
+  - self-hosted
+  - ephemeral
+  - runner-not-found
+  - registration
+  - broker
+  - jit
+patterns:
+  - regex: 'An error occurred: Runner not found'
+    flags: 'i'
+  - regex: 'RunnerNotFoundException'
+    flags: 'i'
+error_messages:
+  - 'An error occurred: Runner not found'
+  - 'GitHub.Actions.RunService.WebApi.RunnerNotFoundException'
+  - 'Listening for Jobs'
+root_cause: |
+  GitHub's broker endpoint returns RunnerNotFoundException immediately after
+  a successful registration and connection for ephemeral self-hosted runners
+  configured with replace mode. The runner completes registration ("Successfully
+  replaced the runner"), establishes connection ("Runner connection is good"),
+  starts listening for jobs, then receives a RunnerNotFoundException from the
+  broker HTTP client within seconds.
+
+  The error originates in BrokerHttpClient.cs where the broker API returns a
+  404/RunnerNotFoundException for the registered runner slot. This can occur
+  when the broker has stale slot state from the previous ephemeral runner
+  iteration that collides with the newly registered runner identity during the
+  brief window between registration and first poll.
+
+  The runner has no graceful retry handling for this condition — it exits with
+  status 1, causing systemd to restart it repeatedly, rapidly exhausting GitHub
+  App installation tokens through frequent re-registration cycles.
+
+  Affects all architectures (x86_64, aarch64, s390x) on various runner versions.
+  Spikes during periods of elevated load on GitHub broker infrastructure.
+  Source: actions/runner#3857 (116 reactions, open May 2025).
+fix: |
+  1. Switch from replace-mode ephemeral runners to JIT (Just-In-Time) runner
+     tokens. JIT runners receive a pre-assigned job ID and avoid the broker
+     slot replacement race entirely.
+  2. Update the runner to the latest version (v2.334.0+) which improves retry
+     behavior around transient broker errors.
+  3. Add restart delay in the systemd service unit to prevent token exhaustion
+     on rapid restart loops:
+       RestartSec=30
+       StartLimitIntervalSec=300
+       StartLimitBurst=5
+  4. Monitor runner diagnostic logs in _diag/Runner_*.log for the
+     RunnerNotFoundException pattern to distinguish broker errors from
+     configuration issues.
+fix_code:
+  - language: yaml
+    label: 'Systemd service unit with restart throttle to prevent token exhaustion'
+    code: |
+      # /etc/systemd/system/actions-runner.service
+      [Unit]
+      Description=GitHub Actions Self-Hosted Runner
+      After=network-online.target
+
+      [Service]
+      ExecStart=/home/runner/actions-runner/run.sh
+      Restart=on-failure
+      RestartSec=30
+      StartLimitIntervalSec=300
+      StartLimitBurst=5
+      User=runner
+
+      [Install]
+      WantedBy=multi-user.target
+  - language: yaml
+    label: 'Workflow using JIT runner token to avoid broker slot collision'
+    code: |
+      jobs:
+        provision-runner:
+          runs-on: ubuntu-latest
+          outputs:
+            runner-token: ${{ steps.jit.outputs.encoded_jit_config }}
+          steps:
+            - name: Generate JIT runner token
+              id: jit
+              uses: actions/github-script@v7
+              with:
+                script: |
+                  const { data } = await github.rest.actions.generateRunnerJitconfigForRepo({
+                    owner: context.repo.owner,
+                    repo: context.repo.repo,
+                    name: 'ephemeral-jit-runner',
+                    runner_group_id: 1,
+                    labels: ['self-hosted', 'ephemeral', 'linux']
+                  });
+                  core.setOutput('encoded_jit_config', data.encoded_jit_config);
+
+        build:
+          needs: provision-runner
+          runs-on: [self-hosted, ephemeral, linux]
+          steps:
+            - uses: actions/checkout@v4
+prevention:
+  - 'Use JIT runner tokens instead of replace-mode registration to eliminate broker slot race'
+  - 'Set systemd RestartSec to at least 30 seconds to avoid GitHub App token exhaustion'
+  - 'Monitor _diag/Runner_*.log for RunnerNotFoundException patterns and alert on restart frequency'
+  - 'Keep runner version current — broker compatibility fixes are regularly backported'
+  - 'Consider Kubernetes ARC ephemeral runners where pod lifecycle handles registration cleanly'
+docs:
+  - url: 'https://github.com/actions/runner/issues/3857'
+    label: 'actions/runner #3857: An error occurred: Runner not found (116 reactions, open May 2025)'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#using-just-in-time-runners'
+    label: 'GitHub Docs: Just-in-time runners (JIT)'
+  - url: 'https://docs.github.com/en/rest/actions/self-hosted-runners#create-configuration-for-a-just-in-time-runner-for-a-repository'
+    label: 'GitHub REST API: Generate JIT runner config'

package/errors/runner-environment/worker-wedged-task-orchestration-job-not-found.yml

@@ -0,0 +1,119 @@
+id: runner-environment-091
+title: 'Self-Hosted Runner Worker Wedges Indefinitely After TaskOrchestrationJobNotFoundException'
+category: runner-environment
+severity: error
+tags:
+  - self-hosted
+  - runner-worker
+  - wedged
+  - slot-starvation
+  - v2-runservice
+  - macos
+  - apple-silicon
+patterns:
+  - regex: 'TaskOrchestrationJobNotFoundException.*workflow instance not found'
+    flags: 'i'
+  - regex: 'Job not found:.*workflow instance not found'
+    flags: 'i'
+  - regex: 'CompleteJobAsync.*TaskOrchestrationJobNotFoundException'
+    flags: 'i'
+error_messages:
+  - 'GitHub.DistributedTask.WebApi.TaskOrchestrationJobNotFoundException: Job not found: <job-guid>. workflow instance not found'
+  - 'TaskOrchestrationJobNotFoundException: workflow instance not found'
+  - 'System.AggregateException: One or more errors occurred. (Job not found:'
+root_cause: |
+  When a self-hosted runner Worker process calls CompleteJobAsync in the V2
+  RunService path (useV2Flow: true, RunServiceHttpClient.CompleteJobAsync)
+  and the GitHub orchestrator has discarded the job record (e.g., due to a
+  server-side timeout, infrastructure failover, or job cancellation during
+  finalization), the Worker receives TaskOrchestrationJobNotFoundException
+  with "workflow instance not found."
+
+  After exhausting configured retry attempts (default maxAttempts), the Worker
+  logs the exception and stops processing — but critically, it fails to call
+  Environment.Exit and the process remains alive at ~0.1% CPU with no active
+  work, no child processes, and no job cleanup activity.
+
+  The parent Runner.Listener treats the still-running Worker process as a busy
+  runner slot and refuses to spawn a new Worker. This causes runner slot
+  starvation: the affected runner stops accepting new jobs until the wedged
+  Worker is externally terminated (kill, reboot, or watchdog).
+
+  On one 3-host Apple Silicon runner pool (v2.334.0), this affected 32.8% of
+  Worker invocations (50 of 152) over three weeks, with one incident wedging
+  all three Workers simultaneously and blocking CI for 3+ hours.
+  Source: actions/runner#4418 (open May 2026).
+fix: |
+  No upstream fix available — the Worker does not exit on non-retryable
+  CompleteJobAsync failures. Mitigations:
+
+  1. Deploy a watchdog that monitors _diag/Worker_*.log for the
+     TaskOrchestrationJobNotFoundException pattern and kills the wedged
+     Worker process by PID.
+
+  2. Use Kubernetes ARC ephemeral runners where the pod lifecycle replaces
+     the entire runner environment after each job — a wedged Worker is
+     automatically cleaned up when the pod is recycled.
+
+  3. Configure a hard systemd runtime limit (RuntimeMaxSec) that terminates
+     any runner process exceeding your longest expected job duration plus a
+     safety margin.
+
+  4. Add an external health-check cron that queries the GitHub API for runner
+     status and restarts the runner service if slots show "busy" longer than
+     expected.
+fix_code:
+  - language: yaml
+    label: 'Kubernetes ARC ephemeral runner configuration (avoids wedged Worker state)'
+    code: |
+      # ARC RunnerDeployment — pods are recycled after each job
+      apiVersion: actions.summerwind.dev/v1alpha1
+      kind: RunnerDeployment
+      metadata:
+        name: ephemeral-runner-deployment
+      spec:
+        replicas: 3
+        template:
+          spec:
+            ephemeral: true          # pod recycled after each job, no wedge possible
+            repository: owner/repo
+            labels:
+              - self-hosted
+              - ephemeral
+  - language: yaml
+    label: 'Scheduled watchdog workflow to detect stalled runner slots via API'
+    code: |
+      on:
+        schedule:
+          - cron: '*/15 * * * *'    # every 15 minutes
+
+      jobs:
+        runner-health-check:
+          runs-on: ubuntu-latest
+          steps:
+            - name: Detect stalled self-hosted runners
+              uses: actions/github-script@v7
+              with:
+                script: |
+                  const runners = await github.rest.actions.listSelfHostedRunnersForRepo({
+                    owner: context.repo.owner,
+                    repo: context.repo.repo
+                  });
+                  const offline = runners.data.runners.filter(r => r.status === 'offline');
+                  if (offline.length > 0) {
+                    core.warning('Offline/stalled runners: ' + offline.map(r => r.name).join(', '));
+                    // Trigger your runner restart webhook here
+                  }
+prevention:
+  - 'Use ephemeral Kubernetes ARC runners — pod recycle eliminates wedged Worker slot starvation'
+  - 'Monitor _diag/Worker_*.log for TaskOrchestrationJobNotFoundException patterns'
+  - 'Set systemd RuntimeMaxSec to maximum expected job duration plus 30 minutes'
+  - 'Track runner slot busy duration — sudden sustained busy state with no job output indicates wedge'
+  - 'Deploy a watchdog process alongside the runner that monitors Worker PID lifetime'
+docs:
+  - url: 'https://github.com/actions/runner/issues/4418'
+    label: 'actions/runner #4418: Worker wedges after TaskOrchestrationJobNotFoundException (open May 2026)'
+  - url: 'https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners'
+    label: 'GitHub Docs: Autoscaling with self-hosted runners'
+  - url: 'https://github.com/actions/actions-runner-controller'
+    label: 'actions/actions-runner-controller: Kubernetes ARC runner controller'

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@htekdev/actions-debugger",
-  "version": "1.0.35",
+  "version": "1.0.36",
   "description": "65+ real GitHub Actions errors, queryable by agents. CLI + MCP server + Copilot skills + error database.",
   "type": "module",
   "main": "./dist/index.js",