@mmerterden/multi-agent-pipeline 15.14.0 → 15.15.0

package/CHANGELOG.md

@@ -16,6 +16,20 @@ Internal file-layout changes that don't affect the slash-command surface are sti
 
 ## [Unreleased]
 
+## [15.15.0] - 2026-08-22
+
+### Added
+- **Graylog has two instances now, because it always did.** Test and production are separate Graylog deployments, and a trx id minted by a tester does not exist in production - so searching production alone answered "no logs" for a complaint that was fully logged one host over, and that answer was indistinguishable from a genuine miss. `hosts.graylogTest` and the optional `keychainMapping.graylog_test` (which falls back to the production key, correct for shared-token deployments) make the second instance addressable. `fetch-graylog.sh --env auto` is the new default: production first, test when production returns nothing or is unreachable. `--env prod` / `--env test` pin one.
+- **The payload names the instance that answered** (`source.environment`, `source.searchedEnvironments[]`), and `/multi-agent:complaint-analysis` now has to cite it. A production complaint corroborated only by test logs is `insufficient-evidence`, not a confirmed `bff` fault, and the old payload gave the triage no way to tell those apart.
+- `/multi-agent:setup` asks for the two things the previous release added a consumer for but no collector: the Graylog test host plus its optional separate token, and `fortify.versionIds`. `versionIds` shipped in 15.14.0 as the only way to resolve an instance-id-only Fortify ticket, and nothing asked for it, so that path silently no-opped for everyone.
+- **`smoke-graylog-environments.sh`**, 15 assertions over the resolution rules that are easy to get subtly wrong: auto stops at production when production answers, falls back on empty AND on unreachable, degrades (never blocks) when both are down, treats a pinned environment with no host as exit 6 rather than silently searching the other one, and lets a 401 on one instance fall through instead of masking a working answer from the other.
+
+### Fixed
+- A pinned `--env` with no configured host now exits 6 naming the exact pref, instead of falling back to whichever host happened to be set. Attaching test logs to a production complaint is worse than attaching none.
+
+### Changed
+- `fetch-graylog.sh` resolves host and token per environment instead of once at the top, and reads prefs through one helper rather than three near-identical inline python blocks.
+
 ## [15.14.0] - 2026-08-22
 
 ### Added

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mmerterden/multi-agent-pipeline",
-  "version": "15.14.0",
+  "version": "15.15.0",
   "description": "8-phase AI development pipeline with full orchestration on Claude Code, Copilot CLI and Codex CLI. Analysis, planning, TDD, CLI-aware parallel review with consensus surfacing + Fable triage, default-FAIL evidence gates, secret + intent guards, per-phase cost ledger, persistent learnings memory, wiki generation, commit automation. Token-preserving uninstall.",
   "type": "module",
   "main": "index.js",

package/pipeline/commands/multi-agent/complaint-analysis/SKILL.md

@@ -81,12 +81,16 @@ For each complaint with at least one id and `skipGraylog: false`:
 ~/.claude/lib/fetch-graylog.sh --trx <trxId> --conv <convId>   # pass whichever exist
 ```
 
-Record per complaint: `graylog: {status: ok|degraded|skipped, totalResults, degradeReason}` plus the top messages (timestamp, source, level, message excerpt) kept in working context for Phase 2/3. Failure handling:
+Environment defaults to `auto`: production first, the test instance (`hosts.graylogTest`) when production returns nothing or is unreachable. A customer complaint is normally a production event, but a tester-minted id only exists on test, and prod-only search reports "no logs" for both cases identically. When the intake itself says which environment a complaint came from, pin it with `--env prod` / `--env test` rather than letting the fallback decide.
+
+Record per complaint: `graylog: {status: ok|degraded|skipped, environment, totalResults, degradeReason}` plus the top messages (timestamp, source, level, message excerpt) kept in working context for Phase 2/3. Failure handling:
 
 - Exit 0 with `degraded: true` -> `status: degraded`, keep the reason, continue.
 - Exit 2 / 3 / 6 (credential / auth / host) -> per `$HOME/.claude/multi-agent-refs/keychain.md` non-critical rule: warn ONCE (`WARN: Graylog unavailable (<reason>); remaining complaints proceed without log evidence.`), mark this and all remaining fetches `status: degraded`, continue. Never halt (Locked 1).
 - `skipGraylog: true` -> `status: skipped`.
 
+**Always name the environment when citing log evidence.** `source.environment` in the fetcher payload says which instance answered; the same lines mean different things depending on whether a production complaint was corroborated by production logs or only by test ones. A complaint whose only evidence came from test is `insufficient-evidence` for a production claim, not `bff`.
+
 Set `phase: "correlating_repos"`.
 
 ### Phase 2 - Repo evidence correlation (read-only)

package/pipeline/commands/multi-agent/setup/SKILL.md

@@ -107,6 +107,7 @@ These are the RECOMMENDED key names. When creating NEW keys, use these. But exis
 | `figma_mcp` | `${USER}_Figma_Mcp_Access_Token` | `figma` + `mcp` |
 | `fortify` | `${USER}_Fortify_Access_Token` | `fortify` |
 | `graylog` | `${USER}_Graylog_Access_Token` | `graylog` |
+| `graylog_test` | `${USER}_Graylog_Test_Access_Token` | `graylogTest` |
 | `firebase` | `${USER}_Firebase_Access_Json` | `firebase` (any variant: `sa`, `service`, `account`, `access`, `json`) |
 | `jenkins` | `${USER}_Jenkins_Access_Token` | `jenkins` |
 | `appstore_connect_key_id` | `${USER}_AppStoreConnect_Key_Id` | (`appstore` or `asc` or `app_store`) + (`key` or `keyid`) |
@@ -288,7 +289,7 @@ The Step 2 discovery summary is never the end of setup. Immediately after showin
 Loop rules:
 
 - **One prompt per service.** Never collapse the list into a single "add missing tokens now?" yes/no, and never offer one multi-select covering all services: each service has its own source URL, key name, identity binding and host prompt, so each gets its own Token Save Flow pass.
-- **Each service pass asks, in order: token → author → host.** Step A/B collect the secret, Step C binds the git identity (author)  -  Bitbucket and GitHub included; every saved token gets an author  -  and Step 3.5 asks the host domain inline for hosted services (jira, confluence, bitbucket, fortify, graylog). GitHub and Figma skip the host question (fixed domains); `bitbucket_user` skips author binding (username, not a PAT). Never end a service pass with the token saved but the author or a required host unasked.
+- **Each service pass asks, in order: token → author → host.** Step A/B collect the secret, Step C binds the git identity (author)  -  Bitbucket and GitHub included; every saved token gets an author  -  and Step 3.5 asks the host domain inline for hosted services (jira, confluence, bitbucket, fortify, graylog) - plus the Fortify project versions and the optional Graylog test instance. GitHub and Figma skip the host question (fixed domains); `bitbucket_user` skips author binding (username, not a PAT). Never end a service pass with the token saved but the author or a required host unasked.
 - **Skip is per-service.** A Skip answer leaves that service's `keychainMapping` at `null` and the loop continues with the next service. Skips never abort the loop.
 - **`figma` and `figma_mcp` are distinct credentials for distinct tiers  -  ask both.** Skipping the PAT does not imply skipping MCP, and vice versa. The `figma_mcp` pass opens with a Remote / Local mode question instead of a clipboard paste (see "Figma MCP Setup" below): Remote generates a `figu_` OAuth token for the current user, Local wires the PAT-based local server.
 - The loop ends only when every missing service has been asked exactly once; then continue to Step 4 (Verify all).
@@ -440,7 +441,9 @@ Service host  -  needed to build API URLs for this token.
   {Service} host (e.g. jira.example.com): ___
 ```
 
-Save to `prefs.global.hosts.{service}`. For Jira specifically, also ask for the default project key (the token is useless without one for new-issue creation):
+Save to `prefs.global.hosts.{service}`. For Fortify, also ask `Fortify project version ids (optional, comma-separated - e.g. 1234,5678)` → `prefs.global.fortify.versionIds`. A ticket naming only an instance id carries no version, so the lookup silently no-ops without these; URL-referenced findings resolve either way.
+
+For Jira specifically, also ask for the default project key (the token is useless without one for new-issue creation):
 
 ```
 Default Jira project key (e.g. PROJ): ___
@@ -476,6 +479,7 @@ Resulting shape:
       "bitbucket":  "bitbucket.example.com",
       "fortify":    "ssc.example.com",
       "graylog":    "graylog.example.com",
+      "graylogTest": "graylog-test.example.com",
       "corpDomain": "example.com"
     },
     "defaultJiraKey": "PROJ"
@@ -483,9 +487,18 @@ Resulting shape:
 }
 ```
 
-Placeholders `{JIRA_HOST}`, `{CONFLUENCE_HOST}`, `{BITBUCKET_HOST}`, `{FORTIFY_HOST}`, `{GRAYLOG_HOST}`, `{CORP_DOMAIN}`, `{JIRA_KEY}` in skills/commands are resolved from this block at runtime. If a host is ever missing when a phase needs it, Phase 0 prompts the same Host Prompt inline.
+Placeholders `{JIRA_HOST}`, `{CONFLUENCE_HOST}`, `{BITBUCKET_HOST}`, `{FORTIFY_HOST}`, `{GRAYLOG_HOST}`, `{GRAYLOG_TEST_HOST}`, `{CORP_DOMAIN}`, `{JIRA_KEY}` in skills/commands are resolved from this block at runtime. If a host is ever missing when a phase needs it, Phase 0 prompts the same Host Prompt inline.
+
+Graylog uses a **PAT** whose API auth is HTTP Basic with the token as username and the literal `token` as password  -  no separate username entry. It is optional by the null-mapping convention: `keychainMapping.graylog` or `hosts.graylog` left `null` degrades the log fetch to empty and never blocks a run.
+
+Graylog is asked **twice**  -  test and production are separate instances, and a tester-minted trx id does not exist in production. After the production host:
+
+```
+Graylog test host (optional  -  Enter to skip): ___
+Does the test instance use a different token? [y/N]
+```
 
-Graylog uses a **PAT (access token)** whose API auth is HTTP Basic with the token as the username and the literal `token` as the password (`<token>:token`)  -  no separate username entry. Its host prompt behaves exactly like the Jira host prompt (saved to `prefs.global.hosts.graylog`, resolved as `{GRAYLOG_HOST}`). Graylog is **automatically optional / non-blocking** via the null-mapping convention: leaving `keychainMapping.graylog` (and/or `hosts.graylog`) `null` simply means the log-fetch adapter degrades to empty and never blocks a run  -  no extra setup code.
+Test host → `hosts.graylogTest` (`{GRAYLOG_TEST_HOST}`); skipped means production-only search, which degrades nothing. Yes to the token question runs one more Token Save Flow pass for `graylog_test`; no (default) leaves it null and the fetcher reuses the production key. Run-time order is `--env auto`: production first, test on empty or unreachable, and the payload names which answered.
 
 **Re-run / update**: `/multi-agent:setup hosts` re-opens the prompt to edit values without touching tokens.
 
@@ -706,7 +719,8 @@ Standard key names (used when creating NEW tokens). Discovery may find existing
 | `figma` | `${USER}_Figma_Access_Token` | PAT (Tier 2, REST API) | Figma Developer Settings (max 90 days). Token Save Flow prompt copy: "Figma Personal Access Token (for Tier 2)". |
 | `figma_mcp` | `${USER}_Figma_Mcp_Access_Token` | OAuth (Tier 1, MCP) | Automatic via Claude Code Figma MCP remote auth. Token Save Flow prompt copy: "Figma MCP token (for Tier 1)". |
 | `fortify` | `${USER}_Fortify_Access_Token` | API Token | Fortify SSC -> Token Management (VPN required) |
-| `graylog` | `${USER}_Graylog_Access_Token` | PAT (API Basic `<token>:token`) | Graylog -> System -> Users and Teams -> Edit Tokens (VPN required). Auth is HTTP Basic with the token as username and literal `token` as password. |
+| `graylog` | `${USER}_Graylog_Access_Token` | PAT (API Basic `<token>:token`) | Graylog -> System -> Users and Teams -> Edit Tokens (VPN required) |
+| `graylog_test` | `${USER}_Graylog_Test_Access_Token` | same | Same page on the TEST instance; optional |
 | `firebase` | `${USER}_Firebase_Access_Json` | Firebase JSON (base64) | Firebase Console -> Project settings -> Service accounts -> Generate new private key. Project id is parsed from the decoded JSON  -  no separate entry. |
 | `jenkins` | `${USER}_Jenkins_Access_Token` | API Token | Jenkins -> User -> Configure -> API Token |
 |  -  | Git Identities | JSON (preferences) | Stored in `$HOME/.claude/multi-agent-preferences.json`  -  not Keychain |

package/pipeline/lib/fetch-graylog.sh

@@ -11,14 +11,29 @@
 #   ./fetch-graylog.sh --trx <transaction-id>
 #   ./fetch-graylog.sh --conv <conversation-id>
 #   ./fetch-graylog.sh --trx <id> --conv <id>
+#   ./fetch-graylog.sh --trx <id> --env test      # force one environment
+#
+# Environments. Test and production are separate Graylog instances, and a trx id
+# minted by a tester simply does not exist in production - searching only prod
+# returns "no logs" for a complaint that is fully logged one host over. The
+# default `--env auto` searches production first and falls back to test when
+# production returns nothing or is unreachable; the answer always names the
+# environment that produced it, so a result is never silently from elsewhere.
+# `--env prod` / `--env test` pin a single instance.
 #
 # Required configuration:
-#   prefs.global.hosts.graylog            -  Graylog host (e.g. logs.example.com)
-#   prefs.global.keychainMapping.graylog  -  keychain key holding the API token
+#   prefs.global.hosts.graylog                 -  production host (e.g. logs.example.com)
+#   prefs.global.keychainMapping.graylog       -  keychain key holding the prod API token
+# Optional (test environment):
+#   prefs.global.hosts.graylogTest             -  test host; without it, test is skipped
+#   prefs.global.keychainMapping.graylog_test  -  test token key; falls back to the prod
+#                                                 key when unset (shared-token deployments)
 #
 # Optional env:
 #   GRAYLOG_TIMEOUT_SECONDS    default 25
-#   GRAYLOG_HOST_OVERRIDE      forces the host (used by tests and CI dry-runs)
+#   GRAYLOG_ENV                auto|prod|test, same as --env (default auto)
+#   GRAYLOG_HOST_OVERRIDE      forces one host, reported as environment "override"
+#                              (used by tests and CI dry-runs)
 #   GRAYLOG_RANGE_SECONDS      relative search window, default 86400 (24h)
 #   GRAYLOG_LIMIT              max messages to return, default 200
 #
@@ -30,7 +45,9 @@
 # Output (stdout, single JSON object):
 #   {
 #     "fetchedAt": "<ISO8601>",
-#     "source": { "host": "<host>", "transactionId": "..."|null,
+#     "source": { "host": "<host>", "environment": "prod|test|override",
+#                 "searchedEnvironments": [ "prod", ... ],
+#                 "transactionId": "..."|null,
 #                 "conversationId": "..."|null, "query": "..." },
 #     "totalResults": <n>,
 #     "messages": [
@@ -45,12 +62,13 @@
 #   2  missing/expired credential (orchestrator handles the Save Flow)
 #   3  auth rejected on a reachable host (4xx)
 #   4  bad usage
-#   6  host not configured (prefs.global.hosts.graylog is empty)
+#   6  host not configured (the requested environment has no host in prefs)
 
 set -euo pipefail
 
 TRX_ID=""
 CONV_ID=""
+ENVIRONMENT="${GRAYLOG_ENV:-auto}"
 TIMEOUT="${GRAYLOG_TIMEOUT_SECONDS:-25}"
 RANGE="${GRAYLOG_RANGE_SECONDS:-86400}"
 LIMIT="${GRAYLOG_LIMIT:-200}"
@@ -59,32 +77,50 @@ while [ $# -gt 0 ]; do
   case "$1" in
     --trx)  TRX_ID="$2";  shift 2 ;;
     --conv) CONV_ID="$2"; shift 2 ;;
+    --env)  ENVIRONMENT="$2"; shift 2 ;;
     -h|--help)
-      echo "usage: $0 --trx <transaction-id> | --conv <conversation-id>" >&2
+      echo "usage: $0 --trx <transaction-id> | --conv <conversation-id> [--env auto|prod|test]" >&2
       exit 4 ;;
     *)
       echo "ERR: unexpected arg $1" >&2; exit 4 ;;
   esac
 done
 
-# Host comes from prefs (private config); skill templates only show {GRAYLOG_HOST}.
+case "$ENVIRONMENT" in
+  auto|prod|test) ;;
+  *) echo "ERR: --env must be auto, prod or test (got '$ENVIRONMENT')" >&2; exit 4 ;;
+esac
+
+# Hosts and token keys come from prefs (private config); skill templates only
+# show {GRAYLOG_HOST} / {GRAYLOG_TEST_HOST}.
 PREFS="$HOME/.claude/multi-agent-preferences.json"
-HOST="${GRAYLOG_HOST_OVERRIDE:-}"
-if [ -z "$HOST" ] && [ -f "$PREFS" ]; then
-  HOST=$(python3 -c "
-import json
+
+pref_str() {
+  # Usage: pref_str <dotted.path.under.global>  -> value or empty
+  [ -f "$PREFS" ] || { printf ''; return 0; }
+  PREF_PATH="$1" python3 -c "
+import json, os
 try:
-    p = json.load(open('$PREFS'))
-    print(p.get('global', {}).get('hosts', {}).get('graylog') or '')
+    node = json.load(open('$PREFS')).get('global', {})
+    for part in os.environ['PREF_PATH'].split('.'):
+        node = node.get(part, {}) if isinstance(node, dict) else {}
+    print(node if isinstance(node, str) else '')
 except Exception:
     print('')
-")
-fi
-if [ -z "$HOST" ]; then
-  printf '%s\n' '{"status":"blocked","reason":"host-not-configured","service":"graylog","expected_pref":"global.hosts.graylog"}' >&2
-  exit 6
-fi
-HOST=${HOST%/}  # strip trailing slash if any
+" 2>/dev/null || printf ''
+}
+
+HOST_PROD=$(pref_str hosts.graylog)
+HOST_TEST=$(pref_str hosts.graylogTest)
+HOST_PROD=${HOST_PROD%/}
+HOST_TEST=${HOST_TEST%/}
+
+KEY_PROD=$(pref_str keychainMapping.graylog)
+[ -z "$KEY_PROD" ] && KEY_PROD="${USER}_Graylog_Access_Token"
+# A test instance usually issues its own token, but plenty of deployments share
+# one; falling back to the prod key beats refusing to search test at all.
+KEY_TEST=$(pref_str keychainMapping.graylog_test)
+[ -z "$KEY_TEST" ] && KEY_TEST="$KEY_PROD"
 
 if [ -z "$TRX_ID" ] && [ -z "$CONV_ID" ]; then
   echo "ERR: no id (pass --trx <transaction-id> and/or --conv <conversation-id>)" >&2
@@ -104,20 +140,6 @@ if [ -n "$CONV_ID" ]; then
   fi
 fi
 
-# Token via keychain mapping.
-TOKEN_KEY=""
-if [ -f "$PREFS" ]; then
-  TOKEN_KEY=$(python3 -c "
-import json
-try:
-    p = json.load(open('$PREFS'))
-    print(p.get('global', {}).get('keychainMapping', {}).get('graylog') or '')
-except Exception:
-    print('')
-")
-fi
-[ -z "$TOKEN_KEY" ] && TOKEN_KEY="${USER}_Graylog_Access_Token"
-
 # Locate the resolver with an existence check, not a `.`-chain.
 #
 # Sourcing a file that does not exist aborts the shell under `set -e` - `||` included -
@@ -142,50 +164,125 @@ for _cred_resolver in \
 done
 unset _cred_resolver
 if [ -z "${CRED_STORE:-}" ]; then
-  printf '%s\n' '{"status":"blocked","reason":"missing-credential-helper","service":"graylog","expected_key":"'"$TOKEN_KEY"'"}' >&2
+  printf '%s\n' '{"status":"blocked","reason":"missing-credential-helper","service":"graylog","expected_key":"'"$KEY_PROD"'"}' >&2
   exit 2
 fi
 
-TOKEN=$("$CRED_STORE" get "$TOKEN_KEY" 2>/dev/null || true)
-if [ -z "$TOKEN" ]; then
-  printf '%s\n' '{"status":"blocked","reason":"missing-token","service":"graylog","expected_key":"'"$TOKEN_KEY"'"}' >&2
-  exit 2
-fi
+# Graylog relative universal search, built once and reused per environment.
+ENC_QUERY=$(QUERY_IN="$QUERY" python3 -c "
+import os, urllib.parse as up
+print(up.quote(os.environ['QUERY_IN']))
+")
+SEARCH_PATH="search/universal/relative?query=$ENC_QUERY&range=$RANGE&limit=$LIMIT&sort=timestamp:desc"
 
-API_BASE="https://$HOST/api"
+# Set by run_search: the outcome of the most recent environment probe.
+RUN_HTTP=""
+RUN_BODY=""
+RUN_HITS=0
 
-graylog_get() {
-  # Usage: graylog_get <path-and-query-relative-to-API_BASE>
-  # Prints "<body>\n<http_code>"; the caller splits the last line off.
-  local path="$1"
-  local url="$API_BASE/$path"
+run_search() {
+  # Usage: run_search <host> <token-key>
+  local host="$1" token_key="$2" token url resp
+  token=$("$CRED_STORE" get "$token_key" 2>/dev/null || true)
+  if [ -z "$token" ]; then
+    RUN_HTTP="no-token"; RUN_BODY=""; RUN_HITS=0
+    return 0
+  fi
+  url="https://$host/api/$SEARCH_PATH"
   # PAT basic auth: username=<token>, password=literal "token". The credential
   # goes through a curl config via process substitution so it never lands in
   # argv (argv is visible to ps).
-  curl -sS --max-time "$TIMEOUT" --connect-timeout 5 -w "\n%{http_code}" \
-    -K <(printf 'user = "%s:token"\n' "$TOKEN") \
+  resp=$(curl -sS --max-time "$TIMEOUT" --connect-timeout 5 -w "\n%{http_code}" \
+    -K <(printf 'user = "%s:token"\n' "$token") \
     -H "Accept: application/json" \
     -H "X-Requested-By: multi-agent-pipeline" \
-    "$url" 2>/dev/null || true
+    "$url" 2>/dev/null || true)
+  RUN_HTTP=$(printf '%s' "$resp" | tail -n1)
+  RUN_BODY=$(printf '%s' "$resp" | sed '$d')
+  RUN_HITS=0
+  if [ "$RUN_HTTP" = "200" ]; then
+    RUN_HITS=$(BODY_IN="$RUN_BODY" python3 -c "
+import json, os
+try:
+    print(len(json.loads(os.environ['BODY_IN']).get('messages') or []))
+except Exception:
+    print(0)
+" 2>/dev/null || echo 0)
+  fi
+  # An empty RUN_HITS would make the caller's numeric test abort under `set -e`.
+  case "$RUN_HITS" in (*[!0-9]*|"") RUN_HITS=0 ;; esac
 }
 
-# Graylog relative universal search.
-ENC_QUERY=$(QUERY_IN="$QUERY" python3 -c "
-import os, urllib.parse as up
-print(up.quote(os.environ['QUERY_IN']))
-")
-SEARCH_PATH="search/universal/relative?query=$ENC_QUERY&range=$RANGE&limit=$LIMIT&sort=timestamp:desc"
+# Which instances to try, in order. `auto` reaches for test only when prod had
+# nothing to say - either no hits or no answer at all.
+CANDIDATES=""
+if [ -n "${GRAYLOG_HOST_OVERRIDE:-}" ]; then
+  CANDIDATES="override"
+else
+  case "$ENVIRONMENT" in
+    prod) CANDIDATES="prod" ;;
+    test) CANDIDATES="test" ;;
+    auto) if [ -n "$HOST_TEST" ]; then CANDIDATES="prod test"; else CANDIDATES="prod"; fi ;;
+  esac
+fi
+
+host_for() {
+  case "$1" in
+    override) printf '%s' "${GRAYLOG_HOST_OVERRIDE%/}" ;;
+    test)     printf '%s' "$HOST_TEST" ;;
+    *)        printf '%s' "$HOST_PROD" ;;
+  esac
+}
+key_for() {
+  case "$1" in
+    test) printf '%s' "$KEY_TEST" ;;
+    *)    printf '%s' "$KEY_PROD" ;;
+  esac
+}
+
+# A pinned environment with no host is a configuration error, not a degrade:
+# silently searching the other instance would attach the wrong logs.
+for _env in $CANDIDATES; do
+  if [ -z "$(host_for "$_env")" ]; then
+    if [ "$_env" = "test" ]; then
+      printf '%s\n' '{"status":"blocked","reason":"host-not-configured","service":"graylog","environment":"test","expected_pref":"global.hosts.graylogTest"}' >&2
+    else
+      printf '%s\n' '{"status":"blocked","reason":"host-not-configured","service":"graylog","environment":"prod","expected_pref":"global.hosts.graylog"}' >&2
+    fi
+    exit 6
+  fi
+done
 
-RESP=$(graylog_get "$SEARCH_PATH")
-HTTP=$(printf '%s' "$RESP" | tail -n1)
-BODY=$(printf '%s' "$RESP" | sed '$d')
+HOST=""
+ENV_USED=""
+SEARCHED=""
+HTTP=""
+BODY=""
+LAST_CANDIDATE="${CANDIDATES##* }"
+for _env in $CANDIDATES; do
+  SEARCHED="${SEARCHED:+$SEARCHED }$_env"
+  run_search "$(host_for "$_env")" "$(key_for "$_env")"
+  HOST=$(host_for "$_env")
+  ENV_USED="$_env"
+  HTTP="$RUN_HTTP"
+  BODY="$RUN_BODY"
+  # An auth rejection on a reachable host is the only hard failure, and it is
+  # per-instance: a dead test token must not mask a working prod answer, so it
+  # aborts only when no candidate is left to try.
+  if [ "$HTTP" = "401" ] || [ "$HTTP" = "403" ]; then
+    if [ "$_env" = "$LAST_CANDIDATE" ]; then
+      echo "ERR: Graylog auth rejected on $_env (HTTP $HTTP)" >&2
+      exit 3
+    fi
+    continue
+  fi
+  if [ "$RUN_HITS" -gt 0 ]; then break; fi
+done
 
-# A genuine auth rejection on a reachable host is the only hard failure.
-case "$HTTP" in
-  401|403)
-    echo "ERR: Graylog auth rejected (HTTP $HTTP)" >&2
-    exit 3 ;;
-esac
+if [ "$HTTP" = "no-token" ]; then
+  printf '%s\n' '{"status":"blocked","reason":"missing-token","service":"graylog","expected_key":"'"$(key_for "$ENV_USED")"'"}' >&2
+  exit 2
+fi
 
 # Any other non-200 (empty code on connection failure, 5xx, timeout) degrades
 # to an empty result so a log fetch never blocks a run.
@@ -197,7 +294,8 @@ if [ "$HTTP" != "200" ]; then
   BODY=""
 fi
 
-SRC_HOST="$HOST" SRC_TRX="$TRX_ID" SRC_CONV="$CONV_ID" SRC_QUERY="$QUERY" \
+SRC_HOST="$HOST" SRC_ENV="$ENV_USED" SRC_SEARCHED="$SEARCHED" \
+SRC_TRX="$TRX_ID" SRC_CONV="$CONV_ID" SRC_QUERY="$QUERY" \
 DEGRADED_IN="$DEGRADED" DEGRADE_REASON_IN="$DEGRADE_REASON" \
 BODY_IN="$BODY" LIMIT_IN="$LIMIT" \
 python3 - <<'PY'
@@ -239,9 +337,11 @@ if not isinstance(total, int):
     total = len(messages)
 
 result = {
-    "fetchedAt": datetime.datetime.utcnow().isoformat() + "Z",
+    "fetchedAt": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"),
     "source": {
         "host": os.environ["SRC_HOST"],
+        "environment": os.environ.get("SRC_ENV") or "prod",
+        "searchedEnvironments": (os.environ.get("SRC_SEARCHED") or "").split(),
         "transactionId": trx,
         "conversationId": conv,
         "query": os.environ.get("SRC_QUERY") or "",

package/pipeline/multi-agent-refs/features/url-enrichment.md

@@ -74,10 +74,11 @@ Two entry shapes reach this step, and `metadata.source` says which:
 
 The extractor emits `graylog` entries from free-text trx / conversation ids (labels like `trx`, `trxId`, `transaction id`, `conversationId`, `convId`, `X-conversationId`), not from a URL  -  so these entries carry `"url": null`. When at least one is present, pull the matching diagnostic logs.
 
-- Host comes from `prefs.global.hosts.graylog` (`GRAYLOG_HOST_OVERRIDE` env forces it for tests/CI); no literal host lives in any source file. Missing host → the fetcher exits `6`; log a setup hint and skip.
-- Token resolution: `prefs.global.keychainMapping.graylog` → `~/.claude/lib/credential-store.sh get "<key>"` → Graylog PAT (fallback key `${USER}_Graylog_Access_Token`). Auth is HTTP Basic with the token as username and the literal `token` as password; the token is fed to `curl` only via a `-K` config file (process substitution), never on argv.
+- Hosts come from `prefs.global.hosts.graylog` (production) and the optional `prefs.global.hosts.graylogTest` (`GRAYLOG_HOST_OVERRIDE` forces a single host for tests/CI); no literal host lives in any source file. Missing host for the requested environment → the fetcher exits `6`; log a setup hint and skip.
+- **Environment**: the fetcher defaults to `--env auto`, which searches production and falls back to test when production returns nothing or is unreachable. A trx id minted by a tester does not exist in production, so prod-only search reports "no logs" for a complaint that is fully logged one host over. The result carries `source.environment` and `source.searchedEnvironments[]`; quote the environment whenever the logs are cited, because evidence from test says something different about a production complaint than the same lines from prod. Pin one instance with `--env prod` / `--env test` when the ticket already says which.
+- Token resolution: `prefs.global.keychainMapping.graylog` → `~/.claude/lib/credential-store.sh get "<key>"` → Graylog PAT (fallback key `${USER}_Graylog_Access_Token`). The test instance uses `keychainMapping.graylog_test`, falling back to the production key when unset. Auth is HTTP Basic with the token as username and the literal `token` as password; the token is fed to `curl` only via a `-K` config file (process substitution), never on argv.
 - Fetch logs (VPN required; respect `vpnServices.graylog`): call `~/.claude/lib/fetch-graylog.sh --trx <id>` (or `--conv <id>` for conversation ids) once per extracted id. The fetcher emits a normalized JSON object of matching log messages.
-- Store as `state.graylogContext = { idType, id, messages[], fetchedAt }` (or an array keyed per id when several were extracted).
+- Store as `state.graylogContext = { idType, id, environment, searchedEnvironments[], messages[], fetchedAt }` (or an array keyed per id when several were extracted).
 - Phase 1 Analysis prepends `state.graylogContext` to the agent prompt under the **Referenced External Sources** section as **diagnostic context, advisory only** (never contradicts code; no task is generated from it, no Phase 4 gate).
 - Soft-fail (non-blocking, mirrors Crashlytics): unreachable host / VPN down / network timeout → the fetcher emits a normalized empty object and exits `0`; mark `state.graylogContext = { skipped: "vpn-unreachable" }` and continue. Only a genuine auth rejection (4xx on a reachable host, fetcher exit `3`) is logged as `skipped` with the auth reason. A log fetch never blocks the run.
 

package/pipeline/multi-agent-refs/keychain.md

@@ -127,7 +127,8 @@ The shell driver auto-delegates to `~/.claude/scripts/keychain.py` on macOS / Li
 | `figma`            | Figma              | `${USER}_Figma_Access_Token`       | PAT               |
 | `figma_mcp`        | Figma MCP          | `${USER}_Figma_Mcp_Access_Token`   | OAuth             |
 | `fortify`          | Fortify            | `${USER}_Fortify_Access_Token`     | API Token         |
-| `graylog`          | Graylog            | `${USER}_Graylog_Access_Token`     | API Token         |
+| `graylog`          | Graylog (prod)     | `${USER}_Graylog_Access_Token`     | API Token         |
+| `graylog_test`     | Graylog (test)     | `${USER}_Graylog_Test_Access_Token` | API Token, optional  -  unset falls back to `graylog` |
 | `firebase`         | Firebase           | `${USER}_Firebase_Access_Json`     | JSON (base64)     |
 | `jenkins`          | Jenkins CI         | `${USER}_Jenkins_Access_Token`     | API Token         |
 |  -                   | Git Identity       | Stored in preferences JSON         | Not Keychain      |
@@ -147,6 +148,7 @@ The shell driver auto-delegates to `~/.claude/scripts/keychain.py` on macOS / Li
       "figma_mcp": "${USER}_Figma_Mcp_Access_Token",
       "fortify": "${USER}_Fortify_Access_Token",
       "graylog": "${USER}_Graylog_Access_Token",
+      "graylog_test": null,
       "firebase": "${USER}_Firebase_Access_Json",
       "jenkins": "${USER}_Jenkins_Access_Token"
     }

package/pipeline/preferences-template.json

@@ -12,6 +12,7 @@
       "figma_mcp": null,
       "fortify": null,
       "graylog": null,
+      "graylog_test": null,
       "usage_ingest": null,
       "firebase": null,
       "jenkins": null

package/pipeline/schemas/complaint-analysis-spec.schema.json

@@ -114,7 +114,12 @@
             "properties": {
               "status": { "type": "string", "enum": ["pending", "ok", "degraded", "skipped"] },
               "totalResults": { "type": "integer", "minimum": 0 },
-              "degradeReason": { "type": ["string", "null"] }
+              "degradeReason": { "type": ["string", "null"] },
+              "environment": {
+                "type": ["string", "null"],
+                "enum": ["prod", "test", "override", null],
+                "description": "v15.15+ - which Graylog instance answered. Test and production are separate instances; a production complaint corroborated only by test logs is insufficient-evidence, not a confirmed bff fault, so the verdict has to know where the lines came from."
+              }
             }
           },
           "repoEvidence": {

package/pipeline/schemas/prefs.schema.json

@@ -143,6 +143,10 @@
             "graylog": {
               "type": ["string", "null"]
             },
+            "graylog_test": {
+              "type": ["string", "null"],
+              "description": "v15.15+ - Keychain key for the TEST Graylog instance. Null or absent falls back to the `graylog` key, which is correct for deployments that share one token across both."
+            },
             "supabase_access": {
               "type": ["string", "null"],
               "description": "Supabase personal access token (Management API). Used by personal-site tooling; not consumed by pipeline phases."
@@ -574,7 +578,11 @@
             },
             "graylog": {
               "type": "string",
-              "description": "Graylog host without scheme, e.g. logs.example.com. Used by the Graylog log-fetch adapter for direct REST API calls. Resolves {GRAYLOG_HOST}."
+              "description": "Graylog PRODUCTION host without scheme, e.g. logs.example.com. Used by the Graylog log-fetch adapter for direct REST API calls. Resolves {GRAYLOG_HOST}."
+            },
+            "graylogTest": {
+              "type": "string",
+              "description": "v15.15+ - Graylog TEST host without scheme. Optional; leaving it unset means fetch-graylog.sh only ever searches production. Test and production are separate instances, so a trx id minted by a tester does not exist in prod and searching prod alone answers 'no logs' for a complaint that is fully logged one host over. Resolves {GRAYLOG_TEST_HOST}."
             },
             "corpDomain": {
               "type": "string",

package/pipeline/scripts/README.md

@@ -26,6 +26,7 @@ Validate contracts. Each emits `══ <name> smoke: N passed, M failed ══`
 - `smoke-handoff-contract.sh`  -  phase-boundary structured handoff + handoff-first resume (v10.8.0)
 - `smoke-update-check.sh`  -  Phase 0 Step 0.6 update-check + required-floor contract (v10.9.0, floor v15.14.0)
 - `smoke-context-links.sh`  -  context-link-extractor classification contract, all types (v15.14.0)
+- `smoke-graylog-environments.sh`  -  Graylog prod / test instance resolution + auto fallback (v15.15.0)
 
 ### Schema + state
 - `smoke-schema-validation.sh`  -  all JSON schemas validate