dv-pipecat-flows 0.0.23.dev12__tar.gz → 0.0.26.dev145__tar.gz

dv_pipecat_flows-0.0.26.dev145/.agents/skills/pipecat-logs/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: pipecat-logs
+description: Fetch and analyze pipecat application logs from VictoriaLogs by call ID or call SID. Use when debugging call issues, investigating errors, or analyzing bot behavior for a specific call. Accepts any call identifier — the Ringg call id, a provider-native id, a telephony provider UUID, or an Asterisk sid (the last two are resolved first) — and optional environment (staging/production).
+argument-hint: <call_id|call_sid> [staging|production]
+allowed-tools: Bash(curl *), Bash(jq *), Bash(kubectl *), Bash(pkill *)
+---
+
+# Pipecat Logs Fetcher
+
+Fetch logs from **VictoriaLogs** for a specific pipecat call and analyze them.
+
+> Pipecat logging moved from Loki to VictoriaLogs in 2026-08. The old Loki
+> instance (`35.154.72.110:3100`) is still up and still answers `200`, but it
+> receives only a residual trickle (~18 lines/hour of pod-shutdown noise). It
+> returns **an empty result set rather than an error** for every real query, so
+> a stale Loki query looks like "this call has no logs" instead of like a
+> broken endpoint. Do not query it.
+
+## Parameters
+
+- `search` — **any call identifier** (required): the Ringg call id (`01a03784-…`), a provider-native id (`3825374100`, `e42e1ec1…`), a telephony provider UUID, or an Asterisk sid. The last two need a resolve hop — see step 3.
+- `for` — **environment** (optional): `production` (default) or `staging`
+
+## Where pipecat logs actually live
+
+VictoriaLogs is **regional** — three independent instances, each storing only
+its own cluster's logs, each behind its own `vmauth` (max 4 concurrent reads).
+There is no cross-region query.
+
+| Instance | Cluster | Holds |
+|---|---|---|
+| **us-east1** | `desivocal-prod-us-e1-cluster` | **all pipecat** — `dv-pipecat` (`production`), `dv-pipecat-canary`, `dv-pipecat-stage` (`staging`) |
+| Mumbai | `desivocal-prod-cluster` | backend / taskiq / temporal only — **no pipecat** |
+| KSA | `desivocal-prod-cluster-ksa` | backend / taskiq only — **no pipecat** |
+
+**Always query us-east1 for pipecat, whatever the call's region.** Both
+production and staging pipecat live there, separated by the `environment`
+field. The Mumbai cluster does have a `dv-pipecat` Deployment, but it is a
+single idle replica (us-east1 runs ~900) — it serves no traffic, and its pods
+lack the `logs.ringg.ai/collect=application` label that Mumbai's Vector agent
+selects on, so nothing from it is collected anyway. An empty Mumbai result is
+therefore expected and means nothing.
+
+Non-pipecat lookups do need the other two: the backend's
+`new-calling-agent-taskiq-call-completion` logs live in Mumbai (India) or KSA.
+See "Cross-checking against the backend" below.
+
+## Instructions
+
+### 1. Parse arguments
+
+```
+IDENTIFIER = search
+ENV = for (default: "production")
+```
+
+Normalize `prod` to `production`. If `IDENTIFIER` is missing, ask the user.
+
+**Do not branch on what the identifier looks like.** A call has several ids and
+they are not interchangeable — but you cannot tell which kind you were handed
+by its shape, so always try the indexed field first and only resolve if it
+comes back empty (step 3). What exists in practice:
+
+| Identifier | Example | `call_id:=` works? |
+|---|---|---|
+| Ringg call id (UUIDv7, `01a0…`) | `01a03784-c580-7811-95ef-20ef317f8352` | **yes** |
+| provider-native numeric | `3825374100` | **yes** |
+| provider-native 32-hex | `e42e1ec169ea4ab5abba0f49480b84ba` | **yes** |
+| telephony provider's own UUID | `db6bd116-5197-4cad-8df6-d39bbbcc4565` | **no — 0 rows** |
+| Asterisk sid | `1787637558.655495` | **no — 0 rows** |
+
+The indexed `call_id` field carries the **Ringg** id — which the logs also call
+`callback_call_id`. The last two rows are the trap: a provider UUID looks
+exactly like a Ringg id and silently returns nothing, and the sid returns only
+a few callback lines. Both are recoverable — see step 3.
+
+### 2. Open a port-forward to the us-east1 VictoriaLogs
+
+The regional read endpoints (`use1.victorialogs.internal:8427` etc.) resolve
+only inside their own cluster, so query `svc/victoria-logs` through a
+port-forward. Start it in the background and poll until it answers — do not
+`sleep` in the foreground:
+
+```bash
+kubectl --context gke_desivocalprod01_us-east1_desivocal-prod-us-e1-cluster \
+  -n victorialogs port-forward svc/victoria-logs 9430:9428 > /tmp/vl-pf.log 2>&1 &
+
+curl -sf --retry 30 --retry-delay 1 --retry-connrefused -m 10 \
+  "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode 'query=_time:5m * | limit 1' -o /dev/null
+```
+
+The readiness check must be `curl`'s own `--retry-connrefused`, not a shell
+retry loop. The forward takes ~1-2s to bind, and until it does `curl` fails
+with connection-refused **instantly** — `-m` never applies, because there is
+nothing to wait on. A bare `for i in 1..10; do curl … && break; done` therefore
+burns all ten attempts in milliseconds and reports failure on a port-forward
+that is about to come up. Only `--retry-delay` actually waits.
+
+Tear it down with `pkill -f "port-forward svc/victoria-logs"` when finished.
+
+### 3. Query
+
+**Timezone note:** user-supplied times are **IST (Asia/Kolkata, UTC+5:30)**.
+LogsQL `_time` accepts relative ranges (`_time:1d`) or absolute UTC
+(`_time:[2026-08-25T04:00:00Z, 2026-08-25T05:00:00Z]`) — convert before querying.
+
+```bash
+curl -s -m 60 "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode "query=_time:1d application:dv-pipecat environment:=<ENV> call_id:=\"<IDENTIFIER>\" | limit 500" \
+  -o /tmp/vl.json
+```
+
+`call_id` is an indexed field, and matching it is **strictly better than a
+substring search** — not just faster. Most lines carry the field without
+printing the id in their text, so substring under-returns badly. Measured on
+three live calls:
+
+| id | `call_id:=` | substring |
+|---|---|---|
+| `01a03784-…` | 82 lines | 27 |
+| `3825374100` | 84 lines | 32 |
+| `e42e1ec1…` | 177 lines | 34 |
+
+**If it returns 0 rows, the identifier is a provider id, not the Ringg id.**
+Recover the real one and re-query — this is the difference between "this call
+barely logged anything" and the actual call:
+
+```bash
+REAL=$(curl -s -m 60 "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode "query=_time:1d application:dv-pipecat \"<IDENTIFIER>\" | fields call_id, _msg | limit 20" \
+  | { grep -oE '"call_id":"[^"u][^"]*"|callback_call_id[=: ]+[0-9a-f-]{8,}' || true; } \
+  | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | sort -u | head -1)
+```
+
+Then re-run the indexed query with `$REAL`. A bare substring filter is the last
+resort, and also the only way to see early-startup lines (`[run_bot] call_id=…`
+and friends), which are logged before the call context is bound and so carry no
+`call_id` field at all — they show as `call_id: "unknown"` or omit it.
+
+If the result is empty, in order: try the other `environment`; drop the
+`environment` filter; widen `_time` (max range is 720h); try
+`application:~"dv-pipecat"` to include `dv-pipecat-canary`.
+
+### 4. Parse and display
+
+The response is **JSON Lines** — one JSON object per line, not a wrapped
+envelope. `jq .` on the whole file fails; parse per line:
+
+```bash
+jq -r '[._time, (.level // "-"), ._msg] | @tsv' /tmp/vl.json | sort
+```
+
+Results come back unordered, so **sort by `_time` yourself**. Display
+chronologically (earliest first) with timestamp, level, and message.
+
+Useful fields: `_time`, `_msg`, `level`, `call_id`, `pod`, `application`,
+`environment`, `cluster`, `container`, `logger`, `module`, `function`, `line`.
+
+Two field names differ from the backend's records and are easy to get wrong:
+pipecat logs the pod as **`pod`**, not `pod_name`, and its **`level` is
+lowercase** (`debug`/`info`/`warning`/`error`) while the parallel `severity`
+field is uppercase. A filter on the wrong one silently matches nothing.
+
+### 5. Analyze
+
+After displaying the logs, provide a brief analysis:
+
+1. **Call timeline**: what happened (connected, greeted, conversation, ended)
+2. **Errors/warnings**: highlight ERROR/WARNING lines
+3. **Tool calls**: function/tool calls made during the call
+4. **TTS/STT activity**: speech-related events
+5. **Call outcome**: normal hangup, error, transfer, timeout
+6. **Anomalies**: long silences, repeated errors, unexpected transitions
+
+Note the session shape while reading — it changes which pipeline is running and
+therefore which failures are possible. `Using single_node bot` vs a FlowManager
+line; `channel=livekit|daily|telephony|smallwebrtc`; and `text mode, starting
+flow` / `Local recording disabled for text-only session` / `lk.chat message ->
+queued to LLM`, which mark a **text-only** session (no STT, no
+`TranscriptionFrame`).
+
+### 6. Grafana link
+
+Give the user a link to the pipecat dashboard (datasource `vl-use1`, i.e. the
+same us-east1 store queried above):
+
+```
+http://10.120.0.49/d/vl-pipecat-logs
+```
+
+Its variables are `environment`, `application`, `container`, `pod`, `call_id`,
+`trace_id`, `message_substring`, `minimum_level`. The host is an internal
+LoadBalancer IP — reachable on the VPN, not from the public internet.
+
+## Cross-checking against the backend
+
+When pipecat logs are thin, the backend's call-completion payload is a good
+second source: it dumps the whole completion object, including
+`function_calls_detailed` with each tool's `response_data` — so a failing
+tool's error string is greppable even without pipecat logs. It lives in the
+**Mumbai** (India) or **KSA** VL instance, not us-east1:
+
+```bash
+kubectl --context gke_desivocalprod01_asia-south1_desivocal-prod-cluster \
+  -n victorialogs port-forward svc/victoria-logs 9429:9428 > /tmp/vl-pf-mum.log 2>&1 &
+
+curl -s -m 60 "http://127.0.0.1:9429/select/logsql/query" \
+  --data-urlencode 'query=_time:1d application:new-calling-agent-taskiq-call-completion "<IDENTIFIER>" | limit 50'
+```
+
+KSA is `gke_desivocalprod01_me-central2_desivocal-prod-cluster-ksa`, same
+namespace and service.
+
+## Troubleshooting
+
+- **Empty result, `200` status** — the usual cause is the wrong region or a
+  missing `_time` window, not a missing call. Work through the fallbacks in
+  step 3 before concluding the call has no logs.
+- **`too big time range selected`** — `_time:` is mandatory and capped at 720h.
+  A query with no `_time` filter is rejected with a `1677-09-21 → 2262-04-11`
+  range in the message; that is the missing-filter error, not a real range.
+- **`jq: parse error`** — the response is JSON Lines; see step 4.
+- **Port-forward drops** — `kubectl port-forward` dies on idle or on pod
+  rotation. Re-run step 2; the retry makes that safe to repeat.
+- **Zero rows, or only a handful** — you were almost certainly handed a
+  provider id rather than the Ringg one. Resolve it (step 3) instead of
+  widening the time range.
+- **Slow or refused reads** — each region's `vmauth` allows 4 concurrent reads.
+  Serialize queries rather than fanning out.
+- **A migrated webcall with no pipecat logs at all** — webcalls with
+  `telephony_provider=livekit` run on LiveKit Agents, not pipecat, and log to
+  GCP Cloud Logging (`desivocalprod01`, container `ringg-webcall`).
+
+## Reference
+
+For LogsQL syntax and more filtering options, see [query-reference.md](query-reference.md).

dv_pipecat_flows-0.0.26.dev145/.agents/skills/pipecat-logs/query-reference.md

@@ -0,0 +1,165 @@
+# LogsQL Query Reference for Pipecat Logs
+
+VictoriaLogs uses **LogsQL**, not LogQL. The two look similar and are not
+compatible: there are no `{label="value"}` stream selectors, no `|=` line
+filters, and no `| json` / `| line_format` stages. Fields are filtered
+directly, and the pipeline after `|` uses named pipes (`limit`, `stats`,
+`sort`, `fields`, `field_names`).
+
+Full syntax: <https://docs.victoriametrics.com/victorialogs/logsql/>
+
+## Base query pattern
+
+```logsql
+_time:1d application:dv-pipecat environment:=production call_id:="<call_id>" | limit 500
+```
+
+`_time:` is **mandatory** on every query and capped at 720h. Omitting it is
+rejected with a `[1677-09-21 … 2262-04-11]` range in the error — that is the
+missing-filter message, not a real range you asked for.
+
+## Filter forms
+
+| Form | Meaning |
+|---|---|
+| `field:="exact"` | exact field match (fastest — use for `call_id`) |
+| `field:"phrase"` | phrase match within the field |
+| `field:~"regex"` | regexp match, e.g. `application:~"dv-pipecat"` |
+| `"substring"` | substring across the whole record — the `\|=` equivalent |
+| `field:*` | field exists |
+| `-field:="x"` | negation |
+| `a AND b`, `a OR b` | boolean (bare space is AND) |
+
+## Time ranges
+
+```logsql
+_time:5m          _time:1h          _time:1d          _time:7d
+_time:[2026-08-25T04:00:00Z, 2026-08-25T05:00:00Z]
+```
+
+Absolute times are **UTC**. User-supplied times are IST (UTC+5:30) — subtract
+5h30m first.
+
+## Common variations
+
+**Errors only** — `level` is lowercase on pipecat records; the uppercase
+values live on the parallel `severity` field, so match case-insensitively or
+pick one field deliberately:
+```logsql
+_time:1d application:dv-pipecat call_id:="<call_id>" level:~"(?i)error|critical" | limit 200
+```
+
+**Warnings and above:**
+```logsql
+_time:1d application:dv-pipecat call_id:="<call_id>" severity:~"WARNING|ERROR|CRITICAL" | limit 200
+```
+
+**Keyword within a call** (LLM / TTS / tool calls / VAD):
+```logsql
+_time:1d application:dv-pipecat call_id:="<call_id>" "LLM" | limit 200
+_time:1d application:dv-pipecat call_id:="<call_id>" _msg:~"function|tool_call|generic_function" | limit 200
+_time:1d application:dv-pipecat call_id:="<call_id>" _msg:~"VAD|speaking|speech|silence|idle" | limit 200
+```
+
+**One error string across all calls** — how you find out whether a bug is
+systemic and how often it fires:
+```logsql
+_time:1d application:dv-pipecat "TaskManager is still not initialized" | limit 200
+```
+
+**By pod** — the field is `pod`; `pod_name` exists on the backend's records
+but **not** on pipecat's, and filtering on it silently matches nothing:
+```logsql
+_time:1h application:dv-pipecat pod:~"dv-pipecat-canary" | limit 200
+```
+
+**Include canary:**
+```logsql
+_time:1d application:~"dv-pipecat" call_id:="<call_id>" | limit 500
+```
+
+## Aggregation and discovery pipes
+
+```logsql
+_time:30m * | stats by (application, cluster, environment) count()   # what's in this instance
+_time:1h application:dv-pipecat | stats by (level) count()           # error-rate breakdown
+_time:1h application:dv-pipecat | limit 500 | field_names            # available fields
+_time:1d application:dv-pipecat "<error>" | fields _time, call_id, pod | limit 200
+```
+
+`stats by (...)` is the cheap way to answer "which region / app / pod is this
+happening in" before pulling raw lines.
+
+## API endpoints
+
+| Endpoint | Purpose |
+|---|---|
+| `POST /select/logsql/query` | Query logs (use `--data-urlencode 'query=…'`) |
+| `POST /select/logsql/hits` | Hit counts bucketed over time |
+| `GET /select/logsql/field_names` | Field names for a query |
+| `GET /select/logsql/field_values` | Values for one field |
+
+Parameters: `query` (LogsQL, includes its own time filter), and optionally
+`start` / `end` / `limit`. There is no `direction` — order with a `sort` pipe
+or sort client-side.
+
+## Response structure
+
+**JSON Lines** — one JSON object per line, no envelope. `jq .` over the whole
+body fails; parse per line:
+
+```
+{"_time":"2026-08-25T04:09:20.448Z","_msg":"Call config: …","level":"info","call_id":"01a0371b-…","pod":"dv-pipecat-…"}
+{"_time":"2026-08-25T04:09:20.455Z","_msg":"BYOK status: …","level":"info","call_id":"01a0371b-…","pod":"dv-pipecat-…"}
+```
+
+```bash
+jq -r '[._time, (.level // "-"), ._msg] | @tsv' /tmp/vl.json | sort
+```
+
+Results are unordered — sort by `_time` yourself.
+
+## Fields on pipecat records
+
+`_time`, `_msg`, `level`, `severity`, `call_id`, `pod`, `node`, `application`,
+`container`, `namespace`, `cluster`, `environment`, `logger`, `module`,
+`function`, `line`, `process`, `thread`, `stream`, `_extras_dropped`,
+`_truncated`.
+
+`level` is lowercase, `severity` is uppercase, and the pod is `pod` — **not**
+`pod_name`. The backend's records carry a different and larger set
+(`pod_name`, `trace_id`, `agent_id`, `workspace_id`, `region`, `host_ip`,
+`correlation_id`, …), so don't carry a working backend filter over to a pipecat
+query unchanged. `| limit N | field_names` on a live window is the way to check.
+
+Vector sets `cluster` / `environment` / the `kubernetes` metadata itself and
+strips any same-named keys from the application payload, so those cannot be
+spoofed by a log line — they are reliable for filtering.
+
+## Infrastructure
+
+Three regional instances, no cross-region query, each behind its own `vmauth`
+(max 4 concurrent reads). All are `svc/victoria-logs:9428` in namespace
+`victorialogs`; the `*.victorialogs.internal:8427` read endpoints resolve only
+inside their own cluster, so port-forward from a laptop.
+
+| Region | kubectl context | Holds |
+|---|---|---|
+| us-east1 | `gke_desivocalprod01_us-east1_desivocal-prod-us-e1-cluster` | **all pipecat**, prod + staging |
+| Mumbai | `gke_desivocalprod01_asia-south1_desivocal-prod-cluster` | backend / taskiq / temporal |
+| KSA | `gke_desivocalprod01_me-central2_desivocal-prod-cluster-ksa` | backend / taskiq |
+
+- **Grafana**: `http://10.120.0.49` (internal LB — VPN only), pipecat dashboard
+  uid `vl-pipecat-logs`, datasources `vl-use1` (default) / `vl-mumbai` / `vl-ksa`.
+- **Collection**: a Vector DaemonSet per cluster ships to the regional `vmauth`.
+  Mumbai, KSA and staging collect only Pods labelled
+  `logs.ringg.ai/collect=application`; us-east1 collects everything except Pods
+  labelled `vector.dev/exclude`. So in Mumbai/KSA, "no logs for this workload"
+  can mean "not labelled for collection", not "not running".
+
+## Environment values
+
+| Environment | `environment` value | Pipecat application |
+|---|---|---|
+| Production | `production` | `dv-pipecat`, `dv-pipecat-canary` |
+| Staging | `staging` | `dv-pipecat-stage` |

dv_pipecat_flows-0.0.26.dev145/.claude/skills/pipecat-logs/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: pipecat-logs
+description: Fetch and analyze pipecat application logs from VictoriaLogs by call ID or call SID. Use when debugging call issues, investigating errors, or analyzing bot behavior for a specific call. Accepts any call identifier — the Ringg call id, a provider-native id, a telephony provider UUID, or an Asterisk sid (the last two are resolved first) — and optional environment (staging/production).
+argument-hint: <call_id|call_sid> [staging|production]
+allowed-tools: Bash(curl *), Bash(jq *), Bash(kubectl *), Bash(pkill *)
+---
+
+# Pipecat Logs Fetcher
+
+Fetch logs from **VictoriaLogs** for a specific pipecat call and analyze them.
+
+> Pipecat logging moved from Loki to VictoriaLogs in 2026-08. The old Loki
+> instance (`35.154.72.110:3100`) is still up and still answers `200`, but it
+> receives only a residual trickle (~18 lines/hour of pod-shutdown noise). It
+> returns **an empty result set rather than an error** for every real query, so
+> a stale Loki query looks like "this call has no logs" instead of like a
+> broken endpoint. Do not query it.
+
+## Parameters
+
+- `search` — **any call identifier** (required): the Ringg call id (`01a03784-…`), a provider-native id (`3825374100`, `e42e1ec1…`), a telephony provider UUID, or an Asterisk sid. The last two need a resolve hop — see step 3.
+- `for` — **environment** (optional): `production` (default) or `staging`
+
+## Where pipecat logs actually live
+
+VictoriaLogs is **regional** — three independent instances, each storing only
+its own cluster's logs, each behind its own `vmauth` (max 4 concurrent reads).
+There is no cross-region query.
+
+| Instance | Cluster | Holds |
+|---|---|---|
+| **us-east1** | `desivocal-prod-us-e1-cluster` | **all pipecat** — `dv-pipecat` (`production`), `dv-pipecat-canary`, `dv-pipecat-stage` (`staging`) |
+| Mumbai | `desivocal-prod-cluster` | backend / taskiq / temporal only — **no pipecat** |
+| KSA | `desivocal-prod-cluster-ksa` | backend / taskiq only — **no pipecat** |
+
+**Always query us-east1 for pipecat, whatever the call's region.** Both
+production and staging pipecat live there, separated by the `environment`
+field. The Mumbai cluster does have a `dv-pipecat` Deployment, but it is a
+single idle replica (us-east1 runs ~900) — it serves no traffic, and its pods
+lack the `logs.ringg.ai/collect=application` label that Mumbai's Vector agent
+selects on, so nothing from it is collected anyway. An empty Mumbai result is
+therefore expected and means nothing.
+
+Non-pipecat lookups do need the other two: the backend's
+`new-calling-agent-taskiq-call-completion` logs live in Mumbai (India) or KSA.
+See "Cross-checking against the backend" below.
+
+## Instructions
+
+### 1. Parse arguments
+
+```
+IDENTIFIER = search
+ENV = for (default: "production")
+```
+
+Normalize `prod` to `production`. If `IDENTIFIER` is missing, ask the user.
+
+**Do not branch on what the identifier looks like.** A call has several ids and
+they are not interchangeable — but you cannot tell which kind you were handed
+by its shape, so always try the indexed field first and only resolve if it
+comes back empty (step 3). What exists in practice:
+
+| Identifier | Example | `call_id:=` works? |
+|---|---|---|
+| Ringg call id (UUIDv7, `01a0…`) | `01a03784-c580-7811-95ef-20ef317f8352` | **yes** |
+| provider-native numeric | `3825374100` | **yes** |
+| provider-native 32-hex | `e42e1ec169ea4ab5abba0f49480b84ba` | **yes** |
+| telephony provider's own UUID | `db6bd116-5197-4cad-8df6-d39bbbcc4565` | **no — 0 rows** |
+| Asterisk sid | `1787637558.655495` | **no — 0 rows** |
+
+The indexed `call_id` field carries the **Ringg** id — which the logs also call
+`callback_call_id`. The last two rows are the trap: a provider UUID looks
+exactly like a Ringg id and silently returns nothing, and the sid returns only
+a few callback lines. Both are recoverable — see step 3.
+
+### 2. Open a port-forward to the us-east1 VictoriaLogs
+
+The regional read endpoints (`use1.victorialogs.internal:8427` etc.) resolve
+only inside their own cluster, so query `svc/victoria-logs` through a
+port-forward. Start it in the background and poll until it answers — do not
+`sleep` in the foreground:
+
+```bash
+kubectl --context gke_desivocalprod01_us-east1_desivocal-prod-us-e1-cluster \
+  -n victorialogs port-forward svc/victoria-logs 9430:9428 > /tmp/vl-pf.log 2>&1 &
+
+curl -sf --retry 30 --retry-delay 1 --retry-connrefused -m 10 \
+  "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode 'query=_time:5m * | limit 1' -o /dev/null
+```
+
+The readiness check must be `curl`'s own `--retry-connrefused`, not a shell
+retry loop. The forward takes ~1-2s to bind, and until it does `curl` fails
+with connection-refused **instantly** — `-m` never applies, because there is
+nothing to wait on. A bare `for i in 1..10; do curl … && break; done` therefore
+burns all ten attempts in milliseconds and reports failure on a port-forward
+that is about to come up. Only `--retry-delay` actually waits.
+
+Tear it down with `pkill -f "port-forward svc/victoria-logs"` when finished.
+
+### 3. Query
+
+**Timezone note:** user-supplied times are **IST (Asia/Kolkata, UTC+5:30)**.
+LogsQL `_time` accepts relative ranges (`_time:1d`) or absolute UTC
+(`_time:[2026-08-25T04:00:00Z, 2026-08-25T05:00:00Z]`) — convert before querying.
+
+```bash
+curl -s -m 60 "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode "query=_time:1d application:dv-pipecat environment:=<ENV> call_id:=\"<IDENTIFIER>\" | limit 500" \
+  -o /tmp/vl.json
+```
+
+`call_id` is an indexed field, and matching it is **strictly better than a
+substring search** — not just faster. Most lines carry the field without
+printing the id in their text, so substring under-returns badly. Measured on
+three live calls:
+
+| id | `call_id:=` | substring |
+|---|---|---|
+| `01a03784-…` | 82 lines | 27 |
+| `3825374100` | 84 lines | 32 |
+| `e42e1ec1…` | 177 lines | 34 |
+
+**If it returns 0 rows, the identifier is a provider id, not the Ringg id.**
+Recover the real one and re-query — this is the difference between "this call
+barely logged anything" and the actual call:
+
+```bash
+REAL=$(curl -s -m 60 "http://127.0.0.1:9430/select/logsql/query" \
+  --data-urlencode "query=_time:1d application:dv-pipecat \"<IDENTIFIER>\" | fields call_id, _msg | limit 20" \
+  | { grep -oE '"call_id":"[^"u][^"]*"|callback_call_id[=: ]+[0-9a-f-]{8,}' || true; } \
+  | grep -oE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | sort -u | head -1)
+```
+
+Then re-run the indexed query with `$REAL`. A bare substring filter is the last
+resort, and also the only way to see early-startup lines (`[run_bot] call_id=…`
+and friends), which are logged before the call context is bound and so carry no
+`call_id` field at all — they show as `call_id: "unknown"` or omit it.
+
+If the result is empty, in order: try the other `environment`; drop the
+`environment` filter; widen `_time` (max range is 720h); try
+`application:~"dv-pipecat"` to include `dv-pipecat-canary`.
+
+### 4. Parse and display
+
+The response is **JSON Lines** — one JSON object per line, not a wrapped
+envelope. `jq .` on the whole file fails; parse per line:
+
+```bash
+jq -r '[._time, (.level // "-"), ._msg] | @tsv' /tmp/vl.json | sort
+```
+
+Results come back unordered, so **sort by `_time` yourself**. Display
+chronologically (earliest first) with timestamp, level, and message.
+
+Useful fields: `_time`, `_msg`, `level`, `call_id`, `pod`, `application`,
+`environment`, `cluster`, `container`, `logger`, `module`, `function`, `line`.
+
+Two field names differ from the backend's records and are easy to get wrong:
+pipecat logs the pod as **`pod`**, not `pod_name`, and its **`level` is
+lowercase** (`debug`/`info`/`warning`/`error`) while the parallel `severity`
+field is uppercase. A filter on the wrong one silently matches nothing.
+
+### 5. Analyze
+
+After displaying the logs, provide a brief analysis:
+
+1. **Call timeline**: what happened (connected, greeted, conversation, ended)
+2. **Errors/warnings**: highlight ERROR/WARNING lines
+3. **Tool calls**: function/tool calls made during the call
+4. **TTS/STT activity**: speech-related events
+5. **Call outcome**: normal hangup, error, transfer, timeout
+6. **Anomalies**: long silences, repeated errors, unexpected transitions
+
+Note the session shape while reading — it changes which pipeline is running and
+therefore which failures are possible. `Using single_node bot` vs a FlowManager
+line; `channel=livekit|daily|telephony|smallwebrtc`; and `text mode, starting
+flow` / `Local recording disabled for text-only session` / `lk.chat message ->
+queued to LLM`, which mark a **text-only** session (no STT, no
+`TranscriptionFrame`).
+
+### 6. Grafana link
+
+Give the user a link to the pipecat dashboard (datasource `vl-use1`, i.e. the
+same us-east1 store queried above):
+
+```
+http://10.120.0.49/d/vl-pipecat-logs
+```
+
+Its variables are `environment`, `application`, `container`, `pod`, `call_id`,
+`trace_id`, `message_substring`, `minimum_level`. The host is an internal
+LoadBalancer IP — reachable on the VPN, not from the public internet.
+
+## Cross-checking against the backend
+
+When pipecat logs are thin, the backend's call-completion payload is a good
+second source: it dumps the whole completion object, including
+`function_calls_detailed` with each tool's `response_data` — so a failing
+tool's error string is greppable even without pipecat logs. It lives in the
+**Mumbai** (India) or **KSA** VL instance, not us-east1:
+
+```bash
+kubectl --context gke_desivocalprod01_asia-south1_desivocal-prod-cluster \
+  -n victorialogs port-forward svc/victoria-logs 9429:9428 > /tmp/vl-pf-mum.log 2>&1 &
+
+curl -s -m 60 "http://127.0.0.1:9429/select/logsql/query" \
+  --data-urlencode 'query=_time:1d application:new-calling-agent-taskiq-call-completion "<IDENTIFIER>" | limit 50'
+```
+
+KSA is `gke_desivocalprod01_me-central2_desivocal-prod-cluster-ksa`, same
+namespace and service.
+
+## Troubleshooting
+
+- **Empty result, `200` status** — the usual cause is the wrong region or a
+  missing `_time` window, not a missing call. Work through the fallbacks in
+  step 3 before concluding the call has no logs.
+- **`too big time range selected`** — `_time:` is mandatory and capped at 720h.
+  A query with no `_time` filter is rejected with a `1677-09-21 → 2262-04-11`
+  range in the message; that is the missing-filter error, not a real range.
+- **`jq: parse error`** — the response is JSON Lines; see step 4.
+- **Port-forward drops** — `kubectl port-forward` dies on idle or on pod
+  rotation. Re-run step 2; the retry makes that safe to repeat.
+- **Zero rows, or only a handful** — you were almost certainly handed a
+  provider id rather than the Ringg one. Resolve it (step 3) instead of
+  widening the time range.
+- **Slow or refused reads** — each region's `vmauth` allows 4 concurrent reads.
+  Serialize queries rather than fanning out.
+- **A migrated webcall with no pipecat logs at all** — webcalls with
+  `telephony_provider=livekit` run on LiveKit Agents, not pipecat, and log to
+  GCP Cloud Logging (`desivocalprod01`, container `ringg-webcall`).
+
+## Reference
+
+For LogsQL syntax and more filtering options, see [query-reference.md](query-reference.md).