vybekiit 0.7.28 → 0.7.32

package/dist/global-skills/use-railway/references/tracing.md

@@ -1,6 +1,6 @@
 # Tracing
 
-Trace a request from Railway's edge through a service and into the services it calls. Turn tracing on with the `set-service-tracing` and `set-project-tracing` MCP tools and read the result with `list-traces` and `get-trace`, or do both on the project's **Traces** tab.
+Trace a request from Railway's edge through a service and into the services it calls. Tracing is switched on per service **and per environment**: with the `set-service-tracing` MCP tool, `railway trace enable`, or the **Tracing setup** drawer on the project's **Traces** tab. Read the result with `list-traces` and `get-trace`, or on the same tab.
 
 Tracing is a preview feature. If the project has no **Traces** tab, the account needs **Tracing** enabled in Priority Boarding first.
 
@@ -13,56 +13,89 @@ Spans from inside the service only appear once the service exports them, through
 
 ## Enable tracing
 
-Tracing has three settings. The project default (`tracingEnabled`) and the sample rate (`tracingSampleRate`) live on the project; a per-service override (`tracingEnabled`, `null` follows the project) and the automatic instrumentation switch (`autoInstrumentationEnabled`) live on the service. Automatic instrumentation only takes effect while the service's tracing is on.
+Tracing is two switches on each service instance, so they are set per service and per environment. `tracingEnabled` makes the edge trace requests to the service and gives its next deploy the OpenTelemetry variables below; `autoInstrumentationEnabled` attaches eBPF probes to the service's processes and only does anything while tracing is on. There is no project-wide default and no sample rate: every client-facing request to a traced service is traced. Enabling a service in `production` says nothing about `staging`, so check and set each environment the user cares about.
 
-**Dashboard:** open the **Traces** tab → **Tracing setup**. Toggle **Trace requests by default** under Project, optionally set a **Sample rate** (percentage), and use each service row's **Traced** switch for overrides and **Automatic instrumentation** / **Manual instrumentation** to pick how it exports spans. The same controls are on the service under **Settings → Tracing**.
+**Dashboard:** open the **Traces** tab → **Tracing setup**. The drawer lists the services of the environment the dashboard is on (the subtitle names it); each row's **Traced** switch and **Automatic instrumentation** / **Manual instrumentation** choice apply to that environment only. Switch environment to configure another. The service's **Settings → Tracing** section links to the same drawer.
 
-**Agent path:** three MCP tools read and change these settings, and `railway trace` does the same from the CLI. Resolve IDs from the URL or `railway status --json` first, and read before writing. On a Railway cloud agent in a dashboard chat session the `railway` CLI is unauthenticated, so resolve IDs with `list-services` and stay on the MCP tools throughout.
+**Agent path:** two MCP tools read and change these settings, and `railway trace` does the same from the CLI. Resolve IDs from the URL or `railway status --json` first, and read before writing. Both tools use the `production` environment when `environmentId` is omitted, so pass it whenever the user is looking at another environment. On a Railway cloud agent in a dashboard chat session the `railway` CLI is unauthenticated, so resolve IDs with `list-services` and stay on the MCP tools throughout.
 
 | Tool | Access | Purpose |
 |---|---|---|
-| `get-tracing` | viewer | The project default (`tracingEnabled`, `sampleRate`) and, per service, `tracingOverride` (`true`/`false` pinned, `null` follows the project), the resolved `tracingEnabled`, `autoInstrumentationEnabled` and whether it is `autoInstrumentationActive`. Pass `serviceId` for one service, omit it for every service in the project |
-| `set-service-tracing` | member | `tracingEnabled` `true`/`false` pins one service regardless of the project default, `null` makes it follow the project again; `autoInstrumentationEnabled` switches OBI for the service. Each is optional and independent; omit what should stay as it is. Service-wide, not per environment |
-| `set-project-tracing` | member | `tracingEnabled` sets the project default for every service without an override; `sampleRate` is the fraction of client-facing requests the edge traces, `0..1`, `null` resets to Railway's default of every request. Each is optional |
+| `get-tracing` | viewer | The `environment` and, per service in it, `tracingEnabled`, `autoInstrumentationEnabled` and whether it is `autoInstrumentationActive` (both on). Pass `serviceId` for one service, omit it for every service in the environment |
+| `set-service-tracing` | member | `tracingEnabled` and `autoInstrumentationEnabled` for one service in one environment. Each is optional and independent; omit what should stay as it is. Returns the service's state in that environment |
 
-All three take `projectId`. `describe-service` reports the same tracing state for one service.
+Both take `projectId` and an optional `environmentId`. `describe-service` reports the same tracing state for one service in the environment it was asked about.
 
 ```text
-Get tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb
+Get tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb in environment <environment-id>
 ```
 
 ```text
-Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>: tracingEnabled true
+Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>, environment <environment-id>: tracingEnabled true
 ```
 
 ```text
-Set project tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb: tracingEnabled true, sampleRate 0.25
+Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>, environment <environment-id>: autoInstrumentationEnabled true
 ```
 
-```text
-Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>: autoInstrumentationEnabled true
+`set-service-tracing` warns when it switches auto-instrumentation on for a service whose tracing is off: the switch does nothing until tracing is enabled too. Without Railway MCP or `railway trace`, the public `serviceInstanceUpdate` mutation sets the same fields through `railway api`, and the environment's service instances carry them for reading; see [request.md](request.md). That fallback needs an authenticated CLI, which a cloud agent's chat session does not have.
+
+```bash
+railway api \
+  'mutation traceInstance($serviceId: String!, $environmentId: String!) {
+    serviceInstanceUpdate(serviceId: $serviceId, environmentId: $environmentId,
+      input: { tracingEnabled: true, autoInstrumentationEnabled: false })
+  }' \
+  --variables '{"serviceId":"<service-id>","environmentId":"<environment-id>"}'
+
+railway api \
+  'query tracing($id: String!) {
+    environment(id: $id) { serviceInstances { edges { node {
+      serviceId serviceName tracingEnabled autoInstrumentationEnabled
+    } } } }
+  }' \
+  --variables '{"id":"<environment-id>"}'
 ```
 
-The sample rate is a fraction from 0 to 1 in the tools and the API; the dashboard shows it as a percentage. `set-service-tracing` warns when it switches auto-instrumentation on for a service whose tracing is off: the switch does nothing until the service, or the project default, is enabled. Without Railway MCP or `railway trace`, the public `projectUpdate` (`tracingEnabled`, `tracingSampleRate`) and `serviceUpdate` (`tracingEnabled`, `autoInstrumentationEnabled`) mutations through `railway api` set the same fields; see [request.md](request.md). That fallback needs an authenticated CLI, which a cloud agent's chat session does not have.
+The older `Project` and `Service` tracing fields still exist as deprecated stubs: `projectUpdate(tracingEnabled)` switches every instance in the project, `serviceUpdate(tracingEnabled)` the service in every environment, and `tracingSampleRate` is ignored. Don't use them; they flip environments the user didn't mention.
 
-**CLI:** `railway trace` (aliases `traces`, `tracing`) sets the same fields. Use it when the user works in a linked repo or wants exact command output; on a cloud agent's chat session the CLI is unauthenticated, so stay on MCP there.
+**CLI:** `railway trace` (aliases `traces`, `tracing`) sets the same switches, in one environment at a time: the linked one, or `--environment <name>`. This needs a CLI release newer than 5.62.1. An older CLI still runs `railway trace enable`, but through the deprecated service field, so it sets the service in every environment, and its `inherit`, `--project-default` and `--sample-rate` belong to the model that is gone. Use the CLI when the user works in a linked repo or wants exact command output; on a cloud agent's chat session stay on MCP.
 
 ```bash
-railway trace status --all                              # project default, every service, last edge/app span
-railway trace enable --service <service>                # pin tracing on for one service
+railway trace status                                    # linked service in the linked environment
+railway trace status --all                              # every service in the environment, last edge/app span
+railway trace enable --service <service>                # trace one service in the linked environment
 railway trace enable --auto-instrument                  # tracing plus OBI for the linked service
-railway trace enable --project-default --sample-rate 0.25
-railway trace disable --auto-instrument                 # pin tracing off and switch OBI off
-railway trace inherit                                   # follow the project default again
+railway trace enable --all --environment staging        # every service in staging
+railway trace disable --auto-instrument                 # tracing and OBI off
 ```
 
-Tracing is service-wide, so `--environment` only scopes `status`, `list` and `get`; pass `--project <id> --environment <name>` when nothing is linked. `--json` prints one document with the project default and the service state. Changing tracing needs a user or workspace token; a project token (`RAILWAY_TOKEN`) can only read. The CLI has no switch for auto-instrumentation alone: `railway trace disable --auto-instrument` followed by `railway trace enable` leaves tracing on and OBI off.
+`--all` and `--service` are mutually exclusive. Pass `--project <id> --environment <name>` together when nothing is linked. `--json` prints one document with `environment: { id, name }` and the service state. Changing tracing needs a user or workspace token; a project token (`RAILWAY_TOKEN`) can only read. The CLI has no switch for auto-instrumentation alone: `railway trace disable --auto-instrument` followed by `railway trace enable` leaves tracing on and OBI off, or use `set-service-tracing` with only `autoInstrumentationEnabled`.
 
 What happens next:
 
-- The edge starts tracing requests to the service's domains within seconds.
+- The edge starts tracing requests to the service's domains in that environment within seconds.
 - Automatic instrumentation reaches the running containers within about a minute. No redeploy.
-- The OpenTelemetry variables below are added on the **next deploy**. An app with an SDK exports nothing until it is redeployed: `railway redeploy --service <service> --yes`.
+- The OpenTelemetry variables below are added on the **next deploy** in that environment. An app with an SDK exports nothing until it is redeployed: `railway redeploy --service <service> --yes`.
+
+### Infrastructure as code
+
+The environment config carries the same two switches as `services[<id>].tracing: { enabled, autoInstrumentation }`, so tracing is part of what `railway config pull`, `plan` and `apply` manage (see [iac.md](iac.md)). Railway serialises only the switches that are on: an untraced service has no `tracing` block, and `enabled: false` is the same as leaving it out. `railway config pull` renders the block for a traced service:
+
+```ts
+const api = service("api", {
+  source: github("owner/repo", { branch: "main" }),
+  tracing: { enabled: true, autoInstrumentation: true },
+});
+```
+
+A plan shows a tracing change as a `resource.update` with `field: "tracing"`. Turning `enabled` on or off has `deployEffect: deploy`, because the `OTEL_*` variables land with a deploy; an `autoInstrumentation`-only change is `deployEffect: none` and reaches the running containers live. A new service carries `tracing` on its create.
+
+**Current limitation: the SDKs don't carry `tracing` yet.** The TypeScript `service()` helper, and the Python and Go ones, whitelist their config keys and `tracing` is not among them, so a `tracing:` the user writes in `railway.ts` is dropped before the CLI sees it and never applies. The other direction is worse: a compiled config with no `tracing` block asks to remove it, so `railway config plan` against a service that is traced shows a `tracing` update with `after: null` and `deployEffect: deploy`, and `railway config apply` turns tracing off and redeploys. Until the SDKs ship the field:
+
+- Set tracing with `set-service-tracing`, `railway trace` or the dashboard, not in `railway.ts`, and say why when the user asks for it in config.
+- Read every plan for a `tracing` change nobody authored, and don't apply one. `railway config pull` reproduces the block, but the SDK drops it again on the next plan, so pulling doesn't fix it.
+- If an apply already disabled tracing, re-enable it with `set-service-tracing`; the redeploy for the variables follows.
 
 ## Choose how the service exports spans
 
@@ -80,7 +113,7 @@ Pick one per service. Running an SDK in a service that also has automatic instru
 
 ## Instrument with an OpenTelemetry SDK
 
-When tracing is on for a service, its next deploy gets these variables. They show up in the service's **Variables** tab alongside the other Railway-provided variables and every OpenTelemetry SDK reads them, so an SDK configured without an explicit endpoint exports to Railway:
+When tracing is on for a service in an environment, its next deploy there gets these variables. They show up in the service's **Variables** tab alongside the other Railway-provided variables and every OpenTelemetry SDK reads them, so an SDK configured without an explicit endpoint exports to Railway:
 
 | Variable | Value |
 |---|---|
@@ -89,8 +122,6 @@ When tracing is on for a service, its next deploy gets these variables. They sho
 | `OTEL_EXPORTER_OTLP_HEADERS` | A header the receiver requires on every export |
 | `OTEL_SERVICE_NAME` | The Railway service name |
 | `OTEL_SERVICE_VERSION` | The commit SHA, or the deployment ID for image and CLI deploys |
-| `OTEL_TRACES_SAMPLER` | `parentbased_traceidratio`, only when the project set its own sample rate |
-| `OTEL_TRACES_SAMPLER_ARG` | The project's sample rate as a fraction, only when the project set its own sample rate |
 
 Rules the agent must apply:
 
@@ -101,7 +132,7 @@ Rules the agent must apply:
   Set variables for project <project-id>, service <service-id>: OTEL_METRICS_EXPORTER=none, OTEL_LOGS_EXPORTER=none, skipDeploys true
   ```
 
-- **User variables win.** A service that sets its own `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (for example to keep exporting to its own collector) gets none of the tracing variables, and its spans don't reach the Traces tab; edge spans still do. A service that sets either `OTEL_TRACES_SAMPLER` or `OTEL_TRACES_SAMPLER_ARG` keeps both of its own and Railway adds neither. Check `railway variable list --service <service> --json` before assuming the provided values apply.
+- **User variables win.** A service that sets its own `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (for example to keep exporting to its own collector) gets none of the tracing variables, and its spans don't reach the Traces tab; edge spans still do. Railway sets no sampler variables, so a sampler the service configures is the only one in play. Check `railway variable list --service <service> --json` before assuming the provided values apply.
 - **Load the SDK first.** Instrumentation libraries (`@opentelemetry/auto-instrumentations-node`, `opentelemetry-instrumentation-*` and the like) patch libraries at import time, so the SDK must be loaded before the app's modules: a `--require`/`--import` flag in the start command, `NODE_OPTIONS`, a Python `opentelemetry-instrument` wrapper, a Java `-javaagent`, and so on. Set it with `railway environment edit --service-config <service> deploy.startCommand "<command>"` or as a variable; see [deploy.md](deploy.md).
 - **Keep W3C Trace Context propagation on** (the SDK default in most languages; Go requires setting the propagator explicitly) so the service continues the edge's trace instead of starting its own.
 - **Don't override `OTEL_SERVICE_NAME`** unless the user wants spans attributed under a different name than the Railway service.
@@ -113,7 +144,15 @@ Per-language install steps, framework notes, and a custom-span example are in th
 Instrumentation libraries give a trace its skeleton: a server span per request and a client span per call a library recognises. On its own that is barely better than automatic instrumentation. The value comes from spans the app opens itself around the units of work its authors think in. When adding tracing to a codebase, or when asked what to instrument, cover these three layers, in this order:
 
 1. **Inbound work.** One span per HTTP handler, gRPC method, queue or job consumer invocation, cron run and WebSocket message. The HTTP and gRPC server instrumentations usually cover handlers. Consumers, cron and background jobs are not covered by the edge or by most instrumentations, so wrap each message or run in its own span (kind `CONSUMER` or `INTERNAL`) and, where the message carries a `traceparent`, continue that context so the trace links back to the producer. Without this, background work is invisible or shows up as orphaned client spans.
-2. **I/O: database queries, cache calls, outgoing HTTP and gRPC calls, queue publishes.** This is where latency and failures hide. Use the driver or client instrumentation where one exists. Where none does (a raw socket client, a vendor SDK the instrumentations don't know), wrap the call in a `CLIENT` span with the standard `db.*`, `server.address` or `url.full` attributes. Every remote call should be visible as its own span.
+2. **I/O: database queries, cache calls, outgoing HTTP and gRPC calls, queue publishes.** This is where latency and failures hide. Use the driver or client instrumentation where one exists. Where none does (a raw socket client, a vendor SDK the instrumentations don't know), wrap the call in a `CLIENT` span with the standard `db.*`, `server.address` or `url.full` attributes. Every remote call should be visible as its own span. List every database driver or ORM, cache, HTTP and queue client in the dependency manifest and check each one, because the bundles miss common ones:
+   - **Node.js**: `@opentelemetry/auto-instrumentations-node` covers `pg`, `mysql2`, `ioredis`, `mongodb`, `mongoose`, `knex` and `undici`/`http`, but not Prisma (add `@prisma/instrumentation` to the SDK's instrumentations) and not postgres.js, Drizzle on postgres.js, `Bun.sql` or the Neon serverless driver (no instrumentation exists; wrap queries in a `CLIENT` span with `db.system`, `db.namespace`, `db.operation.name`). An app written as ES modules needs the `@opentelemetry/instrumentation/hook.mjs` loader, or imported drivers stay unpatched while the `http` server span still appears.
+   - **Python**: `opentelemetry-distro` alone instruments nothing; run `opentelemetry-bootstrap -a install` so the instrumentations for the installed drivers (`psycopg2`, `asyncpg`, `SQLAlchemy`, `redis`, `pymongo`, `requests`, `httpx`) are installed too.
+   - **Go**: nothing is automatic. Wrap `database/sql` with `otelsql`, pgx with `otelpgx`, go-redis with `redisotel`, HTTP clients with `otelhttp.NewTransport`.
+   - **Rust**: `sqlx` emits tracing events, not spans; wrap queries in a span. `reqwest` needs `reqwest-tracing`.
+   - **Deno**: `OTEL_DENO=true` covers `fetch`, `Deno.serve` and `node:http`; npm database drivers need their own instrumentation or a hand-written span.
+   - **Java, Ruby, .NET, PHP**: the agent, `opentelemetry-instrumentation-all`, the per-library packages (`Npgsql.OpenTelemetry`, `OpenTelemetry.Instrumentation.SqlClient`, ...) and `opentelemetry-auto-pdo` cover JDBC, ActiveRecord, ADO.NET and PDO; confirm the package for the driver in use is present.
+   A server span with no client spans under it means this layer was skipped. Before opening the PR, run the app once with the console exporter (`OTEL_TRACES_EXPORTER=console` or the runtime's equivalent) and hit a route that queries the database: a `CLIENT` span carrying `db.system`, even a failed one, proves the driver is patched. List each client and the instrumentation that covers it, or why it is not covered, in the PR description.
+
 3. **Logical units of work.** A function that groups several I/O calls into one meaningful step (`checkout`, `syncUser`, `renderInvoice`), or one that is CPU-heavy on its own (parsing a large payload, image resizing, template rendering, serialising a big response). Give each an `INTERNAL` span carrying the identifiers a person debugging it would want (`order.id`, `tenant`, item counts). This turns twelve sibling `SELECT` spans under a handler into a tree that reads like the code and says which step took the time. Rule of thumb: if you would want log lines saying "starting X" and "finished X in N ms", X is a span.
 
 Keep spans out of tight loops and trivial helpers. A span per item in a loop of thousands hits the per-replica export limit (see Sampling) and adds nothing a `count` attribute on the parent wouldn't; instrument the loop, not the iteration. Name spans with low-cardinality names (`GET /users/{id}`, `db.query users`, `process order`) and put the variable parts in attributes. Set span status to `ERROR` and record the exception when a unit fails, so `@status:error` finds it. Never put secrets, tokens or raw personal data in span names or attributes.
@@ -131,7 +170,7 @@ What differs from a repo service:
 
 Recipe:
 
-1. Turn tracing on for the function with `set-service-tracing` (or `railway trace enable --service <function>`) if `get-tracing` shows it off. Leave `autoInstrumentationEnabled` off; it does nothing for Bun.
+1. Turn tracing on for the function in the environment it runs in with `set-service-tracing` (or `railway trace enable --service <function>`) if `get-tracing` shows it off there. Leave `autoInstrumentationEnabled` off; it does nothing for Bun.
 2. Set the exporter variables with `set-variables`. `NodeSDK` exports metrics and logs over OTLP by default and the receiver rejects both. Pass `skipDeploys: true`; the code push in step 5 is the deploy that picks them up. Check `list-variables` first: a function that sets its own `OTEL_EXPORTER_OTLP_ENDPOINT` gets none of Railway's tracing variables.
 
    ```text
@@ -147,8 +186,8 @@ Recipe:
    import { Hono } from "hono@4";
    import { httpInstrumentationMiddleware } from "@hono/otel@1";
 
-   // Reads OTEL_EXPORTER_OTLP_*, OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER*
-   // from the variables Railway provides. Nothing to configure.
+   // Reads OTEL_EXPORTER_OTLP_* and OTEL_SERVICE_NAME from the variables
+   // Railway provides. Nothing to configure.
    const sdk = new NodeSDK();
    sdk.start();
    process.on("SIGTERM", () => sdk.shutdown().finally(() => process.exit(0)));
@@ -175,7 +214,7 @@ Recipe:
    export default { port: Number(Bun.env.PORT ?? 3000), fetch: app.fetch };
    ```
 
-   For a function that calls `Bun.serve` itself, wrap its `fetch` handler: take the parent from `propagation.extract(context.active(), req.headers, { get: (h, k) => h.get(k) ?? undefined, keys: (h) => [...h.keys()] })` and run the handler inside `tracer.startActiveSpan(name, { kind: SpanKind.SERVER }, parent, ...)`. For an outgoing call, start a `SpanKind.CLIENT` span and `propagation.inject(context.active(), headers, { set: (h, k, v) => h.set(k, v) })` into a `Headers` object before `fetch`. A cron or script function has no server: its spans are new roots (sampled at the project rate through the sampler variables) and it must `await sdk.shutdown()` as its last statement, or the batch never leaves the process. The docs page has all three in full.
+   For a function that calls `Bun.serve` itself, wrap its `fetch` handler: take the parent from `propagation.extract(context.active(), req.headers, { get: (h, k) => h.get(k) ?? undefined, keys: (h) => [...h.keys()] })` and run the handler inside `tracer.startActiveSpan(name, { kind: SpanKind.SERVER }, parent, ...)`. For an outgoing call, start a `SpanKind.CLIENT` span and `propagation.inject(context.active(), headers, { set: (h, k, v) => h.set(k, v) })` into a `Headers` object before `fetch`. A cron or script function has no server: its spans are new roots, recorded on every run, and it must `await sdk.shutdown()` as its last statement, or the batch never leaves the process. The docs page has all three in full.
 
 5. Write it back with `update-function-source-code` (the whole file; pass `staged: true` to stage instead of deploying live) or `railway functions push --path <file>`. This deploy also adds the variables.
 6. Verify with the steps below: `curl -sI https://<domain>/hello/x | grep -i x-railway-trace-id`, then `get-trace` on the ID. A span with `component` `service` and scope `@hono/otel` (or the tracer name) means the function exports. If the deploy logs show `bun install` failing, an import specifier is wrong; if they show OTLP export errors for metrics or logs, step 2 was skipped.
@@ -184,9 +223,8 @@ Docs: [Functions](https://docs.railway.com/observability/tracing/functions).
 
 ## Sampling
 
-- **Default is every request.** With no project sample rate, the edge traces 100% of client-facing requests and Railway adds neither sampler variable, so the SDK's default parent-based sampler follows the edge's decision and records every root span of its own.
-- **A project rate applies everywhere.** The edge draws once per client-facing request and writes the decision into the `traceparent` sampled flag; requests it didn't sample carry a cleared flag, so the SDK records nothing for them. The two sampler variables make the SDK sample roots the edge never saw (cron jobs, queue consumers, private-network calls without a `traceparent`) at the same rate.
-- **A client `traceparent` overrides the draw.** A request that arrives with the sampled flag set is always traced; one with it cleared never is. Use this to force a single trace while debugging a service with a low rate:
+- **Every request is traced.** The edge traces 100% of client-facing requests to a traced service and writes the decision into the `traceparent` sampled flag. Railway sets no sampler variables, so the SDK's default parent-based sampler follows the edge and records every root span of its own (cron jobs, queue consumers, private-network calls without a `traceparent`). There is no project or service rate to set.
+- **A client `traceparent` overrides.** A request that arrives with the sampled flag set is always traced; one with it cleared never is. An instrumented client can start a trace that continues into Railway, and a caller can keep a request out of tracing by sending a cleared flag. To trace one specific request while debugging:
 
   ```bash
   curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" https://<domain>/<path>
@@ -194,7 +232,7 @@ Docs: [Functions](https://docs.railway.com/observability/tracing/functions).
 
   Generate a fresh random 32-hex trace ID (the second field) for each request; reusing one merges requests into a single trace.
 
-- Lower the rate for busy services. Each replica can export 1,000 spans per 10 seconds; exports over the limit are rejected and the SDK reports a partial success.
+- **Sample in the SDK when the volume is too high.** Each replica can export 1,000 spans per 10 seconds; exports over the limit are rejected and the SDK reports a partial success. Disable noisy instrumentations first. If that isn't enough, set `OTEL_TRACES_SAMPLER=traceidratio` and `OTEL_TRACES_SAMPLER_ARG=<fraction>` on the service yourself; a `parentbased_*` sampler follows the edge's flag and would change nothing for edge requests. Traces the SDK drops still show the edge and proxy hops.
 
 ## Read traces
 
@@ -203,7 +241,8 @@ Traces are read through **Remote MCP** (the default agent path), `railway trace
 | Tool | Access | Purpose |
 |---|---|---|
 | `list-traces` | viewer | Traces of an environment, newest first, one row per request with at least one span matching `filter`. Optional `serviceId`, `startDate`/`endDate` (ISO 8601 with timezone; defaults to the last hour), `limit` (default 100, max 500) |
-| `get-trace` | viewer | One trace as an indented span tree, with every span's attributes, events and links in the structured result. Takes `traceId` (32 hex characters); `maxSpans` caps the result and the output says when it was hit |
+| `get-trace` | viewer | One trace as an indented span tree (each line shows the span kind and the attribute that says what it talked to: `db.system`, `http.route`, `url.full` or `server.address`), with every span's attributes, events and links in the structured result. Takes `traceId` (32 hex characters); `maxSpans` caps the result and the output says when it was hit |
+| `get-tracing-coverage` | viewer | What one service's own instrumentation covers: its spans in the window (default the last hour) by span kind, by the remote system they name (`db.system`, `messaging.system`, `rpc.system`, or `http` for HTTP client spans) and by the instrumentation scope that emitted them (`@opentelemetry/instrumentation-pg`, ...). Takes `serviceId`; optional `startDate`/`endDate`. Edge and proxy spans are left out. The one call that answers "are the database queries instrumented?" |
 
 Both take `projectId` and an optional `environmentId`; omit it and the `production` environment is used, so pass the ID explicitly when the user is looking at another environment. Traces belong to the environment they were exported from.
 
@@ -240,22 +279,28 @@ Workflow for "why is this request slow / failing":
    ```
 
 2. **Fetch that trace** with `get-trace` or `railway trace get <trace-id>` and the returned ID. Edge and proxy spans confirm tracing is on; a span with `component` `service` confirms the app is exporting. After enabling an SDK, that only happens once the redeploy that added the variables is live; after enabling automatic instrumentation, allow about a minute.
-3. **Or watch the dashboard.** The Traces tab is at `https://railway.com/project/<project-id>/traces?environmentId=<environment-id>`; its **Trace ID** field accepts a bare 32-hex ID or a whole `traceparent` header. In **Tracing setup**, each service row shows when the edge and the app last exported a span, and the **App** indicator turns green on the first span from the service itself.
+3. **Check what the app covers** with `get-tracing-coverage` for the service, after sending a few requests that hit the database. Every database, cache and queue the service uses must appear under "Remote systems" and the driver instrumentations under "Instrumentation scopes". `SERVER` spans with no database client span mean a driver is still unpatched (see the I/O layer in [What to instrument](#what-to-instrument)); `list-traces` with `@kind:client @db.system:*` is the same check as a trace list.
+4. **Or watch the dashboard.** The Traces tab is at `https://railway.com/project/<project-id>/traces?environmentId=<environment-id>`; its **Trace ID** field accepts a bare 32-hex ID or a whole `traceparent` header. In **Tracing setup**, each service row shows when the edge and the app last exported a span, and the **App** indicator turns green on the first span from the service itself.
 
 ## Troubleshoot
 
-- **No traces at all**: confirm `get-tracing` or `railway trace status` reports tracing on for the service (its override, then the project default), the service has a public domain, and the account has Tracing in Priority Boarding. With a low rate and little traffic, force one with the `traceparent` curl above, then `get-trace` it.
+- **No traces at all**: confirm `get-tracing` or `railway trace status` reports tracing on for the service in the environment the user is looking at (the switches and the traces both belong to an environment), the service has a public domain, and the account has Tracing in Priority Boarding. With little traffic, send a request and check for `x-railway-trace-id` on the response, then `get-trace` it.
+- **Traced in `production` but not elsewhere**: tracing is set per environment. Enable it in the other environment with `set-service-tracing` and its `environmentId`, or `railway trace enable --environment <name>`.
 - **Edge spans only, nothing from the app** (`get-trace` shows only `edge` and `proxy` components): the variables land on the next deploy, so redeploy. Then check the service doesn't set its own `OTEL_EXPORTER_OTLP_ENDPOINT`, the SDK loads before the app serves, and, for OBI, the process is a supported runtime handling HTTP or gRPC.
+- **Server spans but no database or client spans** (`get-tracing-coverage` shows `SERVER` and `INTERNAL` only, or `get-trace` shows a request span with no `CLIENT` children): the driver is not instrumented. In Node.js an ES-module app without the `@opentelemetry/instrumentation/hook.mjs` loader, Prisma without `@prisma/instrumentation`, or a client no instrumentation covers (postgres.js, Drizzle on it, `Bun.sql`); in Python a missing `opentelemetry-bootstrap -a install`; in Go an unwrapped `database/sql`. See the I/O layer in [What to instrument](#what-to-instrument).
 - **App spans appear as separate traces** (`list-traces` shows service-rooted traces with `hasEdge` false next to edge-only ones): the SDK isn't reading `traceparent`. Enable the W3C Trace Context propagator and make sure nothing in front of the handlers strips the header.
 - **SDK logs metrics or logs export errors**: set `OTEL_METRICS_EXPORTER=none` and `OTEL_LOGS_EXPORTER=none`.
 - **Duplicate spans per request**: the service runs an SDK with automatic instrumentation on. Switch it off with `set-service-tracing` (`autoInstrumentationEnabled` false), or on the CLI `railway trace disable --auto-instrument` then `railway trace enable`.
-- **Spans missing from a busy service**: over 1,000 spans per replica per 10 seconds. Disable noisy instrumentations or lower the sample rate.
+- **Spans missing from a busy service**: over 1,000 spans per replica per 10 seconds. Disable noisy instrumentations or sample in the SDK; see [Sampling](#sampling).
 - **A Function shows edge spans only**: automatic instrumentation can't help (Bun); the SDK has to be in the file. Check `get-function-source-code` for the `NodeSDK` block and the request wrapper, that the deploy logs show `bun install` succeeding, and that `OTEL_METRICS_EXPORTER`/`OTEL_LOGS_EXPORTER` are `none`. See [Instrument a Function (Bun)](#instrument-a-function-bun).
-- **Setting the sample rate fails validation**: `set-project-tracing`, `railway trace enable --project-default --sample-rate` and the API take a fraction 0..1, not a percentage.
+- **`railway config plan` wants to remove `tracing`**: the IaC SDKs drop the field, so every plan against a traced service proposes turning it off. Don't apply it; see [Infrastructure as code](#infrastructure-as-code).
 
 ## Validated against
 
-- Docs: [tracing.md](https://docs.railway.com/observability/tracing), [automatic-instrumentation.md](https://docs.railway.com/observability/tracing/automatic-instrumentation), [nodejs.md](https://docs.railway.com/observability/tracing/nodejs), [tracing/functions.md](https://docs.railway.com/observability/tracing/functions), [functions.md](https://docs.railway.com/functions), [variables/reference.md](https://docs.railway.com/variables/reference)
-- Platform source (railwayapp/mono): `common/javascript/models/src/tracingVariables.ts` (provided variables and precedence), `common/javascript/models/src/functions.ts` and `services.ts` (function start command, image prefix), `packages/backboard/src/graphql/v2/schema/schema.graphql` (`Project.tracingEnabled`, `Project.tracingSampleRate`, `Service.tracingEnabled`, `Service.autoInstrumentationEnabled`, `ProjectUpdateInput`, `ServiceUpdateInput`), `packages/hikari/src/settings/tunables.rs` (default sample rate), `packages/stacker-oteld/configs/main.go` (span limit), `packages/backboard/src/handlers/http/routes/mcp/tools/listTraces.ts`, `getTrace.ts`, `getTracing.ts`, `setServiceTracing.ts`, `setProjectTracing.ts`, `tracingMcpHelpers.ts`, `getFunctionSourceCode.ts` and `updateFunctionSourceCode.ts` (MCP tools)
-- Function runtime (railwayapp/code-images): `bun/Dockerfile` (Bun 1.4.0), `bun/run.sh` (import scan, `bun install` per start, `bun run --smol index.tsx`)
+- Docs: [tracing.md](https://docs.railway.com/observability/tracing), [automatic-instrumentation.md](https://docs.railway.com/observability/tracing/automatic-instrumentation), [nodejs.md](https://docs.railway.com/observability/tracing/nodejs), [tracing/functions.md](https://docs.railway.com/observability/tracing/functions), [functions.md](https://docs.railway.com/functions), [variables/reference.md](https://docs.railway.com/variables/reference), [cli/trace.md](https://docs.railway.com/cli/trace), [infrastructure-as-code/reference.md](https://docs.railway.com/infrastructure-as-code/reference)
+- Public GraphQL API (`railway api schema`): `ServiceInstance.tracingEnabled`, `ServiceInstance.autoInstrumentationEnabled` and `ServiceInstanceUpdateInput` as the per-environment path; `Project.tracingEnabled`, `Project.tracingSampleRate`, `Service.tracingEnabled`, `Service.autoInstrumentationEnabled` and their update inputs marked deprecated; the `traces`, `trace` and `tracingStatus` queries
+- Railway MCP (`https://mcp.railway.com`) tool descriptions and schemas: `get-tracing` and `set-service-tracing` with `environmentId`, `describe-service`, `list-traces`, `get-trace`, `get-tracing-coverage`, `get-function-source-code`, `update-function-source-code`
+- CLI source (railwayapp/cli, per-environment `railway trace` and the IaC `tracing` block in #1239): `src/commands/trace.rs` (subcommands, `--all`, `--environment`), `src/iac/compiler.rs` (`services[id].tracing` on service nodes), `src/iac/change_set.rs` (tracing diff, `deployEffect`, removal when the block is absent), `src/commands/config/mod.rs` (pull renderer)
+- Provided variables observed on a traced service's deploy: the five `OTEL_*` variables in the table above and no sampler variables
+- Function runtime: the public `ghcr.io/railwayapp/function-bun:1.4.0` image (Bun 1.4.0, bare imports turned into a `package.json` and installed with `bun install` at every start)
 - Function examples run on Bun 1.4.0 with `@opentelemetry/sdk-node` 0.222.0 and `@hono/otel` 1.1.2 against a stub OTLP receiver: server span continues the incoming `traceparent`, client span propagates it, script flushes on `sdk.shutdown()`

package/dist/global-skills/validate-my-idea/SKILL.md

@@ -5,6 +5,12 @@ metadata:
   vybekiit-generated: buyer-skill-stub
 ---
 
+Prefer the current project instructions and matching project skill when inside a kit app.
+Outside a kit app, resolve guidance relative to this skill directory: ../vybekiitReferences/web/.
+That folder contains language.md, AGENTS.md, CODE-STYLE.md, agent/ and platform-skills/.
+Bare wrapper filenames refer to platform-skills/ there. Do not assume project code or recipe files exist globally.
+Inspect an existing app before changing it. Scaffold a new kit app only when the builder asked for a new app.
+
 <!-- vybekiit:generated:buyer-skill-stub -->
 
 # Skill: validate-my-idea

package/dist/global-skills/vercel-composition-patterns/SKILL.md

@@ -10,6 +10,7 @@ license: MIT
 metadata:
   author: vercel
   version: '1.0.0'
+disable-model-invocation: true
 ---
 
 # React Composition Patterns

package/dist/global-skills/vercel-react-best-practices/SKILL.md

@@ -5,6 +5,7 @@ license: MIT
 metadata:
   author: vercel
   version: "1.0.0"
+disable-model-invocation: true
 ---
 
 # Vercel React Best Practices

package/dist/global-skills/vybekiit-feedback/SKILL.md

@@ -5,6 +5,12 @@ metadata:
   vybekiit-generated: buyer-skill-stub
 ---
 
+Prefer the current project instructions and matching project skill when inside a kit app.
+Outside a kit app, resolve guidance relative to this skill directory: ../vybekiitReferences/web/.
+That folder contains language.md, AGENTS.md, CODE-STYLE.md, agent/ and platform-skills/.
+Bare wrapper filenames refer to platform-skills/ there. Do not assume project code or recipe files exist globally.
+Inspect an existing app before changing it. Scaffold a new kit app only when the builder asked for a new app.
+
 <!-- vybekiit:generated:buyer-skill-stub -->
 
 # Skill: vybekiit-feedback

package/dist/global-skills/vybekiitReferences/backend/AGENTS.md

@@ -0,0 +1,50 @@
+# AGENTS.md — your build agent (read this first)
+
+> **Single source of truth** for Claude Code, Cursor, and Codex. Claude Code and Codex read this
+> file directly; `.cursor/rules` points here. You are talking to a non-technical vibe coder.
+
+Use the plain words in @language.md whenever you talk to the vibe coder.
+
+<!-- vybekiit:generated:start contract -->
+## The contract: Decide + Guide
+
+A request such as "Can people book my classes?" means build the working feature. Inspect the connected provider and run `vybekiit list-presets` before saying the kit lacks the capability. Reuse the booking and organizations presets, add only app-specific behavior, and verify saved bookings and access rules. Read `.vybekiit/agent/session-bootstrap.md` for the persistence workflow. Continue ordinary authorized work without asking the buyer to reconfirm it.
+
+① **One action at a time** — Give the buyer one manual action at a time. Continue authorized work independently, checking each step until the requested work is complete or blocked. Requests to let people book, pay, or join a team mean implement that behavior; do not replace the requested feature with an easier alternative.
+② **Verify before advancing** — Confirm each step actually worked before moving on, so the vibe coder can't get silently stuck. Shared bookings, payments, and team access need real saved information; a preview does not complete them.
+③ **Plain language** — Translate every technical term using language.md — the vibe coder never has to understand or decide.
+④ **Translate errors** — Turn any failure into "what happened + the one thing to do about it" — never paste a raw stack trace.
+⑤ **Celebrate progress** — Celebrate only verified progress. State whether it works locally, in practice, or online.
+⑥ **Record decisions** — After every completing skill, append one entry to checklist.md Decision log via formatChecklistEntry().
+⑦ **Official source fallback** — If MCP or the first debug attempt fails once, run vybekiit doc-fallback and tell the vibe coder the plain stuck phrase only.
+<!-- vybekiit:generated:end contract -->
+
+## How to work
+
+A greeting, a plain "help", or "what is this" in a project whose `checklist.md` Decision log has
+no dated entries starts the onboarding skill (`.vybekiit/skills/onboarding.md`). Other simple
+questions need brief answers, without setup or skill searches. Inspect before making
+project-specific claims. Requests for work mean do the work and verify the requested behavior.
+Reuse instructions already loaded in the conversation. Continue authorized work independently;
+one action at a time limits steps the buyer must take. Prefer this project's skills over global
+copies. Do not use helper agents for simple exchanges.
+
+1. **Find the goal.** Buyer goal skills in `.agents/skills/` (also via `.cursor/skills/` and
+   `.claude/skills/`) load automatically from their descriptions — use the matching skill when the
+   vibe coder's intent fits. If nothing fits or you're unsure, fall back to `.vybekiit/agent/goal-index.md`
+   and read the skill in `.vybekiit/skills/`. Also check `.vybekiit/extensions/goal-index.md` when present.
+   Routine edits need no new skill. Consult `platform-skills/extend-capabilities-vybekiit.md`
+   only for a real uncovered integration.
+2. Use CLI planners: `vybekiit plan-readiness`, `vybekiit plan-setup`, `vybekiit plan-data-model`.
+3. Append routes with `vybekiit backend add-route` or CRUD with `vybekiit backend add-crud`.
+
+## Conventions
+
+- MVC layout: `src/routes/`, `src/controllers/`, `src/middleware/`, `src/services/`.
+- Use `@vybekiit/*` packages — never reinvent auth, db, or payments.
+- Secret settings live in `.env` only — never read values aloud in chat.
+- Settings that are not secret (which services the app uses, public ids, web addresses, ports)
+  live in `settings.env`, saved with the code. A key listed there is set there; secrets and
+  private values stay in `.env`. Never put a secret in `settings.env`. A value the host already
+  sets wins over both files.
+- Session cookies are httpOnly; mobile/extension clients use `APP_URL` pointing here.

package/dist/global-skills/vybekiitReferences/backend/agent/goal-index.md

@@ -0,0 +1,17 @@
+# Goal index — route the builder's request to a skill
+
+| The builder says something like… | Use this skill |
+|---|---|
+| "hi", "help", "what is this", "set up my server", "let's start" | `skills/onboarding.md` |
+| "let people sign in" | `skills/wire-auth.md` |
+| "save my data", "add a database" | `skills/wire-database.md` |
+| "add payments" | `skills/wire-payments.md` |
+| "add an api", "new endpoint" | `skills/add-route.md` |
+| "upload files" | `skills/add-upload.md` |
+| "it's broken", "check my server" | `skills/doctor.md` |
+| "send feedback", "report a kit bug", "the kit is confusing" | `skills/vybekiit-feedback.md` |
+| "update the kit" | `skills/update-kit.md` |
+
+<!-- vybekiit:generated:start goal-index-validation -->
+<!-- Goal drift: run `vybekiit check-goals` — non-zero exit means fix goal-index or add skills -->
+<!-- vybekiit:generated:end goal-index-validation -->