wide_events 0.1.3 → 0.2.1

data/skills/debugging-with-wide-events/SKILL.md

@@ -13,6 +13,17 @@ never needs child spans: filter `main = true` and aggregate.
 
 ## Where the rows live
 
+- **Wide Events store (bundled DuckDB accessory):** the `wide_events` table.
+  `route`, `job_class`, `status`, `error`, `request_id`, `duration_ms`,
+  `kind`, `deployment`, and `source` are promoted columns; everything else
+  the app set is in the `attributes` VARIANT column, read with
+  `attributes['key.name']`. A VARIANT cannot be returned as-is, so cast it
+  to the type you want: `attributes['exception.type']::VARCHAR`,
+  `attributes['account.id']::BIGINT`,
+  `attributes['feature_flag.new_ui']::BOOLEAN`. `IS NULL` works without a
+  cast. Query it with `kamal telemetry` (an interactive prompt, or pipe SQL
+  to it non-interactively) - never SSH in and open the database file
+  directly.
 - **ClickHouse (HyperDX/ClickStack default schema):** the `otel_traces`
   table; wide-event attributes are in the `SpanAttributes` map, resource
   attributes (build, host, environment) in `ResourceAttributes`. Adapt table
@@ -20,12 +31,84 @@ never needs child spans: filter `main = true` and aggregate.
 - **Log sink deployments:** one JSON object per line; query with your log
   store's JSON operators.
 
-Filter by environment when staging and production share storage.
+Filter by environment when staging and production share storage (the
+DuckDB store is single-tenant per deployment, so this mainly applies to the
+ClickHouse/log backends).
 
-## Symptom to query
+## Operational health (DuckDB store)
+
+`SELECT * FROM wide_events_store_status;` is the one-row operational view:
+row counts for the last hour/day, oldest/newest received event, database and
+free-disk bytes, write-readiness and its reason, last checkpoint/retention/
+backup times and errors, and the store/protocol/DuckDB versions. Check this
+first when `bin/rails wide_events:setup:check` reports `degraded`, or before
+trusting a "no events" result - it tells you whether anything has arrived at
+all.
+
+## Standing queries (DuckDB store)
+
+"The reports route is slow":
+
+```sql
+SELECT route,
+       count(*) AS requests,
+       quantile_cont(duration_ms, 0.5) AS p50_ms,
+       quantile_cont(duration_ms, 0.95) AS p95_ms
+FROM wide_events
+WHERE route = 'ReportsController#index'
+  AND occurred_at > current_timestamp - INTERVAL 1 DAY
+GROUP BY route
+ORDER BY p95_ms DESC
+```
+
+"Which jobs are failing":
+
+```sql
+SELECT job_class, count(*) AS failures
+FROM wide_events
+WHERE kind = 'job' AND error
+  AND occurred_at > current_timestamp - INTERVAL 1 DAY
+GROUP BY job_class
+ORDER BY failures DESC
+```
+
+"Unhandled exceptions" - `error` true with no `exception.slug` set is the
+standing "instrument this rescue" query (see instrumenting-wide-events):
+
+```sql
+SELECT route, job_class, attributes['exception.type']::VARCHAR AS exception_type, count(*) AS occurrences
+FROM wide_events
+WHERE error
+  AND coalesce(attributes['exception.slug']::VARCHAR, '') = ''
+  AND occurred_at > current_timestamp - INTERVAL 1 DAY
+GROUP BY route, job_class, exception_type
+ORDER BY occurrences DESC
+```
+
+"Did the deploy change anything" - group by `deployment` (the Kamal version
+string) and compare:
+
+```sql
+SELECT deployment,
+       count(*) AS requests,
+       avg(duration_ms) AS avg_ms,
+       quantile_cont(duration_ms, 0.95) AS p95_ms,
+       sum(error::INT) AS errors
+FROM wide_events
+WHERE route = 'ReportsController#index'
+  AND occurred_at > current_timestamp - INTERVAL 1 DAY
+GROUP BY deployment
+ORDER BY deployment DESC
+```
+
+"Why did this specific request fail" - filter by `request_id` and read
+every promoted column and attribute of the one row before reaching for logs.
+
+## Symptom to query (ClickHouse)
 
 Start from the unit of work, not from logs. One GROUP BY usually replaces a
-cross-tool investigation.
+cross-tool investigation. (On the DuckDB store, use the equivalent queries
+above instead - same symptoms, promoted columns instead of `SpanAttributes`.)
 
 "The reports route is slow":
 
@@ -52,7 +135,7 @@ LIMIT 20
 `SpanAttributes['http.request.id']` and read every attribute of the one row
 before reaching for logs or the trace waterfall.
 
-## Standing queries
+## Standing queries (ClickHouse)
 
 - **Unnamed exception paths**: `SpanAttributes['error'] = 'true' AND
   SpanAttributes['exception.slug'] = '' AND SpanAttributes['exception.type'] != ''`:
@@ -69,6 +152,8 @@ before reaching for logs or the trace waterfall.
 ## Verifying your own fix
 
 After a deploy, don't ask a human to check a dashboard. Query the affected
-route/job filtered to the new `service.version` and compare against the
-prior build: error rate, p50, query count, and the specific attribute your
-change was supposed to move. State the numbers in your summary.
+route/job filtered to the new build (`deployment` on the DuckDB store,
+`ResourceAttributes['service.version']` on ClickHouse) and compare against
+the prior build: error rate, p50/p95, query count, and the specific
+attribute your change was supposed to move. State the numbers in your
+summary.

data/skills/setting-up-wide-events-store/SKILL.md

@@ -0,0 +1,120 @@
+---
+name: setting-up-wide-events-store
+description: Use when a human asks to set up, deploy, or finish deploying the bundled Wide Events DuckDB store (the Kamal accessory that receives production telemetry). Covers the store generator, DNS, `kamal setup`, and post-deploy verification end to end.
+---
+
+# Setting up the Wide Events store
+
+This is the guided path from "wide_events is installed" to "production
+requests are queryable." It wraps deterministic generator/checker code -
+never hand-edit `config/deploy.yml` or `.kamal/secrets` yourself; every
+mutation here is idempotent, so re-running any step (including this whole
+skill) is always safe.
+
+## What to ask for
+
+Exactly three things, nothing else:
+
+1. the SSH address (IP or DNS name) of the VM that will run the store;
+2. the public DNS hostname end users/agents will query through (no scheme,
+   no path); and
+3. retention in days (default 30 if the human has no preference).
+
+## Steps
+
+1. **Generate the accessory.**
+
+   ```bash
+   bin/rails generate wide_events:store --host=<ssh-address> --hostname=<telemetry-hostname> --retention-days=<n>
+   ```
+
+   This edits `config/deploy.yml` (accessory, app environment, `telemetry`
+   and `wide-events-backup` aliases) and writes durable, gitignored tokens to
+   `.kamal/wide-events-ingest-token` and `.kamal/wide-events-query-token`. If
+   it refuses instead of editing (ambiguous YAML, a conflicting existing
+   key), it prints a complete snippet - merge that by hand, then continue.
+
+2. **Show the exact DNS record.** Tell the human to point the telemetry
+   hostname at the store host now, before deploying, so propagation happens
+   in parallel with the next steps instead of after them.
+
+3. **Run the checker.**
+
+   ```bash
+   bin/rails wide_events:setup:check
+   ```
+
+   This is the same idempotent command at every stage - before the first
+   deploy, while waiting on DNS, and after `kamal setup` - so there is never
+   a different command to remember. It reports one of five states:
+
+   - `failed` - a Kamal version/config problem, or a host-directory
+     permission problem. Fix what the message says (it includes the exact
+     one-time administrator command if root is needed to create the data
+     directory) and re-run.
+   - `needs_dns` - DNS has not resolved yet. **This is expected and
+     resumable, not an error.** Nothing generated is reverted or lost;
+     re-running the same command later picks up exactly where it left off.
+     Wait for propagation and run it again.
+   - `ready_to_deploy` - DNS is good and the store isn't answering yet.
+     Proceed to step 4.
+   - `degraded` - the store is routable but not accepting writes (usually
+     low disk space). Do not proceed; investigate on the host.
+   - `healthy` - already fully deployed and working (useful if you're
+     re-running this skill against an existing setup).
+
+4. **Deploy, once the checker says `ready_to_deploy`.**
+
+   ```bash
+   kamal setup
+   ```
+
+5. **Re-run the checker.** `bin/rails wide_events:setup:check` again. On a
+   healthy deploy it sends one synthetic event, queries it back by ID, and
+   prints that row plus a route-count/p95-latency query - proof the whole
+   path (ingest token, query token, TLS, DuckDB) works before any real
+   traffic exists.
+
+6. **Ask the human to exercise one real route** of the app now that the
+   store is live (any request or job is fine).
+
+7. **After they confirm, query it back:**
+
+   ```bash
+   kamal telemetry
+   ```
+
+   opens an interactive SQL prompt against the store (or pipe SQL to it
+   non-interactively). Run a query for the most recent real request, e.g.:
+
+   ```sql
+   SELECT id, occurred_at, route, duration_ms, status
+   FROM wide_events
+   ORDER BY occurred_at DESC
+   LIMIT 1;
+   ```
+
+   Display that captured row, then the route count and p95 latency for the
+   exercised route, and state both numbers back to the human.
+
+## Things to tell the human, unprompted
+
+- **DNS timeouts are resumable.** If `wide_events:setup:check` reports
+  `needs_dns`, that is normal while waiting on propagation, not a failure -
+  running the exact same command again later continues correctly.
+- **Backups are local-only.** The store itself takes a verified local backup
+  once a day in-process (three most recent kept, in `backups/` next to the
+  database); `kamal wide-events-backup` is the on-demand version of the same
+  thing and prints the path it wrote. Either way this protects against
+  logical mistakes and bad upgrades, not loss of the VM or volume itself.
+  There is no off-host backup in v1.
+- **Query access uses a separate token from ingestion.** `.kamal/wide-events-
+  query-token` is read-only and distinct from `.kamal/wide-events-ingest-
+  token`; sharing query access (e.g. with a teammate or another agent) never
+  requires sharing the ability to write events.
+
+## Where full querying guidance lives
+
+Once the store is live, use the `debugging-with-wide-events` skill for the
+standing queries (slow routes, job failures, unhandled exceptions, deploy
+comparisons) and the `wide_events_store_status` operational view.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wide_events
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.2.1
 platform: ruby
 authors:
 - Adam Miribyan
@@ -37,11 +37,26 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.2'
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 'Collects everything a Rails app knows about each request or job execution
   into one flat, high-cardinality event: route, user, build, query counts, cache hits,
-  feature flags, phase timings, errors. Flushes onto the OpenTelemetry root span or
-  a JSON log line. Ships a checkable attribute registry, test helpers, and agent skills
-  for writing and querying instrumentation.'
+  feature flags, phase timings, errors. Ships a bundled DuckDB store you deploy as
+  a Kamal accessory (one generator, then SQL against production), or flushes onto
+  the OpenTelemetry root span or a JSON log line instead. Includes a checkable attribute
+  registry, test helpers, and agent skills for instrumenting, querying, and deploying.'
 email:
 - adam@adam.am
 executables: []
@@ -55,25 +70,41 @@ files:
 - lib/generators/wide_events/install/templates/agents_md_section.md
 - lib/generators/wide_events/install/templates/initializer.rb
 - lib/generators/wide_events/install/templates/registry.yml
+- lib/generators/wide_events/skill_installer.rb
 - lib/generators/wide_events/skills/skills_generator.rb
+- lib/generators/wide_events/store/store_generator.rb
+- lib/generators/wide_events/store/templates/accessory.yml.erb
 - lib/wide_event.rb
 - lib/wide_event/configuration.rb
 - lib/wide_event/job_instrumentation.rb
+- lib/wide_event/kamal/deploy_editor.rb
+- lib/wide_event/kamal/secrets_editor.rb
 - lib/wide_event/middleware.rb
 - lib/wide_event/railtie.rb
 - lib/wide_event/registry.rb
 - lib/wide_event/registry/defaults.yml
+- lib/wide_event/setup/checker.rb
+- lib/wide_event/setup/command_runner.rb
 - lib/wide_event/sinks/log_line.rb
 - lib/wide_event/sinks/memory.rb
 - lib/wide_event/sinks/otel_span.rb
+- lib/wide_event/sinks/store.rb
 - lib/wide_event/span_counter_processor.rb
+- lib/wide_event/store/client.rb
+- lib/wide_event/store/envelope.rb
+- lib/wide_event/store/formatter.rb
+- lib/wide_event/store/query_result.rb
+- lib/wide_event/store/sender.rb
 - lib/wide_event/subscribers.rb
 - lib/wide_event/tasks/registry.rake
+- lib/wide_event/tasks/setup.rake
+- lib/wide_event/tasks/store.rake
 - lib/wide_event/test_helper.rb
 - lib/wide_event/version.rb
 - lib/wide_events.rb
 - skills/debugging-with-wide-events/SKILL.md
 - skills/instrumenting-wide-events/SKILL.md
+- skills/setting-up-wide-events-store/SKILL.md
 homepage: https://github.com/adammiribyan/wide_events
 licenses:
 - MIT
@@ -100,6 +131,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 rubygems_version: 3.6.9
 specification_version: 4
-summary: One wide telemetry event per request or job, on the OTel span you already
-  export.
+summary: One wide telemetry event per Rails request or job, in a database you own.
 test_files: []