e11y 0.2.0 → 1.1.0

data/docs/use_cases/backlog.md

@@ -12,92 +12,20 @@ This document captures promising ideas for future versions of `e11y` that are no
 
 ---
 
-## 1. Quick Start Presets
+## Status: What's Already Done (as of v0.2.0)
 
-### Problem
-
-Configuration complexity can be overwhelming for new users. Setting up production-ready configuration requires understanding multiple subsystems (sampling, compression, retention, payload optimization).
-
-### Proposal
-
-Provide pre-configured profiles for common scenarios:
-
-```ruby
-E11y.configure do |config|
-  # Option 1: Use a preset
-  config.use_preset :production_high_traffic
-  
-  # Option 2: Use a preset with overrides
-  config.use_preset :production_high_traffic do |preset|
-    preset.sampling.max_events_per_sec = 5_000  # Override default
-  end
-end
-```
-
-### Available Presets
-
-| Preset | Description | Sample Rate | Compression | Retention |
-|--------|-------------|-------------|-------------|-----------|
-| `:development` | Local dev, no sampling | 100% | None | 1 day |
-| `:staging` | Pre-prod testing | 50% | Gzip | 7 days |
-| `:production_low_traffic` | < 1K events/sec | 80% | Gzip | 30 days |
-| `:production_high_traffic` | > 10K events/sec | 10% | Zstd | 7 days hot, 90 days warm |
-| `:production_cost_optimized` | Aggressive cost reduction | 5% | Zstd level 9 | 3 days hot, 30 days warm |
-
-### Implementation Example
-
-```ruby
-module E11y
-  module Presets
-    PRODUCTION_HIGH_TRAFFIC = {
-      adaptive_sampling: {
-        enabled: true,
-        load_based: { max_events_per_sec: 10_000 },
-        error_based: { enabled: true },
-        value_based: {
-          high_value_patterns: [/^payment\./, /^order\./, /^error\./],
-          low_value_patterns: [/^debug\./, /^health_check/]
-        }
-      },
-      compression: {
-        enabled: true,
-        algorithm: :zstd,
-        level: 3
-      },
-      retention_tagging: {
-        enabled: true,
-        default_retention: 7.days,
-        retention_by_pattern: {
-          'audit.*' => 7.years,
-          'payment.*' => 1.year,
-          'debug.*' => 1.day
-        }
-      },
-      payload_minimization: {
-        enabled: true,
-        truncate_strings_at: 1000,
-        truncate_arrays_at: 100,
-        remove_null_fields: true
-      }
-    }.freeze
-  end
-end
-```
-
-### Benefits
-
-- ✅ Faster onboarding (< 5 minutes to production-ready config)
-- ✅ Best practices baked in
-- ✅ Easy to customize (override specific settings)
-- ✅ Reduces configuration errors
-
-### Priority
-
-**Medium (v1.1)**
+| Backlog Item | Status | Notes |
+|--------------|--------|-------|
+| **§1 Sampling Budget** | ❌ Not done | |
+| **§2 Global Async** | ❌ Not done | |
+| **§3 Ring Buffer** | ⚠️ Implemented, not integrated | `E11y::Buffers::RingBuffer` exists; Loki uses `[]` + Mutex |
+| **§4 Multi-Tenant** | ⚠️ Partial | Loki adapter has `tenant_id` (X-Scope-OrgID); baggage allows `tenant` key. No full multi-tenant isolation |
 
 ---
 
-## 2. Sampling Budget
+## 1. Sampling Budget
+
+> **Related (implemented):** Adaptive sampling (error-spike, load-based), value-based (`sample_by_value`), per-event `sample_rate`. No budget/cap, no Redis state.
 
 ### Problem
 
@@ -196,15 +124,113 @@ monthly_cost = $1,000 * 30 = $30,000/month
 
 ---
 
-## 3. Additional Ideas (Placeholder)
+## 2. Global Async Mode and Queue
+
+### Problem
+
+Current pipeline is **synchronous**: `Event.track() → Pipeline → Routing → Adapter.write()`. Batching happens only inside individual adapters (Loki, OTel). There is no global async layer between `track()` and adapters.
+
+The [01-SCALE-REQUIREMENTS.md](../prd/01-SCALE-REQUIREMENTS.md) document specifies a global async configuration that is **not implemented**:
+
+```ruby
+# Specified but NOT implemented
+E11y.configure do |config|
+  config.async do
+    queue_size 10_000        # 10k events buffer
+    batch_size 500           # Moderate batching
+    flush_interval 200       # ms
+    worker_threads 1         # Single worker
+  end
+end
+```
+
+### Missing Self-Monitoring Metrics
+
+The scale requirements also specify metrics that are not wired:
+
+- `E11y.stats.drops_total` — total dropped events
+- `E11y.stats.events_processed_total` — total events processed
+- `e11y_internal_queue_utilization_ratio` — buffer fill level (0–1)
+- `e11y_internal_queue_size` / `e11y_internal_queue_capacity`
+
+### Proposal
+
+1. **Add global async config** — optional `config.async` block with `queue_size`, `batch_size`, `flush_interval`, `worker_threads`
+2. **Use `RingBuffer`** — `E11y::Buffers::RingBuffer` exists but is unused; it could be the backing store for the global queue
+3. **Worker thread(s)** — background consumer that pops from `RingBuffer` and writes to adapters in batches
+4. **Self-monitoring** — expose `e11y_internal_*` metrics for queue health, drops, throughput
+
+### Architecture Options
+
+| Option | Description | Complexity |
+|--------|-------------|------------|
+| **A: Keep current** | Batching stays in adapters only | None |
+| **B: Global queue** | Single RingBuffer before routing, worker threads | Medium |
+| **C: Per-adapter queues** | Each adapter has its own RingBuffer + worker | High |
+
+### Benefits
+
+- ✅ Decouples `track()` from I/O (network, disk)
+- ✅ Predictable latency under load (<1ms p99 for track)
+- ✅ Backpressure handling (drop_oldest when full)
+- ✅ Self-monitoring (queue utilization, drops)
+
+### Trade-offs
+
+- ⚠️ Complexity (thread management, shutdown)
+- ⚠️ Data loss window (events in buffer on crash)
+- ⚠️ Memory overhead (buffer capacity × avg event size)
+
+### Priority
+
+**Medium (v1.1)** — Required for small teams at scale; optional for MVP
+
+---
+
+## 3. In-Memory Ring Buffer (Integration)
+
+### Problem
+
+`E11y::Buffers::RingBuffer` is **fully implemented** (`lib/e11y/buffers/ring_buffer.rb`) but **not used** anywhere in the pipeline:
+
+- **Loki adapter** uses `@buffer = []` + `Mutex` (not RingBuffer)
+- **EphemeralBuffer** uses `Concurrent::Array` in `Thread.current`
+- **RingBuffer** exists only in unit tests and benchmarks
+
+PRD Phase 1 (00-ICP-AND-TIMELINE.md) specifies "In-memory ring buffer (SPSC)" as a Week 1–2 deliverable, but it was never integrated.
+
+### RingBuffer Spec
+
+- Lock-free SPSC (Single-Producer, Single-Consumer)
+- 100K events capacity (default)
+- Overflow strategies: `:drop_oldest`, `:drop_newest`, `:block`
+- Target: 100K+ events/sec, <10μs p99 per push/pop
+
+### Proposal
+
+1. **Option A: Integrate into global async** — Use RingBuffer as the backing store when `config.async` is enabled (see §2)
+2. **Option B: Replace Loki buffer** — Swap `[]` + Mutex for RingBuffer in Loki adapter for better throughput
+3. **Option C: Remove from PRD** — Document that RingBuffer is "future-ready" for async mode; adapter-level batching is sufficient for MVP
+
+### Recommendation
+
+**Option A** — RingBuffer is the natural choice for global async queue. Integrate when implementing §2 (Global Async Mode).
+
+### Priority
+
+**Medium (v1.1)** — Depends on §2 (Global Async Mode)
+
+---
+
+## 4. Additional Ideas (Placeholder)
 
 Future ideas to be added:
 
 - **ML-Based Anomaly Detection:** Automatically detect unusual patterns in events
 - **Event Replay from Storage:** Replay events from cold storage for debugging
-- **Multi-Tenant Support:** Isolate events by tenant/customer
-- **Event Transformation Rules:** Transform events before sending to adapters
-- **Custom Retention Policies:** More granular control over data lifecycle
+- **Multi-Tenant Support:** Isolate events by tenant/customer — *Loki has `tenant_id`; baggage allows `tenant`*
+- **Event Transformation Rules:** Transform events before sending to adapters — *routing + PII filter exist*
+- **Custom Retention Policies:** More granular control over data lifecycle — *`retention_period` per event exists*
 
 ---
 
@@ -217,7 +243,13 @@ Future ideas to be added:
 
 ## Related ADRs
 
-- [ADR-009: Cost Optimization](../ADR-009-cost-optimization.md) - Current implementation
+- [ADR-009: Cost Optimization](../architecture/ADR-009-cost-optimization.md) - Current implementation
+- [ADR-001: Architecture](../architecture/ADR-001-architecture.md) - RingBuffer specification (§3.3.1)
+
+## Related Documents
+
+- [01-SCALE-REQUIREMENTS.md](../prd/01-SCALE-REQUIREMENTS.md) - Specifies `config.async`, self-monitoring metrics
+- [00-ICP-AND-TIMELINE.md](../prd/00-ICP-AND-TIMELINE.md) - Phase 1 "In-memory ring buffer (SPSC)"
 
 ---
 

data/e11y.gemspec

@@ -17,7 +17,7 @@ Gem::Specification.new do |spec|
     • Schema-validated events - catch bugs before production with dry-schema
 
     DEVELOPER EXPERIENCE:
-    • 5-minute setup (not 2-week migration)
+    • Minimal setup — one config block, works with stdout out of the box
     • Auto-metrics from events (no manual Yabeda.increment)
     • Rails-first design (follows Rails conventions)
     • Pluggable adapters (Loki, Sentry, OpenTelemetry, custom backends)
@@ -73,7 +73,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "rspec", "~> 3.12"
   spec.add_development_dependency "rubocop", "~> 1.50"
   spec.add_development_dependency "rubocop-rake", "~> 0.6"
-  spec.add_development_dependency "rubocop-rspec", "~> 2.22"
+  spec.add_development_dependency "rubocop-rspec", "~> 3.0"
   spec.add_development_dependency "simplecov", "~> 0.22"
   spec.add_development_dependency "webmock", "~> 3.19" # For HTTP adapter testing
   spec.add_development_dependency "yard", "~> 0.9"

data/gems/e11y-devtools/README.md

@@ -0,0 +1,158 @@
+# e11y-devtools
+
+Developer tools for [e11y](https://github.com/aseletskiy/e11y) — the Rails observability gem.
+
+Three complementary viewers for the same JSONL log:
+
+| Viewer | How to use |
+|--------|-----------|
+| **TUI** (terminal) | `bundle exec e11y` |
+| **Browser Overlay** | Automatic in development — floating badge in bottom-right |
+| **MCP Server** | `bundle exec e11y mcp` — AI integration for Cursor / Claude Code |
+
+## Installation
+
+Add to your Gemfile (development group only):
+
+```ruby
+# Gemfile
+gem "e11y", "~> 1.0"
+gem "e11y-devtools", "~> 0.1.0", group: :development
+```
+
+Then run `bundle install`.
+
+## TUI — Interactive Log Viewer
+
+```bash
+bundle exec e11y           # Open TUI (default)
+bundle exec e11y tui       # Same as above
+bundle exec e11y tail      # Stream events to stdout
+bundle exec e11y help      # Show help
+```
+
+### TUI Keyboard Shortcuts
+
+| Key | Action |
+|-----|--------|
+| `↑` / `k` | Move up |
+| `↓` / `j` | Move down |
+| `Enter` | Drill in (interactions → events → detail) |
+| `Esc` / `b` | Go back |
+| `w` | Filter: web requests only |
+| `j` | Filter: background jobs only |
+| `a` | Filter: all sources |
+| `r` | Reload manually |
+| `c` | Copy event JSON to clipboard (in detail view) |
+| `q` | Quit |
+
+## Browser Overlay
+
+When `gem "e11y-devtools"` is in your Gemfile, the overlay appears automatically in development (Rails mounts `E11y::Devtools::Overlay::Engine` at `/_e11y`).
+
+The UI is a **Svelte** bundle (`overlay.js`) with:
+
+- **FAB** (bottom-right): total events, warn/error counts; brief **pulse** when **new** `warn` or `error`/`fatal` rows appear (not on every poll).
+- **Fullscreen panel**: tabs **Problems** (error/fatal from recent buffer, 1-click → JSON) and **Interactions** (grouped traces like TUI). If there are errors in the recent snapshot, the panel opens on **Problems** first.
+- **Wide viewport** (~900px+): interactions use a **split view** (list left, events right); narrow screens keep the stacked flow.
+- Source filter chips (web / job / all) apply to **Interactions** only.
+- **JSON API** under `/_e11y/v1/`: `GET interactions`, `GET traces/:trace_id/events`, `GET events/recent`.
+
+### Rebuild the overlay asset
+
+After changing `gems/e11y-devtools/frontend/`:
+
+```bash
+cd gems/e11y-devtools/frontend && npm install && npm run build
+```
+
+Output: `lib/e11y/devtools/overlay/assets/overlay.js` (served at `/_e11y/overlay.js`).
+
+### Local UI prototype (mocks)
+
+```bash
+cd gems/e11y-devtools/frontend && npm run dev
+```
+
+Uses `public/mocks/v1/*.json` — see `frontend/README.md`.
+
+## MCP Server — AI Integration
+
+Start the server:
+
+```bash
+# stdio (for Cursor / Claude Code)
+bundle exec e11y mcp
+
+# HTTP (for direct integration)
+bundle exec e11y mcp --port 3099
+```
+
+Add to `.cursor/mcp.json` or `~/.claude/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "e11y": {
+      "command": "bundle",
+      "args": ["exec", "e11y", "mcp"],
+      "cwd": "/path/to/your/rails/app"
+    }
+  }
+}
+```
+
+### Available MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `recent_events` | Get latest N events (filterable by severity) |
+| `events_by_trace` | Get all events for a trace ID |
+| `search` | Full-text search across event names and payloads |
+| `stats` | Aggregate statistics (total, by severity, oldest/newest) |
+| `interactions` | Time-grouped interactions (parallel requests) |
+| `event_detail` | Full payload for a single event by ID |
+| `errors` | Recent error/fatal events only — fastest way to see what went wrong |
+| `clear` | Clear the dev log |
+
+## Configuration
+
+```ruby
+# config/initializers/e11y.rb
+E11y.configure do |config|
+  config.register_adapter :dev_log, E11y::Adapters::DevLog.new(
+    path:         Rails.root.join("log", "e11y_dev.jsonl"),
+    max_size:     ENV.fetch("E11Y_MAX_SIZE", 50).to_i * 1024 * 1024,  # 50 MB
+    max_lines:    ENV.fetch("E11Y_MAX_EVENTS", 10_000).to_i,
+    keep_rotated: ENV.fetch("E11Y_KEEP_ROTATED", 5).to_i
+  )
+end
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `E11Y_MAX_EVENTS` | `10000` | Max lines before rotation |
+| `E11Y_MAX_SIZE` | `50` | Max log size in MB before rotation |
+| `E11Y_KEEP_ROTATED` | `5` | Number of compressed `.gz` files to keep |
+
+## Log Format
+
+Events are stored as JSONL (one JSON object per line) at `log/e11y_dev.jsonl`.
+Rotated files are numbered and gzip-compressed: `e11y_dev.jsonl.1.gz`, `.2.gz`, etc.
+
+## Architecture
+
+```
+log/e11y_dev.jsonl  ←  E11y::Adapters::DevLog (write)
+         ↓
+E11y::Adapters::DevLog::Query (read, cache, search, grouping)
+         ↓
+   ┌─────┴──────┬──────────────┬──────────────┐
+   TUI          Browser        MCP Server
+(ratatui_ruby)  Overlay       (gem 'mcp')
+                (Rack)
+```
+
+The JSONL file is the single source of truth. All viewers are stateless readers — they never write.

data/gems/e11y-devtools/config/routes.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+E11y::Devtools::Overlay::Engine.routes.draw do
+  get    "overlay.js",    to: "e11y/devtools/overlay/rails#overlay_js"
+  get    "events",        to: "e11y/devtools/overlay/rails#events"
+  get    "events/recent", to: "e11y/devtools/overlay/rails#recent"
+  delete "events",        to: "e11y/devtools/overlay/rails#clear"
+  get    "stats",         to: "e11y/devtools/overlay/rails#stats"
+
+  scope "v1" do
+    get "interactions", to: "e11y/devtools/overlay/rails#v1_interactions"
+    get "traces/:trace_id/events", to: "e11y/devtools/overlay/rails#v1_trace_events"
+    get "events/recent", to: "e11y/devtools/overlay/rails#v1_events_recent"
+  end
+end

data/gems/e11y-devtools/e11y-devtools.gemspec

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require_relative "lib/e11y/devtools/version"
+
+Gem::Specification.new do |spec|
+  spec.name    = "e11y-devtools"
+  spec.version = E11y::Devtools::VERSION
+  spec.authors = ["Artur Seletskiy"]
+  spec.summary = "Developer tools for E11y: TUI, Browser Overlay, MCP Server"
+
+  spec.required_ruby_version = ">= 3.2"
+
+  spec.files = Dir["lib/**/*.rb", "lib/**/*.js", "config/**/*.rb", "exe/*", "*.md"]
+  spec.bindir        = "exe"
+  spec.executables   = ["e11y"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "e11y", "~> #{E11y::Devtools::CORE_VERSION}"
+  spec.add_dependency "mcp",          ">= 0.8"
+  spec.add_dependency "ratatui_ruby", "~> 1.4"
+
+  # Optional but recommended for performance
+  spec.add_development_dependency "oj"
+  spec.metadata["rubygems_mfa_required"] = "true"
+end

data/gems/e11y-devtools/exe/e11y

@@ -0,0 +1,34 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "e11y/devtools"
+
+command = ARGV.shift || "tui"
+
+case command
+when "tui"
+  require "e11y/devtools/tui/app"
+  E11y::Devtools::Tui::App.new.run
+when "mcp"
+  require "e11y/devtools/mcp/server"
+  E11y::Devtools::Mcp::Server.new.run(
+    transport: ARGV.include?("--port") ? :http : :stdio,
+    port: (ARGV[ARGV.index("--port") + 1] if ARGV.include?("--port"))&.to_i
+  )
+when "tail"
+  require "e11y/devtools/tui/tail"
+  E11y::Devtools::Tui::Tail.new.run
+when "help", "--help", "-h"
+  puts <<~HELP
+    bundle exec e11y [command]
+
+    Commands:
+      tui   (default)  Interactive TUI — browse events and traces
+      mcp              MCP server for Cursor / Claude Code AI integration
+      tail             Stream new events to stdout (pipe-friendly)
+      help             Show this help
+  HELP
+else
+  warn "Unknown command: #{command}. Run `bundle exec e11y help`."
+  exit 1
+end

data/gems/e11y-devtools/frontend/.gitignore

@@ -0,0 +1,24 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/*
+!.vscode/extensions.json
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?

data/gems/e11y-devtools/frontend/README.md

@@ -0,0 +1,51 @@
+# e11y browser overlay — frontend (Svelte)
+
+Prototype and production bundle for the dev-only overlay injected into Rails apps at `/_e11y/overlay.js`.
+
+## API mocks (Phase 1)
+
+Static JSON mirrors the target `/_e11y/v1/` contract:
+
+| Path | File |
+|------|------|
+| `GET /v1/events/recent` | `public/mocks/v1/events/recent.json` |
+| `GET /v1/interactions` | `public/mocks/v1/interactions.json` |
+| `GET /v1/traces/:trace_id/events` | `public/mocks/v1/traces/<trace_id>/events.json` |
+
+Trace ids are 32-char hex strings (like `DevLog` / OpenTelemetry). Files live under `public/mocks/v1/traces/<trace_id>/events.json`.
+
+**Regenerate mocks** (deterministic `Random.new(42)`):
+
+```bash
+npm run generate-mocks
+# or: E11Y_MOCK_TRACES=52 ruby scripts/generate_mocks.rb   # stress (~300+ events)
+```
+
+Default is `E11Y_MOCK_TRACES=24` (~150 events). `interactions.json` is rebuilt using the same 500ms window grouping as `E11y::Adapters::DevLog::Query`.
+
+**Overlay UX (dev):** Problems tab includes a stacked **volume bar** over `recent` (UTC time scale, drag to **brush** a range and narrow the error list, double-click or **Clear range** to reset), search, and expandable payload rows. Trace event lists support **severity + text filters**, inline **payload expand**, **±2 context** highlight after opening an event and going back, and detail **Copy trace_id / request_id** plus collapsible payload / metadata / full JSON.
+
+## Setup
+
+```bash
+cd gems/e11y-devtools/frontend
+npm install
+```
+
+## Dev server
+
+```bash
+npm run dev
+```
+
+Open the URL Vite prints (default `http://localhost:5173`) to load `index.html` with a fake host page and the overlay.
+
+## Production build
+
+After the Svelte app is scaffolded (plan Task 2+):
+
+```bash
+npm run build
+```
+
+Output is written to `../lib/e11y/devtools/overlay/assets/overlay.js` for the Rails engine to serve.

data/gems/e11y-devtools/frontend/index.html

@@ -0,0 +1,14 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>e11y overlay (dev)</title>
+  </head>
+  <body>
+    <p style="padding: 1rem; font-family: system-ui">Demo host page — overlay FAB bottom-right.</p>
+    <div id="app" class="e11y-demo-host"></div>
+    <script type="module" src="/src/main.ts"></script>
+  </body>
+</html>