tomo-ai 0.8.11 → 0.8.12

package/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.8.12 (2026-07-12)
+
+### Features
+
+- **Prometheus metrics + activity log with Grafana stack** (#238). Two new `WatchBus` subscribers, gated by `config.metrics` (default off): a `prom-client` exporter on `127.0.0.1:9464` (turns, cost, per-session context, tool calls, cron, heartbeats, compactions) and an NDJSON activity log at `~/.tomo/logs/activity.ndjson` for Loki tailing (transcript text local-only, redactable via `metrics.includeMessageText=false`). `contrib/observability/` ships a docker-compose stack (Prometheus, Loki, Alloy, Grafana) with a provisioned Tomo dashboard.
+
+### Dependencies
+
+- Bump `@anthropic-ai/claude-agent-sdk` 0.3.201 → 0.3.207 (#237).
+- Bump the dev-dependencies group (#236). The TypeScript ^6 → ^7 bump in that group was reverted — no `typescript-eslint` release yet supports TypeScript 7 (peer range caps at `<6.1.0`), so it broke `npm ci`.
+
 ## 0.8.11 (2026-07-09)
 
 ### Features

package/contrib/observability/README.md

@@ -0,0 +1,90 @@
+# Tomo observability stack
+
+A self-contained Grafana + Prometheus + Loki + Alloy stack for watching Tomo
+from a dashboard. Data never leaves the machine: Prometheus, Loki, and Alloy
+bind to `127.0.0.1` only. Grafana is reachable from the local network (login
+required) so you can open the dashboard from another computer.
+
+```
+Tomo daemon ──/metrics (127.0.0.1:9464)──▶ Prometheus ──┐
+     │                                                  ├──▶ Grafana (127.0.0.1:3000)
+     └──~/.tomo/logs/activity.ndjson──▶ Alloy ──▶ Loki ─┘
+```
+
+## Setup
+
+1. Enable metrics in `~/.tomo/config.json`:
+
+   ```json
+   {
+     "metrics": { "enabled": true }
+   }
+   ```
+
+   Then restart the daemon (`tomo restart`) and verify:
+
+   ```bash
+   curl -s localhost:9464/metrics | grep tomo_
+   ```
+
+2. Copy the stack somewhere you own, then start it (requires Docker Desktop).
+   Copying matters: run it in place and an `npm upgrade` will overwrite any
+   compose/dashboard customizations.
+
+   ```bash
+   # from a git clone:
+   cp -r contrib/observability ~/tomo-observability
+   # or from a global npm install:
+   cp -r "$(npm root -g)/tomo-ai/contrib/observability" ~/tomo-observability
+
+   cd ~/tomo-observability
+   docker compose up -d
+   ```
+
+3. Open <http://localhost:3000> — or from another computer on your network,
+   `http://<hostname>.local:3000` (e.g. `http://mac-mini.local:3000`). Sign in
+   with `admin` / `admin`; Grafana forces a password change on first login.
+   The **Tomo** dashboard is pre-provisioned.
+
+   To restrict Grafana to this machine only, change its port mapping back to
+   `"127.0.0.1:3000:3000"` in `docker-compose.yml`. For access from *outside*
+   your home network, don't port-forward — use Tailscale or similar.
+
+## Config reference
+
+`~/.tomo/config.json` → `metrics` (env overrides in parens):
+
+| Key | Default | Meaning |
+| --- | --- | --- |
+| `enabled` (`TOMO_METRICS`) | `false` | Serve `/metrics` and write the activity log |
+| `port` (`TOMO_METRICS_PORT`) | `9464` | Prometheus exporter port, loopback-only |
+| `activityLog` | `true` | Write `~/.tomo/logs/activity.ndjson` for Loki |
+| `includeMessageText` | `true` | Include transcript text in the activity log. Disable if you ever ship the log off this machine. |
+
+## What's collected
+
+**Prometheus** (`tomo_*`, plus standard `process_*`/`nodejs_*`): turns by
+source/outcome, turn duration histogram, cumulative cost (USD), per-session
+context usage, tool calls, cron runs + upcoming cron schedule, compactions,
+heartbeats (count, last, next), warn/error issues.
+
+**Loki** (`{job="tomo-activity"}`): every watch-bus event as structured JSON —
+the same feed `tomo watch` renders. Labels: `type`, `session`, `level`.
+Example queries:
+
+```logql
+{job="tomo-activity", type="tool.start"}                # every tool call
+{job="tomo-activity", type="transcript", session="dm:merlin"}
+{job="tomo-activity"} | json | costUsd > 0.10           # expensive turns
+```
+
+## Notes
+
+- Prometheus keeps 90 days of metrics; Loki uses its default retention.
+  Volumes (`prometheus-data`, `loki-data`, `grafana-data`) survive
+  `docker compose down`; add `-v` to wipe them.
+- Counters reset when the daemon restarts — dashboard queries use
+  `increase()`/`rate()`, which handle resets.
+- If `TOMO_HOME` is not `~/.tomo`, fix the Alloy volume mount in
+  `docker-compose.yml`.
+- History starts when you enable the stack; nothing is backfilled.

package/contrib/observability/alloy/config.alloy

@@ -0,0 +1,53 @@
+// Tail Tomo's NDJSON activity feed into Loki. The glob also matches rotated
+// files (activity.ndjson.1, …) so lines written just before a rotation are
+// still picked up if the shipper lags.
+
+local.file_match "tomo_activity" {
+	path_targets = [{ __path__ = "/var/log/tomo/activity.ndjson*" }]
+}
+
+loki.source.file "tomo_activity" {
+	targets    = local.file_match.tomo_activity.targets
+	forward_to = [loki.process.tomo.receiver]
+}
+
+loki.process "tomo" {
+	stage.json {
+		expressions = {
+			type    = "type",
+			session = "sessionKey",
+			level   = "level",
+			ts      = "ts",
+		}
+	}
+
+	// Low-cardinality labels only: event type, session key, issue level.
+	// Everything else stays in the JSON body and is queryable via `| json`.
+	stage.labels {
+		values = {
+			type    = "",
+			session = "",
+			level   = "",
+		}
+	}
+
+	stage.static_labels {
+		values = {
+			job = "tomo-activity",
+		}
+	}
+
+	// Use the event's own millisecond timestamp, not ingestion time.
+	stage.timestamp {
+		source = "ts"
+		format = "UnixMs"
+	}
+
+	forward_to = [loki.write.local.receiver]
+}
+
+loki.write "local" {
+	endpoint {
+		url = "http://loki:3100/loki/api/v1/push"
+	}
+}

package/contrib/observability/docker-compose.yml

@@ -0,0 +1,70 @@
+# Local observability stack for Tomo: Prometheus (metrics), Loki (activity
+# feed), Alloy (log shipper), Grafana (dashboard). See README.md.
+#
+# Grafana listens on the LAN (with login required); everything else binds to
+# 127.0.0.1 — Prometheus/Loki have no auth of their own, and Grafana reaches
+# them over the internal Docker network, so they never need outside exposure.
+services:
+  prometheus:
+    image: prom/prometheus:v3.4.1
+    ports:
+      - "127.0.0.1:9090:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-data:/prometheus
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.retention.time=90d
+    extra_hosts:
+      # Docker Desktop for Mac provides host.docker.internal natively; this
+      # line makes the same name work if the stack ever runs on Linux.
+      - "host.docker.internal:host-gateway"
+    restart: unless-stopped
+
+  loki:
+    image: grafana/loki:3.5.1
+    # No host port: Grafana and Alloy reach Loki over the internal Docker
+    # network, and host 3100 is BlueBubbles' default webhook port — publishing
+    # it would collide with a default iMessage setup. To poke Loki directly:
+    #   docker compose exec loki wget -qO- localhost:3100/ready
+    volumes:
+      - loki-data:/loki
+    restart: unless-stopped
+
+  alloy:
+    image: grafana/alloy:v1.9.1
+    volumes:
+      - ./alloy/config.alloy:/etc/alloy/config.alloy:ro
+      - alloy-data:/var/lib/alloy/data
+      # Tomo's activity log (metrics.activityLog). Adjust if TOMO_HOME moved.
+      - ~/.tomo/logs:/var/log/tomo:ro
+    command:
+      - run
+      - /etc/alloy/config.alloy
+      - --storage.path=/var/lib/alloy/data
+    depends_on:
+      - loki
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:12.0.2
+    ports:
+      # LAN-reachable so other machines can open the dashboard. Because the
+      # activity feed can contain message text, login stays REQUIRED — first
+      # sign-in is admin/admin and Grafana forces a password change. To lock
+      # it back down to this machine only, prefix with 127.0.0.1: again.
+      - "3000:3000"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    depends_on:
+      - prometheus
+      - loki
+    restart: unless-stopped
+
+volumes:
+  prometheus-data:
+  loki-data:
+  alloy-data:
+  grafana-data:

package/contrib/observability/grafana/dashboards/tomo.json

@@ -0,0 +1,328 @@
+{
+  "uid": "tomo-overview",
+  "title": "Tomo",
+  "tags": ["tomo"],
+  "schemaVersion": 39,
+  "editable": true,
+  "time": { "from": "now-24h", "to": "now" },
+  "refresh": "30s",
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Daemon",
+      "gridPos": { "h": 4, "w": 3, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "up{job=\"tomo\"}" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "colorMode": "background" },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "0": { "text": "DOWN", "color": "red" },
+                "1": { "text": "UP", "color": "green" }
+              }
+            }
+          ]
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Cost (24h)",
+      "gridPos": { "h": 4, "w": 3, "x": 3, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "sum(increase(tomo_cost_usd_total[24h]))" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": { "defaults": { "unit": "currencyUSD", "decimals": 2 }, "overrides": [] }
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "Turns (24h)",
+      "gridPos": { "h": 4, "w": 3, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "sum(increase(tomo_turns_total[24h]))" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": { "defaults": { "unit": "short", "decimals": 0 }, "overrides": [] }
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Errors (24h)",
+      "gridPos": { "h": 4, "w": 3, "x": 9, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        { "refId": "A", "expr": "sum(increase(tomo_issues_total{level=\"error\"}[24h])) or vector(0)" }
+      ],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "decimals": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 1 }
+            ]
+          }
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "In flight",
+      "gridPos": { "h": 4, "w": 3, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "tomo_turns_active" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": { "defaults": { "unit": "short", "decimals": 0 }, "overrides": [] }
+    },
+    {
+      "id": 6,
+      "type": "stat",
+      "title": "Memory (RSS)",
+      "gridPos": { "h": 4, "w": 3, "x": 15, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "process_resident_memory_bytes{job=\"tomo\"}" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] }
+    },
+    {
+      "id": 7,
+      "type": "stat",
+      "title": "Next heartbeat",
+      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "tomo_heartbeat_next_timestamp_seconds * 1000" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "dateTimeFromNow",
+          "noValue": "n/a",
+          "mappings": [
+            {
+              "type": "value",
+              "options": { "0": { "text": "continuity off", "color": "text" } }
+            }
+          ]
+        },
+        "overrides": []
+      }
+    },
+    {
+      "id": 8,
+      "type": "stat",
+      "title": "Next cron job",
+      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "refId": "A", "expr": "min(tomo_cron_next_run_timestamp_seconds) * 1000" }],
+      "options": { "reduceOptions": { "calcs": ["lastNotNull"] } },
+      "fieldConfig": { "defaults": { "unit": "dateTimeFromNow", "noValue": "none" }, "overrides": [] }
+    },
+    {
+      "id": 21,
+      "type": "table",
+      "title": "Upcoming cron jobs",
+      "gridPos": { "h": 9, "w": 8, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "tomo_cron_next_run_timestamp_seconds * 1000",
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": { "Time": true, "__name__": true, "instance": true, "job": true },
+            "renameByName": { "cron_job": "Job", "Value": "Next run" }
+          }
+        }
+      ],
+      "options": { "sortBy": [{ "displayName": "Next run", "desc": false }] },
+      "fieldConfig": {
+        "defaults": { "unit": "dateTimeFromNow" },
+        "overrides": [
+          {
+            "matcher": { "id": "byName", "options": "Job" },
+            "properties": [{ "id": "unit", "value": "string" }]
+          }
+        ]
+      }
+    },
+    {
+      "id": 22,
+      "type": "logs",
+      "title": "Tool calls",
+      "gridPos": { "h": 9, "w": 16, "x": 8, "y": 4 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "{job=\"tomo-activity\", type=\"tool.start\"} | json | line_format \"{{.tool}}{{if .agent}} [{{.agent}}]{{end}}{{if .detail}} — {{.detail}}{{end}}\""
+        },
+        {
+          "refId": "B",
+          "expr": "{job=\"tomo-activity\", type=\"tool.end\"} | json | ok=\"false\" | line_format \"{{.tool}} FAILED{{if .durationMs}} after {{.durationMs}}ms{{end}}\""
+        }
+      ],
+      "options": {
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true,
+        "enableLogDetails": true,
+        "dedupStrategy": "none"
+      }
+    },
+    {
+      "id": 10,
+      "type": "timeseries",
+      "title": "Cost per hour",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        { "refId": "A", "expr": "sum(increase(tomo_cost_usd_total[1h]))", "legendFormat": "USD/h" }
+      ],
+      "fieldConfig": { "defaults": { "unit": "currencyUSD" }, "overrides": [] }
+    },
+    {
+      "id": 11,
+      "type": "timeseries",
+      "title": "Turns per hour by source",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (source) (increase(tomo_turns_total[1h]))",
+          "legendFormat": "{{source}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": { "unit": "short", "custom": { "stacking": { "mode": "normal" } } },
+        "overrides": []
+      }
+    },
+    {
+      "id": 12,
+      "type": "timeseries",
+      "title": "Turn duration (p50 / p95)",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 21 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "histogram_quantile(0.5, sum by (le) (rate(tomo_turn_duration_seconds_bucket[$__rate_interval])))",
+          "legendFormat": "p50"
+        },
+        {
+          "refId": "B",
+          "expr": "histogram_quantile(0.95, sum by (le) (rate(tomo_turn_duration_seconds_bucket[$__rate_interval])))",
+          "legendFormat": "p95"
+        }
+      ],
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] }
+    },
+    {
+      "id": 13,
+      "type": "timeseries",
+      "title": "Context usage by session",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 21 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "tomo_context_used_tokens / on(session) tomo_context_max_tokens",
+          "legendFormat": "{{session}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": { "unit": "percentunit", "min": 0, "max": 1 },
+        "overrides": []
+      }
+    },
+    {
+      "id": 14,
+      "type": "timeseries",
+      "title": "Tool calls per hour",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 29 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (tool) (increase(tomo_tool_calls_total[1h]))",
+          "legendFormat": "{{tool}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": { "unit": "short", "custom": { "stacking": { "mode": "normal" } } },
+        "overrides": []
+      }
+    },
+    {
+      "id": 15,
+      "type": "timeseries",
+      "title": "Messages per hour by channel",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 29 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "refId": "A",
+          "expr": "sum by (channel, role) (increase(tomo_messages_total[1h]))",
+          "legendFormat": "{{channel}} / {{role}}"
+        }
+      ],
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] }
+    },
+    {
+      "id": 20,
+      "type": "logs",
+      "title": "Activity feed",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 37 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "targets": [{ "refId": "A", "expr": "{job=\"tomo-activity\"}" }],
+      "options": {
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true,
+        "enableLogDetails": true
+      }
+    }
+  ],
+  "annotations": {
+    "list": [
+      {
+        "name": "Compactions",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "enable": true,
+        "expr": "{job=\"tomo-activity\", type=\"compact\"}",
+        "iconColor": "orange"
+      },
+      {
+        "name": "Heartbeats",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "enable": true,
+        "expr": "{job=\"tomo-activity\", type=\"heartbeat\"}",
+        "iconColor": "blue"
+      },
+      {
+        "name": "Issues",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "enable": false,
+        "expr": "{job=\"tomo-activity\", type=\"issue\"}",
+        "iconColor": "red"
+      }
+    ]
+  }
+}

package/contrib/observability/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,8 @@
+apiVersion: 1
+
+providers:
+  - name: tomo
+    folder: ""
+    type: file
+    options:
+      path: /var/lib/grafana/dashboards

package/contrib/observability/grafana/provisioning/datasources/datasources.yml

@@ -0,0 +1,16 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    uid: prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false
+  - name: Loki
+    uid: loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: false

package/contrib/observability/prometheus.yml

@@ -0,0 +1,9 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: tomo
+    static_configs:
+      # Tomo's metrics exporter on the Docker host (metrics.port, default 9464).
+      - targets: ["host.docker.internal:9464"]

package/dist/cli/start.js

@@ -175,9 +175,35 @@ async function startForeground() {
         }),
         sendChat: (text) => agent.handleWatchChat(text),
     });
+    // Metrics exporter + activity log: two more watch-bus subscribers, for
+    // Prometheus scrapes and Loki tailing. Off unless config.metrics.enabled.
+    const { MetricsExporter } = await import("../metrics/exporter.js");
+    const { ActivityLog } = await import("../metrics/activity-log.js");
+    let metricsExporter = null;
+    let activityLog = null;
+    if (config.metrics.enabled) {
+        metricsExporter = new MetricsExporter({
+            version: getCurrentVersion(),
+            model: config.model,
+            collectors: {
+                cronJobs: () => cronStore.list(),
+                nextHeartbeatAt: () => continuity.nextFireAt(),
+            },
+        });
+        await metricsExporter.start(config.metrics.port);
+        if (config.metrics.activityLog) {
+            activityLog = new ActivityLog({
+                path: join(config.logsDir, "activity.ndjson"),
+                includeMessageText: config.metrics.includeMessageText,
+            });
+            activityLog.start();
+        }
+    }
     // Write PID so `tomo stop` can find us
     writeFileSync(PID_FILE, String(process.pid));
     const shutdown = async () => {
+        activityLog?.stop();
+        metricsExporter?.stop();
         watchServer.stop();
         versionChecker.stop();
         rollupRunner.stop();