@handsforbots/semantic-event-observability 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE.md +9 -0
- package/README.md +141 -2
- package/adapters/genericEventBus.js +20 -0
- package/adapters/instrumentChannel.js +22 -0
- package/core/CorrelationContext.js +49 -0
- package/core/EventBuffer.js +37 -0
- package/core/MetricsRegistry.js +200 -0
- package/core/PhaseTracker.js +143 -0
- package/core/Policy.js +87 -0
- package/core/SessionTracker.js +23 -0
- package/core/TraceContextBridge.js +74 -0
- package/core/TraceMapper.js +167 -0
- package/core/TurnMetricsCollector.js +19 -0
- package/core/TurnModel.js +87 -0
- package/core/baggage.d.ts +12 -0
- package/core/baggage.js +68 -0
- package/core/bucketEventName.js +11 -0
- package/core/createObservability.js +611 -0
- package/core/definePhaseModel.js +14 -0
- package/core/eventInstrumentation.js +21 -0
- package/core/isErrorEvent.js +9 -0
- package/core/telemetryAttributes.js +44 -0
- package/core/turnSpanNaming.js +73 -0
- package/docs/architecture.md +106 -0
- package/docs/custom-exporter.md +135 -0
- package/docs/exporters.md +180 -0
- package/docs/getting-started.md +192 -0
- package/docs/grafana-validation.md +71 -0
- package/docs/handsforbots-adapter.md +64 -0
- package/docs/handsforbots-roadmap.md +193 -0
- package/docs/metrics-roadmap.md +145 -0
- package/docs/publishing.md +55 -0
- package/docs/renaming.md +65 -0
- package/docs/roadmap.md +467 -0
- package/docs/tail-sampling.md +99 -0
- package/exporters/console.js +29 -0
- package/exporters/devPanel.js +204 -0
- package/exporters/faro.js +71 -0
- package/exporters/index.js +58 -0
- package/exporters/langfuse.js +161 -0
- package/exporters/langsmith.js +168 -0
- package/exporters/llmExporterState.js +84 -0
- package/exporters/memory.js +18 -0
- package/exporters/otel.js +236 -0
- package/exporters/otelTraceBackend.js +51 -0
- package/exporters/webVitals.js +73 -0
- package/grafana/semantic-event-observability.json +250 -0
- package/grafana/semantic-event-observability.lgtm.json +399 -0
- package/index.d.ts +312 -0
- package/index.js +31 -0
- package/package.json +83 -3
- package/packageIdentity.js +41 -0
- package/utils/id.js +10 -0
- package/utils/probeOptional.js +42 -0
- package/utils/safeJson.js +74 -0
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Grafana dashboard validation
|
|
2
|
+
|
|
3
|
+
Checklist and automated smoke test for bundled dashboards targeting **`sevo_*`** metrics.
|
|
4
|
+
|
|
5
|
+
## Dashboard files
|
|
6
|
+
|
|
7
|
+
| File | Use |
|
|
8
|
+
|------|-----|
|
|
9
|
+
| `grafana/semantic-event-observability.json` | Import template (select Loki/Tempo/Prometheus) |
|
|
10
|
+
| `grafana/semantic-event-observability.lgtm.json` | Pre-provisioned in `examples/` LGTM stack |
|
|
11
|
+
|
|
12
|
+
## Automated smoke test
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
cd packages/semantic-event-observability
|
|
16
|
+
npm test -- test/grafana-dashboard.test.js
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
The test asserts each dashboard JSON includes PromQL for core P0/P1 metrics:
|
|
20
|
+
|
|
21
|
+
- `sevo_turn_duration_ms`
|
|
22
|
+
- `sevo_phase_duration_ms`
|
|
23
|
+
- `sevo_phase_wait_ms`
|
|
24
|
+
- `sevo_events_dropped_total`
|
|
25
|
+
- `sevo_state_gauge`
|
|
26
|
+
- `sevo_turns_total`
|
|
27
|
+
- `sevo_bus_events_total`
|
|
28
|
+
|
|
29
|
+
## Manual LGTM validation
|
|
30
|
+
|
|
31
|
+
1. Start the example stack with observability enabled (`examples/OBSERVABILITY.md`).
|
|
32
|
+
2. Open Grafana → **Semantic Event Observability** dashboard.
|
|
33
|
+
3. Send a few chat turns in the Vite example.
|
|
34
|
+
4. Confirm within ~30s (metric export interval):
|
|
35
|
+
|
|
36
|
+
| Panel | Expected |
|
|
37
|
+
|-------|----------|
|
|
38
|
+
| Turn duration p95 | Non-empty after turns complete |
|
|
39
|
+
| Phase duration p95 | `backend` (or configured phase) after backend round-trip |
|
|
40
|
+
| Phase wait p95 | Non-zero after turn → backend delay |
|
|
41
|
+
| Turn outcomes rate | `completed` series |
|
|
42
|
+
| Backend queue depth | Reflects orchestrator queue |
|
|
43
|
+
| Bus event rate | `core.input`, `core.output_ready`, etc. |
|
|
44
|
+
| Dropped telemetry | Zero or low with full sample rate |
|
|
45
|
+
| Tempo / Recent turns | Turn traces with phase children |
|
|
46
|
+
|
|
47
|
+
5. Drilldown → Metrics: search `sevo_` — all P0 instruments should appear.
|
|
48
|
+
|
|
49
|
+
## Tempo trace checks
|
|
50
|
+
|
|
51
|
+
- Service: `semantic-event-observability`
|
|
52
|
+
- Root span: `turn:core.input` (or `invoke_agent` when `turnRootSpan` enabled)
|
|
53
|
+
- Child spans: `phase:backend`, `event:*`
|
|
54
|
+
|
|
55
|
+
## When panels stay empty
|
|
56
|
+
|
|
57
|
+
| Symptom | Likely cause |
|
|
58
|
+
|---------|----------------|
|
|
59
|
+
| All Prometheus panels empty | OTel metrics export not wired; check `getMeter` in exporter config |
|
|
60
|
+
| Traces only, no metrics | MeterProvider not registered globally |
|
|
61
|
+
| Histogram panels empty but `sevo_*_count` exists in Explore | PromQL metric name mismatch — OTel must not set `unit: 'ms'` on instruments already named `*_ms` (otherwise Prometheus gets `*_milliseconds_bucket` while dashboards query `*_bucket`) |
|
|
62
|
+
| Phase / status / queue panels empty | Label mismatch — metric labels are `phase`, `status`, `key` (not `sevo_phase`, etc.) |
|
|
63
|
+
| Bus events empty | Bus not instrumented or no traffic |
|
|
64
|
+
| Phase wait empty | No `phases` configured or phases start immediately on turn start |
|
|
65
|
+
| p95 shows NaN briefly | Need ongoing chat traffic in the selected time range; histogram `rate()` needs counter movement across scrapes |
|
|
66
|
+
|
|
67
|
+
## Related
|
|
68
|
+
|
|
69
|
+
- [metrics-roadmap.md](./metrics-roadmap.md)
|
|
70
|
+
- [tail-sampling.md](./tail-sampling.md)
|
|
71
|
+
- [exporters.md](./exporters.md)
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# Hands for Bots adapter
|
|
2
|
+
|
|
3
|
+
Hands for Bots v2 integrates through the plugin [`@handsforbots/observability`](../../observability/src/index.ts). The v1 adapter (`adapters/handsforbots.js`, which instrumented the v1 `eventEmitter`) was removed together with v1.
|
|
4
|
+
|
|
5
|
+
## Scope
|
|
6
|
+
|
|
7
|
+
The plugin instruments the **semantic flow** of the v2 kernel. It does **not** monitor whether the application or its backends are up.
|
|
8
|
+
|
|
9
|
+
| Covered | Not covered (use another stack) |
|
|
10
|
+
|---------|--------------------------------|
|
|
11
|
+
| Turn latency (`turn.started` → `turn.done` / `turn.error` / `turn.aborted`) | Uptime / synthetic HTTP probes |
|
|
12
|
+
| Phase per route: `transport`, `direct` (menu), `capture`, `agent` (WebMCP), `push` | Backend `/health` endpoints |
|
|
13
|
+
| Actions by origin (`action.invoked` / `action.failed`, origin `user` / `assistant` / `agent`) | Kubernetes liveness/readiness |
|
|
14
|
+
| Signals by modality (`signal.text`, `signal.transcript`, `signal.image`…), without content by default | "Site down" alerting |
|
|
15
|
+
| Stimuli (`stimulus.ui.effect`, `stimulus.action.call`, `stimulus.error`…), token deltas excluded | |
|
|
16
|
+
| Perceived latency: `h4b_first_response_ms` (input → first visible response) and `h4b_turn_duration_ms`, labelled by route | |
|
|
17
|
+
| Telemetry export health (`sevo_exporter_errors_total`) | |
|
|
18
|
+
|
|
19
|
+
## Usage
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
import { createH4B } from '@handsforbots/core'
|
|
23
|
+
import { observability } from '@handsforbots/observability'
|
|
24
|
+
|
|
25
|
+
const h4b = createH4B({
|
|
26
|
+
plugins: [
|
|
27
|
+
observability({
|
|
28
|
+
environment: 'production',
|
|
29
|
+
sampleRate: 0.2,
|
|
30
|
+
maxEventsPerMinute: 120,
|
|
31
|
+
exporters: ['memory', 'faro', 'otel', 'langfuse', 'langsmith'],
|
|
32
|
+
exporterConfig: {
|
|
33
|
+
langfuse: { project: 'handsforbots' },
|
|
34
|
+
langsmith: { projectName: 'handsforbots' },
|
|
35
|
+
},
|
|
36
|
+
// includeContent: true, // only if your privacy policy allows logging message text
|
|
37
|
+
}),
|
|
38
|
+
],
|
|
39
|
+
})
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
All `createObservability` options are accepted. The instance is available as a service: `h4b.get('observability')`.
|
|
43
|
+
|
|
44
|
+
### Trace propagation to the backend
|
|
45
|
+
|
|
46
|
+
Pass trace headers to HTTP-based transports so a turn is one trace from the browser to the LLM:
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
agui({ url: '/api/agent', headers: () => h4b.get('observability')?.getTraceHeaders() ?? {} })
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
### Experiments
|
|
53
|
+
|
|
54
|
+
Conversation attributes (`h4b.setConversationAttributes`, ADR 0017) go on every span and, through `getTraceHeaders()`, to the backend as W3C `baggage`. Exposures (`h4b.expose`) are recorded as `experiment.exposed`, also the ones from before the plugin mounted (`late: true`). Two counters, never sampled, are recorded: `h4b_experiment_exposed_total` and `h4b_conversation_engaged_total`, the conversation's first visitor turn. On the server, `readConversationAttributes` from `@handsforbots/observability/server` reads the header. See [Experiments](../../../docs/en-us/experiments.md).
|
|
55
|
+
|
|
56
|
+
## Turn and phase model
|
|
57
|
+
|
|
58
|
+
| Kernel event | Semantic event |
|
|
59
|
+
|--------------|----------------|
|
|
60
|
+
| first `turn.status` of a turn | `turn.started` |
|
|
61
|
+
| `turn.status` `acting` with route R | `route.R.start` |
|
|
62
|
+
| `turn.status` `done` / `error` / `aborted` | `route.R.end`, then `turn.done` / `turn.error` / `turn.aborted` |
|
|
63
|
+
|
|
64
|
+
Exports: `H4B_TURN_START_EVENTS`, `H4B_TURN_END_EVENTS`, `H4B_PHASES` from `@handsforbots/observability`.
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# Hands for Bots roadmap (adapter consumer)
|
|
2
|
+
|
|
3
|
+
> **Note (v2):** this roadmap was written for the v1 integration (`core.input`, `core.calling_backend`…). In v2 the integration is the [`@handsforbots/observability`](../../observability/src/index.ts) plugin; see [handsforbots-adapter.md](./handsforbots-adapter.md) for the current event model. Items that still apply (traceparent, gen_ai spans, feedback per turn, Langfuse) carry over with the v2 names.
|
|
4
|
+
|
|
5
|
+
Metrics, phases, and integration work specific to **Hands for Bots** as a consumer of [Semantic Event Observability](./roadmap.md).
|
|
6
|
+
|
|
7
|
+
> **Library metrics (`sevo_*`):** [metrics-roadmap.md](./metrics-roadmap.md)
|
|
8
|
+
> **Integration guide:** [handsforbots-adapter.md](./handsforbots-adapter.md)
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Relationship to the library
|
|
13
|
+
|
|
14
|
+
Hands for Bots does **not** extend the lib core. It provides:
|
|
15
|
+
|
|
16
|
+
1. **Adapter preset** — `adapters/handsforbots.js`
|
|
17
|
+
2. **Observability output plugin** — `Plugins/Output/Observability/`
|
|
18
|
+
3. **Domain metrics** — `hfb_*` via `recordMetric()` and future plugin instrumentation
|
|
19
|
+
4. **Phase preset** — `HFB_PHASE_MODEL` in adapter (passed to `createObservability`)
|
|
20
|
+
5. **Backend correlation** — trace propagation from orchestrator / backend cores
|
|
21
|
+
|
|
22
|
+
The adapter and Observability plugin do **not** provide uptime monitoring, health endpoints, or platform probes. Those stay in deploy/infra and backend stacks — see [library scope](./architecture.md#scope).
|
|
23
|
+
|
|
24
|
+
```mermaid
|
|
25
|
+
flowchart LR
|
|
26
|
+
HFB["Hands for Bots"]
|
|
27
|
+
Adapter["handsforbots.js adapter"]
|
|
28
|
+
SEO["Semantic Event Observability"]
|
|
29
|
+
Plugin["Observability plugin"]
|
|
30
|
+
|
|
31
|
+
HFB --> Plugin --> Adapter --> SEO
|
|
32
|
+
HFB -->|"hfb_* metrics"| SEO
|
|
33
|
+
HFB -->|"gen_ai.* traces"| Tempo["Tempo / backend"]
|
|
34
|
+
SEO -->|"sevo_* + traces"| LGTM["Grafana LGTM"]
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## HfB turn & phase preset
|
|
40
|
+
|
|
41
|
+
Configured in `adapters/handsforbots.js` and passed to `createObservability`:
|
|
42
|
+
|
|
43
|
+
```javascript
|
|
44
|
+
export const HFB_TURN_START_EVENTS = ['core.input']
|
|
45
|
+
export const HFB_TURN_END_EVENTS = ['core.output_ready']
|
|
46
|
+
|
|
47
|
+
export const HFB_PHASE_MODEL = definePhaseModel([
|
|
48
|
+
{
|
|
49
|
+
id: 'backend',
|
|
50
|
+
startEvent: 'core.calling_backend',
|
|
51
|
+
endEvent: 'core.backend_responded',
|
|
52
|
+
},
|
|
53
|
+
// render: wall-clock after backend_responded until turn.end (lib computes render_ms)
|
|
54
|
+
])
|
|
55
|
+
|
|
56
|
+
export const HFB_SEMANTIC_EVENTS = [ /* allowlist for high-signal events */ ]
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
**State provider (HfB-specific):**
|
|
60
|
+
|
|
61
|
+
```javascript
|
|
62
|
+
{
|
|
63
|
+
queueDepth: bot.orchestrator.queue.length,
|
|
64
|
+
callingBackend: bot.orchestrator.calling_backend,
|
|
65
|
+
redirectInput: bot.redirectInput,
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## Metric naming (`hfb_*`)
|
|
72
|
+
|
|
73
|
+
| Prefix | Owner |
|
|
74
|
+
|---------|-------|
|
|
75
|
+
| `hfb_` | Hands for Bots cores, plugins, adapter |
|
|
76
|
+
| `sevo_` | Library only — never defined in HfB code |
|
|
77
|
+
|
|
78
|
+
**Recommended labels:** `plugin`, `backend`, `modality` (`text` \| `voice`), `environment`.
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Development checklist
|
|
83
|
+
|
|
84
|
+
Status: `[ ]` not started · `[~]` partial · `[x]` done
|
|
85
|
+
|
|
86
|
+
### P0 — Adapter alignment with lib abstractions
|
|
87
|
+
|
|
88
|
+
| # | Item | Status |
|
|
89
|
+
|---|------|--------|
|
|
90
|
+
| H0.1 | Move phase preset from otel exporter to adapter `HFB_PHASE_MODEL` | [x] |
|
|
91
|
+
| H0.2 | Wire adapter to `PhaseModel` when lib Phase B ships | [x] |
|
|
92
|
+
| H0.3 | Enable Observability plugin in example stack with `sevo_*` validation | [~] |
|
|
93
|
+
| H0.4 | Document adapter preset in [handsforbots-adapter.md](./handsforbots-adapter.md) | [~] |
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
### P1 — Orchestration & modality metrics
|
|
98
|
+
|
|
99
|
+
| # | Metric | Source event | Status |
|
|
100
|
+
|---|--------|--------------|--------|
|
|
101
|
+
| H1.1 | `hfb_input_total{plugin}` | `core.input` | [ ] |
|
|
102
|
+
| H1.2 | `hfb_output_ready_total` | `core.output_ready` | [ ] |
|
|
103
|
+
| H1.3 | `hfb_backend_requests_total{backend,status}` | `core.calling_backend` + errors | [ ] |
|
|
104
|
+
| H1.4 | Turn labels `input_plugin` on `sevo_*` via adapter metadata | `core.input` payload | [x] |
|
|
105
|
+
| H1.5 | `sevo_queue_wait_ms` data — emit via phase or custom span | input → calling_backend | [ ] |
|
|
106
|
+
| H1.6 | Backend error → `error.type` on phase span | orchestrator / backend | [ ] |
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
### P2 — UI lifecycle & session
|
|
111
|
+
|
|
112
|
+
| # | Metric | Source event | Status |
|
|
113
|
+
|---|--------|--------------|--------|
|
|
114
|
+
| H2.1 | `hfb_bot_load_duration_ms` | `core.loaded` | [ ] |
|
|
115
|
+
| H2.2 | `hfb_ui_load_duration_ms{plugin}` | `core.ui_loaded` | [ ] |
|
|
116
|
+
| H2.3 | `hfb_all_ui_loaded_total` | `core.all_ui_loaded` | [ ] |
|
|
117
|
+
| H2.4 | `hfb_session_renewals_total` | `core.renew_session` | [ ] |
|
|
118
|
+
| H2.5 | `hfb_history_cleared_total` | `core.history_cleared` | [ ] |
|
|
119
|
+
| H2.6 | `hfb_bc_messages_total` | BroadcastChannel (adapter) | [ ] |
|
|
120
|
+
| H2.7 | `hfb_redirect_input_total` | `redirectInput` state | [ ] |
|
|
121
|
+
| H2.8 | `hfb_action_success_total{plugin}` | `core.action_success` | [ ] |
|
|
122
|
+
| H2.9 | `hfb_mcp_tool_feedback_total` | `mcp.tool_feedback_received` | [ ] |
|
|
123
|
+
| H2.10 | Plugin metrics cookbook in docs | `recordMetric()` examples | [ ] |
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
### P2 — Voice (HfB Voice core)
|
|
128
|
+
|
|
129
|
+
| # | Metric | Source | Status |
|
|
130
|
+
|---|--------|--------|--------|
|
|
131
|
+
| H2.V1 | `hfb_voice_session_duration_s` | Voice input/output | [ ] |
|
|
132
|
+
| H2.V2 | `hfb_voice_stt_duration_ms` | SpeechRecognition / Vosk | [ ] |
|
|
133
|
+
| H2.V3 | `hfb_voice_endpoint_delay_ms` | silence → send | [ ] |
|
|
134
|
+
| H2.V4 | `hfb_voice_tts_ttfb_ms` | Voice output | [ ] |
|
|
135
|
+
| H2.V5 | `hfb_voice_interruptions_total` | barge-in during TTS | [ ] |
|
|
136
|
+
|
|
137
|
+
Industry targets (voice): mouth-to-ear P95 < 1.1–1.4 s; STT < 500 ms; TTS TTFB < 250 ms ([Twilio guide](https://www.twilio.com/en-us/blog/developers/best-practices/guide-core-latency-ai-voice-agents)).
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
### P3 — Backend integration (HfB cores)
|
|
142
|
+
|
|
143
|
+
Backend work stays in **BotOrchestrator** and backend cores. The lib provides trace propagation only.
|
|
144
|
+
|
|
145
|
+
| # | Item | Owner | Status |
|
|
146
|
+
|---|------|-------|--------|
|
|
147
|
+
| H3.1 | Inject `traceparent` on `sendToBackend` fetch | BotOrchestrator | [ ] |
|
|
148
|
+
| H3.2 | `gen_ai.client.operation.duration` | OpenAI / UniversalLLM core | [ ] |
|
|
149
|
+
| H3.3 | `gen_ai.client.token.usage` | OpenAI / UniversalLLM core | [ ] |
|
|
150
|
+
| H3.4 | MCP / tool execution child spans | MCPHelper | [ ] |
|
|
151
|
+
| H3.5 | Rasa backend span or `backend=rasa` label | Rasa core | [ ] |
|
|
152
|
+
| H3.6 | End-to-end trace demo in `examples/OBSERVABILITY.md` | Examples | [ ] |
|
|
153
|
+
| H3.7 | Backend observability guide | `docs/en-us/` or `docs-dev/` | [ ] |
|
|
154
|
+
|
|
155
|
+
| Question | `sevo_*` (lib) | `hfb_*` (HfB) | `gen_ai.*` (backend) |
|
|
156
|
+
|----------|---------------|---------------|----------------------|
|
|
157
|
+
| User wait time | `sevo_turn_duration_ms` | — | — |
|
|
158
|
+
| Backend wall time | `sevo_phase_duration_ms{phase="backend"}` | — | `gen_ai.client.operation.duration` |
|
|
159
|
+
| Render / speak time | `sevo_phase_duration_ms` or render attr | `hfb_voice_tts_ttfb_ms` | — |
|
|
160
|
+
| Input modality | label on seo | `hfb_input_total` | — |
|
|
161
|
+
| Token cost | — | — | `gen_ai.client.token.usage` |
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
### P4 — Quality & product (HfB + Langfuse)
|
|
166
|
+
|
|
167
|
+
| # | Item | Status |
|
|
168
|
+
|---|------|--------|
|
|
169
|
+
| H4.1 | User feedback plugin linked to `turnId` | [ ] |
|
|
170
|
+
| H4.2 | Langfuse project config in Observability plugin | [ ] |
|
|
171
|
+
| H4.3 | Online eval sampling on production turns | [ ] |
|
|
172
|
+
| H4.4 | Unify or deprecate Analytics plugin vs observability pipeline | [ ] |
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Implementation pattern for HfB metrics
|
|
177
|
+
|
|
178
|
+
Prefer thin instrumentation in cores/plugins; emit via the observability instance:
|
|
179
|
+
|
|
180
|
+
```javascript
|
|
181
|
+
// In a plugin
|
|
182
|
+
this.bot.observability?.recordMetric('hfb_gallery_open_total', 1, { plugin: 'ImageGallery' })
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
For automatic bus-driven metrics, a future **HfB metrics helper** (in the adapter, not lib core) may subscribe to `HFB_SEMANTIC_EVENTS` and increment counters — keeps plugins declarative.
|
|
186
|
+
|
|
187
|
+
---
|
|
188
|
+
|
|
189
|
+
## Changelog
|
|
190
|
+
|
|
191
|
+
| Date | Change |
|
|
192
|
+
|------|--------|
|
|
193
|
+
| 2026-06-15 | H0.1–H0.2 done; `HFB_PHASE_MODEL` in adapter |
|
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
# Metrics roadmap (library)
|
|
2
|
+
|
|
3
|
+
Development checklist for **`sevo_*` metrics only** — emitted by Semantic Event Observability.
|
|
4
|
+
|
|
5
|
+
> **Not in this document:** Hands for Bots product metrics (`hfb_*`) → [handsforbots-roadmap.md](./handsforbots-roadmap.md).
|
|
6
|
+
> **Abstractions plan:** [roadmap.md](./roadmap.md).
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Out of scope
|
|
11
|
+
|
|
12
|
+
These concerns are intentionally **not** `sevo_*` metrics:
|
|
13
|
+
|
|
14
|
+
| Concern | Owner | Examples |
|
|
15
|
+
|---------|-------|----------|
|
|
16
|
+
| Service uptime | Infra / synthetics | HTTP probes, Grafana Cloud checks, “site down” alerts |
|
|
17
|
+
| Backend health | Backend deploy | Rasa `/api/health`, LLM server health |
|
|
18
|
+
| Platform probes | Kubernetes / Docker | `livenessProbe`, `readinessProbe` on app containers |
|
|
19
|
+
|
|
20
|
+
`sevo_exporter_errors_total` measures **telemetry export failures**, not application availability. See [architecture.md](./architecture.md#scope).
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Naming rules
|
|
25
|
+
|
|
26
|
+
| Rule | Example |
|
|
27
|
+
|------|---------|
|
|
28
|
+
| Prefix | `sevo_` |
|
|
29
|
+
| Unit suffix | `_ms`, `_total`, `_bytes` |
|
|
30
|
+
| Histograms | Turn duration, phase duration |
|
|
31
|
+
| Counters | Turns, drops, events emitted |
|
|
32
|
+
| Gauges | State keys from `stateProvider` |
|
|
33
|
+
|
|
34
|
+
**Standard labels (low cardinality):**
|
|
35
|
+
|
|
36
|
+
- `environment`
|
|
37
|
+
- `phase` — phase id from PhaseModel
|
|
38
|
+
- `status` — `completed` \| `abandoned` \| `error`
|
|
39
|
+
- `reason` — policy drop reason
|
|
40
|
+
- `event_type` — `bus.trigger` \| `turn.end` \| …
|
|
41
|
+
|
|
42
|
+
Never label with `sessionId`, `turnId`, or raw payload text.
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## Industry mapping (front-end)
|
|
47
|
+
|
|
48
|
+
| Industry metric | Library metric | Notes |
|
|
49
|
+
|-----------------|----------------|-------|
|
|
50
|
+
| End-to-end turn latency | `sevo_turn_duration_ms` | Client-perceived wait |
|
|
51
|
+
| Per-stage latency | `sevo_phase_duration_ms{phase}` | PhaseModel-driven |
|
|
52
|
+
| Turn success rate | `sevo_turns_total{status}` | completed vs abandoned |
|
|
53
|
+
| Telemetry loss | `sevo_events_dropped_total{reason}` | Policy transparency |
|
|
54
|
+
| Queue / backlog pressure | `sevo_state_gauge{key="queueDepth"}` | From stateProvider |
|
|
55
|
+
| Throughput | `rate(sevo_turns_total{status="completed"})` | Derived in Grafana |
|
|
56
|
+
|
|
57
|
+
Backend `gen_ai.*` metrics are **not** emitted by this lib. Hosts join traces via [TraceContextBridge](./roadmap.md#6-tracecontextbridge-new--backend-correlation).
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Checklist
|
|
62
|
+
|
|
63
|
+
Status: `[ ]` not started · `[~]` partial · `[x]` done
|
|
64
|
+
|
|
65
|
+
### P0 — SLO readiness
|
|
66
|
+
|
|
67
|
+
| # | Metric | Type | Trigger | Status |
|
|
68
|
+
|---|--------|------|---------|--------|
|
|
69
|
+
| M0.1 | `sevo_turn_duration_ms` | histogram | `turn.end` → `durationMs` | [x] |
|
|
70
|
+
| M0.2 | `sevo_phase_duration_ms` | histogram | PhaseTracker phase end | [x] |
|
|
71
|
+
| M0.3 | `sevo_turns_total` | counter | `status=completed` on turn end | [x] |
|
|
72
|
+
| M0.4 | `sevo_turns_total` | counter | `status=abandoned` on new turn without end | [x] |
|
|
73
|
+
| M0.5 | `sevo_events_dropped_total` | counter | `Policy.recordDrop(reason)` | [x] |
|
|
74
|
+
| M0.6 | `sevo_state_gauge` | gauge | `stateProvider()` keys | [x] |
|
|
75
|
+
| M0.7 | OTel Metrics API wiring | — | `MetricsRegistry` → exporter | [x] |
|
|
76
|
+
| M0.8 | Grafana dashboard smoke test | — | `grafana/semantic-event-observability.json` | [x] |
|
|
77
|
+
|
|
78
|
+
**Implementation:** single `MetricsRegistry` module; OTel exporter reads from registry (see [roadmap.md](./roadmap.md#4-metricsregistry-new--single-source-of-truth)).
|
|
79
|
+
|
|
80
|
+
**SLO examples:**
|
|
81
|
+
|
|
82
|
+
```promql
|
|
83
|
+
histogram_quantile(0.95, sum(rate(sevo_turn_duration_ms_bucket[5m])) by (le))
|
|
84
|
+
sum(rate(sevo_turns_total{status="abandoned"}[5m])) / sum(rate(sevo_turns_total[5m]))
|
|
85
|
+
sum(increase(sevo_events_dropped_total[5m])) by (reason)
|
|
86
|
+
sum(increase(sevo_session_turns_total[5m])) by (status, reason)
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
### P1 — Reliability & visibility
|
|
92
|
+
|
|
93
|
+
| # | Metric | Type | Trigger | Status |
|
|
94
|
+
|---|--------|------|---------|--------|
|
|
95
|
+
| M1.1 | `sevo_phase_wait_ms` | histogram | turn start → phase start | [x] |
|
|
96
|
+
| M1.2 | `sevo_turns_total` | counter | `status=error` when error heuristic fires | [x] |
|
|
97
|
+
| M1.3 | `sevo_events_emitted_total` | counter | `type` + bucketed `name` | [x] |
|
|
98
|
+
| M1.4 | `sevo_exporter_errors_total` | counter | exporter catch blocks | [x] |
|
|
99
|
+
| M1.5 | `sevo_active_turns` | gauge | open turns (diagnostic) | [x] |
|
|
100
|
+
| M1.6 | Span `error.type` attribute | — | pluggable `isError(event)` | [x] |
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
### P2 — Session & host-agnostic depth
|
|
105
|
+
|
|
106
|
+
| # | Metric | Type | Trigger | Status |
|
|
107
|
+
|---|--------|------|---------|--------|
|
|
108
|
+
| M2.1 | `sevo_session_turns_total` | counter | session boundary (configurable) | [x] |
|
|
109
|
+
| M2.2 | `sevo_custom_metrics_total` | counter | `recordMetric()` with name allowlist | [x] |
|
|
110
|
+
| M2.3 | `sevo_bus_events_total` | counter | `bus.trigger` by bucketed name | [x] |
|
|
111
|
+
| M2.4 | `sevo_listener_duration_ms` | histogram | `wrapListeners: true` | [x] |
|
|
112
|
+
| M2.5 | `sevo_web_vital` | histogram | `webVitals` exporter | [x] |
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## `recordMetric()` convention
|
|
117
|
+
|
|
118
|
+
Host apps may call `observability.recordMetric(name, value, labels)` for domain metrics.
|
|
119
|
+
|
|
120
|
+
| Rule | Detail |
|
|
121
|
+
|------|--------|
|
|
122
|
+
| Lib does not rename | Host owns prefix (`hfb_`, `myapp_`) |
|
|
123
|
+
| Policy | Subject to `metric.gauge` policy gate |
|
|
124
|
+
| Export | All exporters receive `onMetric` |
|
|
125
|
+
| Recommendation | Document host prefix in adapter; avoid `sevo_` in `recordMetric` |
|
|
126
|
+
| Allowlist counter | Set `customMetricAllowlist: ['hfb_*']` to increment `sevo_custom_metrics_total` |
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## Grafana
|
|
131
|
+
|
|
132
|
+
Dashboard templates: `grafana/semantic-event-observability.json`, `grafana/semantic-event-observability.lgtm.json`.
|
|
133
|
+
|
|
134
|
+
Panels expect P0 metrics. Host-specific panels belong in host repos, not lib dashboards.
|
|
135
|
+
|
|
136
|
+
Validation: [grafana-validation.md](./grafana-validation.md) · `npm test -- test/grafana-dashboard.test.js`
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## Changelog
|
|
141
|
+
|
|
142
|
+
| Date | Change |
|
|
143
|
+
|------|--------|
|
|
144
|
+
| 2026-06-15 | Initial combined doc (lib + HfB) |
|
|
145
|
+
| 2026-06-15 | Phase A metrics implemented (`MetricsRegistry`, `TurnMetricsCollector`, OTel Metrics API) |
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Publishing to npm
|
|
2
|
+
|
|
3
|
+
The package lives at `packages/semantic-event-observability/` and is named **`@handsforbots/semantic-event-observability`**.
|
|
4
|
+
|
|
5
|
+
## Single source of truth
|
|
6
|
+
|
|
7
|
+
| Field | Source |
|
|
8
|
+
|-------|--------|
|
|
9
|
+
| Package name / scope | `packageIdentity.js` → `PACKAGE_NAME` |
|
|
10
|
+
| Version | `packageIdentity.js` → `PACKAGE_VERSION` **and** `package.json` → `version` |
|
|
11
|
+
| Global debug key | Derived from `PACKAGE_SLUG` via `getPackageIdentity()` |
|
|
12
|
+
| OTel service name | `identity.otelServiceName` (= slug) |
|
|
13
|
+
|
|
14
|
+
**Before every publish**, keep `package.json` version in sync with `PACKAGE_VERSION` in `packageIdentity.js`.
|
|
15
|
+
|
|
16
|
+
## Pre-publish checklist
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
cd packages/semantic-event-observability
|
|
20
|
+
npm test
|
|
21
|
+
npm pack --dry-run # verify files list
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Confirm `files` in `package.json` includes everything consumers need (`index.js`, `index.d.ts`, `core/`, `adapters/`, `exporters/`, `utils/`).
|
|
25
|
+
|
|
26
|
+
## Publish (maintainers)
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
npm login
|
|
30
|
+
npm publish --access public
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Scoped package `@handsforbots/*` requires `--access public` on first publish.
|
|
34
|
+
|
|
35
|
+
## Consumer install
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
npm i @handsforbots/semantic-event-observability
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Optional peers (install only what you use):
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
npm i @opentelemetry/api @grafana/faro-web-sdk
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
TypeScript consumers get types from `index.d.ts` automatically (`"types"` field in `package.json`).
|
|
48
|
+
|
|
49
|
+
## Monorepo / local development
|
|
50
|
+
|
|
51
|
+
Hands for Bots imports the lib from the repo path. No publish required for local work. The Vite example uses the same source tree.
|
|
52
|
+
|
|
53
|
+
## Renaming the package
|
|
54
|
+
|
|
55
|
+
If the public npm name changes, update **`PACKAGE_SLUG`** in `packageIdentity.js` only. Storage keys, globals, and dashboard UIDs derive from it. See [renaming.md](./renaming.md).
|
package/docs/renaming.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Renaming the package
|
|
2
|
+
|
|
3
|
+
The library is published under a **working title**. To rename without breaking integrations:
|
|
4
|
+
|
|
5
|
+
## 1. Edit `packageIdentity.js`
|
|
6
|
+
|
|
7
|
+
```javascript
|
|
8
|
+
export const PACKAGE_SLUG = 'your-new-name'
|
|
9
|
+
export const PACKAGE_DISPLAY_NAME = 'Your New Name'
|
|
10
|
+
export const PACKAGE_SCOPE = '@your-org' // optional
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## 2. Update `package.json`
|
|
14
|
+
|
|
15
|
+
```json
|
|
16
|
+
{
|
|
17
|
+
"name": "@your-org/your-new-name"
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## 3. Derived identifiers (automatic)
|
|
22
|
+
|
|
23
|
+
`getPackageIdentity()` builds these from `PACKAGE_SLUG`:
|
|
24
|
+
|
|
25
|
+
| Derived | Example |
|
|
26
|
+
|---------|---------|
|
|
27
|
+
| npm name | `@your-org/your-new-name` |
|
|
28
|
+
| Global debug API | `__YOUR_NEW_NAME__` (slug uppercased, `-` → `_`) |
|
|
29
|
+
| localStorage debug key | `your-new-name:debug` |
|
|
30
|
+
| localStorage panel key | `your-new-name:panel` |
|
|
31
|
+
| OTel `service.name` | `your-new-name` |
|
|
32
|
+
| Grafana dashboard UID | `your-new-name` |
|
|
33
|
+
|
|
34
|
+
**Do not hardcode these strings elsewhere.** Import from `packageIdentity.js` or `getPackageIdentity()`.
|
|
35
|
+
|
|
36
|
+
## 4. After rename checklist
|
|
37
|
+
|
|
38
|
+
- [ ] Update README and docs references (human-readable name only)
|
|
39
|
+
- [ ] Re-import Grafana dashboard (UID changes with slug)
|
|
40
|
+
- [ ] Update collector rules / Loki labels if they filter on `service_name`
|
|
41
|
+
- [ ] Notify consumers to update localStorage debug key
|
|
42
|
+
- [ ] Publish new npm package (old name can remain as deprecated alias if needed)
|
|
43
|
+
|
|
44
|
+
## 5. Hands for Bots import path
|
|
45
|
+
|
|
46
|
+
Hands for Bots consumes the library through its plugin `@handsforbots/observability`, which imports `createObservability` from the package name. After a rename, only that import changes:
|
|
47
|
+
|
|
48
|
+
```javascript
|
|
49
|
+
import { createObservability } from '@your-org/your-new-name'
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## 6. Alias strategy for npm
|
|
53
|
+
|
|
54
|
+
When renaming publicly, you can publish both packages for one major cycle:
|
|
55
|
+
|
|
56
|
+
```json
|
|
57
|
+
// old package.json
|
|
58
|
+
{ "name": "@handsforbots/semantic-event-observability", "deprecated": "Use @handsforbots/new-name" }
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Re-export from the old entry point:
|
|
62
|
+
|
|
63
|
+
```javascript
|
|
64
|
+
export * from '@handsforbots/new-name'
|
|
65
|
+
```
|