@stjbrown/agent-knowledge 0.1.0-beta.4 → 0.1.0-beta.5

package/OBSERVABILITY.md

@@ -0,0 +1,163 @@
+# Janet observability design
+
+## Goals
+
+Janet is a local CLI agent, not a web application. Observability therefore belongs in the CLI
+runtime and must not require Mastra Studio, a Mastra development server, or any other always-on
+Janet process.
+
+The foundation has five constraints:
+
+1. Tracing is strictly off by default.
+2. Metadata-only capture is the recommended mode.
+3. Local inspection works without a collector.
+4. Remote export uses standard OTLP so Phoenix is the first supported backend, not the only one.
+5. Secrets come from the process environment and are never written to Janet settings.
+
+## Runtime architecture
+
+Each interactive or headless Janet process creates one observability runtime during controller
+startup:
+
+```text
+session.sendMessage
+  -> Mastra tracing options
+  -> Mastra Observability
+     -> local Mastra storage exporter -> ~/.agent-knowledge/observability.db
+     -> generic OTLP exporter         -> Phoenix or another OTLP backend
+```
+
+No observability object or exporter is constructed while capture is off. The ordinary
+`threads.db` continues to hold Janet thread history. Local traces use a separate
+`observability.db`, routed through Mastra composite storage, with a seven-day default retention
+window.
+
+Completed spans are flushed before Janet destroys its controller. Export and pruning failures are
+best effort and must not turn a successful Janet response into a failed response.
+
+## Capture modes
+
+| Mode | Captured | Excluded |
+|---|---|---|
+| `off` | Nothing | All spans and exports |
+| `metadata` | Timing, hierarchy, model and tool identity, token usage, status, and errors | Prompts, responses, tool arguments, and tool results |
+| `full` | Metadata plus prompt, response, and tool payload content | Nothing beyond Mastra's sensitive-data filtering and serialization limits |
+
+Full capture requires a second explicit confirmation in the TUI. Both captured modes exclude
+streaming model-chunk spans and cap serialized string, object, array, and nesting sizes.
+
+Project identity is represented by Janet's existing hashed resource ID. Settings and status output
+never show authentication headers. Endpoint status output strips credentials, query parameters,
+and fragments.
+
+## Destinations
+
+### Local history
+
+Local history uses Mastra's storage exporter and libSQL. `/traces` lists recent root traces and
+renders their agent, model, and tool hierarchy without printing captured payloads.
+
+### Phoenix
+
+Phoenix uses the same generic OTLP/HTTP protobuf path as any other compatible collector. Janet
+adds the Phoenix project name as both an OpenInference resource attribute and the
+`x-project-name` request header. A base collector endpoint such as `http://localhost:6006` is
+normalized by Mastra's OTLP exporter to `/v1/traces`.
+
+Phoenix runs separately from Janet. Follow the
+[Phoenix local deployment documentation](https://arize.com/docs/phoenix) to run its collector and
+UI.
+
+### Custom OTLP
+
+Custom OTLP accepts any HTTP or HTTPS base endpoint compatible with OTLP/HTTP protobuf. Credentials
+and vendor headers use the standard `OTEL_EXPORTER_OTLP_HEADERS` environment variable. This keeps
+the runtime backend-neutral and avoids storing secrets or adding vendor-specific code to Janet.
+
+## Configuration
+
+The TUI is the primary interactive setup:
+
+```text
+/observability
+/observability status
+/observability off
+/traces
+```
+
+The TUI persists only nonsecret preferences in `~/.agent-knowledge/settings.json`. Changes apply
+after restart so a process never has two competing observability lifecycles.
+
+Environment variables override saved settings for headless runs and automation:
+
+| Variable | Purpose |
+|---|---|
+| `JANET_OBSERVABILITY` | `off`, `metadata`, or `full` |
+| `JANET_OBSERVABILITY_BACKEND` | `local`, `phoenix`, or `otlp` |
+| `JANET_OBSERVABILITY_SAMPLE_RATE` | Number from `0` through `1` |
+| `PHOENIX_COLLECTOR_ENDPOINT` | Phoenix base collector endpoint |
+| `PHOENIX_PROJECT_NAME` | Phoenix project, default `janet` |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Generic OTLP base endpoint |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Trace-specific OTLP endpoint |
+| `OTEL_EXPORTER_OTLP_HEADERS` | Runtime-only comma-separated headers |
+
+Precedence is:
+
+1. Janet environment overrides
+2. Saved Janet settings
+3. Strictly off defaults
+
+Standard `OTEL_*` variables can configure an explicitly enabled run, but cannot enable tracing by
+themselves. Janet does not automatically load a project `.env`.
+
+## Cancellation
+
+Cancellation is part of the observability foundation because a trace is not useful if a runaway
+turn cannot be stopped. Keyboard handling is global rather than tied to the focused editor:
+
+- Esc or the first Ctrl+C calls `session.abort()` for an active turn.
+- A second Ctrl+C within the double-press window force exits if abort is not completing.
+- `/cancel` uses the same active-turn abort path.
+- A single idle Ctrl+C clears editor input or shows the exit hint; a second exits Janet.
+
+## Verification
+
+The automated suite covers:
+
+- default-off resolution even when standard OTEL variables exist
+- metadata and full privacy flags
+- malformed and missing configuration
+- separate local trace storage
+- local trace persistence and concurrent Janet processes
+- content-free local trace rendering
+- global cancellation and force-exit behavior
+- endpoint and header redaction
+
+An opt-in integration test opens a temporary local collector and verifies a nonempty
+Phoenix-compatible protobuf request, `/v1/traces`, and `x-project-name`:
+
+```bash
+JANET_OTLP_INTEGRATION=1 \
+corepack pnpm --filter @stjbrown/agent-knowledge \
+  exec vitest run test/observability-runtime.test.ts
+```
+
+The release test plan in [`TESTING.md`](./TESTING.md) adds a real TUI, Phoenix UI, and clean-install
+pass.
+
+## Evals roadmap
+
+Tracing comes first because it supplies the run records needed to design useful evals. The next
+layer should remain backend-neutral:
+
+1. Define a small Janet evaluator interface over completed run records.
+2. Start with deterministic checks already owned by this project: OKF conformance, citation
+   integrity, expected file changes, repeated tool attempts, and successful cancellation.
+3. Store evaluator name, version, score, label, and explanation as trace metadata or linked score
+   records.
+4. Add a fixture corpus for init, ingest, query, lint, and failure-recovery scenarios.
+5. Add optional model-graded evaluators only after the deterministic baseline is stable.
+6. Export the same results to Phoenix or another backend without changing evaluator logic.
+
+Tool extensibility is intentionally a separate follow-up. Traces should tell us which capabilities
+Janet lacks before the project commits to a tool-provider interface or bundled defaults.

package/README.md

@@ -79,7 +79,9 @@ token-free OKF conformance check before the agent's drift audit, so it is usable
 |---|---|
 | `/models` · `/model [id]` | pick a model from an arrow-key list, or switch by id |
 | `/login <anthropic\|openai-codex> [browser\|device]` · `/logout` · `/auth` | subscription sign-in and status; device mode is available for remote OpenAI login |
-| `/help` · `/quit` | help; exit (or double Ctrl+C) |
+| `/observability` · `/traces` | configure opt-in tracing and browse local trace history |
+| `/cancel` | cancel the active turn; Esc or Ctrl+C does the same while Janet is working |
+| `/help` · `/quit` | help; exit (or press Ctrl+C twice) |
 
 Just type to talk to Janet; ↑/↓ recalls previous prompts.
 
@@ -88,6 +90,32 @@ Gemini, via ADC/service account), Amazon Bedrock (AWS credential chain), Anthrop
 key **or** subscription OAuth), and Google Gemini (API key). Set the choice once (`--model`,
 `JANET_MODEL`, or the first-run picker) and it persists.
 
+**Observability.** Tracing is strictly off by default. Run `/observability` to choose local trace
+history, Phoenix, or a custom OTLP endpoint. Metadata-only capture records timing, model and tool
+activity, token usage, status, and errors without prompt or response bodies. Full capture requires
+an explicit warning and confirmation. Settings take effect after restarting Janet.
+
+Local history is stored separately at `~/.agent-knowledge/observability.db` and can be inspected
+with `/traces`. Phoenix runs as a separate local or remote service; Janet sends it standard
+OTLP/HTTP protobuf traces and does not run a web or development server. Custom OTLP supports other
+compatible collectors and backends.
+
+Headless runs and automation can use environment configuration:
+
+```bash
+JANET_OBSERVABILITY=metadata \
+JANET_OBSERVABILITY_BACKEND=phoenix \
+PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006 \
+PHOENIX_PROJECT_NAME=janet \
+janet query "What happened?" --print
+```
+
+Use `JANET_OBSERVABILITY_BACKEND=otlp` with `OTEL_EXPORTER_OTLP_ENDPOINT` for a custom collector.
+Authentication headers can be supplied with `OTEL_EXPORTER_OTLP_HEADERS`; Janet never writes them
+to `settings.json`. Standard `OTEL_*` variables configure an explicitly enabled run but do not
+enable tracing on their own. Janet also does not load a project's `.env` automatically. See
+[`OBSERVABILITY.md`](./OBSERVABILITY.md) for the architecture, privacy model, and eval roadmap.
+
 Janet is built on [Mastra](https://mastra.ai) and lives in
 [`packages/janet`](https://github.com/stjbrown/agent-knowledge/tree/janet-agent/packages/janet)
 (published as `@stjbrown/agent-knowledge`, bins `janet` + `ding`). She also reports lifecycle state