@stjbrown/agent-knowledge 0.1.0-beta.1 → 0.1.0-beta.10

package/NOTICE

@@ -17,10 +17,11 @@ and used under those terms.
 
 ------------------------------------------------------------------------
 
-Portions of packages/janet/src (the Amazon Bedrock gateway, and — when added —
-the OAuth auth subsystem under src/auth) are adapted from MastraCode
-(https://github.com/mastra-ai/mastra, the mastracode package), licensed under
-the Apache License, Version 2.0. Adapted and used under those terms.
+Portions of packages/janet/src (the Amazon Bedrock gateway, the OAuth auth
+subsystem under src/auth, and the Observational Memory configuration) are
+adapted from MastraCode (https://github.com/mastra-ai/mastra, the mastracode
+package), licensed under the Apache License, Version 2.0. Adapted and used under
+those terms.
 
 ------------------------------------------------------------------------
 

package/OBSERVABILITY.md

@@ -0,0 +1,163 @@
+# Janet observability design
+
+## Goals
+
+Janet is a local CLI agent, not a web application. Observability therefore belongs in the CLI
+runtime and must not require Mastra Studio, a Mastra development server, or any other always-on
+Janet process.
+
+The foundation has five constraints:
+
+1. Tracing is strictly off by default.
+2. Metadata-only capture is the recommended mode.
+3. Local inspection works without a collector.
+4. Remote export uses standard OTLP so Phoenix is the first supported backend, not the only one.
+5. Secrets come from the process environment and are never written to Janet settings.
+
+## Runtime architecture
+
+Each interactive or headless Janet process creates one observability runtime during controller
+startup:
+
+```text
+session.sendMessage
+  -> Mastra tracing options
+  -> Mastra Observability
+     -> local Mastra storage exporter -> ~/.agent-knowledge/observability.db
+     -> generic OTLP exporter         -> Phoenix or another OTLP backend
+```
+
+No observability object or exporter is constructed while capture is off. The ordinary
+`threads.db` continues to hold Janet thread history. Local traces use a separate
+`observability.db`, routed through Mastra composite storage, with a seven-day default retention
+window.
+
+Completed spans are flushed before Janet destroys its controller. Export and pruning failures are
+best effort and must not turn a successful Janet response into a failed response.
+
+## Capture modes
+
+| Mode | Captured | Excluded |
+|---|---|---|
+| `off` | Nothing | All spans and exports |
+| `metadata` | Timing, hierarchy, model and tool identity, token usage, status, and errors | Prompts, responses, tool arguments, and tool results |
+| `full` | Metadata plus prompt, response, and tool payload content | Nothing beyond Mastra's sensitive-data filtering and serialization limits |
+
+Full capture requires a second explicit confirmation in the TUI. Both captured modes exclude
+streaming model-chunk spans and cap serialized string, object, array, and nesting sizes.
+
+Project identity is represented by Janet's existing hashed resource ID. Settings and status output
+never show authentication headers. Endpoint status output strips credentials, query parameters,
+and fragments.
+
+## Destinations
+
+### Local history
+
+Local history uses Mastra's storage exporter and libSQL. `/traces` lists recent root traces and
+renders their agent, model, and tool hierarchy without printing captured payloads.
+
+### Phoenix
+
+Phoenix uses the same generic OTLP/HTTP protobuf path as any other compatible collector. Janet
+adds the Phoenix project name as both an OpenInference resource attribute and the
+`x-project-name` request header. A base collector endpoint such as `http://localhost:6006` is
+normalized by Mastra's OTLP exporter to `/v1/traces`.
+
+Phoenix runs separately from Janet. Follow the
+[Phoenix local deployment documentation](https://arize.com/docs/phoenix) to run its collector and
+UI.
+
+### Custom OTLP
+
+Custom OTLP accepts any HTTP or HTTPS base endpoint compatible with OTLP/HTTP protobuf. Credentials
+and vendor headers use the standard `OTEL_EXPORTER_OTLP_HEADERS` environment variable. This keeps
+the runtime backend-neutral and avoids storing secrets or adding vendor-specific code to Janet.
+
+## Configuration
+
+The TUI is the primary interactive setup:
+
+```text
+/observability
+/observability status
+/observability off
+/traces
+```
+
+The TUI persists only nonsecret preferences in `~/.agent-knowledge/settings.json`. Changes apply
+after restart so a process never has two competing observability lifecycles.
+
+Environment variables override saved settings for headless runs and automation:
+
+| Variable | Purpose |
+|---|---|
+| `JANET_OBSERVABILITY` | `off`, `metadata`, or `full` |
+| `JANET_OBSERVABILITY_BACKEND` | `local`, `phoenix`, or `otlp` |
+| `JANET_OBSERVABILITY_SAMPLE_RATE` | Number from `0` through `1` |
+| `PHOENIX_COLLECTOR_ENDPOINT` | Phoenix base collector endpoint |
+| `PHOENIX_PROJECT_NAME` | Phoenix project, default `janet` |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Generic OTLP base endpoint |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Trace-specific OTLP endpoint |
+| `OTEL_EXPORTER_OTLP_HEADERS` | Runtime-only comma-separated headers |
+
+Precedence is:
+
+1. Janet environment overrides
+2. Saved Janet settings
+3. Strictly off defaults
+
+Standard `OTEL_*` variables can configure an explicitly enabled run, but cannot enable tracing by
+themselves. Janet does not automatically load a project `.env`.
+
+## Cancellation
+
+Cancellation is part of the observability foundation because a trace is not useful if a runaway
+turn cannot be stopped. Keyboard handling is global rather than tied to the focused editor:
+
+- Esc or the first Ctrl+C calls `session.abort()` for an active turn.
+- A second Ctrl+C within the double-press window force exits if abort is not completing.
+- `/cancel` uses the same active-turn abort path.
+- A single idle Ctrl+C clears editor input or shows the exit hint; a second exits Janet.
+
+## Verification
+
+The automated suite covers:
+
+- default-off resolution even when standard OTEL variables exist
+- metadata and full privacy flags
+- malformed and missing configuration
+- separate local trace storage
+- local trace persistence and concurrent Janet processes
+- content-free local trace rendering
+- global cancellation and force-exit behavior
+- endpoint and header redaction
+
+An opt-in integration test opens a temporary local collector and verifies a nonempty
+Phoenix-compatible protobuf request, `/v1/traces`, and `x-project-name`:
+
+```bash
+JANET_OTLP_INTEGRATION=1 \
+corepack pnpm --filter @stjbrown/agent-knowledge \
+  exec vitest run test/observability-runtime.test.ts
+```
+
+The release test plan in [`TESTING.md`](./TESTING.md) adds a real TUI, Phoenix UI, and clean-install
+pass.
+
+## Evals roadmap
+
+Tracing comes first because it supplies the run records needed to design useful evals. The next
+layer should remain backend-neutral:
+
+1. Define a small Janet evaluator interface over completed run records.
+2. Start with deterministic checks already owned by this project: OKF conformance, citation
+   integrity, expected file changes, repeated tool attempts, and successful cancellation.
+3. Store evaluator name, version, score, label, and explanation as trace metadata or linked score
+   records.
+4. Add a fixture corpus for init, ingest, query, lint, and failure-recovery scenarios.
+5. Add optional model-graded evaluators only after the deterministic baseline is stable.
+6. Export the same results to Phoenix or another backend without changing evaluator logic.
+
+Tool extensibility is intentionally a separate follow-up. Traces should tell us which capabilities
+Janet lacks before the project commits to a tool-provider interface or bundled defaults.

package/README.md

@@ -77,22 +77,85 @@ token-free OKF conformance check before the agent's drift audit, so it is usable
 
 | Command | |
 |---|---|
-| `/models` · `/model [id]` | pick a model from an arrow-key list, or switch by id |
+| `/models` · `/model [id]` | pick a configured provider and model, or switch directly by id |
+| `/providers` | show detected providers and the environment variables that enable more |
 | `/login <anthropic\|openai-codex> [browser\|device]` · `/logout` · `/auth` | subscription sign-in and status; device mode is available for remote OpenAI login |
-| `/help` · `/quit` | help; exit (or double Ctrl+C) |
+| `/observability` · `/traces` | configure opt-in tracing and browse local trace history |
+| `/compact` | flush the current conversation into Observational Memory now |
+| `/clear` | start a blank conversation; the previous thread stays saved and recallable |
+| `/cancel` | cancel the active turn; Esc or Ctrl+C does the same while Janet is working |
+| `/help` · `/quit` | help; exit (or press Ctrl+C twice) |
 
 Just type to talk to Janet; ↑/↓ recalls previous prompts.
 
-**Models & providers.** No default provider — you choose. Janet supports Google Vertex AI (Claude +
-Gemini, via ADC/service account), Amazon Bedrock (AWS credential chain), Anthropic and OpenAI (API
-key **or** subscription OAuth), and Google Gemini (API key). Set the choice once (`--model`,
-`JANET_MODEL`, or the first-run picker) and it persists.
+**Models & providers.** No default provider — you choose. Janet discovers configured providers and
+their current model catalogs through Mastra's native model router. The first provider cohort is
+OpenAI, Anthropic, Google AI Studio, DeepSeek, Groq, Mistral, xAI, OpenRouter, Together AI,
+Fireworks AI, and Cerebras. Set the provider's standard environment variable, restart Janet, and
+use `/models`; `/providers` shows the exact variable names without revealing their values.
+
+Vertex AI (ADC/service account) and Amazon Bedrock (AWS credential chain) use Janet's dedicated
+cloud gateways. OpenAI and Anthropic additionally support ChatGPT/Codex and Claude Max subscription
+OAuth. An explicitly exported `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` takes precedence over stored
+OAuth for that process; unset it to return to subscription authentication. Any other configured
+Mastra-native provider remains usable through `/model provider/model` or `--model provider/model`
+even when it is not in the initial cohort. The selected model persists across restarts.
+
+**Memory.** Janet uses Mastra Observational Memory (OM) by default. The Observer compresses older
+messages and noisy tool output into durable observations as the conversation grows; the Reflector
+condenses those observations over longer sessions. Raw messages remain in local storage and OM's
+recall tool can recover exact details when a compressed observation is insufficient.
+
+Memory work stays on the provider you already authenticated: Vertex and Google use Gemini Flash,
+Anthropic uses Claude Haiku, OpenAI uses a mini model, and Bedrock uses Claude Haiku. Providers
+without a dependable fast default use the selected model itself. To override this policy, set
+`JANET_MEMORY_MODEL=provider/model`, or set `JANET_OBSERVER_MODEL` and
+`JANET_REFLECTOR_MODEL` independently. `/compact` forces the current unobserved tail through the
+same OM pipeline; automatic buffering and compaction remain active either way. `/clear` rotates to
+a blank thread without deleting the old one or changing the knowledge bundle.
+
+**Observability.** Tracing is strictly off by default. Run `/observability` to choose local trace
+history, Phoenix, or a custom OTLP endpoint. Metadata-only capture records timing, model and tool
+activity, token usage, status, and errors without prompt or response bodies. Full capture requires
+an explicit warning and confirmation. Settings take effect after restarting Janet.
+
+Local history is stored separately at `~/.agent-knowledge/observability.db` and can be inspected
+with `/traces`. Phoenix runs as a separate local or remote service; Janet sends it standard
+OTLP/HTTP protobuf traces and does not run a web or development server. Custom OTLP supports other
+compatible collectors and backends.
+
+Headless runs and automation can use environment configuration:
+
+```bash
+JANET_OBSERVABILITY=metadata \
+JANET_OBSERVABILITY_BACKEND=phoenix \
+PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006 \
+PHOENIX_PROJECT_NAME=janet \
+janet query "What happened?" --print
+```
+
+Use `JANET_OBSERVABILITY_BACKEND=otlp` with `OTEL_EXPORTER_OTLP_ENDPOINT` for a custom collector.
+Authentication headers can be supplied with `OTEL_EXPORTER_OTLP_HEADERS`; Janet never writes them
+to `settings.json`. Standard `OTEL_*` variables configure an explicitly enabled run but do not
+enable tracing on their own. Janet also does not load a project's `.env` automatically. See
+[`OBSERVABILITY.md`](./OBSERVABILITY.md) for the architecture, privacy model, and eval roadmap.
 
 Janet is built on [Mastra](https://mastra.ai) and lives in
 [`packages/janet`](https://github.com/stjbrown/agent-knowledge/tree/janet-agent/packages/janet)
 (published as `@stjbrown/agent-knowledge`, bins `janet` + `ding`). She also reports lifecycle state
 natively to [Herdr](https://herdr.dev) when run inside a Herdr pane.
 
+Janet reads local PDFs through a dedicated TypeScript extractor. Small documents return
+page-delimited text directly; larger documents use a cached Markdown artifact read in bounded
+chunks. Raw PDF bytes never enter model history. Visual/OCR fallback remains optional and is not
+enabled yet.
+
+Janet also fetches known public HTTP(S) URLs through a provider-neutral local reader. It validates
+every redirect, blocks private and metadata networks, never executes page JavaScript, and returns
+readable Markdown through the same bounded artifact/chunk pattern. This baseline needs no API key.
+Web search providers (such as Tavily, Firecrawl, or Exa) and interactive browser automation remain
+separate, optional capabilities and are not enabled yet.
+
 ---
 
 ## The skills
@@ -183,7 +246,7 @@ the interactive view. Start at
 ## Layout
 
 ```
-skills/                 # source of truth for both Janet and the skills.sh / plugin installs
+skills/                 # source of truth for the portable skills.sh / plugin collection
   kb/                   # hub: SKILL.md + references/ (SPEC, glossary, trust-model) + templates/ + example-bundle/
   kb-init/  kb-ingest/  kb-query/
   kb-lint/              # + scripts/conformance.mjs  (deterministic §9 check, zero-dep)
@@ -191,6 +254,8 @@ skills/                 # source of truth for both Janet and the skills.sh / plu
 knowledge/              # this project's own OKF bundle (self-documenting)
 packages/
   janet/                # the standalone agent (published as "agent-knowledge")
+    src/skills/         # Janet-only inline skills (not exposed to skills installers)
+    src/tools/          # Janet-only deterministic tools
   kb-tools/             # deterministic TS conformance + graph (compiles the committed skill .mjs)
 .claude-plugin/         # plugin manifest
 ```
@@ -209,4 +274,5 @@ can run `pnpm pack:janet` to execute the release checks and write the package to
 
 [MIT](./LICENSE). The vendored OKF specification (`skills/kb/references/SPEC.md`) is from
 GoogleCloudPlatform/knowledge-catalog under Apache-2.0; portions of `packages/janet` (the auth
-subsystem and Bedrock gateway) are adapted from MastraCode under Apache-2.0. See [NOTICE](./NOTICE).
+subsystem, Bedrock gateway, and Observational Memory configuration) are adapted from MastraCode
+under Apache-2.0. See [NOTICE](./NOTICE).