engineering-behavior-observatory 0.2.1

package/docs/evaluation/uniform-events.md

@@ -0,0 +1,131 @@
+# Uniform events and adapter contract
+
+EBO normalizes only capture-qualified native evidence. The native session,
+hook, protocol, telemetry, workspace, verifier, and manifest records remain the
+authority; an `ebo.uniform-event/v1` record is a bounded projection with an
+audit link back to those bytes.
+
+## Event projection
+
+[`schemas/uniform-events/v1.json`](../../schemas/uniform-events/v1.json) defines the
+event and `ebo.adapter-capability-profile/v1` schemas. Every event carries its
+event, run, and attempt identities; source harness and native type; native
+evidence reference; native order and time; actor; family; phase; scope;
+relations; bounded attributes; and content references.
+
+An event may retain up to 4,096 known relations. Native model-request and
+assembled-message events can reference many source chunks; the live DeepSeek
+smoke required 119 relations on one event. Relations are preserved rather than
+silently truncated, and larger sets fail validation explicitly.
+
+The initial families are `message`, `model-request`, `tool`, `context`,
+`permission`, `delegation`, `artifact`, `validation`, `runtime`, and `outcome`.
+They are observation categories, not a shared transport or lifecycle protocol.
+The four phases (`before`, `during`, `after`, and `instant`) locate an
+observation around its native operation. They do not claim common completion
+semantics across harnesses.
+
+Native order, native time, parentage, and content are explicit evidence values:
+
+- `known` carries a native value, an empty list for known-empty content, or
+  content references.
+- `unknown` records that the run did not establish the value and gives a reason.
+- `unsupported` records that the source capability cannot provide it and gives
+  a reason.
+
+Adapters must not substitute a local clock, array position, empty content, or a
+false parent for missing native evidence. Attributes accept at most 32 shallow
+scalar fields, strings are bounded, and nested or large content is represented
+only through a native evidence reference.
+
+A known native order includes its source-local `domain`. Sequence values from
+independent session, hook, protocol, or telemetry streams are never compared or
+combined unless retained native evidence establishes that ordering.
+
+## Native references
+
+A native reference contains the retained `artifactId` and a source-owned
+`recordLocator`, such as a JSONL line, JSON pointer, or workspace path. EBO does
+not interpret the locator as a universal protocol address. The
+`NativeEvidenceResolver` verifies that each source and content reference
+resolves in the retained bundle before normalized events are accepted.
+
+DeepSeek JSON-RPC method names remain native types and record locators.
+OpenHands REST resources, operations, and WebSocket events remain native types
+and record locators. None of them become uniform event families. The same rule
+applies to Agent SDK message and hook names.
+
+## Adapter boundary
+
+`NativeCaptureAdapter<Request, NativeRecord>` owns a typed, source-specific
+request and returns one `QualifiedNativeCapture`: the run and attempt identity,
+qualification status, and the native records qualified under that identity. It
+does not impose shared methods, tool names, limits, or completion states.
+`UniformEventNormalizationAdapter<NativeRecord>` consumes that same envelope
+and produces uniform events plus an explicit list of unmapped native references.
+
+`assertAdapterContract` checks that:
+
+1. capture, normalizer, and capability-profile identities agree;
+2. every event and content reference resolves;
+3. every captured native record is either mapped or retained as unmapped; and
+4. a normalizer does not invent source references.
+
+`AdapterRegistry` receives a fixed list and rejects duplicate harnesses. This is
+explicit registration, not dynamic discovery or a plugin lifecycle. Adding an
+adapter supplies its own native types and capability profile; it does not add
+convenience fields to the event schema.
+
+The golden examples in
+`test/fixtures/uniform-events/all-families.v1.json`
+cover all initial families using representative retained harness, verifier,
+telemetry, workspace, and terminal evidence shapes. They deliberately preserve unknown
+time, parentage, and content where the qualified fixtures do not establish
+those facts.
+
+## Agent SDK mapping
+
+`normalizeClaudeAgentSdkRunBundle` applies the concrete Agent SDK adapter to a
+retained bundle. It runs structural capture qualification first, verifies every
+artifact digest, normalizes the qualified native records, and resolves every
+emitted source/content reference. An unqualified bundle is rejected before
+normalization.
+
+| Native evidence | Uniform families |
+| --- | --- |
+| SDK assistant/user/result and structural system records | `message`, `tool`, `context`, `delegation`, `permission`, `runtime`, `outcome` |
+| Agent SDK callback records | `message`, `tool`, `context`, `permission`, `delegation`, `artifact`, `runtime` |
+| Agent SDK telemetry summary | `runtime` |
+| Workspace manifest descriptor and verifier artifact | `artifact`, `validation` |
+| Run-manifest assessment mode and terminal records | `outcome` |
+
+Session and callback sequence numbers keep separate
+`session:<artifact-id>` and `hooks:<artifact-id>` order domains. An originating
+SDK message timestamp is retained when present and valid RFC 3339;
+the adapter never promotes the stream capture clock or hook callback clock to a
+native occurrence time. Tool call/result and hook relations are added only when
+one unique emitted call is supported by the same `tool_use_id`; the same rule
+applies to stable subagent/task and workspace artifact identities.
+
+Workspace metadata is sourced from its exact run-manifest evidence-descriptor
+pointer; the qualified workspace bytes remain a content reference. Raw prompts,
+messages, tool inputs/results, compact summaries, and verifier
+assertions stay in their native artifacts. Events contain bounded structural
+attributes and pointers to those bytes. Unknown message/hook records and
+qualification metadata remain reachable through the returned `unmapped`
+report.
+
+The returned per-run capability profile records whether detailed-beta hook
+spans were configured. This never changes semantic hook events because
+`hooks.jsonl` is the occurrence authority. That separation follows the Agent
+SDK's documented [callback correlation](https://code.claude.com/docs/en/agent-sdk/hooks)
+and [beta tracing](https://code.claude.com/docs/en/agent-sdk/observability)
+boundaries.
+
+Model-switch callbacks remain `runtime` evidence. The retained Agent SDK
+capability record does not expose inference-request lifecycle events, so the
+`model-request` family is reported as unsupported rather than as an observed
+zero.
+
+Dataset-wide native-reference, ordering, coverage, and comparison gates are
+documented in [Normalization integrity and comparison gates](normalization-integrity.md).

package/docs/guides/atlas.md

@@ -0,0 +1,144 @@
+# Behavior Atlas
+
+The Atlas combines native Grafana dashboards, a local evidence workbench and a
+standalone HTML research report. It consumes the [aggregation request](../evaluation/aggregation.md)
+and revalidates corpus, structural observations, assertion citations, human
+review lineage and comparison gates. It never runs a judge or edits reviews.
+
+## Rebuild and explore
+
+Use Node 24.19.0 and `npm ci && npm run build`. Put this request beside the
+aggregation request. Paths are relative to their containing request.
+
+```json
+{
+  "schemaVersion": "ebo.atlas-request/v1",
+  "aggregationRequest": "aggregation.json",
+  "title": "Behavior study",
+  "operatorNarrative": "Optional operator-authored research notes.",
+  "atlasUrl": "http://127.0.0.1:13011",
+  "grafanaUrl": "http://127.0.0.1:13010",
+  "reviewPackets": ["review-packet/index.html"]
+}
+```
+
+```sh
+ebo atlas build atlas.json atlas-output
+ebo atlas serve atlas.json
+```
+
+Open `http://127.0.0.1:13011`. Model, harness, task, trial, capture, outcome,
+review, category, assessment and text filters select a reproducible cohort.
+Apply with the button or Enter; controls and evidence disclosures use native
+keyboard behavior. Expand a case for rationale, alternative explanation,
+normalized events, cited native records and human decision lineage.
+
+Filters select attempts through matching cases. Aggregation keeps **every
+judgment on those attempts**, including conflicting assessments. Filtering for
+constructive cases cannot erase an adverse rerun from the aggregate. Matching
+case and aggregate attempt populations are named separately. Rates expose both
+counts and exclusion reasons; structural means retain their units and are not
+percentages. Observational completion is not task success.
+
+The server binds only to `127.0.0.1`, accepts read-only named routes and known
+filter keys, rejects foreign hosts/origins and never maps URLs to arbitrary
+files. It reloads explicitly configured sources for each query. Stale or
+incompatible input fails rather than returning an old aggregate. `--port <port>`
+changes the listener; set `atlasUrl` to the same port before rebuilding dashboards.
+This is a local research workbench, not a hosted multi-user service.
+
+## Grafana
+
+Install official **Grafana OSS 13.2.0** for your operating system, then start a
+fresh profile using the generated provisioning:
+
+```sh
+sh scripts/atlas-grafana.sh /path/to/grafana-13.2.0 atlas-output
+```
+
+The script installs **Infinity 4.0.0** from Grafana's official registry in this
+output's private plugin directory. Grafana binds `127.0.0.1:13010` with local
+anonymous Viewer access. Its data, logs, plugins and provisioning stay under
+this output; no existing dashboard, Docker container or volume changes.
+Rebuild into a new output to preserve previous outputs and local changes.
+
+Open `/d/ebo-atlas-overview` or `/d/ebo-atlas-behavior` on that Grafana instance.
+Native tables, cohort variables and data links query EBO-derived JSON and open
+the evidence workbench at a selected case. There is no custom plugin code or
+study-specific TestData snapshot. Grafana's time picker affects Tempo only; it
+does not filter the retained EBO cohort.
+
+For existing captured traces, point `grafanaUrl` and `tempoDatasourceUid` to a
+profile containing the relevant Tempo data source. Optional `traces` entries
+name `runId`, `attemptId`, a 32-character hexadecimal `traceId`, `originalStart`
+and optional `replayStart` (ISO timestamps). These are operator-configured
+references. Atlas labels both timestamps and neither replays nor invents trace
+data. Without a configured backend and reference it shows trace viewing as
+unavailable. A fresh profile needs its own Tempo data source before trace links
+can resolve.
+
+## Reports and sharing
+
+“Open cohort report” freezes the selected filters, exact aggregates, evidence
+cases, operator narrative, method, limitations and source/cohort digests in
+standalone HTML. Save it and use “Print / save PDF.” The JSON link retains
+supporting data. No server or frontend build is needed to open the saved report.
+
+```sh
+ebo atlas build atlas.json selected-report \
+  --filter model=model-a --filter review=confirmed
+```
+
+Output must be a new directory outside immutable sources. The offline report
+declares its frozen population and never recalculates semantic populations in
+browser JavaScript. Cited records use the existing export sanitizer to omit
+secrets, hidden fields and local paths; original evidence remains authoritative.
+Assertion/source digests identify the original. Existing human packet links are
+local file URLs; browsers may block them from HTTP, so open them from the saved
+local report.
+
+This does not require paid Grafana reporting: Grafana's scheduled reports and
+dashboard PDF exports are Enterprise features. EBO uses standalone HTML and
+browser printing. See [Grafana's official sharing documentation](https://grafana.com/docs/grafana/latest/visualizations/dashboards/share-dashboards-panels/).
+
+Ordinary reports are **restricted-local-only**, even when cited displays are
+sanitized. A shareable summary requires existing portable exports covering every
+selected source manifest. Add `sharing` to the request using the exact policy
+used to create those exports:
+
+```json
+{
+  "sharing": {
+    "policy": { "sharingClass": "partner", "maxArtifactBytes": 16777216, "maxStringBytes": 8192 },
+    "approvedExports": ["approved/run-a", "approved/run-b"],
+    "fields": ["cohort", "aggregate-metrics", "source-digests"]
+  }
+}
+```
+
+`ebo atlas build atlas.json shareable-summary --share` reuses portable export
+readback, policy/digest validation, sanitization and final secret scanning.
+Source correlations are rewritten. Unsupported fields fail closed. Semantic
+assertions/distributions, human prose, native records, raw prompts, trace/review
+links and operator narrative lack portable field classifications and are
+excluded. Local review access never grants sharing approval. Nothing uploads,
+emails or publishes the report.
+
+Shared summaries accept model, harness, task, trial, capture and outcome
+filters. Review, category, assessment and free-text evidence filters are
+rejected: even without case details, their selected-cohort counts would disclose
+semantic or human-review findings that have no export approval.
+
+## Synthetic verification
+
+```sh
+node dist/test/atlas-fixture.js .ebo/atlas-fixture
+ebo atlas build .ebo/atlas-fixture/atlas.json .ebo/atlas-report
+ebo atlas serve .ebo/atlas-fixture/atlas.json
+node --test dist/test/atlas.test.js
+```
+
+The generator injects synthetic capture streams and fixture review decisions;
+it never calls a model or labels real research. It covers opposite assessments,
+confirmed/disputed/rejected/proposed/abstained/unavailable states, a retry,
+missing observations, an unsupported comparison and hostile text.

package/docs/guides/evidence-and-sharing.md

@@ -0,0 +1,100 @@
+# Inspect and share evidence
+
+A run bundle is the retained record of one attempt. Begin with its manifest,
+not a filename guessed from another harness.
+
+## Read a run
+
+| Read | Question |
+| :--- | :--- |
+| CLI summary | Where is the bundle? Did cleanup leave a workspace to recover? |
+| `manifest.json` | Which run/attempt, runtime, configuration, terminal and evidence descriptors belong together? |
+| Capture report referenced by the manifest | Which evidence qualified? What is missing or unsupported? |
+| Native session/store/hooks | What did the agent and harness actually emit? |
+| Workspace descriptor and patch/snapshot | What changed from the frozen starting tree? |
+| Derived observations | Which mechanical facts are supported, and by which native records? |
+
+Each evidence descriptor gives the retained path, media type, size, digest, and
+authority. Names differ by harness. A Pi session tree and a Cursor JSONL store
+are not interchangeable with an Agent SDK stream.
+
+```sh
+ebo validate <bundle-root>/manifest.json
+ebo observations create <bundle-root> study/observations.json
+ebo corpus build study/runs study/index.jsonl
+ebo corpus validate study/runs study/index.jsonl
+ebo corpus query study/index.jsonl --capture qualified
+```
+
+`validate` checks supported artifact schemas; it is not a claim that every
+referenced byte or capability was verified. Observation creation and corpus
+validation perform their documented retained-evidence checks. Review
+`qualified-with-gaps` separately, using the gaps relevant to your question.
+
+The corpus index is a rebuildable JSONL view, not the source of truth. It
+indexes identities and capture/outcome facts rather than prompt and tool bodies.
+
+## Preserve partial attempts
+
+Do not discard an interrupted run because it lacks a final answer. Its native
+events can still explain a stop. Keep terminal state, infrastructure failure,
+and capture quality separate. Preserve `retainedWorkspacePath` when outcome
+packaging failed; that path is a recovery location, not qualified evidence.
+
+Use a new output destination for derived records and reruns. Do not edit a
+native bundle to make a validator accept it.
+
+## Create a portable export
+
+Review the sharing authorization and source material first. Sanitization reduces
+disclosure risk; it does not grant permission to distribute someone else's code.
+
+Save a caller-owned policy as `study/export-policy.json`:
+
+```json
+{
+  "sharingClass": "partner",
+  "maxArtifactBytes": 16777216,
+  "maxStringBytes": 8192,
+  "sensitiveValues": []
+}
+```
+
+Add known confidential strings to `sensitiveValues` when necessary. Keep
+that policy restricted if it contains them. Bounds are examples: choose values
+appropriate to the evidence, not a reason to silently truncate it.
+
+```sh
+ebo export create <bundle-root> study/export-policy.json study/exports/run-a
+ebo corpus pack study/exports/run-a study/export-policy.json study/run-a.tar.gz
+```
+
+Export creates a separate derivative, removes hidden reasoning and detected
+secrets/local identifiers, rewrites correlations, and validates the output.
+Unknown classifications, unsupported content, or failed integrity/secret checks
+stop creation. Native source bytes remain unchanged. Inspect the export manifest
+for transformations and exclusions: opaque workspace snapshots, for example,
+are not made shareable by passing through a text sanitizer.
+
+Packing revalidates the derivative with the same policy, follows the export
+manifest allowlist, and records only approved members. It does not publish.
+
+To inspect a received archive:
+
+```sh
+ebo corpus unpack study/run-a.tar.gz study/received-run-a
+```
+
+The destination must not exist. Unpacking validates archive bounds, membership,
+paths, and digests. Do not replace this with manual extraction of an untrusted
+archive into your checkout.
+
+## Reports are a separate sharing surface
+
+An ordinary Atlas report is **restricted-local-only**, even if displayed
+citations are sanitized. Shareable summaries require approved exports for every
+selected source and an explicit Atlas sharing policy. See
+[reports and sharing](atlas.md#reports-and-sharing) for supported fields and
+exclusions.
+
+For field-level details, see the [run-bundle contract](../reference/run-bundle-contract.md).