@pasko70/pibo 1.16.6 → 2.0.0

package/skills/builtin/pibo-agent-runtime-adapter/references/testing-migration-and-validation.md

@@ -0,0 +1,256 @@
+# Testing, Migration, and Pibo2 Validation
+
+Use this reference when planning adapter verification, writing full/partial harness evals, migrating existing sessions, packaging, or validating the exact candidate on Pibo2.
+
+## Verification layers
+
+Do not substitute one layer for another.
+
+1. **Static contracts** — typecheck, config schema, capabilities, import boundaries.
+2. **Deterministic adapter tests** — fake/native protocol fixtures, lifecycle, events, failures.
+3. **Product integration tests** — router, binding store, profiles, Agent Designer, tools/resources, trace/debug.
+4. **Migration tests** — old stores, ids/revisions, rollback boundary.
+5. **Full regression suite** — preserve existing Pi and product behavior.
+6. **Package verification** — installed package contains runtime/skill/protocol assets.
+7. **Exact-candidate Pibo2 validation** — real gateway, auth, persistence, restart, browser, cleanup.
+8. **Real model proof** — required where streaming/model/tool parity is claimed; auth-failed turns do not count.
+
+## Shared adapter contract
+
+Run `exerciseAgentRuntimeAdapterContract()` against the adapter with deterministic inputs. It checks:
+
+- configured instance/adapter identity;
+- diagnostics and profile validation;
+- open-session identity and workspace;
+- live session contract;
+- binding identity;
+- prompt lifecycle with turn start and terminal event;
+- settled status;
+- abort;
+- unsubscribe;
+- double disposal.
+
+Add adapter-specific tests for every optional capability and protocol behavior not covered by the shared contract.
+
+## Deterministic protocol fixtures
+
+For an RPC/process adapter, fixtures should cover:
+
+- initialize/ready handshake and version negotiation;
+- native session/thread create and resume;
+- assistant deltas/final content;
+- reasoning and content indexes;
+- tool call/start/update/result;
+- usage/context updates;
+- model/reasoning option lists;
+- approvals and structured user input;
+- native history read/pagination;
+- interrupt/abort;
+- process crash and EOF;
+- malformed JSON/message/schema;
+- unknown notification/method;
+- duplicate/out-of-order/late events;
+- startup timeout and backpressure/overload;
+- missing native session;
+- cleanup after partial startup.
+
+Pin fixtures to the exact official schema/version that generated them. Store provenance and supported version range.
+
+## Capability contract tests
+
+For each `true` capability, assert the operation and method exist and work. For each delivery capability, assert selected contributions actually arrive by the declared mode.
+
+For each unsupported capability:
+
+- profile selection returns an error diagnostic with a reason;
+- Agent Designer remains intelligible and does not silently drop existing selections;
+- the live session does not expose a misleading control method;
+- direct API calls fail with capability-unavailable behavior;
+- Context Build reports unsupported/failed delivery accurately.
+
+Test degraded behavior for the documented fidelity loss, not merely successful startup.
+
+## Full-harness evaluation
+
+A full-harness assessment/scaffold should include:
+
+- exact protocol/version evidence table;
+- complete Pibo capability matrix;
+- driver/config/adapter/session module map;
+- binding state and resume design;
+- semantic event and correlation map;
+- native prompt/tool preservation plan;
+- Pibo tool MCP and external MCP plan;
+- selected skills/context delivery and verification;
+- models/auth/reasoning/approvals/input/history mapping;
+- Agent Designer and disabled-state behavior for any remaining gaps;
+- contract/protocol/product/migration/full-suite tests;
+- exact Pibo2 restart/resume/browser/failure/cleanup plan.
+
+The output still may leave optional capabilities unsupported when evidence does not prove them.
+
+## Partial-harness evaluation
+
+A partial-harness assessment must:
+
+- classify unsupported and unknown surfaces explicitly;
+- avoid invented native ids, resume, history, tools, MCP, skills, context, approvals, model catalog, or reasoning controls;
+- use the smallest truthful capability matrix;
+- explain useful supported scope;
+- map unsupported selections to visible disabled reasons and validation errors;
+- avoid terminal scraping or global config hacks;
+- identify the exact evidence/API needed to enable each pending capability;
+- add negative tests that prevent accidental support claims.
+
+A prompt demanding "full support" does not override protocol facts.
+
+## Product integration matrix
+
+At minimum test:
+
+| Area | Required checks |
+|---|---|
+| Registry | duplicate ids, malformed config, unavailable instance, diagnostics |
+| Router | selected instance, queue, events, abort, disposal, no hard-coded adapter branch |
+| Binding | initial state, CAS bind, resume, missing/error, repair/rebind, restart |
+| Profiles | default selection, frozen existing sessions, invalid options/resources |
+| Designer | runtime catalog, support modes, disabled reasons, stale selection removal |
+| Tools | direct/MCP delivery, non-portable rejection, progress, cancellation, payloads |
+| MCP | selected-only, cross-session denial, revocation, actual connection inventory |
+| Skills/context | selected-only, unselected absence, symlink/size bounds, native prompt intact |
+| Subagents | child Pibo Session, cross-runtime target, correlation, cleanup |
+| Events | assistant/reasoning/tool/usage/errors, ordering, duplicate/late handling |
+| History | product-primary fresh trace, native provider, partial page, missing state, cursor scope |
+| Debug/telemetry | runtime identity, payload hydration, redaction, native drill-down |
+| Product jobs | Cron, Loop, workflow, run/goal control where supported |
+
+Use the deterministic fake adapter for generic behavior so tests do not depend on a real harness.
+
+## Import-boundary tests
+
+Prevent adapter dependencies from leaking into generic modules. Scan imports or enforce module boundaries so:
+
+- generic runtime/router/history/trace/debug/data/tools code does not import Pi/Codex/other harness packages;
+- adapter directories may import their own SDK/protocol;
+- compatibility facades are narrow and allowlisted;
+- product code does not branch on literal adapter ids where capability dispatch is sufficient.
+
+## Migration tests
+
+When bindings or history metadata change, construct databases from prior schema versions and verify:
+
+- schema upgrades once and is idempotent;
+- every existing Pi session receives the correct compatibility binding/metadata;
+- fresh sessions do not receive legacy fallback markers;
+- Pibo Session ids remain unchanged;
+- native ids and transcript paths remain unchanged;
+- binding state and revision remain unchanged unless the migration explicitly requires it;
+- message/event/payload counts and integrity remain stable;
+- uniqueness is adapter-scoped;
+- old binaries can ignore additive data within the documented rollback boundary;
+- native non-Pi sessions are never silently converted to Pi during rollback.
+
+Use row counts and stable digests before/after on real disposable data.
+
+## Package verification
+
+Build and inspect the tarball:
+
+```text
+npm run typecheck
+npm run build
+npm test
+npm pack --dry-run --json
+```
+
+Verify it contains:
+
+- compiled adapter/runtime modules;
+- protocol schemas/fixtures needed at runtime;
+- built-in skills and every referenced skill resource;
+- no local credentials, generated runtime state, eval workspaces, or server fixtures.
+
+Import the installed tarball for package-level integration rather than importing the source worktree.
+
+## Exact Pibo2 workflow
+
+Follow the server-development skill. The minimum sequence is:
+
+1. Inspect current version, gateway/process, deployment commit, browser target, database schema/counts/integrity, and host resources.
+2. Pack the clean exact implementation commit and record SHA-256.
+3. Install under a commit-addressed candidate path.
+4. Activate through the approved Pibo2 workflow.
+5. Verify the active executable path and deployment commit immediately before validation requests.
+6. Run authenticated API/CLI/browser scenarios through the real public gateway.
+7. Restart the gateway and prove the same native binding resumes where supported.
+8. Exercise missing native state, invalid profile/options, unsupported capability, abort/failure, and cleanup paths.
+9. Recheck active executable/commit after requests because another deployment can replace the candidate.
+10. Delete temporary sessions/agents/config/payloads/generated directories and restore disposable fixtures.
+11. Record final counts, integrity, active processes, and public health.
+
+Do not hard-code server addresses or expose machine keys/cookies/tokens in docs or logs.
+
+## Pibo2 scenario checklist
+
+For a full adapter, validate where supported:
+
+- fresh native session/thread and first turn;
+- resumed turn after gateway restart with identical native id;
+- native assistant/reasoning/tool streaming;
+- native standard tools unchanged;
+- Pibo MCP tool in the same session;
+- selected external MCP with actual connected inventory;
+- selected built-in/plugin/user skill and selected context;
+- unselected resource absence;
+- model/reasoning/options and auth status;
+- approvals/structured input;
+- Pibo-managed and cross-runtime subagents;
+- Cron/Loop/workflow compatibility;
+- abort and process failure;
+- missing native session diagnostic;
+- product history, native history, trace, debug, telemetry;
+- Agent Designer and Context Build;
+- browser-visible rendering and performance;
+- session/agent deletion and generation cleanup.
+
+For a partial adapter, validate the supported path plus every visible disabled/rejected path. Do not run a scenario that assumes an unsupported feature and then count failure as support.
+
+## Parity evidence
+
+When preserving an existing adapter such as Pi, parity requires:
+
+- full existing suite and build;
+- old native sessions reopen without id/path rewrite;
+- routing, controls, tools, jobs, subagents, TUI, Chat Web, context, trace, debug, telemetry, and reliability behavior;
+- real model streaming/tool behavior on Pibo2.
+
+A turn that reaches queue/idle but fails provider authentication proves only routing/failure handling. It does not prove assistant streaming, tool calls, compaction, or model parity.
+
+## Validation report
+
+Write `docs/reports/<adapter-or-milestone>-validation-YYYY-MM-DD.md` containing:
+
+- status and scope;
+- branch, PR, commits, package version/path/checksum;
+- exact harness/protocol versions;
+- local typecheck/build/focused/full-suite results;
+- migration before/after counts/digests/integrity;
+- exact Pibo2 process/commit checks;
+- API, debug, telemetry, restart, browser, performance, failure, and cleanup evidence;
+- screenshots cropped/redacted to omit identities and unrelated data;
+- defects found and fixes made;
+- remaining uncertainty and unsupported capabilities.
+
+Do not write the report as if a blocked or untested capability passed.
+
+## Completion gate
+
+An adapter milestone is ready for review only when:
+
+- code and docs agree with the capability matrix;
+- all declared capability paths have tests/evidence;
+- all unsupported selections have negative tests and visible reasons;
+- full regression checks pass;
+- the exact candidate was validated where real integration matters;
+- cleanup completed;
+- branch is clean, pushed, and represented by a focused/stacked PR.