omnius 1.0.558 → 1.0.560

package/prompts/reference-contracts/33_contract_evaluation_and_regression_design.md

@@ -0,0 +1,109 @@
+# Contract Evaluation and Regression Design
+
+## Purpose
+
+System-message and orchestration work is finished only when a model-facing
+behavioral harness proves the intended contract. Static source assertions can
+guard a literal wire, but they cannot show whether evidence arrived, whether a
+tool was blocked, or whether a model could integrate a steering change.
+
+## Four evaluation layers
+
+### 1. Compiler unit tests
+
+Given typed runtime state, assert the request compiler selects the intended
+contract IDs, puts dynamic frames in the required order, excludes retired
+frames, and reports normalized context accounting. Use fixtures with unique
+sentinels so a missing or duplicate frame is unambiguous.
+
+### 2. Orchestration integration tests
+
+Run the real runner path with a deterministic scripted backend and temporary
+workspace. The backend must see the same assembled request that an actual
+model would see, issue normal tool calls, and receive normal tool results.
+Assert user-visible outcomes, not only method calls.
+
+### 3. Adversarial behavioral tests
+
+Test the failure modes the runtime is designed to prevent:
+
+- inject typed “do not mutate; explain next action” steering during a
+  multi-tool old-plan response and a brute-force loop;
+- issue a full-file read, compact, and reread after exact source body left the
+  request;
+- run a partial branch extraction with multiple requirements, including an
+  unresolved requirement; and
+- make an ordinary edit without controller hash/todo preconditions and assert
+  the native tool result, not a synthetic admission denial.
+
+Each test must prove both what happens and what does *not* happen: no old-plan
+mutation after steering, no extract labelled full, no missing required branch
+evidence silently treated as complete, and no exploration gate blocking an
+authorized read/edit.
+
+### 4. Live inference acceptance harness
+
+Use a real model only after the deterministic harness passes. The live harness
+should use a controlled repository fixture that requires reading more source
+than can be conveniently injected at once, then ask a realistic parent task
+whose answer requires evidence from at least 20 oversized files.
+
+Before generating a token, record:
+
+```text
+endpoint + exact model
+selected GPU UUID/model process placement
+available context window and token settings
+fixture source identities and requirements
+contract-selection manifest
+```
+
+The harness must refuse to launch when a capable, explicitly approved device
+cannot be verified. It must never select a weaker incidental accelerator simply
+because it is available.
+
+## Cross-tier acceptance matrix
+
+| Behavior | Small | Medium | Large |
+| --- | --- | --- | --- |
+| Current objective/source visibility | Required | Required | Required |
+| Full read remains first class | Required | Required | Required |
+| Branch extraction returns all requirements or justified unresolved items | Required | Required | Required |
+| Natural edit result reaches model | Required | Required | Required |
+| Generic exploration/todo/verifier gate | Off by default | Off by default | Off by default |
+| Optional concise coaching | Allowed, visible, non-fabricating | Advisory only | Advisory only |
+| Model-selected valid tool sequence | Preserve where possible | Preserve | Preserve |
+| Typed steering safe-boundary reconciliation | Required | Required | Required |
+
+## Measurements
+
+Record metrics that answer whether the system actually helped:
+
+- request-level source-body visibility and rehydration rate;
+- full-read versus shell-fallback rate;
+- extraction requirement coverage and unresolved-reason quality;
+- synthetic-denial count (target: zero outside real authority/atomic/schema/
+  resource boundaries);
+- repeated action/result signature rate, reset reasons, and actual recovery;
+- tool-result-to-next-action grounding rate;
+- verification freshness at completion; and
+- user-visible timeouts, packet/network load, GPU placement, and daemon boot
+  identity for live tests.
+
+Do not optimize a metric by hiding the behavior it measures. For example,
+reducing reread count by denying reads is a regression if source visibility was
+lost; reducing failures by relabelling truncation as success is a regression if
+the unresolved requirement remains.
+
+## Failure triage for the harness
+
+When an evaluation fails, preserve the request dump, tool trace, compacted
+audit, active evidence frame, selected contracts, model output, and fixture
+state. Determine whether the defect is in contract selection, frame rendering,
+context budgeting, tool dispatch, tool implementation, model behavior, or the
+test fixture. Fix the earliest causal layer and retain the test as a regression
+case.
+
+Do not weaken a behavioral assertion because a prompt change makes a model
+avoid the path. The scenario must still force the requested evidence,
+steering, edit, or recovery behavior to surface naturally.

package/prompts/reference-contracts/README.md

@@ -0,0 +1,84 @@
+# Agent Development Lifecycle Reference Contracts
+
+This directory is an operational transmutation of the contracts an engineering
+agent works under: how it receives work, uses context and tools, changes files,
+validates results, and hands work back. It preserves actionable obligations,
+branch conditions, and evidence requirements in language a medium or small
+model can follow.
+
+## Lifecycle map
+
+1. [Always-on core: load every turn](00_always_on_core.md)
+2. [Model-ready instruction set](00_model_ready_instruction_set.md)
+3. [Model-ready verification contract](01_model_ready_verification_contract.md)
+4. [Development decision tables](11_development_decision_tables.md)
+5. [Verification harness specification](12_verification_harness_spec.md)
+6. [Contextual contract loading index](13_contextual_contract_loading.md)
+7. [Runtime interaction protocol](14_runtime_interaction_protocol.md)
+8. [Authority, state, and steering](15_authority_state_and_steering.md)
+9. [Workspace, shell, and Git operations](16_workspace_shell_and_git.md)
+10. [Research, web, and documentation](17_research_web_and_documentation.md)
+11. [Modalities and rendering](18_modalities_and_rendering.md)
+12. [Task execution protocol](19_task_execution_protocol.md)
+13. [Quality review and handback](20_quality_review_and_handback.md)
+14. [Release and publication procedure](21_release_and_publication_procedure.md)
+15. [Live inference hardware procedure](22_live_inference_hardware_procedure.md)
+16. [Thinking and tool-call procedure](23_thinking_and_tool_call_procedure.md)
+17. [Local tool and file-edit procedure](24_local_tool_and_file_edit_procedure.md)
+18. [Runtime-injected state contract](25_runtime_injected_state_contract.md)
+19. [Skill procedure in full](26_skill_procedure_in_full.md)
+20. [Repository-local instruction procedure](27_repository_local_instruction_procedure.md)
+21. [Scope and precedence](00_scope_and_precedence.md)
+22. [Collaboration and communication](01_collaboration_and_communication.md)
+23. [Discovery and understanding](02_discovery_and_understanding.md)
+24. [Implementation and file safety](03_implementation_and_file_safety.md)
+25. [Tools, shell, and external state](04_tools_shell_and_external_state.md)
+26. [Evidence and verification](05_evidence_and_verification.md)
+27. [Context, memory, and compaction](06_context_memory_and_compaction.md)
+28. [Parallel and delegated work](07_parallel_and_delegated_work.md)
+29. [Skills and specialized procedures](08_skills_and_specialized_procedures.md)
+30. [Delivery, release, and completion](09_delivery_release_and_completion.md)
+31. [Observed development-cycle lessons](10_observed_cycle_lessons.md)
+32. [Security, sensitive data, and high-impact operations](28_security_and_sensitive_operations.md)
+33. [Untrusted content and provenance](29_untrusted_content_and_provenance.md)
+34. [Model tiers, autonomy, and intervention](30_model_tiers_autonomy_and_intervention.md)
+35. [Failure recovery and non-blocking observability](31_failure_recovery_and_nonblocking_observability.md)
+36. [System-message compilation and request assembly](32_system_message_compilation_spec.md)
+37. [Contract evaluation and regression design](33_contract_evaluation_and_regression_design.md)
+
+## Reading rule
+
+When two instructions conflict, follow the higher-precedence applicable one.
+When an instruction is ambiguous, preserve user intent, avoid irreversible
+state changes, and gather the smallest amount of authoritative evidence needed
+to proceed.
+
+## Loading policy
+
+Do not load this whole directory into every request. That would repeat rules,
+bury current evidence, and teach a model to quote process instead of doing
+work.
+
+- Load `00_always_on_core.md` every turn.
+- Load one or two phase-specific contracts when the task enters discovery,
+  implementation, verification, delegation, release, or external research.
+- Load `11_development_decision_tables.md` only when the agent is uncertain
+  whether to inspect, edit, test, ask, or stop.
+- Load `12_verification_harness_spec.md` when planning or reviewing a behavior
+  change.
+- Load `28_security_and_sensitive_operations.md` before handling credentials,
+  personal or customer data, privileged commands, public network operations,
+  production services, publication, or irreversible changes.
+- Load `29_untrusted_content_and_provenance.md` whenever source text, logs,
+  retrieved documents, tool output, web pages, or an external agent could be
+  mistaken for task authority.
+- Load `30_model_tiers_autonomy_and_intervention.md` when configuring
+  controller behavior for a model class or deciding whether the runtime may
+  intervene in a model-selected action.
+- Load `31_failure_recovery_and_nonblocking_observability.md` when designing
+  retries, loop detection, recovery, or operational telemetry.
+- Load `32_system_message_compilation_spec.md` and
+  `33_contract_evaluation_and_regression_design.md` only while implementing or
+  testing the request-assembly layer itself.
+- Keep current task evidence and tool results later in the request than these
+  contracts. Current evidence must win over generic procedure.