open-research-protocol 0.4.7 → 0.4.9

package/docs/AGENT_LOOP.md

@@ -21,6 +21,9 @@ Use this loop when an AI agent is the primary operator of an ORP-enabled repo.
   - or `orp pack fetch --source <git-url> --pack-id <pack-id> --install-target . --json`
 - If the workflow depends on public Erdos data, sync it first:
   - `orp erdos sync --problem-id <id> --out-problem-dir <dir> --json`
+- If the task begins from a public YouTube link, normalize it first:
+  - `orp youtube inspect <youtube-url> --json`
+  - or `orp youtube inspect <youtube-url> --save --json` when the source artifact should stay with the repo
 
 ## 3. Run
 

package/docs/ORP_REASONING_KERNEL_AGENT_PILOT.md

@@ -0,0 +1,125 @@
+# ORP Reasoning Kernel Agent Pilot
+
+This document records the first live in-environment Codex pickup simulation for
+the ORP Reasoning Kernel.
+
+Supporting artifact:
+
+- [docs/benchmarks/orp_reasoning_kernel_agent_pilot_v0_1.json](/Volumes/Code_2TB/code/orp/docs/benchmarks/orp_reasoning_kernel_agent_pilot_v0_1.json)
+
+Supporting corpus and harness:
+
+- [examples/kernel/comparison/comparison-corpus.json](/Volumes/Code_2TB/code/orp/examples/kernel/comparison/comparison-corpus.json)
+- [scripts/orp-kernel-agent-pilot.py](/Volumes/Code_2TB/code/orp/scripts/orp-kernel-agent-pilot.py)
+
+## What This Pilot Measures
+
+This pilot asks a real fresh Codex session to act like a downstream handoff
+consumer with no repo context.
+
+For each matched case and condition, the agent sees only the artifact and must
+recover the **full required field set** for that artifact class:
+
+- `task`: `object`, `goal`, `boundary`, `constraints`, `success_criteria`
+- `decision`: `question`, `chosen_path`, `rejected_alternatives`, `rationale`, `consequences`
+- `hypothesis`: `claim`, `boundary`, `assumptions`, `test_path`, `falsifiers`
+- `experiment`: `objective`, `method`, `inputs`, `outputs`, `evidence_expectations`, `interpretation_limits`
+- `checkpoint`: `completed_unit`, `current_state`, `risks`, `next_handoff_target`, `artifact_refs`
+- `policy`: `scope`, `rule`, `rationale`, `invariants`, `enforcement_surface`
+- `result`: `claim`, `evidence_paths`, `status`, `interpretation_limits`, `next_follow_up`
+
+The agent is instructed to use `null` unless a field is explicit enough to
+carry forward into a canonical artifact without invention.
+
+This is a stronger standard than the earlier deterministic pickup proxy because
+it uses an actual fresh Codex session rather than a local rubric only.
+
+## What This Is And Is Not
+
+This is a **live internal agent simulation**, not a human handoff study.
+
+It does **not** prove:
+
+- human pickup speed
+- human clarification count
+- downstream implementation quality
+- cross-team or cross-model outcome superiority
+
+It **does** prove something narrower and important:
+
+- how much of the kernel’s required structure remains explicitly recoverable to
+  a fresh downstream agent
+- whether the kernel’s structural advantages survive contact with a real agent,
+  not just a deterministic local scorer
+
+## Current Result
+
+On the matched `7`-case, `5`-domain live Codex corpus, the current report
+shows:
+
+- kernel mean pickup score: `1.000`
+- generic checklist mean pickup score: `0.810`
+- free-form mean pickup score: `0.695`
+
+Pairwise result:
+
+- kernel beats free-form on `7/7` cases
+- kernel beats generic checklist on `4/7` cases and ties on `3/7`
+- generic checklist beats free-form on average, but with `1` loss and `2` ties
+
+Additional result:
+
+- kernel keeps all required fields explicitly recoverable on the matched corpus
+- kernel mean invention rate: `0.000`
+- free-form and generic checklist both leave recoverability gaps for at least
+  some artifact classes
+
+## Why This Matters
+
+This pilot matters because it is the first evidence layer that uses a real
+fresh agent rather than only local deterministic scoring.
+
+That gives us a stronger internal claim:
+
+- the kernel’s structural advantage is not decorative
+- the advantage remains visible to a downstream Codex session
+- the benefit is strongest against free-form artifacts
+- the generic checklist baseline is helpful, but it does not match the kernel’s
+  full recoverability
+
+The generic-checklist result is especially useful because it is **not** a
+strawman. It performs reasonably well and even ties the kernel on some cases.
+That makes the kernel’s wins more credible.
+
+## Honest Caveat
+
+This pilot still does not seal the kernel as a universally outcome-superior
+methodology.
+
+It is stronger than the earlier deterministic proxy, but it remains:
+
+- internal
+- model-specific
+- artifact-recoverability-focused
+
+The next real evidence bar is still:
+
+- blinded human pickup studies
+- downstream execution or review studies
+- broader cross-model or cross-team replication
+
+## Bottom Line
+
+The live agent pilot makes the kernel materially harder to dismiss.
+
+We now have evidence across three levels:
+
+- deterministic structural comparison
+- deterministic pickup proxy
+- live fresh-agent recoverability simulation
+
+Together, those support a strong internal claim:
+
+ORP kernel artifacts preserve more explicit canonical structure for downstream
+agents than free-form artifacts, and more than a generic checklist on average,
+while keeping full required-field recoverability on the matched corpus.

package/docs/ORP_REASONING_KERNEL_AGENT_REPLICATION.md

@@ -0,0 +1,97 @@
+# ORP Reasoning Kernel Agent Replication
+
+This document records the completed `10`-repeat full-corpus repeatability
+pilot for the live ORP kernel agent evaluation.
+
+Supporting artifact:
+
+- [docs/benchmarks/orp_reasoning_kernel_agent_replication_v0_2.json](/Volumes/Code_2TB/code/orp/docs/benchmarks/orp_reasoning_kernel_agent_replication_v0_2.json)
+
+Supporting harness:
+
+- [scripts/orp-kernel-agent-replication.py](/Volumes/Code_2TB/code/orp/scripts/orp-kernel-agent-replication.py)
+
+The harness now supports:
+
+- per-field stability tables
+- progress reporting for long runs
+- shard-and-merge execution for higher-repeat studies
+- confidence-interval reporting for repeated live runs
+
+## What This Measures
+
+The original live agent pilot showed that a fresh Codex session could recover
+the kernel’s required fields more completely than the simpler alternatives.
+
+This replication harness asks a different question:
+
+- does that result survive across repeated fresh-agent runs?
+
+The current completed pilot covers:
+
+- the full matched `7`-case corpus
+- `10` independent fresh Codex repetitions of that corpus
+
+## Current Result
+
+Across the repeated full matched corpus:
+
+- kernel mean pickup score: `1.000`
+- generic checklist mean pickup score: `0.790`
+- free-form mean pickup score: `0.718`
+
+Stability result:
+
+- kernel stayed above checklist on all `10/10` run-level repeats
+- kernel stayed above free-form on all `10/10` run-level repeats
+- kernel won `47/70` case-level comparisons against checklist and tied the other `23/70`
+- kernel won `70/70` case-level comparisons against free-form
+- kernel invention rate remained `0.000` across all `10` repeats
+
+Confidence and stability result:
+
+- kernel pickup CI95 half-width: `0.000`
+- generic checklist pickup CI95 half-width: `0.023`
+- free-form pickup CI95 half-width: `0.007`
+- kernel per-field stability gap stayed at `0.000` for every tracked required field
+
+The per-field tables are especially useful because they show where the simpler
+alternatives still degrade under repetition:
+
+- checklist repeatedly misses `decision.question` and `decision.rationale`
+- checklist repeatedly misses `experiment.inputs`, `experiment.outputs`, and `hypothesis.falsifiers`
+- checklist still struggles with `policy.enforcement_surface` and `task.object`
+- free-form most often drops `experiment.outputs`, `result.evidence_paths`, and sometimes `checkpoint.next_handoff_target`
+
+## Why This Matters
+
+The replication pilot is now much stronger than a single-case smoke, because
+it covers the full matched corpus, repeats it `10` times with fresh sessions,
+and exposes field-level stability instead of only overall means.
+
+That strengthens the evidence story in an agent-first way:
+
+- the kernel advantage can survive fresh-session variation
+- the kernel continues to look safe on invention
+- the checklist baseline is helpful, but it still leaves repeatable structural
+  holes that the kernel does not
+
+## Honest Boundary
+
+This is now a strong internal replication pilot, but it is still not a final
+external replication study.
+
+What remains:
+
+- compare multiple agent models when practical
+- run blinded human or mixed human/agent handoff replications
+- extend the repeated continuation benchmark in the same style
+
+## Bottom Line
+
+The replication pilot makes the live agent story much more credible:
+
+the kernel’s recoverability advantage appears stable under repeated fresh-agent
+execution across the full matched corpus, not only on a single representative
+case, and its field-level stability now looks materially cleaner than both
+free-form and checklist alternatives.

package/docs/ORP_REASONING_KERNEL_CANONICAL_CONTINUATION_PILOT.md

@@ -0,0 +1,100 @@
+# ORP Reasoning Kernel Canonical Continuation Pilot
+
+This document records the first full-corpus live canonical continuation pilot
+for the ORP Reasoning Kernel.
+
+Supporting artifact:
+
+- [docs/benchmarks/orp_reasoning_kernel_canonical_continuation_v0_1.json](/Volumes/Code_2TB/code/orp/docs/benchmarks/orp_reasoning_kernel_canonical_continuation_v0_1.json)
+
+Supporting harness:
+
+- [scripts/orp-kernel-canonical-continuation.py](/Volumes/Code_2TB/code/orp/scripts/orp-kernel-canonical-continuation.py)
+
+## What This Pilot Measures
+
+The earlier continuation pilot asked whether a fresh downstream agent could
+continue work safely in a general sense.
+
+This harder pilot asks a more demanding question:
+
+- can a fresh downstream agent turn the source artifact into the next canonical
+  kernel task artifact?
+
+For each source artifact, the agent must produce a real task-shaped follow-on
+object with:
+
+- `object`
+- `goal`
+- `boundary`
+- `constraints`
+- `success_criteria`
+
+The benchmark then scores:
+
+- field alignment against the expected next canonical task
+- unsupported invention
+- whether missing structure is reported explicitly instead of silently filled
+
+## Current Result
+
+On the matched `7`-case, `5`-domain live canonical continuation corpus:
+
+- kernel mean total score: `0.738`
+- generic checklist mean total score: `0.663`
+- free-form mean total score: `0.484`
+
+Invention result:
+
+- kernel mean invention rate: `0.386`
+- generic checklist mean invention rate: `0.495`
+- free-form mean invention rate: `0.748`
+
+Pairwise result:
+
+- kernel beat free-form on `7/7` cases
+- kernel beat generic checklist on `4/7` cases
+- kernel tied generic checklist on `1/7` cases
+- kernel lost to generic checklist on `2/7` cases
+
+## Why This Matters
+
+This pilot is stronger than the softer continuation benchmark because it asks
+the downstream agent to produce a real canonical artifact rather than only a
+safe next action.
+
+The result is more discriminative:
+
+- free-form falls off sharply once a real next artifact must be constructed
+- checklist stays meaningfully competitive
+- kernel still leads on aggregate score and invention control
+
+That makes the evidence more believable, not less. The kernel is not winning
+against a straw baseline here.
+
+## Honest Boundary
+
+The canonical continuation pilot does not prove universal methodological
+superiority.
+
+It does show a narrower and important result:
+
+- kernel structure gives a fresh downstream agent a stronger base for producing
+  the next canonical task artifact than free-form notes
+- checklist can still be strong on some cases, so the kernel advantage is real
+  but not absolute
+
+What remains:
+
+- repeat the canonical continuation benchmark across multiple fresh runs
+- add per-field stability summaries for canonical continuation itself
+- compare multiple models when practical
+
+## Bottom Line
+
+The harder continuation benchmark makes the kernel evidence more grounded:
+
+the kernel still wins on average, stays safest on invention, and clearly beats
+free-form when the downstream task is “produce the next canonical artifact,”
+while revealing that a strong checklist baseline remains competitive enough to
+matter on some cases.

package/docs/ORP_REASONING_KERNEL_COMPARISON_PILOT.md

@@ -0,0 +1,116 @@
+# ORP Reasoning Kernel Comparison Pilot
+
+This document records the first in-repo side-by-side comparison between three
+artifact styles:
+
+1. free-form artifact writing
+2. generic checklist artifact writing
+3. ORP typed kernel artifact writing
+
+Supporting artifact:
+
+- [docs/benchmarks/orp_reasoning_kernel_comparison_v0_1.json](/Volumes/Code_2TB/code/orp/docs/benchmarks/orp_reasoning_kernel_comparison_v0_1.json)
+
+Supporting corpus and harness:
+
+- [examples/kernel/comparison/comparison-corpus.json](/Volumes/Code_2TB/code/orp/examples/kernel/comparison/comparison-corpus.json)
+- [scripts/orp-kernel-comparison.py](/Volumes/Code_2TB/code/orp/scripts/orp-kernel-comparison.py)
+
+## What This Pilot Measures
+
+This pilot is intentionally narrow.
+
+It does **not** measure downstream execution quality, human review outcomes, or
+team handoff performance directly.
+
+It **does** measure structural clarity on a matched internal corpus using a
+deterministic rubric.
+
+Each condition is scored on:
+
+- artifact type clarity
+- objective clarity
+- limits clarity
+- evaluation clarity
+- handoff readiness
+- class-specific completeness
+
+The corpus spans:
+
+- `7` cases
+- `5` domains
+- all `7` v0.1 kernel artifact classes
+
+## Why This Comparison Matters
+
+The current technical validation package proves that the kernel works.
+
+What it did not yet prove by itself was whether the kernel offers more usable
+structure than simpler alternatives on the same prompts.
+
+This pilot addresses that specific gap by comparing matched artifacts instead
+of evaluating the kernel in isolation.
+
+## Current Result
+
+On the matched internal corpus, the current report shows:
+
+- kernel mean total score: `1.000`
+- generic checklist mean total score: `0.687`
+- free-form mean total score: `0.275`
+
+Pairwise result:
+
+- kernel beats generic checklist on all `7/7` cases
+- kernel beats free-form on all `7/7` cases
+- generic checklist beats free-form on all `7/7` cases
+
+Additional result:
+
+- kernel class-specific completeness mean: `1.000`
+- generic checklist class-specific completeness mean: `0.657`
+- free-form class-specific completeness mean: `0.328`
+
+## What This Supports
+
+This pilot supports a **narrow but real** claim:
+
+On a matched internal comparison corpus, ORP kernel artifacts preserve more
+explicit structural coverage than both free-form artifacts and a generic
+checklist alternative.
+
+That is stronger than a pure rationale-only claim.
+
+## What This Does Not Yet Support
+
+This pilot does **not** prove that the kernel:
+
+- improves downstream implementation success
+- improves human pickup speed in live handoffs
+- reduces rework in actual projects
+- is universally superior across all teams or domains
+
+Those still require the larger studies in
+[docs/ORP_REASONING_KERNEL_EVALUATION_PLAN.md](/Volumes/Code_2TB/code/orp/docs/ORP_REASONING_KERNEL_EVALUATION_PLAN.md).
+
+## Why The Scoring Is Structured This Way
+
+The scoring deliberately rewards explicit structure rather than latent meaning
+buried in prose.
+
+That reflects the actual ORP goal:
+
+- humans can stay loose at the boundary
+- but promotable repository artifacts should remain structurally legible
+
+So this pilot is not a writing-style contest. It is a test of how much
+canonically useful structure survives into the artifact itself.
+
+## Bottom Line
+
+This comparison pilot does not seal the kernel as a universally outcome-better
+methodology.
+
+It does provide the first comparative evidence that the kernel is not only
+valid in isolation, but also materially stronger as a structural artifact
+surface than the simpler alternatives tested here.

package/docs/ORP_REASONING_KERNEL_CONTINUATION_PILOT.md

@@ -0,0 +1,86 @@
+# ORP Reasoning Kernel Continuation Pilot
+
+This document records the first full-corpus live continuation pilot for the ORP
+Reasoning Kernel.
+
+Supporting artifact:
+
+- [docs/benchmarks/orp_reasoning_kernel_continuation_v0_1.json](/Volumes/Code_2TB/code/orp/docs/benchmarks/orp_reasoning_kernel_continuation_v0_1.json)
+
+Supporting harness:
+
+- [scripts/orp-kernel-continuation-pilot.py](/Volumes/Code_2TB/code/orp/scripts/orp-kernel-continuation-pilot.py)
+
+Related harder benchmark:
+
+- [docs/ORP_REASONING_KERNEL_CANONICAL_CONTINUATION_PILOT.md](/Volumes/Code_2TB/code/orp/docs/ORP_REASONING_KERNEL_CANONICAL_CONTINUATION_PILOT.md)
+
+## What This Pilot Measures
+
+The earlier live agent pilot measured recoverability: can a fresh downstream
+agent reconstruct the required kernel fields?
+
+This continuation pilot asks a slightly different question:
+
+- can a fresh downstream agent continue the work safely?
+
+For each artifact, the agent must:
+
+- propose a recommended next action
+- identify handoff-critical fields to carry forward
+- surface what is explicitly missing instead of inventing it
+
+The scoring emphasizes:
+
+- carry-forward coverage
+- invention rate
+- whether a concrete next action is present
+
+This remains the softer downstream benchmark. The harder follow-on test is the
+canonical continuation pilot, where the downstream agent must produce the next
+full kernel task artifact rather than only a safe continuation.
+
+## Current Result
+
+On the matched `7`-case, `5`-domain live continuation corpus:
+
+- kernel continuation score: `1.000`
+- generic checklist continuation score: `0.984`
+- free-form continuation score: `0.968`
+
+Invention result:
+
+- kernel invention rate: `0.000`
+- generic checklist invention rate: `0.000`
+- free-form invention rate: `0.048`
+
+## Why This Matters
+
+This pilot pushes the evaluation one step closer to real downstream agent work.
+
+The continuation corpus suggests:
+
+- the kernel clearly supports safer continuation than free-form artifacts
+- the kernel slightly exceeds the generic checklist on continuation score at
+  corpus level, while never doing worse on any matched case
+- the kernel preserves a safety advantage by keeping invention at `0.000`
+
+## Honest Boundary
+
+This is now a real full-corpus pilot, but it is still not a full external
+continuation study.
+
+What remains:
+
+- replicate across more fresh sessions
+- compare additional agent models when practical
+- extend from artifact continuation to fuller downstream execution quality
+
+## Bottom Line
+
+The continuation pilot strengthens the kernel in a well-rounded way:
+
+it suggests the kernel is not only easier to recover, but also a strong and
+safe surface for downstream agent continuation across the matched corpus, while
+showing that a generic checklist remains competitive enough to be a meaningful
+baseline rather than a strawman.