@bjoernrochel/sevro 0.1.0-rc.20 → 0.1.0-rc.22

package/docs/extension-protocol-v1.md

@@ -238,6 +238,15 @@ body exposure and successful Claude `Skill` calls.
 This replaces the retired Codex skill-read and Claude nested-skill observations.
 Explicit invocation receipts remain dispatch evidence and do not contribute to
 `observedSkills`; extensions must migrate activation checks to the common ID.
+Codex additionally supplies per-skill `skillExposures` and independent
+`collectionStatus` in that observation. Its optional
+`sevro.host.skill-exposure-claims` capability provides complete evidence only
+for listed positive full or partial body exposures, even when collection is
+partial. Extensions must check the requested skill and acceptable exposure
+status explicitly; this evidence cannot establish absence or primary selection.
+See the [exposure contract](native-hosts.md#skill-activation-evidence) for scope,
+unknown states and host limitations. Required aggregate activation evidence
+still needs a `complete` observation as before.
 Host adapters can also supply bounded evidence artifacts. The engine persists
 their bytes outside the candidate workspace before `evaluate`, then passes
 file URLs and digests through the existing artifact references. A required host

package/docs/installing.md

@@ -23,10 +23,10 @@ Local installation records the selected version in the project lockfile.
 
 ## Current release
 
-The current release is `0.1.0-rc.20`. Install this exact version:
+The current release is `0.1.0-rc.22`. Install this exact version:
 
 ```sh
-bun add --exact @bjoernrochel/sevro@0.1.0-rc.20
+bun add --exact @bjoernrochel/sevro@0.1.0-rc.22
 ```
 
 <!-- sevro-current-release:end -->

package/docs/native-hosts.md

@@ -90,8 +90,55 @@ Newly created or changed skill bodies are outside the prepared snapshot. A
 complete empty observation is limited to this scope and the supported local
 tool hook paths.
 
+Codex separates per-skill exposure from collection health in the same
+observation. `skillExposures` contains one `{ skill, status }` entry per prepared
+catalog entry, in catalog order, with qualified skill names:
+
+| Status      | Meaning                                                                 |
+| ----------- | ----------------------------------------------------------------------- |
+| `full`      | The complete prepared body was exposed, in one result or exact pages.   |
+| `partial`   | Recognizable fragments were exposed without establishing the full body. |
+| `none`      | No exposure was detected within a complete collection's matching scope. |
+| `unknown`   | No exposure was detected, but collection gaps prevent an absence claim. |
+| `ambiguous` | Body exposure could not distinguish identically registered bodies.      |
+
+`collectionStatus` is `complete`, `partial`, or `unavailable`. No parsed hook
+receipts means `unavailable`; otherwise collection gaps mean `partial`.
+Recognized partial bodies and ambiguous identities alone do not indicate a
+collection failure. Their existing `incompleteReasons` still keep the aggregate
+activation observation partial for compatibility. Other reason codes, including
+truncated call diagnostics, keep collection partial. Older observations without
+these new fields have unknown exposure and collection status; do not infer them
+from an empty `observedSkills` list.
+
+Exposure statuses use all parsed receipts before the diagnostic `calls` list is
+bounded. A proven `full` or `partial` exposure survives collection gaps, including
+an earlier pending call, missing lifecycle hooks or an invalid receipt tail.
+Full exposure takes precedence over fragments; ambiguous identities cannot
+establish exposure of a particular named skill. `none` does not establish that
+the skill was never used outside the prepared catalog and matching scope.
+Recognizable fragments may be shared by several skill bodies. These statuses
+describe content exposure to a tool caller, not instruction compliance, model
+visibility or selection order.
+
+The Codex adapter also advertises `sevro.host.skill-exposure-claims`. When at
+least one skill has `full` or `partial` exposure, it emits that observation with
+`completeness: complete`, `method: tool_result_body_match`,
+`scope: positive_exposure`, the original `collectionStatus`, and `skillExposures`
+containing only those positive entries. It is absent when no unambiguous
+positive exposure is available. Extensions can require this evidence for a
+specific positive assertion: accept only `full` for complete body exposure,
+or explicitly accept `partial` when recognizable exposure suffices. A `partial`
+entry is not a full activation. This receipt cannot establish absence, first
+selection, a complete sequence or successful execution; collection remains
+independently qualified. Existing activation and primary-selection requirements
+are unchanged. Claude's native `Skill` seam does not emit body-exposure statuses
+or these positive body-exposure claims.
+
 Codex observations include `incompleteReasons`: an empty array for a complete
-collection, or all applicable reason codes for a partial collection. Older
+aggregate activation observation, or all applicable reason codes for a partial
+aggregate observation. The separate `collectionStatus` isolates collection
+health from partial bodies and body ambiguity. Older
 observations without this field have unknown reasons; their missing details
 cannot be reconstructed from balanced native call/return counts.
 

package/docs/reading-results.md

@@ -24,6 +24,26 @@ Preparation/candidate/grading/evidence progress appears on stderr. Stage complet
 means the attempt ended; read the three result states to determine its outcome.
 See the [evaluation workflow](evaluation-workflow.md) for the complete progression.
 
+## Interpret skill exposure
+
+In retained Codex `sevro.host.skill-activations` observations, read
+`data.skillExposures` alongside `data.collectionStatus`. A skill may have `full`
+or `partial` exposure even when collection is incomplete. `none` means no
+exposure was detected within a complete collection's scope; `unknown` means
+collection gaps prevent that conclusion. `ambiguous` means identical bodies
+could not identify a named skill. Missing fields in older evidence remain
+unknown.
+
+Full exposure establishes that a tool caller received the complete prepared
+body, not that the model followed it. Partial exposure establishes only
+recognizable fragments. Cases that accept partial exposure must say so in
+their extension policy. The separate `sevro.host.skill-exposure-claims`
+observation supports positive exposure checks without treating gaps as absence
+or first selection. [Native host evidence](native-hosts.md#skill-activation-evidence)
+defines the fields and limits; Claude's native skill-tool evidence has a
+different scope. Task verdicts and existing activation requirements retain their
+current meaning.
+
 ## Summarize runs
 
 ```sh

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bjoernrochel/sevro",
-  "version": "0.1.0-rc.20",
+  "version": "0.1.0-rc.22",
   "license": "SEE LICENSE IN LICENSE",
   "description": "A standalone evaluation runner for agentic coding capabilities",
   "type": "module",

package/src/hosts/codex-state.ts

@@ -28,6 +28,7 @@ import {
   candidateTranscriptRoots,
 } from "../native-transcripts";
 import { prepareHookMounts } from "../runtime/hooks";
+import { requireNativeHookRoots } from "../native-transcript-state";
 import {
   requireRuntimeReadRoots,
   runtimeExecutableProtection,
@@ -102,9 +103,28 @@ export async function codexHookMounts(
     await rm(root, { recursive: true });
     await cp(curated, root, { recursive: true });
   }
+  await grantCodexHookReads(state, hooks.directories);
   return hooks;
 }
 
+/** Hooks advertise paths relative to these curated copies, not the install cache. */
+async function grantCodexHookReads(
+  state: CodexState,
+  roots: string[],
+): Promise<void> {
+  if (roots.length === 0) return;
+  await requireNativeHookRoots(roots, state.codexHome);
+  const path = join(state.codexHome, "config.toml");
+  const config = await readFile(path, "utf8");
+  const header = `[permissions.${state.profileId}.filesystem]\n`;
+  const grants = roots
+    .map((root) => `${JSON.stringify(root)} = "read"\n`)
+    .join("");
+  await writeFile(path, config.replace(header, header + grants), {
+    mode: 0o600,
+  });
+}
+
 async function privateCodexDirectories(stateRoot: string) {
   const codexHome = join(stateRoot, "codex-home"),
     parentHome = join(stateRoot, "home"),

package/src/hosts/codex.ts

@@ -45,6 +45,7 @@ import {
   prepareSkillTracking,
   skillTrackingObservation,
   skillActivationClaims,
+  skillExposureClaims,
   type SkillTracking,
 } from "./skill-hooks";
 import {
@@ -141,6 +142,7 @@ function codexCapabilities(
     "sevro.host.hooks",
     "sevro.host.skill-activations",
     "sevro.host.skill-activation-claims",
+    "sevro.host.skill-exposure-claims",
     ...(entrypoint === "app-server" ? ["sevro.host.native-goal"] : []),
     "sevro.codex.plugin-marketplace",
     "sevro.codex.explicit-invocation",
@@ -761,6 +763,7 @@ async function codexObservations(context: ExecutionContext, turns: TurnResult) {
     ...(explicit ? [explicit] : []),
     tracking,
     ...skillActivationClaims(tracking),
+    ...skillExposureClaims(tracking),
   ];
 }
 

package/src/hosts/skill-hooks.ts

@@ -449,6 +449,66 @@ export function skillActivationClaims(
   ];
 }
 
+export function skillExposureClaims(
+  observation: Awaited<ReturnType<typeof skillTrackingObservation>>,
+) {
+  const skillExposures = observation.data.skillExposures.filter(
+    (exposure) => exposure.status === "full" || exposure.status === "partial",
+  );
+  if (skillExposures.length === 0) return [];
+  return [
+    {
+      id: "sevro.host.skill-exposure-claims",
+      completeness: "complete" as const,
+      data: {
+        method: "tool_result_body_match",
+        scope: "positive_exposure",
+        collectionStatus: observation.data.collectionStatus,
+        skillExposures,
+      },
+    },
+  ];
+}
+
+function exposureSummary(
+  skills: HookSkill[],
+  events: SkillHookReceipt[],
+  collectionStatus: string,
+) {
+  const exposure = {
+    full: new Set(events.flatMap((event) => event.invocations)),
+    partial: new Set(incompleteSkills(events)),
+    ambiguous: new Set(events.flatMap((event) => event.ambiguousInvocations)),
+  };
+  return skills.map(({ invocation: skill }) => ({
+    skill,
+    status: skillExposureStatus(skill, exposure, collectionStatus),
+  }));
+}
+
+function skillExposureStatus(
+  skill: string,
+  exposure: { full: Set<string>; partial: Set<string>; ambiguous: Set<string> },
+  collectionStatus: string,
+) {
+  if (exposure.full.has(skill)) return "full" as const;
+  if (exposure.ambiguous.has(skill)) return "ambiguous" as const;
+  if (exposure.partial.has(skill)) return "partial" as const;
+  return collectionStatus === "complete"
+    ? ("none" as const)
+    : ("unknown" as const);
+}
+
+function exposureCollectionStatus(reasons: string[], eventCount: number) {
+  if (eventCount === 0) return "unavailable" as const;
+  return reasons.some(
+    (reason) =>
+      !["partial_skill_bodies", "ambiguous_skill_bodies"].includes(reason),
+  )
+    ? ("partial" as const)
+    : ("complete" as const);
+}
+
 export async function skillTrackingObservation(
   tracking: SkillTracking,
   sessionId: string | null,
@@ -489,6 +549,10 @@ export async function skillTrackingObservation(
     ...conditions.filter(([, applies]) => applies).map(([reason]) => reason),
   ];
   const complete = incompleteReasons.length === 0;
+  const collectionStatus = exposureCollectionStatus(
+    incompleteReasons,
+    events.length,
+  );
   return {
     id: "sevro.host.skill-activations",
     completeness: complete ? ("complete" as const) : ("partial" as const),
@@ -497,6 +561,12 @@ export async function skillTrackingObservation(
       visibility: "tool_result",
       scope: "prepared_skill_bodies",
       catalogComplete: tracking.configuration.catalogComplete,
+      collectionStatus,
+      skillExposures: exposureSummary(
+        tracking.configuration.skills,
+        events,
+        collectionStatus,
+      ),
       observedSkills: [...new Set(calls.flatMap((call) => call.invocations))],
       ...primaryEvidence(events, incompleteReasons),
       partialSkills,