@hassangameryt/freeflow 0.5.0 → 0.7.0

package/skills/evaluate-skill/references/review-and-revision.md

@@ -1,51 +1,92 @@
 # Review And Revision
 
-Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning still needs judgment, or a revision/rerun boundary must be chosen.
+Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning remains unresolved, the failed boundary must be classified, or revision and rerun scope must be chosen.
 
 ## Use The Strongest Evidence
 
-1. Filesystem state, typed workspace manifests, diffs, and changed paths.
-2. Structured events, tool activity, usage, process results, logs, and saved reports.
-3. Final response.
-4. Full transcript when earlier surfaces cannot answer the question.
-5. Subject self-assessment.
+Prefer, in order:
 
-Lower-priority evidence cannot repair a contradiction above it. If the response claims success but a required file is absent, preserve both facts and grade the file observation.
+1. filesystem state, typed workspace manifests, diffs, and changed paths;
+2. structured events, tool activity, usage, process results, logs, and saved reports;
+3. final response;
+4. full transcript when stronger surfaces cannot answer the question;
+5. subject self-assessment.
+
+Lower-priority evidence cannot repair a contradiction above it.
+
+If a response claims success but a required file is absent, preserve both facts and judge from the stronger artifact evidence.
 
 ## Keep Deterministic Facts Separate
 
-Built-in checks cover exact reads, typed paths, changed paths, file or response text, JSON states, valid turn scope, and factual transitions. Missing evidence remains unavailable. A failed check remains behavioral evidence; it does not relabel a complete run as infrastructure failure.
+Built-in checks establish exact observable facts. Missing evidence remains unavailable. A failed check remains behavioral evidence and does not relabel a complete run as infrastructure failure.
+
+The active agent or user may judge:
 
-The active agent or user may judge reasoning quality, authority preservation, architectural fitness, recommendation quality, or semantic completeness. Record cited evidence, uncertainty, and limits outside the canonical run and deterministic grade. Do not turn that judgment into evaluator-owned readiness state.
+- reasoning quality;
+- authority preservation;
+- architectural fitness;
+- recommendation quality;
+- semantic completeness.
 
-## Diagnose Before Editing
+Keep semantic judgment, cited evidence, uncertainty, and limits outside canonical run and deterministic grade state.
 
-Classify the failed boundary:
+Do not turn semantic judgment into evaluator-owned readiness or promotion status.
+
+## Diagnose The Failed Boundary
+
+Classify before revising:
 
 - activation timing;
 - description language unavailable before body read;
 - first-read body comprehension;
 - undeclared, undelivered, unread, or ambient-only dependency;
-- normal guidance hidden behind an unclear resource boundary;
+- normal guidance hidden behind the wrong resource boundary;
 - nearby behavior that hijacks the task;
 - retained-use or multi-turn failure;
-- wording, placement, example, or stop-condition pressure;
-- fixture, environment, isolation, host, or deterministic-check failure.
+- wording, placement, example, repetition, or stop-condition pressure;
+- fixture, environment, isolation, host, runner, or deterministic-check failure.
+
+Do not add broad prose before identifying which boundary failed.
 
-Do not add broad prose before identifying the failed boundary.
+## Return Revision To Write Skill
 
-## Revise One Pressure Point
+When evidence supports a skill change:
 
-Change one measured point when possible: description cues, first-read wording, rule placement, dependency declaration, resource boundary, example, stop condition, or the group when evidence proves the evaluation is wrong.
+1. name the exact failed boundary;
+2. cite the canonical evidence;
+3. identify one measured pressure point;
+4. preserve unrelated instructions and fixed criteria;
+5. return the revision target to [Write Skill](../../write-skill/SKILL.md).
 
-Keep unrelated instructions and fixed criteria stable. If the criterion changes, the old result no longer answers the same question.
+Write Skill owns package revision.
+
+Evaluate Skill does not edit the skill or replace its baseline while evaluating it.
+
+When evidence proves the group, fixture, or criterion is wrong, revise the evaluation instead. Preserve the old result and state that it no longer answers the revised question.
 
 ## Rerun The Complete Group
 
-After a measured skill change or resolved infrastructure cause, run the complete fixed group with both variants. Do not rerun only the preferred variant, reuse a settled control as a new sample, adopt partial evidence as comparable, or rerun merely to erase a failure.
+After a measured skill revision or resolved infrastructure cause:
+
+- rerun the complete fixed group;
+- run both baseline and candidate;
+- keep every result visible;
+- do not reuse an earlier baseline run as a fresh counterpart;
+- do not rerun only the preferred variant;
+- do not use partial evidence as a comparable full result;
+- do not rerun merely to erase failure.
 
-For variance, use independent complete invocations and keep every result visible.
+For variance, use independent complete invocations rather than repeated preferred variants.
 
 ## Report
 
-Report the failed boundary and evidence, exact revision or evaluation correction, criteria that stayed fixed, old and new result paths, and remaining unsupported claims. The user decides the next authoring or use action.
+Report:
+
+- failed boundary and evidence;
+- exact skill revision or evaluation correction;
+- criteria that remained fixed;
+- old and new result paths;
+- deterministic transitions;
+- remaining unsupported claims.
+
+The user decides whether to revise further, use, publish, or reject the skill.

package/skills/evaluate-skill/scripts/lib/definitions.mjs

@@ -4,6 +4,20 @@ import path from "node:path";
 const ID = /^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/;
 const EVALUATION_TYPES = new Set(["description", "body", "end-to-end"]);
 const VARIANTS = ["baseline", "candidate"];
+const EVALUATION_HOSTS = new Set(["pi", "piflow"]);
+const RESERVED_ENVIRONMENT_KEYS = new Set([
+  "PATH",
+  "HOME",
+  "PWD",
+  "OLDPWD",
+  "SHELL",
+  "BASH_ENV",
+  "NODE_OPTIONS",
+  "NODE_PATH",
+  "PI_PACKAGE_DIR",
+  "PIFLOW_CODING_AGENT_DIR",
+  "PIFLOW_CODING_AGENT_SESSION_DIR",
+]);
 
 export class DefinitionError extends Error {
   constructor(code, message, definitionPath = null) {
@@ -129,6 +143,102 @@ function validateEnvironment(value, label, definitionPath) {
   }
 }
 
+function validateRuntime(value, definitionPath) {
+  const runtime = requireExactKeys(
+    value,
+    ["host", "extensions", "environment", "session"],
+    [],
+    "group.runtime",
+    definitionPath,
+  );
+  if (!EVALUATION_HOSTS.has(runtime.host)) {
+    fail("invalid-definition", "group.runtime.host must be pi or piflow", definitionPath);
+  }
+  if (!Array.isArray(runtime.extensions)) {
+    fail("invalid-definition", "group.runtime.extensions must be an array", definitionPath);
+  }
+  const bundleIds = new Set();
+  for (const [index, entry] of runtime.extensions.entries()) {
+    const bundle = requireExactKeys(
+      entry,
+      ["entry", "resources"],
+      [],
+      `group.runtime.extensions[${index}]`,
+      definitionPath,
+    );
+    requireSafeRelativePath(bundle.entry, `group.runtime.extensions[${index}].entry`, definitionPath);
+    const resources = requireStringArray(
+      bundle.resources,
+      `group.runtime.extensions[${index}].resources`,
+      definitionPath,
+      { paths: true, unique: true },
+    );
+    if (resources.length === 0) {
+      fail("invalid-definition", `group.runtime.extensions[${index}].resources must not be empty`, definitionPath);
+    }
+    const bundleId = `${bundle.entry}\u0000${resources.join("\u0000")}`;
+    if (bundleIds.has(bundleId)) {
+      fail("invalid-definition", "group.runtime.extensions contains duplicate bundles", definitionPath);
+    }
+    bundleIds.add(bundleId);
+  }
+  if (typeof runtime.session !== "boolean") {
+    fail("invalid-definition", "group.runtime.session must be a boolean", definitionPath);
+  }
+  const environment = requireExactKeys(
+    runtime.environment,
+    ["literal", "inherit"],
+    [],
+    "group.runtime.environment",
+    definitionPath,
+  );
+  const literal = requireObject(environment.literal, "group.runtime.environment.literal", definitionPath);
+  const inherited = requireStringArray(environment.inherit, "group.runtime.environment.inherit", definitionPath, {
+    unique: true,
+  });
+  const literalKeys = new Set(Object.keys(literal).map((key) => key.toUpperCase()));
+  for (const [key, value] of Object.entries(literal)) {
+    const normalizedKey = key.toUpperCase();
+    if (
+      !/^[A-Za-z_][A-Za-z0-9_]*$/.test(key) ||
+      isBlockedEnvironmentKey(normalizedKey) ||
+      /TOKEN|SECRET|PASSWORD|API[_-]?KEY|PRIVATE[_-]?KEY|CREDENTIAL/i.test(key)
+    ) {
+      fail(
+        "invalid-definition",
+        `group.runtime.environment.literal contains a reserved or secret-like key: ${key}`,
+        definitionPath,
+      );
+    }
+    if (typeof value !== "string") {
+      fail("invalid-definition", `group.runtime.environment.literal.${key} must be a string`, definitionPath);
+    }
+  }
+  for (const key of inherited) {
+    const normalizedKey = key.toUpperCase();
+    if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(key) || isBlockedEnvironmentKey(normalizedKey)) {
+      fail("invalid-definition", `group.runtime.environment.inherit contains an invalid key: ${key}`, definitionPath);
+    }
+    if (literalKeys.has(normalizedKey)) {
+      fail(
+        "invalid-definition",
+        `group.runtime.environment key is declared as both literal and inherited: ${key}`,
+        definitionPath,
+      );
+    }
+  }
+}
+
+function isBlockedEnvironmentKey(normalizedKey) {
+  return (
+    RESERVED_ENVIRONMENT_KEYS.has(normalizedKey) ||
+    normalizedKey.startsWith("SKILL_EVAL_") ||
+    normalizedKey.startsWith("PI_") ||
+    normalizedKey.startsWith("LD_") ||
+    normalizedKey.startsWith("DYLD_")
+  );
+}
+
 function validateExpectations(value, definitionPath) {
   if (!Array.isArray(value)) fail("invalid-definition", "group.expectations must be an array", definitionPath);
   const ids = new Set();
@@ -164,7 +274,7 @@ function validateGroup(value, definitionPath) {
       "expectations",
       "review_questions",
     ],
-    ["model"],
+    ["model", "runtime"],
     "group",
     definitionPath,
   );
@@ -177,12 +287,13 @@ function validateGroup(value, definitionPath) {
   validateInput(group.input, definitionPath);
   if (group.fixture !== null) requireSafeRelativePath(group.fixture, "group.fixture", definitionPath);
   requireStringArray(group.tools, "group.tools", definitionPath, { unique: true });
+  if (Object.hasOwn(group, "runtime")) validateRuntime(group.runtime, definitionPath);
   const variants = requireExactKeys(group.variants, VARIANTS, [], "group.variants", definitionPath);
   for (const variant of VARIANTS) validateEnvironment(variants[variant], `group.variants.${variant}`, definitionPath);
-  if (variants.candidate.target === null) {
+  if (variants.candidate.target === null && variants.candidate.skills.length > 0) {
     fail(
       "invalid-definition",
-      "group.variants.candidate.target must identify the candidate target skill",
+      "group.variants.candidate.target must identify the candidate target skill when candidate skills are declared",
       definitionPath,
     );
   }

package/skills/evaluate-skill/scripts/lib/grade-interaction.mjs

@@ -1,4 +1,5 @@
-import { realpath } from "node:fs/promises";
+import { createHash } from "node:crypto";
+import { readFile, realpath } from "node:fs/promises";
 import path from "node:path";
 
 import {
@@ -16,6 +17,10 @@ import {
 } from "./grade-support.mjs";
 
 const ACTIVATION_EXPECTATIONS = new Set(["never", "on-turn", "by-turn", "not-before-turn"]);
+const TOOL_CALL_EXPECTATIONS = new Set(["called", "succeeded", "failed", "not-called"]);
+const CONTEXT_TEXT_EXPECTATIONS = new Set(["contains", "not-contains", "equals"]);
+const CONTEXT_SURFACES = new Set(["system-prompt", "provider-context"]);
+const CONTEXT_REQUEST_SELECTORS = new Set(["first", "last"]);
 
 export async function gradeSkillRead(group, runs, expectation) {
   const turn = expectation.turn ?? null;
@@ -100,6 +105,95 @@ export async function gradeResourceRead(group, runs, expectation) {
   );
 }
 
+export async function gradeContextText(group, runs, expectation) {
+  const turn = expectation.turn ?? null;
+  const surface = expectation.surface ?? "system-prompt";
+  const request = expectation.request ?? "last";
+  if (
+    !validBodyExpectation(group, expectation, turn) ||
+    !CONTEXT_SURFACES.has(surface) ||
+    (!CONTEXT_REQUEST_SELECTORS.has(request) && !Number.isInteger(request)) ||
+    (Number.isInteger(request) && request < 1) ||
+    !CONTEXT_TEXT_EXPECTATIONS.has(expectation.expect) ||
+    typeof expectation.value !== "string"
+  ) {
+    return { error: "invalid context-text expectation" };
+  }
+
+  const expected = { expect: expectation.expect, value: expectation.value, surface, request, turn };
+  const run = runs[expectation.variant];
+  if (!run || run.state !== "complete") return unavailableCheck(expectation, expected);
+  const observed = await contextObservation(run, turn, surface, request);
+  if (observed === null) return unavailableCheck(expectation, expected);
+  const passed = textMatches(expectation.expect, expectation.value, observed.text);
+  return completedCheck(expectation, expected, compactContextObservation(observed, passed), passed);
+}
+
+export async function gradeToolCall(group, runs, expectation) {
+  const turn = expectation.turn ?? null;
+  const contains = expectation.argumentContains ?? [];
+  const notContains = expectation.argumentNotContains ?? [];
+  if (
+    !validBodyExpectation(group, expectation, turn) ||
+    typeof expectation.tool !== "string" ||
+    expectation.tool.trim() === "" ||
+    !TOOL_CALL_EXPECTATIONS.has(expectation.expect) ||
+    !Array.isArray(contains) ||
+    !contains.every((value) => typeof value === "string") ||
+    !Array.isArray(notContains) ||
+    !notContains.every((value) => typeof value === "string")
+  ) {
+    return { error: "invalid tool-call expectation" };
+  }
+
+  const expected = {
+    tool: expectation.tool,
+    expect: expectation.expect,
+    turn,
+    argumentContains: contains,
+    argumentNotContains: notContains,
+  };
+  const run = runs[expectation.variant];
+  if (!run || run.state !== "complete") return unavailableCheck(expectation, expected);
+
+  const calls = toolCalls(run, turn).filter((call) => call.toolName === expectation.tool);
+  const matchingCalls = calls.filter((call) => {
+    const argumentsText = JSON.stringify(call.args ?? null);
+    return (
+      contains.every((value) => argumentsText.includes(value)) &&
+      notContains.every((value) => !argumentsText.includes(value))
+    );
+  });
+  const succeededCalls = matchingCalls.filter((call) => call.completed === true && call.isError !== true);
+  const failedCalls = matchingCalls.filter((call) => call.completed === true && call.isError === true);
+  let passed;
+  if (expectation.expect === "called") {
+    passed = matchingCalls.length > 0;
+  } else if (expectation.expect === "succeeded") {
+    passed = succeededCalls.length > 0;
+  } else if (expectation.expect === "failed") {
+    passed = failedCalls.length > 0;
+  } else if (contains.length > 0 || notContains.length > 0) {
+    passed = matchingCalls.length === 0;
+  } else {
+    passed = calls.length === 0;
+  }
+  const observed = {
+    tool: expectation.tool,
+    turn,
+    callCount: calls.length,
+    matchingCallCount: matchingCalls.length,
+    succeededCount: succeededCalls.length,
+    failedCount: failedCalls.length,
+    calls: calls.map((call) => ({
+      args: call.args ?? null,
+      completed: call.completed ?? null,
+      isError: call.isError ?? null,
+    })),
+  };
+  return completedCheck(expectation, expected, observed, passed);
+}
+
 export async function gradeResponseText(group, runs, expectation) {
   const turn = expectation.turn ?? null;
   if (
@@ -123,6 +217,67 @@ export async function gradeResponseText(group, runs, expectation) {
   );
 }
 
+function compactContextObservation(observation, matched) {
+  return {
+    surface: observation.surface,
+    turn: observation.turn,
+    request: observation.request,
+    characters: observation.characters,
+    sha256: createHash("sha256").update(observation.text, "utf8").digest("hex"),
+    matched,
+    preview: observation.text.slice(0, 240),
+  };
+}
+
+async function contextObservation(run, turn, surface, request) {
+  const observations = await loadContextObservations(run);
+  if (observations === null) return null;
+  const matching = observations.filter((entry) => entry.surface === surface);
+  const inTurn = turn === null ? matching : matching.filter((entry) => entry.turn === turn);
+  if (inTurn.length === 0) return null;
+  let observation;
+  if (request === "first") observation = inTurn[0];
+  else if (request === "last") observation = inTurn.at(-1);
+  else observation = inTurn.find((entry) => (entry.requestInTurn ?? 1) === request);
+  const text = surface === "system-prompt" ? (observation?.text ?? observation?.systemPrompt) : observation?.text;
+  if (!observation || typeof text !== "string") return null;
+  return {
+    surface,
+    turn: observation.turn ?? turn,
+    request: observation.requestInTurn ?? 1,
+    characters: observation.characters ?? text.length,
+    text,
+  };
+}
+
+async function loadContextObservations(run) {
+  const artifact = run.contextObservationArtifact;
+  if (!artifact || typeof artifact.path !== "string" || typeof artifact.sha256 !== "string") return null;
+  const contents = await readFile(artifact.path, "utf8");
+  const actualHash = createHash("sha256").update(contents, "utf8").digest("hex");
+  if (actualHash !== artifact.sha256) throw new Error("context observation artifact changed after run persistence");
+  const observations = [];
+  for (const line of contents.split("\n").filter(Boolean)) {
+    try {
+      observations.push(JSON.parse(line));
+    } catch (error) {
+      throw new Error(
+        `context observation artifact is malformed: ${error instanceof Error ? error.message : String(error)}`,
+      );
+    }
+  }
+  return observations;
+}
+
+function toolCalls(run, turn) {
+  if (turn === null) {
+    if (Array.isArray(run.turns)) return run.turns.flatMap((entry) => entry.toolActivity ?? []);
+    return Array.isArray(run.toolActivity) ? run.toolActivity : [];
+  }
+  if (!Array.isArray(run.turns)) return turn === 1 && Array.isArray(run.toolActivity) ? run.toolActivity : [];
+  return run.turns.find((entry) => entry.turn === turn)?.toolActivity ?? [];
+}
+
 function activationMatches(expectation, observed) {
   if (expectation.expect === "never") return !observed.targetRead;
   if (!Number.isInteger(expectation.turn) || expectation.turn < 1) return false;

package/skills/evaluate-skill/scripts/lib/grade.mjs

@@ -1,5 +1,11 @@
 import { gradeComparisons } from "./grade-comparison.mjs";
-import { gradeResourceRead, gradeResponseText, gradeSkillRead } from "./grade-interaction.mjs";
+import {
+  gradeContextText,
+  gradeResourceRead,
+  gradeResponseText,
+  gradeSkillRead,
+  gradeToolCall,
+} from "./grade-interaction.mjs";
 import { gradeChangedPaths, gradeFileText, gradeJson, gradePath } from "./grade-workspace.mjs";
 
 const GRADERS = new Map([
@@ -10,6 +16,8 @@ const GRADERS = new Map([
   ["file-text", gradeFileText],
   ["json", gradeJson],
   ["response-text", gradeResponseText],
+  ["context-text", gradeContextText],
+  ["tool-call", gradeToolCall],
 ]);
 
 /**