@hassangameryt/freeflow 0.3.0 → 0.5.0

package/skills/evaluate-skill/references/execution-and-evidence.md

@@ -0,0 +1,85 @@
+# Execution And Evidence
+
+Read this when operating `run|view`, resolving paths, interpreting states, or reasoning about Pi execution, isolation, persistence, cancellation, and safeguards.
+
+## Command And Path Contract
+
+Run commands from the repository or fixture root that owns the definitions:
+
+```text
+node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate]
+node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
+```
+
+- The current working directory is the definition root and result root.
+- The `run` target resolves from that directory.
+- Suite group references resolve relative to the suite file.
+- Fixture, skill, and context declarations resolve from the definition root.
+- Results are stored under `<cwd>/.skill-eval/runs/<result-id>`.
+- A bare `view` target is tried first as a stored result ID, then as a path from the working directory.
+- An absolute or path-like `view` target resolves as a directory path.
+
+Examples:
+
+```text
+skill-eval run .skill-eval/my-skill/suite.json
+skill-eval run .skill-eval/my-skill/suite.json --variant candidate
+skill-eval view 20260720173209-a3649b8b
+skill-eval view 20260720173209-a3649b8b --group 2 --variant candidate
+skill-eval view /absolute/path/to/result
+```
+
+With no selectors, `run` or `view` selects every suite group and both variants. `--group` is invalid for a direct group definition/result.
+
+## Current Execution Boundary
+
+`run` executes description and body groups. `end-to-end` is a reserved definition type and is rejected before subject execution. Description groups allow no tools or `read`; body groups allow no tools or path-guarded `read`, `write`, and `edit`. `bash` and definition-supplied commands are unsupported.
+
+Use fresh JSON-mode execution for one-shot descriptions. Use one persistent RPC process per selected variant for ordered description turns and all body groups. RPC correlates responses, waits for `agent_settled`, disables automatic retry and compaction, preserves directly observed partial-turn evidence, and cleans the process tree.
+
+A body target is matched to the exact snapshotted `SKILL.md` and explicitly delivered on turn one. Later turns remain unchanged. Description groups use natural prompts and never explicit target delivery.
+
+## Isolation And Resources
+
+For each selected variant:
+
+- snapshot exact declared skills and context;
+- snapshot the group fixture once from the definition-root working tree—never from a variant Git source—then create an independent writable copy;
+- preserve skill order and target index;
+- replace ambient instructions with evaluator-owned declared context;
+- keep criteria and review questions outside the subject prompt;
+- allow reads only from workspace and declared immutable resources;
+- allow writes and edits only inside the workspace;
+- reject traversal, symlink escape, unsupported Git entries, and invalid UTF-8;
+- verify immutable resource fingerprints after execution.
+
+Working-tree sources snapshot current paths. Git sources resolve one exact commit and record both the declared ref and resolved identity.
+
+## Persistence And States
+
+Each selected run is persisted before its counterpart. Grade and group evidence are persisted before later groups.
+
+Run states are `complete`, `invalid`, `infrastructure-failed`, `cancelled`, and `not-selected`. Group and batch states are `complete`, `partially-complete`, `invalid` where applicable, and `cancelled`.
+
+- Failed deterministic checks are ordinary complete behavioral evidence and exit zero when no infrastructure/grade failure exists.
+- Variant-local invalidity or infrastructure failure preserves evidence and does not stop safe queued work.
+- Shared fixture failure invalidates the group.
+- Thrown grading or recoverable grade/group publication failure becomes explicit `grade-error` fallback evidence.
+- If a run or required fallback cannot be preserved, queued work stops and no completed summary is published.
+- Cancellation starts no queued subject and preserves observed diagnostics.
+
+Invalidity, infrastructure failure, cancellation, or `grade-error` makes the final command exit nonzero after safe queued work finishes.
+
+## Views And Raw Evidence
+
+Views can render a complete result, all baseline or candidate variants, one group, or one group/variant. Grade appears before selected run evidence. The displayed absolute result path anchors result-relative artifact and workspace paths.
+
+Path cells escape literal backslashes before tab, carriage return, and newline. Variant views exclude expectation-owned errors from the other variant while retaining comparison, group, and system errors.
+
+Use ordinary file tools for raw `run.json`, events, transcript, final response, stderr, workspace, definition, grade, and group artifacts. Views remove transport noise; they do not replace canonical evidence.
+
+## Safeguards And Limits
+
+Normal completion follows settlement. Path guards, no-progress detection, cancellation, process-tree cleanup, and very high emergency ceilings stop runaway or unsafe infrastructure. Do not impose ordinary guessed turn, token, spend, output, or short time caps.
+
+When an unsupported operation changes the question, reject it or run a clearly separate direct comparison with explicit limits. Never invoke archived evaluators as a fallback.

package/skills/evaluate-skill/references/review-and-revision.md

@@ -0,0 +1,51 @@
+# Review And Revision
+
+Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning still needs judgment, or a revision/rerun boundary must be chosen.
+
+## Use The Strongest Evidence
+
+1. Filesystem state, typed workspace manifests, diffs, and changed paths.
+2. Structured events, tool activity, usage, process results, logs, and saved reports.
+3. Final response.
+4. Full transcript when earlier surfaces cannot answer the question.
+5. Subject self-assessment.
+
+Lower-priority evidence cannot repair a contradiction above it. If the response claims success but a required file is absent, preserve both facts and grade the file observation.
+
+## Keep Deterministic Facts Separate
+
+Built-in checks cover exact reads, typed paths, changed paths, file or response text, JSON states, valid turn scope, and factual transitions. Missing evidence remains unavailable. A failed check remains behavioral evidence; it does not relabel a complete run as infrastructure failure.
+
+The active agent or user may judge reasoning quality, authority preservation, architectural fitness, recommendation quality, or semantic completeness. Record cited evidence, uncertainty, and limits outside the canonical run and deterministic grade. Do not turn that judgment into evaluator-owned readiness state.
+
+## Diagnose Before Editing
+
+Classify the failed boundary:
+
+- activation timing;
+- description language unavailable before body read;
+- first-read body comprehension;
+- undeclared, undelivered, unread, or ambient-only dependency;
+- normal guidance hidden behind an unclear resource boundary;
+- nearby behavior that hijacks the task;
+- retained-use or multi-turn failure;
+- wording, placement, example, or stop-condition pressure;
+- fixture, environment, isolation, host, or deterministic-check failure.
+
+Do not add broad prose before identifying the failed boundary.
+
+## Revise One Pressure Point
+
+Change one measured point when possible: description cues, first-read wording, rule placement, dependency declaration, resource boundary, example, stop condition, or the group when evidence proves the evaluation is wrong.
+
+Keep unrelated instructions and fixed criteria stable. If the criterion changes, the old result no longer answers the same question.
+
+## Rerun The Complete Group
+
+After a measured skill change or resolved infrastructure cause, run the complete fixed group with both variants. Do not rerun only the preferred variant, reuse a settled control as a new sample, adopt partial evidence as comparable, or rerun merely to erase a failure.
+
+For variance, use independent complete invocations and keep every result visible.
+
+## Report
+
+Report the failed boundary and evidence, exact revision or evaluation correction, criteria that stayed fixed, old and new result paths, and remaining unsupported claims. The user decides the next authoring or use action.

package/skills/evaluate-skill/scripts/lib/definitions.mjs

@@ -0,0 +1,296 @@
+import { readFile, realpath } from "node:fs/promises";
+import path from "node:path";
+
+const ID = /^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/;
+const EVALUATION_TYPES = new Set(["description", "body", "end-to-end"]);
+const VARIANTS = ["baseline", "candidate"];
+
+export class DefinitionError extends Error {
+  constructor(code, message, definitionPath = null) {
+    super(message);
+    this.name = "DefinitionError";
+    this.code = code;
+    this.definitionPath = definitionPath;
+  }
+}
+
+function fail(code, message, definitionPath) {
+  throw new DefinitionError(code, message, definitionPath);
+}
+
+function isPlainObject(value) {
+  return value !== null && typeof value === "object" && !Array.isArray(value);
+}
+
+function requireObject(value, label, definitionPath) {
+  if (!isPlainObject(value)) fail("invalid-definition", `${label} must be an object`, definitionPath);
+  return value;
+}
+
+function requireExactKeys(value, required, optional, label, definitionPath) {
+  const object = requireObject(value, label, definitionPath);
+  const allowed = new Set([...required, ...optional]);
+  for (const key of required) {
+    if (!Object.hasOwn(object, key)) fail("invalid-definition", `${label} is missing ${key}`, definitionPath);
+  }
+  for (const key of Object.keys(object)) {
+    if (!allowed.has(key)) fail("invalid-definition", `${label} contains unsupported field ${key}`, definitionPath);
+  }
+  return object;
+}
+
+function requireId(value, label, definitionPath) {
+  if (typeof value !== "string" || value.length > 64 || !ID.test(value)) {
+    fail(
+      "invalid-definition",
+      `${label} must start with a lowercase letter and use lowercase letters, numbers, or single hyphens up to 64 characters`,
+      definitionPath,
+    );
+  }
+  return value;
+}
+
+function requireString(value, label, definitionPath) {
+  if (typeof value !== "string" || value.trim() === "") {
+    fail("invalid-definition", `${label} must be a non-empty string`, definitionPath);
+  }
+  return value;
+}
+
+function requireStringArray(value, label, definitionPath, { paths = false, unique = false } = {}) {
+  if (!Array.isArray(value)) fail("invalid-definition", `${label} must be an array`, definitionPath);
+  const seen = new Set();
+  return value.map((entry, index) => {
+    const string = requireString(entry, `${label}[${index}]`, definitionPath);
+    if (paths) requireSafeRelativePath(string, `${label}[${index}]`, definitionPath);
+    if (unique && seen.has(string)) {
+      fail("invalid-definition", `${label} contains duplicate ${string}`, definitionPath);
+    }
+    seen.add(string);
+    return string;
+  });
+}
+
+function requireSafeRelativePath(value, label, definitionPath) {
+  const string = requireString(value, label, definitionPath);
+  const normalized = path.normalize(string);
+  if (path.isAbsolute(string) || normalized === "." || normalized === ".." || normalized.startsWith(`..${path.sep}`)) {
+    fail("unsafe-path", `${label} must stay inside the definition root`, definitionPath);
+  }
+  return string;
+}
+
+function requireSuiteReference(value, label, definitionPath) {
+  const string = requireString(value, label, definitionPath);
+  if (path.isAbsolute(string) || path.normalize(string) === ".") {
+    fail("unsafe-path", `${label} must be relative to the suite file`, definitionPath);
+  }
+  return string;
+}
+
+function validateInput(value, definitionPath) {
+  const input = requireExactKeys(value, [], ["prompt", "turns"], "group.input", definitionPath);
+  const hasPrompt = Object.hasOwn(input, "prompt");
+  const hasTurns = Object.hasOwn(input, "turns");
+  if (hasPrompt === hasTurns) {
+    fail("invalid-definition", "group.input must contain exactly one of prompt or turns", definitionPath);
+  }
+  if (hasPrompt) requireString(input.prompt, "group.input.prompt", definitionPath);
+  if (hasTurns) {
+    const turns = requireStringArray(input.turns, "group.input.turns", definitionPath);
+    if (turns.length === 0) fail("invalid-definition", "group.input.turns must not be empty", definitionPath);
+  }
+}
+
+function validateSource(value, label, definitionPath) {
+  const source = requireObject(value, label, definitionPath);
+  if (source.kind === "working-tree") {
+    requireExactKeys(source, ["kind"], [], label, definitionPath);
+    return;
+  }
+  if (source.kind === "git") {
+    requireExactKeys(source, ["kind", "ref"], [], label, definitionPath);
+    requireString(source.ref, `${label}.ref`, definitionPath);
+    return;
+  }
+  fail("invalid-definition", `${label}.kind must be working-tree or git`, definitionPath);
+}
+
+function validateEnvironment(value, label, definitionPath) {
+  const environment = requireExactKeys(value, ["source", "skills", "target", "context"], [], label, definitionPath);
+  validateSource(environment.source, `${label}.source`, definitionPath);
+  const skills = requireStringArray(environment.skills, `${label}.skills`, definitionPath, { paths: true });
+  requireStringArray(environment.context, `${label}.context`, definitionPath, { paths: true });
+  if (
+    environment.target !== null &&
+    (!Number.isInteger(environment.target) || environment.target < 0 || environment.target >= skills.length)
+  ) {
+    fail("invalid-definition", `${label}.target must be null or an index into skills`, definitionPath);
+  }
+}
+
+function validateExpectations(value, definitionPath) {
+  if (!Array.isArray(value)) fail("invalid-definition", "group.expectations must be an array", definitionPath);
+  const ids = new Set();
+  for (const [index, entry] of value.entries()) {
+    const expectation = requireObject(entry, `group.expectations[${index}]`, definitionPath);
+    requireId(expectation.id, `group.expectations[${index}].id`, definitionPath);
+    requireString(expectation.kind, `group.expectations[${index}].kind`, definitionPath);
+    if (ids.has(expectation.id)) {
+      fail("invalid-definition", `group.expectations contains duplicate ID ${expectation.id}`, definitionPath);
+    }
+    ids.add(expectation.id);
+  }
+}
+
+function validateModel(value, definitionPath) {
+  const model = requireExactKeys(value, ["model"], ["thinking"], "group.model", definitionPath);
+  requireString(model.model, "group.model.model", definitionPath);
+  if (Object.hasOwn(model, "thinking")) requireString(model.thinking, "group.model.thinking", definitionPath);
+}
+
+function validateGroup(value, definitionPath) {
+  const group = requireExactKeys(
+    value,
+    [
+      "schema_version",
+      "kind",
+      "id",
+      "type",
+      "input",
+      "fixture",
+      "tools",
+      "variants",
+      "expectations",
+      "review_questions",
+    ],
+    ["model"],
+    "group",
+    definitionPath,
+  );
+  if (group.schema_version !== 1) fail("unsupported-schema", "group.schema_version must be 1", definitionPath);
+  if (group.kind !== "group") fail("invalid-definition", "group.kind must be group", definitionPath);
+  requireId(group.id, "group.id", definitionPath);
+  if (!EVALUATION_TYPES.has(group.type)) {
+    fail("invalid-definition", "group.type must be description, body, or end-to-end", definitionPath);
+  }
+  validateInput(group.input, definitionPath);
+  if (group.fixture !== null) requireSafeRelativePath(group.fixture, "group.fixture", definitionPath);
+  requireStringArray(group.tools, "group.tools", definitionPath, { unique: true });
+  const variants = requireExactKeys(group.variants, VARIANTS, [], "group.variants", definitionPath);
+  for (const variant of VARIANTS) validateEnvironment(variants[variant], `group.variants.${variant}`, definitionPath);
+  if (variants.candidate.target === null) {
+    fail(
+      "invalid-definition",
+      "group.variants.candidate.target must identify the candidate target skill",
+      definitionPath,
+    );
+  }
+  validateExpectations(group.expectations, definitionPath);
+  requireStringArray(group.review_questions, "group.review_questions", definitionPath);
+  if (Object.hasOwn(group, "model")) validateModel(group.model, definitionPath);
+  return group;
+}
+
+function validateSuite(value, definitionPath) {
+  const suite = requireExactKeys(value, ["schema_version", "kind", "id", "groups"], [], "suite", definitionPath);
+  if (suite.schema_version !== 1) fail("unsupported-schema", "suite.schema_version must be 1", definitionPath);
+  if (suite.kind !== "suite") fail("invalid-definition", "suite.kind must be suite", definitionPath);
+  requireId(suite.id, "suite.id", definitionPath);
+  const groups = requireStringArray(suite.groups, "suite.groups", definitionPath, { unique: true });
+  for (const [index, reference] of groups.entries()) {
+    requireSuiteReference(reference, `suite.groups[${index}]`, definitionPath);
+  }
+  if (groups.length === 0) fail("invalid-definition", "suite.groups must not be empty", definitionPath);
+  return suite;
+}
+
+function isContained(root, target) {
+  const relative = path.relative(root, target);
+  return relative === "" || (!relative.startsWith(`..${path.sep}`) && relative !== ".." && !path.isAbsolute(relative));
+}
+
+async function readJsonFile(file, root) {
+  const lexicalFile = path.resolve(file);
+  if (!isContained(root, lexicalFile)) fail("unsafe-path", `Definition escapes root: ${file}`, lexicalFile);
+
+  let canonicalRoot;
+  let canonicalFile;
+  try {
+    [canonicalRoot, canonicalFile] = await Promise.all([realpath(root), realpath(lexicalFile)]);
+  } catch {
+    fail("unreadable-definition", `Cannot resolve definition: ${lexicalFile}`, lexicalFile);
+  }
+  if (!isContained(canonicalRoot, canonicalFile)) {
+    fail("unsafe-path", `Definition symlink escapes root: ${lexicalFile}`, lexicalFile);
+  }
+
+  let contents;
+  try {
+    contents = await readFile(canonicalFile, "utf8");
+  } catch {
+    fail("unreadable-definition", `Cannot read definition: ${lexicalFile}`, lexicalFile);
+  }
+  try {
+    return { value: JSON.parse(contents), file: lexicalFile };
+  } catch {
+    fail("invalid-json", `Definition is not valid JSON: ${lexicalFile}`, lexicalFile);
+  }
+}
+
+export async function loadDefinition(file, { root = process.cwd() } = {}) {
+  const definitionRoot = path.resolve(root);
+  const loaded = await readJsonFile(file, definitionRoot);
+  const object = requireObject(loaded.value, "definition", loaded.file);
+  if (object.kind === "group") return validateGroup(object, loaded.file);
+  if (object.kind !== "suite") fail("invalid-definition", "definition.kind must be group or suite", loaded.file);
+
+  const suite = validateSuite(object, loaded.file);
+  const groups = [];
+  const ids = new Set();
+  for (const reference of suite.groups) {
+    const groupFile = path.resolve(path.dirname(loaded.file), reference);
+    const groupLoaded = await readJsonFile(groupFile, definitionRoot);
+    const group = validateGroup(groupLoaded.value, groupLoaded.file);
+    if (ids.has(group.id)) fail("invalid-definition", `suite contains duplicate group ID ${group.id}`, loaded.file);
+    ids.add(group.id);
+    groups.push(group);
+  }
+  return { ...suite, groups };
+}
+
+function variantSelection(selected) {
+  return {
+    baseline: selected === null || selected === "baseline" ? "selected" : "not-selected",
+    candidate: selected === null || selected === "candidate" ? "selected" : "not-selected",
+  };
+}
+
+export function selectDefinition(definition, { group = null, variant = null } = {}) {
+  if (variant !== null && !VARIANTS.includes(variant)) {
+    fail("invalid-selector", "variant must be baseline or candidate", null);
+  }
+  if (definition.kind === "group" && group !== null) {
+    fail("invalid-selector", "--group cannot be used with a direct group definition", null);
+  }
+
+  const available = definition.kind === "group" ? [definition] : definition.groups;
+  let selected = available;
+  if (group !== null) {
+    const position = typeof group === "number" || /^\d+$/.test(group) ? Number(group) : null;
+    selected =
+      position === null ? available.filter((entry) => entry.id === group) : [available[position - 1]].filter(Boolean);
+    if (selected.length === 0) fail("unknown-group", `Unknown group selector: ${group}`, null);
+  }
+
+  return {
+    kind: "selection",
+    definitionKind: definition.kind,
+    id: definition.id,
+    groups: selected.map((entry) => ({
+      position: available.indexOf(entry) + 1,
+      group: entry,
+      variants: variantSelection(variant),
+    })),
+  };
+}

package/skills/evaluate-skill/scripts/lib/evidence.mjs

@@ -0,0 +1,46 @@
+import { createHash, randomBytes } from "node:crypto";
+import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+
+export function createInvocationId(now = new Date()) {
+  const timestamp = now
+    .toISOString()
+    .replaceAll(/[-:.TZ]/g, "")
+    .slice(0, 14);
+  return `${timestamp}-${randomBytes(4).toString("hex")}`;
+}
+
+export async function writeJson(file, value) {
+  const contents = `${JSON.stringify(value, null, 2)}\n`;
+  await writeAtomic(file, contents);
+  return { contents, sha256: sha256(contents) };
+}
+
+export async function writeText(file, contents) {
+  await writeAtomic(file, contents);
+  return { contents, sha256: sha256(contents) };
+}
+
+export async function fileIdentity(file) {
+  const contents = await readFile(file);
+  return {
+    path: file,
+    sha256: createHash("sha256").update(contents).digest("hex"),
+  };
+}
+
+export function sha256(value) {
+  return createHash("sha256").update(value).digest("hex");
+}
+
+async function writeAtomic(file, contents) {
+  await mkdir(path.dirname(file), { recursive: true });
+  const temporary = `${file}.${process.pid}.${randomBytes(4).toString("hex")}.tmp`;
+  try {
+    await writeFile(temporary, contents);
+    await rename(temporary, file);
+  } catch (error) {
+    await rm(temporary, { force: true }).catch(() => {});
+    throw error;
+  }
+}

package/skills/evaluate-skill/scripts/lib/grade-comparison.mjs

@@ -0,0 +1,60 @@
+import { isDeepStrictEqual } from "node:util";
+
+import { hasOwn } from "./grade-support.mjs";
+
+export function gradeComparisons(expectations, checks) {
+  /** @type {Map<string, any[]>} */
+  const grouped = new Map();
+  const errors = /** @type {any[]} */ ([]);
+  for (const expectation of expectations) {
+    if (!hasOwn(expectation, "comparison")) continue;
+    if (typeof expectation.comparison !== "string" || !/^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/.test(expectation.comparison)) {
+      errors.push({ id: expectation.id, reason: "invalid comparison identity" });
+      continue;
+    }
+    const entries = grouped.get(expectation.comparison) ?? [];
+    entries.push(expectation);
+    grouped.set(expectation.comparison, entries);
+  }
+
+  /** @type {Map<string, any>} */
+  const byCheck = new Map(checks.map((check) => [check.id, check]));
+  const comparisons = /** @type {any[]} */ ([]);
+  for (const [id, entries] of grouped) {
+    const baselineExpectation = entries.find((entry) => entry.variant === "baseline");
+    const candidateExpectation = entries.find((entry) => entry.variant === "candidate");
+    if (entries.length !== 2 || !baselineExpectation || !candidateExpectation) {
+      errors.push({ id, reason: "comparison requires one baseline and one candidate expectation" });
+      continue;
+    }
+    const baseline = byCheck.get(baselineExpectation.id);
+    const candidate = byCheck.get(candidateExpectation.id);
+    if (!sameCheck(baseline, candidate)) {
+      errors.push({ id, reason: "comparison expectations must describe the same deterministic check" });
+      continue;
+    }
+    comparisons.push({
+      id,
+      kind: baseline.kind,
+      baseline: { check: baseline.id, state: baseline.state },
+      candidate: { check: candidate.id, state: candidate.state },
+      transition: transition(baseline.state, candidate.state),
+    });
+  }
+  return { comparisons, errors };
+}
+
+function sameCheck(baseline, candidate) {
+  return (
+    baseline !== undefined &&
+    candidate !== undefined &&
+    baseline.kind === candidate.kind &&
+    isDeepStrictEqual(baseline.expected, candidate.expected)
+  );
+}
+
+function transition(baseline, candidate) {
+  return [baseline, candidate].every((state) => state === "pass" || state === "fail")
+    ? `${baseline}-to-${candidate}`
+    : "unavailable";
+}