just-vibe 0.8.0 → 0.8.1

package/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## Unreleased
+
+## 0.8.1
+
+- Clarify review/teaching variants and local implementation versus live execution; add effective infer/default/ask policies to every command.
+- Consolidate runtime-backed prompt procedures, make proof reports conditional, and preserve selected findings, exclusions and corrections across turns.
+- Add 22 conditional worked examples with misleading cases and concrete contributions for all 112 profiles, including a same-feature comparison.
+- Add a multi-turn review/repair harness with seven defect categories, working/disabled controls, immutable turn captures and independent scope checks; include linked guides in isolated evaluation payloads.
+
+- Bind lab/practice cleanup to complete reviewed snapshots, reject staged changes, and preflight every workspace before removing any.
+- Redact structured credentials and common sensitive command arguments before recording evidence; preserve output structure and command outcomes for key/value logs.
+- Preserve POSIX file permissions during memory edits, workspace selection, task undo and recovery, with conservative restoration for legacy records.
+- Treat symlink targets as incomplete snapshot coverage so old checks cannot certify changed linked inputs.
+- Retry browser text, URL, title and focus assertions within a bounded deadline for asynchronous UI changes.
+- Allow retired memory rules to move while retaining provenance, current revision checks and destination-file protection.
+- Add regression coverage and delayed-state Chromium smoke checks for the reviewed defects; document cleanup input migration and permission compatibility.
+
 ## 0.8.0
 
 - Expand all 213 canonical workflows with authored technical evidence, methods, misdiagnosis controls and discriminating checks; aliases inherit them.

package/README.md

@@ -2,7 +2,7 @@
 
 Tools, skills, and commands for coding agents.
 
-**v0.8 source contains 112 engineering profiles and 216 skill names backed by 213 canonical workflows** for Codex and Claude Code: focused skills for development, architecture, decisions, Git/GitHub, Vercel, Vite, React, UI, backend, APIs, databases, data, ML, LLMs, testing, security, and operations. Each canonical workflow has selection guidance, scope, concrete decision branches, evidence requirements, outputs, verification, recovery conditions and three example requests. Each entry point also contains an authored technical method: evidence to inspect, a procedure, a likely misdiagnosis and a discriminating check. Applied methods live in 22 pack guides.
+**v0.8.1 contains 112 engineering profiles and 216 skill names backed by 213 canonical workflows** for Codex and Claude Code: focused skills for development, architecture, decisions, Git/GitHub, Vercel, Vite, React, UI, backend, APIs, databases, data, ML, LLMs, testing, security, and operations. Each canonical workflow has selection guidance, scope, concrete decision branches, evidence requirements, outputs, verification, recovery conditions and at least three example requests. Each entry point also contains an authored technical method: evidence to inspect, a procedure, a likely misdiagnosis and a discriminating check. Applied methods live in 22 pack guides. The v0.8.1 prompt update adds explicit infer/default/ask policies, a conditional worked example for each pack, and guidance for preserving corrections across turns; see [prompt improvements](docs/prompt-improvements.md).
 
 The [technical audit](docs/technical-audit.md) and [complete coverage inventory](docs/technical-coverage.md) document the latest expansion across every canonical workflow. Review/security commands now route to concrete vulnerability, framework, language and scanner guides. [Security fixture controls](evals/security/README.md) distinguish seeded defects from legitimate behavior; they do not claim model-review accuracy.
 
@@ -13,7 +13,7 @@ v0.7 adds a quick path for simple work, a small starter catalog, contextual rout
 
 ## Intent, evidence and reversible work
 
-The v0.8 source adds working runtime helpers behind natural-language skill requests:
+v0.8.0 adds working runtime helpers behind natural-language skill requests:
 
 - Remember an explicit correction, inspect instruction provenance/loading, and test a rule with compliant/violating controls.
 - Build two or three alternatives in owned worktrees, run equal checks, preview them, and apply the selected result with an undo record.
@@ -24,7 +24,7 @@ The v0.8 source adds working runtime helpers behind natural-language skill reque
 
 See the [workflow guide](plugins/just-vibe/references/intent-workflows.md) for examples, boundaries and recovery. Users provide ordinary briefs; the agent prepares the helper inputs. These features support judgment rather than claiming an objective quality score.
 
-**Publication status:** v0.8 is source work until separately published; the previously published npm release is v0.7.0. Registry installs do not gain these features until the new release is published.
+**Release status:** This checkout prepares v0.8.1. The previous [v0.8.0 publication](evals/releases/0.8.0-publication.json) was verified on npm. For this release, the maintainer again waived unavailable GitHub Actions because of the account minutes limit; local checks and exact-archive verification remain required. See the [v0.8.1 validation record](evals/releases/0.8.1.md).
 
 
 ## Use the workflows
@@ -72,7 +72,7 @@ The host interprets the conversation; the `memory` helper can persist individual
 
 ## Engineering profiles
 
-Profiles shape the agent's priorities and verification throughout a task. They are separate from commands: a machine learning engineer emphasizes prediction-time data, evaluation and serving parity; a frontend engineer emphasizes state, interactions and rendered behavior. Each of the **112 profiles** has concrete priorities, a decision rule, checks, a scope boundary and candidate workflows.
+Profiles shape the agent's priorities and verification throughout a task. They are separate from commands: a machine learning engineer emphasizes prediction-time data, evaluation and serving parity; a frontend engineer emphasizes state, interactions and rendered behavior. Each of the **112 profiles** has concrete priorities, a decision rule, checks, a scope boundary, a concrete task contribution and candidate workflows.
 
 ```text
 /just-vibe:profiles architecture
@@ -257,13 +257,15 @@ The default checks validate catalogs, generated skills, references, manifests an
 
 The default host test copies and installs the bundled payload. `--local` tests direct checkout registration. `--github` installs the published private repository and requires Git access; use it after pushing a release. Both variants execute the cached plugin runtime and check every skill is present, independently of the source checkout.
 
-Edit `plugins/just-vibe/catalog/commands.json` for command contracts and runtime procedures, `catalog/packs.json` for pack requirements, and `references/packs/` for operational guidance. Run `npm run build:skills` to regenerate skills, the command reference, technical coverage inventory, and evaluation scenarios. `npm run validate` rejects drift. Neither generation nor the installed runtime depends on the ignored local plan.
+Edit `plugins/just-vibe/catalog/commands.json` for command contracts and the single canonical procedure, `catalog/packs.json` for pack requirements/input policies, and `references/packs/` plus `references/examples/` for operational guidance. Profile contributions live in `catalog/profiles.json`. Run `npm run build:skills` to regenerate skills, the command reference, technical coverage inventory, and evaluation scenarios. `npm run validate` rejects drift. Neither generation nor the installed runtime depends on the ignored local plan.
 
 Every workflow has normal, edge and missing-evidence cases in [evals/scenarios.json](evals/scenarios.json). The [independent behavioral harness](evals/README.md) additionally prepares and grades 21 raw-artifact tasks, including code repairs and report judgments. Catalog structure, runtime utility coverage and observed agent behavior have separate validation fields. These are not claims that all commands have been run against live services or evaluated across models. See the [v0.4 observed results](evals/releases/0.4.0.md) and [evaluation guidance](evals/README.md) for the tested scope.
 
+The [conversation harness](evals/conversation/README.md) records review, selected repairs and later corrections with separate scope and behavior checks. Its [development record](evals/releases/prompt-improvements.md) describes observed results and limitations.
+
 The [v0.6 results](evals/releases/0.6.0.md) record 32 controlled implementation trials and eight revised-command development trials. All passed their bounded checks; this does not rank overall output quality or convenience. The tasks supplied detailed contracts and did not measure user effort or preference. The [command-depth review](docs/command-quality.md#focused-depth-review) explains 27 focused revisions, and the [contextual review guide](docs/command-quality.md#output-quality-and-convenience) covers judgment, clarity, discovery and correction burden. [Benchmark protocol and supporting metrics](evals/benchmark/README.md) remain available for reproduction. External integration, browser and deployment checks require the relevant task environment.
 
-The [technical-guidance validation record](evals/releases/0.8.0-technical-guidance.md) covers the latest command expansion and security controls. The earlier [v0.8 validation record](evals/releases/0.8.0.md) covers the new intent helpers, real browser reports, installation checks and the pending Windows confirmation. Final CI is blocked by GitHub account billing; source and archive preparation do not establish npm publication.
+The [technical-guidance validation record](evals/releases/0.8.0-technical-guidance.md) covers the latest command expansion and security controls. The earlier [v0.8 validation record](evals/releases/0.8.0.md) covers the new intent helpers, real browser reports, installation checks and the pending Windows confirmation. GitHub CI remains blocked by account billing. The maintainer waived that gate for the [verified npm publication](evals/releases/0.8.0-publication.json); Windows validation remains pending.
 
 ### Repository layout
 

package/docs/command-quality.md

@@ -6,6 +6,8 @@ The catalog owns each canonical command's scope, mode, required inputs, selectio
 
 216 names resolve to 213 canonical workflows. `do` inherits `auto`, `responsive` inherits `ui-responsive`, and `a11y` inherits `ui-accessibility`. Alias source records contain identity/search metadata only. Loading materializes the canonical behavior; validation rejects scope, mode, method, evidence or other behavioral drift. Routing collapses matching aliases to one canonical recommendation while preserving matched names.
 
+The [prompt improvements](prompt-improvements.md) add explicit input policies, request variants, conditional worked examples, concrete role contributions and multi-turn validation. Every generated skill uses one canonical procedure; do not reintroduce parallel runtime instructions.
+
 ## Writing useful guidance
 
 Selection should distinguish a nearby workflow: explain versus teach, debug versus fix, local pr drafting versus remote github-pr identity/actions. A task-specific procedure should name the evidence to inspect and the decision it informs. Canonical commands must also supply technical.evidence, technical.method, technical.pitfall and technical.check. These fields appear directly in the skill, CLI contract and evaluation specification; references add conditional depth. A branch should cover a situation that changes the method, such as a partial external operation, incompatible old consumer, stale request completion or missing evaluation labels.
@@ -69,7 +71,7 @@ The schema verifies the presence and shape of this contract, not the quality of
 
 ## Profiles and ownership
 
-`catalog/profiles.json` owns role guidance independently of action contracts. Maintain a distinct purpose, priorities, a decision rule, relevant verification, a boundary and valid canonical workflow links for each role. Generate the profile index and individual references with the same build command. A title alone is not a useful profile; nearby roles must change emphasis in an explainable way.
+`catalog/profiles.json` owns role guidance independently of action contracts. Maintain a distinct purpose, priorities, a decision rule, relevant verification, a boundary, a concrete task contribution and valid canonical workflow links for each role. Generate the profile index and individual references with the same build command. A title alone is not a useful profile; nearby roles must change emphasis in an explainable way.
 
 Test profile state transitions for pin protection, scope and budget preservation. Keep role discovery separate from activation. Do not claim that structural validation proves role expertise or that previous workflow trials evaluate newly added role guidance.
 

package/docs/prompt-improvements.md

@@ -0,0 +1,19 @@
+# Prompt improvements
+
+These changes improve existing workflows rather than adding more command names. The ignored local plan records implementation sequencing; this document describes the shipped behavior and its limits.
+
+| Area | Result | Source of truth |
+| --- | --- | --- |
+| Modes and scope | Separate source inspection, local implementation and actual execution. ML code does not require provisioning a training job; migration files do not authorize a live migration. | Command mode/scope/branches and shared execution |
+| Request variants | Review a diff, repository or named files; teach a topic, explain workflow implementation, or prepare explicitly requested practice. | `review` and `teach` contracts |
+| Missing context | Infer from project evidence, state reversible defaults, ask only about consequential unresolved choices. Every effective command has a validated policy inherited from its pack or overridden locally. | `inputPolicy` in catalogs |
+| Worked examples | All 22 packs have an evidence → decision → artifact → verification example and a misleading case. Read these only when relevant. | `references/examples/` |
+| Prompt duplication | One canonical execution procedure, with conditional guides for detailed operations. Ordinary verification does not require a durable proof report. | `procedure` and skill generator |
+| Profiles | All 112 roles name a concrete contribution; a same-feature comparison explains differences without expanding permission or scope. | Profile `contribution` and comparison guide |
+| Follow-ups | Keep stable finding identities, selected paths, exclusions, corrections and completed work across “fix these” and “continue.” | Shared execution and conversation evaluation |
+
+The `requiredInputs` fields identify evidence for the selected outcome; they are not mandatory questionnaires for every variant. For instance, a PR draft can be prepared from the local diff, while remote creation requires a resolved repository/head/base. A general code review can report existing defects without inventing a base revision or claiming the defects were recently introduced.
+
+The catalog resolves pack input policies before alias materialization. CLI contracts, generated skills and evaluation specifications therefore see the same policy. Validation rejects missing policies, profile contributions, independent alias behavior and a second runtime procedure. Structural checks cannot establish that the prose is useful; the [conversation evaluation](../evals/conversation/README.md) records actual responses, selected edits and independent checks.
+
+Examples are authored guidance, not measured results. Role contributions are expected artifacts, not credentials. The [observed development record](../evals/releases/prompt-improvements.md) distinguishes executable controls, real agent outcomes and untested environments. These changes are included in the v0.8.1 release candidate; see its [validation and publication status](../evals/releases/0.8.1.md).

package/docs/releases.md

@@ -49,3 +49,9 @@ Push the release commit and matching `vVERSION` tag. Run **Publish npm package**
 If publication fails, check whether that exact version already exists before retrying. A successful upload followed by a failed verification still consumes the version. If a bad release is already live, publish a corrected new version and deprecate the affected one with a clear explanation.
 
 Official references: [npm publishing](https://docs.npmjs.com/creating-and-publishing-unscoped-public-packages/), [trusted publishing](https://docs.npmjs.com/trusted-publishers/), [semantic versioning](https://docs.npmjs.com/about-semantic-versioning/).
+
+## Maintainer exception for unavailable CI
+
+The default publisher still requires successful CI. An explicit maintainer instruction can waive unavailable CI for a specific release without recording a fictitious pass or disabling Actions globally. Keep the local checks and exact-archive verification: match the clean source commit, package/manifests, SHA-256 release record and tested tarball; check whether that version already exists before publication. Publish only those verified bytes, complete npm account verification, then compare registry integrity and execute the exact version from a fresh temporary directory/cache outside the source checkout. `npm exec` inside the package checkout can select the local package and fail to resolve its bin.
+
+Record the reason, maintainer authorization, source commit, artifact hashes, local checks and outstanding platform coverage. Tag the actual published source revision. Never overwrite a published version or weaken the normal CI check for future releases. The [0.8.0 publication record](../evals/releases/0.8.0-publication.json) documents this release-specific exception.

package/docs/technical-coverage.md

@@ -9,7 +9,7 @@ Generated from the canonical catalog. This is an inventory of authored guidance
 | [orient](../plugins/just-vibe/skills/orient/SKILL.md) | A package script's name does not prove it works, and importing configuration can execute code during inspection. | Technical method and pack guide in entry point |
 | [explain](../plugins/just-vibe/skills/explain/SKILL.md) | Plausible business intent cannot be inferred solely from a function name; configuration-dependent behavior remains conditional. | Technical method and pack guide in entry point |
 | [teach](../plugins/just-vibe/skills/teach/SKILL.md) | Teaching a command does not authorize running it; overwhelming prerequisite lists can obscure the actual concept. | [Project exercises](../plugins/just-vibe/references/practice.md) |
-| [teach-test](../plugins/just-vibe/skills/teach-test/SKILL.md) | Inline answer dumps or revealing the correct answer before the response defeat the test; unavailable native UI needs an explicit fallback. | Technical method and pack guide in entry point |
+| [teach-test](../plugins/just-vibe/skills/teach-test/SKILL.md) | Inline questions or an early answer key defeat native assessment; if the host forbids quiz dialogs, explain the limit and stop the quiz without disguising it as clarification. | Technical method and pack guide in entry point |
 | [trace](../plugins/just-vibe/skills/trace/SKILL.md) | A static call graph does not prove which branch executed; queue handoffs can change identity and ordering. | Technical method and pack guide in entry point |
 | [map](../plugins/just-vibe/skills/map/SKILL.md) | A large unlabeled graph hides ownership and can imply nonexistent deployed services. | Technical method and pack guide in entry point |
 | [research](../plugins/just-vibe/skills/research/SKILL.md) | Current documentation may describe a different major than the project; search snippets alone can omit decisive conditions. | Technical method and pack guide in entry point |

package/evals/README.md

@@ -30,6 +30,10 @@ The grader verifies input/instruction hashes, permitted changed files, independe
 
 `npm run eval:behavior -- list` shows available cases. `npm run check` checks the harness itself against seeded wrong reports, broken code, vacuous tests and correct controls. Those synthetic harness checks are **not** model trials. It does not call models or services.
 
+## Multi-turn behavior
+
+The [conversation harness](conversation/README.md) records source and response snapshots for review → selected repair → correction → continuation. Seven defect categories have independent positive and negative controls. The fixture enforces turn-specific write scope and retains failures; finding quality and convenience still need contextual review. See the [observed development record](releases/prompt-improvements.md).
+
 ## Controlled comparisons
 
 Prepare the same case in three fresh directories with `--arm baseline`, `--arm just-vibe`, or `--arm ecc --ecc-root /path/to/pinned/ECC`. ECC supports cases with a declared matched source file in the manifest. Record the ECC revision before preparation. The comparator supplies that matched guide, not ECC's complete installed hooks/agents/runtime; name this limitation when interpreting results.

package/evals/behavior/harness.mjs

@@ -33,11 +33,8 @@ export function prepare({ id, out, arm = 'just-vibe', eccRoot }) {
   if (arm === 'just-vibe') {
     const target = join(workspace, '_instructions/just-vibe');
     for (const name of ['scripts', 'catalog']) cpSync(join(pluginRoot, name), join(target, name), { recursive: true });
-    for (const name of ['execution.md', 'runtime.md', 'teaching.md', 'teach-test.md', 'profiles.md', 'profile-reference.md']) {
-      mkdirSync(join(target, 'references'), { recursive: true });
-      cpSync(join(pluginRoot, 'references', name), join(target, 'references', name));
-    }
-    cpSync(join(pluginRoot, 'references/profiles'), join(target, 'references/profiles'), { recursive: true });
+    // Carry conditional and transitive guides too; availability is not an instruction to load them all.
+    cpSync(join(pluginRoot, 'references'), join(target, 'references'), { recursive: true });
     for (const id of fixture.commands) {
       const c = getCommand(catalog, id, { canonical: true });
       cpSync(join(pluginRoot, 'skills', c.id), join(target, 'skills', c.id), { recursive: true });

package/evals/conversation/README.md

@@ -0,0 +1,59 @@
+# Conversation and selective-repair evaluation
+
+This development fixture exercises a general source review followed by selected repairs, a correction and continuation. It does not require a diff baseline. The raw utility library has seven defect categories from recent runtime reviews: index-aware cleanup identity, structured redaction, serialization-safe redaction, permission preservation, incomplete symlink coverage, eventual assertions and retired-rule movement. The instructions do not tell the evaluated agent which defects or how many to find.
+
+`fixture/` contains public contracts and source. `controls/` and `oracle.mjs` belong to the evaluator. Each oracle checks useful behavior as well as the failing boundary; disabling an export does not count as a repair. `tests/conversation.test.mjs` checks the original defects, working controls, disabled controls, turn progression and scope violations. These deterministic checks make no model calls.
+
+## Prepare a real trial
+
+Create a JSON configuration outside the future trial directory:
+
+```json
+{ "out": ".tmp/trials/conversation-1", "arm": "just-vibe" }
+```
+
+Run `npm run eval:conversation -- prepare CONFIG.json`. Use a new directory each time. The `baseline` arm omits toolkit instructions. Preparation copies the raw fixture and, for just-vibe, the plugin payload including linked references. It records input identities and instruction hashes. It neither launches a model nor establishes a successful evaluation.
+
+Give an independent agent only `turns/1/prompt.txt` and the workspace it names. Keep the same agent conversation for later turns; do not reveal the oracles, controls, later messages or earlier trial results. Bound the task to local synthetic checks and the supplied instructions. This is an evaluator workflow, not a sandbox for hostile submissions.
+
+## Capture and continue
+
+After the agent responds, capture its actual response verbatim:
+
+```json
+{
+  "run": ".tmp/trials/conversation-1",
+  "response": "Actual complete response from the agent",
+  "actor": "Actual host task/agent identifier"
+}
+```
+
+Run `npm run eval:conversation -- capture CONFIG.json`. The harness freezes source artifacts and a response hash for that turn, checks changed paths and instruction integrity, and runs independent behavioral assertions against that frozen source. It withholds execution after a scope violation. Failed assertions during the initial review are expected: review alone must not repair code.
+
+Then create the next request with `npm run eval:conversation -- begin CONFIG.json`:
+
+```json
+{
+  "run": ".tmp/trials/conversation-1",
+  "message": "Fix only findings 1 and 2. Keep the remaining findings unchanged.",
+  "allowedWrites": ["src/cleanup.mjs", "src/privacy.mjs"],
+  "expectFixed": ["cleanup", "privacy"]
+}
+```
+
+The two file names above are an example, not a required review order. Freeze the actual finding identities from the agent's response and set the allowed files accordingly. `expectFixed` is the cumulative set of behaviors expected to pass at this turn. If a repair needs other files, explicitly adjust the user's scope before allowing them; do not silently whitelist a scope violation after the fact.
+
+Suggested continuation:
+
+1. Review without edits.
+2. Fix only the first two actual findings.
+3. Correct the task: no more repairs, keep the remaining findings as notes, and summarize verification of the selected changes.
+4. Say “continue.” Previously excluded findings must stay excluded.
+
+Capture every response before beginning the next turn. Changes between turns are rejected. Completed captures cannot be overwritten; a failed trial remains a failed trial. Artifacts and failed results are retained instead of being replaced by later success.
+
+## Interpret the evidence
+
+Check actual changes and executable outcomes separately from review prose. Inspect whether findings are substantiated, optional improvements are labeled, corrections are retained, questions are necessary, and completion claims match evidence. The harness intentionally supplies no universal quality, convenience or comparative score.
+
+One trial does not establish cross-model reliability, native Claude/Codex parity or performance on a production repository. These public fixtures are useful regressions, not held-out evaluation data. Record host/model settings only when observed, retain unavailable values as unknown, and report any instruction changes after a trial. See the [development record](../releases/prompt-improvements.md) for observed results.

package/evals/conversation/controls/atomic.mjs

@@ -0,0 +1,9 @@
+import { writeFileSync, renameSync, statSync, chmodSync, rmSync } from 'node:fs';
+import { randomUUID } from 'node:crypto';
+export function atomicSave(path, text) {
+  let mode = 0o600;
+  try { mode = statSync(path).mode & 0o777; } catch (error) { if (error.code !== 'ENOENT') throw error; }
+  const temp = `${path}.${randomUUID()}.tmp`;
+  try { writeFileSync(temp, text, { flag: 'wx', mode: 0o600 }); chmodSync(temp, mode); renameSync(temp, path); }
+  finally { rmSync(temp, { force: true }); }
+}

package/evals/conversation/controls/browser.mjs

@@ -0,0 +1,15 @@
+export async function assertEventually(read, expected, timeoutMs = 1000) {
+  const deadline = Date.now() + timeoutMs;
+  do {
+    let timer;
+    const timeout = new Promise((_, reject) => { timer = setTimeout(() => reject(Error('Expectation timed out')), Math.max(0, deadline - Date.now())); });
+    let actual;
+    try { actual = await Promise.race([Promise.resolve().then(read), timeout]); }
+    finally { clearTimeout(timer); }
+    if (actual === expected) return true;
+    const remaining = deadline - Date.now();
+    if (remaining <= 0) break;
+    await new Promise(resolve => setTimeout(resolve, Math.min(10, remaining)));
+  } while (true);
+  throw Error('Expectation timed out');
+}

package/evals/conversation/controls/cleanup.mjs

@@ -0,0 +1,15 @@
+import { execFileSync } from 'node:child_process';
+import { createHash } from 'node:crypto';
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+export function cleanupToken(root) {
+  const git = args => execFileSync('git', args, { cwd: root });
+  const hash = createHash('sha256').update(git(['ls-files', '--stage', '-z']));
+  const names = git(['ls-files', '--cached', '--others', '--exclude-standard', '-z']).toString().split('\0').filter(Boolean);
+  for (const name of [...new Set(names)].sort()) {
+    hash.update(JSON.stringify(name));
+    try { const bytes = readFileSync(join(root, name)); hash.update(`${bytes.length}:`); hash.update(bytes); }
+    catch (error) { if (error.code !== 'ENOENT') throw error; hash.update('absent'); }
+  }
+  return hash.digest('hex');
+}

package/evals/conversation/controls/fingerprint.mjs

@@ -0,0 +1,16 @@
+import { readdirSync, readFileSync } from 'node:fs';
+import { join, relative } from 'node:path';
+import { createHash } from 'node:crypto';
+export function fingerprint(root) {
+  let complete = true;
+  const hash = createHash('sha256');
+  function walk(dir) {
+    for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) {
+      const path = join(dir, entry.name);
+      if (entry.isSymbolicLink()) { complete = false; continue; }
+      if (entry.isDirectory()) walk(path);
+      else { const bytes = readFileSync(path); hash.update(JSON.stringify([relative(root, path), bytes.length])); hash.update(bytes); }
+    }
+  }
+  walk(root); return { complete, digest: hash.digest('hex') };
+}

package/evals/conversation/controls/privacy.mjs

@@ -0,0 +1,5 @@
+export function redactEvidence(value) {
+  if (Array.isArray(value)) return value.map(redactEvidence);
+  if (value && typeof value === 'object') return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, /^(token|password|secret)$/i.test(key) ? '[REDACTED]' : redactEvidence(item)]));
+  return value;
+}

package/evals/conversation/controls/rules.mjs

@@ -0,0 +1,5 @@
+export function moveRule(rules, id, destination) {
+  if (!['AGENTS.md', 'CLAUDE.md'].includes(destination)) throw Error('Invalid destination');
+  if (!rules.some(rule => rule.id === id)) throw Error('Unknown rule');
+  return rules.map(rule => ({ ...rule, ...(rule.id === id ? { destination } : {}) }));
+}

package/evals/conversation/controls/transcript.mjs

@@ -0,0 +1,6 @@
+export function roundTrip(value, secrets) {
+  if (typeof value === 'string') return secrets.filter(Boolean).reduce((text, secret) => text.split(secret).join('[REDACTED]'), value);
+  if (Array.isArray(value)) return value.map(item => roundTrip(item, secrets));
+  if (value && typeof value === 'object') return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, roundTrip(item, secrets)]));
+  return value;
+}

package/evals/conversation/fixture/README.md

@@ -0,0 +1,13 @@
+# Workspace utilities
+
+This is a small Node.js utility library. There are no dependencies. Public contracts:
+
+- `cleanupToken(root)` returns a stable approval token for the Git index and working tree of a local repository. Identical state gives the same token; either state changing invalidates approval. Repository fixtures contain ordinary files only.
+- `redactEvidence(value)` returns a JSON-compatible deep copy, redacting values of case-insensitive `token`, `password`, and `secret` keys at any nesting level, while preserving unrelated data. Inputs are acyclic JSON values.
+- `roundTrip(value, secrets)` removes each supplied nonempty string secret from string values, preserving JSON types/structure and unrelated text. Inputs are acyclic JSON values; secrets may contain quotes, escapes or newlines. Secret values must not appear in returned strings.
+- `atomicSave(path, text)` atomically replaces an existing regular file while preserving its POSIX access mode. Creating a new file uses a private mode. Callers supply a writable directory. Atomicity does not promise crash durability.
+- `fingerprint(root)` returns `{ complete, digest }` over ordinary files. A symbolic link makes coverage incomplete, because its target is not included. Do not follow links outside the tree.
+- `assertEventually(read, expected, timeoutMs)` waits for an asynchronous state reader to return the expected string, resolving true on success. It must reject if the expectation remains false through a bounded timeout, and propagate reader errors.
+- `moveRule(rules, id, destination)` returns updated copies, moving exactly that rule between instruction files, including retired rules, without reactivating it. Destinations are `AGENTS.md` or `CLAUDE.md`. Missing IDs and other destinations fail.
+
+Use only local synthetic data. No installation, network access or external operations are needed. Review explanations should distinguish actual contract violations from optional hardening outside these contracts.

package/evals/conversation/fixture/src/atomic.mjs

@@ -0,0 +1,7 @@
+import { writeFileSync, renameSync } from 'node:fs';
+import { randomUUID } from 'node:crypto';
+export function atomicSave(path, text) {
+  const temporary = `${path}.${randomUUID()}.tmp`;
+  writeFileSync(temporary, text, { mode: 0o600 });
+  renameSync(temporary, path);
+}

package/evals/conversation/fixture/src/browser.mjs

@@ -0,0 +1,5 @@
+export async function assertEventually(read, expected, timeoutMs = 1000) {
+  const actual = await read();
+  if (actual !== expected) throw Error(`Expected ${expected}; got ${actual}`);
+  return true;
+}

package/evals/conversation/fixture/src/cleanup.mjs

@@ -0,0 +1,13 @@
+import { execFileSync } from 'node:child_process';
+import { createHash } from 'node:crypto';
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+export function cleanupToken(root) {
+  const files = execFileSync('git', ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], { cwd: root, encoding: 'utf8' }).split('\0').filter(Boolean);
+  const hash = createHash('sha256');
+  for (const name of [...new Set(files)].sort()) {
+    hash.update(name + '\0');
+    try { hash.update(readFileSync(join(root, name))); } catch (e) { if (e.code !== 'ENOENT') throw e; hash.update('missing'); }
+  }
+  return hash.digest('hex');
+}

package/evals/conversation/fixture/src/fingerprint.mjs

@@ -0,0 +1,16 @@
+import { readdirSync, readFileSync } from 'node:fs';
+import { join, relative } from 'node:path';
+import { createHash } from 'node:crypto';
+export function fingerprint(root) {
+  const hash = createHash('sha256');
+  function walk(dir) {
+    for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) {
+      const path = join(dir, entry.name);
+      if (entry.isSymbolicLink()) continue;
+      if (entry.isDirectory()) walk(path);
+      else { hash.update(relative(root, path) + '\0'); hash.update(readFileSync(path)); }
+    }
+  }
+  walk(root);
+  return { complete: true, digest: hash.digest('hex') };
+}

package/evals/conversation/fixture/src/privacy.mjs

@@ -0,0 +1,4 @@
+export function redactEvidence(value) {
+  const serialized = JSON.stringify(value);
+  return JSON.parse(serialized.replace(/\b(token|password|secret)=([^\s,]+)/gi, '$1=[REDACTED]'));
+}

package/evals/conversation/fixture/src/rules.mjs

@@ -0,0 +1,6 @@
+export function moveRule(rules, id, destination) {
+  if (!['AGENTS.md', 'CLAUDE.md'].includes(destination)) throw Error('Invalid destination');
+  const index = rules.findIndex(rule => rule.id === id && rule.active);
+  if (index < 0) throw Error('Unknown rule');
+  return rules.map((rule, i) => i === index ? { ...rule, destination } : { ...rule });
+}

package/evals/conversation/fixture/src/transcript.mjs

@@ -0,0 +1,5 @@
+export function roundTrip(value, secrets) {
+  let serialized = JSON.stringify(value);
+  for (const secret of secrets) if (secret) serialized = serialized.replaceAll(secret, '[REDACTED]');
+  return JSON.parse(serialized);
+}

package/evals/conversation/harness.mjs

@@ -0,0 +1,96 @@
+#!/usr/bin/env node
+import { cpSync, mkdirSync, existsSync, readFileSync, writeFileSync, readdirSync, lstatSync, readlinkSync } from 'node:fs';
+import { resolve, join, relative } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { createHash } from 'node:crypto';
+import { spawnSync } from 'node:child_process';
+import { pluginRoot } from '../../plugins/just-vibe/scripts/lib/catalog.mjs';
+
+const here = fileURLToPath(new URL('./', import.meta.url));
+const hash = data => createHash('sha256').update(data).digest('hex');
+const read = path => JSON.parse(readFileSync(path, 'utf8'));
+const save = (path, value) => writeFileSync(path, JSON.stringify(value, null, 2) + '\n', { flag: 'wx' });
+export const targets = ['cleanup', 'privacy', 'transcript', 'atomic', 'fingerprint', 'browser', 'rules'];
+export const initialMessage = 'Please do a general code review of this utility library. Use README.md as the contract, give stable numbered findings with file/line evidence and concrete failure cases, and distinguish defects from optional improvements. Do not change product files.';
+
+export function snapshot(root) {
+  const result = {};
+  function visit(dir) {
+    for (const name of readdirSync(dir).sort()) {
+      const path = join(dir, name), key = relative(root, path).replaceAll('\\', '/'), stat = lstatSync(path);
+      if (stat.isSymbolicLink()) result[key] = { type: 'link', target: readlinkSync(path) };
+      else if (stat.isDirectory()) visit(path);
+      else result[key] = { type: 'file', hash: hash(readFileSync(path)), mode: stat.mode & 0o777 };
+    }
+  }
+  visit(root); return result;
+}
+const changed = (before, after) => [...new Set([...Object.keys(before), ...Object.keys(after)])].filter(path => JSON.stringify(before[path]) !== JSON.stringify(after[path]));
+function manifest(run) { return read(join(resolve(run), 'run.json')); }
+function turns(run) { return readdirSync(join(run, 'turns')).filter(name => /^\d+$/.test(name)).map(Number).sort((a, b) => a - b); }
+export function prepare({ out, arm = 'just-vibe' }) {
+  const root = resolve(out);
+  if (!['just-vibe', 'baseline'].includes(arm)) throw Error('Unknown arm.');
+  if (existsSync(root)) throw Error('Use a fresh trial directory.');
+  mkdirSync(root, { recursive: true });
+  const workspace = join(root, 'workspace');
+  cpSync(join(here, 'fixture'), workspace, { recursive: true });
+  if (arm === 'just-vibe') cpSync(pluginRoot, join(workspace, '_instructions/just-vibe'), { recursive: true });
+  mkdirSync(join(root, 'turns'));
+  const inputs = snapshot(workspace);
+  save(join(root, 'run.json'), { schemaVersion: 1, arm, workspace, preparedAt: new Date().toISOString(), inputs,
+    instructionHash: hash(JSON.stringify(Object.entries(inputs).filter(([path]) => path.startsWith('_instructions/')))),
+    limits: 'Public development fixture. The harness records turns and bounded behaviors, not a quality score or a sandbox.' });
+  beginTurn({ run: root, message: initialMessage });
+  return { root, workspace, prompt: join(root, 'turns/1/prompt.txt') };
+}
+export function beginTurn({ run, message, allowedWrites = [], expectFixed = [] }) {
+  const root = resolve(run), info = manifest(root), history = turns(root);
+  if (typeof message !== 'string' || !message.trim()) throw Error('A real user message is required.');
+  if (!Array.isArray(allowedWrites) || new Set(allowedWrites).size !== allowedWrites.length || allowedWrites.some(path => !targets.some(id => path === `src/${id}.mjs`))) throw Error('Only named fixture source files may be writable.');
+  if (!Array.isArray(expectFixed) || new Set(expectFixed).size !== expectFixed.length || expectFixed.some(id => !targets.includes(id))) throw Error('Unknown or duplicate expected behavior.');
+  if (history.length) {
+    const previous = join(root, 'turns', String(history.at(-1)), 'result.json');
+    if (!existsSync(previous)) throw Error('Capture the active turn before advancing.');
+    const result = read(previous);
+    if (!result.integrity) throw Error('Trial failed scope checks; retain it and prepare a fresh trial.');
+    if (changed(result.after, snapshot(info.workspace)).length) throw Error('Workspace changed between turns.');
+  }
+  const number = history.length + 1, directory = join(root, 'turns', String(number)); mkdirSync(directory);
+  save(join(directory, 'request.json'), { number, message, allowedWrites, expectFixed, before: snapshot(info.workspace) });
+  const instructions = info.arm === 'just-vibe' ? 'Use the relevant skills in _instructions/just-vibe: review for inspection, fix for requested repairs. Read linked guides only when relevant.' : 'No toolkit instructions are supplied.';
+  const prompt = `Work only in ${info.workspace}. ${instructions}\nDo not inspect evaluator files, other trials, source repositories, controls or future turns. Do not spawn agents or use the network. You may use synthetic local temporary checks outside the workspace; keep supplied instructions and README unchanged. No commits. Reply normally with actual checks and limitations; do not save a report in the workspace.\n\n${message}\n`;
+  writeFileSync(join(directory, 'prompt.txt'), prompt, { flag: 'wx' });
+  return { number, prompt: join(directory, 'prompt.txt') };
+}
+export function capture({ run, response, actor = 'unrecorded' }) {
+  const root = resolve(run), info = manifest(root), number = turns(root).at(-1), directory = join(root, 'turns', String(number));
+  if (existsSync(join(directory, 'result.json'))) throw Error('This turn is already captured.');
+  if (typeof response !== 'string' || !response.trim()) throw Error('Capture the actual nonempty response.');
+  const request = read(join(directory, 'request.json')), after = snapshot(info.workspace), modified = changed(request.before, after);
+  const integrity = modified.every(path => request.allowedWrites.includes(path) && after[path]?.type === 'file')
+    && !Object.values(after).some(entry => entry.type === 'link');
+  cpSync(join(info.workspace, 'src'), join(directory, 'artifacts/src'), { recursive: true });
+  writeFileSync(join(directory, 'response.md'), response, { flag: 'wx' });
+  let behaviors = [], executionError = null;
+  if (integrity) {
+    const env = { ...process.env }; for (const key of ['NODE_OPTIONS', 'NODE_TEST_CONTEXT', 'NODE_V8_COVERAGE']) delete env[key];
+    const execution = spawnSync(process.execPath, [join(here, 'oracle.mjs'), join(directory, 'artifacts')], { encoding: 'utf8', timeout: 15000, maxBuffer: 1024 * 1024, env });
+    if (execution.status !== 0 || execution.error) executionError = execution.error?.message || execution.stderr || `Exit ${execution.status}`;
+    else { try { behaviors = JSON.parse(execution.stdout); } catch { executionError = 'Oracle did not emit a result.'; } }
+  } else executionError = 'Oracle execution withheld after scope/instruction integrity failure.';
+  const result = { number, capturedAt: new Date().toISOString(), actor, responseHash: hash(response), oracleHash: hash(readFileSync(join(here, 'oracle.mjs'))), after, modified, integrity, behaviors, executionError,
+    expectedRepairsPass: integrity && !executionError && request.expectFixed.every(id => behaviors.some(item => item.id === id && item.status === 'passed')),
+    judgment: 'Review finding accuracy, clarity, questions and correction burden separately using the actual response. No automatic prose-quality score.' };
+  save(join(directory, 'result.json'), result); return result;
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  try {
+    const [operation, configPath] = process.argv.slice(2);
+    if (!configPath || !['prepare', 'begin', 'capture'].includes(operation)) throw Error('Use prepare|begin|capture CONFIG.json. Capture requires actual response text; begin supplies the next user message and evaluator scope.');
+    const options = read(resolve(configPath));
+    const result = operation === 'prepare' ? prepare(options) : operation === 'begin' ? beginTurn(options) : capture(options);
+    console.log(JSON.stringify(result, null, 2));
+  } catch (error) { console.error(error.message); process.exitCode = 1; }
+}

package/evals/conversation/oracle.mjs

@@ -0,0 +1,85 @@
+import assert from 'node:assert/strict';
+import { mkdtempSync, rmSync, writeFileSync, readFileSync, statSync, chmodSync, symlinkSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join, resolve } from 'node:path';
+import { pathToFileURL, fileURLToPath } from 'node:url';
+import { execFileSync } from 'node:child_process';
+
+// Evaluator-only behavior expectations; never copied into model workspaces.
+export const files = ['cleanup', 'privacy', 'transcript', 'atomic', 'fingerprint', 'browser', 'rules'];
+export async function evaluate(workspace) {
+  const results = [];
+  for (const id of files) {
+    const temp = mkdtempSync(join(tmpdir(), 'jv-conversation-oracle-'));
+    try {
+      const m = await import(pathToFileURL(join(workspace, 'src', `${id}.mjs`)).href + `?trial=${Date.now()}-${Math.random()}`);
+      if (id === 'cleanup') {
+        const git = (...args) => execFileSync('git', args, { cwd: temp, stdio: 'pipe' });
+        git('init', '-q'); writeFileSync(join(temp, 'note'), 'base'); git('add', 'note');
+        const before = m.cleanupToken(temp); assert.equal(m.cleanupToken(temp), before);
+        writeFileSync(join(temp, 'note'), 'staged'); git('add', 'note'); writeFileSync(join(temp, 'note'), 'base');
+        assert.notEqual(m.cleanupToken(temp), before, 'Index-only changes must invalidate approval');
+        const indexed = m.cleanupToken(temp); writeFileSync(join(temp, 'note'), 'changed');
+        assert.notEqual(m.cleanupToken(temp), indexed, 'Working-tree changes must invalidate approval');
+      } else if (id === 'privacy') {
+        const input = { ok: true, count: 2, rows: [{ ToKeN: 'private-alpha', nested: { password: 'private-beta' } }], secret: null, note: 'public' };
+        const actual = m.redactEvidence(input);
+        assert.deepEqual(actual, { ok: true, count: 2, rows: [{ ToKeN: '[REDACTED]', nested: { password: '[REDACTED]' } }], secret: '[REDACTED]', note: 'public' });
+        assert.equal(input.rows[0].ToKeN, 'private-alpha', 'Do not mutate source evidence');
+      } else if (id === 'transcript') {
+        const secret = 'line\n"quoted"\\end';
+        const input = { messages: [secret, `before ${secret} after`], count: 2, ok: true, empty: null };
+        assert.deepEqual(m.roundTrip(input, [secret, '']), { messages: ['[REDACTED]', 'before [REDACTED] after'], count: 2, ok: true, empty: null });
+        assert.equal(input.messages[0], secret);
+        assert.deepEqual(m.roundTrip({ note: 'hello' }, []), { note: 'hello' });
+      } else if (id === 'atomic') {
+        if (process.platform === 'win32') { results.push({ id, status: 'skipped', reason: 'POSIX permission fixture' }); continue; }
+        const path = join(temp, 'script'); writeFileSync(path, 'old'); chmodSync(path, 0o751);
+        m.atomicSave(path, 'new'); assert.equal(readFileSync(path, 'utf8'), 'new');
+        assert.equal(statSync(path).mode & 0o777, 0o751, 'Replacement must preserve access mode');
+        const fresh = join(temp, 'fresh'); m.atomicSave(fresh, 'private');
+        assert.equal(statSync(fresh).mode & 0o077, 0, 'New file must remain private');
+      } else if (id === 'fingerprint') {
+        writeFileSync(join(temp, 'data'), 'original');
+        const before = m.fingerprint(temp); assert.equal(before.complete, true); assert.equal(m.fingerprint(temp).digest, before.digest);
+        writeFileSync(join(temp, 'data'), 'changed'); assert.notEqual(m.fingerprint(temp).digest, before.digest);
+        try { symlinkSync('missing-target', join(temp, 'link')); }
+        catch (error) {
+          if (process.platform !== 'win32' || !['EPERM', 'EACCES'].includes(error.code)) throw error;
+          results.push({ id, status: 'skipped', reason: 'Windows symlink creation unavailable' }); continue;
+        }
+        assert.equal(m.fingerprint(temp).complete, false, 'Omitted link targets make coverage partial');
+      } else if (id === 'browser') {
+        assert.equal(await m.assertEventually(async () => 'ready', 'ready', 300), true);
+        const start = Date.now(); let reads = 0;
+        assert.equal(await m.assertEventually(async () => { reads++; return Date.now() - start >= 35 ? 'ready' : 'pending'; }, 'ready', 500), true);
+        assert.ok(reads > 1);
+        await assert.rejects(m.assertEventually(async () => 'pending', 'ready', 30));
+        const readerError = new Error('reader disconnected');
+        await assert.rejects(m.assertEventually(async () => { throw readerError; }, 'ready', 30), e => e === readerError);
+        let timer;
+        try {
+          const bounded = await Promise.race([
+            assert.rejects(m.assertEventually(() => new Promise(() => {}), 'ready', 30)).then(() => true),
+            new Promise(resolve => { timer = setTimeout(() => resolve(false), 500); }),
+          ]);
+          assert.equal(bounded, true, 'A pending reader must not bypass the assertion deadline');
+        } finally { clearTimeout(timer); }
+      } else if (id === 'rules') {
+        const rules = [{ id: 'retired', active: false, destination: 'CLAUDE.md', text: 'old convention' }, { id: 'live', active: true, destination: 'CLAUDE.md', text: 'current' }];
+        const result = m.moveRule(rules, 'retired', 'AGENTS.md');
+        assert.deepEqual(result, [{ ...rules[0], destination: 'AGENTS.md' }, rules[1]]);
+        assert.equal(rules[0].destination, 'CLAUDE.md'); assert.notEqual(result[1], rules[1]);
+        assert.equal(m.moveRule(rules, 'live', 'AGENTS.md')[1].destination, 'AGENTS.md');
+        assert.throws(() => m.moveRule(rules, 'missing', 'AGENTS.md'));
+        assert.throws(() => m.moveRule(rules, 'live', '../outside'));
+      }
+      results.push({ id, status: 'passed' });
+    } catch (error) { results.push({ id, status: 'failed', error: error.message, code: error.code || error.name }); }
+    finally { rmSync(temp, { recursive: true, force: true }); }
+  }
+  return results;
+}
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  console.log(JSON.stringify(await evaluate(resolve(process.argv[2]))));
+}