just-vibe 0.7.0

package/evals/releases/0.6.0.md

@@ -0,0 +1,94 @@
+# v0.6.0 command depth and repeated evaluations
+
+This release revises 27 command contracts and strengthens shared execution guidance. Changes replace repeated descriptions with concrete ownership, timing, failure, recovery and verification decisions. The [depth review](../../docs/command-quality.md#focused-depth-review) lists the commands and what changed. The inventory remains 215 skill names, 212 canonical workflows and 112 profiles.
+
+## What quality and convenience mean here
+
+Output quality and convenience are the product goals. We can check behavior, regression sensitivity and preserved Git state against explicit requirements. Whether the agent made helpful architectural choices, explained the result well or saved the user effort requires contextual review and real use. These results do not produce an overall product ranking.
+
+The trials supplied complete written contracts. They therefore removed much of the context preparation and discovery burden that a toolkit should help with. No participating users rated the experience, and no user-preference or convenience scores were collected. Zero interventions means the protocol supplied none; it does not mean the agent never needed clarification. The [contextual review guide](../../docs/command-quality.md#output-quality-and-convenience) covers task fit, maintainability, clarity, discovery, correction burden and defaults, with concrete evidence rather than a composite score.
+
+The revised contracts aim to make useful decisions explicit: distinguish one caller's cancellation from shared cache work, preserve partially staged edits during a commit, and select historical feature revisions before applying time windows. These are reviewed instruction improvements. Their effect on everyday convenience has not been measured.
+
+## Original controlled comparison
+
+The original study ran 32 fresh native CLI trials: four newly authored multi-file repositories × four instruction arms × two repetitions. It compared just-vibe v0.5.0 at `8ed8fcb5b2c690dcff2409268c1d4e3079f59823` with a plain-agent baseline, selected task profiles and matched [ECC instructions](https://github.com/affaan-m/ECC/tree/934195f955cf0da847d59fcd6f68856bce112d8b). All entry points, profiles, permissions and limits are in the [registered protocol](../benchmark/README.md).
+
+| Repository task | Main behavior exercised | Selected profile |
+| --- | --- | --- |
+| Async cache | Tenant identity, coalescing, independent cancellation, TTL, generation invalidation and view disposal | Frontend engineer |
+| Local ledger | Atomic transfers, exact validation, replay/conflict semantics, caller transaction ownership and contention | Backend engineer |
+| Temporal ML data | Visible revisions, time windows, label maturity, training-only transforms and unknown outcomes | Machine learning engineer |
+| Scoped Git commit | Fix and tests committed while preserving unrelated same-file staged/unstaged changes | Senior software engineer |
+
+All runs requested `gpt-6-astra` with medium reasoning through isolated Codex CLI 0.155.1. The macOS arm64 host used Node 26.7.0, Python 3.9.6 and SQLite 3.51.0. Order was shuffled with seed 5192026, concurrency capped at two and each agent limited to 240 seconds. User config/rules, host skill discovery, plugins, hooks, apps, web and subagents were disabled; the skill-discovery flag is experimental. No user replies were supplied during a trial.
+
+| Instructions | Passed bounded checks |
+| --- | ---: |
+| Plain agent | 8/8 |
+| just-vibe, profiles off | 8/8 |
+| just-vibe, selected profile | 8/8 |
+| Matched ECC guides | 8/8 |
+
+Every arm passed all eight assigned trials under the final scorer. Both conditions also passed in all eight paired profile comparisons. These checks did not distinguish the instruction arms. They do **not** show that the products are equal, that just-vibe beats ECC, or that profiles improve coding judgment. See the [32 per-trial results](0.6.0-before-results.json) for assertions, paired outcomes, earlier scores and evidence hashes.
+
+## Revised-command development check
+
+The revised v0.6 instructions match the plugin payload committed in `68c1ac23d9d4496f14ca78042f9ea0f224e4314f`. Eight separate fresh trials then used the same four tasks, one repetition with and without profiles, seed 5192027 and the same model/CLI/limits. The task and instruction copies were frozen before execution.
+
+| Instructions | Passed bounded checks |
+| --- | ---: |
+| just-vibe, profiles off | 4/4 |
+| just-vibe, selected profile | 4/4 |
+
+8/8 revised-command trials passed. These are development regression results on now-known tasks, not held-out evidence that the changes caused an improvement. The original and revised studies have different repetition counts and were run sequentially; do not compare their raw totals as equal-size samples. The [eight per-trial results](0.6.0-after-results.json) remain separate from the original arms.
+
+## Evaluator audit and retained results
+
+Final results use scorer version 3. Known-correct controls pass and seeded defects fail. The scorer verifies protected inputs, executable behavior, meaningful regression sensitivity, completion and—in the Git case—the committed implementation and tests independently of the mixed worktree.
+
+The initial scorer misclassified some regression failures: Node's default reporter withheld assertion details before later hangs, and Python behavior exceptions or completed assertions before a later hang were not consistently retained. Streaming failure evidence fixes those classifications. A timeout without completed behavior evidence, a setup-only failure or a successful mutant run still does not establish regression sensitivity. The Git grader was strengthened to execute the committed tree, and the evaluator's final SQLite observer connection now closes before temporary-file cleanup on Windows. These changes preserve the original task requirements and behavior assertions.
+
+10 of the original 32 outcome labels changed between scorer versions 1 and 3. Original and intermediate grades, failed attempts and raw workspaces are retained; no model output was repaired or replaced during rescoring. Both original and final scorer hashes are recorded. Two excluded setup pilots are documented in the protocol: the first exposed Git metadata write protection, and both initially inferred an incorrect default without an explicit README reference in the task. The registered task gave every arm that reference. An unexecuted v0.6 preparation predating the Windows cleanup fix was retained separately and contributes no trial results.
+
+## Output review
+
+All 40 final answers were read alongside artifact results; recorded actions were checked for unsupported claims and scope issues. In the eight revised-command answers, each named the repair and local verification. The ML answers explicitly avoided claiming a measured model-score improvement; the ledger answers distinguished local SQLite checks from process-crash or production-reliability testing. The Git answers identified the local commit and preservation of unrelated work, which the grader independently checked.
+
+Those are useful examples of appropriately bounded reporting. They are not blind ratings of maintainability or user satisfaction, and we did not score every sentence or independently audit every internal action. No qualitative winner is claimed. A practical follow-up would compare ordinary starting briefs in real repositories, review anonymized outputs where feasible, and record what users had to clarify, correct or look up. Different users and tasks may prefer different workflows.
+
+## Scope and further evidence
+
+The four tasks load 11 distinct workflows in bundles and exercise four of the 112 profiles. They do not establish standalone success rates for each command or expertise for every role. The other command revisions have instruction review plus structural, generation and packaging validation; existing per-command historical labels are not inflated by this bundled study.
+
+These are authored laboratory repositories with explicit contracts, not third-party production repositories. The ECC arm supplies matched skill/agent documents, not its complete installed hooks, memory and integrations. Fresh agent contexts and restricted prompts reduce contamination; the filesystem read boundary is not hardened. Public fixtures also cease to be a durable blind benchmark. Recorded actions, final reports and artifacts were reviewed, but an absent event cannot prove an unobserved action never occurred.
+
+Further evidence should include held-out work in real repositories, multiple models, missing/ambiguous-context tasks, browser interaction, authenticated deployment recovery and actual integration environments. No live Vercel deployment, database service, training job or native teaching dialog is claimed here.
+
+## Supporting operational data
+
+Tokens and elapsed time are retained for reproducibility and troubleshooting. They do not measure output quality or convenience. No cost-efficiency winner is inferred.
+
+Original study:
+
+| Instructions | Passed | Median agent seconds | Total input tokens | Cached input subset | Output tokens |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Plain agent | 8/8 | 147.0 | 723,179 | 541,312 | 31,627 |
+| just-vibe, profiles off | 8/8 | 153.7 | 956,052 | 785,792 | 33,968 |
+| just-vibe, selected profile | 8/8 | 158.8 | 1,116,010 | 892,928 | 35,641 |
+| Matched ECC guides | 8/8 | 144.5 | 1,142,256 | 916,864 | 31,801 |
+
+Revised-command development check:
+
+| Instructions | Passed | Median agent seconds | Total input tokens | Cached input subset | Output tokens |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| just-vibe, profiles off | 4/4 | 163.4 | 557,715 | 472,064 | 17,689 |
+| just-vibe, selected profile | 4/4 | 189.7 | 600,937 | 502,016 | 18,307 |
+
+Cached input is already included in total input. Dollar cost is unavailable from this authenticated CLI and remains null. Wall time includes CLI/service delays and excludes grading; other development validation ran on the host, so this was not an isolated latency experiment. Recognized tool-event counts do not account for every internal operation. The studies have different sample sizes and were run sequentially; their raw totals are not comparable as equal-size samples.
+
+## Release validation
+
+Local release checks passed: 107 tests, 106 passed and one Windows-specific skip on macOS, plus release metadata, license, archive-content, link and credential-pattern checks. They exercise catalog generation, workflow/runtime behavior, installation, packaging and evaluator controls. Native Codex and Claude installation lifecycles passed in isolated host configuration on macOS, including repeat setup, doctor, update, uninstall and reinstall. npm, pnpm 10/12 and Yarn archive execution passed, including payload survival after package-cache deletion. These validate delivery and utility behavior; they do not substitute for a user convenience study.
+
+The [GitHub CI history](https://github.com/Zachshotamartin/just-vibe/actions/workflows/ci.yml) records Node 22/24 checks on Ubuntu, macOS and Windows. A Windows evaluator cleanup failure and a macOS cold-start timing assumption in an evaluator test were corrected during this release. Final release preparation checks the exact archive and performs a publication dry run; it does not publish to npm.

package/evals/releases/0.7.0.md

@@ -0,0 +1,32 @@
+# v0.7.0 daily workflow improvements
+
+This release implements the seven workstreams identified after the v0.6 review. The inventory remains 215 skill names, 212 canonical workflows, 112 profiles and 22 pack guides. Seventeen existing skill entries change; six scenario guides and the daily workflow reference provide conditional detail. No overall product superiority or measured user preference is claimed.
+
+| Workstream | What changed | Requirements and limits |
+| --- | --- | --- |
+| Proportionate auto | Quick work keeps brief, scope, profile and verification in conversation; tracked work uses existing validated stages. Escalation carries actual evidence and consumed limits. | The host resolves real scope/effects; strategy heuristics do not enforce permissions or prove agent compliance. |
+| Discovery | Small starter catalog, full inventory through --all, contextual ranking, selection reasons, ambiguity and concise human output. | Bounded manifest inspection; JS/Python dependency signals and explicit intent rules, not a semantic model. Saved preferences remain context. |
+| Scenario depth | Cookie/OAuth/refresh/recovery authentication; dialog/combobox/date interaction; classical/tensor/distributed training; GitHub/Vercel/migration evidence. | Read only the relevant branch and current provider/framework documentation. New prose was reviewed and packaged, not independently scored as expert behavior for every scenario. |
+| Continuity | Explicit preferences, revision-checked notes, named checkpoints and comparison of repository, branch, HEAD, index and bounded content. | Local .just-vibe state. Partial coverage requires revalidation; prior verification/authorization is never restored automatically. |
+| Collectors | GitHub PR checks with head/base recheck, Vercel build diagnosis, real browser steps, SQL migration/hash reconciliation. | Installed provider tools and applicable access. Browser requires project Playwright/Chromium; SQL collector does not contact a database. |
+| Optional hooks | Configured Stop checks, single-file formatters and bounded continuation summaries; local exact-config trust, overlap prevention, timeouts and deduplication. | Inactive by default. Host hook trust is separate. Configured commands execute local project code; this is not a sandbox. Staged files are skipped; check results become stale if project content changes during execution. |
+| Distribution | v0.7 package/manifests, exact-archive preparation, source-commit record and guarded publication command with registry-integrity reconciliation. | Actual publication requires the maintainer's npm login and completed CI. A prepared archive or passing dry run is not a published package. |
+
+## Verification performed
+
+The local release check ran 128 tests: 127 passed and one Windows-specific skip on macOS, plus catalog/generation, archive, license, link and credential-pattern checks.
+
+- Local tests exercise routing, revisions and stale checkpoints, path escapes, large-file coverage, provider failures/pending/stale identity, migration drift, browser cleanup, command time/output limits, hook trust changes, partial staging, overlap, deduplication and checks against changing source.
+- The final v0.7 payload also extends remember with conversation-to-instruction merging, shared CLAUDE.md/AGENTS.md guidance and an optional checkpoint in the same invocation. Routing regressions, generated skill validation and packaged references passed. These checks do not establish model adherence or native instruction reload behavior; the revised remember workflow has no independent agent trial.
+- A real hook entry-point test runs an actual failing local check and verifies its advisory output and retained exit status. No fabricated model trial is counted as hook verification.
+- An isolated Chromium test runs the public CLI against a local HTTP fixture: dialog open/close, visible/hidden states, input, text and focus return. A deliberately wrong title fails with exit 2. Reproduce with `npm run test:browser -- PROJECT_WITH_PLAYWRIGHT_AND_CHROMIUM`; the script installs nothing.
+- A live, read-only GitHub PR observation resolved identity and retained both successful and failing checks. This validates collection, not the quality of that repository. Stale-head and authentication failures are also covered through deterministic provider fixtures.
+- Installed Vercel inspect options were checked; metadata/error/timeout behavior is covered through fixtures. No supplied deployment target was available for a live deployment test, and no deployment was created.
+- Native Codex and Claude installation lifecycles passed using isolated configuration on macOS with the v0.7 payload. This establishes installation, updates, inventory and cached runtime execution. It does not establish native hook-event delivery/trust or browser behavior on every host/OS.
+- Package tests execute the extracted archive, including the new project and hook-status utilities. Release preparation separately executes its exact archive through npm, pnpm and Yarn and performs an npm publication dry run.
+
+The [CI history](https://github.com/Zachshotamartin/just-vibe/actions/workflows/ci.yml) records runtime/package checks on Node 22/24 and Ubuntu/macOS/Windows. Provider authentication, real database rollout, training runs and host-native hook trust remain environment-specific. The [v0.6 comparison](0.6.0.md) is historical evidence for those instruction versions; it is not relabeled as a model evaluation of v0.7.
+
+## Review and use
+
+The [daily workflow reference](../../plugins/just-vibe/references/daily-workflows.md) contains the CLI schemas, configuration and prerequisites. The [quality review guide](../../docs/command-quality.md#output-quality-and-convenience) remains the basis for reviewing real output and user effort. Concrete behavior and packaging checks support this release; claims about improved everyday judgment and convenience still need actual use and contextual review.