systemlink-cli 1.27.7__tar.gz → 1.28.1__tar.gz

{systemlink_cli-1.27.7 → systemlink_cli-1.28.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: systemlink-cli
-Version: 1.27.7
+Version: 1.28.1
 Summary: SystemLink Integrator CLI - cross-platform CLI for SystemLink workflows and templates.
 License-File: LICENSE
 Author: Fred Visser

{systemlink_cli-1.27.7 → systemlink_cli-1.28.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "systemlink-cli"
-version = "1.27.7"
+version = "1.28.1"
 description = "SystemLink Integrator CLI - cross-platform CLI for SystemLink workflows and templates."
 authors = ["Fred Visser <fred.visser@emerson.com>"]
 packages = [{ include = "slcli" }]
@@ -68,6 +68,7 @@ pytest = ">=7.0"
 pytest-cov = ">=3.0"
 pytest-timeout = ">=2.1.0"
 pyinstaller = "^6.14.2"
+jsonschema = ">=4.0"
 
 # Typed stubs for third-party libraries used by mypy in CI
 types-requests = "*"

{systemlink_cli-1.27.7 → systemlink_cli-1.28.1}/slcli/_version.py

@@ -1,4 +1,4 @@
 """Version information for slcli."""
 
 # This file is auto-generated. Do not edit manually.
-__version__ = "1.27.7"
+__version__ = "1.28.1"

systemlink_cli-1.28.1/slcli/skills/slcli/evals/COPILOT_BATCH_RUN_PROMPT.md

@@ -0,0 +1,127 @@
+# Copilot Batch Eval Prompt
+
+Use this prompt in a single parent Copilot Chat session when you want Copilot
+to run the entire prepared `gating` iteration, populate the `outputs/`
+folders, grade the results, aggregate the benchmark, and regenerate the static
+review page.
+
+This is the recommended orchestration pattern because the repo scripts handle
+workspace preparation, grading, aggregation, and review rendering, but they do
+not execute the model runs themselves.
+
+Important: the parent chat is shared context. Do not execute both
+`with_skill` and `old_skill` runs directly in the same running thread of
+conversation if you want a meaningful baseline. Use fresh stateless subagents
+per run so the merge-base skill stays isolated from candidate context.
+
+Also use a fail-fast budget per run. Do not let one struggling eval burn the
+whole batch. A good default for the gating suite is a maximum of about 8 tool
+calls or about 3 minutes of active work for a single run, whichever comes
+first.
+
+## When to use this
+
+- You already created an iteration workspace with `prepare_eval_workspace.py`
+- You already generated per-run `executor_prompt.txt` files with
+  `prepare_eval_prompts.py`
+- You want one parent Copilot chat to orchestrate all prepared gating evals end
+  to end
+
+## What this does
+
+In one parent chat, Copilot should:
+
+1. Read `iteration_manifest.json`.
+2. Find every `executor_prompt.txt` under the iteration directory.
+3. For each prepared run directory:
+   - execute the prompt in a fresh stateless subagent
+
+- load the candidate `slcli` skill from the run-specific candidate repo for `with_skill` runs
+- load the merge-base `slcli` skill from the run-specific baseline repo for `old_skill` runs
+- stop early when the per-run budget is exhausted and record the failure
+- save the final answer to `outputs/response.txt`
+- save the complete executor trace to `outputs/transcript.jsonl`
+- save executor identity, configuration, and completion status to `outputs/run_metadata.json`
+- save `total_tokens`, `duration_ms`, and derived `total_duration_seconds` from the subagent completion notification to the run's `timing.json`; do not estimate these values
+- retry infrastructure failures once; after a second failure, set status to `infrastructure_error` and continue
+- optionally save `outputs/notes.txt` for assumptions
+
+4. Run `benchmark_iteration.py`.
+5. Run `render_eval_review.py`.
+6. Summarize which runs were populated and where the review HTML was written.
+
+## Paste This Into Copilot Chat
+
+```text
+Run the prepared gating eval iteration end to end.
+
+Use one parent conversation only as the orchestrator. For each executor prompt,
+spawn a fresh stateless subagent so the runs do not share prompt history.
+For `with_skill`, load the skill path inside the run-specific candidate repo
+named by the executor prompt instead of using the working checkout.
+For `old_skill`, load the skill path inside the run-specific baseline repo named
+by the executor prompt instead of using the candidate checkout. Use only the
+neutral input paths named by the executor prompt for attached fixtures.
+Run independent evals in parallel when possible, but keep concurrency modest:
+typically 2 to 4 subagents at a time.
+
+Iteration workspace:
+<ITERATION_DIR>
+
+Instructions:
+1. Read iteration_manifest.json in that workspace.
+2. Discover every executor_prompt.txt under the iteration directory.
+3. For each executor prompt:
+   - execute the task described in the prompt in a fresh stateless subagent
+   - use a maximum budget of about 8 tool calls or about 3 minutes of active work for that run, whichever comes first
+   - if the run does not converge inside that budget, stop, write the best grounded response you have to response.txt, and write notes.txt explaining the failure briefly
+   - save the final user-facing answer to the sibling outputs/response.txt path named in the prompt
+  - save the complete executor trace to the sibling outputs/transcript.jsonl path named in the prompt
+   - save optional outputs/notes.txt only if assumptions or caveats matter
+  - save outputs/run_metadata.json with executor_provider, exact executor_model, harness, configuration, and status
+  - when the subagent completion notification arrives, immediately save its total_tokens and duration_ms plus derived total_duration_seconds to the sibling timing.json named in the executor prompt
+  - use status `completed` only for a completed model run; retry an infrastructure failure once, then use `infrastructure_error` and preserve partial artifacts
+4. Do both configurations for every eval:
+   - with_skill
+   - old_skill
+5. After all outputs are populated, run:
+  - poetry run python -m slcli.skills.slcli.scripts.benchmark_iteration <ITERATION_DIR>
+  - poetry run python -m slcli.skills.slcli.scripts.render_eval_review <ITERATION_DIR>
+6. Report:
+   - which run directories were populated
+   - whether grading and benchmark generation succeeded
+   - where review.html was written
+
+Execution rules:
+- Use the existing executor_prompt.txt files as the source of truth for each run.
+- Do not answer multiple eval runs in the parent chat context.
+- Do not reuse a subagent across runs.
+- Parallelize independent runs when useful, but keep concurrency to roughly 2 to 4 subagents at a time.
+- For with_skill runs, the subagent must load only the candidate skill path named in the executor prompt.
+- For old_skill runs, the subagent must load only the merge-base skill path named in the executor prompt.
+- If a run exceeds its budget without a grounded answer, declare it failed quickly, persist the best grounded partial result plus a short note, and continue.
+- Do not overwrite populated outputs unless the existing file is only a placeholder.
+- Keep each response grounded in supported slcli commands and workflows.
+- Save response artifacts only inside the specified outputs/ directories; save timing.json at the run path named in the executor prompt.
+- If a single run fails, continue with the remaining runs and report the failure at the end.
+```
+
+## Notes
+
+- You do not need a new top-level chat for every eval. One parent chat can
+  orchestrate the whole prepared gating suite if each run is delegated to a
+  fresh stateless subagent.
+- Yes, the subagents can be parallelized because the prepared runs are
+  independent. Keep the batch size small so one bad run does not hide the rest.
+- A plain single-thread conversation is not a clean `with_skill` versus
+  `old_skill` comparison because the parent chat shares context across
+  turns.
+- You also do not need a repo script per eval. The intended split is:
+  - repo scripts prepare, grade, aggregate, and render
+  - Copilot orchestrates isolated subagent executions and writes the output
+    artifacts
+- If you want broader coverage, swap the iteration path to a prepared
+  `regression` iteration.
+
+Replace `<ITERATION_DIR>` with the prepared iteration path before using the
+template.

systemlink_cli-1.28.1/slcli/skills/slcli/evals/README.md

@@ -0,0 +1,236 @@
+# slcli Eval Workflow
+
+This directory contains the eval corpus for the repo-local `slcli` skill.
+Use the `gating` suite to compare a candidate skill with its merge-base version
+after a skill edit. The default is three paired trials per configuration.
+
+## Gating Workflow
+
+Run the following from the repository root.
+
+### 1. Run the single prompt
+
+Use the repo prompt in Copilot Chat:
+
+```text
+/eval-skill-gating
+```
+
+Optional examples:
+
+```text
+/eval-skill-gating iteration_dir="slcli/skills/slcli-workspace/iteration-4"
+/eval-skill-gating max_parallel=2 max_tool_calls=6 max_minutes=2.5
+```
+
+The prompt performs the full gating workflow:
+
+- prepare or reuse a gating iteration
+- generate executor prompts
+- execute `with_skill` and `old_skill` runs via isolated subagents
+- grade and aggregate the iteration
+- apply the regression gate
+- regenerate `review.html`
+
+### 2. What the prompt runs under the hood
+
+If you need to run the flow manually, these are the underlying steps.
+
+#### Prepare a fresh gating workspace
+
+```bash
+poetry run python -m slcli.skills.slcli.scripts.prepare_eval_workspace --suite gating
+```
+
+This prints a new iteration directory such as:
+
+```text
+slcli/skills/slcli-workspace/iteration-1
+```
+
+The iteration contains paired candidate and baseline repository templates from
+the current candidate checkout. Only the `slcli` skill in the baseline template
+is replaced with its version from the merge base of `origin/main` and `HEAD`.
+Each trial receives its own copy of the appropriate template, so parallel runs
+cannot modify another trial's repository. File-backed fixtures are copied into
+a neutral `inputs/` directory for each run. Use
+`--baseline-ref` when comparing against another branch. Use
+`--baseline without_skill` only when measuring whether a new skill adds value;
+that comparison is not a regression test. The legacy `--isolate-baseline` option
+is accepted but no longer needed because both configurations are always isolated.
+
+#### Generate executor prompts
+
+```bash
+poetry run python -m slcli.skills.slcli.scripts.prepare_eval_prompts \
+  slcli/skills/slcli-workspace/iteration-1 \
+  --max-tool-calls 8 \
+  --max-minutes 3
+```
+
+This writes one `executor_prompt.txt` per prepared run.
+
+#### Execute the runs in Copilot
+
+Use [COPILOT_BATCH_RUN_PROMPT.md](./COPILOT_BATCH_RUN_PROMPT.md) together with
+the generated `executor_prompt.txt` files.
+
+Recommended execution pattern:
+
+1. Use one parent chat as the orchestrator.
+2. Run each prepared eval in a fresh stateless subagent.
+3. Point `with_skill` runs at the run-specific candidate skill path in its prompt.
+4. Point `old_skill` runs at the run-specific baseline skill path in its prompt.
+5. Keep concurrency modest, usually 2 to 4 runs at a time.
+6. If a run exceeds its budget, save the best grounded `response.txt`, add a short `notes.txt`, and continue.
+
+Each run saves response artifacts under its own `outputs/` directory and timing
+metadata at the run root. Every run must write:
+
+- `response.txt`: final user-facing response
+- `transcript.jsonl`: complete executor trace for audit and diagnosis
+- `run_metadata.json`: `executor_provider`, exact `executor_model`, `harness`,
+  `configuration`, and `status` (`completed` or `infrastructure_error`)
+- `../timing.json`: `total_tokens` and `duration_ms` from the subagent completion
+  notification plus derived `total_duration_seconds`; these values must not be estimated
+
+Candidate and baseline trials are incomparable when provider, model, or harness
+metadata differs.
+
+The gate treats a run as inconclusive when candidate/baseline hashes, the eval
+manifest hash, executor prompt hash, run identity, classification, detailed
+grading payload, or other recorded evidence differs from the preparation-time
+provenance in the iteration manifest or `run_record.json`.
+
+Retry an infrastructure failure once. If the retry also fails, preserve the
+partial artifacts with `status: infrastructure_error`; the gate reports that
+trial as inconclusive instead of scoring it as a skill failure.
+
+## Harness Invariants
+
+- Every trial uses a fresh repository sandbox and a fresh stateless subagent.
+- Candidate and baseline sandboxes begin from the same candidate repository;
+  only the baseline skill directory differs.
+- Input fixtures live outside both skill trees and are identical for both arms.
+- Each trial receives its own fixture copy so one executor cannot mutate another trial's input.
+- The eval corpus and grading harness are excluded from both runtime skill snapshots so
+  executor agents cannot use the answer key while comparing candidate and baseline skills.
+- Copied fixture hashes must match their preparation-time input manifest.
+- Executor prompt hashes must match both the iteration manifest and live prompt files.
+- Critical compound workflows must pass within one command invocation.
+- Positive controls must use supported command paths and required arguments.
+- Executor errors and provenance mismatches are inconclusive, never passing evidence.
+
+#### Grade and aggregate the iteration
+
+```bash
+poetry run python -m slcli.skills.slcli.scripts.benchmark_iteration --force \
+  slcli/skills/slcli-workspace/iteration-1
+```
+
+This grades populated runs, writes `run_record.json` per trial, generates
+`benchmark.json` plus `benchmark.md`, and applies the regression gate. It also
+writes `regression.json` and `regression.md`.
+
+The gate exits with:
+
+- `0`: no detected regression
+- `1`: regression detected
+- `2`: inconclusive because runs or required metadata are missing or incompatible
+
+The default gate fails when the baseline passes the critical rules in a
+majority of trials and any paired candidate trial introduces a critical
+failure. It also fails when mean assertion pass rate drops by more than five
+percentage points.
+
+#### Regenerate the review page
+
+```bash
+poetry run python -m slcli.skills.slcli.scripts.render_eval_review \
+  slcli/skills/slcli-workspace/iteration-1
+```
+
+This writes `review.html` in the iteration directory.
+
+## What Good Looks Like
+
+- `with_skill` should be non-inferior to `old_skill` on the gating suite.
+- Failures should point to missing skill behavior, not to ambiguous eval wording.
+- If both configs pass easily, keep the case for regression coverage and add
+  harder cases to a separate capability suite.
+- If both configs fail, tighten the prompt or grading rules before changing the skill.
+
+## Files in This Directory
+
+- `evals.json`: prompts, fixtures, and grading rules
+- `evals.schema.json`: machine-readable corpus contract
+- `trigger_evals.json`: balanced should-trigger and should-not-trigger prompts
+- `files/`: input fixtures for file-backed evals
+- `../../../../.github/prompts/eval-skill-gating.prompt.md`: one-shot gating eval prompt
+- `COPILOT_BATCH_RUN_PROMPT.md`: parent-chat orchestration prompt
+- `../scripts/prepare_eval_workspace.py`: scaffolds an iteration directory
+- `../scripts/prepare_eval_prompts.py`: writes executor prompts
+- `../scripts/benchmark_iteration.py`: grades and aggregates a full iteration
+- `../scripts/render_eval_review.py`: writes the static review page
+
+## Broader Coverage
+
+Use `regression` with five trials before a release:
+
+```bash
+poetry run python -m slcli.skills.slcli.scripts.prepare_eval_workspace \
+  --suite regression \
+  --runs-per-config 5
+```
+
+## Trigger Evaluation
+
+Trigger evaluation is separate from task execution because task prompts inject
+the skill explicitly. When the Claude CLI is available, run the repository's
+trigger evaluator with balanced thresholds:
+
+```bash
+python .github/skills/skill-creator/scripts/run_eval.py \
+  --eval-set slcli/skills/slcli/evals/trigger_evals.json \
+  --skill-path slcli/skills/slcli \
+  --runs-per-query 5 \
+  --trigger-threshold 0.8 \
+  --negative-trigger-threshold 0.2
+```
+
+This is a local or scheduled check until the production Copilot harness exposes
+skill-selection events through a non-interactive executor. Do not infer routing
+from final-answer text.
+
+## CI Coverage
+
+The existing CI workflow runs the deterministic manifest, grader, snapshot,
+and comparator unit tests on every pull request. Model trials remain an
+explicit developer workflow because this repository does not yet have a
+credentialed non-interactive Copilot executor. Attach `benchmark.json`,
+`regression.json`, `review.html`, run records, transcripts, and outputs to a PR
+when reviewing a skill change.
+
+## What To Commit
+
+Commit the eval harness and definitions:
+
+- `slcli/skills/slcli/SKILL.md`
+- `slcli/skills/slcli/evals/evals.json`
+- `slcli/skills/slcli/evals/files/`
+- `slcli/skills/slcli/evals/README.md`
+- `slcli/skills/slcli/evals/COPILOT_BATCH_RUN_PROMPT.md`
+- `slcli/skills/slcli/scripts/`
+- `slcli/skills/slcli/.github/prompts/` or repo-level prompt files that drive the workflow
+
+Do not usually commit run artifacts:
+
+- `slcli/skills/slcli-workspace/`
+- per-run `outputs/response.txt`
+- per-run `grading.json`
+- generated `benchmark.json` and `benchmark.md`
+- generated `review.html`
+
+Treat the checked-in files as the reproducible test harness and the workspace
+artifacts as local experiment output unless you intentionally want to preserve a
+specific benchmark snapshot for review or release documentation.