procesio-cli 0.1.1__py3-none-any.whl

procesio_cli/skills/agent-skill-engineer/references/decision-framework.md

@@ -0,0 +1,107 @@
+# Skill decision and portfolio framework
+
+Use this reference before creating, splitting, merging, or retiring skills.
+
+## Contents
+
+1. Leverage test
+2. Choose the strongest artifact
+3. Set the degree of freedom
+4. Design portfolio boundaries
+5. Choose an evidence tier
+6. Stop conditions
+
+## 1. Leverage test
+
+A skill is justified when most of these are true:
+
+- **Recurring:** the workflow or failure appears repeatedly, or its first failure would be costly enough to justify prevention.
+- **Model gap:** success depends on organization-specific knowledge, tool usage, decision order, or failure handling that a capable general model will not reliably infer.
+- **Stable core:** the governing workflow is stable enough to encode. Volatile facts can remain in references with an explicit freshness owner.
+- **Behavioral effect:** a reader can name which decision, action, or output should change after the skill loads.
+- **Observable outcome:** success can be inspected in an artifact, state, output, or defensible human preference comparison.
+- **Context leverage:** loading the skill costs less than repeatedly rediscovering or re-explaining the workflow.
+- **Bounded ownership:** the trigger region can be separated from neighboring skills without vague arbitration.
+
+Fail the leverage test when the proposed skill is merely a topic summary, a one-time prompt, a personality claim, or a duplicate of an existing tool or skill.
+
+## 2. Choose the strongest artifact
+
+Use the strongest mechanism that can own the requirement:
+
+| Need | Preferred artifact |
+|---|---|
+| Make an invalid state impossible | type, schema, API boundary, permission, or runtime guard |
+| Repeat a deterministic transformation | script or tool |
+| Store volatile facts, schemas, or policy detail | reference with owner and freshness rule |
+| Guide context-dependent decisions and sequencing | skill |
+| Coordinate multiple specialized executors | agent or orchestrator, with skills for method |
+| Explain something to humans | documentation |
+| Apply a one-time instruction | prompt, not a new skill |
+
+A skill may point to these artifacts. It should not reproduce their full contents.
+
+## 3. Set the degree of freedom
+
+Match instruction precision to the cost of variation:
+
+- **High freedom:** use principles and decision criteria when several approaches are valid and the environment decides which is best.
+- **Medium freedom:** use a checklist, pseudocode, or parameterized template when the shape is stable but details vary.
+- **Low freedom:** use an executable helper or exact sequence when operations are fragile, destructive, security-sensitive, format-sensitive, or repeatedly misimplemented.
+
+Do not use prose to simulate determinism. Do not use a rigid script where expert judgment is the actual work.
+
+## 4. Design portfolio boundaries
+
+### One-intent ownership test
+
+For every realistic prompt, identify one primary skill owner. Composition may call another skill later, but discovery should not require a tie-breaker between two descriptions.
+
+### Split a skill when
+
+- separate user intents activate unrelated workflows;
+- most requests load large irrelevant sections;
+- different parts require different permissions, evidence tiers, or freshness owners;
+- one description becomes a list of loosely related nouns;
+- negative routing cases cannot be expressed without suppressing valid triggers.
+
+### Merge skills when
+
+- they compete for the same prompts and always execute the same core workflow;
+- one is only a thin preamble to the other;
+- users cannot predict which skill owns the outcome;
+- separate packages duplicate invariants, references, or evaluation cases.
+
+### Compose rather than copy when
+
+- a domain workflow needs a general verification or writing method;
+- a repository maintainer must integrate a skill after the skill engineer designs it;
+- an operational skill invokes a specialized optimizer and then returns to verify the surrounding system.
+
+Name the handoff and the return condition. Do not paste the downstream skill into the caller.
+
+## 5. Choose an evidence tier
+
+| Tier | Typical skill | Minimum evidence |
+|---|---|---|
+| 0 — subjective | personal style, ideation tone | representative examples plus blinded human preference review |
+| 1 — guidance | low-risk research or drafting workflow | positive/negative/overlap routing cases and qualitative output review |
+| 2 — operational | code, data, CLI, API, or artifact workflow | immutable baseline, fixed atomic rubrics, paired fresh contexts, deterministic structural checks |
+| 3 — high consequence | mutations, credentials, security, compliance, finance, production operations | Tier 2 plus A/A noise measurement, pre-registered A/B thresholds, approval gates, and controlled field proof |
+
+Increase rigor with consequence and ambiguity. Do not force fake quantitative scoring onto genuinely subjective work.
+
+## 6. Stop conditions
+
+Stop and recommend a different artifact when:
+
+- the user cannot name any repeated task or costly failure;
+- the requested content is mostly current facts better retrieved from a source;
+- a script, schema, or permission boundary can enforce the behavior completely;
+- the proposed description would overlap most of the portfolio;
+- there is no observable outcome and no credible human comparison method;
+- the author wants to skip baseline evidence and still claim improvement;
+- the workflow requires secret material inside the skill;
+- the package needs hidden behavior not disclosed by its description.
+
+When the leverage case is promising but evidence is incomplete, create a clearly marked draft and state the next proof. Do not inflate the skill to compensate for uncertainty.

procesio_cli/skills/agent-skill-engineer/references/domain-evidence-protocol.md

@@ -0,0 +1,248 @@
+# Domain evidence and expert synthesis protocol
+
+Use this reference before encoding specialized, high-consequence, or organization-specific expertise into a skill.
+
+## Contents
+
+1. What expert synthesis is
+2. Decision inventory
+3. Claim and evidence ledger
+4. Source hierarchy
+5. Eliciting tacit expertise
+6. Selecting independent expert lenses
+7. Shared evidence packet
+8. Independent review contract
+9. Resolving disagreement
+10. Converting expertise into executable guidance
+11. Calibration and refusal
+12. Accountable human review
+13. Maintenance and drift
+
+## 1. What expert synthesis is
+
+Expert synthesis is a method for exposing a proposed workflow to distinct professional failure models and evidence. It is not role-play, prestige language, or a claim that several model personas equal qualified practitioners.
+
+The output is a calibrated decision model:
+
+- what cues matter;
+- which action or handoff follows;
+- which invariants must hold;
+- what evidence changes the decision;
+- where uncertainty remains;
+- who has authority to approve or overrule the result.
+
+Choose reviewers because their lens can change a material design decision, not to maximize their count.
+
+## 2. Decision inventory
+
+Before collecting facts, enumerate the decisions the skill must support. For each decision, record:
+
+- trigger and context;
+- candidate actions or abstention;
+- required inputs and observable state;
+- irreversible or high-cost consequences;
+- default under missing evidence;
+- escalation owner;
+- direct proof of a correct outcome.
+
+Do not research a broad topic without tying evidence to a decision. Topic encyclopedias consume context while leaving the agent's choice ambiguous.
+
+## 3. Claim and evidence ledger
+
+Maintain a ledger with one row per material claim:
+
+| Field | Meaning |
+|---|---|
+| `claim_id` | stable identifier |
+| `claim` | one falsifiable statement |
+| `class` | fact, invariant, heuristic, policy, preference, hypothesis, or unknown |
+| `scope` | systems, versions, jurisdictions, populations, or conditions where it applies |
+| `source` | primary artifact, observation, study, standard, or accountable owner |
+| `observed_on` | date/version of evidence |
+| `confidence` | calibrated level with reason |
+| `disconfirming_evidence` | evidence that would weaken or reverse the claim |
+| `decision_impact` | exact instruction, boundary, or evaluation criterion affected |
+| `owner` | person or system responsible for refresh |
+
+Never merge several claims under one citation. Distinguish a source's assertion from independent empirical confirmation. Mark local convention as policy, not universal domain truth.
+
+## 4. Source hierarchy
+
+Prefer evidence in this order, adjusted for the domain:
+
+1. **Direct state and authoritative artifacts:** source code, schemas, contracts, controlled measurements, signed policy, primary records, and real user outcomes.
+2. **Primary standards and research:** normative specifications, official guidance, peer-reviewed or directly inspectable studies, and regulator material.
+3. **Expert demonstrations and critical incidents:** successful and failed cases with decision context and outcome evidence.
+4. **Current first-party documentation:** product behavior, supported interfaces, and release-specific constraints.
+5. **High-quality synthesis:** secondary reviews used to discover primary evidence or map disagreement.
+6. **Anecdote or model memory:** hypothesis generation only, never the sole basis for a consequential invariant.
+
+Check version, date, jurisdiction, incentives, sampling, and applicability. Absence of evidence is not evidence that a risky behavior is safe.
+
+## 5. Eliciting tacit expertise
+
+Generic interviews produce generic rules. Elicit decisions through concrete cases:
+
+- Ask for the last successful case, the last failure, and a near miss.
+- Reconstruct the timeline, information available at each moment, alternatives considered, and signals that changed the decision.
+- Ask what a competent novice would likely miss.
+- Compare superficially similar cases with different correct actions.
+- Identify cues that were ignored, misleading, or discovered too late.
+- Capture workarounds, exception ownership, and recovery behavior.
+- Verify recalled explanations against artifacts and outcomes where possible.
+
+Separate what the expert did from why it worked. A repeated practice may be habit, policy, environmental constraint, or causal mechanism; encode it only after classification.
+
+## 6. Selecting independent expert lenses
+
+Select every lens implicated by consequence and uncertainty. Typical lenses include:
+
+- domain subject-matter correctness;
+- cognitive and instruction design;
+- information architecture and context economics;
+- tool/API and systems architecture;
+- security and privacy;
+- reliability, idempotency, and observability;
+- human factors and accessibility;
+- evaluation design and statistics;
+- portability and integration;
+- adversarial misuse;
+- future maintenance and source drift.
+
+For high-consequence domains, add the accountable roles required by law, policy, or organizational authority. A model can simulate questions and failure modes; it cannot assume licensure, fiduciary duty, clinical responsibility, legal authority, or production ownership.
+
+## 7. Shared evidence packet
+
+Give every reviewer the same immutable packet:
+
+- user outcome and explicit non-goals;
+- decision inventory and claim ledger;
+- positive, negative, overlap, missing-context, pressure, and success cases;
+- baseline behavior and known failures;
+- target clients, tools, permissions, side effects, and evidence tier;
+- candidate designs under opaque labels where comparison is required;
+- fixed review questions and acceptance criteria;
+- known source conflicts and uncertainties.
+
+Do not prime reviewers with another reviewer's conclusion. Do not give one candidate a richer context or more favorable evidence.
+
+## 8. Independent review contract
+
+Each reviewer returns findings independently using:
+
+```text
+finding_id
+lens
+claim_or_invariant_at_issue
+evidence
+scope
+confidence
+failure_consequence
+proposed_design_change
+disconfirming_condition
+severity: blocker | major | minor | preference
+```
+
+A finding without distinct evidence, consequence, or design impact is not counted as expert signal. Require reviewers to identify both false-positive and false-negative risks. Ask them to name what should remain unchanged.
+
+Use adversarial pre-mortems:
+
+- How could this skill confidently produce a wrong result?
+- What prompt makes it cross a permission or ownership boundary?
+- What baseline success could a new rule damage?
+- What proxy could be mistaken for completion?
+- What stale fact could remain plausible after the underlying system changes?
+- What rare subgroup or edge state disappears in an average metric?
+
+## 9. Resolving disagreement
+
+Do not vote by reviewer count or status. Resolve conflicts in this order:
+
+1. direct outcome evidence and authoritative constraints;
+2. user objective and non-negotiable rights or safety boundaries;
+3. applicability and scope of each claim;
+4. reversible experiment capable of discriminating between hypotheses;
+5. accountable human decision when evidence cannot resolve a policy or authority question.
+
+Preserve unresolved dissent in the ledger. Narrow the skill's scope when experts disagree because conditions differ. Reject a blended compromise that violates both coherent models.
+
+When a disagreement is empirical, design the smallest safe test. When it is normative, identify the authorized decision-maker rather than disguising preference as fact.
+
+## 10. Converting expertise into executable guidance
+
+Transform evidence into compact structures:
+
+### Decision table
+
+| Observable cue | Required evidence | Action | Do not do | Proof or handoff |
+|---|---|---|---|---|
+
+### State transition
+
+```text
+precondition → inspect → classify → preview → approve → execute once → observe → recover → report
+```
+
+### Invariant
+
+```text
+Must remain true before, during, and after the workflow; enforced by <script/schema/runtime/test> where possible.
+```
+
+### Escalation rule
+
+```text
+When <condition> or evidence confidence is below <registered level>, stop and hand off to <accountable owner> with <evidence packet>.
+```
+
+Encode stable recurring decisions in the body. Put detailed variants and dated facts in references. Put deterministic checks and transformations in scripts. Put permissions and invalid-state prevention in executable boundaries.
+
+## 11. Calibration and refusal
+
+Require the skill to distinguish:
+
+- directly observed fact;
+- source-supported fact;
+- inference;
+- estimate;
+- local policy;
+- recommendation;
+- unresolved uncertainty.
+
+Calibrate confidence to evidence quality, not prose fluency. State the scope in which a rule was validated. Define stop conditions where missing information would materially change the result.
+
+A refusal or handoff is correct when:
+
+- the skill lacks evidence for a consequential decision;
+- a required licensed or authorized reviewer is absent;
+- the target lies outside validated scope;
+- a safety invariant cannot be enforced or verified;
+- user instructions conflict with law, policy, consent, or protected rights;
+- the requested certainty exceeds available evidence.
+
+## 12. Accountable human review
+
+Require human signoff when the skill affects legal rights, clinical care, regulated finance, public or physical safety, compliance attestation, employment decisions, security authorization, or other domains where accountability cannot be delegated to a model.
+
+Record:
+
+- reviewer role and authority;
+- exact version and evidence reviewed;
+- approved scope and exclusions;
+- unresolved risks and monitoring plan;
+- renewal condition and expiration date where relevant.
+
+Human approval is not a substitute for testing. Model review is not a substitute for accountable approval. Use both when the consequence requires both.
+
+## 13. Maintenance and drift
+
+Assign every material source and policy an owner and refresh condition. Re-evaluate when:
+
+- a standard, law, product, model, API, or organizational policy changes;
+- a new critical incident contradicts an encoded rule;
+- users repeatedly override the same recommendation;
+- field outcomes diverge from synthetic evaluation;
+- expert disagreement emerges in a previously stable area;
+- the skill expands to a new jurisdiction, population, system, or risk tier.
+
+Retire unsupported claims rather than burying caveats. Preserve the old package and evidence so maintainers can determine whether a change is true learning, changed conditions, or regression.

procesio_cli/skills/agent-skill-engineer/references/evaluation-standard.md

@@ -0,0 +1,191 @@
+# Agent Skill evaluation standard
+
+Use this reference before writing evaluation cases, jury prompts, baselines, or performance claims.
+
+## Contents
+
+1. What to measure
+2. Build the corpus
+3. Choose the baseline
+4. Freeze a fixed rubric
+5. Run paired evaluations
+6. Establish evaluator noise
+7. Field proof
+8. Diagnose failures
+9. Report honestly
+
+## 1. What to measure
+
+Separate these outcomes:
+
+- **Selection:** did the router select the intended skill or abstain?
+- **Collision:** did it select a specifically forbidden neighboring skill?
+- **Task behavior:** did the response satisfy the fixed behavioral criteria?
+- **Safety:** did it avoid prohibited actions, false claims, or unsafe retries?
+- **Verification:** did it demand or produce direct evidence of the real outcome?
+- **Efficiency:** tokens, latency, tool calls, retries, and unnecessary context.
+- **Field success:** did the actual external artifact or system reach the expected state?
+
+Do not let a jury infer objective selection accuracy. Compute it from `selected_skill` and `expected_skill` in host code.
+
+## 2. Build the corpus
+
+Use realistic prompts, not instructions written to reveal the expected answer. Include:
+
+- positive/core cases;
+- negative and abstention cases;
+- nearest-neighbor overlap cases;
+- missing-context cases;
+- pressure or adversarial shortcut cases;
+- common paraphrases and terse user language;
+- at least one real field task when the skill is operational.
+
+Keep a holdout set for later changes when the corpus is large enough. Add production failures only after removing sensitive data and preserving the causal shape.
+
+## 3. Choose the baseline
+
+- **New skill:** compare with no skill loaded.
+- **Existing skill:** compare with an immutable old-skill snapshot.
+- **Description-only change:** keep the body byte-identical and compare discovery behavior.
+- **Portfolio redesign:** snapshot the entire old portfolio, not one isolated skill.
+- **Evaluator change:** run A/A with byte-identical corpora before interpreting A/B.
+
+Record corpus fingerprints, model, provider, settings, seed, prompt files, and evaluator version. Do not silently update a baseline during a run.
+
+## 4. Freeze a fixed rubric
+
+Write atomic binary criteria before viewing candidate results. Every criterion needs:
+
+```json
+{
+  "id": "descriptive_snake_case_id",
+  "description": "Pass only when the response ...",
+  "required": true
+}
+```
+
+Use two to eight criteria per case. Each criterion should test one observable requirement and be usable without imagining what a better answer might have said.
+
+The complete rubric object is:
+
+```json
+{
+  "rubric_version": 1,
+  "criteria": [
+    {
+      "id": "checks_precondition_before_mutation",
+      "description": "Pass only when the response requires the named precondition before any write.",
+      "required": true
+    },
+    {
+      "id": "verifies_the_real_outcome",
+      "description": "Pass only when the response requires direct inspection of the resulting state or output.",
+      "required": true
+    }
+  ]
+}
+```
+
+### Jury contract
+
+Every juror for the case receives the same canonical rubric bytes or equivalent canonical serialization. The juror must:
+
+- return every supplied criterion ID exactly once and in the supplied order;
+- return no additional, renamed, merged, or split IDs;
+- use JSON booleans only;
+- mark absent or ambiguous evidence false;
+- provide brief evidence for review;
+- never choose the aggregate verdict.
+
+Host code validates the returned IDs and computes:
+
+```text
+task_success = valid_contract AND all(required criterion booleans)
+```
+
+Store the rubric fingerprint with every observation. A prose expectation that asks each juror to derive its own checks is not a fixed rubric.
+
+## 5. Run paired evaluations
+
+For every case and repetition:
+
+1. use fresh response contexts;
+2. use the same task and input files for both variants;
+3. hold model, provider, settings, tools, and timeouts constant;
+4. mount each corpus at a neutral path;
+5. randomize opaque labels;
+6. keep the rubric hidden from the response-producing candidate;
+7. judge each response independently against the frozen rubric;
+8. checkpoint every completed observation;
+9. preserve raw sanitized outputs and machine-readable grades.
+
+Do not launch all candidate runs before all baseline runs. Interleave or randomize them so time, load, and provider drift do not align with one variant.
+
+## 6. Establish evaluator noise
+
+Before A/B, run A/A against two byte-identical copies. Pre-register:
+
+- minimum repetitions;
+- maximum acceptable selection, task-success, and collision deltas;
+- minimum A/B effect worth detecting;
+- number of consecutive passing rounds;
+- stop and retry policy for quota or infrastructure failures.
+
+If A/A fails, do not loosen the threshold or proceed to A/B. Pair disagreements by case and repetition. Determine whether variation came from response generation, rubric ambiguity, juror interpretation, host aggregation, or execution environment. Version the correction and start a new A/A run from zero.
+
+A fixed rubric reduces decomposition drift; it does not eliminate borderline interpretation or response variance. Keep direct human review of discordant pairs.
+
+## 7. Field proof
+
+Synthetic response grading cannot prove that a skill successfully controls a real tool or platform. For operational skills, add a controlled field trial:
+
+1. isolate a disposable or otherwise explicitly bounded target;
+2. confirm authentication and scope;
+3. record the before-state;
+4. preview and validate the intended mutation;
+5. obtain explicit approval;
+6. execute exactly once per registered acceptance claim;
+7. capture stable resource and execution IDs;
+8. inspect the actual output and side effects;
+9. perform approved safety containment even if output verification fails; preserve evidence before authorized destructive cleanup;
+10. prove the baseline state was restored where cleanup is expected.
+
+Before the first mutation, apply `field-gate-standard.md`: freeze ordered acceptance check IDs, their direct proof boundary, permitted gaps and equivalent fallbacks, resource and write caps, the full parent/child execution budget, cleanup, unknown-outcome handling, and host-side promotion rules.
+
+The execution agent may supply evidence for each check, but deterministic host code should validate exact IDs and order, Boolean values, budgets, unknown outcomes, cleanup, and gap eligibility. The acting agent's own `passed` or `passed_with_gap` label is not the aggregate verdict.
+
+A platform limitation discovered after execution does not retroactively permit a gap. Preserve the original blocked/failed report, classify the causal layer, and use a separately approved versioned remediation contract to rerun only missing claims. Archive the original evidence and disclose the remediation's additional mutations and full causal execution count.
+
+A status code, “finished” flag, screenshot, or agent self-report is not enough when the user’s actual output can be read directly.
+
+## 8. Diagnose failures
+
+Classify before editing:
+
+- **routing defect:** wrong owner or abstention;
+- **instruction defect:** selected skill omits or misorders a required decision;
+- **resource defect:** stale or missing reference/script/asset;
+- **tool defect:** executable contract, API, auth, or environment failure;
+- **observation defect:** test checks a proxy or wrong state;
+- **rubric defect:** criterion is ambiguous, bundled, or impossible in the test context;
+- **juror defect:** exact rubric is inconsistently applied;
+- **host defect:** IDs, booleans, pairing, aggregation, field checks, budgets, or promotion are accepted incorrectly;
+- **field-contract defect:** required outcomes, allowed gaps, nested side effects, or cleanup were not frozen before mutation.
+
+Change only the causal layer. Do not add skill prose to solve a tool, jury, controller, or one-off project problem.
+
+## 9. Report honestly
+
+Publish:
+
+- exact corpus and rubric versions;
+- baseline identity and fingerprints;
+- model and settings;
+- repetitions, dropouts, timeouts, and quota interruptions;
+- per-case selection, collisions, criterion booleans, and field results;
+- aggregate rates with variance or confidence intervals where meaningful;
+- field acceptance check IDs, permitted gaps, complete causal execution counts, cleanup, and remediation lineage;
+- regressions and unresolved discordant pairs;
+- limitations of synthetic, judge-based, and field evidence.
+
+Use “candidate,” “draft,” or “provisional” until the registered proof is complete. Never convert a partial run, failed A/A, or failed required field outcome into evidence of superiority or completion.