@buaa_smat/hometrans 0.1.7 → 0.1.8

package/skills/skill-quality-evaluator/references/report-template.md

@@ -1,163 +0,0 @@
-# Report Template
-
-ALWAYS use this exact structure for the evaluation report. Replace bracketed placeholders with actual content. Skip sections marked N/A only when the corresponding dimension was entirely skipped (e.g., no scripts).
-
----
-
-# Skill Quality Report: [skill-name]
-
-**Evaluated on:** [date]
-**Skill path:** [path]
-**Overall score:** [X.X] / 5.0 — [Rating]
-
----
-
-## Executive Summary
-
-[2-3 sentences: what the skill does, its strongest quality, and the single most impactful improvement it needs.]
-
----
-
-## Dimension Scores
-
-| Dimension | Score | Weight | Weighted |
-|-----------|-------|--------|----------|
-| 1. Spec Compliance | X.X | 10% | X.X |
-| 2. Progressive Disclosure | X.X | 15% | X.X |
-| 3. Content Efficiency | X.X | 20% | X.X |
-| 4. Instruction Quality | X.X | 25% | X.X |
-| 5. Description Effectiveness | X.X | 15% | X.X |
-| 6. Script Quality | X.X / N/A | 5% | X.X |
-| 7. Evaluability | X.X | 10% | X.X |
-| **Overall** | | | **X.X** |
-
----
-
-## Dimension 1 — Specification Compliance: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 1.1 | Frontmatter validity | X | [Specific observation] |
-| 1.2 | Name rules | X | [Specific observation] |
-| 1.3 | Description length | X | [Specific observation] |
-| 1.4 | Directory structure | X | [Specific observation] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 2 — Progressive Disclosure: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 2.1 | SKILL.md size | X | [Line count / observation] |
-| 2.2 | Reference usage | X | [Which references exist, are they focused?] |
-| 2.3 | Load triggers | X | [Does the skill say when to load each reference?] |
-| 2.4 | Path conventions | X | [Relative paths? Chain depth?] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 3 — Content Efficiency: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 3.1 | No generic filler | X | [Examples of filler found, or confirmation of none] |
-| 3.2 | Coherent scope | X | [Scope assessment] |
-| 3.3 | Appropriate detail | X | [Detail level assessment] |
-| 3.4 | Domain grounding | X | [What domain-specific knowledge is present?] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 4 — Instruction Quality: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 4.1 | Calibrated specificity | X | [Examples of good or poor calibration] |
-| 4.2 | Defaults over menus | X | [Where does it provide defaults? Where does it list options?] |
-| 4.3 | Procedural over declarative | X | [Does it teach approach or give specific answers?] |
-| 4.4 | Explains why | X | [Examples of reasoning in instructions] |
-| 4.5 | Gotchas | X | [Presence and quality of gotchas section] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 5 — Description Effectiveness: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 5.1 | Imperative framing | X | [Quote the description or its key phrases] |
-| 5.2 | Intent-focused | X | [Mechanics vs. user intent analysis] |
-| 5.3 | Trigger coverage | X | [What triggers are covered? What's missing?] |
-| 5.4 | Conciseness | X | [Character count and density assessment] |
-| 5.5 | Keyword specificity | X | [Would this false-trigger or miss-trigger?] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 6 — Script Quality: [X.X / 5.0 or N/A]
-
-[If N/A: "No scripts/ directory — this dimension is not applicable."]
-
-[If scored:]
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 6.1 | Self-contained | X | [Dependency handling] |
-| 6.2 | Non-interactive | X | [Any TTY prompts?] |
-| 6.3 | Help output | X | [--help quality] |
-| 6.4 | Error messages | X | [Error message quality] |
-| 6.5 | Structured output | X | [Output format assessment] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Dimension 7 — Evaluability: X.X / 5.0
-
-| # | Criterion | Score | Evidence |
-|---|-----------|-------|----------|
-| 7.1 | Test cases exist | X | [evals/evals.json present? How many cases?] |
-| 7.2 | Realistic prompts | X | [Example prompt quality] |
-| 7.3 | Verifiable assertions | X | [Are assertions specific and checkable?] |
-| 7.4 | Edge case coverage | X | [Any boundary condition tests?] |
-
-**What's good:** [1 sentence]
-**What to improve:** [1 sentence with action]
-
----
-
-## Top 3 Strengths
-
-1. **[Strength title]** — [1-2 sentences with evidence]
-2. **[Strength title]** — [1-2 sentences with evidence]
-3. **[Strength title]** — [1-2 sentences with evidence]
-
----
-
-## Top 3 Improvement Areas
-
-1. **[Issue title]** — [Why it matters] → **Fix:** [Specific, actionable change]
-2. **[Issue title]** — [Why it matters] → **Fix:** [Specific, actionable change]
-3. **[Issue title]** — [Why it matters] → **Fix:** [Specific, actionable change]
-
----
-
-## Quick Wins
-
-[2-3 one-line fixes that would take under 5 minutes and yield immediate improvement. If none are obvious, say "No quick wins identified — improvements require structural changes."]
-
----
-
-*Report generated by skill-quality-evaluator*

package/skills/skill-quality-evaluator/references/scoring-rubric.md

@@ -1,269 +0,0 @@
-# Scoring Rubric
-
-Each sub-criterion is scored on a 1-5 scale. The anchors below define what each score means per criterion. When evidence sits between two anchors, use the intermediate score (2 or 4).
-
-## 1. Specification Compliance
-
-### 1.1 Frontmatter validity
-| Score | Anchor |
-|-------|--------|
-| 1 | YAML is malformed or missing entirely; required fields absent |
-| 3 | YAML parses but has minor issues (e.g., indentation warnings); required fields present but one may be empty |
-| 5 | YAML is well-formed; `name` and `description` both present and non-empty |
-
-### 1.2 Name rules
-| Score | Anchor |
-|-------|--------|
-| 1 | Name is missing, contains uppercase, starts/ends with hyphen, or has consecutive hyphens |
-| 3 | Name follows most rules but has one violation (e.g., trailing hyphen, doesn't match directory) |
-| 5 | Name follows all rules: 1-64 chars, lowercase alphanumeric + hyphens, no leading/trailing/consecutive hyphens, matches directory |
-
-### 1.3 Description length
-| Score | Anchor |
-|-------|--------|
-| 1 | Missing or over 1024 characters |
-| 3 | Present but too short (under ~20 chars, too vague to trigger) or uncomfortably close to the 1024 limit |
-| 5 | Within limit, informative length (typically 100-600 chars) |
-
-### 1.4 Directory structure
-| Score | Anchor |
-|-------|--------|
-| 1 | No SKILL.md at root |
-| 3 | Has SKILL.md but auxiliary directories are disorganized (e.g., scripts mixed into root, references inside assets) |
-| 5 | Clean structure: SKILL.md at root, optional dirs named conventionally (`scripts/`, `references/`, `assets/`) |
-
-### 1.5 No broken references
-| Score | Anchor |
-|-------|--------|
-| 1 | Multiple broken references — SKILL.md references files in scripts/, references/, or assets/ that don't exist; or the skill's resource tables list non-existent files |
-| 3 | One broken or suspicious reference (e.g., the path looks right but the file is missing, or a reference exists in prose but isn't listed in the resource table) |
-| 5 | Every file path mentioned in SKILL.md resolves to an actual file on disk. Every entry in scripts/references/assets tables exists. No dangling references. |
-
-## 2. Progressive Disclosure
-
-### 2.1 SKILL.md size
-| Score | Anchor |
-|-------|--------|
-| 1 | Over 1000 lines — bloated, loads massive context on every invocation |
-| 3 | 500-700 lines — slightly over the recommended 500, or approaching the limit with dense content |
-| 5 | Under 500 lines, tightly scoped to core instructions |
-
-### 2.2 Reference usage
-| Score | Anchor |
-|-------|--------|
-| 1 | All detail crammed into SKILL.md; no references/ even when content is clearly reference-grade (long tables, exhaustive lists, multi-domain variants) |
-| 3 | Has references/ but some material that belongs there is still inline, or reference files are unfocused dumps |
-| 5 | Reference-grade material lives in references/; each reference file has a clear, focused topic |
-
-### 2.3 Load triggers
-| Score | Anchor |
-|-------|--------|
-| 1 | References files without saying when to read them ("see references/ for details") |
-| 3 | Some references have conditional triggers, others are just listed |
-| 5 | Every reference file has a clear conditional trigger ("if X happens, read `references/Y.md`") |
-
-### 2.4 Path conventions
-| Score | Anchor |
-|-------|--------|
-| 1 | Uses absolute paths or deeply nested reference chains (A → B → C → D) |
-| 3 | Uses relative paths but has one case of a 2+ level deep reference chain |
-| 5 | All paths are relative from skill root; references are at most one level deep from SKILL.md |
-
-## 3. Content Efficiency
-
-### 3.1 No generic filler
-| Score | Anchor |
-|-------|--------|
-| 1 | Large sections explain basic concepts the agent already knows (e.g., "JSON is a data format...", "Git is a version control system...") |
-| 3 | A few instances of unnecessary explanation mixed with useful content |
-| 5 | Every sentence earns its place. The "would the agent get this wrong without it?" test passes throughout. |
-
-### 3.2 Coherent scope
-| Score | Anchor |
-|-------|--------|
-| 1 | Scope is either a single trivial action (too narrow) or a sprawling collection of unrelated workflows (too broad) |
-| 3 | Scope is generally coherent but has one notable inclusion/exclusion that doesn't fit |
-| 5 | Encapsulates one coherent unit of work that composes well with other skills |
-
-### 3.3 Appropriate detail
-| Score | Anchor |
-|-------|--------|
-| 1 | Either a vague one-liner with no structure OR an exhaustive reference manual covering every edge case |
-| 3 | Generally good detail level but with sections that are too sparse or too exhaustive |
-| 5 | Concise, stepwise guidance with enough structure to follow reliably; leaves reasonable judgment calls to the agent |
-
-### 3.4 Domain grounding
-| Score | Anchor |
-|-------|--------|
-| 1 | Entirely generic — could have been written by an LLM with no project-specific knowledge |
-| 3 | Some domain-specific content but also significant generic portions |
-| 5 | Rich with project-specific conventions, environment-specific gotchas, non-obvious patterns — things only an expert on *this* project would know |
-
-## 4. Instruction Quality
-
-### 4.1 Calibrated specificity
-| Score | Anchor |
-|-------|--------|
-| 1 | Uniformly rigid (ALWAYS/NEVER for everything) or uniformly vague ("handle this appropriately") |
-| 3 | Mostly well-calibrated but with a few instructions that are too rigid or too loose for their context |
-| 5 | Specificity matches fragility: prescriptive for brittle operations, flexible where multiple approaches work |
-
-### 4.2 Defaults over menus
-| Score | Anchor |
-|-------|--------|
-| 1 | Presents undifferentiated lists of tools/approaches with no guidance on which to pick |
-| 3 | Provides defaults for some choices but presents menus for others |
-| 5 | Every choice point has a clear default with alternatives noted briefly |
-
-### 4.3 Procedural over declarative
-| Score | Anchor |
-|-------|--------|
-| 1 | Gives specific answers for specific cases rather than reusable methods (e.g., exact SQL instead of a query-building approach) |
-| 3 | Has some reusable method descriptions but also some overly specific one-shot instructions |
-| 5 | Teaches *how to approach* a class of problems; instructions generalize beyond the examples |
-
-### 4.4 Explains why
-| Score | Anchor |
-|-------|--------|
-| 1 | Pure directives with zero reasoning; "just do X" throughout |
-| 3 | Some instructions include reasoning but most are bare directives |
-| 5 | Non-obvious instructions consistently include the reasoning behind them |
-
-### 4.5 Gotchas
-| Score | Anchor |
-|-------|--------|
-| 1 | No gotchas section, even though the domain likely has non-obvious pitfalls |
-| 3 | Has gotchas but they are too generic ("handle errors") or miss important ones |
-| 5 | Concrete, environment-specific corrections that would prevent real mistakes; every gotcha is testable |
-
-## 5. Description Effectiveness
-
-### 5.1 Imperative framing
-| Score | Anchor |
-|-------|--------|
-| 1 | Passive description ("This skill helps with PDFs") |
-| 3 | Mix of declarative and imperative |
-| 5 | Consistently imperative: "Use this skill when...", "Trigger on...", clearly instructing the agent |
-
-### 5.2 Intent-focused
-| Score | Anchor |
-|-------|--------|
-| 1 | Describes internal mechanics ("This skill runs a Python script that parses XML and...") |
-| 3 | Mentions both mechanics and user intent |
-| 5 | Describes what the user wants to achieve; implementation details are absent from the description |
-
-### 5.3 Trigger coverage
-| Score | Anchor |
-|-------|--------|
-| 1 | Only triggers on exact keyword matches; misses obvious related requests |
-| 3 | Covers common triggers but has notable gaps (e.g., covers English but not Chinese, covers formal but not casual phrasing) |
-| 5 | Pushy about when to trigger; covers synonyms, casual phrasing, cross-language triggers, and implicit mentions |
-
-### 5.4 Conciseness
-| Score | Anchor |
-|-------|--------|
-| 1 | Either a 5-word fragment or a 1000+ character paragraph |
-| 3 | Reasonable length but could be tighter or slightly more expansive |
-| 5 | Dense with signal; covers scope fully without bloat; well under 1024 chars |
-
-### 5.5 Keyword specificity
-| Score | Anchor |
-|-------|--------|
-| 1 | Keywords are so generic they'd cause false triggers on unrelated tasks, or so narrow they'd miss most real uses |
-| 3 | Keywords are reasonable but would either false-trigger on some adjacent domains or miss some relevant ones |
-| 5 | Keywords precisely carve out the skill's domain; would match relevant prompts and not match adjacent-but-different ones |
-
-## 6. Script Quality
-
-Skip this dimension if no `scripts/` directory exists.
-
-### 6.1 Self-contained
-| Score | Anchor |
-|-------|--------|
-| 1 | Scripts have undocumented external dependencies; fail on first run without manual setup |
-| 3 | Dependencies documented but not declared inline; requires an install step |
-| 5 | Dependencies declared inline (PEP 723, `package.json`, `bundler/inline`) or scripts use only stdlib |
-
-### 6.2 Non-interactive
-| Score | Anchor |
-|-------|--------|
-| 1 | Scripts prompt for input interactively (TTY read, confirm dialog) |
-| 3 | Mostly non-interactive but one script has an interactive fallback |
-| 5 | All input via flags, env vars, or stdin; no TTY dependency |
-
-### 6.3 Help output
-| Score | Anchor |
-|-------|--------|
-| 1 | No `--help` or equivalent |
-| 3 | `--help` exists but is minimal (just lists flags, no examples) |
-| 5 | `--help` shows usage, flag descriptions, and working examples |
-
-### 6.4 Error messages
-| Score | Anchor |
-|-------|--------|
-| 1 | Opaque errors ("Error: invalid input") or silent failures |
-| 3 | Some errors are descriptive, others are cryptic |
-| 5 | Every error says what went wrong, what was expected, and what to try |
-
-### 6.5 Structured output
-| Score | Anchor |
-|-------|--------|
-| 1 | Free-form text output only; no structured format available |
-| 3 | Has a structured output option but it's not the default, or the structured format is poorly chosen |
-| 5 | Structured output (JSON/CSV) is the default or easily selectable; diagnostics go to stderr |
-
-## 7. Evaluability
-
-### 7.1 Test cases exist
-| Score | Anchor |
-|-------|--------|
-| 1 | No evals/ directory or evals.json |
-| 3 | Has evals.json but with only 1 test case |
-| 5 | Has evals/evals.json with 2+ test cases |
-
-### 7.2 Realistic prompts
-| Score | Anchor |
-|-------|--------|
-| 1 | Prompts are generic placeholders ("test the skill", "run the workflow") |
-| 3 | Some prompts are realistic but others are too abstract |
-| 5 | Prompts look like real user messages: file paths, personal context, varied formality, specific details |
-
-### 7.3 Verifiable assertions
-| Score | Anchor |
-|-------|--------|
-| 1 | Assertions are vague ("output is good", "works correctly") or missing entirely |
-| 3 | Mix of specific and vague assertions |
-| 5 | Every assertion is specific, objective, and verifiable — you could write a script to check most of them |
-
-### 7.4 Edge case coverage
-| Score | Anchor |
-|-------|--------|
-| 1 | All test cases are happy-path only |
-| 3 | Has one edge case but it's superficial |
-| 5 | At least one test case thoroughly probes a boundary condition (malformed input, ambiguous request, missing data) |
-
-## Dimension Weights
-
-The overall score is a weighted average of dimension scores. Weights reflect the relative importance of each dimension to skill effectiveness:
-
-| Dimension | Weight | Rationale |
-|-----------|--------|-----------|
-| 1. Spec Compliance | 10% | Table stakes — must pass, but doesn't differentiate great from good |
-| 2. Progressive Disclosure | 15% | Directly affects agent context efficiency and skill composability |
-| 3. Content Efficiency | 20% | The core value question: does this skill add information the agent lacks? |
-| 4. Instruction Quality | 25% | The highest-impact dimension — how well the skill guides the agent |
-| 5. Description Effectiveness | 15% | Determines whether the skill gets used at all |
-| 6. Script Quality | 5% | Optional; important when present but many good skills have no scripts |
-| 7. Evaluability | 10% | Indicates maturity and maintainability |
-
-When a dimension is N/A (e.g., no scripts), redistribute its weight proportionally across the remaining dimensions.
-
-## Overall Score Interpretation
-
-| Range | Rating | Meaning |
-|-------|--------|---------|
-| 4.5 - 5.0 | Excellent | Production-ready; minor polish at most |
-| 3.5 - 4.4 | Good | Solid skill; a few specific improvements would elevate it |
-| 2.5 - 3.4 | Adequate | Functional but has notable gaps; needs targeted work |
-| 1.5 - 2.4 | Weak | Major issues across multiple dimensions; significant revision needed |
-| 1.0 - 1.4 | Poor | Fundamentally incomplete or misaligned with best practices |