conectese 0.1.14

package/skills/conectese-skill-creator/references/schemas.md

@@ -0,0 +1,430 @@
+# JSON Schemas
+
+This document defines the JSON schemas used by skill-creator.
+
+---
+
+## evals.json
+
+Defines the evals for a skill. Located at `evals/evals.json` within the skill directory.
+
+```json
+{
+  "skill_name": "example-skill",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "User's example prompt",
+      "expected_output": "Description of expected result",
+      "files": ["evals/files/sample1.pdf"],
+      "expectations": [
+        "The output includes X",
+        "The skill used script Y"
+      ]
+    }
+  ]
+}
+```
+
+**Fields:**
+- `skill_name`: Name matching the skill's frontmatter
+- `evals[].id`: Unique integer identifier
+- `evals[].prompt`: The task to execute
+- `evals[].expected_output`: Human-readable description of success
+- `evals[].files`: Optional list of input file paths (relative to skill root)
+- `evals[].expectations`: List of verifiable statements
+
+---
+
+## history.json
+
+Tracks version progression in Improve mode. Located at workspace root.
+
+```json
+{
+  "started_at": "2026-01-15T10:30:00Z",
+  "skill_name": "pdf",
+  "current_best": "v2",
+  "iterations": [
+    {
+      "version": "v0",
+      "parent": null,
+      "expectation_pass_rate": 0.65,
+      "grading_result": "baseline",
+      "is_current_best": false
+    },
+    {
+      "version": "v1",
+      "parent": "v0",
+      "expectation_pass_rate": 0.75,
+      "grading_result": "won",
+      "is_current_best": false
+    },
+    {
+      "version": "v2",
+      "parent": "v1",
+      "expectation_pass_rate": 0.85,
+      "grading_result": "won",
+      "is_current_best": true
+    }
+  ]
+}
+```
+
+**Fields:**
+- `started_at`: ISO timestamp of when improvement started
+- `skill_name`: Name of the skill being improved
+- `current_best`: Version identifier of the best performer
+- `iterations[].version`: Version identifier (v0, v1, ...)
+- `iterations[].parent`: Parent version this was derived from
+- `iterations[].expectation_pass_rate`: Pass rate from grading
+- `iterations[].grading_result`: "baseline", "won", "lost", or "tie"
+- `iterations[].is_current_best`: Whether this is the current best version
+
+---
+
+## grading.json
+
+Output from the grader agent. Located at `<run-dir>/grading.json`.
+
+```json
+{
+  "expectations": [
+    {
+      "text": "The output includes the name 'John Smith'",
+      "passed": true,
+      "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
+    },
+    {
+      "text": "The spreadsheet has a SUM formula in cell B10",
+      "passed": false,
+      "evidence": "No spreadsheet was created. The output was a text file."
+    }
+  ],
+  "summary": {
+    "passed": 2,
+    "failed": 1,
+    "total": 3,
+    "pass_rate": 0.67
+  },
+  "execution_metrics": {
+    "tool_calls": {
+      "Read": 5,
+      "Write": 2,
+      "Bash": 8
+    },
+    "total_tool_calls": 15,
+    "total_steps": 6,
+    "errors_encountered": 0,
+    "output_chars": 12450,
+    "transcript_chars": 3200
+  },
+  "timing": {
+    "executor_duration_seconds": 165.0,
+    "grader_duration_seconds": 26.0,
+    "total_duration_seconds": 191.0
+  },
+  "claims": [
+    {
+      "claim": "The form has 12 fillable fields",
+      "type": "factual",
+      "verified": true,
+      "evidence": "Counted 12 fields in field_info.json"
+    }
+  ],
+  "user_notes_summary": {
+    "uncertainties": ["Used 2023 data, may be stale"],
+    "needs_review": [],
+    "workarounds": ["Fell back to text overlay for non-fillable fields"]
+  },
+  "eval_feedback": {
+    "suggestions": [
+      {
+        "assertion": "The output includes the name 'John Smith'",
+        "reason": "A hallucinated document that mentions the name would also pass"
+      }
+    ],
+    "overall": "Assertions check presence but not correctness."
+  }
+}
+```
+
+**Fields:**
+- `expectations[]`: Graded expectations with evidence
+- `summary`: Aggregate pass/fail counts
+- `execution_metrics`: Tool usage and output size (from executor's metrics.json)
+- `timing`: Wall clock timing (from timing.json)
+- `claims`: Extracted and verified claims from the output
+- `user_notes_summary`: Issues flagged by the executor
+- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising
+
+---
+
+## metrics.json
+
+Output from the executor agent. Located at `<run-dir>/outputs/metrics.json`.
+
+```json
+{
+  "tool_calls": {
+    "Read": 5,
+    "Write": 2,
+    "Bash": 8,
+    "Edit": 1,
+    "Glob": 2,
+    "Grep": 0
+  },
+  "total_tool_calls": 18,
+  "total_steps": 6,
+  "files_created": ["filled_form.pdf", "field_values.json"],
+  "errors_encountered": 0,
+  "output_chars": 12450,
+  "transcript_chars": 3200
+}
+```
+
+**Fields:**
+- `tool_calls`: Count per tool type
+- `total_tool_calls`: Sum of all tool calls
+- `total_steps`: Number of major execution steps
+- `files_created`: List of output files created
+- `errors_encountered`: Number of errors during execution
+- `output_chars`: Total character count of output files
+- `transcript_chars`: Character count of transcript
+
+---
+
+## timing.json
+
+Wall clock timing for a run. Located at `<run-dir>/timing.json`.
+
+**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.
+
+```json
+{
+  "total_tokens": 84852,
+  "duration_ms": 23332,
+  "total_duration_seconds": 23.3,
+  "executor_start": "2026-01-15T10:30:00Z",
+  "executor_end": "2026-01-15T10:32:45Z",
+  "executor_duration_seconds": 165.0,
+  "grader_start": "2026-01-15T10:32:46Z",
+  "grader_end": "2026-01-15T10:33:12Z",
+  "grader_duration_seconds": 26.0
+}
+```
+
+---
+
+## benchmark.json
+
+Output from Benchmark mode. Located at `benchmarks/<timestamp>/benchmark.json`.
+
+```json
+{
+  "metadata": {
+    "skill_name": "pdf",
+    "skill_path": "/path/to/pdf",
+    "executor_model": "claude-sonnet-4-20250514",
+    "analyzer_model": "most-capable-model",
+    "timestamp": "2026-01-15T10:30:00Z",
+    "evals_run": [1, 2, 3],
+    "runs_per_configuration": 3
+  },
+
+  "runs": [
+    {
+      "eval_id": 1,
+      "eval_name": "Ocean",
+      "configuration": "with_skill",
+      "run_number": 1,
+      "result": {
+        "pass_rate": 0.85,
+        "passed": 6,
+        "failed": 1,
+        "total": 7,
+        "time_seconds": 42.5,
+        "tokens": 3800,
+        "tool_calls": 18,
+        "errors": 0
+      },
+      "expectations": [
+        {"text": "...", "passed": true, "evidence": "..."}
+      ],
+      "notes": [
+        "Used 2023 data, may be stale",
+        "Fell back to text overlay for non-fillable fields"
+      ]
+    }
+  ],
+
+  "run_summary": {
+    "with_skill": {
+      "pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90},
+      "time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0},
+      "tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100}
+    },
+    "without_skill": {
+      "pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45},
+      "time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0},
+      "tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500}
+    },
+    "delta": {
+      "pass_rate": "+0.50",
+      "time_seconds": "+13.0",
+      "tokens": "+1700"
+    }
+  },
+
+  "notes": [
+    "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
+    "Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent",
+    "Without-skill runs consistently fail on table extraction expectations",
+    "Skill adds 13s average execution time but improves pass rate by 50%"
+  ]
+}
+```
+
+**Fields:**
+- `metadata`: Information about the benchmark run
+  - `skill_name`: Name of the skill
+  - `timestamp`: When the benchmark was run
+  - `evals_run`: List of eval names or IDs
+  - `runs_per_configuration`: Number of runs per config (e.g. 3)
+- `runs[]`: Individual run results
+  - `eval_id`: Numeric eval identifier
+  - `eval_name`: Human-readable eval name (used as section header in the viewer)
+  - `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding)
+  - `run_number`: Integer run number (1, 2, 3...)
+  - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`
+- `run_summary`: Statistical aggregates per configuration
+  - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields
+  - `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"`
+- `notes`: Freeform observations from the analyzer
+
+**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.
+
+---
+
+## comparison.json
+
+Output from blind comparator. Located at `<grading-dir>/comparison-N.json`.
+
+```json
+{
+  "winner": "A",
+  "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.",
+  "rubric": {
+    "A": {
+      "content": {
+        "correctness": 5,
+        "completeness": 5,
+        "accuracy": 4
+      },
+      "structure": {
+        "organization": 4,
+        "formatting": 5,
+        "usability": 4
+      },
+      "content_score": 4.7,
+      "structure_score": 4.3,
+      "overall_score": 9.0
+    },
+    "B": {
+      "content": {
+        "correctness": 3,
+        "completeness": 2,
+        "accuracy": 3
+      },
+      "structure": {
+        "organization": 3,
+        "formatting": 2,
+        "usability": 3
+      },
+      "content_score": 2.7,
+      "structure_score": 2.7,
+      "overall_score": 5.4
+    }
+  },
+  "output_quality": {
+    "A": {
+      "score": 9,
+      "strengths": ["Complete solution", "Well-formatted", "All fields present"],
+      "weaknesses": ["Minor style inconsistency in header"]
+    },
+    "B": {
+      "score": 5,
+      "strengths": ["Readable output", "Correct basic structure"],
+      "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"]
+    }
+  },
+  "expectation_results": {
+    "A": {
+      "passed": 4,
+      "total": 5,
+      "pass_rate": 0.80,
+      "details": [
+        {"text": "Output includes name", "passed": true}
+      ]
+    },
+    "B": {
+      "passed": 3,
+      "total": 5,
+      "pass_rate": 0.60,
+      "details": [
+        {"text": "Output includes name", "passed": true}
+      ]
+    }
+  }
+}
+```
+
+---
+
+## analysis.json
+
+Output from post-hoc analyzer. Located at `<grading-dir>/analysis.json`.
+
+```json
+{
+  "comparison_summary": {
+    "winner": "A",
+    "winner_skill": "path/to/winner/skill",
+    "loser_skill": "path/to/loser/skill",
+    "comparator_reasoning": "Brief summary of why comparator chose winner"
+  },
+  "winner_strengths": [
+    "Clear step-by-step instructions for handling multi-page documents",
+    "Included validation script that caught formatting errors"
+  ],
+  "loser_weaknesses": [
+    "Vague instruction 'process the document appropriately' led to inconsistent behavior",
+    "No script for validation, agent had to improvise"
+  ],
+  "instruction_following": {
+    "winner": {
+      "score": 9,
+      "issues": ["Minor: skipped optional logging step"]
+    },
+    "loser": {
+      "score": 6,
+      "issues": [
+        "Did not use the skill's formatting template",
+        "Invented own approach instead of following step 3"
+      ]
+    }
+  },
+  "improvement_suggestions": [
+    {
+      "priority": "high",
+      "category": "instructions",
+      "suggestion": "Replace 'process the document appropriately' with explicit steps",
+      "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
+    }
+  ],
+  "transcript_insights": {
+    "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script",
+    "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods"
+  }
+}
+```

package/skills/conectese-skill-creator/references/skill-format.md

@@ -0,0 +1,235 @@
+# Conectese Skill Format Reference
+
+## SKILL.md Structure
+
+Every Conectese skill consists of a `SKILL.md` file with YAML frontmatter and a Markdown body.
+
+### Common Frontmatter Fields (all types)
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `name` | Yes | Skill identifier (lowercase, hyphenated) |
+| `description` | Yes | What the skill does and when to use it |
+| `type` | Yes | `mcp`, `script`, `hybrid`, or `prompt` |
+| `version` | Yes | Semver version string (e.g., `1.0.0`) |
+| `categories` | No | Classification tags array (e.g., `["social-media", "content"]`) |
+| `env` | No | Required environment variable names array |
+
+### Type: mcp
+
+MCP skills connect to external APIs via Model Context Protocol servers.
+
+Additional fields under `mcp:`:
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `server_name` | Yes | Key used in MCP config |
+| `command` | Yes (stdio) | Command to run the server |
+| `args` | Yes (stdio) | Command arguments array |
+| `transport` | Yes | `stdio` or `http` |
+| `url` | Yes (http) | Server URL for HTTP transport |
+| `headers` | No | Auth headers mapping (key-value pairs) |
+
+### Type: script
+
+Script skills run custom code via a specific runtime.
+
+Additional fields under `script:`:
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `path` | Yes | Script path relative to skill directory |
+| `runtime` | Yes | `node`, `python`, or `bash` |
+| `dependencies` | No | Package list to install before running |
+| `invoke` | Yes | Full invocation command template |
+
+### Type: hybrid
+
+Hybrid skills combine both MCP and script components. Include both `mcp:` and `script:` sections as defined above.
+
+### Type: prompt
+
+Prompt skills are pure behavioral instructions for agents with no external integration. No additional fields beyond the common fields are required — the Markdown body is the entire skill.
+
+---
+
+## Markdown Body
+
+The body contains instructions for agents. Recommended sections:
+
+- `## When to use` — describes when agents should activate this skill
+- `## Instructions` — step-by-step usage guide
+- `## Available operations` — what the skill can do (especially for MCP/script types)
+- `## Examples` — input/output examples to guide agent behavior
+- `## Error handling` — what to do when things go wrong
+
+---
+
+## Complete Examples
+
+### Example: MCP Skill (Canva Integration)
+
+```yaml
+---
+name: canva-design
+description: Create and manage Canva designs. Use when users want to generate social media graphics, presentations, or any visual content via Canva.
+type: mcp
+version: 1.0.0
+categories:
+  - design
+  - social-media
+env:
+  - CANVA_API_KEY
+mcp:
+  server_name: canva
+  command: npx
+  args: ["-y", "@anthropic/canva-mcp-server"]
+  transport: stdio
+---
+
+# Canva Design Skill
+
+## When to use
+
+Use this skill when the user or squad pipeline needs to create visual content — social media posts, presentations, flyers, or any graphic design task that benefits from Canva's template library.
+
+## Instructions
+
+1. Start by understanding the design requirements (type, dimensions, brand colors)
+2. Search for relevant templates using `search-brand-templates`
+3. Generate the design with `generate-design`
+4. Present candidates to the user for selection
+5. Export in the requested format
+
+## Available operations
+
+- Search templates and designs
+- Generate new designs from prompts
+- Edit existing designs (text, images, layout)
+- Export to PDF, PNG, JPG, PPTX
+```
+
+### Example: Script Skill (Data Processor)
+
+```yaml
+---
+name: csv-analyzer
+description: Analyze CSV files and generate summary reports. Use when users need data analysis, column statistics, or trend detection from CSV/spreadsheet data.
+type: script
+version: 1.0.0
+categories:
+  - data
+  - analytics
+script:
+  path: scripts/analyze.py
+  runtime: python
+  dependencies:
+    - pandas
+    - matplotlib
+  invoke: python scripts/analyze.py --input {input_file} --output {output_dir}
+---
+
+# CSV Analyzer Skill
+
+## When to use
+
+Use this skill when the pipeline needs to analyze tabular data from CSV files — generating summaries, detecting trends, computing statistics, or creating visualizations.
+
+## Instructions
+
+1. Read the input CSV file to understand its structure
+2. Run the analysis script: `python scripts/analyze.py --input <file> --output <dir>`
+3. Review the generated report in the output directory
+4. Present findings to the user with key insights highlighted
+
+## Available operations
+
+- Column statistics (mean, median, mode, std dev)
+- Trend detection across time-series columns
+- Correlation analysis between numeric columns
+- Chart generation (bar, line, scatter)
+```
+
+### Example: Prompt Skill (Tone of Voice)
+
+```yaml
+---
+name: professional-tone
+description: Write content in a professional corporate tone. Use when squad agents need to produce business communications, reports, or client-facing materials that require formal, polished language.
+type: prompt
+version: 1.0.0
+categories:
+  - writing
+  - tone
+---
+
+# Professional Tone Skill
+
+## When to use
+
+Apply this skill whenever generating content that will be seen by clients, executives, or external stakeholders. This includes emails, reports, proposals, and presentation scripts.
+
+## Instructions
+
+When writing in professional tone:
+
+1. Use formal but accessible language — avoid jargon unless the audience expects it
+2. Lead with the conclusion or recommendation, then support with evidence
+3. Keep sentences concise (aim for 15-20 words average)
+4. Use active voice by default; passive voice only when the actor is unknown or irrelevant
+5. Structure content with clear headings and bullet points for scannability
+
+## Examples
+
+**Input:** "Hey, just wanted to let you know the project is running behind. We messed up the timeline."
+
+**Output:** "I'm writing to provide an update on the project timeline. Due to unforeseen complexity in the integration phase, we are currently tracking two weeks behind our original schedule. I've prepared a revised timeline and mitigation plan for your review."
+
+## Avoid
+
+- Casual language ("gonna", "kinda", "hey")
+- Hedging without substance ("I think maybe we could possibly...")
+- Exclamation marks in formal contexts
+- Emojis in any client-facing content
+```
+
+### Example: Hybrid Skill (Social Media Publisher)
+
+```yaml
+---
+name: social-publisher
+description: Draft social media content and publish via API. Use when squads need to create and post content to Instagram, Twitter/X, or LinkedIn with approval workflows.
+type: hybrid
+version: 1.0.0
+categories:
+  - social-media
+  - publishing
+env:
+  - SOCIAL_API_KEY
+mcp:
+  server_name: social-api
+  command: npx
+  args: ["-y", "@company/social-mcp"]
+  transport: stdio
+script:
+  path: scripts/format_post.py
+  runtime: python
+  dependencies:
+    - pillow
+  invoke: python scripts/format_post.py --platform {platform} --input {draft_file}
+---
+
+# Social Media Publisher
+
+## When to use
+
+Use this skill when the pipeline needs to both create and publish social media content. The script handles formatting and image processing, while the MCP server handles the actual publishing API calls.
+
+## Instructions
+
+1. Draft the content using squad agent context (topic, tone, audience)
+2. Run the formatting script to prepare platform-specific versions
+3. Present drafts for user approval at the checkpoint
+4. After approval, use the MCP server to publish to the target platforms
+5. Log the published URLs in the output directory
+```