@litfamily/lithermes 1.0.0

package/assets/lithermes-plugin/skills/autoconference/references/results-logging.md

@@ -0,0 +1,165 @@
+# Results Logging Protocol
+
+> **LitHermes child boundary:** This is inert Hermes reference data. A conference child has `delegation_allowed=false`, returns a bounded packet only, and never writes shared or project state; only the root chair writes approved logs.
+
+Structured logging for machine-readable experiment tracking at two levels: per-researcher (matching autoresearch format) and conference-level (adding researcher identity, round context, and peer review verdicts).
+
+---
+
+## Section 1 — Per-Researcher TSV (`researcher_{ID}_results.tsv`)
+
+One file per researcher (e.g., `researcher_A_results.tsv`, `researcher_B_results.tsv`). Same schema as autoresearch's `autoresearch-results.tsv` for compatibility.
+
+### Schema
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `iteration` | int | 0-indexed iteration number within the current round |
+| `metric_value` | float | Measured metric value (or self-assessment score 1-10 in qualitative mode) |
+| `delta` | float or `-` | Change from baseline (iteration 0 of this researcher) |
+| `delta_pct` | string | Percentage change from baseline |
+| `status` | enum | `baseline`, `kept`, `reverted` |
+| `description` | string | One-line description of the change |
+| `evaluator_source` | string | Source of the evaluation — e.g., `script:evaluate.py`, `llm:reviewer`, `self-assessment` |
+| `timestamp` | ISO 8601 | When the experiment completed |
+
+### Example
+
+```tsv
+iteration	metric_value	delta	delta_pct	status	description	evaluator_source	timestamp
+0	2.3991	-	-	baseline	Recursive quicksort with list comprehensions	script:benchmark.py	2026-03-18T10:00:00Z
+1	1.8845	-0.5146	-21.4%	kept	Bottom-up iterative merge sort	script:benchmark.py	2026-03-18T10:05:00Z
+2	1.7265	-0.6726	-28.0%	kept	Merge sort + insertion sort for subarrays < 32	script:benchmark.py	2026-03-18T10:10:00Z
+3	1.9504	-0.4487	-18.7%	reverted	Natural merge sort with run detection	script:benchmark.py	2026-03-18T10:15:00Z
+4	0.9817	-1.4174	-59.1%	kept	LSD radix sort base 256	script:benchmark.py	2026-03-18T10:20:00Z
+```
+
+### Notes
+
+- **Qualitative mode:** `metric_value` holds the researcher's self-assessed quality score (1-10). The Reviewer's authoritative scores appear in the conference-level TSV.
+- **Iteration numbering:** Resets to 0 at the start of each round for simplicity. The round context is captured in the conference-level TSV.
+- **Baseline row:** The first row of each round uses `status: baseline` and `delta: -`.
+
+---
+
+## Section 2 — Conference-Level TSV (`conference_results.tsv`)
+
+One file for the entire conference. Adds researcher identity, round context, and peer review verdicts on top of the per-researcher schema.
+
+### Schema
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `round` | int | Conference round number (1-indexed) |
+| `researcher` | string | Researcher identifier (`A`, `B`, `C`, ...) |
+| `iteration` | int | Iteration within the round (0-indexed) |
+| `metric_value` | float | Measured metric value |
+| `delta` | float or `-` | Change from round baseline |
+| `delta_pct` | string | Percentage change from round baseline |
+| `status` | enum | `baseline`, `kept`, `reverted`, `failed` |
+| `description` | string | One-line description of the change |
+| `evaluator_source` | string | Source of the evaluation — e.g., `script:evaluate.py`, `llm:reviewer`, `self-assessment` |
+| `peer_review_verdict` | enum | `validated`, `challenged`, `overturned`, `-` |
+| `timestamp` | ISO 8601 | When the experiment completed |
+
+### Example
+
+```tsv
+round	researcher	iteration	metric_value	delta	delta_pct	status	description	evaluator_source	peer_review_verdict	timestamp
+1	A	0	2.3991	-	-	baseline	Recursive quicksort baseline	script:benchmark.py	-	2026-03-18T10:00:00Z
+1	A	1	1.8845	-0.5146	-21.4%	kept	Bottom-up iterative merge sort	script:benchmark.py	validated	2026-03-18T10:05:00Z
+1	A	2	1.7265	-0.6726	-28.0%	kept	Merge sort + insertion sort < 32	script:benchmark.py	validated	2026-03-18T10:10:00Z
+1	B	0	2.4012	-	-	baseline	Recursive quicksort baseline	script:benchmark.py	-	2026-03-18T10:00:00Z
+1	B	1	2.1500	-0.2512	-10.5%	kept	Heap sort implementation	script:benchmark.py	challenged	2026-03-18T10:07:00Z
+1	B	2	1.9800	-0.4212	-17.5%	kept	Heap sort with Floyd's algorithm	script:benchmark.py	validated	2026-03-18T10:12:00Z
+1	C	0	2.3950	-	-	baseline	Recursive quicksort baseline	script:benchmark.py	-	2026-03-18T10:00:00Z
+1	C	1	0.9817	-1.4133	-59.1%	kept	LSD radix sort base 256	script:benchmark.py	validated	2026-03-18T10:08:00Z
+2	A	0	1.7265	-	-	baseline	Round 2 start from best known	script:benchmark.py	-	2026-03-18T10:30:00Z
+2	A	1	1.5100	-0.2165	-12.5%	kept	Merge sort + radix hybrid	script:benchmark.py	validated	2026-03-18T10:35:00Z
+```
+
+### Status Values
+
+| Value | Meaning |
+|-------|---------|
+| `baseline` | Starting point for this researcher in this round |
+| `kept` | Change improved the metric and was kept |
+| `reverted` | Change did not improve metric; reverted to previous best |
+| `failed` | Researcher crashed, timed out, or self-terminated before completing the iteration |
+
+### Peer Review Verdict Values
+
+| Value | Meaning |
+|-------|---------|
+| `validated` | Reviewer confirmed the claim holds |
+| `challenged` | Reviewer flagged the claim as questionable; needs more evidence |
+| `overturned` | Reviewer determined the claim is invalid |
+| `-` | Not yet reviewed (e.g., baseline row, or round not yet complete) |
+
+---
+
+## Section 3 — Event Log (`conference_events.jsonl`)
+
+An append-only JSONL file recording all significant conference events. Each line is a JSON object. External tools can tail this file for real-time monitoring.
+
+### Schema
+
+```json
+{
+  "event": "<event_type>",
+  "timestamp": "<ISO 8601>",
+  "payload": { ... }
+}
+```
+
+### Event Types
+
+| Event | When | Payload Fields |
+|-------|------|----------------|
+| `conference.started` | Conference Chair initializes | `researchers`, `mode`, `goal`, `config_summary` |
+| `round.started` | Each round begins | `round`, `researcher_states` |
+| `researcher.iteration` | Researcher completes an iteration | `researcher`, `round`, `iteration`, `metric_value`, `delta`, `status` |
+| `round.poster_session` | Poster session complete | `round`, `summary` |
+| `round.peer_review` | Peer review complete | `round`, `validated_count`, `challenged_count`, `overturned_count` |
+| `round.completed` | Round finishes | `round`, `best_metric`, `best_researcher`, `converged` |
+| `researcher.stuck` | Researcher hits stuck Level 2+ | `researcher`, `round`, `stuck_level` |
+| `conference.converged` | Convergence detected | `final_best_metric`, `round_count`, `reason` |
+| `conference.completed` | Synthesis done | `synthesis_path`, `final_report_path`, `total_iterations` |
+
+### Example
+
+```jsonl
+{"event":"conference.started","timestamp":"2026-03-18T10:00:00Z","payload":{"researchers":3,"mode":"metric","goal":"Optimize inference latency","config_summary":"metric=p95_latency_ms, direction=minimize, target=<50ms"}}
+{"event":"round.started","timestamp":"2026-03-18T10:00:05Z","payload":{"round":1,"researcher_states":["A:ready","B:ready","C:ready"]}}
+{"event":"researcher.iteration","timestamp":"2026-03-18T10:05:00Z","payload":{"researcher":"A","round":1,"iteration":1,"metric_value":1.8845,"delta":-0.5146,"status":"kept"}}
+{"event":"round.poster_session","timestamp":"2026-03-18T10:20:00Z","payload":{"round":1,"summary":"A: merge sort -28%, B: heap sort -17.5%, C: radix sort -59.1%"}}
+{"event":"round.peer_review","timestamp":"2026-03-18T10:25:00Z","payload":{"round":1,"validated_count":5,"challenged_count":1,"overturned_count":0}}
+{"event":"round.completed","timestamp":"2026-03-18T10:26:00Z","payload":{"round":1,"best_metric":0.9817,"best_researcher":"C","converged":false}}
+{"event":"conference.completed","timestamp":"2026-03-18T11:45:00Z","payload":{"synthesis_path":"synthesis.md","final_report_path":"final_report.md","total_iterations":45}}
+```
+
+---
+
+## Usage
+
+The TSV files and event log enable:
+
+- **Programmatic analysis:** Load `conference_results.tsv` into pandas; plot per-researcher convergence curves; compute cross-researcher strategy comparison.
+- **CI integration:** Parse the last row of `conference_results.tsv` to check if the target was met. Same pattern as autoresearch-skill.
+- **Cross-project comparison:** Standardized format across all autoconference runs.
+- **Bounded monitoring:** Prefer a supplied, finite, redacted event snapshot. If fresh observation is explicitly approved, the root runs one attended foreground probe with declared duration and line/byte caps, then stops it and records cleanup; children never open or maintain a live stream.
+- **Peer review accuracy analysis:** Compare `peer_review_verdict` against final synthesis outcomes to measure reviewer quality over time.
+
+## Relationship to Other Output Files
+
+| File | Purpose | Format | Audience |
+|------|---------|--------|----------|
+| `conference.md` | User configuration + Conference Log | Markdown | Humans |
+| `researcher_{ID}_log.md` | Detailed per-iteration reasoning | Markdown | Humans |
+| `researcher_{ID}_results.tsv` | Per-researcher machine-readable results | TSV | Scripts/CI |
+| `conference_results.tsv` | Conference-level results with verdicts | TSV | Scripts/CI |
+| `conference_events.jsonl` | Event stream for real-time monitoring | JSONL | Scripts/monitoring |
+| `poster_session_round_N.md` | Session Chair's round summary | Markdown | Humans |
+| `peer_review_round_N.md` | Reviewer's verdicts per round | Markdown | Humans |
+| `synthesis.md` | Synthesizer's unified result | Markdown | Humans |
+| `final_report.md` | Executive summary | Markdown | Humans |

package/assets/lithermes-plugin/skills/autoconference/references/visualization-guide.md

@@ -0,0 +1,99 @@
+# Visualization Guide
+
+> **LitHermes child boundary:** This is inert Hermes reference data. A conference child has `delegation_allowed=false`, returns a bounded packet only, and never writes shared or project state; only the root chair writes approved figures.
+
+Mandatory rules for all `results.png` figures generated by this family. Use the registered `lithermes:lit-scientific-visualization` standards.
+
+## Setup
+
+Every `visualize.py` MUST start with:
+
+```python
+import sys, os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'scripts'))
+from style_presets import rcparams
+rcparams()  # MUST be called before plt.subplots()
+```
+
+This loads `scripts/style_presets.py` which sets Pretendard/Arial fonts, inward ticks, Okabe-Ito colors, white background, and DPI 600.
+
+## 7 Mandatory Rules
+
+### 1. Axis endpoints must show numbers
+First and last tick on both axes must be visible. Use `ax.set_xlim()` and `ax.set_ylim()` explicitly.
+
+### 2. Subplots must maintain spacing
+Use `fig.tight_layout(pad=1.5)` or `constrained_layout=True`. No overlapping labels.
+
+### 3. Legend must never obscure data
+Place legend where it doesn't cover data points. Use `loc='upper right'` or `bbox_to_anchor` if needed.
+
+### 4. No titles
+Journal figures use captions, not titles. Do NOT call `ax.set_title()` or `fig.suptitle()`.
+
+### 5. Legend text color must match data color
+Each legend entry's text color matches its data series color:
+```python
+leg = ax.legend()
+colors = ['#0072B2', '#D55E00', '#009E73']
+for i, text in enumerate(leg.get_texts()):
+    text.set_color(colors[i])
+```
+
+### 6. Legend border: black, framealpha=1
+```python
+ax.legend(frameon=True, edgecolor='black', framealpha=1)
+```
+
+### 7. DPI > 500
+Save at 600 DPI. Export both PNG and PDF:
+```python
+fig.savefig('results.png', dpi=600, bbox_inches='tight')
+fig.savefig('results.pdf', dpi=600, bbox_inches='tight')
+```
+
+## Figure Structure for Autoresearch
+
+Two-panel layout with panel labels:
+
+```python
+fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
+
+# Panel labels
+ax1.text(-0.08, 1.08, 'A', transform=ax1.transAxes, fontsize=16, fontweight='bold', va='top')
+ax2.text(-0.08, 1.08, 'B', transform=ax2.transAxes, fontsize=16, fontweight='bold', va='top')
+```
+
+**Panel A — Metric Convergence:**
+- Scatter: kept iterations (blue `#0072B2`) vs reverted (vermillion `#D55E00`)
+- Line: best-so-far envelope (green `#009E73`)
+- Dashed: target threshold (orange `#E69F00`)
+- Annotate baseline and best values
+
+**Panel B — Domain-specific result:**
+- Code optimization: bar chart of execution times
+- Function fitting: prediction curve vs data scatter
+- Literature review: category coverage
+- Skill elaboration: component breakdown
+
+## Okabe-Ito Palette Reference
+
+| Color | Hex | Use |
+|-------|-----|-----|
+| Orange | `#E69F00` | Target line, warnings |
+| Sky Blue | `#56B4E9` | Train data, secondary |
+| Bluish Green | `#009E73` | Best-so-far, success |
+| Yellow | `#F0E442` | Highlights |
+| Blue | `#0072B2` | Kept iterations, primary |
+| Vermillion | `#D55E00` | Reverted, errors |
+| Reddish Purple | `#CC79A7` | Prediction curves |
+| Black | `#000000` | Baselines, text |
+
+## Conference-Specific Panels
+
+For autoconference visualizations, use multi-panel layouts:
+
+**Panel A:** Per-researcher metric convergence (one line per researcher, different colors)
+**Panel B:** Best-of-conference envelope (across all researchers)
+**Panel C:** Peer review verdict distribution (validated/challenged/overturned per round)
+**Panel D (optional):** Researcher strategy heatmap or cross-pollination graph

package/assets/lithermes-plugin/skills/autoconference/scripts/check_conference.sh

@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+# check_conference.sh - Lightweight progress monitor for autoconference runs
+# Usage: check_conference.sh [conference_dir]
+
+CONF_DIR="${1:-.}"
+CONF_DIR="${CONF_DIR%/}"  # strip trailing slash
+
+CONF_MD="$CONF_DIR/conference.md"
+RESULTS_TSV="$CONF_DIR/conference_results.tsv"
+EVENTS_JSONL="$CONF_DIR/conference_events.jsonl"
+SYNTHESIS_MD="$CONF_DIR/synthesis.md"
+
+# ── Parse conference.md ───────────────────────────────────────────────────────
+
+if [[ -f "$CONF_MD" ]]; then
+    # First non-empty line after "## Goal"
+    goal=$(awk '/^## Goal/{found=1; next} found && /[^[:space:]]/{print; exit}' \
+           "$CONF_MD")
+    # max_rounds: line matching "max_rounds", "Rounds:", or scaffolded "Max rounds:"
+    max_rounds=$(grep -m1 -iE 'max_rounds|Rounds:' "$CONF_MD" \
+                  | grep -oE '[0-9]+' \
+                  | head -1)
+    # Researcher count: prefer scaffolded "Count", fall back to named researcher entries
+    researcher_count=$(grep -m1 -iE '\*\*Count:\*\*|^count:' "$CONF_MD" \
+                       | grep -oE '[0-9]+' \
+                       | head -1)
+    if [[ -z "$researcher_count" ]]; then
+        researcher_count=$(grep -cE '^\s*-\s+\*\*Name\*\*:|^\s*-\s+name:' "$CONF_MD" 2>/dev/null) || researcher_count=0
+    fi
+    # Direction
+    direction=$(grep -m1 -i '\*\*Direction:\*\*' "$CONF_MD" \
+                | sed 's/.*\*\*Direction:\*\*[[:space:]]*//' \
+                | sed 's/[[:space:]]*$//' \
+                | tr '[:upper:]' '[:lower:]')
+    # Target
+    target=$(grep -m1 -i '\*\*Target:\*\*' "$CONF_MD" \
+              | sed 's/.*\*\*Target:\*\*[[:space:]]*//' \
+              | sed 's/[[:space:]]*$//')
+else
+    goal="?"; max_rounds="?"; researcher_count="?"; direction="maximize"; target="?"
+fi
+
+[[ -z "$goal"              ]] && goal="?"
+[[ -z "$max_rounds"        ]] && max_rounds="?"
+[[ -z "$researcher_count"  ]] && researcher_count="?"
+[[ -z "$direction"          ]] && direction="maximize"
+[[ -z "$target"            ]] && target="?"
+
+# ── Parse conference_results.tsv ──────────────────────────────────────────────
+
+best_metric="?"; best_researcher="?"
+
+if [[ -f "$RESULTS_TSV" ]]; then
+    # Expect conference-level columns:
+    # round, researcher, iteration, metric_value, delta, delta_pct, status, description,
+    # evaluator_source, peer_review_verdict, timestamp
+    # Track best metric value and which researcher achieved it
+    best_val=""
+    while IFS=$'\t' read -r f_round f_researcher f_iteration f_metric f_rest; do
+        [[ -z "$f_researcher" ]] && continue
+        val="$f_metric"
+        if [[ "$val" =~ ^-?[0-9]+(\.[0-9]+)?$ ]]; then
+            if [[ -z "$best_val" ]]; then
+                best_val="$val"
+                best_researcher="$f_researcher"
+            else
+                if [[ "$direction" == "minimize" ]]; then
+                    better=$(awk -v a="$val" -v b="$best_val" 'BEGIN{print (a<b)?1:0}')
+                else
+                    better=$(awk -v a="$val" -v b="$best_val" 'BEGIN{print (a>b)?1:0}')
+                fi
+                if [[ "$better" == "1" ]]; then
+                    best_val="$val"
+                    best_researcher="$f_researcher"
+                fi
+            fi
+        fi
+    done < <(tail -n +2 "$RESULTS_TSV" | grep -v '^[[:space:]]*$')
+
+    if [[ -n "$best_val" ]]; then
+        best_metric="$best_val"
+    fi
+    [[ -z "$best_researcher" ]] && best_researcher="?"
+fi
+
+# ── Parse conference_events.jsonl ─────────────────────────────────────────────
+
+current_round="0"; last_event_type="?"; last_event_ts="?"; stuck_count="0"
+
+if [[ -f "$EVENTS_JSONL" ]]; then
+    # Count current and legacy round completion event names.
+    completed_rounds=$(grep -Ec '"round\.completed"|"round\.[0-9]+\.completed"' "$EVENTS_JSONL" 2>/dev/null) || completed_rounds=0
+    current_round="$completed_rounds"
+
+    # Count "researcher.stuck" events
+    stuck_count=$(grep -c '"researcher\.stuck"' "$EVENTS_JSONL" 2>/dev/null) || stuck_count=0
+
+    # Last event: extract type and timestamp from the last non-empty line
+    last_line=$(grep -v '^[[:space:]]*$' "$EVENTS_JSONL" | tail -1)
+    if [[ -n "$last_line" ]]; then
+        # Extract event type — value of "event" or "type" key
+        last_event_type=$(printf '%s' "$last_line" \
+                          | grep -oE '"(event|type)"\s*:\s*"[^"]+"' \
+                          | head -1 \
+                          | grep -oE '"[^"]+"\s*$' \
+                          | tr -d '"' \
+                          | sed 's/^[[:space:]]*//')
+        # Extract timestamp — value of "ts", "timestamp", or "time" key
+        last_event_ts=$(printf '%s' "$last_line" \
+                        | grep -oE '"(ts|timestamp|time)"\s*:\s*"[^"]+"' \
+                        | head -1 \
+                        | grep -oE '"[^"]+"\s*$' \
+                        | tr -d '"' \
+                        | sed 's/^[[:space:]]*//')
+    fi
+fi
+
+[[ -z "$last_event_type" ]] && last_event_type="?"
+[[ -z "$last_event_ts"   ]] && last_event_ts=""
+
+# ── Determine run status ──────────────────────────────────────────────────────
+
+if [[ -f "$SYNTHESIS_MD" ]]; then
+    run_status="COMPLETE"
+elif [[ -f "$EVENTS_JSONL" ]] && [[ "$current_round" -gt 0 ]]; then
+    run_status="running"
+else
+    run_status="no data"
+fi
+
+# ── Build display strings ─────────────────────────────────────────────────────
+
+round_display="${current_round} / ${max_rounds}"
+
+# Format last event with timestamp if available
+if [[ -n "$last_event_ts" ]]; then
+    last_event_display="${last_event_type} @ ${last_event_ts}"
+else
+    last_event_display="$last_event_type"
+fi
+
+# Add stuck researcher info if any
+if [[ "$stuck_count" -gt 0 ]]; then
+    researcher_display="${researcher_count} (${stuck_count} stuck)"
+else
+    researcher_display="$researcher_count"
+fi
+
+# Truncate long strings to keep box tidy (max ~44 chars for value column)
+_trunc() {
+    local s="$1" max="${2:-44}"
+    if [[ ${#s} -gt $max ]]; then
+        printf '%s' "${s:0:$((max-1))}…"
+    else
+        printf '%s' "$s"
+    fi
+}
+
+val_w=44
+
+conf_val=$(_trunc "${CONF_DIR}/" $val_w)
+goal_val=$(_trunc "$goal" $val_w)
+round_val=$(_trunc "$round_display" $val_w)
+researcher_val=$(_trunc "$researcher_display" $val_w)
+metric_val=$(_trunc "${best_metric} (${best_researcher})" $val_w)
+target_val=$(_trunc "$target" $val_w)
+status_val=$(_trunc "$run_status" $val_w)
+last_val=$(_trunc "$last_event_display" $val_w)
+
+# ── Box drawing ───────────────────────────────────────────────────────────────
+
+inner_w=54  # fixed inner width for clean alignment
+border=$(printf '─%.0s' $(seq 1 $inner_w))
+
+_pad_row() {
+    local label="$1" value="$2"
+    local content="  ${label}${value}"
+    local pad=$(( inner_w - ${#content} ))
+    if [[ $pad -lt 0 ]]; then
+        local max_val=$(( inner_w - ${#label} - 2 ))
+        value="${value:0:$((max_val-1))}…"
+        content="  ${label}${value}"
+        pad=0
+    fi
+    printf '│%s%*s│\n' "$content" "$pad" ""
+}
+
+printf '┌─ autoconference progress ─%s┐\n' "$(printf '─%.0s' $(seq 1 $((inner_w - 28))))"
+_pad_row "Conference:   " "$conf_val"
+_pad_row "Goal:         " "$goal_val"
+_pad_row "Round:        " "$round_val"
+_pad_row "Researchers:  " "$researcher_val"
+_pad_row "Best metric:  " "$metric_val"
+_pad_row "Target:       " "$target_val"
+_pad_row "Status:       " "$status_val"
+_pad_row "Last event:   " "$last_val"
+printf '└%s┘\n' "$border"