@r1z4x/tezgah 1.0.0

package/skills/ai-research/0-autoresearch-skill/references/progress-reporting.md

@@ -0,0 +1,166 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 0-autoresearch-skill/references/progress-reporting.md; MIT (c) Orchestra Research -->
+# Progress Reporting: Research Presentations
+
+When the research produces something worth sharing, create a compelling presentation — not a status dump, but a research story with visuals.
+
+## When to Report
+
+You decide when progress is meaningful enough to report. Consider reporting:
+
+- After an outer loop reflection that identified a significant pattern
+- When the optimization trajectory shows clear, sustained improvement
+- After a pivot — explain why the direction changed
+- Before requesting human input on a major decision
+- When concluding the research, before paper writing
+
+Maximum frequency: once per /loop tick or heartbeat cycle. Minimum: whenever you have something a human would find interesting.
+
+## What Makes a Good Research Presentation
+
+A good progress report reads like a research talk, not a database query. It should:
+
+1. **Tell a story**: why we started, what we tried, what we found, what it means
+2. **Show, don't just tell**: include plots, tables, comparisons — not just text
+3. **Be selective**: highlight the interesting findings, don't exhaustively list every experiment
+4. **End with direction**: what happens next and why
+
+## Recommended Sections
+
+Adapt these to what's compelling from your current research. Skip sections that aren't relevant. Add sections the research demands.
+
+### 1. Research Question and Motivation
+- What are we investigating and why does it matter?
+- One paragraph, accessible to someone unfamiliar with the project
+
+### 2. Approach
+- What's our method? What are we optimizing?
+- The two-loop architecture in one sentence
+
+### 3. Optimization Trajectory (The Karpathy Plot)
+- X-axis: experiment number or wall-clock time
+- Y-axis: proxy metric value
+- Show baseline as a horizontal line
+- Annotate significant jumps with what change caused them
+- This is often the most compelling visual — include it whenever possible
+
+### 4. Key Findings
+- The 2-3 most significant results with supporting evidence
+- Include plots, metric tables, comparison charts
+- Explain WHY results are significant, not just WHAT they are
+
+### 5. What We Tried (Decision Map)
+- A selective view of the hypothesis tree
+- Focus on the reasoning: why each direction was chosen, what it taught us
+- Include both successes and informative failures
+
+### 6. Current Understanding
+- The findings.md narrative, but presented compellingly
+- What's our best explanation for the patterns we see?
+
+### 7. Next Steps
+- What experiments are planned and why
+- What questions remain open
+- Any decisions that need human input
+
+## The Optimization Trajectory Plot
+
+This is the signature visual of autoresearch — a chart showing metric improvement over experiments.
+
+Minimal implementation (SVG-based, no dependencies):
+
+```python
+def generate_trajectory_svg(trajectory_data, width=800, height=400):
+    """Generate an SVG optimization trajectory chart.
+
+    trajectory_data: list of {"run": int, "metric": float, "label": str}
+    """
+    if not trajectory_data:
+        return "<p>No experiments yet.</p>"
+
+    metrics = [d["metric"] for d in trajectory_data]
+    min_m, max_m = min(metrics), max(metrics)
+    margin = (max_m - min_m) * 0.1 or 0.1
+    y_min, y_max = min_m - margin, max_m + margin
+
+    padding = 60
+    plot_w = width - 2 * padding
+    plot_h = height - 2 * padding
+    n = len(trajectory_data)
+
+    def x_pos(i):
+        return padding + (i / max(n - 1, 1)) * plot_w
+
+    def y_pos(v):
+        return padding + plot_h - ((v - y_min) / (y_max - y_min)) * plot_h
+
+    # Build SVG
+    svg = f'<svg width="{width}" height="{height}" xmlns="http://www.w3.org/2000/svg">'
+    svg += f'<rect width="{width}" height="{height}" fill="#1a1a2e" rx="8"/>'
+
+    # Grid lines
+    for i in range(5):
+        y = padding + i * plot_h / 4
+        val = y_max - i * (y_max - y_min) / 4
+        svg += f'<line x1="{padding}" y1="{y}" x2="{width-padding}" y2="{y}" stroke="#333" stroke-dasharray="4"/>'
+        svg += f'<text x="{padding-8}" y="{y+4}" fill="#888" text-anchor="end" font-size="11">{val:.3f}</text>'
+
+    # Baseline line
+    baseline = trajectory_data[0]["metric"]
+    by = y_pos(baseline)
+    svg += f'<line x1="{padding}" y1="{by}" x2="{width-padding}" y2="{by}" stroke="#ff6b6b" stroke-dasharray="6" opacity="0.7"/>'
+    svg += f'<text x="{width-padding+5}" y="{by+4}" fill="#ff6b6b" font-size="10">baseline</text>'
+
+    # Data line
+    points = " ".join(f"{x_pos(i)},{y_pos(d['metric'])}" for i, d in enumerate(trajectory_data))
+    svg += f'<polyline points="{points}" fill="none" stroke="#4ecdc4" stroke-width="2"/>'
+
+    # Data points
+    for i, d in enumerate(trajectory_data):
+        cx, cy = x_pos(i), y_pos(d["metric"])
+        svg += f'<circle cx="{cx}" cy="{cy}" r="4" fill="#4ecdc4"/>'
+
+    # Title
+    svg += f'<text x="{width/2}" y="24" fill="#eee" text-anchor="middle" font-size="14" font-weight="bold">Optimization Trajectory</text>'
+    svg += f'<text x="{width/2}" y="{height-10}" fill="#888" text-anchor="middle" font-size="11">Experiment Run</text>'
+    svg += '</svg>'
+    return svg
+```
+
+Embed the SVG output directly in the HTML report. Annotate significant jumps with brief labels.
+
+## HTML Presentation Template
+
+Use [templates/progress-presentation.html](../templates/progress-presentation.html) as a starting point. It provides:
+
+- Clean, dark-themed styling suitable for research presentations
+- Responsive layout
+- Section scaffolding matching the recommended structure
+- Placeholder for the trajectory chart
+
+Replace placeholder content with your actual research data. Add, remove, or rearrange sections as the research demands. The template is a scaffold, not a constraint.
+
+### Claude Code
+
+Generate the HTML, then show it to the human:
+
+```bash
+open to_human/progress-001.html
+```
+
+### OpenClaw
+
+Generate a PDF version. Options:
+- Use Python `weasyprint` to convert HTML to PDF
+- Use `matplotlib` to generate plots directly as PDF
+- Create a simple markdown → PDF pipeline
+
+Note the PDF path in HEARTBEAT.md so the human knows to look at it.
+
+## Presentation Quality Tips
+
+- **One insight per section** — don't overload
+- **Label axes and units** on all plots
+- **Use color consistently** — one color for improvements, another for baselines
+- **Include confidence intervals** or error bars where meaningful
+- **Show the trajectory early** — it's the hook that tells the reader "this is working"
+- **End with a clear next step** — the human should know what happens next without asking

package/skills/ai-research/0-autoresearch-skill/references/skill-routing.md

@@ -0,0 +1,219 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 0-autoresearch-skill/references/skill-routing.md; MIT (c) Orchestra Research -->
+# Skill Routing: When to Use Which Domain Skill
+
+The autoresearch skill orchestrates — domain skills execute. This reference maps research activities to the skills library.
+
+## Routing Principle
+
+When you encounter a domain-specific task during research, search the skills library for the right tool. Read the SKILL.md of the relevant skill before starting — it contains workflows, common issues, and production-ready code examples.
+
+## Complete Routing Map
+
+### Data and Preprocessing
+
+| Task | Skill | Location |
+|---|---|---|
+| Large-scale data processing | Ray Data | `05-data-processing/ray-data/` |
+| Data curation and filtering | NeMo Curator | `05-data-processing/nemo-curator/` |
+| Custom tokenizer training | HuggingFace Tokenizers | `02-tokenization/hf-tokenizers/` |
+| Subword tokenization | SentencePiece | `02-tokenization/sentencepiece/` |
+
+### Model Architecture and Training
+
+| Task | Skill | Location |
+|---|---|---|
+| Large-scale pretraining | Megatron-Core | `01-model-architecture/megatron-core/` |
+| Lightweight LLM training | LitGPT | `01-model-architecture/litgpt/` |
+| State-space models | Mamba | `01-model-architecture/mamba/` |
+| Linear attention models | RWKV | `01-model-architecture/rwkv/` |
+| Small-scale pretraining | NanoGPT | `01-model-architecture/nanogpt/` |
+
+### Fine-tuning
+
+| Task | Skill | Location |
+|---|---|---|
+| Multi-method fine-tuning | Axolotl | `03-fine-tuning/axolotl/` |
+| Template-based fine-tuning | LLaMA-Factory | `03-fine-tuning/llama-factory/` |
+| Fast LoRA fine-tuning | Unsloth | `03-fine-tuning/unsloth/` |
+| PyTorch-native fine-tuning | Torchtune | `03-fine-tuning/torchtune/` |
+
+### Post-training (RL / Alignment)
+
+| Task | Skill | Location |
+|---|---|---|
+| PPO, DPO, SFT pipelines | TRL | `06-post-training/trl/` |
+| Group Relative Policy Optimization | GRPO | `06-post-training/grpo-rl-training/` |
+| Scalable RLHF | OpenRLHF | `06-post-training/openrlhf/` |
+| Reference-free alignment | SimPO | `06-post-training/simpo/` |
+
+### Interpretability
+
+| Task | Skill | Location |
+|---|---|---|
+| Transformer circuit analysis | TransformerLens | `04-mechanistic-interpretability/transformerlens/` |
+| Sparse autoencoder training | SAELens | `04-mechanistic-interpretability/saelens/` |
+| Intervention experiments | NNsight | `04-mechanistic-interpretability/nnsight/` |
+| Causal tracing | Pyvene | `04-mechanistic-interpretability/pyvene/` |
+
+### Distributed Training
+
+| Task | Skill | Location |
+|---|---|---|
+| ZeRO optimization | DeepSpeed | `08-distributed-training/deepspeed/` |
+| Fully sharded data parallel | FSDP | `08-distributed-training/fsdp/` |
+| Multi-GPU abstraction | Accelerate | `08-distributed-training/accelerate/` |
+| Training framework | PyTorch Lightning | `08-distributed-training/pytorch-lightning/` |
+| Distributed data + training | Ray Train | `08-distributed-training/ray-train/` |
+
+### Evaluation
+
+| Task | Skill | Location |
+|---|---|---|
+| Standard LLM benchmarks | lm-evaluation-harness | `11-evaluation/lm-eval-harness/` |
+| NeMo-integrated evaluation | NeMo Evaluator | `11-evaluation/nemo-evaluator/` |
+| Custom eval tasks | Inspect AI | `11-evaluation/inspect-ai/` |
+
+### Inference and Serving
+
+| Task | Skill | Location |
+|---|---|---|
+| High-throughput serving | vLLM | `12-inference-serving/vllm/` |
+| NVIDIA-optimized inference | TensorRT-LLM | `12-inference-serving/tensorrt-llm/` |
+| CPU / edge inference | llama.cpp | `12-inference-serving/llama-cpp/` |
+| Structured generation serving | SGLang | `12-inference-serving/sglang/` |
+
+### Experiment Tracking
+
+| Task | Skill | Location |
+|---|---|---|
+| Full experiment tracking | Weights & Biases | `13-mlops/wandb/` |
+| Open-source tracking | MLflow | `13-mlops/mlflow/` |
+| Training visualization | TensorBoard | `13-mlops/tensorboard/` |
+
+### Optimization Techniques
+
+| Task | Skill | Location |
+|---|---|---|
+| Efficient attention | Flash Attention | `10-optimization/flash-attention/` |
+| 4/8-bit quantization | bitsandbytes | `10-optimization/bitsandbytes/` |
+| GPTQ quantization | GPTQ | `10-optimization/gptq/` |
+| AWQ quantization | AWQ | `10-optimization/awq/` |
+| GGUF format (llama.cpp) | GGUF | `10-optimization/gguf/` |
+| PyTorch-native quantization | Quanto | `10-optimization/quanto/` |
+
+### Safety and Alignment
+
+| Task | Skill | Location |
+|---|---|---|
+| Constitutional AI training | Constitutional AI | `07-safety-alignment/constitutional-ai/` |
+| Content safety classification | LlamaGuard | `07-safety-alignment/llamaguard/` |
+| Guardrail pipelines | NeMo Guardrails | `07-safety-alignment/nemo-guardrails/` |
+| Prompt injection detection | Prompt Guard | `07-safety-alignment/prompt-guard/` |
+
+### Infrastructure
+
+| Task | Skill | Location |
+|---|---|---|
+| Serverless GPU compute | Modal | `09-infrastructure/modal/` |
+| Multi-cloud orchestration | SkyPilot | `09-infrastructure/skypilot/` |
+| GPU cloud instances | Lambda Labs | `09-infrastructure/lambda-labs/` |
+
+### Agents and RAG
+
+| Task | Skill | Location |
+|---|---|---|
+| Agent pipelines | LangChain | `14-agents/langchain/` |
+| Knowledge retrieval agents | LlamaIndex | `14-agents/llamaindex/` |
+| Lightweight agents | Smolagents | `14-agents/smolagents/` |
+| Claude-based agents | Claude Agent SDK | `14-agents/claude-agent-sdk/` |
+| Vector store (local) | Chroma | `15-rag/chroma/` |
+| Vector similarity search | FAISS | `15-rag/faiss/` |
+| Text embeddings | Sentence Transformers | `15-rag/sentence-transformers/` |
+| Managed vector DB | Pinecone | `15-rag/pinecone/` |
+| Scalable vector DB | Milvus | `15-rag/milvus/` |
+
+### Prompt Engineering and Structured Output
+
+| Task | Skill | Location |
+|---|---|---|
+| Prompt optimization | DSPy | `16-prompt-engineering/dspy/` |
+| Structured LLM output | Instructor | `16-prompt-engineering/instructor/` |
+| Constrained generation | Guidance | `16-prompt-engineering/guidance/` |
+| Grammar-based generation | Outlines | `16-prompt-engineering/outlines/` |
+
+### Multimodal
+
+| Task | Skill | Location |
+|---|---|---|
+| Vision-language models | CLIP | `18-multimodal/clip/` |
+| Speech recognition | Whisper | `18-multimodal/whisper/` |
+| Visual instruction tuning | LLaVA | `18-multimodal/llava/` |
+| Vision-language (Qwen) | Qwen2-VL | `18-multimodal/qwen2-vl/` |
+| Vision-language (Mistral) | Pixtral | `18-multimodal/pixtral/` |
+| Visual understanding | Florence-2 | `18-multimodal/florence-2/` |
+| Document retrieval | ColPali | `18-multimodal/colpali/` |
+
+### Observability
+
+| Task | Skill | Location |
+|---|---|---|
+| LLM tracing and debugging | LangSmith | `17-observability/langsmith/` |
+| LLM observability platform | Phoenix | `17-observability/phoenix/` |
+
+### Emerging Techniques
+
+| Task | Skill | Location |
+|---|---|---|
+| Mixture of Experts training | MoE Training | `19-emerging-techniques/moe-training/` |
+| Combining trained models | Model Merging | `19-emerging-techniques/model-merging/` |
+| Extended context windows | Long Context | `19-emerging-techniques/long-context/` |
+| Faster inference via drafting | Speculative Decoding | `19-emerging-techniques/speculative-decoding/` |
+| Teacher-student compression | Knowledge Distillation | `19-emerging-techniques/knowledge-distillation/` |
+| Reducing model size | Model Pruning | `19-emerging-techniques/model-pruning/` |
+
+### Research Output
+
+| Task | Skill | Location |
+|---|---|---|
+| Generate research ideas | Research Ideation | `21-research-ideation/` |
+| Write publication-ready paper | ML Paper Writing | `20-ml-paper-writing/` |
+
+## Common Research Workflows
+
+### "I need to fine-tune a model and evaluate it"
+
+1. Pick fine-tuning skill based on needs (Unsloth for speed, Axolotl for flexibility)
+2. Use lm-evaluation-harness for standard benchmarks
+3. Track with W&B or MLflow
+
+### "I need to understand what the model learned"
+
+1. Use TransformerLens for circuit-level analysis
+2. Train SAEs with SAELens for feature-level understanding
+3. Run interventions with NNsight or Pyvene
+
+### "I need to do RL training"
+
+1. Start with TRL for standard PPO/DPO
+2. Use GRPO skill for DeepSeek-R1 style training
+3. Scale with OpenRLHF if needed
+
+### "I need to run experiments on cloud GPUs"
+
+1. Modal for quick serverless runs
+2. SkyPilot for multi-cloud optimization
+3. Lambda Labs for dedicated instances
+
+## Finding Skills
+
+If you're not sure which skill to use:
+
+```bash
+# Search by keyword in skill names
+ls */*/SKILL.md | head -20
+
+# Search skill descriptions for a keyword
+grep -l "keyword" */*/SKILL.md
+```
+
+Or search the repository's README.md which lists all skills with descriptions.

package/skills/ai-research/0-autoresearch-skill/templates/findings.md

@@ -0,0 +1,44 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 0-autoresearch-skill/templates/findings.md; MIT (c) Orchestra Research -->
+# Research Findings
+
+## Research Question
+
+<!-- What are we trying to discover? One clear sentence. -->
+
+## Current Understanding
+
+<!-- Updated after each outer loop cycle. What do we know so far?
+     What patterns explain our results? What's the mechanism?
+     This section should read like the core argument of a paper. -->
+
+## Key Results
+
+<!-- Significant experimental findings. Include metrics, comparisons, and
+     brief interpretation. Link to experiment directories for full details. -->
+
+## Patterns and Insights
+
+<!-- What emerges across multiple experiments? What types of changes
+     consistently work or fail? Why? -->
+
+## Lessons and Constraints
+
+<!-- Specific actionable learnings that should guide future experiments.
+     Things you tried that didn't work and WHY, so you don't repeat them.
+     Constraints you discovered about the problem space.
+
+     Examples:
+     - Weight decay > 0.1 causes training instability at 125M param scale
+     - SwiGLU and RoPE improvements stack because they're orthogonal (FFN vs positional)
+     - Baseline only reproduces published numbers with batch_size=64, not 32
+     - Sleep phases before memorization completion hurt — model needs memories to consolidate -->
+
+## Open Questions
+
+<!-- What remains unanswered? What would strengthen or challenge
+     our current understanding? -->
+
+## Optimization Trajectory
+
+<!-- Summary of inner loop progress. How has the metric evolved?
+     Note inflection points and what caused them. -->

package/skills/ai-research/0-autoresearch-skill/templates/research-log.md

@@ -0,0 +1,41 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 0-autoresearch-skill/templates/research-log.md; MIT (c) Orchestra Research -->
+# Research Log
+
+Chronological record of research decisions and actions. Append-only.
+
+| # | Date | Type | Summary |
+|---|------|------|---------|
+| | | | |
+
+<!-- Entry types:
+  bootstrap    — initial scoping, literature search, hypothesis formation
+  inner-loop   — experiment run and result
+  outer-loop   — synthesis, reflection, direction decision
+  pivot        — change in research direction
+  report       — progress presentation generated
+  conclude     — decision to finalize and write paper
+
+Example entries:
+| 1 | 2026-03-15 | bootstrap | Searched Semantic Scholar + arXiv for efficient transformer architectures. Found 8 relevant papers. Gap: no systematic comparison of GLU variants on small models. Formed 3 hypotheses. Baseline: NanoGPT 5-min run, val_loss=4.82. |
+| 2 | 2026-03-15 | inner-loop | H1 run_001: swapped ReLU for SwiGLU in FFN. 5-min training run. val_loss=4.61 (baseline 4.82, delta -0.21). Kept. |
+| 3 | 2026-03-15 | inner-loop | H1 run_002: increased FFN hidden dim from 4x to 5.3x to match SwiGLU param count. val_loss=4.58 (-0.03 vs run_001). Marginal — SwiGLU benefit mostly from gating, not extra params. |
+| 4 | 2026-03-15 | inner-loop | H1 run_003: tried GEGLU instead of SwiGLU. val_loss=4.63. Slightly worse than SwiGLU. SwiGLU wins for this scale. |
+| 5 | 2026-03-15 | inner-loop | H2 run_004: replaced learned positional embeddings with RoPE. val_loss=4.55 (-0.06 vs SwiGLU baseline). Promising — stacks with SwiGLU. |
+| 6 | 2026-03-15 | inner-loop | H2 run_005: RoPE + SwiGLU combined. val_loss=4.41 (-0.41 vs original baseline). Best so far. |
+| 7 | 2026-03-16 | outer-loop | Reviewed 5 runs. Pattern: gating mechanisms (SwiGLU) and rotary embeddings (RoPE) give independent gains that stack. Combined improvement ~9%. But WHY do they stack? Hypothesis: they operate on orthogonal aspects (FFN expressiveness vs positional encoding). Direction: DEEPEN — test if adding RMSNorm also stacks independently. |
+| 8 | 2026-03-16 | inner-loop | H3 run_006: replaced LayerNorm with RMSNorm. val_loss=4.39 (-0.02). Small gain. Stacks but diminishing returns on normalization. |
+| 9 | 2026-03-17 | outer-loop | 8 runs complete. Optimization plateau around val_loss=4.38. The easy architectural wins (SwiGLU, RoPE) are captured. Searched literature on training dynamics — found papers on warmup schedules at small scale. Direction: BROADEN — shift from architecture to training recipe. |
+| 10 | 2026-03-17 | report | Generated progress-001.html with trajectory plot showing 9% improvement from architectural changes. |
+
+Example entries (discovery-type research — understanding grokking):
+| 1 | 2026-03-20 | bootstrap | Searched literature on grokking and delayed generalization. Found Nanda et al. progress measures, Grokfast spectral filtering. Gap: no connection to memory consolidation theory from neuroscience. 3 hypotheses formed. |
+| 2 | 2026-03-20 | inner-loop | H1 run_001: trained modular addition transformer to memorization (100% train acc, 0% test). Steps to memorize: 1200. Baseline established. |
+| 3 | 2026-03-20 | inner-loop | H1 run_002: continued training with standard weight decay. Grokking at step 48000. Measured progress measure throughout — sharp transition at step 44000. |
+| 4 | 2026-03-20 | inner-loop | H1 run_003: inserted "sleep phase" at step 20000 (elevated weight decay + oscillatory LR for 500 steps). Grokking now at step 31000. 35% acceleration. |
+| 5 | 2026-03-20 | inner-loop | H1 run_004: sleep phase at step 10000. Grokking at step 27000. Earlier sleep = earlier grokking. |
+| 6 | 2026-03-20 | inner-loop | H1 run_005: sleep phase at step 5000 (before full memorization). Grokking at step 38000. Too early hurts — model hadn't memorized enough for consolidation to work. |
+| 7 | 2026-03-21 | outer-loop | Reviewed 5 runs. Clear pattern: sleep phases accelerate grokking but only AFTER memorization is complete. This matches memory consolidation theory exactly — you need memories formed before consolidation can reorganize them. Searched for neural slow-wave sleep literature. The weight decay + oscillatory LR during sleep phases mimics synaptic downscaling. Direction: DEEPEN — sweep sleep timing relative to memorization completion. |
+| 8 | 2026-03-21 | inner-loop | H1.1 run_006-010: swept sleep insertion at 80%, 100%, 120%, 150%, 200% of memorization step. Sweet spot at 110-120%. Consistent across 3 seeds. |
+| 9 | 2026-03-22 | outer-loop | 10 runs complete. The story is clear: neural networks "dream to learn" just like brains — consolidation after encoding, not during. Grokfast achieves similar acceleration through a different mechanism (gradient spectral filtering). Next: compare gradient spectra during our sleep phases vs Grokfast filtering to see if they converge on the same signal. Direction: BROADEN. |
+| 10 | 2026-03-22 | report | Generated progress-001.html with sleep timing vs grokking step plot. Key visual: sweet spot curve mirrors neuroscience memory consolidation window. |
+-->