researchloop 0.1.0

package/docs/getting-started.md

@@ -0,0 +1,283 @@
+# ResearchLoop Getting Started
+
+ResearchLoop is an open source npm package that helps an AI agent run a disciplined research loop inside a machine learning repo.
+
+The shortest way to think about it:
+
+- you install the CLI
+- you point it at a repo
+- it creates a durable `.researchloop/` workspace
+- your AI agent uses that workspace to plan, run, compare, and record experiments
+
+## 1. Install
+
+From your own machine:
+
+```bash
+npm install -g researchloop
+```
+
+For local development from this repo:
+
+```bash
+cd /Users/vukrosic/my-life/researchloop
+npm link
+researchloop --help
+```
+
+If you want to hand this to an AI agent, the simplest instruction is:
+
+```text
+Install ResearchLoop, initialize the repo, inspect the project, then use the generated prompt to start the research loop.
+```
+
+## 2. Initialize a repo
+
+Run this inside a blank folder or inside an existing ML repo:
+
+```bash
+researchloop init --agent codex
+```
+
+That creates:
+
+```text
+.researchloop/
+  AGENTS.md
+  goal.md
+  plan.md
+  repo-profile.json
+  scratchpad/
+    THREAD.md
+    runs.jsonl
+    ideas/
+    papers/
+    variants/
+    sweeps/
+```
+
+If you want the agent-specific file for another tool, use:
+
+```bash
+researchloop init --agent claude-code
+researchloop init --agent hermes
+researchloop init --agent cursor
+```
+
+## 3. Set the research goal
+
+Tell ResearchLoop what the agent should optimize:
+
+```bash
+researchloop goal "lower validation loss"
+```
+
+You can also add structure:
+
+```bash
+researchloop goal "lower validation loss" --metric val_loss --direction lower
+```
+
+That saves the objective into `.researchloop/goal.md`, which the agent and the prompt command can read later.
+
+## 4. Generate experiment ideas
+
+```bash
+researchloop idea --write
+```
+
+This prints a ranked list of small experiments for the current repo shape. For `llm-research-kit`, that usually means baseline checks, learning-rate sweeps, and tiny architecture changes. For a generic repo, it starts with finding the baseline and metric plumbing.
+
+## 5. Inspect the repo
+
+```bash
+researchloop inspect
+```
+
+This writes a repo profile into `.researchloop/repo-profile.json` and helps the agent understand:
+
+- possible training files
+- possible eval files
+- config files
+- log folders
+- likely adapters
+
+## 6. Generate the agent prompt
+
+```bash
+researchloop prompt --agent codex
+```
+
+Paste the output into your AI agent.
+
+You can also attach a focused playbook:
+
+```bash
+researchloop prompt --agent codex --focus hyperparameters
+researchloop prompt --agent codex --focus architecture
+researchloop prompt --agent codex --focus attention
+```
+
+That prompt tells the agent to:
+
+- read the `.researchloop/` files
+- establish a baseline
+- propose small experiments
+- record runs
+- compare results
+- keep the loop moving
+
+You can still pass `--goal` for a one-off override, but the normal flow is to save the goal once and let the prompt command read it back.
+
+If you want the prompt to narrow in on a family of experiments, use one of the built-in focus playbooks:
+
+- `hyperparameters`
+- `architecture`
+- `attention`
+
+## 7. Record and compare runs
+
+After a run finishes:
+
+```bash
+researchloop record --id first-run --status complete --metric val_loss=2.31 --note "first logged experiment"
+```
+
+To compare runs:
+
+```bash
+researchloop compare --metric val_loss --direction lower
+```
+
+For metrics where higher is better:
+
+```bash
+researchloop compare --metric accuracy --direction higher
+```
+
+Then summarize the current state:
+
+```bash
+researchloop report
+```
+
+## 8. Open the dashboard
+
+Serve a local dashboard for the current repo:
+
+```bash
+researchloop dashboard
+```
+
+Then open the localhost URL it prints. The dashboard reads the repo's `.researchloop/` files and shows:
+
+- the saved goal
+- the run ledger
+- the best run so far
+- the latest run
+- a small trend chart for the main metric
+
+It does not need accounts or auth because it stays on your machine.
+
+## 9. Test the setup before you trust it
+
+Run the local checks from this repo:
+
+```bash
+npm run smoke
+npm run test:compare
+npm run test:setup
+npm run test:prompts
+npm run test:site
+npm run smoke:e2e
+```
+
+These checks verify that:
+
+- the CLI starts
+- the setup flow works in a blank folder
+- `compare` ranks runs
+- prompt templates are clean
+- the website copy matches the product
+- the end-to-end flow works
+
+## 10. Use it in a real ML repo
+
+Once the basics work, move into a real project:
+
+```bash
+cd /path/to/your/ml-repo
+researchloop init --agent codex
+researchloop inspect
+researchloop prompt --agent codex --goal "improve validation loss"
+```
+
+Then give the prompt to your AI agent and let it run the loop.
+
+ResearchLoop is not trying to magically solve the model for you. It gives the agent the operating system for research: goals, baseline, logs, comparison, and continuation.
+
+## 11. Publish to npm
+
+The package is published to the public npm registry at [npmjs.com](https://www.npmjs.com/).
+
+Before publishing:
+
+```bash
+npm login
+npm whoami
+npm pack --dry-run
+```
+
+Make sure the package name is available and the contents look right.
+
+For this repo, the package name is currently `researchloop` in `package.json`. If that name is available in your npm account, publish with:
+
+```bash
+npm publish
+```
+
+If you later switch to a scoped package like `@yourname/researchloop`, publish with:
+
+```bash
+npm publish --access public
+```
+
+Common release flow:
+
+```bash
+npm version patch
+git push --follow-tags
+npm publish
+```
+
+Typical release checklist:
+
+1. run the local tests
+2. check `npm pack --dry-run`
+3. bump the version
+4. publish to npm
+5. update the website and README if the usage changed
+
+## 12. Where users install it from
+
+Users install it from the npm registry with:
+
+```bash
+npm install -g researchloop
+```
+
+If they prefer local use inside one repo:
+
+```bash
+npm install researchloop
+```
+
+Then they run the CLI from that environment.
+
+## 13. The one-line handoff to an AI agent
+
+If you want the shortest possible instruction for Codex, Claude Code, Hermes, or a similar agent, give it this:
+
+```text
+Use ResearchLoop: run init, inspect the repo, read .researchloop/AGENTS.md and goal.md, establish the baseline, then run small experiments, record results, compare runs, and keep the research loop moving.
+```

package/package.json

@@ -0,0 +1,37 @@
+{
+  "name": "researchloop",
+  "version": "0.1.0",
+  "description": "Install an autonomous AI research harness for Codex, Claude Code, Hermes, and other coding agents.",
+  "type": "module",
+  "bin": {
+    "researchloop": "./bin/researchloop.js"
+  },
+  "files": [
+    "bin",
+    "templates",
+    "README.md",
+    "docs/getting-started.md",
+    "CHANGELOG.md"
+  ],
+  "scripts": {
+    "smoke": "node ./bin/researchloop.js --help",
+    "smoke:e2e": "bash ./scripts/smoke-e2e.sh",
+    "test:goal": "bash ./scripts/test-goal.sh",
+    "test:idea": "bash ./scripts/test-idea.sh",
+    "test:dashboard": "bash ./scripts/test-dashboard.sh",
+    "test:setup": "bash ./scripts/test-setup.sh",
+    "test:compare": "bash ./scripts/test-compare.sh",
+    "test:prompts": "bash ./scripts/test-prompts.sh",
+    "test:focus-prompts": "bash ./scripts/test-focus-prompts.sh",
+    "test:site": "bash ./scripts/test-site.sh"
+  },
+  "keywords": [
+    "ai-research",
+    "codex",
+    "claude-code",
+    "agent",
+    "experiments",
+    "ablation"
+  ],
+  "license": "MIT"
+}

package/templates/adapters/generic.md

@@ -0,0 +1,18 @@
+# Generic Adapter
+
+Use this when the repository type is unknown.
+
+Inspection targets:
+- README and docs
+- package manager files
+- train, eval, benchmark, and test scripts
+- config files
+- logs and checkpoint folders
+
+Required fields:
+- baseline command
+- evaluation command
+- primary metric
+- target direction
+- allowed changes
+- forbidden changes

package/templates/adapters/huggingface.md

@@ -0,0 +1,15 @@
+# Hugging Face Trainer Adapter
+
+Useful files:
+- `training_args`
+- `Trainer`
+- `accelerate`
+- `config.json`
+- dataset loading scripts
+- metric callbacks
+
+Default first experiments:
+1. Confirm a tiny run works.
+2. Identify evaluation metric and save path.
+3. Sweep learning rate and schedule before model architecture.
+4. Preserve dataset and split unless the goal allows data changes.

package/templates/adapters/llm-research-kit.md

@@ -0,0 +1,20 @@
+# LLM Research Kit Adapter
+
+Useful files:
+- `train_llm.py`
+- `configs/llm_config.py`
+- `configs/dataset_config.py`
+- `training/trainer.py`
+- `training/evaluation.py`
+- `optimizers/`
+- `plots/metrics_*.json`
+
+MacBook mode:
+- Use MPS or CPU for smoke tests.
+- Keep `torch.compile` disabled outside CUDA.
+- Prefer tiny configs for local proof-of-life.
+
+CUDA mode:
+- Enable compile and mixed precision when supported.
+- Use repeated seeds before claiming wins.
+- Run pruning before promoting stacked changes.

package/templates/adapters/pytorch.md

@@ -0,0 +1,26 @@
+# PyTorch Adapter
+
+Useful files:
+- `train.py`
+- `eval.py`
+- `configs/`
+- `requirements.txt`
+- `pyproject.toml`
+- `checkpoints/`
+- `logs/`
+
+Default first experiments:
+1. Run a one-batch smoke test.
+2. Establish baseline metric parsing.
+3. Try one optimizer or schedule ablation.
+4. Reproduce any apparent win.
+
+Common knobs:
+- optimizer
+- learning rate
+- weight decay
+- schedule
+- warmup
+- precision
+- initialization
+- gradient clipping

package/templates/base/AGENTS.md

@@ -0,0 +1,47 @@
+# Research Loop Agent Rules
+
+You are an autonomous research engineer working in this repository.
+
+## Mission
+
+Improve the target metric through small, documented experiments.
+Read `.researchloop/goal.md`, `.researchloop/plan.md`, and `.researchloop/scratchpad/THREAD.md` before making changes.
+
+## Hard Rules
+
+1. Do not claim a result unless you ran the command or can point to an existing log.
+2. Establish or identify a baseline before optimizing.
+3. Keep each experiment small enough to isolate the causal change.
+4. Log every meaningful action to `.researchloop/scratchpad/THREAD.md`.
+5. Add every run to `.researchloop/scratchpad/runs.jsonl`.
+6. Write idea notes before coding non-trivial experiments.
+7. Define a kill criterion before launching a sweep.
+8. Reproduce promising wins before treating them as real.
+9. Run pruning or leave-one-out checks before promoting a stacked recipe.
+10. Preserve user work and avoid unrelated refactors.
+
+## Autonomy
+
+If the next step is clear, take it. If two paths are reasonable, choose one, log why, and continue.
+Do not stop only because the current idea failed. A failed idea should produce a note, a lesson, and the next candidate.
+
+## Scratchpad
+
+- `THREAD.md`: append-only chronological mission log.
+- `runs.jsonl`: structured run ledger.
+- `ideas/`: one file per idea, with mechanism, prior art, ablation plan, and kill criterion.
+- `papers/`: paper notes with exact recipe details and a "how to port this" section.
+- `variants/`: generated code/config variants.
+- `sweeps/`: grouped sweep notes and outputs.
+- `picklist.md`: prioritized candidates and ruled-out families.
+- `audits.md`: benchmark-rule checks, portability checks, and claim audits.
+
+## Experiment Loop
+
+1. Inspect the repo and find the baseline command.
+2. Define the allowed and forbidden change surfaces.
+3. Propose 3-7 ranked experiments.
+4. Run the cheapest useful experiment first.
+5. Parse and record metrics.
+6. Decide: reproduce, refine, prune, or pivot.
+7. Keep `plan.md` current and `THREAD.md` chronological.

package/templates/base/goal.md

@@ -0,0 +1,22 @@
+# Research Goal
+
+## Goal
+
+Describe the metric to improve and the constraints.
+
+Example:
+
+- Target metric: validation loss
+- Direction: lower is better
+- Baseline command: unknown
+- Evaluation command: unknown
+- Allowed changes: optimizer, schedules, initialization, hyperparameters
+- Forbidden changes: data, architecture, batch size, benchmark definition
+
+## Current Best
+
+Unknown.
+
+## Notes
+
+Use `researchloop inspect` to generate a repo profile, then ask an agent to fill in the missing benchmark details.

package/templates/base/plan.md

@@ -0,0 +1,22 @@
+# Research Plan
+
+This is mutable state. Keep it short and current.
+The chronological log belongs in `scratchpad/THREAD.md`.
+
+## Current State
+
+- Baseline: unknown
+- Best valid result: unknown
+- Active family: none
+- Running jobs: none
+- Next action: inspect repo and establish baseline
+
+## Picklist
+
+1. Establish baseline.
+2. Identify metric extraction.
+3. Run one smoke experiment.
+
+## Ruled Out
+
+None yet.

package/templates/base/scratchpad/THREAD.md

@@ -0,0 +1,13 @@
+# Research Thread
+
+Append one entry per meaningful event.
+
+Format:
+
+```text
+YYYY-MM-DD HH:MM - event
+- What changed:
+- Evidence:
+- Decision:
+- Next:
+```

package/templates/base/scratchpad/audits.md

@@ -0,0 +1,14 @@
+# Audits
+
+Use this file for benchmark-rule checks, portability checks, and claim audits.
+
+## Benchmark Rules
+
+- Baseline command: unknown
+- Evaluation command: unknown
+- Allowed changes: unknown
+- Forbidden changes: unknown
+
+## Claims
+
+No claims yet.

package/templates/base/scratchpad/ideas/.gitkeep

@@ -0,0 +1 @@
+

package/templates/base/scratchpad/papers/.gitkeep

@@ -0,0 +1 @@
+

package/templates/base/scratchpad/picklist.md

@@ -0,0 +1,15 @@
+# Picklist
+
+Promote ideas here only after they have a note, a minimal test plan, and a kill criterion.
+
+## Active
+
+- Establish baseline.
+
+## Backlog
+
+- Add candidates after repo inspection.
+
+## Ruled Out
+
+- None yet.

package/templates/base/scratchpad/runs.jsonl

@@ -0,0 +1 @@
+

package/templates/base/scratchpad/sweeps/.gitkeep

@@ -0,0 +1 @@
+

package/templates/base/scratchpad/variants/.gitkeep

@@ -0,0 +1 @@
+