academic-army 0.1.0

package/src/evolve-skill/README.md

@@ -0,0 +1,116 @@
+# Evolve Skill Pipeline
+
+[`src/evolve-skill`](.) implements the self-evolution loop used by the metaskill evolution scripts described in [`../../metaskills/README.md`](../../metaskills/README.md). It is for improving an existing skill by repeatedly testing it on a fixed task, evaluating the produced artifact, and applying targeted revisions.
+
+[中文说明](README.zh-CN.md)
+
+For the overall TypeScript pipeline usage and entry points, see [`src/README.md`](../README.md).
+
+For the user-facing optimization workflow, see [`../../metaskills/README.md`](../../metaskills/README.md).
+
+## Quick Start
+
+When a skill output is weak, add concrete tips to the matching metaskill file, then run that skill's evolution script from the repository root. See [`../../metaskills/README.md`](../../metaskills/README.md) for the prepared scripts and path mapping.
+
+```bash
+bash metaskills/academic-army-architect/envolve.sh
+```
+
+## Direct Command
+
+Run the pipeline directly from the repository root:
+
+```bash
+npm run evolve-skill -- \
+  --config agent-forge.yaml \
+  --skill-path skills/academic-army-architect \
+  --artifact-path output/evolve-academic-army-architect \
+  --metaskill-path metaskills/academic-army-architect/METASKILL.md \
+  --task-path metaskills/academic-army-architect/ENVOLVETASK.md
+```
+
+## Options Reference
+
+Required arguments:
+
+| Option | Description |
+|---|---|
+| `--skill-path` | The skill directory or file to revise. |
+| `--artifact-path` | The output folder cleared and reused by each runner round. |
+| `--metaskill-path` | The metaskill design document used by evaluator and modifier. |
+| `--task-path` | The fixed task used by the runner to test the skill. Repeat this option to run multiple fixed tasks per round. |
+
+Optional arguments:
+
+```text
+--rounds 5
+  Number of self-evolve rounds to run.
+```
+
+`--rounds` defaults to `3`.
+
+## Main Flow
+
+[`pipeline.ts`](pipeline.ts) parses:
+
+- `--skill-path`: the skill directory or file to revise.
+- `--artifact-path`: the output directory cleared and reused each round.
+- `--metaskill-path`: the design document and tips used to judge and revise the skill.
+- `--task-path`: one or more fixed tasks used to test the skill.
+- `--rounds`: the number of self-evolve rounds, defaulting to `3`.
+
+Each round does the following:
+
+1. Clear and recreate `--artifact-path`.
+2. For each configured `--task-path`, create a fresh `skill-runner` agent to run the target skill on that fixed task and write artifacts.
+3. Use `skill-evaluator` to evaluate the artifact against the current metaskill guidance file.
+4. Pass the evaluator review to `skill-modifier`, which revises the target skill using the same metaskill guidance.
+
+The fresh runner keeps each artifact independent from earlier runner context. The evaluator and modifier are invoked through the shared `AgentTeam` so their configured agent behavior is centralized in the pipeline config.
+
+## Loop Behavior
+
+The loop keeps skill development grounded in concrete output. Instead of rewriting a skill from vague impressions, it tests the skill on a stable task, evaluates the resulting artifact against the metaskill, and then asks Codex to revise the skill based on concrete feedback.
+
+The pipeline keeps two long-lived Codex threads through the shared team:
+
+1. `skill-evaluator`: reviews artifacts across rounds.
+2. `skill-modifier`: edits the target skill across rounds.
+
+Each round also creates a fresh one-time `skill-runner` thread. The runner has no memory from previous rounds, so earlier artifacts do not pollute the next run.
+
+The loop deliberately stays simple:
+
+1. A fresh runner thread runs the target skill for each configured task and writes artifacts to an output folder.
+2. A long-lived evaluator thread reviews the artifact using the metaskill.
+3. A long-lived modifier thread edits the skill according to the evaluator feedback.
+4. The next round starts with a fresh runner thread.
+
+This avoids LangGraph, state machines, registries, and defensive wrapper code. The important state lives in the long-lived evaluator/modifier Codex sessions, the current artifact folder, and the files being revised.
+
+## Inputs And Outputs
+
+| Item | Path source |
+|---|---|
+| Target skill | `--skill-path` |
+| Metaskill | `--metaskill-path` |
+| Fixed task | `--task-path` |
+| Generated artifact folder | `--artifact-path`, cleared and reused each round |
+
+## Important Files
+
+| Path | Purpose |
+|---|---|
+| [`pipeline.ts`](pipeline.ts) | Argument parsing and round orchestration. |
+| [`agents/factory.ts`](agents/factory.ts) | Registers `skill-runner`, `skill-evaluator`, and `skill-modifier`. |
+| [`agents/runner.ts`](agents/runner.ts) | Reads each fixed task file configured by `--task-path` and asks the target skill to write artifacts. |
+| [`agents/evaluator.ts`](agents/evaluator.ts) | Reads the metaskill file configured by `--metaskill-path` and critiques the produced artifact. |
+| [`agents/modifier.ts`](agents/modifier.ts) | Reads the metaskill file and the evaluator review, then revises the target skill. |
+
+## Troubleshooting
+
+| Problem | Likely cause | Fix |
+|---|---|---|
+| Artifacts disappear between rounds | `--artifact-path` is cleared and reused each runner round. | Use a dedicated `output/evolve-*` folder. |
+| The output still feels weak | The loop needs concrete metaskill guidance. | Add concrete tips to the matching metaskill file and run the script again. |
+| Runner context seems to influence results | The runner should be fresh each round. | Check the pipeline config and archive the generated artifacts for comparison. |

package/src/evolve-skill/README.zh-CN.md

@@ -0,0 +1,116 @@
+# Evolve Skill Pipeline
+
+[`src/evolve-skill`](.) 实现 [`../../metaskills/README.zh-CN.md`](../../metaskills/README.zh-CN.md) 中说明的 metaskill evolution scripts 所使用的 self-evolution loop。它用于优化已有 skill：反复在固定任务上测试该 skill，评价产出的 artifact，并根据评价进行有针对性的修改。
+
+[English README](README.md)
+
+TypeScript pipeline 的整体用法和入口见 [`src/README.zh-CN.md`](../README.zh-CN.md)。
+
+面向用户的优化流程见 [`../../metaskills/README.zh-CN.md`](../../metaskills/README.zh-CN.md)。
+
+## 快速开始
+
+当某个 skill 的输出不理想时，先把具体 tips 加到对应的 metaskill 文件，然后在仓库根目录运行该 skill 的 evolution 脚本。预设脚本和路径对应关系见 [`../../metaskills/README.zh-CN.md`](../../metaskills/README.zh-CN.md)。
+
+```bash
+bash metaskills/academic-army-architect/envolve.sh
+```
+
+## 直接命令
+
+可以在仓库根目录直接运行 pipeline：
+
+```bash
+npm run evolve-skill -- \
+  --config agent-forge.yaml \
+  --skill-path skills/academic-army-architect \
+  --artifact-path output/evolve-academic-army-architect \
+  --metaskill-path metaskills/academic-army-architect/METASKILL.md \
+  --task-path metaskills/academic-army-architect/ENVOLVETASK.md
+```
+
+## 参数参考
+
+必填参数：
+
+| 参数 | 说明 |
+|---|---|
+| `--skill-path` | 要修改的 skill 目录或文件。 |
+| `--artifact-path` | 每轮 runner 清空并复用的输出文件夹。 |
+| `--metaskill-path` | evaluator 和 modifier 使用的 metaskill 设计文档。 |
+| `--task-path` | runner 用来测试 skill 的固定任务文件；可重复传入多个固定任务。 |
+
+可选参数：
+
+```text
+--rounds 5
+  要运行的 self-evolve 轮数。
+```
+
+`--rounds` 默认是 `3`。
+
+## 主流程
+
+[`pipeline.ts`](pipeline.ts) 解析以下参数：
+
+- `--skill-path`：要修改的 skill 目录或文件。
+- `--artifact-path`：每轮都会清空并复用的输出目录。
+- `--metaskill-path`：用于评价和修改 skill 的设计文档与 tips。
+- `--task-path`：用于测试 skill 的一个或多个固定任务。
+- `--rounds`：self-evolve 轮数，默认是 `3`。
+
+每一轮执行以下步骤：
+
+1. 清空并重新创建 `--artifact-path`。
+2. 对每个配置的 `--task-path` 创建新的 `skill-runner` agent，让目标 skill 在该固定任务上运行并写出 artifacts。
+3. 使用 `skill-evaluator` 根据当前 metaskill 指导文件评价 artifact。
+4. 把 evaluator review 交给 `skill-modifier`，让它基于同一份 metaskill 指导修改目标 skill。
+
+新建 runner 可以避免前一轮 runner 上下文污染下一轮 artifact。evaluator 和 modifier 通过共享的 `AgentTeam` 调用，因此它们的 agent 配置集中在 pipeline config 中。
+
+## Loop 行为
+
+这个 loop 让 skill 的迭代基于具体产物，而不是凭模糊感觉重写 skill。它会让 skill 在一个固定任务上产出 artifact，再根据 metaskill 检查这个 artifact，最后让 Codex 根据具体反馈修改 skill。
+
+pipeline 通过共享 team 保留两个长生命周期 Codex thread：
+
+1. `skill-evaluator`：跨轮次评价 artifact。
+2. `skill-modifier`：跨轮次修改目标 skill。
+
+每一轮还会新建一个一次性的 `skill-runner` thread。runner 不保留上一轮上下文，避免旧 artifact 或旧对话污染下一轮输出。
+
+整个 loop 保持简单：
+
+1. 针对每个配置的 task 新建一次性的 runner thread，运行目标 skill，并把产物写入输出文件夹。
+2. 长生命周期的 evaluator thread 根据 metaskill 评价 artifact。
+3. 长生命周期的 modifier thread 根据评价修改 skill。
+4. 下一轮重新新建 runner thread。
+
+这里刻意不引入 LangGraph、状态机、registry 或复杂的 defensive wrapper。关键状态只保留在 evaluator/modifier 两个长期 Codex session、当前 artifact 文件夹，以及被修改的文件里。
+
+## 输入和输出
+
+| 项目 | 路径来源 |
+|---|---|
+| 目标 skill | `--skill-path` |
+| Metaskill | `--metaskill-path` |
+| 固定任务 | `--task-path` |
+| 生成 artifact 文件夹 | `--artifact-path`，每轮清空并复用 |
+
+## 重要文件
+
+| 路径 | 作用 |
+|---|---|
+| [`pipeline.ts`](pipeline.ts) | 参数解析和轮次编排。 |
+| [`agents/factory.ts`](agents/factory.ts) | 注册 `skill-runner`、`skill-evaluator` 和 `skill-modifier`。 |
+| [`agents/runner.ts`](agents/runner.ts) | 读取每个 `--task-path` 配置的固定任务文件，并要求目标 skill 写出 artifacts。 |
+| [`agents/evaluator.ts`](agents/evaluator.ts) | 读取 `--metaskill-path` 配置的 metaskill 文件，并评价产出的 artifact。 |
+| [`agents/modifier.ts`](agents/modifier.ts) | 读取 metaskill 文件和 evaluator review，然后修改目标 skill。 |
+
+## 常见问题
+
+| 问题 | 常见原因 | 解决办法 |
+|---|---|---|
+| Artifacts 在轮次之间消失 | `--artifact-path` 每轮都会清空并复用。 | 使用专门的 `output/evolve-*` 文件夹。 |
+| 输出仍然不理想 | Loop 需要具体 metaskill guidance。 | 把具体 tips 加到对应 metaskill 文件，然后再次运行脚本。 |
+| Runner 上下文似乎影响结果 | Runner 应该每轮新建。 | 检查 pipeline config，并归档生成 artifacts 方便比较。 |

package/src/evolve-skill/agents/evaluator.ts

@@ -0,0 +1,28 @@
+import { Agent } from "coding-agent-forge/agent";
+import { readFileSync } from "node:fs";
+
+export type SkillEvaluatorVariables = {
+  artifactPath: string;
+  metaskillPath: string;
+  taskDescriptions: string;
+};
+
+export class SkillEvaluatorAgent extends Agent<SkillEvaluatorVariables> {
+  protected buildPrompt(variables: Readonly<SkillEvaluatorVariables>): string {
+    const metaskill = readFileSync(variables.metaskillPath, "utf8");
+    return `
+Evaluate the artifact at ${variables.artifactPath}. It was produced by a skill.
+
+The artifacts were created based on the following task descriptions:
+
+${variables.taskDescriptions}
+
+The metaskill below contains the design goals and tips of this skill:
+
+${metaskill}
+
+Based on these goals and tips, are there any problems in the artifact produced by this skill? Are there any redundant parts?
+Carefully inspect both the language and the content, and use that analysis to explain how this skill can be optimized.
+`;
+  }
+}

package/src/evolve-skill/agents/factory.ts

@@ -0,0 +1,11 @@
+import type { AgentFactoryMap } from "coding-agent-forge";
+
+import { SkillEvaluatorAgent } from "./evaluator.js";
+import { SkillModifierAgent } from "./modifier.js";
+import { SkillRunnerAgent } from "./runner.js";
+
+export const agentFactories: AgentFactoryMap = {
+  "skill-runner": (thread, constants) => new SkillRunnerAgent(thread, constants),
+  "skill-evaluator": (thread, constants) => new SkillEvaluatorAgent(thread, constants),
+  "skill-modifier": (thread, constants) => new SkillModifierAgent(thread, constants),
+};

package/src/evolve-skill/agents/index.ts

@@ -0,0 +1,4 @@
+export { agentFactories } from "./factory.js";
+export { SkillEvaluatorAgent, type SkillEvaluatorVariables } from "./evaluator.js";
+export { SkillModifierAgent, type SkillModifierVariables } from "./modifier.js";
+export { SkillRunnerAgent, type SkillRunnerVariables } from "./runner.js";

package/src/evolve-skill/agents/modifier.ts

@@ -0,0 +1,27 @@
+import { Agent } from "coding-agent-forge/agent";
+import { readFileSync } from "node:fs";
+
+export type SkillModifierVariables = {
+  skillPath: string;
+  metaskillPath: string;
+  review: string;
+};
+
+export class SkillModifierAgent extends Agent<SkillModifierVariables> {
+  protected buildPrompt(variables: Readonly<SkillModifierVariables>): string {
+    const metaskill = readFileSync(variables.metaskillPath, "utf8");
+    return `
+Revise the skill at ${variables.skillPath} using the feedback below. The feedback is based on an artifact produced by this skill.
+
+The metaskill below contains the design goals and tips of this skill:
+
+${metaskill}
+
+Consider these design goals and tips when revising.
+
+Feedback:
+
+${variables.review}
+`;
+  }
+}

package/src/evolve-skill/agents/runner.ts

@@ -0,0 +1,19 @@
+import { Agent } from "coding-agent-forge/agent";
+import { readFileSync } from "node:fs";
+
+export type SkillRunnerVariables = {
+  skillPath: string;
+  artifactPath: string;
+  taskPath: string;
+};
+
+export class SkillRunnerAgent extends Agent<SkillRunnerVariables> {
+  protected buildPrompt(variables: Readonly<SkillRunnerVariables>): string {
+    const task = readFileSync(variables.taskPath, "utf8");
+    return `
+Use the skill at ${variables.skillPath} to complete the task below. Save all relevant output files in ${variables.artifactPath}.
+
+${task}
+`;
+  }
+}

package/src/evolve-skill/index.ts

@@ -0,0 +1 @@
+export * from "./pipeline.js";

package/src/evolve-skill/pipeline.ts

@@ -0,0 +1,140 @@
+import { AgentTeam, type RecordCallback } from "coding-agent-forge";
+import { mkdir, readFile, rm } from "node:fs/promises";
+import { parseArgs } from "node:util";
+import { definePipeline } from "../pipeline.js";
+import type { ParsedPipelineArgs } from "../pipeline.js";
+import { agentFactories } from "./agents/index.js";
+import type {
+  SkillEvaluatorVariables,
+  SkillModifierVariables,
+  SkillRunnerVariables,
+} from "./agents/index.js";
+
+export type EvolveSkillAgentVariables = {
+  "skill-runner": SkillRunnerVariables;
+  "skill-evaluator": SkillEvaluatorVariables;
+  "skill-modifier": SkillModifierVariables;
+};
+
+export type EvolveSkillOptions = {
+  skillPath: string;
+  artifactPath: string;
+  metaskillPath: string;
+  taskPaths: readonly string[];
+  rounds: number;
+};
+
+const USAGE =
+  "Usage: npm run evolve-skill -- --config <path> --skill-path <path> --artifact-path <folder> --metaskill-path <path> --task-path <path> [--task-path <path> ...] [--rounds <positive-integer>]";
+
+export function parseEvolveSkillArgs(
+  args: readonly string[],
+): ParsedPipelineArgs<EvolveSkillOptions> {
+  const {
+    values: {
+      config,
+      "skill-path": skillPath,
+      "artifact-path": artifactPath,
+      "metaskill-path": metaskillPath,
+      "task-path": taskPath,
+      rounds,
+    },
+  } = parseArgs({
+    args: [...args],
+    options: {
+      config: { type: "string", multiple: true },
+      "skill-path": { type: "string" },
+      "artifact-path": { type: "string" },
+      "metaskill-path": { type: "string" },
+      "task-path": { type: "string", multiple: true },
+      rounds: { type: "string" },
+    },
+  });
+
+  if (
+    config === undefined ||
+    skillPath === undefined ||
+    artifactPath === undefined ||
+    metaskillPath === undefined ||
+    taskPath === undefined
+  ) {
+    throw new Error(USAGE);
+  }
+
+  return {
+    configPaths: config,
+    runningOptions: {
+      skillPath,
+      artifactPath,
+      metaskillPath,
+      taskPaths: taskPath,
+      rounds: Number(rounds ?? 3),
+    },
+  };
+}
+
+export async function evolveSkill(
+  team: AgentTeam<EvolveSkillAgentVariables>,
+  options: EvolveSkillOptions,
+): Promise<void> {
+  const logRecord: RecordCallback = (thread, record) => {
+    console.log(thread.recordToPrettyString(record));
+  };
+  const taskDescriptions = (
+    await Promise.all(
+      options.taskPaths.map(async (taskPath, index) => {
+        const task = await readFile(taskPath, "utf8");
+        return `Task ${String(index + 1)}: ${task}`;
+      }),
+    )
+  ).join("\n\n");
+
+  for (let round = 1; round <= options.rounds; round++) {
+    await rm(options.artifactPath, { recursive: true, force: true });
+    await mkdir(options.artifactPath, { recursive: true });
+
+    for (const taskPath of options.taskPaths) {
+      const runner = await team.createAgent("skill-runner");
+      await runner.runStreamed(
+        {
+          skillPath: options.skillPath,
+          artifactPath: options.artifactPath,
+          taskPath,
+        },
+        logRecord,
+      );
+    }
+
+    const review = (
+      await team.runStreamed(
+        "skill-evaluator",
+        {
+          artifactPath: options.artifactPath,
+          metaskillPath: options.metaskillPath,
+          taskDescriptions,
+        },
+        logRecord,
+      )
+    ).trim();
+
+    console.log(`\n# Review\n${review}\n`);
+
+    const edit = await team.runStreamed(
+      "skill-modifier",
+      {
+        skillPath: options.skillPath,
+        metaskillPath: options.metaskillPath,
+        review,
+      },
+      logRecord,
+    );
+
+    console.log(`# Edit\n${edit}\n`);
+  }
+}
+
+export const evolveSkillPipeline = definePipeline({
+  agentFactories,
+  parseArgs: parseEvolveSkillArgs,
+  run: evolveSkill,
+});

package/src/pipeline.ts

@@ -0,0 +1,65 @@
+import {
+  AgentTeam,
+  isPlainObject,
+  loadYamls,
+  mergeConfig,
+  type AgentFactoryMap,
+  type AgentVariablesByName,
+  type PlainObject,
+} from "coding-agent-forge";
+
+export type ParsedPipelineArgs<Options> = {
+  configPaths: readonly string[];
+  runningOptions: Options;
+};
+
+export type PipelineDefinition<VariablesByName extends AgentVariablesByName, Options> = {
+  agentFactories: AgentFactoryMap;
+  parseArgs: (args: readonly string[]) => ParsedPipelineArgs<Options>;
+  run: (team: AgentTeam<VariablesByName>, options: Options) => Promise<void>;
+};
+
+export function definePipeline<VariablesByName extends AgentVariablesByName, Options>(
+  definition: PipelineDefinition<VariablesByName, Options>,
+): PipelineDefinition<VariablesByName, Options> {
+  return definition;
+}
+
+function buildPipelineAgentTeam<VariablesByName extends AgentVariablesByName, Options>(
+  rawConfig: PlainObject,
+  definition: PipelineDefinition<VariablesByName, Options>,
+): AgentTeam<VariablesByName> {
+  if (!isPlainObject(rawConfig.agents)) {
+    throw new Error("Config must define an agents object");
+  }
+
+  const configuredAgents = Object.fromEntries(
+    Object.entries(rawConfig.agents).filter(([name]) =>
+      Object.hasOwn(definition.agentFactories, name),
+    ),
+  );
+
+  const agents = Object.fromEntries(
+    Object.keys(definition.agentFactories).map((name) => [name, { kind: name }]),
+  );
+
+  return new AgentTeam<VariablesByName>(
+    mergeConfig({ ...rawConfig, agents: configuredAgents }, { agents }),
+    definition.agentFactories,
+  );
+}
+
+export async function runPipelineCli<VariablesByName extends AgentVariablesByName, Options>(
+  definition: PipelineDefinition<VariablesByName, Options>,
+  args: readonly string[],
+): Promise<void> {
+  const { configPaths, runningOptions } = definition.parseArgs(args);
+  const rawConfig = await loadYamls(...configPaths);
+  const team = buildPipelineAgentTeam(rawConfig, definition);
+
+  try {
+    await definition.run(team, runningOptions);
+  } finally {
+    await team.close();
+  }
+}

package/tsconfig.json

@@ -0,0 +1,22 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext",
+    "strict": true,
+    "declaration": true,
+    "declarationMap": true,
+    "sourceMap": true,
+    "outDir": "dist",
+    "rootDir": "src",
+    "exactOptionalPropertyTypes": true,
+    "noFallthroughCasesInSwitch": true,
+    "noImplicitReturns": true,
+    "noUncheckedIndexedAccess": true,
+    "noUnusedLocals": true,
+    "noUnusedParameters": true,
+    "skipLibCheck": true,
+    "types": ["node"]
+  },
+  "include": ["src/**/*.ts"]
+}