@jadelau/ace-rsi-toolkit 0.0.0-stage → 0.4.0

package/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/icon.svg

@@ -0,0 +1,7 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round">
+  <rect x="4.5" y="3" width="15" height="18" rx="2" />
+  <path d="M8.5 3v18" />
+  <path d="M11.5 8h5" />
+  <path d="M11.5 12h5" />
+  <path d="M11.5 16h3" />
+</svg>

package/package.json

@@ -1,6 +1,24 @@
 {
   "name": "@jadelau/ace-rsi-toolkit",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.4.0",
+  "description": "The ACE RSI Toolkit: Agentic Context Engineering (Zhang et al., 2025, https://arxiv.org/abs/2510.04618), the sequential offline algorithm — a Reflector reads each training Trace, a Curator folds the lessons into a playbook of rules the Target reads before every task, and the final playbook is measured on the frozen Benchmark. Self-contained: it creates the Target's playbook Skill, evaluates its own cells and writes the Evaluation Center's scoreboard.",
+  "license": "Apache-2.0",
+  "keywords": [
+    "jiumate-plugin"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/qiaiqian67/jiumate",
+    "directory": "plugins/ace-rsi-toolkit"
+  },
+  "files": [
+    "plugin.json",
+    "icon.svg",
+    "skills",
+    "hooks",
+    "LICENSE"
+  ],
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/plugin.json

@@ -0,0 +1,13 @@
+{
+  "description": "The ACE RSI Toolkit: Agentic Context Engineering (Zhang et al., 2025, https://arxiv.org/abs/2510.04618), the sequential offline algorithm — a Reflector reads each training Trace, a Curator folds the lessons into a playbook of rules the Target reads before every task, and the final playbook is measured on the frozen Benchmark. Self-contained: it creates the Target's playbook Skill, evaluates its own cells and writes the Evaluation Center's scoreboard.",
+  "version": "2026-10-12.1",
+  "category": "ai-app-development",
+  "description_zh": "ACE RSI 工具包：按论文《Agentic Context Engineering》（Zhang 等，2025，https://arxiv.org/abs/2510.04618）的顺序离线算法——Reflector 逐条阅读训练轨迹，Curator 把教训并入 Target 每次任务前都会读的规则手册（playbook），最终手册在冻结的 Benchmark 上测量。自成一体：自行创建 Target 的手册 Skill、自行评测每个单元、写入评估中心的记分板。",
+  "short_description": "ACE RSI Toolkit: evolve a playbook from Traces.",
+  "short_description_zh": "ACE RSI 工具包：从轨迹进化规则手册。",
+  "quick_start": {
+    "prompt": "Use the `ace-rsi-toolkit` Skill to run ACE on an agent of this Project: ask me for the Test Agent and the published Benchmark to train on, initialize the playbook, run one epoch at the smoke budget, and report the baseline and final scores.",
+    "prompt_zh": "使用 `ace-rsi-toolkit` 技能在本 Project 的一个智能体上运行 ACE：先问我被测智能体与用于训练的已发布 Benchmark，初始化规则手册，以冒烟预算跑一个 epoch，报告基线分与最终分。",
+    "skills": ["ace-rsi-toolkit"]
+  }
+}

package/skills/ace-rsi-toolkit/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: ace-rsi-toolkit
+description: Run ACE (Agentic Context Engineering, Zhang et al. 2025) on an agent the sequential offline way - a Reflector reads each training Trace, a Curator adds the lessons to a playbook of rules the agent reads before every task, and the final playbook is measured on a published Benchmark.
+---
+
+# ACE
+
+[Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://arxiv.org/abs/2510.04618), Zhang et al., 2025 ([arXiv:2510.04618v1](https://arxiv.org/abs/2510.04618v1), §3–4 and Appendix B). GitHub: [ace-agent/ace](https://github.com/ace-agent/ace). Verified implementation revision `82709de050e1db6e6ef2f07bcb0393560b94992a`, chiefly `ace/ace.py` (`_train_single_sample`, `_offline_train`), `ace/core/reflector.py`, `ace/core/curator.py`, `ace/prompts/` and `playbook_utils.py`. This Skill runs the **sequential offline** algorithm: the Target (the paper's Generator) solves one training case at a time with the current playbook, a Reflector diagnoses that execution, and a Curator adds the lessons the playbook lacks as itemized rules; the final playbook is then measured on the frozen Benchmark. Run the method the way the paper does; every departure is a declared adaptation reported at the end.
+
+## Before you start
+
+This Skill needs `run_subagent`, so it runs in a top-level Session. If the message only names it, ask which agent to train and on which published Benchmark. Otherwise resolve the inputs below and ask only for a missing Target or Benchmark; everything else has a default.
+
+- `test_agent_id` — the Target (the Evaluation Center's Test Agent): an existing agent of this Project, or `create: <id>` when the user wants a fresh one, created with no plugins by `penguin agent create --agent-id <id> --project-id "$PROJECT_ID" --json` (an id matches `^[a-z][a-z0-9_-]{1,63}$`). If that id exists already, stop and ask; never reinitialize an agent. The Target is never the agent you run as: training rewrites its State while you work.
+- `benchmark_id` — a Benchmark of this Project whose `benchmark_config.toml` status is published (not a literal `draft` or `failed`). Its cases (every directory of it holding a `statement/README.md`) are both the training samples and the measured case set. Without one, list the published Benchmarks and ask; this Skill never builds one.
+- `test_benchmark_id` (optional) — a second published Benchmark held out from training: H1 (the initialized State) and the final playbook are measured on it too, and nothing of it reaches the Reflector or the Curator.
+- `runs` — runs per case in every measurement; default 1.
+- The runtime — the `provider` / `model_id` pair the user names, else your Environment's, fixed for the whole experiment (`references/evaluation.md`, controller step 2). The Reflector and the Curator run on your own model; record it.
+- The settings below. Use the smoke profile unless the user asks for other values. A target score or a round limit is not an ACE setting: say so and leave it out.
+
+| Setting | Smoke profile (default) | Source |
+| --- | --- | --- |
+| `num_epochs` | 1 | release default 1; the paper's experiments use up to 5 |
+| `max_num_rounds`: reflections per incorrect sample | 3 | release default 3; the paper's experiments use up to 5 |
+| `curator_frequency`: curate after every k-th sample | 1 | batch size 1, as in the paper's experiments |
+| `playbook_token_budget` | 80000 | release default |
+| `use_bulletpoint_analyzer`: embedding deduplication | false | release default |
+| Training order | the case ids, sorted | fixed before the first execution |
+
+Budget, for C training cases, E epochs, M rounds, k = `curator_frequency` and R runs: at most `E × C × (2 + M)` training executions of the Target, `2 × C × R` executions for the H1 and final measurements (plus `2 × C' × R` on a test Benchmark of C' cases), at most `E × C × M` Reflector calls and `E × ⌊C / k⌋` Curator calls. Before the first execution, state the profile and these numbers. Never launch the paper's full budget (5 epochs, 5 rounds) implicitly: a profile above the smoke one runs only when the user asks for it, after you have restated its arithmetic.
+
+## Paths and records
+
+Resolve every path from your Environment's `App Data Dir`, as `references/evaluation.md` does; never search for the Project:
+
+```text
+PROJECT_DIR  = <app_data_dir>
+PROJECT_ID   = basename of PROJECT_DIR
+PENGUIN_HOME = dirname of PROJECT_DIR
+TARGET       = PROJECT_DIR/agents/<test_agent_id>
+STATE        = TARGET/agent_state
+BENCHMARK    = PROJECT_DIR/benchmarks/<benchmark_id>
+CASE         = BENCHMARK/<case_id>       # statement/README.md is public; rubric/README.md is private
+SCOREBOARD   = BENCHMARK/scoreboard.yaml
+SKILL_DIR    = PROJECT_DIR/agents/<your Agent ID>/agent_state/skills/ace-rsi-toolkit
+PLAYBOOK     = STATE/skills/ace-playbook # SKILL.md + rules.yaml, the only learned slot
+OUT          = <this Session's scratchpad>/ace-rsi-toolkit/<experiment_id>
+```
+
+`<experiment_id>` is `<test_agent_id>-<UTC time as YYYYMMDDTHHMMSSZ>`. Before the first execution, write `OUT/experiment.yaml`; its `dimensions` place the run in the Awesome-RSI taxonomy:
+
+```yaml
+method: ACE
+paper: "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models"
+arxiv_url: https://arxiv.org/abs/2510.04618
+github_url: https://github.com/ace-agent/ace
+source_revision: 82709de050e1db6e6ef2f07bcb0393560b94992a
+settings: { num_epochs: 1, max_num_rounds: 3, curator_frequency: 1, playbook_token_budget: 80000, use_bulletpoint_analyzer: false, correct: "score = 100" }
+test_agent_id: <id>
+benchmark_id: <id>
+test_benchmark_id: <id or null>
+training_order: [<case ids>]
+runs: 1
+runtime: { provider: <provider>, model_id: <model_id>, thinking_level: <level> }
+roles_model: { provider: <your Provider>, model_id: <your Model ID> }
+h1: { version: <n>, sha256: { <every file of STATE but .vault.toml>: <hash> } }
+dimensions:
+  artifact: playbook of itemized rules
+  source: the Target's own training executions
+  feedback: Rubric score and correctness
+  updater: Reflector and Curator, merged deterministically
+  frequency: every curator_frequency samples
+  topology: sequential, one Target
+  selection: none, the final playbook is kept
+  mode: offline
+  scope: the ace-playbook Skill
+adaptations: []
+```
+
+Keep in OUT every role request and raw reply, every cell result, every published `rules.yaml` as `OUT/playbook/v<version>.yaml`, every failure and adaptation, and `OUT/state.yaml` (epoch, sample, phase, State version), so that a compacted or resumed Session continues the run instead of restarting it.
+
+Only the evaluation worker reads a `rubric/`; you, the Reflector and the Curator never do. A Harbor case's session id `harbor:<trial>` names a trial directory under `BENCHMARK/.jobs/<job>/`: its Trace is under `agent/penguin/traces/`, it has no Workspace on this machine, and its `verifier/` output and the tasks' `tests/` and `solution/` are private.
+
+Snapshots: before the first change to a measured State, make sure `TARGET/snapshots/v<version>.tar.gz` exists: an archive of `agent_state/` without `.vault.toml`, its entries rooted at `agent_state/`. Never overwrite one, and stop if it cannot be created. State versions only increase and are never reused.
+
+## Initialize
+
+Follow `references/initialization.md`. It creates the empty playbook Skill and one fixed reader line in the Target's `AGENTS.md`, and changes nothing else. The initialized State is H1.
+
+## Measure
+
+Every measurement is a full case × `runs` matrix through `references/evaluation.md`, following its controller section, and lands in that Benchmark's `SCOREBOARD`. Measure H1 first. A prior record is reusable only when its State version, case set, runtime and runs all match. Training executions are single cells and never enter a scoreboard: the per-sample trajectory runs on a playbook that changes as it goes, so it is never reported as a matrix.
+
+## Train the playbook
+
+Make `num_epochs` passes over the training order, one sample at a time; two training cells never run at once. Every execution of the Target is one cell through `references/evaluation.md`: the request of its controller step 3 on the current State version, with a fresh `run` label for each execution of the same case (1, 2, … across the experiment), and the result handled as its controller steps 4 and 5 say. A cell is **correct** when it scores 100, or when it meets another predicate the user set before the first execution (record it). After every cell, check that `PLAYBOOK` still matches what you last published; a Target that edits it fails the run's integrity.
+
+For each sample:
+
+1. **Generate.** Run one cell on the current playbook.
+2. **Reflect.** Run the Reflector on the latest execution, then publish the counter updates its tags give. If the execution was correct, go to 3. Otherwise **regenerate**: run a training cell on the same case with that reflection in its Workspace (Regeneration, below). If it is correct, go to 3; if not, reflect on it and regenerate again, up to `max_num_rounds` reflections in all. Never add rounds because a score disappoints.
+3. **Curate.** After every `curator_frequency`-th sample, run the Curator on the latest reflection and the current playbook, and publish its ADD operations. An empty list is a valid answer.
+4. **Observe.** Run one more cell on the same case, without reflection, on the updated playbook. Record it as the post-curation observation: it gates nothing.
+
+Then go on to the next sample with the updated playbook.
+
+### Regeneration
+
+A regeneration is the one cell that differs from `references/evaluation.md`: the reflection reaches the Target as a file in its Workspace. Send the worker:
+
+```text
+Read <SKILL_DIR>/references/evaluation.md and follow its "The worker" section with one training change: after copying `statement/`, write everything below the line `--- REFLECTION.md ---` into the Workspace as `REFLECTION.md`, and launch with the message "Read README.md in the current Workspace and complete the task exactly as specified there. REFLECTION.md holds feedback on an earlier attempt at this task; use it." Evaluate one cell and reply with the protocol YAML only.
+<the eight request fields of controller step 3>
+--- REFLECTION.md ---
+<the latest reflection's error identification, root cause analysis, correct approach and key insight>
+```
+
+A Harbor case runs from its benchmark repository and has no Workspace here to carry the file: on Harbor cases, skip regeneration (one reflection per sample) and declare it.
+
+### Reflector
+
+Run one with `run_subagent` per reflection (omit `agent_id`), each in a fresh context:
+
+```text
+Read <SKILL_DIR>/SKILL.md, section "Reflector", and act as the ACE Reflector. Reply with the JSON only.
+statement: <CASE/statement/README.md>
+execution: session <session_id>, score <score>, correct <true | false>
+trace: <its root Trace files, TARGET/traces/*/<session_id>_*.jsonl, or the Harbor trial's>
+rules_used: <TARGET/scratchpad/<session_id>/ace-rules-used.txt>
+playbook: <PLAYBOOK/rules.yaml>
+```
+
+As the Reflector, read only these inputs and the Workspace the Trace's `session_meta` names: never a `rubric/`, a gold answer, another case, another Trace or OUT. When `rules_used` is missing, take the ids from the Trace's write of that file, else treat them as none. Work from what the execution actually did and produced, not from what it said it did. Reply with one JSON object: `error_identification` (what went wrong, or `none`), `root_cause_analysis`, `correct_approach`, `key_insight` (a lesson that holds beyond this case, with no case id, instance value or answer), and `bullet_tags`, a list of `{"id": "<rule id>", "tag": "helpful" | "harmful" | "neutral"}` covering only the rules used.
+
+### Curator
+
+```text
+Read <SKILL_DIR>/SKILL.md, section "Curator", and act as the ACE Curator. Reply with the JSON only.
+playbook: <PLAYBOOK/rules.yaml>
+reflection: <the OUT file of the latest reflection>
+statement: <CASE/statement/README.md>
+progress: sample <i> of <C>, epoch <e> of <E>; playbook token budget <playbook_token_budget>
+```
+
+As the Curator, read only these inputs. Add only what the playbook lacks: reusable insights from the reflection that no entry already covers, with no quota to fill. Reply `{"reasoning": "…", "operations": [{"type": "ADD", "section": "<snake_case section>", "content": "<one rule>"}]}`; an empty `operations` list is valid. ADD is the only operation, as in the release: never restate, edit, merge or delete an entry. Never copy a case id, an instance value or an answer, and keep the playbook within the token budget.
+
+### Publish
+
+Only you write `PLAYBOOK`, and only while no cell runs. Apply a change deterministically: a `helpful` or `harmful` tag adds 1 to that counter of an existing entry, and `neutral` changes nothing; an ADD appends `{id, section, content, helpful: 0, harmful: 0}` with the next fresh id (`r-0001`, `r-0002`, …, never reused) and the Curator's `content` verbatim, with no second rewrite. Drop, and record, an ADD whose content names a case id. Then bump the `ace-playbook` Skill's `version` (the next `N` when it carries today's date, else today's `.1`) and the State `version` in `STATE/system_config.yaml` by 1, replacing the field rather than adding a second one; parse both files back, and copy `rules.yaml` to `OUT/playbook/v<version>.yaml`. A change that leaves `rules.yaml` as it was publishes nothing.
+
+Deduplication stays off, as in the release: its optional BulletpointAnalyzer needs an embedding model this toolkit does not configure, so decline a request for it rather than imitate it with a model's similarity judgement and call that the paper's grow-and-refine. If a publication would push `rules.yaml` past `playbook_token_budget` (about 4 characters per token), stop training at the last published State; never compress the playbook.
+
+### Stops
+
+Stop when the budget is spent, on `version_changed` or `benchmark_invalid`, on an `evaluation_failed` with no repair left, on a runtime mismatch, on a Target edit to `PLAYBOOK`, and when private scoring material enters a role's context (discard that reflection or curation as contaminated). The last published State stays; never publish a partial or contaminated change, and say what stopped the run.
+
+## Freeze and report
+
+After the last sample, the live State is the final playbook. ACE keeps it whatever it scores: with no validation split here, the release's best-validation selection does not apply, and a training score never selects a version. Make sure its snapshot exists, then measure it on `benchmark_id`, and on `test_benchmark_id` when given, with a `summary_title` such as `ACE · final playbook, 1 epoch, 14 rules · 71.50`. After an early stop, measure the last published State only when the user asks, and name the stop in its summary. Keep every snapshot, record and OUT file; the user can bring H1 back from its snapshot.
+
+Report:
+
+- the H1 and final State versions, the initial and final `rules.yaml` with their SHA-256 hashes, and the entries per section;
+- the baseline and final scores per Benchmark, and per case;
+- per sample, the initial, regenerated and post-curation results, kept apart from the frozen matrices; the reflections, counter events, ADD operations and empty curations; the rules the Traces show the Target applying (a read is not a use, and a use is not a gain);
+- the settings and budget used against the plan, and every stop or skip;
+- the cost of every role: the Target cells' costs from the worker results, and the usage the Reflector and Curator Sessions recorded;
+- the source links and revision, and every adaptation: the playbook is a target-owned Skill read through a fixed `AGENTS.md` line; the Generator is a full agent run with tools that reports the rules it applied; correctness is a full Rubric score, and the Reflector sees the score but never a gold answer (the paper's setting without ground-truth labels, plus a score); the final playbook is kept without validation; training and measurement share `benchmark_id` unless `test_benchmark_id` was given; Harbor cases skip regeneration. This is not a reproduction of the paper's AppWorld or financial-analysis numbers;
+- that the Evaluation Center's chart shows the curve under the label `<agent_id> · <model_id> · <thinking_level>`.

package/skills/ace-rsi-toolkit/references/evaluation.md

@@ -0,0 +1,130 @@
+# Evaluate one cell
+
+How this toolkit measures a Target Agent on a Benchmark without any other plugin: the toolkit's Skill is the controller, every Case × run cell is scored by a worker the controller spawns through `run_subagent`, and every complete evaluation lands in the Benchmark's `scoreboard.yaml` in the shape the Evaluation Center reads. The same file ships in every RSI toolkit; keep it byte-identical.
+
+## Paths
+
+Resolve every path from the Environment's `App Data Dir`; never search for the Project:
+
+```text
+PROJECT_DIR  = <app_data_dir>
+PROJECT_ID   = basename of PROJECT_DIR
+PENGUIN_HOME = dirname of PROJECT_DIR
+TARGET       = PROJECT_DIR/agents/<test_agent_id>
+STATE        = TARGET/agent_state
+BENCHMARK    = PROJECT_DIR/benchmarks/<benchmark_id>
+CASE         = BENCHMARK/<case_id>       # statement/README.md is public; rubric/README.md is private
+SCOREBOARD   = BENCHMARK/scoreboard.yaml
+```
+
+Reject traversal, symlink escape, or any path outside TARGET and BENCHMARK. Never read `.project_config.toml`, `.vault.toml`, the server's `api-token` file or `web.db`, and never call the server's HTTP API with a token read from disk.
+
+## The controller
+
+1. Require `benchmark_config.toml` and read its `status`: a literal `draft` is not frozen and a literal `failed` one never calibrated (stop and say so); a missing or any other value is published. Read the Agent State `version` from `STATE/system_config.yaml` (1 when absent) and `model.thinking_level` (`medium` when absent).
+2. Fix the runtime once per experiment: `provider` and `model_id` are the pair the user named, else the `Provider` and `Model ID` lines of your own Environment. Never read a pair from the Target's files, a Project default or the server. A half pair stops the run.
+3. For every Case in the declared case set and every run index `1..runs`, call `run_subagent` once (omit `agent_id`: the worker is this Agent, so it can read this file) with exactly:
+
+```text
+Read <absolute path of this references/evaluation.md> and follow its "The worker" section. Evaluate one cell and reply with the protocol YAML only.
+protocol_version: 1
+case_id: <case_id>
+run: <1_based_run_index>
+expected_version: <agent_state_version>
+test_agent_id: <test_agent_id>
+benchmark_id: <benchmark_id>
+provider: <provider>
+model_id: <model_id>
+```
+
+Cases of one frozen State may run in parallel within capacity; two States of the same Target never run at the same time. A Case whose statement has a `## How this case is run` section naming the Harbor framework is a Harbor case, as every built-in Benchmark's are: run its cells in the background, at most four at a time unless the user named another limit (never above eight), launching the next cell the moment one returns rather than in batches and without polling, and retry once, at lower concurrency, a cell Docker could not give a network (its trial's `result.json` under `BENCHMARK/.jobs/` names an exhausted address pool; that is never a score).
+
+4. Before reading any field, check that the worker-authored text is exactly one plain YAML document. If not, ask the same worker to resend the YAML from its existing result; never rerun the Target for a formatting repair. Require every scored result's `agent_id`, `provider`, `model_id` and `thinking_level` to equal the fixed runtime, else the matrix is invalid: stop.
+5. `invalid_request`: correct and resend. `version_changed` or `benchmark_invalid`: stop. `evaluation_failed`: ask the same worker to diagnose and repair, and rerun only that cell when Workspace and Trace evidence proves the Target never started; every retry applies a new, specific repair. A matrix with a missing cell has no score: never average a partial matrix.
+6. Append one evaluation per complete matrix to `SCOREBOARD` (create the file with `evaluations: []` if the Benchmark has none), then re-parse the file and verify the appended record:
+
+```yaml
+- time: <ISO-8601 UTC, from `date -u +"%Y-%m-%dT%H:%M:%SZ"`>
+  agent_id: <test_agent_id>
+  version: <agent_state_version measured>
+  provider: <provider>
+  model_id: <model_id>
+  thinking_level: <thinking_level>
+  summary_title: >-
+    <public title: the method, the step or candidate, and the result>
+  summary: >-
+    <public summary; never the rubric, gold answers or private scoring>
+  score: <average of the Case scores, two decimals>
+  cost: <average of known Case costs (six decimals), or null when every Case cost is null>
+  duration_ms: <average of the Case durations, integer>
+  cases:
+    - case: <case_id>
+      score: <average of the Run scores>
+      cost: <average of known Run costs, or null>
+      duration_ms: <average of the Run durations>
+      runs:
+        - score: <Run score>
+          cost: <Run cost or null>
+          duration_ms: <Run duration>
+          session_id: <Test Session id>
+```
+
+Every score is on the fixed `0..100` scale; write no `max_score`, no `aggregate`, and no field the shape above does not have. The stored averages are authoritative. Record only complete measurements of a State that still exists as a snapshot; a method's own history (proposals, skipped candidates, partial matrices, failures) stays in the method's output directory, not in the scoreboard.
+
+## The worker
+
+You evaluate exactly one cell. Do not spawn another worker, do not write `SCOREBOARD`, do not change the Target or the Benchmark. Operate silently: across streamed and final responses, the only text you author is the final protocol YAML — no narration, headings, fences or scoring details.
+
+1. Validate the request: every field present once and consistent, else return `invalid_request`. Require `STATE/system_config.yaml`, `benchmark_config.toml`, `CASE/statement/README.md` and `CASE/rubric/README.md`; `status = "failed"` is `benchmark_invalid`. The State `version` (1 when absent) must equal `expected_version`, else `version_changed`. Snapshot `model.thinking_level` (`medium` when absent) and every file under `statement/` and `rubric/`. The Rubric's items must total exactly 100 points.
+2. **Workspace cases.** Create a unique Workspace under `TARGET/workspaces/`, resolve it to an absolute canonical path under that directory, and copy only `statement/` into it. The Target may see the statement and its own State, never the rubric. Run one foreground execution as separate shell statements in this order:
+
+```bash
+PROJECT_DIR="<app_data_dir>"
+PROJECT_ID="$(basename "$PROJECT_DIR")"
+PENGUIN_HOME="$(dirname "$PROJECT_DIR")"
+export PENGUIN_HOME
+penguin run \
+  --message "Read README.md in the current Workspace and complete the task exactly as specified there." \
+  --provider "<provider>" --model-id "<model_id>" --project-id "$PROJECT_ID" \
+  --agent-id "<test_agent_id>" --workspace "<absolute_unique_workspace_path>" \
+  --approve allow-all
+```
+
+   (Never pass `--source`: it is retired; every Session `penguin run` creates is a CLI Session and lands in the Web App's Background folder.) Use the exact requested Agent, Project, Workspace and model pair; never fall back to a Project default. Retry a launch only when unchanged Workspace and Trace evidence proves the Target did not start, each retry with a new diagnosis and a specific correction. Return `evaluation_failed` when no safe repair remains or it is unclear whether the Target started.
+3. **Harbor cases.** If the statement's `## How this case is run` section names the Harbor framework, links the benchmark repository at a 40-character commit and gives the `harbor run` launch, run that launch instead of step 2, from the root of a checkout at that commit, following the repository README section the statement links (fetching, concurrency, retries, `result.json`, credentials). The trial must run the Target, not a stock agent: derive `PROJECT_ID` and `PENGUIN_HOME` as in step 2, pick a job name not used before (`<case_id>-run<run>-<UTC timestamp>`), pack the State into `BENCHMARK/.jobs/<job>.agent-state.tar.gz` (create `.jobs/` when missing) with `tar --exclude=./.vault.toml --exclude=./memory --exclude=./schedule -czf <archive> -C "STATE" .`, and insert `--ak agent_state_tar=<archive> --ak host_penguin_home="$PENGUIN_HOME" --ak host_project_id="$PROJECT_ID"` and, when the checkout holds the install bundle the README describes, `--ak install_bundle=<that file>` before `--agent-setup-timeout-multiplier`, with `--job-name <job> -o "BENCHMARK/.jobs"`; remove the archive after the run. The verifier's reward × 100 is the score; `result.json`'s `agent_result.cost_usd` the cost; `agent_execution` start/finish the duration; `harbor:<trial directory name>` the session id. The one configuration value you look up is the model's API host, through the repository's helper, as that section says.
+4. After the run, verify that the State `version`, `model.thinking_level` and both material snapshots are unchanged (`version_changed` / `benchmark_invalid` otherwise). Bind exactly one root Trace whose Workspace, Agent State path, provider and model match this request (`evaluation_failed` if there is no unique match); read the actual `provider` / `model_id` from its `session_meta`. A Harbor case binds its trial instead: the actual pair is the key of `agent_result.model_usage`, and a trial the README says has no score is `evaluation_failed`.
+5. Score against the private Rubric from the Workspace, the bound root Trace and its directly referenced child Traces only. A wrong, missing or malformed answer is scored behavior (`status: ok`); a launcher, binding or scoring failure is not. `duration_ms` is the root Session's; `cost` only from usage or cost already recorded in those Traces, else `null`.
+6. Return the YAML as the only worker-authored text, without fences. Scored:
+
+```text
+protocol_version: 1
+status: ok
+case_id: <case_id>
+run: <run>
+agent_id: <test_agent_id>
+expected_version: <version>
+provider: <actual_provider>
+model_id: <actual_model_id>
+thinking_level: <configured_thinking_level>
+score: <0_to_100, two decimals>
+cost: <number_or_null>
+duration_ms: <non_negative_integer>
+session_id: <test_session_id>
+```
+
+   Failed (`null` for an identity field that was missing or conflicting; never a score, cost, duration, session id, private data or advice):
+
+```text
+protocol_version: 1
+status: failed
+case_id: <case_id_or_null>
+run: <run_or_null>
+agent_id: <test_agent_id_or_null>
+expected_version: <version_or_null>
+provider: <provider_or_null>
+model_id: <model_id_or_null>
+thinking_level: <thinking_level_or_null>
+failure_code: <invalid_request | benchmark_invalid | version_changed | evaluation_failed>
+```
+
+If the controller reports invalid formatting, resend only the clean YAML from the result already in this Session; do not call tools, relaunch or rescore.

package/skills/ace-rsi-toolkit/references/initialization.md

@@ -0,0 +1,42 @@
+# Initialize the ACE playbook
+
+Prepare the Target before H1 is measured: one target-owned Skill holding an empty playbook, and one fixed line in the Target's `AGENTS.md`. Run no execution here. The model settings, tools, hooks, memory and every other Skill stay as they are. `STATE`, `PLAYBOOK` and `OUT` are the paths of the Skill's "Paths and records".
+
+1. **Learned state.** If `PLAYBOOK` exists already, stop. It is either an experiment to resume, which needs the matching `OUT/experiment.yaml`, or learning that must not be lost: ask the user, and never empty a learned `rules.yaml`.
+2. **Version.** A Target created for this experiment is initialized in place, at `version` 1. For an existing agent, make sure the snapshot of its current version exists (the Skill's snapshot rule); the initialized State then takes that version + 1 as `version` in `STATE/system_config.yaml`.
+3. **The Skill.** Write `PLAYBOOK/SKILL.md`, with today's date version (`YYYY.MM.DD.1`) in place of the placeholder, exactly:
+
+```markdown
+---
+name: ace-playbook
+description: Playbook of rules learned from earlier tasks - read it in full before your first substantive action on a task.
+version: <YYYY.MM.DD.1>
+---
+
+# Playbook
+
+`rules.yaml` beside this file holds rules learned from earlier tasks. Each entry has an `id`, a `section`, the rule's `content`, and `helpful` / `harmful` counts of how often it helped or hurt before. Read the whole file before your first substantive action. Apply the entries that fit the current task, weigh them by their counts, and check them against the task's own evidence; where none fits, solve the task as you normally would. Keep the output the task asks for.
+
+Before you finish, write the ids of the entries you applied, comma-separated, or `none`, as the only line of `ace-rules-used.txt` in this Session's scratchpad. Never change this Skill, `rules.yaml` or anything else in your Agent State.
+```
+
+4. **The playbook.** Write `PLAYBOOK/rules.yaml` as `entries: []`. Every entry the Curator adds later has this form, and only its two counters ever change:
+
+```yaml
+entries:
+  - id: r-0001 # fresh, never reused
+    section: common_mistakes # the Curator's section
+    content: <the Curator's rule, verbatim>
+    helpful: 0
+    harmful: 0
+```
+
+5. **The reader.** Append this line to `STATE/AGENTS.md`, after a blank line when the file is not empty, exactly:
+
+```text
+Before your first substantive action on any task, read `skills/ace-playbook/SKILL.md` and `skills/ace-playbook/rules.yaml` in your Agent State in full, and follow that Skill.
+```
+
+6. **Verify.** Parse `STATE/system_config.yaml` (the expected `version`, no duplicate key), the frontmatter of `PLAYBOOK/SKILL.md` (`name: ace-playbook`, matching its directory) and `PLAYBOOK/rules.yaml` (`entries: []`), and check that the reader line appears exactly once in `STATE/AGENTS.md`. Record the SHA-256 of these four files in `OUT/experiment.yaml`.
+
+The Skill body and the reader line stay unchanged through H1, training and the final measurement: only `rules.yaml` learns, and only through the Skill's Publish step.

package/README.md

@@ -1,3 +0,0 @@
-# Temporary Holding Version
-
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.