@xingular/harness 0.1.7 → 1.1.1

package/docs/study/arms/10-refinement.md

@@ -0,0 +1,110 @@
+# Arm 10 — Refinement / self-improvement
+
+Status: **audited (self + upstream, in person)** — this arm's headline finding changes
+the shape of the study, see §1.
+
+## Design space
+
+Who is improved (prompt files, tools, tree files), how candidates are made
+(dimension-split editors?), how they are probed (isolated, unattended; approvals
+refused), how selection stays answer-free, how experience accumulates across
+tasks, and what no step may do (act in the world with a person's assets).
+
+## 1. Headline finding: this harness already ports harness²
+
+Every module of `src/refine/` is an explicit port of harness²'s core, by its own
+header comments:
+
+| this harness | upstream |
+|---|---|
+| `src/refine/recursion.ts:5` | `harness2/core/recursion.py` |
+| `src/refine/selection.ts:5` | `harness2/core/selection.py` |
+| `src/refine/prompts.ts:5` | `harness2/core/prompts.py` (the **adapt** pack only — `prompts.ts:1`) |
+| `src/refine/ledger.ts:5` | `harness2/core/ledger.py` |
+| `src/refine/library.ts:9` | `harness2/core/library.py` |
+| `src/refine/evidence.ts:13` | `harness2/core/context.py` |
+| `src/refine/runner.ts:6` | `harness2/core/schedule.py::run_task` (shaped after) |
+
+Identities preserved: R = 4k rollouts / P = 3k sessions per mode
+(`src/refine/ledger.ts:7-11`), matching upstream `recursion.py:15-19`; Stage
+shape (`Stage {label, dim, tree, keptBase, fellBack…}` `src/refine/recursion.ts:26-37`)
+with one-retry-then-inherit upstream `recursion.py` docstring; sequential-round
+checkpoint anchoring incl. the "KEEP_BASE means the CURRENT checkpoint" rule
+(`src/refine/prompts.ts:316` vs upstream `prompts.py:852-857`); the answer-free
+selector (`src/refine/prompts.ts:399` mirrors `prompts.py:791-793`); KEEP_BASE
+doctrine (`src/refine/prompts.ts:232` mirrors `prompts.py:333-335`).
+
+The merge the study was asked to plan is therefore **already largely executed**.
+What remains is a mature-upstream delta analysis, below.
+
+## 2. Comparison with harness² as it now stands
+
+Converged (ported): parallel/sequential recursion, INV/DOM candidate split with
+independent same-evidence samplers, aggregator/composer with MERGE / ADOPT-ONE /
+KEEP_BASE, anonymized answer-free selection, per-domain experience library
+carrying the **full adopted tree plus an EXPERIENCE.ms-style why/lesson** (ours:
+`src/refine/library.ts`; theirs: `external/harness2/harness2/core/library.py:15-23`,
+"the whole tree is carried, never a summary"), reflection writing cue/lesson rows
+(`src/refine/prompts.ts:418-425` mirrors their `_EXP`-driven reflector).
+
+Deltas (upstream beyond our port):
+
+1. **Second, hardened prompt pack ("tts").** Upstream ships two packs per stage
+   (`_tts_propose` / `_tts_aggregate` / `_tts_select` vs the base pack,
+   `external/harness2/harness2/core/prompts.py:838-839,896-897`); ours carries
+   only the adapt pack (`src/refine/prompts.ts:1`). The tts pack adds: an
+   explicit **`<notes>` decision audit format** (`kept|revised|dropped -> edit ->
+   evidence`, `.../prompts.py:765-769`), a `yardstick` slot ("WHAT GOOD PROCESS
+   LOOKS LIKE HERE", `.../prompts.py:705-706,757-761`), sharper always-on
+   economics and on-demand-first framing, and pool-anonymity emphases
+   (`PICKING AN UNMODIFIED HARNESS IS VALID`, `.../prompts.py:66-74`).
+2. **Screenshot-era surface dialects.** Upstream maintains this c1–c8 tree for
+   two substrates (opencode and codex) with per-substrate hook/plugin mechanics
+   (`.../prompts.py:617-661`, `tree.py:41-60 SurfaceSpec`) — ours compiles one
+   Pi surface (`src/refine/recursion.ts` imports `PI_SURFACE`).
+3. **A decoupled judge.** `harness2/core/judge.py:15-19`: measurement of the
+   chosen run happens *after selection*, **"Nothing in the method imports this
+   module"**; a judge failure is None, excluded from denominators
+   (`.../judge.py:37-40`). **Not ported** — `src/refine/` has no judge module.
+   This is the one structural piece of upstream we do not have.
+4. **Substrate contract vs our RefineHost.** Their substrate wrap is a
+   file-level (`context/harness/` editable tree + workspace dirs); ours isolates
+   through `RefineHost` (workspace, probe session, deliverable) with probe runs
+   as unattended sessions whose approvals are refused (repo contract). Same
+   shape, different isolation vocabulary — ours is the stronger primitive, but
+   note upstream's `keep_base` inherits untouched files with no re-read
+   (`.../prompts.py:186-189`); ours should verify the same semantic survived the
+   port (pending test-level confirmation).
+
+## 3. Trade-offs
+
+- **Judge in or out.** Keeping measurement out of the harness (our family rule:
+  measurement belongs to Evaluation) preserves layering but forfeits private
+  experience labeling — the `<experience>` reflection then rests only on
+  trajectory contrast, which ours already guards ("A lesson resting only on
+  `confounded` rows is a guess", `src/refine/prompts.ts:425`). Upstream's
+  compromise: a judge that labels the *chosen* run only, never feeds forward, and
+  is excluded from denominators on failure — it cannot leak into selection
+  because selection already happened.
+- **Prompt packs.** Two packs cost maintenance but let upstream evolve hardening
+  without destabilizing the published method. Ours froze one pack at port time;
+  upstream drift since then is invisible to our selectors.
+- **One surface vs many.** Ours targets Pi only; upstream's SurfaceSpec proves
+  the same method bends to codex/opencode cheaply — a bridge test for whether
+  our `RefineHost` boundary is truly substrate-agnostic.
+
+## 4. Adoption candidates
+
+| # | Idea | From | Tier | Land as |
+|---|---|---|---|---|
+| R1 | Diff the ported prompts against upstream's `adapt` lines and pull changed sentences (e.g. pool-anonymity emphasis, deliverable-shape rule) into `src/refine/prompts.ts` | harness² drift since the port | T1 | in-place prompt text sync (tests measure wording-insensitive behavior) |
+| R2 | Add the `<notes>` audit format to compose/selected stages so adoption decisions carry their evidence | tts pack | T1 | our prompts' notes sections |
+| R3 | Consider the yardstick slot as an optional, app-supplied `BenchPrompts` field — *only if* an app needs a domain yardstick; it must remain prose, never a score | tts pack via `Spec` | T2 | `RefineHost` launch input, thinking hard before adding |
+| R4 | A post-selection judge module, ported with upstream's exact decoupling (not imported by the method; None on failure excluded from denominators) **or** explicitly glued to Evaluation | judge.py | T2 with a layering decision first — see trade-off above | new `src/refine/judge.ts` behind `RefineHost`, or nothing |
+| R5 | Re-run upstream-vs-ours substrate parity test: can our tree/validation accept an upstream-authored candidate tree without edits? (`isAllowed`, caps, autotrigger semantics) | both | T1 verification task, no code change | a test |
+
+Anti-candidates, recorded so nobody re-proposes them: do **not** move measurement
+or grading into `src/refine/` (family boundary: Evaluation owns measurement);
+do **not** let any judge label reach improver or selector (answer-free is
+invariant); do **not** carry per-task facts into the durable library without the
+cue/lesson narrowing ours already enforces.

package/docs/study/autonomous-engineering-sources.md

@@ -0,0 +1,277 @@
+# Autonomous engineering: evidence from other harnesses
+
+Companion evidence for [STUDY-0002: proactive autonomy and three feature ideas](proactive-autonomy.md).
+
+Research date: 2026-10-23. Scope: autonomous software maintenance, recurring execution,
+human-readable continuity, visual inspection, and the limits of product inference.
+This is research evidence and design interpretation, not an accepted architecture change.
+
+## Finding
+
+The strongest transferable pattern is a worker that repeatedly receives new evidence,
+selects a concrete obligation, exercises the product, changes a branch, and leaves a
+legible record for the next worker. The evidence supports automating that engineering
+loop. It does not establish that a scheduler or a capable coding model can independently
+discover which new feature customers need.
+
+Three distinctions matter when comparing products:
+
+1. A scheduled prompt, a recoverable conversation, and a continuously available execution
+   service solve different problems.
+2. A screenshot, a browser interaction, and evidence that an edit fixed the correct source
+   component provide different levels of confidence.
+3. A working pull request, a justified product experiment, and an improved customer outcome
+   are different claims.
+
+These distinctions are the researcher's synthesis of the cases below, rather than vendor
+terminology or a benchmark result.
+
+## Evidence standard
+
+**Documented capability** means the vendor or project documents the behavior; it was not
+executed in this study. **Source observation** means the public implementation or README
+was inspected. **Author-reported experiment** means the system's authors describe their
+own run or evaluation. **Design inference** means a proposed lesson for this harness.
+No model runs, production deployments, repository tests, or independent comparative
+benchmarks were performed for this document.
+
+## 1. Anthropic: continuity artifacts and an evaluator that uses the application
+
+**Author-reported experiment, November 2025.** Anthropic found that compaction alone did
+not keep a coding agent coherent across many sessions. Its example used an initializer,
+a startup script, Git history, a progress log, and a JSON feature list with explicit
+acceptance steps and completion flags. A subsequent worker selected unfinished work,
+verified basic behavior, implemented an increment, and handed off artifacts. Browser
+interaction caught failures missed by unit checks or HTTP requests, although the
+particular Puppeteer integration had blind spots such as native alert dialogs. These
+are observations from Anthropic's harness experiments, not a general reliability rate.
+[Source: effective harnesses, published 2025-11-26](https://www.anthropic.com/research/effective-harnesses-for-long-running-agents).
+
+**Author-reported experiment, March 2026.** A later design separated planning, generation,
+and evaluation. The evaluator exercised the running application with Playwright and
+graded it against explicit criteria. One reported failure connected an unsuccessful
+delete interaction to a condition in `LevelEditor.tsx:892`, demonstrating a useful
+browser-to-source investigation. The author also found generous, superficial QA and
+had to tune its judgment. With Opus 4.6, the author removed sprint scaffolding and moved
+QA later; evaluation was valuable near the generator's capability limit and overhead
+for easier tasks. An illustrative updated run took 3 hr 50 min and $124.70. These are
+case studies and author-reported costs, not controlled evidence of general superiority.
+[Source: harness design, published 2026-03-24](https://www.anthropic.com/research/harness-design-long-running-apps).
+
+**Design inference.** Human readability belongs in the persisted work product: what was
+observed, which obligation was chosen, what changed, what was verified, and what remains.
+A separate critic is most useful when evidence is subjective or the change is difficult.
+Making every tiny cleanup pass through a heavyweight planner and browser evaluator can
+consume more time and money than the maintenance itself.
+
+## 2. Claude Code: maintenance becomes a default behavior, but scheduling has modes
+
+**Documented capability, checked on the research date.** Claude Code's current `/loop`
+can choose its own wakeup interval. Its default maintenance instructions continue
+unfinished work, address the branch's PR, CI failures and conflicts, then do cleanup.
+The documented scope excludes new initiatives, and irreversible actions must continue
+existing authorization. A project can replace the default with `.claude/loop.md`.
+Session cron runs while the process is active and idle, has no catch-up per missed tick,
+and recurring tasks expire after seven days. Fixed cron tasks can be restored with
+exceptions; self-paced loops are not restored. Cloud and Desktop scheduling are separate
+options: cloud scheduling needs no open session or running laptop, while Desktop jobs
+require the machine. Cloud tasks work from a fresh clone rather than local files.
+[Source: scheduled tasks documentation](https://code.claude.com/docs/en/scheduled-tasks).
+
+**Design inference.** A useful default work order can create the feeling that an agent
+knows what to do next, without pretending it has an unlimited product mandate. Recovery
+must belong to the execution system: a loop that only resumes while someone has an
+interactive process open is not an unattended maintenance service. Seven-day expiry is
+the current documented session behavior; older articles describing a three-day limit
+should not be used as the current specification.
+
+## 3. OpenHands: scheduling and recoverable execution have explicit ownership
+
+**Source observation.** OpenHands' Software Agent SDK owns conversations, workspaces,
+events, tools and Agent Server execution. Its README explicitly assigns scheduling,
+webhooks, run history and dispatch to a separate automation service. The automation
+README also assigns sandbox lifecycle orchestration to that service and labels it beta.
+This is an architectural precedent for separating *when work should happen* from
+*executing an agent conversation*.
+[SDK README](https://github.com/OpenHands/software-agent-sdk/blob/main/README.md),
+[automation README](https://github.com/OpenHands/automation/blob/main/README.md).
+
+**Source observation.** The automation scheduler polls persistent definitions and creates
+pending runs. The inspected implementation uses PostgreSQL row locks with
+`FOR UPDATE SKIP LOCKED`; its SQLite path assumes a single process. Dispatch is a separate
+poller, and completion is reported asynchronously by the execution environment. The
+inspected dispatcher distinguishes retryable HTTP statuses from permanent errors.
+These details demonstrate queueing and failure handling; they do not prove exactly-once
+execution of arbitrary external side effects.
+[Scheduler source](https://github.com/OpenHands/automation/blob/main/openhands/automation/scheduler.py),
+[dispatcher source](https://github.com/OpenHands/automation/blob/main/openhands/automation/dispatcher.py).
+
+**Documented capability.** SDK persistence stores configuration and runtime status in
+`base_state.json` and individual conversation events under `events/`. The same
+conversation ID and persistence directory can restore a conversation for another run.
+Separately, metrics expose per-LLM token usage, costs and latency, with conversation
+aggregation that includes auxiliary LLMs such as a context condenser.
+[Persistence guide](https://docs.openhands.dev/sdk/guides/convo-persistence),
+[metrics guide](https://docs.openhands.dev/sdk/guides/metrics).
+
+**Design inference.** A cron definition should create a durable work request that has
+its own identity and lifecycle. A human-readable narrative should be derived from
+those facts. Saving messages alone is not enough to decide whether a failed deployment
+was retried, whether a run still owns a branch, or whether a successful tool call was
+already applied before a worker crashed. Cost accounting must include scouting and
+evaluation, not only the worker that edits code.
+
+## 4. SWE-agent and mini-swe-agent: evidence against unnecessary harness machinery
+
+**Author-reported benchmark.** The original SWE-agent paper studies how the interface
+given to an agent affects repository navigation, editing and tool execution. It reports
+12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix in its 2024 setup. These historical
+figures establish a scoped research result; they do not describe current model quality,
+an autonomous product manager, or long-running production maintenance.
+[Paper, revised 2024-11-11](https://arxiv.org/abs/2405.15793v3).
+
+**Source observation.** The SWE-agent README now recommends mini-swe-agent as its default
+successor. mini-swe-agent's README describes a simple agent with bash actions, a linear
+message history, independent subprocess execution, configurable local or sandbox
+environments, and a trajectory browser. Its current README advertises a verified-benchmark
+score above 74%; this study did not reproduce that claim or equate it to the original
+paper's different benchmark setting.
+[SWE-agent README](https://github.com/SWE-agent/SWE-agent/blob/main/README.md),
+[mini-swe-agent README](https://github.com/SWE-agent/mini-swe-agent/blob/main/README.md).
+
+**Design inference.** Durable execution and good evidence do not require a large society
+of agents. Keep a small worker loop and add structure where observed failures require
+it. A useful trajectory is both a debugging aid and training evidence for improving
+future task selection. Benchmark issue repair supplies the problem statement; it does
+not test whether the agent chose the right problem to solve without a human.
+
+## 5. Cursor and Copilot: unattended work increasingly arrives with review artifacts
+
+**Documented capability: Cursor.** Cursor Automations supports schedules and external
+events, configurable tools, repository or multiple-repository work, and PR creation.
+Scheduled starts may be delayed. Persistent notes can be viewed and edited outside the
+worker filesystem, and computer use is included for automations. Cursor explicitly warns
+that untrusted input can influence memories and thereby later runs. An automation with
+no repository does not clone code and cannot edit code or open PRs.
+[Source: Automations](https://cursor.com/docs/cloud-agent/automations).
+
+Cloud agents have isolated desktop VMs and can exercise software, then attach screenshots,
+videos and log references. Cursor documents that artifacts embedded in GitHub PR
+descriptions use unauthenticated, unguessable URLs. Screenshots make changes reviewable,
+but their storage and publication are also deliberate product decisions.
+[Source: capabilities](https://cursor.com/docs/cloud-agent/capabilities).
+
+**Documented capability: Copilot.** GitHub's application card describes asynchronous
+cloud execution in ephemeral environments, branch and PR creation, iteration from
+feedback, and automated security scanning. It also documents missed defects, false
+positives and potentially incorrect suggestions. GitHub's browser announcement describes
+Playwright MCP for reproducing and validating web-app bugs and sharing screenshots in PRs.
+[Application card](https://docs.github.com/en/copilot/responsible-use/agents),
+[browser announcement, 2025-07-02](https://github.blog/changelog/2025-07-02-copilot-coding-agent-now-has-its-own-web-browser/).
+
+VS Code's current browser guide describes a loop using page content, screenshots,
+console errors and interaction results. Human-selected element context can include HTML
+and CSS. The guide distinguishes isolated agent-opened pages from explicitly shared,
+authenticated pages. This describes VS Code's tools; it is not a claim that every
+Copilot cloud session exposes the identical integration.
+[Source: VS Code browser tools, page dated 2026-10-21](https://code.visualstudio.com/docs/agents/run/browser-tools).
+
+**Design inference.** The novel opportunity is the quality of the autonomous judgment
+and the explanation it leaves. Cron, browser control and PR creation are already
+established features. A compelling harness should explain why it chose this change,
+show the failed journey, link the source ownership, and expose the resulting behavior.
+
+## What “knows where to fix the UI” should mean
+
+The following is a proposed evidence chain, informed by the browser cases above:
+
+| Step | Evidence the worker should preserve | What it establishes |
+|---|---|---|
+| Observe | Route, viewport, interaction trace, screenshot and console output | A specific condition was seen |
+| Reproduce | A repeatable journey and its expected result | The suspected issue can be exercised |
+| Locate | Component or handler, source revision and explanation of ownership | A plausible causal source exists |
+| Change | Patch, branch and a statement of intended behavior | The intervention is inspectable |
+| Recheck | The same journey before and after, plus relevant neighboring behavior | The intervention changed observed behavior |
+| Learn | User correction or measured post-release outcome | The repair helped outside the agent's imagined task |
+
+The first five steps can often operate against a development environment. The sixth
+requires information from real usage or a human. A screenshot cannot by itself prove
+which source file owns an element, and fixing one click path does not establish that
+every customer path works. Framework metadata, source maps, route manifests and code
+search are candidate techniques for source ownership, proposed here rather than claimed
+as a universal capability of the surveyed products.
+
+## What “knows where to clean up” should mean
+
+These are design judgments, not comparative benchmark findings:
+
+- Prefer an observed maintenance obligation: a failing check, repeated exception, unused
+  export supported by repository analysis, duplicated implementation with clear ownership,
+  or a broken journey.
+- For an apparent dead module, examine dynamic loading, scripts, plugins, generated
+  references and public package entry points before treating absence of ordinary imports
+  as evidence of non-use.
+- Preserve a compact explanation: what made the code suspect, why the behavior should
+  remain equivalent, what was exercised, and how the change can be reverted.
+- Treat a cleaner-looking diff as a hypothesis. A refactor that increases abstraction or
+  churn while changing no user outcome may be a poor use of the next unattended run.
+
+Cleanup confidence should come from converging evidence, rather than a model's confidence
+adjective. The failure mode is an endless sequence of plausible edits that produces
+activity without making the software easier to maintain.
+
+## What “suggests features from analytics” needs beyond an agent runtime
+
+The surveyed execution and browser capabilities provide tools for investigating product
+questions. They do not supply this application's success definition. The following input
+contract is a design inference:
+
+| Input | Why feature invention needs it |
+|---|---|
+| Declared product direction and excluded work | Prevents successful optimization of the wrong objective |
+| Journey events with stable definitions | Shows what users attempted, completed or abandoned |
+| Errors linked to application revision and journey | Distinguishes product friction from release failures |
+| Qualitative feedback and customer segment | Gives context for ambiguous behavior and conflicting needs |
+| Baselines, experiment assignment and outcome windows | Makes a claimed improvement assessable |
+| A permitted intervention scope | Defines where an agent may spend its own initiative |
+
+A feature idea should state the observed signal, the user problem it suggests, an
+alternative explanation, the proposed intervention, and an outcome that could prove the
+idea wrong. Low usage alone is insufficient to justify deleting a capability: it may
+serve a rare but essential workflow. Likewise, an apparent funnel improvement is not
+automatically causal if releases, traffic composition or instrumentation also changed.
+
+## Where to trade safety for more autonomous behavior
+
+The requested adventurous direction can be made explicit as product choices. These are
+proposed tradeoffs, not features already implemented in this harness:
+
+| Autonomy granted | What becomes more exciting | What the owner accepts |
+|---|---|---|
+| Let the worker choose an unassigned maintenance obligation | Useful work appears without a task queue curated by a person | Some work will be low value or based on a false diagnosis |
+| Let it change presentation and interaction details in a preview | The agent can discover and demonstrate a better experience | Taste can drift and previews may include failed experiments |
+| Let it prepare its own feature experiment from usage evidence | The harness can behave like an opinionated product collaborator | Analytics can be misread and an experiment can worsen its target metric |
+
+The interesting freedom is initiative within an understandable operating agreement.
+None of the research establishes that disabling account isolation, erasing the tool gate,
+or allowing refinement to act externally improves autonomous product judgment. Those
+would be separate architecture changes rather than a prerequisite for these ideas.
+
+## Source trace
+
+All URLs were accessed on 2026-10-23. Public documentation can change. GitHub reads used
+the connected GitHub tool. Inspected file blob SHAs, recorded for source identification:
+
+| Repository and file | Blob SHA |
+|---|---|
+| `OpenHands/software-agent-sdk/README.md` | `386dc8945133cb51969192b0b29c8396719cbff6` |
+| `OpenHands/automation/README.md` | `eb02e7d60fa72a9cd4207ca8841d37b675094643` |
+| `OpenHands/automation/openhands/automation/scheduler.py` | `d9c17e3f6bc8a6942ff82d7b1c437fe553c31842` |
+| `OpenHands/automation/openhands/automation/dispatcher.py` | `18d3f583e7f67a0fae0cde23d4ec447522c6a618` |
+| `SWE-agent/SWE-agent/README.md` | `e241ede47c338b69eb13cbc5423047e1edf1d50f` |
+| `SWE-agent/mini-swe-agent/README.md` | `7db1d82dfa92eaa00adeb10c8681677a9e6ec639` |
+
+Blob SHAs identify the retrieved file contents; they are not repository commit SHAs.
+The OpenHands scheduler and dispatcher observations above are limited to the retrieved
+opening portions of those files (through lines 250 and 240 respectively), rather than
+an audit of every backend, retry path or production deployment.

package/docs/study/autonomy-sources.md

@@ -0,0 +1,147 @@
+# Proactive autonomy precedents: OpenClaw, Paperclip, and Letta
+
+Companion evidence for [STUDY-0002: proactive autonomy and three feature ideas](proactive-autonomy.md).
+
+Research date: **2026-10-23**. Status: external evidence and design interpretation; this document changes no Harness contract or runtime behavior.
+
+The useful combination is a readable mandate, a durable wake mechanism, explicit work ownership, and memory that improves between conversations. Scheduling alone supplies none of the judgment needed to identify valuable work. The three systems below implement different parts of that combination; their documentation and source establish mechanisms, not evidence that an unattended agent consistently improves a real product.
+
+## Evidence and version boundaries
+
+Official documentation, upstream code, and original research were inspected. Source links are pinned to these upstream snapshots where possible:
+
+| System | Snapshot inspected | What the evidence establishes |
+|---|---|---|
+| OpenClaw | [`42c0d5a8`](https://github.com/openclaw/openclaw/commit/42c0d5a8f9d00e113c48090c3d259977e67b5b5a), 2026-10-22 | Automation, prospective memory, and consolidation contracts; selected implementation checks |
+| Paperclip | [`def4726b`](https://github.com/paperclipai/paperclip/commit/def4726bea0e3b340457100038d80ca55c2b7d4f), 2026-10-22 | Routine dispatch, ownership protocol, accounting controls, and suppression logic |
+| Letta Code | [`035bb8c8`](https://github.com/letta-ai/letta-code/commit/035bb8c8917b5dd94a3989d4adf04860a2ce8fea), 2026-10-22 | Reflection triggers, memory editing instructions, merge behavior, and retry controls |
+
+Two freshness traps matter. OpenClaw's [fetched live heartbeat page](https://docs.openclaw.ai/heartbeat) described a system-owned monitor, while repository HEAD documents its migration into ordinary automation jobs. Paperclip's [live routine guide](https://docs.paperclip.ing/guides/projects-workflow/routines/) described coalescing as a queued follow-up, while its API reference and dispatch implementation finalize the incoming occurrence against existing work without creating another issue. This note uses the pinned source for those disputed details. [OpenClaw migration][o1], [Paperclip API][p2], [Paperclip dispatch][p3]
+
+## OpenClaw: make recurring work inspectable
+
+### Current direction: ordinary jobs replace a special heartbeat mechanism
+
+At the inspected revision, legacy heartbeat monitors become editable automation jobs with schedules, prompts, scratch context, policies, and run history. Migration preserves identity and history and verifies the destination before retiring old configuration. Runtime no longer interprets `HEARTBEAT.md` or scratch `tasks:` blocks as schedules. Initial setup provisions one owner-associated proactive check; deleting it is lasting because a provisioning receipt prevents its recreation. The worker implementation performs receipt inspection and job creation in a state transaction. [Migration and setup][o1], [`default-proactive-job.worker.ts:20`][o2]
+
+**Interpretation:** an appealing Markdown checklist can express intent, but a machine-owned occurrence record should decide what is due. A self-editing checklist that also acts as the scheduler makes deleted tasks, concurrent edits, and restart behavior ambiguous. OpenClaw's evolution offers a stronger precedent than copying an old `HEARTBEAT.md` example.
+
+### Keep the mandate readable; make updates and silence structured
+
+Each job can retain private scratch context. Scratch updates require revision comparison; an agent rereads and reconciles a conflicting revision. During a scheduled turn, scratch and result actions are restricted to the current job. Scratch is prompt context, so its private storage does not make it an appropriate secret store. [Scratch contract][o3]
+
+The model can produce no outward message while recording an outcome such as `no_change`, `progress`, `done`, `blocked`, or `needs_attention`. A quiet outcome stays in run history and bounded session context. Silence does not erase execution failure. Explicitly empty scratch can suppress a run when that policy is enabled; absent scratch has different semantics. These are small distinctions with large consequences for an unattended system's readability. [Scratch and quiet results][o3]
+
+**Interpretation:** provide a compact human view of “what I care about” and a separate machine record of “what happened.” Show consequential changes and blocked work; let routine all-clear checks remain quiet but inspectable. Preserve the difference between a quiet success, a deliberate skip, and an error.
+
+### Durable scheduling requires recovery rules, not merely persistence
+
+OpenClaw documents persisted job definitions, runtime state, and history in shared SQLite state; its Gateway process must be running for schedules to fire. Startup reschedules overdue agent work rather than executing it during initialization, and the worker coalesces missed ticks. Execution and primary-delivery outcomes are persisted together before subsequent notification work. A terminal result recovers by exact run identity without replaying execution or delivery. [Runtime and recovery][o4]
+
+For an interrupted, unchanged timed agent job before delivery, recovery permits one fresh attempt and asks the agent to reconcile prior effects. The documentation explicitly excludes an exactly-once guarantee for arbitrary interrupted tool effects. Unknown or possible delivery, command/script payloads, and event schedules have different replay rules. This is a useful honesty boundary: a durable scheduler cannot infer whether an external write succeeded just before a crash. [Run reconciliation][o4]
+
+Policies support idle-only admission and timezone-aware active hours. The inspected migration also removes earlier heartbeat-specific duplicate suppression and no-route skipping. New designs should therefore state their own deduplication and delivery semantics rather than assuming a heartbeat label provides them. [Monitoring policies][o5], [Migration behavior][o1]
+
+### Future intent is not all the same kind of state
+
+OpenClaw distinguishes clock work, event-conditioned intent, and aspirations kept in readable notes with review dates. Standing intents live in agent SQLite state and are matched before eligible interactive replies using a keyword prefilter and authoritative scope, expiry, cooldown, and fire-budget checks. Matching does not call a model. Default cooldown, fire count, and expiry bound repeated reminders. Cancellation is explicit durable state. These intents react to eligible incoming turns; they are not independent background event monitors. [Standing intents][o6]
+
+Its memory system additionally separates dated working notes, curated durable knowledge, and a human Dream Diary. Dreaming applies evidence and provenance gates before consolidation; the model chooses operations while a writer constructs grounded entries. Untrusted and system-derived candidates are excluded from promotion. Prior memory is backed up, and the diary records additions, merges, and supersessions; the diary itself is not promoted as evidence. [Memory organization][o7], [Consolidation and diary][o8]
+
+**Interpretation:** track “when to act,” “what I believe,” and “what I learned” separately. Otherwise generated reports become their own evidence and stale intentions keep reappearing.
+
+## Paperclip: organize autonomy around owned work
+
+### Goals provide direction; assigned tasks select execution
+
+Paperclip models organizations, goals, agents, projects, and issues. Its core concepts describe a goal-to-task hierarchy and atomic task checkout, with a conflict preventing two agents from owning the same issue. The current agent skill is stricter than a free-roaming assistant: it selects assigned work, prioritizes existing work before new todo items, skips blocked tasks without new context, and exits if nothing is assigned. A scoped wake can bypass general inbox discovery. [Core concepts][p1], [`SKILL.md:94`][p4]
+
+This matters for the user's “knows what to do next” request: Paperclip demonstrates a durable work queue and allocation protocol. It does **not** establish that any worker should invent unbounded new work whenever idle. Autonomous opportunity discovery would require a separate product-level mandate, candidate selection process, and resource policy. That is a proposed extension, not an observed property of the worker protocol.
+
+The skill asks for compact incremental context rather than full thread replay. A blocked-status comment without subsequent replies suppresses another checkout or comment. Progress should identify what changed and the next owner, and waiting for a human uses a saved interaction that can wake the assignee later. A prose claim that a watcher exists is insufficient: an actual scheduled continuation must exist. [Heartbeat procedure][p4]
+
+### A routine creates accountable work with explicit overlap behavior
+
+Routine definitions contain a readable title and description, assignee, project, optional goal, priority, triggers, and policies. They support schedule, webhook, and API activation, execution history, definition revisions, and optional revision checks on updates. Agents can manage routines assigned to themselves, while assignment to another agent is an operator capability. Webhook requests can use stable idempotency keys; acceptance means queued asynchronous work, not completion. [Routine API][p2]
+
+At the inspected revision, `coalesce_if_active` finalizes the incoming occurrence as coalesced and links it to existing active execution; `skip_if_active` records a skip; `always_enqueue` creates separate work. The implementation checks for existing execution and handles the corresponding uniqueness conflict before creating and linking a new issue to an assignment wake. Catch-up is bounded, with an internal limit of 25; the scheduler has additional behavior for subhourly schedules. [Dispatch implementation][p3]
+
+**Interpretation:** decide whether a missed tick represents discarded work, merged observation, or a required follow-up. Those are different semantics. A maintenance scan usually does not deserve a backlog entry for every missed hour.
+
+### Suppress work that feeds on its own bookkeeping
+
+Paperclip's activity gate can require external activity since the last dispatched non-skipped occurrence. It excludes the routine's own execution issues, its scheduler bookkeeping, and read/archive activity, while work by other agents on delegated tasks counts as external input. Quiet skipped occurrences do not advance the observation window. This supports a watcher that runs while genuine work continues and becomes cheap when the environment settles. [Activity-gated runs][p5]
+
+Source also implements an independent no-progress wake throttle: after two consecutive successful runs without issue-visible progress, eligible empty wakes receive a two-minute cooldown that doubles up to thirty minutes. Fresh input, human comment wakes, and explicit operator resumption bypass it; agent-authored comment wakes remain throttled. Failures break the successful-no-progress streak so recovery is not mistaken for redundant polling. [Wake throttle implementation][p6]
+
+**Interpretation:** durable “no new evidence” state is as valuable as a task queue. Otherwise one autonomous agent's status comment can become another agent's reason to run, producing activity without useful progress.
+
+### Budgets must account for unfinished work
+
+The inspected accounting source evaluates policies across organization, agent, and project scopes. It distinguishes recorded spend, held reservations, pending work, and unpriced usage, with policy-dependent blocking. Admission rechecks a held reservation before provider dispatch; budget enforcement pauses applicable scope and schedules cancellation through a durable path. This is more substantial than logging costs after a cron run completes. These are source-level controls, not a measurement that real provider spend can never overshoot a limit. [Budget admission and enforcement][p7]
+
+**Interpretation:** a system allowed to discover its own work needs a resource envelope for discovery as well as implementation. Concurrent agents can exhaust a budget before any completed run reports a bill.
+
+## Letta: improve future work during idle periods
+
+### Use the current filesystem model, with historical sleep-time work as context
+
+The 2025 sleep-time architecture used a primary agent and a separate memory editor. Letta's later first-party roadmap explicitly moves away from server memory tools and server-side sleep-time agents toward filesystem operations, context repositories, and runtime subagents. Current documentation describes MemFS as Markdown memory in an agent-owned Git repository. Root memories enter active context; deeper indexed directories can be read when needed. Memory subagents use worktrees to avoid blocking the foreground agent. [Historical design][l1], [Migration direction][l2], [Current MemFS][l3]
+
+Current dreaming reviews recent conversations, extracts lessons, and updates memory. It can trigger after completed steps or compaction. The optional agent-review stage is another background conversation that can revise proposed edits; it consumes more tokens and does not require human approval. These are event/step triggers, not evidence of an always-running wall-clock daemon. Local schedules also require an active Letta session, whereas cloud schedules run in the managed environment. [Dream settings][l4], [Schedule execution targets][l5]
+
+### Autonomous learning already includes editing procedures
+
+The inspected reflection prompt prioritizes mistakes and corrections, preferences, durable facts, contradictions, and reusable workflows. It directs the agent to skip ephemeral or duplicate material, prefer existing procedures, limit skill changes, make no commit when nothing survives, and report exact memory changes and issues. It permits autonomous memory and skill edits within the memory directory, with a commit recording why. This is implemented instruction, not evidence that every learned rule improves later performance. [`reflection-v2.md:27`][l6]
+
+The launcher prevents overlapping reflection for the same agent, skips empty transcript work, and suppresses automatic retries for invalid model configuration. Worktree integration handles dirty parent state and conflicts explicitly. Failed integration has exponential retry delay from one to thirty minutes and notification deduplication; that retry state is process-local and resets on restart. [Launcher][l7], [Worktree integration][l8], [Retry implementation][l9]
+
+**Interpretation:** the bold precedent is letting experience edit future procedure without a human approving each memory diff. Git history makes those changes reviewable and reversible. It does not make them correct, authorize unrelated source edits, or justify letting a background learner rewrite its own resource and permission limits.
+
+## What has actually been measured
+
+The original sleep-time paper reports roughly five times less test-time compute at comparable accuracy on its stateful math variants, and amortized savings across related queries. Its software case study uses only 33 feature examples from two repositories and evaluates overlap with ground-truth modified files, not implementation correctness. Predictable future queries benefit more; the paper acknowledges the simplified split between preparation and later queries. These results motivate reusable preparation and memory, but do not establish reliable autonomous UI repair, product prioritization, or profitability. [Original paper, Sections 5–7 and Appendix D][r1]
+
+TriggerBench finds proactive recall can degrade with longer context even when explicitly queried recall remains strong; enhanced reasoning can produce an excessive-reminder strategy. ProEvent evaluates synthetic ongoing chats and finds overaction and cancellation failures across tested systems. These benchmarks justify measuring missed opportunities **and** false actions, timing, cancellation, and cost. Their environments differ from this Harness; the findings are directional evidence rather than predicted local failure rates. [TriggerBench][r2], [ProEvent][r3]
+
+## Design consequences for this research
+
+The following are inferences from the precedents, not existing Harness features:
+
+| Capability | Grounded precedent | Deliberate autonomy trade |
+|---|---|---|
+| Scheduled self-directed work | OpenClaw jobs plus Paperclip owned issues, activity gates, and budgets | Let the agent rank and start bounded local work from a standing mandate without a fresh human prompt |
+| Human-readable operating intent | Readable routine descriptions, scratch, explicit standing intents, and outcome diaries | Let the agent revise working plans while preserving owner intent and an inspectable change history |
+| Learning between sessions | Letta reflection, versioned memory, procedure edits, and optional agent review | Automatically apply learned procedures or candidate improvements while keeping changes attributable and reversible |
+
+The novel opportunity is combining those mechanisms with real product evidence and a human-readable explanation of why the next action was selected. The studied sources do not establish an end-to-end system that identifies a product's highest-value UI defect from analytics, repairs it correctly, and invents a desirable feature without supervision. That remains a hypothesis requiring local evaluation.
+
+The most informative first evaluation would replay real work episodes with known useful actions and explicit “do nothing” cases. Record whether the right work was chosen, duplicate or canceled work was suppressed, artifacts stayed reviewable, and spending matched value. A system that runs frequently while producing low-value changes has not demonstrated useful autonomy.
+
+[o1]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/gateway/heartbeat.md#L27
+[o2]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/src/cron/default-proactive-job.worker.ts#L20
+[o3]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/automation/cron-jobs/payloads.md#L268
+[o4]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/automation/cron-jobs/how-it-works.md#L14
+[o5]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/automation/cron-jobs/schedules.md#L61
+[o6]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/concepts/standing-intents.md#L14
+[o7]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/concepts/memory.md
+[o8]: https://github.com/openclaw/openclaw/blob/42c0d5a8f9d00e113c48090c3d259977e67b5b5a/docs/concepts/dreaming.md#L106
+[p1]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/docs/start/core-concepts.md
+[p2]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/docs/api/routines.md
+[p3]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/server/src/services/routines.ts#L1859
+[p4]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/skills/paperclip/SKILL.md#L94
+[p5]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/skills/paperclip/references/routines.md#L87
+[p6]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/server/src/services/issue-rewake-throttle.ts
+[p7]: https://github.com/paperclipai/paperclip/blob/def4726bea0e3b340457100038d80ca55c2b7d4f/server/src/services/budgets.ts
+[l1]: https://www.letta.com/blog/sleep-time-compute/
+[l2]: https://www.letta.com/blog/our-next-phase/
+[l3]: https://docs.letta.com/concepts/memfs
+[l4]: https://docs.letta.com/configuration/memory
+[l5]: https://docs.letta.com/configuration/schedules
+[l6]: https://github.com/letta-ai/letta-code/blob/035bb8c8917b5dd94a3989d4adf04860a2ce8fea/src/agent/subagents/builtin/reflection-v2.md#L27
+[l7]: https://github.com/letta-ai/letta-code/blob/035bb8c8917b5dd94a3989d4adf04860a2ce8fea/src/cli/helpers/reflection-launcher.ts
+[l8]: https://github.com/letta-ai/letta-code/blob/035bb8c8917b5dd94a3989d4adf04860a2ce8fea/src/agent/memory-worktree.ts#L363
+[l9]: https://github.com/letta-ai/letta-code/blob/035bb8c8917b5dd94a3989d4adf04860a2ce8fea/src/cli/helpers/reflection-retry.ts
+[r1]: https://arxiv.org/html/2504.13171v1
+[r2]: https://arxiv.org/abs/2606.23459
+[r3]: https://arxiv.org/abs/2607.17701

package/docs/study/btw-comparison.md

@@ -0,0 +1,37 @@
+# `/btw` side-question comparison
+
+Status: source-reviewed comparison of the checked-out workspace snapshots; no live model or UI runs were performed. This is a study note, not a change to the Harness contract.
+
+## Scope and conclusion
+
+This compares the terminal implementation in this repository, Xingular Agent's app-owned chat feature, and the external projects with an actual `/btw` implementation in the checked-out `external/` sources. The external set is Oh My Pi (`omp`), Kimi CLI, OpenCode, and OpenHands. OpenHands' SDK is its backend, not a separate UI implementation.
+
+The similarly named features are not one shared implementation. **Xingular Harness terminal creates a separate, in-memory Pi session with a bounded transcript excerpt and hard tool suppression. Oh My Pi's native `/btw` uses the main session's ephemeral side-turn pipeline, with much more context and persistent BTW history.** Xingular Agent's `/btw` is instead owned by the App: it starts a tracked, tool-free side run using a deliberately bounded public snapshot.
+
+## Comparison
+
+| Implementation | Context supplied | Isolation and persistence | Interaction |
+|---|---|---|---|
+| **Xingular Harness terminal** | Up to six visible user/assistant turns; each is clipped to 1,200 characters and the excerpt to 6,000. Includes tool names and a running flag, not tool inputs/results. | Creates an in-memory Pi session using the active model, with tools, extensions, skills, prompt templates, themes, and context files disabled; thinking is off. The side exchange is not added to the main transcript or a Harness run record. | One dock at a time. A new question replaces the old one; an empty `/btw` or Escape dismisses it and aborts the side session. The displayed answer is limited to eight lines. (`src/terminal/tui/sideQuestion.ts:13-21,39-53,69-85,106-165`; wiring: `src/terminal/opentui/app.ts:376-399`.) |
+| **Xingular Agent (App-owned)** | Up to eight visible conversation messages plus a bounded public-progress snapshot: current request, visible partial answer, status, and up to six public tool-activity summaries. Excludes hidden reasoning, tool arguments/results, and attachments. | Separate tool-free run using the conversation's account/model/effort and an opaque one-use session identity. It does not control or enter the main transcript; it has an ordinary run record/events for Trajectory and Usage. | One temporary dock per one-to-one Agent/Solo conversation. Stop and Dismiss affect only the side run; a new question replaces it. Groups and attachments are unsupported. (`../App/docs/features/FEAT-0014-side-questions.md:25-43`; snapshot and run launch: `../App/src-tauri/src/agents.rs:2748-2881`; dock lifecycle: `../App/src/modes/chat/useSideQuestion.ts:15-110`.) |
+| **Oh My Pi (`external/omp`)** | Uses the main system prompt and session history, and includes the in-progress assistant text. The snapshot preserves thinking blocks too. Its main tool catalog is supplied for the request, but the side turn does not execute tool calls. | Ephemeral side turn does not mutate the main session history. BTW questions and answers are checkpointed in a separate on-disk history sidecar; topics support follow-ups. | TUI history panel, cancellation, copy, optional promotion of a single answer to a branch, and RPC commands/stream events. (`external/omp/packages/coding-agent/src/session/agent-session.ts:11363-11550,11555-11600`; turn/follow-up context: `external/omp/packages/coding-agent/src/session/btw-turn.ts:13-93`; sidecar: `external/omp/packages/coding-agent/src/session/btw-history.ts:80-146`; controller: `external/omp/packages/coding-agent/src/modes/controllers/btw-controller.ts:121-190,356-448`; RPC: `external/omp/packages/coding-agent/src/modes/rpc/rpc-btw.ts:1-18`.) |
+| **Kimi CLI** | Uses the same system prompt, normalized conversation history, and tool definitions as the main agent, explicitly to align the prompt cache. It does not explicitly add the current partial stream to that history. | Tool definitions are visible, but a deny-all toolset rejects calls. If the model tries a tool, it gets one retry (two model steps maximum). The side exchange is not written into main conversation history. | Streams into a scrollable modal; Escape dismisses. Interactive shell and wire events are supported. (`external/kimi-cli/src/kimi_cli/soul/btw.py:33-93,102-190,233-247`; behavior: `external/kimi-cli/docs/en/reference/slash-commands.md:247-256`; modal handling: `external/kimi-cli/src/kimi_cli/ui/shell/visualize/_interactive.py:120-189,281-324`.) |
+| **OpenCode** | Makes a one-shot generation from the selected session's previewed history, system transcript, and tool definitions, followed by the BTW instruction and question. | The generation explicitly does not mutate the session; it returns text rather than entering a tool-execution loop. The question/answer live in a separate per-session extension store until the tab closes. | One side tab per question; supports retry and copy. The command is enabled only with an open session in a desktop-width window. (`external/opencode/packages/core/src/session/generate.ts:25-65`; extension state: `external/opencode/packages/gui-extensions/src/btw/model.ts:7-110`; command/panel: `external/opencode/packages/gui-extensions/src/btw/renderer.tsx:32-101`.) |
+| **OpenHands** | The default SDK path prepares a request from the enforced conversation snapshot plus the side question, using the agent's model and tool definitions. | `ask_agent` is documented as stateless: it does not alter conversation state, persist events, or change execution status. It makes a direct generation rather than running tool actions. The UI entries are held in a client-side store. | The UI shows a pending card and then the answer, which can be expanded or dismissed; it waits for the response rather than rendering token deltas. (`external/openhands-agent-sdk/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py:2903-2966`; interception: `external/openhands/src/hooks/chat/use-btw-interceptor.ts:13-43`; client state and card: `external/openhands/src/stores/btw-store.ts:1-68`, `external/openhands/src/components/features/chat/btw-messages.tsx:11-55`.) |
+
+## Cross-cutting trade-offs
+
+- **Context breadth:** Harness terminal and Agent App construct bounded excerpts. Oh My Pi, Kimi CLI, OpenCode, and OpenHands reuse broader session context. Oh My Pi is the most explicit about carrying live state—and its code includes current thinking blocks—so it is also the least restrictive about what private context the side request can see.
+- **Tool boundary:** Harness terminal and Agent App suppress tools at launch. Kimi advertises the main tool definitions but denies execution and retries after a tool attempt. Oh My Pi, OpenCode, and OpenHands make direct side generations without dispatching the returned answer through the main agent's tool-execution loop; some still send tool definitions for context or cache alignment.
+- **Retention:** Harness terminal is intentionally transient. Agent App retains an ordinary run record for observability. Oh My Pi has durable, followable BTW topics; OpenCode retains one tab's question and answer in its session store. OpenHands' displayed entries are client-side state, not main conversation events.
+- **Cache versus least privilege:** Kimi CLI explicitly aligns system prompt and tool definitions for cache reuse; Oh My Pi reuses the main session's prompt/cache path. This means broader context and tool metadata may be sent than the bounded-snapshot designs require.
+
+## Study implications
+
+1. Keep the Harness terminal's bounded context and hard no-tools setup as the conservative generic default.
+2. If a side question needs better awareness of an active run, prefer a bounded summary of public progress over forwarding hidden reasoning or raw tool inputs/results. Agent App already demonstrates that application-specific snapshot at the App layer; keep that ownership boundary intact.
+3. Treat persistent follow-up topics, branching, and RPC support as optional product capabilities, not requirements of a basic side question.
+
+## Coverage note
+
+A source scan found no `/btw` implementation in the checked-out `external/deepseek-harness`, `external/harness2`, `external/smolagents`, or `external/swe-agent` trees. Matches in `external/orca` were fixture/test references, not an implementation. These are observations about the local snapshots, not claims about current upstream releases.