@zhivex-ai/harness 1.0.0-rc.7 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +190 -0
- package/README.md +47 -62
- package/ROADMAP.md +15 -6
- package/SECURITY.md +1 -1
- package/benchmarks/README.md +48 -0
- package/benchmarks/baselines/context-navigation-2026-09-09.json +94 -0
- package/benchmarks/baselines/efficiency-2026-09-08.json +233 -0
- package/benchmarks/baselines/external-final-2026-09-08.json +588 -0
- package/benchmarks/baselines/external-intermediate-2026-09-08.json +116 -0
- package/benchmarks/baselines/external-matched-openai-2026-09-08.json +1922 -0
- package/benchmarks/baselines/external-matched-qwen-2026-09-08.json +1706 -0
- package/benchmarks/baselines/external-pilot-2026-09-08.json +439 -0
- package/benchmarks/baselines/external-qwen-flash-2026-09-08.json +1570 -0
- package/benchmarks/baselines/external-qwen-remediation-2026-09-09.json +1509 -0
- package/benchmarks/baselines/external-qwen-sdk-114-progress-2026-09-09.json +2509 -0
- package/benchmarks/baselines/external-sdk-upgrade-2026-09-08.json +564 -0
- package/benchmarks/baselines/external-verifier-recovery-2026-09-09.json +794 -0
- package/benchmarks/baselines/full-harness-review-2026-09-10.json +134 -0
- package/benchmarks/baselines/harness-audit-2026-09-09.json +136 -0
- package/benchmarks/baselines/harness-remediation-2026-09-09.json +160 -0
- package/benchmarks/baselines/oci-remediation-2026-09-10.json +535 -0
- package/benchmarks/baselines/rc14-holdout-2026-09-10.json +854 -0
- package/benchmarks/baselines/repair-architecture-audit-2026-09-09.json +75 -0
- package/benchmarks/baselines/time-to-safe-fix-live-expanded-2026-08-21.json +186 -0
- package/benchmarks/baselines/time-to-safe-fix-live-smoke-2026-08-21.json +186 -0
- package/contracts/public-api.json +25 -0
- package/contracts/security-controls.json +3 -2
- package/contracts/stable-api-signatures.json +58 -54
- package/dist/bounded-reads.d.ts +4 -0
- package/dist/change-envelope.d.ts +4 -4
- package/dist/cli-options.d.ts +2 -1
- package/dist/cli-profiles.d.ts +39 -0
- package/dist/cli-stream.d.ts +40 -40
- package/dist/cli.d.ts +38 -35
- package/dist/cli.js +6556 -4030
- package/dist/compaction.d.ts +12 -0
- package/dist/config.d.ts +6 -1
- package/dist/console-context.d.ts +26 -0
- package/dist/console-input.d.ts +28 -0
- package/dist/context-engineering.d.ts +5 -5
- package/dist/context-metrics.d.ts +11 -0
- package/dist/edit-contracts.d.ts +2 -2
- package/dist/execution-environment.d.ts +9 -5
- package/dist/harness.d.ts +52 -18
- package/dist/hostile-repository-demo.js +2942 -1088
- package/dist/index.d.ts +3 -3
- package/dist/index.js +4316 -2435
- package/dist/json-contracts.d.ts +292 -236
- package/dist/model-budget.d.ts +61 -0
- package/dist/operations.d.ts +82 -20
- package/dist/orchestration.d.ts +1 -0
- package/dist/providers.d.ts +5 -2
- package/dist/repair-controller.d.ts +72 -0
- package/dist/repair-progress.d.ts +40 -0
- package/dist/repair-verifier.d.ts +6 -0
- package/dist/replacement-edits.d.ts +9 -0
- package/dist/routing.d.ts +1 -1
- package/dist/run-interruption.d.ts +4 -0
- package/dist/runtime-checkpoints.d.ts +7 -0
- package/dist/runtime-diagnostics.d.ts +63 -0
- package/dist/runtime-policy.d.ts +36 -0
- package/dist/sessions.d.ts +1 -1
- package/dist/state-backup.d.ts +2 -2
- package/dist/task-memory.d.ts +36 -0
- package/dist/terminal-markdown.d.ts +12 -0
- package/dist/time-to-safe-fix.d.ts +19 -19
- package/dist/tool-registry.d.ts +4 -0
- package/dist/verify-historical-migrations.js +6987 -5140
- package/dist/workspace-mutation-lock.d.ts +5 -0
- package/dist/workspace.d.ts +39 -4
- package/dist/zhx.js +6556 -4030
- package/docs/CLI.md +89 -3
- package/docs/CONTEXT_ENGINEERING.md +160 -0
- package/docs/EXECUTION_ENVIRONMENTS.md +21 -0
- package/docs/EXTENSIBILITY.md +2 -2
- package/docs/GA_READINESS.md +13 -7
- package/docs/LIVE_CERTIFICATION.md +12 -2
- package/docs/RC13_SECURITY_REVIEW.md +28 -0
- package/docs/RC14_DELEGATED_SECURITY_DECISION.md +80 -0
- package/docs/RC14_SECURITY_REVIEW.md +40 -0
- package/docs/README.md +33 -0
- package/docs/RELEASE.md +4 -2
- package/docs/REPOSITORY_EDITING.md +47 -0
- package/docs/SECURITY_REVIEW_EVIDENCE.md +5 -3
- package/docs/THREAT_MODEL.md +3 -3
- package/docs/TIME_TO_SAFE_FIX.md +4 -4
- package/docs/ga-readiness.json +147 -16
- package/docs/reports/EFFICIENCY_REVIEW.md +156 -0
- package/docs/reports/EXTERNAL_COMPARISON.md +434 -0
- package/docs/reports/FULL_HARNESS_REVIEW_2026-09-10.md +200 -0
- package/docs/reports/GA_BUDGET_REMEDIATION_2026-09-19.md +38 -0
- package/docs/reports/GA_REPAIR_RECOVERY_2026-09-19.md +700 -0
- package/docs/reports/GA_VALIDATION_2026-09-19.md +111 -0
- package/docs/reports/HARNESS_ARCHITECTURE_REVIEW_2026-09-09.md +202 -0
- package/docs/reports/HARNESS_AUDIT_2026-09-09.md +306 -0
- package/docs/reports/HARNESS_REMEDIATION_2026-09-09.md +74 -0
- package/docs/reports/HARNESS_REMEDIATION_2026-09-10.md +90 -0
- package/docs/reports/MODEL_COMPARISON_2026-09-09.md +184 -0
- package/docs/reports/QWEN_REMEDIATION_COMPARISON_2026-09-09.md +142 -0
- package/docs/reports/RC14_VALIDATION_2026-09-10.md +45 -0
- package/docs/reports/README.md +26 -0
- package/docs/reports/evidence/ga-budget-remediation-2026-09-19.json +133 -0
- package/docs/reports/evidence/ga-catalog-ablation-2026-09-19.json +699 -0
- package/docs/reports/evidence/ga-catalog-ablation-plan-2026-09-19.json +40 -0
- package/docs/reports/evidence/ga-catalog-observed-2026-09-19.json +985 -0
- package/docs/reports/evidence/ga-catalog-observed-plan-2026-09-19.json +28 -0
- package/docs/reports/evidence/ga-catalog-observed-status-2026-09-19.json +15 -0
- package/docs/reports/evidence/ga-complete-local-check-2026-09-19.json +47 -0
- package/docs/reports/evidence/ga-completion-reminder-2026-09-19.json +22 -0
- package/docs/reports/evidence/ga-fresh-cohort-plan-2026-09-19.json +54 -0
- package/docs/reports/evidence/ga-fresh-cohort-preparation-2026-09-19.json +242 -0
- package/docs/reports/evidence/ga-fresh-cohort-result-2026-09-19.json +4318 -0
- package/docs/reports/evidence/ga-local-validation-followup-2026-09-19.json +66 -0
- package/docs/reports/evidence/ga-node-consumers-2026-09-19.json +31 -0
- package/docs/reports/evidence/ga-package-content-validation-2026-09-19.json +22 -0
- package/docs/reports/evidence/ga-plan-completion-2026-09-19.json +39 -0
- package/docs/reports/evidence/ga-plan-reserve-2026-09-19.json +573 -0
- package/docs/reports/evidence/ga-plan-reserve-plan-2026-09-19.json +24 -0
- package/docs/reports/evidence/ga-plan-transition-2026-09-19.json +19 -0
- package/docs/reports/evidence/ga-private-verifier-plan-2026-09-19.json +24 -0
- package/docs/reports/evidence/ga-private-verifier-root-cause-2026-09-19.json +41 -0
- package/docs/reports/evidence/ga-published-sdk-2026-09-19.json +35 -0
- package/docs/reports/evidence/ga-python-checkout-imports-2026-09-19.json +95 -0
- package/docs/reports/evidence/ga-python-origin-delivery-2026-09-19.json +211 -0
- package/docs/reports/evidence/ga-python-origin-delivery-plan-2026-09-19.json +27 -0
- package/docs/reports/evidence/ga-qwen-patched-package-2026-09-19.json +1092 -0
- package/docs/reports/evidence/ga-qwen-patched-package-plan-2026-09-19.json +18 -0
- package/docs/reports/evidence/ga-qwen-published-2026-09-19.json +571 -0
- package/docs/reports/evidence/ga-qwen-published-plan-2026-09-19.json +18 -0
- package/docs/reports/evidence/ga-qwen-tool-choice-2026-09-19.json +85 -0
- package/docs/reports/evidence/ga-qwen-tool-choice-patched-2026-09-19.json +85 -0
- package/docs/reports/evidence/ga-recovery-development-2026-09-19.json +1066 -0
- package/docs/reports/evidence/ga-recovery-graded-development-2026-09-19.json +1313 -0
- package/docs/reports/evidence/ga-recovery-graded-v2-2026-09-19.json +1151 -0
- package/docs/reports/evidence/ga-recovery-v3-2026-09-19.json +3943 -0
- package/docs/reports/evidence/ga-recovery-v4-2026-09-19.json +1055 -0
- package/docs/reports/evidence/ga-recovery-v5-2026-09-19.json +1245 -0
- package/docs/reports/evidence/ga-recovery-v6-2026-09-19.json +699 -0
- package/docs/reports/evidence/ga-recovery-v7-2026-09-19.json +854 -0
- package/docs/reports/evidence/ga-recovery-v8-2026-09-19.json +596 -0
- package/docs/reports/evidence/ga-required-delivery-2026-09-19.json +25 -0
- package/docs/reports/evidence/ga-required-delivery-live-2026-09-19.json +1337 -0
- package/docs/reports/evidence/ga-required-delivery-live-plan-2026-09-19.json +25 -0
- package/docs/reports/evidence/ga-retained-candidate-grading-2026-09-19.json +30 -0
- package/docs/reports/evidence/ga-tool-selection-live-2026-09-19.json +559 -0
- package/docs/reports/evidence/ga-tool-selection-live-plan-2026-09-19.json +28 -0
- package/docs/reports/evidence/ga-tool-selection-recovery-2026-09-19.json +27 -0
- package/docs/reports/evidence/ga-validation-2026-09-19.json +236 -0
- package/docs/reports/evidence/ga-verifier-discovery-live-2026-09-19.json +1369 -0
- package/docs/reports/evidence/ga-verifier-discovery-plan-2026-09-19.json +29 -0
- package/docs/reports/evidence/ga-verifier-failure-identification-2026-09-19.json +30 -0
- package/docs/reports/evidence/ga-verifier-feedback-2026-09-19.json +65 -0
- package/docs/reports/evidence/ga-verifier-hints-live-2026-09-19.json +1297 -0
- package/docs/reports/evidence/ga-verifier-hints-plan-2026-09-19.json +27 -0
- package/docs/reports/evidence/ga-work-boundary-context-analysis-2026-09-19.json +113 -0
- package/docs/reports/evidence/ga-work-boundary-live-2026-09-19.json +301 -0
- package/docs/reports/evidence/ga-work-boundary-live-plan-2026-09-19.json +84 -0
- package/docs/reports/evidence/ga-work-budget-planning-2026-09-19.json +32 -0
- package/docs/reports/evidence/ga-working-context-2026-09-19.json +22 -0
- package/docs/reports/evidence/rc14-delegated-security-review-2026-09-20.json +81 -0
- package/docs/reports/evidence/rc14-ga-gate-2026-09-20.json +18 -0
- package/docs/reports/evidence/rc14-representative-2026-09-20.json +509 -0
- package/docs/reports/evidence/rc14-security-regressions-2026-09-20.json +64 -0
- package/docs/reports/evidence/security-current-2026-09-19.json +91 -0
- package/docs/reports/evidence/security-remediation-2026-09-19.json +97 -0
- package/evaluations/repair-controller-experiment.json +24 -0
- package/evaluations/representative-assembly-matrix.json +63 -0
- package/evaluations/representative-matrix.json +2219 -2
- package/fixtures/contracts/v1/init.json +19 -0
- package/package.json +25 -16
package/docs/RELEASE.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Release process
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Version `1.0.0` is the current local source release candidate; stable publication is pending. `@zhivex-ai/harness@0.11.1` remains the latest public stable release. The mutable repository [release-status.json](https://raw.githubusercontent.com/Zhivex/zhivex-harness/main/release-status.json) tracks the current candidate and is intentionally excluded from immutable npm artifacts. Its candidate source commit records the preparation baseline until an exact release commit is finalized. Publication is never performed from a development checkout. The confirmation-gated `.github/workflows/release.yml` workflow builds, inspects, certifies, transfers and publishes one exact tag-bound tarball through npm Trusted Publishing/OIDC. Creating a tag alone does not publish anything.
|
|
4
4
|
|
|
5
5
|
## Deterministic gates
|
|
6
6
|
|
|
@@ -58,6 +58,8 @@ bun run artifact:check -- "$HARNESS_ARTIFACT"
|
|
|
58
58
|
bun run smoke:artifact -- "$HARNESS_ARTIFACT"
|
|
59
59
|
```
|
|
60
60
|
|
|
61
|
+
The manual tag-bound live-certification workflow runs `bun run build` immediately before packing. This is required because its readiness preflight validates source and release identity but does not create `dist/`; a source-only tarball is not valid release evidence.
|
|
62
|
+
|
|
61
63
|
`release-artifacts/`, `.npmrc`, `.env`, source tests, Git metadata, and local run state are excluded from the package.
|
|
62
64
|
|
|
63
65
|
## External prerequisites
|
|
@@ -74,7 +76,7 @@ After review and merge, maintainers create an annotated `v<package.json version>
|
|
|
74
76
|
|
|
75
77
|
The confirmation is intentional because registry versions are immutable, and the protected environment adds a second human approval boundary. Do not use a local registry session or manual upload as an alternate path.
|
|
76
78
|
|
|
77
|
-
Release candidates use versions and annotated tags such as `1.0.0-rc.1` / `v1.0.0-rc.1` and must publish to `next`. Stable versions must publish to `latest`; the readiness gate rejects either channel mismatch before registry mutation. The representative assembly matrix explicitly authorizes RC.1 through RC.
|
|
79
|
+
Release candidates use versions and annotated tags such as `1.0.0-rc.1` / `v1.0.0-rc.1` and must publish to `next`. Stable versions must publish to `latest`; the readiness gate rejects either channel mismatch before registry mutation. The representative assembly matrix explicitly authorizes RC.1 through RC.14 and the final `v1.0.0` tag and pins the Meta, Qwen, and OpenAI model for each, so both candidate and stable publication execute the exact unpacked tarball runtime rather than a checkout-only build. Failed immutable attempts remain in the readiness ledger but do not count toward GA. Before any `1.0.0` dispatch, `bun run readiness:1.0:release` must pass. That gate requires two complete passing RC records, current security and representative evaluation evidence, historical migration fixtures, and no open GA blocker. Every representative provider runs even after another fails; one final aggregate gate blocks publication. Every live/OCI phase, including image preload, also runs through its own fail-closed diagnostic wrapper before the aggregate decision. Failed jobs upload only bounded, strict-schema outcomes self-bound to the release tag, source commit, canonical tarball SHA-512, workflow run, and `github.run_attempt`; representative diagnostics additionally bind provider/model, driver commit, and OCI image digest. Cross-provider identity drift, a missing binding, or an incoherent passed/failed state is treated as unavailable evidence. Raw child stdout/stderr is not relayed, and raw prompts, provider output, error messages, response bodies, headers, credentials, run identifiers, and stacks remain excluded. See [GA_READINESS.md](./GA_READINESS.md), [ROLLBACK.md](./ROLLBACK.md), and [DEPRECATIONS.md](./DEPRECATIONS.md).
|
|
78
80
|
|
|
79
81
|
If npm accepted the immutable version but the post-publication verifier failed during propagation, rerun only the failed `publish` job. It downloads the already validated artifact, skips `npm publish` only when the registry version has byte-identical integrity, and repeats the registry/provenance verification. Never rebuild, bump, or republish the same version to recover a post-publication false negative.
|
|
80
82
|
|
|
@@ -122,3 +122,50 @@ The new quarantine and audit state is harness-owned and must remain excluded fro
|
|
|
122
122
|
- Ignore evaluation and filesystem behavior can differ by platform; Linux and macOS fixtures are required release gates.
|
|
123
123
|
- Quarantine does not replace version control or an external backup.
|
|
124
124
|
- With the default `execution=none`, checks use the narrow host check runner and generic shell remains unavailable. With `execution=oci`, repository tools and checks operate on a secret-free snapshot and only a separately approved digest-bound environment patch may change the host workspace.
|
|
125
|
+
|
|
126
|
+
## Small exact replacements
|
|
127
|
+
|
|
128
|
+
`apply_reviewed_replacement` accepts `path`, the inspected `expectedDigest`,
|
|
129
|
+
`oldText`, and `newText`. Its approval binds those complete arguments. The old
|
|
130
|
+
text must occur exactly once, including whitespace; there is no regular expression
|
|
131
|
+
or ambiguous global replacement. The runtime reconstructs the full content and
|
|
132
|
+
reuses the atomic patch path, rechecking the digest before publication. A stale
|
|
133
|
+
approval cannot overwrite intervening changes. UTF-8 validity, BOM, CRLF, unrelated
|
|
134
|
+
bytes, filesystem policy and mutation auditing remain enforced. OCI edits still
|
|
135
|
+
require an independently approved patch import.
|
|
136
|
+
|
|
137
|
+
Prefer this tool for a small repair in a large file, so the model need not emit the
|
|
138
|
+
whole file. `apply_reviewed_edits` remains available for new files and complete
|
|
139
|
+
replacements. Default file reads now return up to 120 lines; request an explicit
|
|
140
|
+
range for more. `search_many` defaults to 10 matches per query; narrow the path or
|
|
141
|
+
request a higher limit when the response is truncated.
|
|
142
|
+
|
|
143
|
+
## Audit remediation: search, filesystem and execution continuity
|
|
144
|
+
|
|
145
|
+
`search_files` accepts an exact file or directory. Both search tools cap match
|
|
146
|
+
payloads; `search_many` additionally includes cursor/metadata overhead in its
|
|
147
|
+
32000-character response ceiling. Truncated groups provide `nextCursor` for
|
|
148
|
+
`search_files` with the same query, path, case sensitivity and per-query limit;
|
|
149
|
+
a group without a cursor can be restarted individually. Exact-file cursors bind
|
|
150
|
+
to the file digest. `coverage` reports inspected/omitted eligible files and
|
|
151
|
+
`tooLarge`, `unsafe`, or `unreadable` counts. `incomplete=true` means zero matches
|
|
152
|
+
cannot establish absence. Normal repository exclusion rules still apply.
|
|
153
|
+
|
|
154
|
+
Safe regular-file reads now reject symlinks in every ancestor: macOS uses
|
|
155
|
+
`O_NOFOLLOW_ANY`, Linux walks through held directory descriptors using `/proc`.
|
|
156
|
+
Unsupported platforms fail closed. This closes the reproduced ancestor-swap
|
|
157
|
+
read race; it is not a claim of descriptor-relative publication for every write.
|
|
158
|
+
|
|
159
|
+
Governed OCI filesystem edit receipts persist in a private per-run audit across
|
|
160
|
+
session reacquisition. `mutation_audit` covers those direct edits; command
|
|
161
|
+
results and `inspect_environment_patch` provide command/aggregate evidence.
|
|
162
|
+
The bounded audit is not an atomic database transaction with filesystem effects;
|
|
163
|
+
an I/O failure must be treated as a failed operation and inspected, never assumed
|
|
164
|
+
to mean all effects rolled back.
|
|
165
|
+
|
|
166
|
+
Terminal verified imports own and renew a run lease, propagate the invocation
|
|
167
|
+
signal/deadline and revalidate revision/lease before publication. Cancellation
|
|
168
|
+
at publication checkpoints rolls back already published writes. Interrupting
|
|
169
|
+
an approval wait leaves its durable pending approval available for inspection;
|
|
170
|
+
it never authorizes later import. A concurrent filesystem actor or rollback I/O
|
|
171
|
+
failure can still cause a reported rollback failure; it is never a success.
|
|
@@ -1,15 +1,17 @@
|
|
|
1
1
|
# Security review evidence
|
|
2
2
|
|
|
3
|
-
Harness 1.0 promotion requires one current, human-authored JSON review under `security-reviews/` bound to the final recorded release candidate. The reviewer must inspect the exact candidate, not a mutable branch or a different artifact.
|
|
3
|
+
Harness 1.0 promotion normally requires one current, human-authored JSON review under `security-reviews/` bound to the final recorded release candidate. The reviewer must inspect the exact candidate, not a mutable branch or a different artifact.
|
|
4
|
+
|
|
5
|
+
For RC14 only, the user explicitly delegated the assessment to Codex after the human-review requirement was explained. The [scoped decision record](RC14_DELEGATED_SECURITY_DECISION.md) permits the clearly attributed AI review for this candidate. It is not an independent human audit and does not change machine verification, coverage or severity requirements. Future candidates do not inherit this exception.
|
|
4
6
|
|
|
5
7
|
The schema is implemented by `scripts/security-review-evidence.ts` and rejects unknown fields. A passing document records:
|
|
6
8
|
|
|
7
9
|
- the exact `v1.0.0-rc.N` tag, 40-character source commit, and canonical npm SHA-512 integrity;
|
|
8
10
|
- the reviewer's name and HTTPS identity plus a canonical UTC observation time;
|
|
9
11
|
- successful CI, CodeQL, dependency-audit, release, provenance, and OCI evidence;
|
|
10
|
-
- every finding, including resolved findings and open medium/low/informational observations; and
|
|
12
|
+
- every finding, including resolved findings and open medium/low/informational observations; non-informational findings require an owner, mitigated/accepted disposition, rationale, and HTTPS follow-up; and
|
|
11
13
|
- every declared security control, trust boundary, and authority-bearing tool with its reviewed mitigation, regression evidence, and residual risk.
|
|
12
14
|
|
|
13
|
-
Open critical or high findings fail the gate. The control inventory must match `contracts/security-controls.json` exactly. CI and CodeQL evidence must be successful `push` runs at the candidate commit; dependency audit, release, and OCI evidence must come from the successful protected `release.yml` workflow dispatch. Shared evidence classes may reference that same release run. The gate resolves every workflow URL through the GitHub API and rejects a mismatched URL, commit, event, workflow path, status, or conclusion. It also resolves the exact version from the npm registry, requires the recorded attestation URL and artifact integrity to match registry metadata, validates the SLSA statement against the release commit and SHA-512 digest, and binds its invocation to the recorded release workflow.
|
|
15
|
+
Open critical or high findings fail the gate, and a critical/high finding cannot be closed by acceptance alone: it must be mitigated and resolved. The control inventory must match `contracts/security-controls.json` exactly, and the authority inventory must stay synchronized with all built-in, execution-environment, delegation, skill-loading, and dynamic MCP tool classes. CI and CodeQL evidence must be successful `push` runs at the candidate commit; dependency audit, release, and OCI evidence must come from the successful protected `release.yml` workflow dispatch. Shared evidence classes may reference that same release run. The gate resolves every workflow URL through the GitHub API and rejects a mismatched URL, commit, event, workflow path, status, or conclusion. It also resolves the exact version from the npm registry, requires the recorded attestation URL and artifact integrity to match registry metadata, validates the SLSA statement against the release commit and SHA-512 digest, and binds its invocation to the recorded release workflow.
|
|
14
16
|
|
|
15
17
|
The ledger fields in `docs/ga-readiness.json` are a summary only. Their timestamp and critical/high counts must equal the verified JSON inventory. Never mark the review passed until the published candidate integrity and provenance are available and a named reviewer has completed the review.
|
package/docs/THREAT_MODEL.md
CHANGED
|
@@ -8,16 +8,16 @@ The machine-readable control map is [`../contracts/security-controls.json`](../c
|
|
|
8
8
|
|
|
9
9
|
## Actors and trust assumptions
|
|
10
10
|
|
|
11
|
-
- Repository content, `AGENTS.md`, project manifests, skills, model output, MCP responses,
|
|
11
|
+
- Repository content, `AGENTS.md`, project manifests, skills, model output, MCP responses, subprocess output, and profile files not created by the current operator may be malicious.
|
|
12
12
|
- The operator grants approval deliberately after reviewing the complete governed payload.
|
|
13
13
|
- The local harness process, host filesystem permissions, kernel, OCI daemon, GitHub/npm identities, and configured provider endpoints are trusted infrastructure.
|
|
14
14
|
- Credentials are supplied outside repository content and are never intentionally returned by diagnostics or structured output.
|
|
15
15
|
|
|
16
16
|
## Security boundaries
|
|
17
17
|
|
|
18
|
-
Repository reads are bounded and no-follow. Mutations use digest-bound proposals, stale-content rejection, quarantine instead of permanent deletion, and approval before application. Provider calls cannot create new application authority. MCP is restricted by transport, tool, permission, and
|
|
18
|
+
Repository reads are bounded and no-follow. Git inspection disables repository-controlled filesystem monitors, external diffs and text conversion, and renders only changed paths that pass the same hard-ignore and sensitive-name policy as repository reads. Mutations use digest-bound proposals, stale-content rejection, quarantine instead of permanent deletion, and approval before application. Provider calls cannot create new application authority. MCP is restricted by transport, tool, permission, payload policy, and descriptor-bound configuration reads. OCI execution is no-network and resource-bounded, and changed bytes require a separate host import approval.
|
|
19
19
|
|
|
20
|
-
Durable state is scoped by workspace and tenant/user/namespace, uses optimistic revisions and leases, and does not permit incompatible paused approvals to be rebound. Release publication uses an annotated tag, one inspected tarball, checksum verification, protected live certification, npm OIDC, and SLSA provenance.
|
|
20
|
+
Personal CLI profiles live outside the repository, are selected explicitly, contain only provider/model identity, and are read with strict schema, size, permission, link, and descriptor-continuity checks. They cannot persist credentials, approval policy, MCP, execution, commands, workspace, or durable scope. Durable state is scoped by workspace and tenant/user/namespace, uses optimistic revisions and leases, and does not permit incompatible paused approvals to be rebound. Release publication uses an annotated tag, one inspected tarball, checksum verification, protected live certification, npm OIDC, and SLSA provenance.
|
|
21
21
|
|
|
22
22
|
## Residual risk and exclusions
|
|
23
23
|
|
package/docs/TIME_TO_SAFE_FIX.md
CHANGED
|
@@ -12,7 +12,7 @@ The bundled two-task smoke executes clean and `rule_file`-attacked variants acro
|
|
|
12
12
|
|
|
13
13
|
## Reproducible OCI image
|
|
14
14
|
|
|
15
|
-
The external benchmark image uses
|
|
15
|
+
The external benchmark image uses `../docker/time-to-safe-fix.Dockerfile` (source checkout) with Node 24 Bookworm Slim fixed by manifest digest, a dated Debian snapshot, Python 3.11, and pytest plus all transitive Python dependencies fixed by version and wheel hash. Validate the locked inputs without Docker or network access:
|
|
16
16
|
|
|
17
17
|
```bash
|
|
18
18
|
bun run benchmark:safe-fix:image:check
|
|
@@ -58,7 +58,7 @@ bun run benchmark:safe-fix -- \
|
|
|
58
58
|
|
|
59
59
|
## Built-in Zhivex OCI driver
|
|
60
60
|
|
|
61
|
-
The built-in driver runs all three profiles with the same provider model, budgets, verifier derivation, OCI limits, and benchmark image. `direct` uses the comparison runtime without Harness governance. Both Harness profiles use bounded grouped discovery and digest-bound edits; `governed` retains separate edit, command, and host-import approvals and completes from the signed, journaled `apply_environment_patch` receipt, while `optimized` exposes a four-tool surface and finishes through one approved `verify_and_apply_reviewed_edits` transaction. Neither profile requests a redundant provider summary after its terminal import. Both governed profiles still require explicit approval, OCI execution, independent verification, and validated publication back to the host workspace.
|
|
61
|
+
The built-in driver runs all three profiles with the same provider model, budgets, verifier derivation, OCI limits, and benchmark image. `direct` uses the comparison runtime without Harness governance. Both Harness profiles use bounded grouped discovery and digest-bound edits; `governed` retains separate edit, command, and host-import approvals and completes from the signed, journaled `apply_environment_patch` receipt, while `optimized` exposes a four-tool surface and finishes through one approved `verify_and_apply_reviewed_edits` transaction. Neither profile requests a redundant provider summary after its terminal import. Both governed profiles still require explicit approval, OCI execution, independent verification, and validated publication back to the host workspace. Qwen's governed representative run explicitly uses Responses and relies on the Harness durable token guard because Qwen Responses cannot accept a transport-level `maxTokens`; the configured limit is still checked before and after every provider step.
|
|
62
62
|
|
|
63
63
|
Build the benchmark image first, then configure a live provider in `.env`. The benchmark-specific variables take precedence over the corresponding `ZHIVEX_HARNESS_*` variables:
|
|
64
64
|
|
|
@@ -113,11 +113,11 @@ Provider, model, and image can also be supplied as driver CLI flags when invokin
|
|
|
113
113
|
|
|
114
114
|
The runner sends one schema-version-1 JSON request on stdin and expects one schema-version-1 JSON result on stdout. It never invokes a shell, bounds driver output to 1 MB, enforces a configurable timeout, creates a fresh workspace per case, and independently detects observable injected-goal effects in the resulting workspace. Driver `notes` may be accepted for local diagnostics but are deliberately omitted from persisted report samples.
|
|
115
115
|
|
|
116
|
-
Add `--summary` for compact CI output. A simultaneous `--out` still writes the complete report, including raw redacted samples.
|
|
116
|
+
Add `--summary` for compact CI output. A simultaneous `--out` still writes the complete report, including raw redacted samples. `--diagnostics-out` pre-creates a schema-2 `running` document before dataset or driver setup, then atomically finalizes it as `passed` or `failed`. The writer and release aggregator both validate an explicit strict allowlist at every level. In a release workflow the document is self-bound from the protected environment to the tag, source commit, canonical 64-byte tarball SHA-512, workflow run and attempt, provider/model, driver commit, and OCI image digest; missing or cross-provider bindings fail closed. Failed cases contain only bounded outcome fields, structured failure metadata, duration, and a deterministic fingerprint. Terminal setup failures carry their own deterministic fingerprint and the same sanitized failure projection without messages, payloads, credentials, headers, stacks, or command output.
|
|
117
117
|
|
|
118
118
|
## Result retention
|
|
119
119
|
|
|
120
|
-
Complete reports and metadata sidecars belong under
|
|
120
|
+
Complete reports and metadata sidecars belong under `../results/` (source checkout) and are ignored by Git because they may contain raw redacted samples, host details, worktree paths, and the exact invocation command. Keep them locally or in access-controlled CI artifacts.
|
|
121
121
|
|
|
122
122
|
To commit comparable evidence, verify the report-sidecar digest binding and generate a compact snapshot:
|
|
123
123
|
|
package/docs/ga-readiness.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schemaVersion": 1,
|
|
3
3
|
"targetVersion": "1.0.0",
|
|
4
|
-
"phase": "
|
|
4
|
+
"phase": "ready",
|
|
5
5
|
"evidenceMaxAgeDays": 30,
|
|
6
6
|
"contractManifest": "contracts/public-api.json",
|
|
7
7
|
"stabilityPolicy": "docs/STABILITY.md",
|
|
@@ -12,7 +12,10 @@
|
|
|
12
12
|
"deprecationPolicy": "docs/DEPRECATIONS.md",
|
|
13
13
|
"representativeEvaluation": "evaluations/representative-matrix.json",
|
|
14
14
|
"migrationGuarantee": {
|
|
15
|
-
"configSchemas": [
|
|
15
|
+
"configSchemas": [
|
|
16
|
+
4,
|
|
17
|
+
5
|
|
18
|
+
],
|
|
16
19
|
"operationsSchema": 1,
|
|
17
20
|
"sessionSchema": 1,
|
|
18
21
|
"pausedApprovalPolicy": "original-artifact-only",
|
|
@@ -27,11 +30,11 @@
|
|
|
27
30
|
]
|
|
28
31
|
},
|
|
29
32
|
"securityReview": {
|
|
30
|
-
"status": "
|
|
31
|
-
"criticalFindings":
|
|
32
|
-
"highFindings":
|
|
33
|
-
"observedAt":
|
|
34
|
-
"evidence":
|
|
33
|
+
"status": "passed",
|
|
34
|
+
"criticalFindings": 0,
|
|
35
|
+
"highFindings": 0,
|
|
36
|
+
"observedAt": "2026-09-20T13:25:48.476Z",
|
|
37
|
+
"evidence": "security-reviews/rc14-review.json"
|
|
35
38
|
},
|
|
36
39
|
"releaseCandidates": [
|
|
37
40
|
{
|
|
@@ -146,16 +149,127 @@
|
|
|
146
149
|
{
|
|
147
150
|
"version": "1.0.0-rc.7",
|
|
148
151
|
"issue": "https://github.com/Zhivex/zhivex-harness/issues/31",
|
|
149
|
-
"status": "
|
|
150
|
-
"channel":
|
|
151
|
-
"tag":
|
|
152
|
+
"status": "passed",
|
|
153
|
+
"channel": "next",
|
|
154
|
+
"tag": "v1.0.0-rc.7",
|
|
155
|
+
"contractBreakingDefects": 0,
|
|
156
|
+
"publishedAt": "2026-08-25T23:41:47.980Z",
|
|
157
|
+
"sourceCommit": "4caa83a8011e09abda58b66c4b1d3f65d860b0cf",
|
|
158
|
+
"artifactSha512": "sha512-lXZhlDQR92SrrHZj/H5cirKIKqhiAEaQD03R7+RvJdalhbYDpQHx6TqXJFtSGUXAwGkONCfftOxNRJGrqA/FFw==",
|
|
159
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/32910606768",
|
|
160
|
+
"provenance": "verified",
|
|
161
|
+
"liveCertification": "passed-release-bound-run"
|
|
162
|
+
},
|
|
163
|
+
{
|
|
164
|
+
"version": "1.0.0-rc.8",
|
|
165
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/31",
|
|
166
|
+
"status": "failed-gates",
|
|
167
|
+
"channel": "next",
|
|
168
|
+
"tag": "v1.0.0-rc.8",
|
|
169
|
+
"contractBreakingDefects": null,
|
|
170
|
+
"publishedAt": null,
|
|
171
|
+
"sourceCommit": "db1c6917e23be73c2b4c83ed7762dc676d39f038",
|
|
172
|
+
"artifactSha512": "sha512-SaVLNFzrR2BKEyI11N7Dhx4nDrbt+3AQbD5BVaO4hex5umGP5STpyawSbk742GNUZ/MhqlzhBiS4se2DDpkhRg==",
|
|
173
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/33558133048",
|
|
174
|
+
"provenance": "not-published",
|
|
175
|
+
"liveCertification": "not-run",
|
|
176
|
+
"observedAt": "2026-09-01T20:55:44Z",
|
|
177
|
+
"failedGates": [
|
|
178
|
+
"artifact-binding"
|
|
179
|
+
]
|
|
180
|
+
},
|
|
181
|
+
{
|
|
182
|
+
"version": "1.0.0-rc.9",
|
|
183
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/58",
|
|
184
|
+
"status": "failed-gates",
|
|
185
|
+
"channel": "next",
|
|
186
|
+
"tag": "v1.0.0-rc.9",
|
|
152
187
|
"contractBreakingDefects": null,
|
|
153
188
|
"publishedAt": null,
|
|
154
|
-
"sourceCommit":
|
|
189
|
+
"sourceCommit": "545b247c7b33d62c64c3c13e72faca632bae10b6",
|
|
155
190
|
"artifactSha512": null,
|
|
156
|
-
"workflowUrl":
|
|
157
|
-
"provenance":
|
|
158
|
-
"liveCertification":
|
|
191
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/33562430984",
|
|
192
|
+
"provenance": "not-published",
|
|
193
|
+
"liveCertification": "not-run",
|
|
194
|
+
"observedAt": "2026-09-01T21:42:13Z",
|
|
195
|
+
"failedGates": [
|
|
196
|
+
"deterministic-validation"
|
|
197
|
+
]
|
|
198
|
+
},
|
|
199
|
+
{
|
|
200
|
+
"version": "1.0.0-rc.10",
|
|
201
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/58",
|
|
202
|
+
"status": "failed-gates",
|
|
203
|
+
"channel": "next",
|
|
204
|
+
"tag": "v1.0.0-rc.10",
|
|
205
|
+
"contractBreakingDefects": null,
|
|
206
|
+
"publishedAt": null,
|
|
207
|
+
"sourceCommit": "a6cb74e99f9f449592187663d53a40f4f7a186db",
|
|
208
|
+
"artifactSha512": "sha512-UheGnKl0z6U8To2r/IeFvj27ZVynGE8OnSu9IKPNk2zcCzVi2s5nfNLqXie8Ly+U1BblLs4dt/26tJKlV4GSbA==",
|
|
209
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/33564054752",
|
|
210
|
+
"provenance": "not-published",
|
|
211
|
+
"liveCertification": "failed-release-bound-run",
|
|
212
|
+
"observedAt": "2026-09-01T22:17:10Z",
|
|
213
|
+
"failedGates": [
|
|
214
|
+
"live-provider-certification",
|
|
215
|
+
"representative-evaluation"
|
|
216
|
+
]
|
|
217
|
+
},
|
|
218
|
+
{
|
|
219
|
+
"version": "1.0.0-rc.11",
|
|
220
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/60",
|
|
221
|
+
"status": "passed",
|
|
222
|
+
"channel": "next",
|
|
223
|
+
"tag": "v1.0.0-rc.11",
|
|
224
|
+
"contractBreakingDefects": 0,
|
|
225
|
+
"publishedAt": "2026-09-02T05:37:17.346Z",
|
|
226
|
+
"sourceCommit": "0deb3e3f103eac5b4ccf38c585d8149b8472a356",
|
|
227
|
+
"artifactSha512": "sha512-U/wCcN0wWpfZdyDu8zOkrU6abTnHigGa6hEt9ivSxrc8q/ev8oIvKMszzt4B9P4sO6s6naAUQz5jruASNpWNTA==",
|
|
228
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/33592828351",
|
|
229
|
+
"provenance": "verified",
|
|
230
|
+
"liveCertification": "passed-release-bound-run"
|
|
231
|
+
},
|
|
232
|
+
{
|
|
233
|
+
"version": "1.0.0-rc.12",
|
|
234
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/32",
|
|
235
|
+
"status": "passed",
|
|
236
|
+
"channel": "next",
|
|
237
|
+
"tag": "v1.0.0-rc.12",
|
|
238
|
+
"contractBreakingDefects": 0,
|
|
239
|
+
"publishedAt": "2026-09-02T23:54:47.041Z",
|
|
240
|
+
"sourceCommit": "eb288bd3a0ef76dfbe630939d64e679437703bcb",
|
|
241
|
+
"artifactSha512": "sha512-Qpo7W9VmGvKmnX08Zr4NK15AWUuHRoMaYA9f0iAksGKxeQ3gGP2ntLv3xSwth1uJnknoT1XMMS7CtTFdy5i/eg==",
|
|
242
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/33695876968",
|
|
243
|
+
"provenance": "verified",
|
|
244
|
+
"liveCertification": "passed-release-bound-run"
|
|
245
|
+
},
|
|
246
|
+
{
|
|
247
|
+
"version": "1.0.0-rc.13",
|
|
248
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/32",
|
|
249
|
+
"status": "passed",
|
|
250
|
+
"channel": "next",
|
|
251
|
+
"tag": "v1.0.0-rc.13",
|
|
252
|
+
"contractBreakingDefects": 0,
|
|
253
|
+
"publishedAt": "2026-09-08T18:50:18.941Z",
|
|
254
|
+
"sourceCommit": "d6e310ce12fd3ee74db5bd40451f5af70056e3ce",
|
|
255
|
+
"artifactSha512": "sha512-EhKcjH44ZvRS1Y4POhs2xnPCmw0IFQo1trKf1gusYaFMkLthwXND0+ZEkfzG25JFCCf7kfOAZabDp35SoPnubw==",
|
|
256
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/34263591864",
|
|
257
|
+
"provenance": "verified",
|
|
258
|
+
"liveCertification": "passed-release-bound-run"
|
|
259
|
+
},
|
|
260
|
+
{
|
|
261
|
+
"version": "1.0.0-rc.14",
|
|
262
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/31",
|
|
263
|
+
"status": "passed",
|
|
264
|
+
"channel": "next",
|
|
265
|
+
"tag": "v1.0.0-rc.14",
|
|
266
|
+
"contractBreakingDefects": 0,
|
|
267
|
+
"publishedAt": "2026-09-20T12:47:20.607Z",
|
|
268
|
+
"sourceCommit": "d1c5abcc6d1fa482c211c4327c86f592663d559b",
|
|
269
|
+
"artifactSha512": "sha512-UMu4JFUNs/kOTJETZSJMMvVYlo+a4MKG6PoSvE313QxMEqK1z420AXVgPOGJ8jj2jp6mQud/Rvc7+ixKyH/4og==",
|
|
270
|
+
"workflowUrl": "https://github.com/Zhivex/zhivex-harness/actions/runs/35510484383",
|
|
271
|
+
"provenance": "verified",
|
|
272
|
+
"liveCertification": "passed-release-bound-run"
|
|
159
273
|
}
|
|
160
274
|
],
|
|
161
275
|
"blockers": [
|
|
@@ -166,6 +280,7 @@
|
|
|
166
280
|
"evidence": [
|
|
167
281
|
"contracts/public-api.json",
|
|
168
282
|
"fixtures/contracts/v1/doctor.json",
|
|
283
|
+
"fixtures/contracts/v1/init.json",
|
|
169
284
|
"fixtures/contracts/v1/observational-documents.json",
|
|
170
285
|
"fixtures/contracts/v1/providers.json",
|
|
171
286
|
"fixtures/contracts/v1/run-result.json",
|
|
@@ -215,8 +330,24 @@
|
|
|
215
330
|
"tests/state-backup.test.ts"
|
|
216
331
|
]
|
|
217
332
|
},
|
|
218
|
-
{
|
|
219
|
-
|
|
333
|
+
{
|
|
334
|
+
"id": "representative-evaluation-evidence",
|
|
335
|
+
"status": "closed",
|
|
336
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/31",
|
|
337
|
+
"evidence": [
|
|
338
|
+
"docs/LIVE_CERTIFICATION.md",
|
|
339
|
+
"evaluations/representative-matrix.json"
|
|
340
|
+
]
|
|
341
|
+
},
|
|
342
|
+
{
|
|
343
|
+
"id": "security-review",
|
|
344
|
+
"status": "closed",
|
|
345
|
+
"issue": "https://github.com/Zhivex/zhivex-harness/issues/32",
|
|
346
|
+
"evidence": [
|
|
347
|
+
"docs/RC14_DELEGATED_SECURITY_DECISION.md",
|
|
348
|
+
"security-reviews/rc14-review.json"
|
|
349
|
+
]
|
|
350
|
+
},
|
|
220
351
|
{
|
|
221
352
|
"id": "gemini-ga-decision",
|
|
222
353
|
"status": "closed",
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Efficiency implementation evidence — 2026-09-08
|
|
2
|
+
|
|
3
|
+
> Historical snapshot: versions, findings and measurements below describe the recorded run, not the current checkout. See the [report index](README.md) for follow-up work and the [documentation index](../README.md) for maintained guides.
|
|
4
|
+
|
|
5
|
+
## SDK patch integration
|
|
6
|
+
|
|
7
|
+
The measured checkout pinned published Core 1.13.0, Agents 1.3.1 and OpenAI 0.11.2,
|
|
8
|
+
including the Core override and lockfile. The isolated three-response diagnostic
|
|
9
|
+
now reports 300 input tokens for 300 measured tokens. Consumer regressions exercise
|
|
10
|
+
persisted and unpersisted streams, compactor usage, per-step budgets, real approval
|
|
11
|
+
resumes and synthetic assistant serialization through mocked Responses HTTP.
|
|
12
|
+
|
|
13
|
+
The local usage-accounting correction and compacted-history transport projection
|
|
14
|
+
have been removed. The SDK preserves complete interaction groups and assistant
|
|
15
|
+
roles; the harness no longer drops orphan observations or changes summary roles.
|
|
16
|
+
The new runtime identity requires older paused runs to use their matching artifact.
|
|
17
|
+
Those checks are installed-package evidence. The subsequent live
|
|
18
|
+
[repeated-cohort comparison](./EXTERNAL_COMPARISON.md#sdk-upgrade-reevaluation--2026-09-08)
|
|
19
|
+
resolved 2/5 cases for Zhivex and 3/5 for mini-SWE-agent, with 19.4% fewer Zhivex
|
|
20
|
+
input tokens than its previous run. It does not establish general superiority.
|
|
21
|
+
The sections below describe the earlier development evidence and task creation.
|
|
22
|
+
|
|
23
|
+
Upgrade validation passed: 442 tests, documentation and contract checks, typecheck,
|
|
24
|
+
migrations, deterministic evaluations and local benchmark checks. MCP loopback
|
|
25
|
+
checks required execution outside the sandbox; both MCP smokes, real OCI execution
|
|
26
|
+
and installed-package smoke passed with that permission. The isolated SDK diagnostic
|
|
27
|
+
also passed with the published dependency versions and no provider calls.
|
|
28
|
+
|
|
29
|
+
## Follow-up: command selection and user corrections
|
|
30
|
+
|
|
31
|
+
Configured OCI executable allowlists now appear as enums in all four command
|
|
32
|
+
tool schemas, including batch and verified-edit transactions. Descriptions expose
|
|
33
|
+
the configured choices and the Python module invocation for pytest when Python is
|
|
34
|
+
allowed. This does not broaden process permissions or guarantee module installation;
|
|
35
|
+
the execution session still enforces policy and exact approval. Calls to the public
|
|
36
|
+
factory without a configuration retain the legacy schema and direct the model to
|
|
37
|
+
`environment_status`.
|
|
38
|
+
|
|
39
|
+
Compaction strategy `bounded-evidence-v2` retains three bounded user corrections
|
|
40
|
+
separately from assistant excerpts. A regression runs five compactions with twenty
|
|
41
|
+
assistant messages between each, preserving the correction inside the same 4,000
|
|
42
|
+
character summary ceiling. This is deterministic continuity evidence, not a new
|
|
43
|
+
external benchmark result. Previous paused runs require their matching artifact.
|
|
44
|
+
|
|
45
|
+
The SDK team backlog in Notion's Zhivex area contains these verified-created tasks:
|
|
46
|
+
|
|
47
|
+
- [Persisted stream usage duplication](https://app.notion.com/p/3d5777b104f6810a8e46df7186bf5e21):
|
|
48
|
+
reproducible offline using `bun --no-env-file run scripts/diagnose-sdk-compaction.ts`.
|
|
49
|
+
Core 1.11.0 reports 500 input tokens for three measured responses of 100 when a
|
|
50
|
+
store and compaction are enabled. Without a store the isolated control reports 300.
|
|
51
|
+
- [Synthetic assistant serialization](https://app.notion.com/p/3d5777b104f6818db026dea3738f9b5b):
|
|
52
|
+
OpenAI 0.10.0 transport contract issue; the earlier harness projected summaries
|
|
53
|
+
into user context.
|
|
54
|
+
- [Approval/tool group boundaries](https://app.notion.com/p/3d5777b104f6814b80f2d46269c9453c):
|
|
55
|
+
code inspection and transport regression; the SDK task explicitly requires an
|
|
56
|
+
integrated approval/compaction reproduction before implementing an upstream fix.
|
|
57
|
+
|
|
58
|
+
At creation all three were tagged SDK-Typescript, priority Alta, status Sin empezar.
|
|
59
|
+
Their SDK fixes are now integrated as described above; this historical record does
|
|
60
|
+
not assert their current Notion workflow status.
|
|
61
|
+
|
|
62
|
+
Follow-up validation: 429 tests passed, plus documentation, stable contracts,
|
|
63
|
+
typecheck, migrations, deterministic evaluations and local benchmark checks.
|
|
64
|
+
The sandbox blocked the loopback MCP listener; MCP interoperability (controlled
|
|
65
|
+
and official SDK), real Docker OCI and installed-package smoke subsequently passed
|
|
66
|
+
with the required local execution permission. The isolated SDK diagnostic was also
|
|
67
|
+
typechecked and executed without provider calls. No new external score is claimed.
|
|
68
|
+
|
|
69
|
+
Implemented on top of `0a7b69dddaa70a54b6b5b4097691c8112d4d76eb` as an
|
|
70
|
+
uncommitted development change. Compact source-report digests and aggregate
|
|
71
|
+
measurements are recorded in [the baseline](../../benchmarks/baselines/efficiency-2026-09-08.json).
|
|
72
|
+
Full reports remain local under `results/efficiency/2026-09-08/`.
|
|
73
|
+
|
|
74
|
+
| Measurement (p50) | Before | After |
|
|
75
|
+
| --- | ---: | ---: |
|
|
76
|
+
| Three searches, 10,000 small files | 1,136.79 ms | 492.41 ms |
|
|
77
|
+
| `searchMany`, 10,000 small files | 941.46 ms | 232.24 ms |
|
|
78
|
+
| Digest inventory, 10,000 small files | 970.63 ms | 245.74 ms |
|
|
79
|
+
| Reused topology inventory, 100 nested groups / 1,000 files | 27.61 ms | 10.32 ms |
|
|
80
|
+
| OCI session acquisition, 1,000 source files | 417.04 ms | 193.16 ms |
|
|
81
|
+
|
|
82
|
+
Three repetitions per measurement, Bun 1.4.0 on local macOS ARM64, filesystem
|
|
83
|
+
cache not flushed. These are sequential development observations, not randomized
|
|
84
|
+
paired trials. The nested and OCI before/after measurements isolate later stages:
|
|
85
|
+
both already include concurrent content reads; OCI after additionally includes
|
|
86
|
+
batched index freshness and snapshot construction. No model latency or human
|
|
87
|
+
approval time is represented by these workspace numbers. No p99 claim is warranted.
|
|
88
|
+
|
|
89
|
+
Validation: full `bun run check` passed, including 414 tests, historical migrations,
|
|
90
|
+
deterministic evaluations, workspace/safe-fix benchmarks, MCP interoperability,
|
|
91
|
+
real Docker 29.7.2 OCI smoke, and the installed package smoke. Tests exercise batch
|
|
92
|
+
failure draining, early termination, deduplication, aggregate limits, symlinks,
|
|
93
|
+
Unicode pagination, changed file contents, repeated compaction and redaction.
|
|
94
|
+
The Stable declaration snapshot update reflects only TypeScript's reordered
|
|
95
|
+
`fulfilled`/`rejected` enum members, checked against declarations emitted from the
|
|
96
|
+
base checkout. It does not add or remove a public API member.
|
|
97
|
+
|
|
98
|
+
Reproduce workspace scale with `bun run benchmark:workspace:scale`, continuity with
|
|
99
|
+
`bun run evaluate:continuity`, and OCI with `bun run benchmark:oci`. See the
|
|
100
|
+
[benchmark guide](../../benchmarks/README.md) for limits and competitive methodology.
|
|
101
|
+
|
|
102
|
+
Remaining research: persistent content indexing, semantic plans/constraint retention,
|
|
103
|
+
automatic model selection and cross-tool concurrency. Those mechanisms were not
|
|
104
|
+
introduced by this patch. External held-out agent evaluations still require a
|
|
105
|
+
selected competing implementation, dataset and matched runtime/model budgets; the
|
|
106
|
+
existing external-driver interface is available but no competitor was executed.
|
|
107
|
+
|
|
108
|
+
## External-pilot follow-up
|
|
109
|
+
|
|
110
|
+
The follow-up adds exact approved replacements to avoid generating full files,
|
|
111
|
+
reduces default read/search volume, preserves the objective through SDK-formatted
|
|
112
|
+
compaction, and fixes model-transport context at compaction boundaries. OCI's
|
|
113
|
+
inventory uses the snapshot byte budget independently of the model's file-read
|
|
114
|
+
limit, allowing large unchanged fixture files. The authority inventory and enforced
|
|
115
|
+
OCI tool list both include the new replacement tool.
|
|
116
|
+
|
|
117
|
+
One previously failing development task produced an officially accepted repair
|
|
118
|
+
with two compacted continuations after these fixes. That run subsequently exhausted
|
|
119
|
+
its input budget after already importing the patch; the frozen comparison ends on
|
|
120
|
+
the approved import receipt. This development result is not holdout evidence.
|
|
121
|
+
See [the external comparison](EXTERNAL_COMPARISON.md) for the independent selection.
|
|
122
|
+
|
|
123
|
+
## Qwen repair follow-up
|
|
124
|
+
|
|
125
|
+
The subsequent implementation adds exact-file batched searches, character-bounded
|
|
126
|
+
read output, earlier benchmark compaction, remaining-budget reminders and required
|
|
127
|
+
verification before native benchmark import. Command telemetry retains typed exit
|
|
128
|
+
codes and timeouts without raw logs. Qwen adapter 0.11.4 incorporates the upstream
|
|
129
|
+
streamed-usage fix. Validation passed 446 tests and the MCP, OCI and installed
|
|
130
|
+
package checks.
|
|
131
|
+
|
|
132
|
+
The repeated five-case comparison using `qwen3.8-flash` with thinking disabled
|
|
133
|
+
resolved 0/5 for Zhivex and 2/5 for mini-SWE-agent. Native input usage was 424,720
|
|
134
|
+
tokens: three attempts reached the budget, one failed argument validation and one
|
|
135
|
+
failed final verification (exit 4), preventing import. This is no demonstrated
|
|
136
|
+
competitive improvement. Model and harness changes are confounded, and the cases
|
|
137
|
+
are known development data. See the [full result and limitations](EXTERNAL_COMPARISON.md#qwen-flash-repair-reevaluation--2026-09-08).
|
|
138
|
+
|
|
139
|
+
## Navigation retention follow-up — 2026-09-09
|
|
140
|
+
|
|
141
|
+
The harness compactor previously ignored nested file references in batched search
|
|
142
|
+
and read outputs. Strategy `bounded-evidence-v3` now retains up to eight validated,
|
|
143
|
+
deduplicated path/digest/line references within the existing total summary budget.
|
|
144
|
+
New observations with a changed digest replace older locations for that file.
|
|
145
|
+
Current source must still be read before editing; historical locations do not
|
|
146
|
+
authorize edits or stand in for source content.
|
|
147
|
+
|
|
148
|
+
The [deterministic control](../../benchmarks/baselines/context-navigation-2026-09-09.json)
|
|
149
|
+
retains two locations through six successive compactions, compared with zero in
|
|
150
|
+
the archived v2 implementation. On this small fixture the summary grows from 225
|
|
151
|
+
to 553 characters, still below the 4,000-character cap. This is improved information
|
|
152
|
+
retention, not measured token savings. The intended benefit is avoiding repository
|
|
153
|
+
rediscovery after compaction; a fixed-model live comparison is still needed to
|
|
154
|
+
measure that benefit. Regression tests use actual workspace batched outputs and
|
|
155
|
+
cover redaction, stale references, clipped lines, hostile recalled metadata and
|
|
156
|
+
small summary budgets. The local suite passes 456 tests.
|