@agentsfleet/orly 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -4
- package/docs/CHANGELOG.md +8 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -42,9 +42,19 @@ Works with Claude Code, Codex, OpenCode, and Amp.
|
|
|
42
42
|
|
|
43
43
|
---
|
|
44
44
|
|
|
45
|
-
## 0.
|
|
45
|
+
## 0.14: retained Jev answers explained
|
|
46
46
|
|
|
47
|
-
The
|
|
47
|
+
The [offline diagnosis](https://github.com/agentsfleet/orly/blob/main/evals/judgments/comparison/diagnosis/report.md) explains all 188 retained TypeSafe Jev attempts.
|
|
48
|
+
It separates native-answer agreement, confidence withholding, grading disagreements and unavailable replies.
|
|
49
|
+
Each attempt links to its saved inputs and a bounded explanation; unknown provider causes stay unknown.
|
|
50
|
+
|
|
51
|
+
The diagnosis uses existing receipts and sends no new provider requests.
|
|
52
|
+
All five proposed checks remain evaluation-only while fixture meanings and independent adoption evidence need review.
|
|
53
|
+
These contributor records live in the source checkout; installed packages link to the same published source.
|
|
54
|
+
|
|
55
|
+
## Measured judgment comparison
|
|
56
|
+
|
|
57
|
+
The source-checkout examiner checks unsupported completion claims.
|
|
48
58
|
It freezes the six runtime questions and evaluates five proposed checks separately.
|
|
49
59
|
Those checks cover obligations, evidence, production wiring, failure tests and finding resolution.
|
|
50
60
|
|
|
@@ -55,7 +65,9 @@ A successful run reports `passed: true`; seeded false completions remain failed
|
|
|
55
65
|
|
|
56
66
|
A separate explicit `jev-run.ts --live` command measures TypeSafe Jev answers against the frozen expectations.
|
|
57
67
|
Its retained report lists every measured answer, grading match, disagreement and unavailable response.
|
|
58
|
-
|
|
68
|
+
The saved replay reproduced every result with zero new requests at historical source revision [`751ee303`](https://github.com/agentsfleet/orly/tree/751ee303d0c97a403863f12f33adf8fbc20ff433).
|
|
69
|
+
Replay requires that revision's unchanged source inventory; changed bytes fail admission.
|
|
70
|
+
Autonomous agent improvement remains unmeasured.
|
|
59
71
|
All five candidates remain evaluation-only; uncertain and failed replies stay in the report.
|
|
60
72
|
An additional rehearsal uses the packed candidate and actual `agentsfleet` source in an isolated worktree.
|
|
61
73
|
See the [evaluation design](docs/architecture/judgment-evaluation.md) for commands, evidence and limits.
|
|
@@ -65,7 +77,8 @@ The [0.12 report](evals/release/release-report.md) retains the earlier package a
|
|
|
65
77
|
TypeSafe's Jev model supplies runtime advice through explicit `orly judge` requests; contributor measurement uses the separate live evaluator.
|
|
66
78
|
|
|
67
79
|
See [releases](https://github.com/agentsfleet/orly/releases) for published versions.
|
|
68
|
-
The install commands below fetch the published package
|
|
80
|
+
The install commands below fetch the published package.
|
|
81
|
+
The release workflow publishes 0.14.0 after its version change merges to `main`.
|
|
69
82
|
|
|
70
83
|
---
|
|
71
84
|
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.14.0 — 2026-10-08
|
|
4
|
+
|
|
5
|
+
Retained TypeSafe Jev measurements now carry per-attempt explanations and explicit adoption limits.
|
|
6
|
+
|
|
7
|
+
- **Diagnosis** — accounts for all 188 saved attempts and separates native-answer agreement from confidence withholding and grading.
|
|
8
|
+
- **Evidence** — preserves original replies and labels; names uncertain fixture meanings and unavailable provider explanations.
|
|
9
|
+
- **Guidance** — links contributors to the source diagnosis while keeping all five proposed checks evaluation-only.
|
|
10
|
+
|
|
3
11
|
## 0.13.0 — 2026-10-07
|
|
4
12
|
|
|
5
13
|
A source-checkout examiner checks unsupported completion claims and grades actual TypeSafe Jev answers against frozen expectations.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentsfleet/orly",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary \u2014 for any coding agent.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|