@agentsfleet/orly 0.13.0 → 0.14.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -4
- package/docs/CHANGELOG.md +15 -0
- package/docs/LOGGING_STANDARD.md +23 -6
- package/docs/RULE_ENFORCEMENT.md +2 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -42,9 +42,19 @@ Works with Claude Code, Codex, OpenCode, and Amp.
|
|
|
42
42
|
|
|
43
43
|
---
|
|
44
44
|
|
|
45
|
-
## 0.
|
|
45
|
+
## 0.14: retained Jev answers explained
|
|
46
46
|
|
|
47
|
-
The
|
|
47
|
+
The [offline diagnosis](https://github.com/agentsfleet/orly/blob/main/evals/judgments/comparison/diagnosis/report.md) explains all 188 retained TypeSafe Jev attempts.
|
|
48
|
+
It separates native-answer agreement, confidence withholding, grading disagreements and unavailable replies.
|
|
49
|
+
Each attempt links to its saved inputs and a bounded explanation; unknown provider causes stay unknown.
|
|
50
|
+
|
|
51
|
+
The diagnosis uses existing receipts and sends no new provider requests.
|
|
52
|
+
All five proposed checks remain evaluation-only while fixture meanings and independent adoption evidence need review.
|
|
53
|
+
These contributor records live in the source checkout; installed packages link to the same published source.
|
|
54
|
+
|
|
55
|
+
## Measured judgment comparison
|
|
56
|
+
|
|
57
|
+
The source-checkout examiner checks unsupported completion claims.
|
|
48
58
|
It freezes the six runtime questions and evaluates five proposed checks separately.
|
|
49
59
|
Those checks cover obligations, evidence, production wiring, failure tests and finding resolution.
|
|
50
60
|
|
|
@@ -55,7 +65,9 @@ A successful run reports `passed: true`; seeded false completions remain failed
|
|
|
55
65
|
|
|
56
66
|
A separate explicit `jev-run.ts --live` command measures TypeSafe Jev answers against the frozen expectations.
|
|
57
67
|
Its retained report lists every measured answer, grading match, disagreement and unavailable response.
|
|
58
|
-
|
|
68
|
+
The saved replay reproduced every result with zero new requests at historical source revision [`751ee303`](https://github.com/agentsfleet/orly/tree/751ee303d0c97a403863f12f33adf8fbc20ff433).
|
|
69
|
+
Replay requires that revision's unchanged source inventory; changed bytes fail admission.
|
|
70
|
+
Autonomous agent improvement remains unmeasured.
|
|
59
71
|
All five candidates remain evaluation-only; uncertain and failed replies stay in the report.
|
|
60
72
|
An additional rehearsal uses the packed candidate and actual `agentsfleet` source in an isolated worktree.
|
|
61
73
|
See the [evaluation design](docs/architecture/judgment-evaluation.md) for commands, evidence and limits.
|
|
@@ -65,7 +77,8 @@ The [0.12 report](evals/release/release-report.md) retains the earlier package a
|
|
|
65
77
|
TypeSafe's Jev model supplies runtime advice through explicit `orly judge` requests; contributor measurement uses the separate live evaluator.
|
|
66
78
|
|
|
67
79
|
See [releases](https://github.com/agentsfleet/orly/releases) for published versions.
|
|
68
|
-
The install commands below fetch the published package
|
|
80
|
+
The install commands below fetch the published package.
|
|
81
|
+
The release workflow publishes 0.14.1 after its version change merges to `main`.
|
|
69
82
|
|
|
70
83
|
---
|
|
71
84
|
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,20 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.14.1 — 2026-10-11
|
|
4
|
+
|
|
5
|
+
The logging standard names the field for a second failure's code.
|
|
6
|
+
|
|
7
|
+
- **Role-prefixed codes** — §5 defines `cause_error_code`, `rollback_error_code` and `config_error_code`, and the `<role>_error_code` shape for new roles.
|
|
8
|
+
- **Rust binding** — §8A points a record that reports a second failure at §5.
|
|
9
|
+
|
|
10
|
+
## 0.14.0 — 2026-10-08
|
|
11
|
+
|
|
12
|
+
Retained TypeSafe Jev measurements now carry per-attempt explanations and explicit adoption limits.
|
|
13
|
+
|
|
14
|
+
- **Diagnosis** — accounts for all 188 saved attempts and separates native-answer agreement from confidence withholding and grading.
|
|
15
|
+
- **Evidence** — preserves original replies and labels; names uncertain fixture meanings and unavailable provider explanations.
|
|
16
|
+
- **Guidance** — links contributors to the source diagnosis while keeping all five proposed checks evaluation-only.
|
|
17
|
+
|
|
3
18
|
## 0.13.0 — 2026-10-07
|
|
4
19
|
|
|
5
20
|
A source-checkout examiner checks unsupported completion claims and grades actual TypeSafe Jev answers against frozen expectations.
|
package/docs/LOGGING_STANDARD.md
CHANGED
|
@@ -184,6 +184,21 @@ Embed the code as a struct field on the logger call: `log.err("event_name", .{ .
|
|
|
184
184
|
|
|
185
185
|
System-level failures with no domain meaning (e.g. raw `EACCES` from a syscall before we attribute it to a tenant operation) emit without `error_code`. The follow-up rule: if a syscall failure surfaces to a user, it gets attributed to a registry code at the boundary.
|
|
186
186
|
|
|
187
|
+
**One record, two failures.** Some records report a decision about another
|
|
188
|
+
failure: a gate that fails open, a rollback, a refused configuration. The
|
|
189
|
+
record's `error_code` names the decision. The failure behind it keeps its own
|
|
190
|
+
registry code in a field named for its role:
|
|
191
|
+
|
|
192
|
+
| Field | Carries the code of |
|
|
193
|
+
|---|---|
|
|
194
|
+
| `cause_error_code` | the failure the decision answers |
|
|
195
|
+
| `rollback_error_code` | a failure in the recovery that followed |
|
|
196
|
+
| `config_error_code` | the configuration read that could not be used |
|
|
197
|
+
|
|
198
|
+
A new role takes the same `<role>_error_code` shape. `error_code` always names
|
|
199
|
+
the record's own code, so never overwrite it with the cause's code. Never
|
|
200
|
+
embed a code inside a reason sentence either; the field carries it.
|
|
201
|
+
|
|
187
202
|
## §6 · PII / secret discipline
|
|
188
203
|
|
|
189
204
|
> [UNENFORCED → broad principle; no scan covers every log call site in the codebase for credential-shaped values]
|
|
@@ -324,12 +339,14 @@ arguments, never positional message formatting.
|
|
|
324
339
|
|
|
325
340
|
> [JUDGMENT → RUST-STYLE]
|
|
326
341
|
|
|
327
|
-
Every `warn` or `error` that maps to a registry code supplies
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
342
|
+
Every `warn` or `error` that maps to a registry code supplies `error_code =
|
|
343
|
+
code` per §5. A record that reports a second failure adds its
|
|
344
|
+
`<role>_error_code` per §5, for example `cause_error_code`. Hoist field
|
|
345
|
+
expressions into locals before the macro call. The log bridge duplicates field
|
|
346
|
+
expressions, and LLVM coverage instrumentation scores the dead copy; locals
|
|
347
|
+
preserve evaluation and coverage behavior. A trailing string message is §3's
|
|
348
|
+
optional `msg` and is always the last macro argument. Prefer fields alone when
|
|
349
|
+
they already explain the event.
|
|
333
350
|
|
|
334
351
|
```rust
|
|
335
352
|
let code = error_code::INTERNAL_DB_QUERY.as_str();
|
package/docs/RULE_ENFORCEMENT.md
CHANGED
|
@@ -6,7 +6,7 @@ regenerates this file and fails on any difference. To update it, run
|
|
|
6
6
|
|
|
7
7
|
| Rule document | enforced by | judged | acknowledged | classified | trigger |
|
|
8
8
|
|---|---|--:|--:|--:|---|
|
|
9
|
-
| `docs/LOGGING_STANDARD.md` | `LOG` → `logging.sh` | 10 | 13 |
|
|
9
|
+
| `docs/LOGGING_STANDARD.md` | `LOG` → `logging.sh` | 10 | 13 | 30/30 | mechanical |
|
|
10
10
|
| `docs/REST_API_DESIGN_GUIDELINES.md` | — | 0 | 0 | 0/67 | mechanical |
|
|
11
11
|
| `docs/SCHEMA_CONVENTIONS.md` | — | 0 | 0 | 0/17 | mechanical |
|
|
12
12
|
| `docs/DOCUMENTATION_RULES.md` | — | 0 | 0 | 0/22 | latent |
|
|
@@ -14,7 +14,7 @@ regenerates this file and fails on any difference. To update it, run
|
|
|
14
14
|
| `docs/CHANGELOG_VOICE.md` | — | 0 | 0 | 0/3 | latent |
|
|
15
15
|
| `docs/VERIFY_TIERS.md` | — | 0 | 0 | 0/4 | uncited |
|
|
16
16
|
| `docs/greptile-learnings/RULES.md` | — | 0 | 0 | 0/115 | mechanical |
|
|
17
|
-
| **corpus** | | | | **
|
|
17
|
+
| **corpus** | | | | **30/280** | |
|
|
18
18
|
|
|
19
19
|
## How to read this
|
|
20
20
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentsfleet/orly",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.1",
|
|
4
4
|
"description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary \u2014 for any coding agent.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|