akm-cli 0.9.19-alpha.1 → 0.9.19-alpha.2

package/CHANGELOG.md

@@ -6,6 +6,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.9.19-alpha.2] - 2026-09-30
+
+### Changed
+
+- **Distill's grounding check vetoes only a score of 1; a 2 goes to a person.**
+  0.9.19-alpha.1 made a grounding score of 2 or less `quality_rejected`. A
+  calibration of the lesson judge on a local llama.cpp model (34 cases, 5
+  passes, 199 calls) scored all 4 off-subject lessons grounding 1 in nearly
+  every pass (one scored 2 in 4 of 5 full passes and 1 otherwise), and its only
+  false vetoes, 2 of 120 legitimate judgments, were one on-subject lesson that
+  adds advice beyond its source and scored 2. Temperature 0 did not make the
+  judge repeatable on that server either: scores moved by up to a point between
+  passes. A grounding score of 1 is therefore still `quality_rejected` (an
+  `improve_ledger` row and a `distill_invoked` event, no proposal, reason
+  `Off-subject for its source (grounding 1/5): …`), and a 2 is now
+  `review_needed`, a pending proposal for a person with the reason
+  `Borderline on grounding (2/5), routed to review: …`, even when the mean of
+  novelty and non-redundancy alone would pass it. A mean that alone rejects the
+  lesson stays `quality_rejected`, and a grounding score of 3 to 5 is
+  unchanged.
+
 ## [0.9.19-alpha.1] - 2026-09-30
 
 ### Added

package/dist/commands/improve/stage.js

@@ -235,15 +235,20 @@ export function buildReflectJudgePrompt(candidateContent, sourceContent, feedbac
  * `grounding` is scored with the other lesson criteria but left out of their
  * mean: a lesson about a different subject than its source reads as novel and
  * non-redundant, so the mean would pass it (or, in the review band, mint it as
- * a pending proposal). A score of {@link UNGROUNDED_MAX_SCORE} or less is a
- * rejection whatever the mean says (#999). Only a different subject scores that
- * low. A lesson that goes beyond or corrects its source is on its subject:
- * distill folds feedback into the lesson, and the judge is never shown it. A
- * contradiction of the source is the optional fidelity check's to send to a
- * human (`judgeAndQueue` in distill.ts), so the rubric must not pre-empt it.
+ * a pending proposal). The rubric reserves 1-2 for a different subject. A score
+ * of {@link UNGROUNDED_MAX_SCORE} or less is a rejection whatever the mean says
+ * (#999). A higher score up to {@link BORDERLINE_GROUNDING_MAX_SCORE} is only
+ * borderline: a lesson on its source's subject that advises beyond it has scored
+ * 2, and a score can move a point between runs (see `runQualityJudge`), so it
+ * goes to a person unless the mean alone already rejects it. A lesson that goes
+ * beyond or corrects its source is on its subject: distill folds feedback into
+ * the lesson, and the judge is never shown it. A contradiction of the source is
+ * the optional fidelity check's to send to a human (`judgeAndQueue` in
+ * distill.ts), so the rubric must not pre-empt it.
  */
 const GROUNDING_CRITERION = "grounding";
-const UNGROUNDED_MAX_SCORE = 2;
+const UNGROUNDED_MAX_SCORE = 1;
+const BORDERLINE_GROUNDING_MAX_SCORE = 2;
 const LESSON_JUDGE_CRITERIA = ["novelty", "nonRedundancy", GROUNDING_CRITERION];
 const REFLECT_JUDGE_CRITERIA = ["feedbackAlignment", "preservation", "quality"];
 /**
@@ -296,7 +301,12 @@ function judgeResponseSchema(keys) {
  * The quality judge. Fails closed: no runner, an unparseable verdict or a
  * provider failure never passes content. Bands: >= 3.5 pass, 2.5-3.5 review,
  * < 2.5 reject; a `grounding` score of {@link UNGROUNDED_MAX_SCORE} or less
- * rejects whatever the mean is. Temperature is pinned to 0 so verdicts do not flip.
+ * rejects whatever the mean is, and one of {@link BORDERLINE_GROUNDING_MAX_SCORE}
+ * routes a lesson the mean would pass to review (a mean that rejects stays a
+ * rejection). Temperature is set to 0, which reduces run-to-run variation but
+ * does not remove it: on some servers (llama.cpp batching, for one) the same
+ * request can score a point apart, so the routing rules are chosen with that
+ * margin in mind.
  */
 async function runQualityJudge(feature, config, prompt, keys, chat, options) {
     const resolved = !options.runnerSelectionFrozen && !options.llmRunner
@@ -339,6 +349,19 @@ async function runQualityJudge(feature, config, prompt, keys, chat, options) {
         };
     }
     const verdict = score >= 3.5 ? { pass: true } : score >= 2.5 ? { pass: false, reviewNeeded: true } : { pass: false };
+    // Borderline grounding is a person's call even when the mean would pass; a mean that rejects stays rejected.
+    if (criteria &&
+        grounding !== undefined &&
+        grounding <= BORDERLINE_GROUNDING_MAX_SCORE &&
+        (verdict.pass || verdict.reviewNeeded)) {
+        return {
+            pass: false,
+            reviewNeeded: true,
+            score,
+            reason: `Borderline on grounding (${grounding}/5), routed to review: ${reason}`,
+            criteria,
+        };
+    }
     return { ...verdict, score, reason, ...(criteria ? { criteria } : {}) };
 }
 /** Judge a proposed lesson (or knowledge promotion) against its source. */

package/docs/migration/release-notes/0.9.19.md

@@ -107,10 +107,13 @@ longer offers a TODO: verify placeholder: when feedback asks for something the
 asset lacks, reflect is told to leave the section unchanged. TODO lines earlier
 runs already put in your assets stay until you remove them; grep -rniE
 "TODO:? *verify" over the bundle finds them. Distill's quality judge now also
-scores whether a lesson is about what its source is about, and a lesson judged
-off-subject is dropped as quality_rejected (a ledger row and a distill_invoked
-event, no proposal) instead of passing or waiting in the queue as
-review_needed; the judge also reads the same first 3000 characters of the
+scores whether a lesson is about what its source is about. A lesson scored 1
+(off-subject) is dropped as quality_rejected (a ledger row and a
+distill_invoked event, no proposal) instead of passing or waiting in the queue
+as review_needed. A lesson scored 2 is borderline: it is queued as
+review_needed for you to decide, even when its novelty and non-redundancy
+alone would pass it, unless those alone would reject it, which stays
+quality_rejected. The judge also reads the same first 3000 characters of the
 source body, without frontmatter, that the generator saw. The shipped agent
 guidance changed with it: akm help agents now says to record feedback about an
 asset's content, that it helped or turned out wrong, stale or unhelpful, and

package/docs/reference/cli.md

@@ -3045,8 +3045,9 @@ Write the reason about the asset's content. Reflect treats it as an unverified
 report to investigate, not a fact to insert, and is told to leave the section
 unchanged when the reason asks for information the asset lacks. Distill's
 quality gate rejects a lesson that is off-subject for the asset it was
-distilled from. A command that failed (`akm show` erroring on the ref, say)
-says nothing about the asset, so it is not a reason to record against it.
+distilled from and sends a borderline one to review. A command that failed
+(`akm show` erroring on the ref, say) says nothing about the asset, so it is
+not a reason to record against it.
 
 ### task
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "akm-cli",
-  "version": "0.9.19-alpha.1",
+  "version": "0.9.19-alpha.2",
   "type": "module",
   "description": "akm (Agent Knowledge Manager) — a portable, local-first capability library for AI agents. Discover, load, share, and improve reusable skills, scripts, workflows, and knowledge across any shell-capable coding agent, including Claude Code, OpenCode, and Cursor.",
   "keywords": [