@heyramzi/social 1.0.0 → 1.1.0

package/README.md

@@ -1,15 +1,34 @@
 # @heyramzi/social
 
-The heuristics for scripts and posts, written once. CutKit's script check and UpSys's Jev loop
-are different apps with their own copy, weights and contracts, and both import their rules from
-here so a rule changes in one place.
+Content evaluation, written once. CutKit's script check and UpSys's Jev loop are different apps
+with their own weights and contracts, and both import what's the same from here, so a rule
+changes in one place.
 
 | Entry | What it holds | Read by |
 | --- | --- | --- |
 | `./story` | Checks with no model: `openerFault` (greeting, hidden point, scene first), `hedgeScan`, `flatRun`, `HOOK_MAX_WORDS` | CutKit's offline lint, `./shorts` |
-| `./questions` | `storyQuestions({ field, piece, reader })`: the story questions a judge model answers, in the surface's own words | CutKit's check, Jev's `short`, `opening`, `post` and `x` surfaces |
+| `./questions` | `storyQuestions({ field, piece, reader })`: the story questions a judge model answers, in the surface's own words | CutKit's check, Jev's `opening`, `linkedin`, `x` and `reddit` surfaces |
 | `./hooks` | One regex per hook-library shape, and the banned shape | `./shorts` |
 | `./shorts` | `matchHook` and `auditShorts`, the skeleton of a Short | `heyramzi-slop --register=shorts` in `@heyramzi/lint` |
+| `./jev` | The wire shape of a question (`questionBody`), the `Answer` type, and `askJev`: one call that takes the key, model, state and questions, and returns the answers | CutKit's check, UpSys's `tools/social` client |
+| `./grade` | A trained model plus Jev's answers to a grade: `gradeAnswers` (grade 0 to 10, confidence, the three weakest answers with their fixes, the nearest winners, the pass gate), `encode`, `predict`, `rankOn`, `letterGrade`, `PASS_GRADE` | UpSys's loop, training and draft grading, CutKit's letter |
+
+`./jev` reads no env and keeps no cache. The caller passes the key, the model and, if it wants
+one, `fetch` and `retries` (a dropped connection, a 429, a 529 or a 5xx waits 0.5s, 1s, 2s and
+on). Every call gives up after 60 seconds. It throws when a question comes back unanswered, so
+nobody grades a partial set. A disk or database cache wraps it in the app, keyed on
+`questionBody`.
+
+`./grade` is pure. Training reads the same `encode` and `predict` that grade a draft, so a model
+grades the way it was fitted.
+
+Every evaluator grades on one scale. `rankOn(ruler, value)` is the share of a sorted corpus
+ruler a value beats, in tenths, and `letterGrade` reads that as A+ to F with 8 as the B- bar.
+So a B- beats 80% of its corpus, whether it's a Short, a LinkedIn post or a Reddit reply. A
+trained model's ruler holds its predicted probabilities. CutKit's check maps answers to
+hand-written items with fixed weights, so its ruler holds the craft scores of its corpus
+scripts (`pnpm eval:ruler` in cutkit). Training (logistic regression, the corpus rows) stays in
+UpSys, because it reads local corpora.
 
 Fix copy never lives here. A question in the bank says what to ask; the app says what to tell
 the person when the answer drags the grade down.
@@ -17,8 +36,8 @@ the person when the answer drags the grade down.
 ## Adding a question
 
 Add it to `storyQuestions` with its `source`. It earns a place in a surface once
-`pnpm jev train <surface>` in UpSys keeps it, and in CutKit's check once it has done that on
-`short`. The `script` wording CutKit already ships is pinned in `questions.test.mjs`, because the
+`pnpm jev train <surface>` in UpSys keeps it, and in CutKit's check once its rating bench
+(`evals/rating`) keeps it. The `script` wording CutKit already ships is pinned in `questions.test.mjs`, because the
 judge's answers move when a word does.
 
 ## Calibration
@@ -28,6 +47,13 @@ over 5,089 punctuated Short transcripts from 23 creators, a sentence-rhythm rule
 a row at one length) fired on 30.5% of winners and 27.9% of the rest, so it isn't here. `flatRun`
 flags 57% of winners and 60% of the rest: CutKit shows it as a nudge, never a fail.
 
+The same day, a Jev discovery round asked every question in the bank on each surface. Only
+`villain` earned a place: on `post` (+0.0020 AUC, coefficient +0.16) and `x` (+0.0025, +0.12).
+`intro_order` helped `opening`'s ranking with a negative coefficient (W-order openings did worse
+there), and `stakes`, `visual_first`, `head_fake` and `atomic_share` added nothing or pointed
+backwards. They stay in the bank to test again. UpSys's `short` surface, whose song-lyric
+transcripts pulled its AUC to 0.52, is gone: CutKit's check grades Shorts.
+
 ```
 pnpm test
 ```

package/grade.d.ts

@@ -0,0 +1,75 @@
+import type { Answer, JevQuestion } from "#social/jev.js";
+
+export type Question = JevQuestion & {
+  /** Code-only name. Jev never sees it, so the instructions carry the whole meaning. */
+  id: string;
+  /** Where the question came from: a control-tested lever, Kallaway, a hook-library shape, a discovery round. */
+  source?: string;
+  /** Said to the writer when this answer is what's pulling the grade down. */
+  fix?: string;
+  /** Training checks fix direction against the learned coefficient because advice must counter the trait dragging the grade. */
+  fixPushes?: "more" | "less";
+  /** Answered and weighed so the other coefficients aren't skewed by a tactic we ban, but never offered as advice. */
+  control?: true;
+};
+
+export type Fit = {
+  mean: number[];
+  sd: number[];
+  coef: number[];
+  intercept: number;
+};
+
+export type Model<Surface extends string = string> = Fit & {
+  surface: Surface;
+  trainedAt: string;
+  questions: Question[];
+  features: string[];
+  /** Sorted predicted probabilities over the training corpus, the ruler a grade is read off. */
+  ruler: number[];
+  metrics: {
+    cvAuc: number;
+    testAuc: number | null;
+    ownAuc: number | null;
+    rows: number;
+    positives: number;
+  };
+};
+
+export type TrainedModel<Surface extends string = string> = Model<Surface> & {
+  winners: { creator: string; text: string; breakout: number; x: number[] }[];
+};
+
+export type Weak = {
+  question: string;
+  answer: string;
+  pull: number;
+  fix?: string;
+};
+
+export type Grade<Surface extends string = string> = {
+  surface: Surface;
+  grade: number;
+  confidence: number;
+  pass: boolean;
+  /** False when the model failed its held-out test: the grade is advice and can't block a ship. */
+  gates: boolean;
+  testAuc: number | null;
+  weakest: Weak[];
+  winners: { creator: string; text: string; breakout: number }[];
+};
+
+export declare const PASS_GRADE: number;
+export declare const MIN_CONFIDENCE: number;
+export declare const MIN_TEST_AUC: number;
+/** The 0 to 10 grade as A+ to F. 8 is the B- bar. */
+export declare function letterGrade(grade: number): string;
+/** A value's rank on a sorted ruler of corpus scores, as a 0 to 10 grade: the share it beats. */
+export declare function rankOn(ruler: number[], value: number): number;
+export declare function encode(questions: Question[], answers: Record<string, Answer>): number[];
+export declare function predict(fit: Fit, x: number[]): number;
+export declare function contributions(fit: Fit, x: number[]): number[];
+export declare function gradeAnswers<Surface extends string>(
+  model: TrainedModel<Surface>,
+  answers: Record<string, Answer>,
+): Grade<Surface>;

package/grade.js

@@ -0,0 +1,166 @@
+/**
+ * Jev's answers to a grade on a trained model's corpus ruler. Pure. Training reads the same
+ * encode and predict, so a model grades the way it was fitted.
+ */
+
+/** The grade a piece needs before it ships. */
+export const PASS_GRADE = 8;
+/** Under this confidence Jev is closer to guessing, so a gating model won't pass the piece. */
+export const MIN_CONFIDENCE = 0.6;
+/** Only a model that ranks unseen winners above controls can block shipping. The rest advise. */
+export const MIN_TEST_AUC = 0.6;
+
+/**
+ * The grade times ten, so the bar of 8 reads B-. Every grade comes off `rankOn`, so a B-
+ * beats 80% of its corpus on any surface.
+ */
+const LETTERS = [
+  [97, "A+"],
+  [93, "A"],
+  [90, "A-"],
+  [87, "B+"],
+  [83, "B"],
+  [80, "B-"],
+  [77, "C+"],
+  [73, "C"],
+  [70, "C-"],
+  [67, "D+"],
+  [63, "D"],
+  [60, "D-"],
+];
+
+/** @param {number} grade 0 to 10 */
+export function letterGrade(grade) {
+  const score = Math.round(grade * 10);
+  return LETTERS.find(([floor]) => score >= floor)?.[1] ?? "F";
+}
+
+/**
+ * The share of a sorted corpus ruler a value beats, as a 0 to 10 grade.
+ * @param {number[]} ruler sorted ascending
+ * @param {number} value
+ */
+export function rankOn(ruler, value) {
+  if (!ruler.length) return 0;
+  const below = ruler.filter((r) => r < value).length;
+  return Number(((below / ruler.length) * 10).toFixed(1));
+}
+
+/** A score answer as a fraction of the scale; no answer sits mid scale. */
+function encodeScore(levels, a) {
+  return (a?.score ?? (levels - 1) / 2) / (levels - 1);
+}
+
+/** One column per choice past the first (the baseline): its probability, else 1 for the pick. */
+function encodeChoice(criteria, a) {
+  return Object.keys(criteria)
+    .slice(1)
+    .map((k) => a?.probabilities?.[k] ?? (a?.choice === k ? 1 : 0));
+}
+
+function encodeOne(q, a) {
+  if (q.type === "noul") return [a?.noul ?? 0.5];
+  if (q.type === "score") return [encodeScore(q.criteria.length, a)];
+  return encodeChoice(q.criteria, a);
+}
+
+/** One number per feature, in the order a model's `features` lists them. */
+export function encode(questions, answers) {
+  return questions.flatMap((q) => encodeOne(q, answers[q.id]));
+}
+
+const sigmoid = (z) => 1 / (1 + Math.exp(-z));
+const standardized = (f, x, j) => ((x[j] - (f.mean[j] ?? 0)) / (f.sd[j] ?? 1)) * (f.coef[j] ?? 0);
+
+export function predict(f, x) {
+  return sigmoid(f.intercept + x.reduce((sum, _, j) => sum + standardized(f, x, j), 0));
+}
+
+/** Each feature's pull on the logit, against an average corpus row. */
+export function contributions(f, x) {
+  return x.map((_, j) => standardized(f, x, j));
+}
+
+function describe(a, q) {
+  if (!a) return "no answer";
+  if (q.type === "noul") return `yes ${Math.round((a.noul ?? 0) * 100)}%`;
+  if (q.type === "score") {
+    const levels = q.criteria;
+    const i = Math.max(0, Math.min(levels.length - 1, Math.round(a.score ?? 0)));
+    return `level ${i + 1} of ${levels.length}: "${levels[i]}"`;
+  }
+  return `picked "${a.choice}"`;
+}
+
+/** Each question's pull: the sum over its features, against an average corpus row. */
+function pullsOf(model, contrib) {
+  return model.questions.map((q) => {
+    const cols = model.features
+      .map((f, j) => [f, j])
+      .filter(([f]) => f === q.id || f.startsWith(`${q.id}=`));
+    return { q, pull: cols.reduce((sum, [, j]) => sum + (contrib[j] ?? 0), 0) };
+  });
+}
+
+/** Confidence that counts is the confidence on the answers doing the work. */
+function confidenceOf(pulls, answers) {
+  let weight = 0;
+  let sum = 0;
+  for (const { q, pull } of pulls) {
+    const c = answers[q.id]?.confidence;
+    if (c === undefined) continue;
+    weight += Math.abs(pull);
+    sum += Math.abs(pull) * c;
+  }
+  return Number((weight ? sum / weight : 1).toFixed(2));
+}
+
+/** The winners nearest the answers, with distance weighted by each feature's coefficient. */
+function nearestWinners(model, x) {
+  const z = (v, j) =>
+    ((v - (model.mean[j] ?? 0)) / (model.sd[j] ?? 1)) * Math.abs(model.coef[j] ?? 0);
+  const zx = x.map(z);
+  return model.winners
+    .map((w) => ({ w, d: w.x.reduce((sum, v, j) => sum + (z(v, j) - (zx[j] ?? 0)) ** 2, 0) }))
+    .toSorted((a, b) => a.d - b.d)
+    .slice(0, 3)
+    .map(({ w }) => ({ creator: w.creator, text: w.text, breakout: w.breakout }));
+}
+
+/**
+ * Where the answers rank, what drags them down, and the nearest winners. Under `MIN_TEST_AUC`
+ * a model only advises: `gates` false, `pass` true.
+ */
+export function gradeAnswers(model, answers) {
+  const x = encode(model.questions, answers);
+  const fitted = { mean: model.mean, sd: model.sd, coef: model.coef, intercept: model.intercept };
+  const p = predict(fitted, x);
+  const grade = rankOn(model.ruler, p);
+
+  const pulls = pullsOf(model, contributions(fitted, x));
+  const confidence = confidenceOf(pulls, answers);
+  const weakest = pulls
+    .filter((e) => e.pull < 0 && !e.q.control)
+    .toSorted((a, b) => a.pull - b.pull)
+    .slice(0, 3)
+    .map(({ q, pull }) => ({
+      question: q.instructions,
+      answer: describe(answers[q.id], q),
+      pull: Number(pull.toFixed(2)),
+      fix: q.fix,
+    }));
+
+  const testAuc = model.metrics.testAuc;
+  const gates = testAuc !== null && testAuc >= MIN_TEST_AUC;
+  const pass = !gates || (grade >= PASS_GRADE && confidence >= MIN_CONFIDENCE);
+  return {
+    surface: model.surface,
+    grade,
+    confidence,
+    pass,
+    gates,
+    testAuc,
+    weakest,
+    winners: nearestWinners(model, x),
+  };
+}

package/jev.d.ts

@@ -0,0 +1,29 @@
+/** One answer from Jev. Which fields are set depends on the question's type. */
+export type Answer = {
+  type?: string;
+  noul?: number;
+  score?: number;
+  choice?: string;
+  probabilities?: Record<string, number>;
+  confidence?: number;
+};
+
+export type JevQuestion =
+  | { type: "noul"; instructions: string; criteria?: undefined }
+  /** Ordered low to high, 2 to 10 levels. */
+  | { type: "score"; instructions: string; criteria: string[] }
+  /** Option key to its meaning. */
+  | { type: "choice"; instructions: string; criteria: Record<string, string> };
+
+export declare const JEV_URL: string;
+export declare function questionBody(q: JevQuestion): JevQuestion;
+
+export declare function askJev(request: {
+  key: string;
+  model: string;
+  state: unknown;
+  questions: Record<string, JevQuestion>;
+  url?: string;
+  fetch?: typeof fetch;
+  retries?: number;
+}): Promise<{ answers: Record<string, Answer>; model: string }>;

package/jev.js

@@ -0,0 +1,71 @@
+/**
+ * The wire shape of a Jev question and the one call that answers it. No env reads and no
+ * cache: the app passes its key and model and caches.
+ */
+
+/** TypeSafe's /v1/systemone behind Kilo, which bills our Kilo credits. */
+export const JEV_URL = "https://api.kilo.ai/api/gateway/typesafe/v1/systemone";
+
+/** A dead socket hung a run for 15 hours on 14 Oct 2026, so every call gives up at this. */
+const TIMEOUT_MS = 60_000;
+
+/**
+ * What Jev sees of a question. The app's id, fix and source stay out, so a cache key hashed
+ * from this survives a change to them.
+ */
+export function questionBody(q) {
+  return q.type === "noul"
+    ? { type: "noul", instructions: q.instructions }
+    : { type: q.type, instructions: q.instructions, criteria: q.criteria };
+}
+
+const retryable = (status) => status === 429 || status === 529 || status >= 500;
+const backoff = (attempt) => new Promise((done) => setTimeout(done, 500 * 2 ** attempt));
+
+async function post(request, attempt) {
+  const { key, model, state, questions, url = JEV_URL, fetch: send = fetch, retries = 0 } = request;
+  let res;
+  try {
+    res = await send(url, {
+      method: "POST",
+      headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
+      body: JSON.stringify({ state, model, questions }),
+      signal: AbortSignal.timeout(TIMEOUT_MS),
+    });
+  } catch (error) {
+    if (attempt >= retries) throw error;
+    return backoff(attempt).then(() => post(request, attempt + 1));
+  }
+  if (retryable(res.status) && attempt < retries)
+    return backoff(attempt).then(() => post(request, attempt + 1));
+  if (!res.ok) {
+    const detail = await res.text().catch(() => "");
+    throw new Error(`Jev answered ${res.status}: ${detail.slice(0, 200)}`);
+  }
+  return res.json();
+}
+
+/**
+ * @typedef {{
+ *   key: string,
+ *   model: string,
+ *   state: unknown,
+ *   questions: Record<string, ReturnType<typeof questionBody>>,
+ *   url?: string,
+ *   fetch?: typeof fetch,
+ *   retries?: number,
+ * }} JevCall
+ */
+
+/**
+ * Asks every question about one state. A drop, 429, 529 or 5xx retries after 0.5s, 1s, 2s and
+ * on. A missing answer throws.
+ * @param {JevCall} request
+ */
+export async function askJev(request) {
+  const reply = await post(request, 0);
+  for (const id of Object.keys(request.questions)) {
+    if (!reply.answers?.[id]) throw new Error(`Jev returned no answer for ${id}`);
+  }
+  return { answers: reply.answers, model: reply.model ?? request.model };
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@heyramzi/social",
-  "version": "1.0.0",
-  "description": "The social-media heuristics CutKit and UpSys share: hook shapes, the Shorts audit, the story checks with no model in them, and the story question bank a judge model answers",
+  "version": "1.1.0",
+  "description": "Content evaluation CutKit and UpSys share: hook shapes, the Shorts audit, story checks with no model, the story question bank, and the Jev call and trained-model grade",
   "license": "MIT",
   "repository": {
     "type": "git",
@@ -9,8 +9,12 @@
     "directory": "packages/social"
   },
   "files": [
+    "grade.js",
+    "grade.d.ts",
     "hooks.js",
     "hooks.d.ts",
+    "jev.js",
+    "jev.d.ts",
     "questions.js",
     "questions.d.ts",
     "shorts.js",
@@ -24,10 +28,18 @@
     "#social/*": "./*"
   },
   "exports": {
+    "./grade": {
+      "types": "./grade.d.ts",
+      "default": "./grade.js"
+    },
     "./hooks": {
       "types": "./hooks.d.ts",
       "default": "./hooks.js"
     },
+    "./jev": {
+      "types": "./jev.d.ts",
+      "default": "./jev.js"
+    },
     "./questions": {
       "types": "./questions.d.ts",
       "default": "./questions.js"