@littlebigbrain/create 0.5.0 → 0.6.0

package/templates/review/README.md

@@ -10,9 +10,12 @@ others to a person: one review item per instance.
 
 The LBB-run version is the starter workflow `review@v1`
 (`docs/reference/starter-workflows.md`, "Review"; design
-`docs/architecture/document-output-mapping.md`). This template is the same
-decision in TypeScript: the workflow in `workflow.ts`, the rules and the
-facts in `decision.ts`. It calls no model.
+`docs/architecture/document-output-mapping.md`). This template takes its
+rules, notes, counters and review items: the workflow in `workflow.ts`, the
+rules and the facts in `decision.ts`. It calls no model. Its eval runs both
+on the samples and on a fixture of every rule, and finds the same
+decisions, notes, facts and items
+([`../_evals/README.md`](../_evals/README.md)).
 
 ## How it runs
 
@@ -32,29 +35,36 @@ proposal is decided once: one that has a `fact_status` already is skipped.
    `rdfs:label`, a relation between two instances (the entry's `DENOTES`
    included), or a property value in its type. The ends of a relation must fit
    its domain and range, superclasses included.
-3. **The decision** (`decide`), in this order:
+3. **The decision** (`decide`), in this order, with LBB's notes:
    - **refused** when the graph cannot hold the statement: a predicate that
      is no relation or property of the records ontology (nor `rdf:type` or
-     `rdfs:label`), a value of another type, an object that is no instance,
-     an instance without a class, an end its relation does not take, a second
-     class, or a statement about a proposal;
+     `rdfs:label`), a value that does not take its property's type (LBB's
+     words: `` `vierzehn` is not a number``), an object that is no
+     instance, an instance without a class, an end its relation does not
+     take, a second class, or a statement about a proposal;
    - **to a person** when a rule of `alwaysReview` names it (`new_instance`:
      a statement whose subject or object the graph does not have yet; or a
-     predicate IRI), when it would replace another writer's value (a records
-     graph holds one value per property) or name, or when its confidence is
-     below the gate. The proposal goes into the **review item of its
-     instance** (`groupOf`): its subject; the object of a link whose subject
-     the graph has and whose object is new; and always the object of the
-     trace (`DENOTES`). One item holds every proposal of its instance that
-     the turn sends to a person, in order: the type, the label, values,
-     links, the trace (`fileItem`). Its title names the instance (`New
+     predicate IRI), when it would replace another writer's value or name, or
+     when its gate value is below the gate. The gate value is the
+     proposal's confidence, or the typing answer's probability `p` of a
+     source whose link is in the graph already (`gateValue`). The proposal
+     goes into the **review item of its instance** (`groupOf`): its subject;
+     the object of a link whose subject the graph has and whose object is
+     new; and always the object of the trace (`DENOTES`). One item holds every
+     proposal of its instance that the turn sends to a person, in order: the
+     type, the label, values, links, the trace (`fileItem`). The item is
+     LBB's: its key (`starter:<workflow>:instance:<hash of the
+     instance>:<hash of its proposals>`, `itemKey`), its title (`New
      Equipment M 7 Medienwagen`, or `Room Raum 0.12 Klassenraum: 1 fact to
-     check`), its rationale says why, and it has one evidence sample per
-     fact (what it states, its confidence, its line). Its facts are each
-     statement and its proposal's `fact_status: accepted`, and each sample
-     names the rows and triplets of its fact (`row`, `rows`, `triplets`), so
-     a person can leave facts out of the accept. The type of a new instance
-     is a required row: a person keeps it or dismisses the item;
+     check`), its rationale, its summary (`is a Equipment · label · M 7
+     Medienwagen · 1 link`), one status row per fact (the proposal's
+     `fact_status: accepted`) and one evidence sample per fact (what it
+     states, its gate value, its line, the other new instance a link waits
+     for). The type of a new instance is a required row: a person keeps it or
+     dismisses the item. After the status rows the item holds each
+     statement, so its accept commits the facts (design G8); each sample names
+     the rows and triplets of its fact (`rows`, `triplets`), so a person can
+     leave facts out of the accept;
    - **written** otherwise: a relation as an edge whose evidence names the
      proposal (`source_id` is the proposal's IRI, the text its evidence); a
      property as its value; a new instance's `rdf:type` as the instance of
@@ -64,11 +74,17 @@ proposal is decided once: one that has a `fact_status` already is skipped.
      name does not depend on which statement is written first. A label is a
      text: an IRI or an empty label is refused. A statement the graph holds
      already is not written again. Nothing is taken back, a name included: a
-     records instance keeps one.
+     records instance keeps one. When the graph refuses the commit, the
+     proposal is `refused` with `The graph refused the statement: …`.
 4. **Write** (`writeOwned`): the proposal's `fact_status` (`written`,
    `in_review` or `refused`) and `review_note`, owned by this workflow, and in
    the same commit the facts of a written statement.
 
+The state counts what LBB's version counts: `proposals`, `written` (a
+statement the graph held already included), `held_already`, `in_review`,
+`refused`, `new_instance` (proposals for a person that name a new instance)
+and `items`.
+
 ## Settings
 
 `defaults` in `workflow.ts`, those of LBB's `review`:
@@ -94,7 +110,8 @@ beside an LBB-run `review` on the same graph: they own the same properties.
   only; its sweep writes each kept fact, the type first, and dismisses the
   facts a person left out. A link to another new instance waits until that
   instance's item is accepted, and is dismissed with it. The template's item
-  commits its facts on accept, so such a link makes the other instance too.
+  commits its facts on accept (design G8), so such a link makes the other
+  instance too.
 - **One item per instance over time.** Its items follow the open proposals
   of an instance across messages: when they change, it supersedes the open
   item with a new one, and an accept of the old item fails (409). The
@@ -122,16 +139,26 @@ beside an LBB-run `review` on the same graph: they own the same properties.
   graph kinds: the records ontology's relations (the records `DENOTES`
   included), properties and classes, and `rdfs:label`. An instance is new
   when it has no class of the records ontology. It refuses any other
-  predicate with a note that says so (`owl:sameAs`, `lbb:denotes`).
+  predicate with LBB's note (`owl:sameAs`, `lbb:denotes`).
 - **Ownership by statement.** Its facts count as owned by statement, so it
   conflicts with no other workflow on the predicates it writes. The template
   owns `fact_status` and `review_note` only and writes the facts without
   owning them.
-- **The graph refuses.** It lets the commit refuse a statement (a domain, a
-  range) and marks the proposal `refused`. The template checks domain and
-  range before it writes; another refusal stops the turn.
-- **Status and preview.** Counters per outcome, the open items it follows,
-  and a preview of what it would decide.
+- **Status and preview.** The open items it follows in its status, and a
+  preview of what it would decide.
+
+The template differs from it in two places, and its eval names both:
+
+- **Domain and range.** LBB's review leaves them to the commit, and a
+  records graph commits such an edge, so it writes p11 of the samples (a
+  room as the equipment of a room). The template checks a relation's ends
+  against its domain and range, superclasses included, and refuses it, as
+  the answer key says.
+- **What it reads.** The template reads the graph with the commits of its
+  own turn; LBB's review reads the published generation. A statement written
+  earlier in the same turn (the label of a new instance written with its
+  type) counts as `held_already` in the template, and LBB's review writes it
+  again. The facts are the same.
 
 ## Samples and test
 

package/templates/review/decision.ts

@@ -1,7 +1,14 @@
 // The parts of review that need no I/O: what a proposal's statement is in the graph's records
-// ontology, the decision (write, a person, or refused), and the facts that write it. Adapted
-// from the LBB-run `review@v1` (database/crates/lbb-graph-query/src/review.rs).
-import { ENTITY_ID_KEY_SENTINEL, ENTITY_IRI_NAMESPACE, type Schemas } from "@littlebigbrain/client";
+// ontology, the decision (write, a person, or refused), and the facts that write it. The rules,
+// notes and item shapes are those of the LBB-run `review@v1`
+// (database/crates/lbb-graph-query/src/review.rs); its eval (sdk/templates/_evals) runs both on
+// the samples.
+import {
+  ENTITY_ID_KEY_SENTINEL,
+  ENTITY_IRI_NAMESPACE,
+  stableHashHex,
+  type Schemas,
+} from "@littlebigbrain/client";
 
 export const RDF_TYPE = "http://www.w3.org/1999/02/22-rdf-syntax-ns#type";
 export const RDFS_LABEL = "http://www.w3.org/2000/01/rdf-schema#label";
@@ -125,25 +132,76 @@ const ENTITY = /^https:\/\/littlebigbrain\.com\/e\/[0-9a-f]{32}$/;
 /** Whether an IRI names an instance of a records graph. */
 export const isEntity = (iri: string) => ENTITY.test(iri.trim());
 
-/** A value in its property's type, or null when it does not take the type. */
-export function coerce(raw: string, type: string): string | number | boolean | null {
+const NUMBER = /^[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/;
+const RFC3339 = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})$/;
+
+/** A value in its property's type, or why it does not take the type (LBB's `coerce_value`). */
+export function coerceValue(
+  raw: string,
+  type: string,
+): { value: string | number | boolean } | { error: string } {
   const text = raw.trim();
-  if (!text) return null;
-  const number = Number(text);
+  if (!text) return { error: "the value is empty" };
+  const cleaned = text.replaceAll(",", ".").replace(/\s+/g, "");
+  const number = NUMBER.test(cleaned) ? Number(cleaned) : NaN;
   switch (type) {
     case "i64":
-      return Number.isInteger(number) ? number : null;
+      if (!Number.isFinite(number)) return { error: `\`${text}\` is not a number` };
+      return Number.isInteger(number)
+        ? { value: number }
+        : { error: `\`${text}\` is not a whole number` };
     case "f64":
-      return Number.isFinite(number) ? number : null;
-    case "bool":
-      return text === "true" ? true : text === "false" ? false : null;
-    case "date_time":
-      return Number.isNaN(Date.parse(text)) ? null : text;
+      return Number.isFinite(number) ? { value: number } : { error: `\`${text}\` is not a number` };
+    case "bool": {
+      const lower = text.toLowerCase();
+      if (["true", "yes", "ja"].includes(lower)) return { value: true };
+      if (["false", "no", "nein"].includes(lower)) return { value: false };
+      return { error: `\`${text}\` is not true or false` };
+    }
+    case "datetime":
+    case "date_time": {
+      const value = new TextEncoder().encode(text).length === 10 ? `${text}T00:00:00Z` : text;
+      return RFC3339.test(value) && !Number.isNaN(Date.parse(value))
+        ? { value }
+        : { error: `\`${text}\` is not a date` };
+    }
+    case "keyword":
     default:
-      return text;
+      return { value: text };
   }
 }
 
+/** A value in its property's type, or null when it does not take the type. */
+export function coerce(raw: string, type: string): string | number | boolean | null {
+  const coerced = coerceValue(raw, type);
+  return "value" in coerced ? coerced.value : null;
+}
+
+/**
+ * The confidence the review gates a proposal on (LBB's `gate_value`): its `fact_confidence`,
+ * unless a source of its evidence is a link the graph holds already (`documents.map` writes
+ * `"link": {"written": true}` beside the typing answer's probability `p`); then that `p` when it
+ * is higher. A value that is not a finite number reads as 0.
+ */
+export function gateValue(confidence: number, evidence: string | undefined): number {
+  let value = Number.isFinite(confidence) ? Math.min(1, Math.max(0, confidence)) : 0;
+  const text = evidence?.trim();
+  if (!text?.startsWith("{")) return value;
+  let parsed: unknown;
+  try {
+    parsed = JSON.parse(text);
+  } catch {
+    return value;
+  }
+  const sources = (parsed as { sources?: unknown }).sources;
+  for (const source of Array.isArray(sources) ? sources : []) {
+    const s = source as { link?: { written?: unknown }; p?: unknown };
+    if (s?.link?.written === true && typeof s.p === "number" && Number.isFinite(s.p))
+      value = Math.max(value, Math.min(1, Math.max(0, Math.fround(s.p))));
+  }
+  return value;
+}
+
 /** Everything `classify` reads about a statement's entities and values. */
 export interface Seen {
   /** The entities the statement names, by IRI. */
@@ -152,11 +210,16 @@ export interface Seen {
   current?: string[];
 }
 
+/** Why an entity cannot be written: the graph has no instance of it and no proposal types it. */
+export const untyped = (e: Known) =>
+  `<${e.iri}> is not an instance of the graph, and no proposal of its rdf:type gives its class`;
+
 /**
- * The terms of a statement in a records graph, or why it cannot hold it:
- * `rdf:type` of a class, `rdfs:label`, a relation between two instances, a
- * property value. `proposalClass` is the class of the proposals: the review
- * writes no statement about a proposal.
+ * The terms of a statement in a records graph, or why it cannot hold it, in the order and words
+ * of LBB's review: `rdf:type` of a class, `rdfs:label`, a relation between two instances, a
+ * property value. `proposalClass` is the class of the proposals: the review writes no statement
+ * about a proposal. A relation's ends must fit its domain and range, superclasses included (LBB's
+ * review leaves that check to the commit).
  */
 export function classify(
   s: Statement,
@@ -169,26 +232,25 @@ export function classify(
     return refused(`The subject <${s.subject}> is not an instance IRI of this graph.`);
   const known = (iri: string): Known => seen.entities.get(iri) ?? { iri, exists: false };
   const subject = known(s.subject);
-  if (subject.class === proposalClass)
-    return refused("The review writes no statement about a proposal.");
-  const untyped = (e: Known) =>
-    `<${e.iri}> is not an instance of the graph, and no proposal of its rdf:type gives its class.`;
+  const aboutProposal = refused("The review writes no statement about a proposal.");
   const fits = (cls: string | undefined, allowed: string[]) =>
     !allowed.length || allowed.some((a) => ontology.ancestors.get(cls ?? "")?.has(a));
   if (s.predicate === RDF_TYPE) {
     const cls = ontology.classes.get(s.object.trim());
     if (!cls) return refused(`<${s.object}> is not a class of the graph's ontology.`);
+    if (subject.class === proposalClass) return aboutProposal;
     if (!subject.exists) return { kind: "type", class: cls, subject, holds: false };
     const holds = ontology.ancestors.get(subject.class ?? "")?.has(cls) ?? false;
     if (!holds)
       return refused(
-        `<${s.subject}> is an instance of ${subject.class} already; a records graph gives an instance one class.`,
+        `<${s.subject}> is an instance of ${subject.class ?? ""} already; a records graph gives an instance one class.`,
       );
     return { kind: "type", class: cls, subject, holds };
   }
   if (s.predicate === RDFS_LABEL) {
     if (!s.datatype || !s.object.trim())
       return refused("A label is a text; the object is an IRI or empty.");
+    if (subject.class === proposalClass) return aboutProposal;
     if (!subject.class) return refused(untyped(subject));
     return { kind: "label", label: s.object.trim(), subject };
   }
@@ -198,6 +260,7 @@ export function classify(
       return refused(
         `The relation ${relation.name} links two instances; the object is not an instance IRI.`,
       );
+    if (subject.class === proposalClass) return aboutProposal;
     const object = known(s.object.trim());
     for (const end of [subject, object]) if (!end.class) return refused(untyped(end));
     if (!fits(subject.class, relation.domain))
@@ -212,21 +275,21 @@ export function classify(
   }
   const property = ontology.properties.get(s.predicate);
   if (property) {
+    if (subject.class === proposalClass) return aboutProposal;
+    const coerced = coerceValue(s.object, property.type);
+    if ("error" in coerced) return refused(coerced.error);
     if (!subject.class) return refused(untyped(subject));
-    const value = coerce(s.object, property.type);
-    if (value === null)
-      return refused(`"${s.object}" is not a value of ${property.name} (${property.type}).`);
     return {
       kind: "property",
       field: property.name,
       type: property.type,
       subject,
-      value,
-      current: seen.current ?? [],
+      value: coerced.value,
+      current: subject.exists ? (seen.current ?? []) : [],
     };
   }
   return refused(
-    `<${s.predicate}> is not a relation or a property of the graph's records ontology (nor rdf:type or rdfs:label). The template writes with POST /v1/graph/commit, which takes those only.`,
+    `<${s.predicate}> is not a relation or a property of the graph's ontology (nor rdf:type or rdfs:label).`,
   );
 }
 
@@ -305,7 +368,7 @@ export function decide(
   if (t.kind === "label" && t.subject.exists && t.subject.label && t.subject.label !== t.label)
     return {
       status: "in_review",
-      why: `The subject is named ${t.subject.label} already: writing ${t.label} would rename it (a records entity has one name).`,
+      why: `The subject is named ${t.subject.label} already: writing ${t.label} would rename it.`,
     };
   if (!Number.isFinite(confidence) || confidence < rules.gate)
     return {
@@ -416,6 +479,8 @@ export interface Shown {
   kind: string;
   /** The fact in words, seen from its instance: `is a Room`, `HAS_EQUIPMENT → T 1`. */
   text: string;
+  /** A link to an instance the graph does not have: its name. The link waits for that instance. */
+  waitsFor?: string;
 }
 
 /** What an item shows of a fact about the instance `group`. */
@@ -425,21 +490,37 @@ export function shownOf(
   group: string,
 ): Shown {
   const name = (e: Known) => e.label?.trim() || e.iri.slice(e.iri.lastIndexOf("/") + 1);
+  const shown = (kind: string, text: string, waitsFor?: string): Shown => ({
+    kind,
+    text: clip(text, 200),
+    ...(waitsFor !== undefined ? { waitsFor } : {}),
+  });
   switch (t.kind) {
     case "type":
-      return { kind: "type", text: `is a ${t.class}` };
+      return shown("type", `is a ${t.class}`);
     case "label":
-      return { kind: "label", text: `label · ${t.label}` };
+      return shown("label", `label · ${t.label}`);
     case "property":
-      return { kind: "value", text: `${t.field} · ${t.value}` };
+      return shown("value", `${t.field} · ${t.value}`);
     case "relation":
-      if (isTrace(s.predicate)) return { kind: "source", text: name(t.subject) };
-      return t.object.iri === group && t.subject.iri !== group
-        ? { kind: "link", text: `${t.relation} ← ${name(t.subject)}` }
-        : { kind: "link", text: `${t.relation} → ${name(t.object)}` };
+      if (isTrace(s.predicate)) return shown("source", name(t.subject));
+      if (t.object.iri === group && t.subject.iri !== group)
+        return shown("link", `${t.relation} ← ${name(t.subject)}`);
+      return shown(
+        "link",
+        `${t.relation} → ${name(t.object)}`,
+        !t.object.exists && t.object.iri !== t.subject.iri ? name(t.object) : undefined,
+      );
   }
 }
 
+/** The key of an instance's review item: LBB's, from the instance and the proposals it holds. */
+export function itemKey(workflow: string, instance: string, proposals: string[]): string {
+  const group = stableHashHex(["review_group_v1", instance]).slice(0, 24);
+  const held = stableHashHex(["review_item_v1", ...[...proposals].sort()]).slice(0, 12);
+  return `starter:${workflow}:instance:${group}:${held}`;
+}
+
 /** The order of the kinds in an item: the type, the label, values, links, the trace. */
 export const KIND_ORDER = ["type", "label", "value", "link", "source"];
 

package/templates/review/workflow.test.ts

@@ -188,7 +188,16 @@ test("the samples get the statuses and the facts of the answer key", async () =>
   assert.deepEqual(edges, want);
   assert.equal(lbb.graph.entities.get(iris.get("room-012")!)!.properties.area_m2, 62.4);
   assert.equal(lbb.reviews.size, EXPECTED.review_items);
-  assert.deepEqual(state, { proposals: 14, written: 4, held: 1, in_review: 5, refused: 4 });
+  // LBB's counters: written counts a statement the graph held already too.
+  assert.deepEqual(state, {
+    proposals: 14,
+    written: 5,
+    held_already: 1,
+    in_review: 5,
+    refused: 4,
+    new_instance: 3,
+    items: 3,
+  });
   assert.equal(turns.length, 1);
   assert.ok(turns[0]!.keys.length <= 64);
   // Each step reports its phase as the LBB-run review words it.
@@ -252,12 +261,17 @@ test("one item per instance: a new instance's facts in order, its type required"
     ["p7", "p8", "p9"].map((id) => proposals.get(id)),
   );
   assert.equal(samples[0]!.line, "M 7 Medienwagen, 1 Stück");
-  // The type's write and its status are required; the link is not.
+  // The type's status and its write are required; the link is not. The status rows come
+  // first, one per fact, as in LBB's item; the statements after them.
   const required = (m7.facts as { required?: { row: number }[] }).required!.map((r) => r.row);
-  assert.deepEqual(required, [samples[0]!.rows, samples[0]!.row].flat());
+  assert.deepEqual(required, [samples[0]!.row, samples[0]!.rows].flat());
+  assert.deepEqual(
+    samples.map((sample) => sample.row),
+    [0, 1, 2],
+  );
   assert.equal(samples[2]!.required, undefined);
   assert.deepEqual(samples[2]!.triplets, [0]);
-  assert.match(m7.key!, /^starter:review:instance:[0-9a-f]{32}:[0-9a-f]{12}$/);
+  assert.match(m7.key!, /^starter:review:instance:[0-9a-f]{24}:[0-9a-f]{12}$/);
   assert.deepEqual((m7.mapping as { review: { new: boolean } }).review.new, true);
 });
 
@@ -294,11 +308,9 @@ test("accepting a review item writes its statement and the proposal's accepted",
   const rename = [...lbb.reviews.values()].find((r) =>
     r.title.startsWith("Equipment S 2.1 Spiegel"),
   )!;
-  assert.deepEqual(rename.facts!.entity_properties![0]!.name, "S 2.1 Spiegel, Kristallglas");
-  assert.equal(
-    rename.facts!.entity_properties![0]!.key,
-    `\u0000lbb-eid:${iris.get("s21")!.slice(-32)}`,
-  );
+  const renamed = rename.facts!.entity_properties!.find((row) => row.type === "Equipment")!;
+  assert.deepEqual(renamed.name, "S 2.1 Spiegel, Kristallglas");
+  assert.equal(renamed.key, `\u0000lbb-eid:${iris.get("s21")!.slice(-32)}`);
 });
 
 test("by default the gate decides a new instance: written at the gate, an item below it", async () => {
@@ -389,7 +401,15 @@ test("a message of more proposals than a turn decides continues in a new turn",
   assert.equal(turns.length, 2);
   assert.ok(turns.every((t) => t.keys.length <= 64));
   // The first value of each room is written; each later one would replace it, so it goes to a person.
-  assert.deepEqual(state, { proposals: 40, written: 2, held: 0, in_review: 38, refused: 0 });
+  assert.deepEqual(state, {
+    proposals: 40,
+    written: 2,
+    held_already: 0,
+    in_review: 38,
+    refused: 0,
+    new_instance: 0,
+    items: 4,
+  });
 });
 
 test("the decision reads the statement's terms as the LBB review does", () => {