@littlebigbrain/create 0.3.0 → 0.4.0

package/templates/sections/README.md

@@ -0,0 +1,118 @@
+# sections
+
+Reads each Markdown file of a graph into its sections. A handbook gives one
+`Section` per heading, with its title, its level, its own text, the path of
+headings above it and its position. Each section has an edge `PART_OF` to
+its parent section, or to the file's `Document` when it has none. The
+edge's evidence is the heading line, and its source is the file's IRI with
+the section's lines: `https://littlebigbrain.com/e/41ab…#line=39,44` is
+lines 40 to 44 (RFC 5147,
+[evidence.md](../../../docs/reference/evidence.md#evidence-from-a-file)).
+
+The template has no LBB-run version. It is a named workflow with a
+**content trigger**: LBB sends it each upload of `text/markdown`,
+`text/x-markdown` and `text/plain` within seconds, without a wait for a
+publication ([workflows.md](../../../docs/reference/workflows.md#content-triggers)).
+A `text/plain` file is read when its name ends with `.md` or `.markdown`.
+
+## How it runs
+
+Each message holds one file. The first turn of a file:
+
+1. **Check.** Reads the file's name with a strong read, and the
+   derivation of the content for `sections@v1` and the settings
+   ([Derivations](../../../docs/reference/workflows.md#derivations)). A
+   content with a derivation is skipped, so the same bytes under a second
+   name are read once. So is a file whose name the settings' `match` does
+   not match, and a `text/plain` file without a Markdown name.
+2. **Plan.** Reads the bytes (`client.files.content`) and splits them at
+   their ATX headings (`#` to `######`, up to 3 spaces before them, a
+   space after them). A line in a fenced code block (three or more
+   backticks or tildes) is text. Closing hashes (`## Escalation ##`) are
+   dropped.
+3. **Represents.** With `represents`, when the file's name without its
+   extension is the label of exactly one instance of that class, writes
+   `Document REPRESENTS <instance>`. `support-handbook.md` represents the
+   guide `Support handbook`: the labels compare in lower case, with `-`,
+   `_` and spaces as one space. No match, or more than one, writes nothing
+   and proposes nothing.
+
+Then each turn writes at most `sectionsPerTurn` sections in owned writes of
+100. A section is one instance of the settings' class, keyed
+`<sha256>:<position>`:
+
+| Property | Holds |
+| --- | --- |
+| `title` | The heading's text |
+| `level` | 1 to 6 |
+| `text` | The lines after the heading up to the next heading, at most 8,000 characters |
+| `heading_path` | The titles from the top heading to its own, joined by ` > ` |
+| `position` | Its place in the file: 1, 2, … |
+
+The parent of a section is the nearest heading above it of a lower level.
+The text before the first heading, when it is not blank, is the section at
+position 0, of level 0, titled with the file's name without its extension.
+A section's lines run from its heading to its last line that is not
+blank. The last turn records the derivation: the sections, their count per
+level, the instance the file represents, and the commits of the writes.
+
+## Settings
+
+`defaults` in `workflow.ts`:
+
+| Setting | Default | Meaning |
+| --- | --- | --- |
+| `className` | `Section` | The class of the sections |
+| `represents` | none | Write `REPRESENTS` to the one instance of this class that the file's name names. The samples' guides are of the class `Guide` |
+| `match` | none | A regular expression on the file's name. Without it, every Markdown file |
+| `sectionsPerTurn` | 1,000 | Sections one turn writes, 1 to 5,000 |
+
+`trigger(settings)` names the workflow `sections`, with the content trigger
+`sections`. It owns the class, `PART_OF`, `REPRESENTS` when `represents`
+is set, and the five properties. `ontology(settings)` adds the class, the
+properties and `PART_OF`.
+
+The settings digest is the SHA-256 of the settings' canonical JSON: a
+change of any setting reads every file again.
+
+Two content workflows own one relation when no upload can reach both:
+`rows` and `sections`, both with `represents`, go on one graph, because no
+media type of one meets a media type of the other. A second workflow that
+writes `REPRESENTS` for a type that meets one of these, such as `text/*`, is
+refused with `409 workflow_owns_conflict`
+([starter-workflows.md](../../../docs/reference/starter-workflows.md#cycles-and-conflicts)).
+
+## Commands
+
+`setup.ts`, `worker.ts` and `try.ts` are the template's own:
+`npm run setup` creates the named workflow with
+`client.workflows.named.put(trigger())`, and `npm start` serves
+`createWorkflow(lbbActivities(client), client)`: the client reads the
+files' bytes and the derivations. `npm run try` creates the graph
+`try-sections`, commits the two sample guides, uploads the sample handbook
+and creates the workflow.
+
+## Limits
+
+- Setext headings (a line of `===` or `---` under the text) start no
+  section.
+- Another `className` in the settings writes the sections again as new
+  instances; the instances of the earlier class stay.
+- A file deleted later keeps its sections; their `PART_OF` edges to the
+  file go with its `Document`.
+
+## Samples and test
+
+`samples/support-handbook.md` is an invented support handbook: text before
+the first heading, headings of levels 1 to 3, a heading without text of
+its own, a fenced code block with a line that starts with `#`, closing
+hashes, and a `#` without a space. `samples/guides.json` holds two guides
+the file's name can name. `samples/expected.json` gives the 8 sections a
+correct run writes, each with its parent and its lines.
+
+`workflow.test.ts` runs the workflow without a server (`MockContent` holds
+the files and the derivations). It checks the answer key, the evidence of
+each section, `text/plain` with and without a Markdown name, a file
+without a heading, an empty file, a file of more sections than a turn
+writes, the skip by derivation, `represents` with zero, one and two
+matching instances, the phases, and the heading rules.

package/templates/sections/markdown.ts

@@ -0,0 +1,121 @@
+// The sections of a Markdown file, split at its ATX headings (`#` to
+// `######`, as CommonMark reads them). Each section holds its heading, its
+// own text up to the next heading, its parent (the nearest heading above it
+// of a lower level, else the document) and its lines, which its evidence
+// names as RFC 5147 line positions: `#line=39,44` is lines 40 to 44.
+
+/** One section of a file. */
+export interface Section {
+  /** Its place in the file: 0 for the text before the first heading, then 1, 2, … */
+  position: number;
+  /** 1 to 6; 0 for the text before the first heading. */
+  level: number;
+  title: string;
+  /** Its own text: the lines after its heading, up to the next heading. */
+  text: string;
+  /** The titles of its parents and its own, from the top. */
+  headingPath: string[];
+  /** The position of its parent section; null when its parent is the document. */
+  parent: number | null;
+  /** Its first and last line, from 1. */
+  from: number;
+  to: number;
+  /** The heading line, or the first line of the text before the first heading. */
+  quote: string;
+}
+
+/** The lines of a text: split at CRLF, LF and CR, without a byte order mark. A line break at the end adds no line. */
+export function splitLines(text: string): string[] {
+  const body = text.charCodeAt(0) === 0xfeff ? text.slice(1) : text;
+  if (!body) return [];
+  const lines = body.split(/\r\n|\n|\r/);
+  if (lines[lines.length - 1] === "") lines.pop();
+  return lines;
+}
+
+/** An ATX heading: up to 3 spaces, 1 to 6 `#`, then a space or the end. A closing run of `#` is dropped. */
+export function atxHeading(line: string): { level: number; title: string } | null {
+  const m = /^ {0,3}(#{1,6})(?:[ \t]+(.*?))?[ \t]*$/.exec(line);
+  if (!m) return null;
+  const title = (m[2] ?? "").replace(/(?:^|[ \t]+)#+$/, "").trim();
+  return { level: m[1]!.length, title };
+}
+
+/** The opening of a fenced code block: up to 3 spaces and 3 or more backticks or tildes. */
+const FENCE = /^ {0,3}(`{3,}|~{3,})/;
+
+/**
+ * The sections of a Markdown text. Lines inside a fenced code block are
+ * text, never headings. The text before the first heading, when it is not
+ * blank, is section 0 of level 0, titled `untitled` (the caller names it
+ * after the file). A section's lines run from its heading to its last line
+ * that is not blank before the next heading.
+ */
+export function markdownSections(text: string, untitled = ""): Section[] {
+  const lines = splitLines(text);
+  const headings: { line: number; level: number; title: string }[] = [];
+  let fence: { char: string; length: number } | null = null;
+  lines.forEach((line, i) => {
+    if (fence) {
+      const close = new RegExp(`^ {0,3}\\${fence.char}{${fence.length},}[ \\t]*$`);
+      if (close.test(line)) fence = null;
+      return;
+    }
+    const open = FENCE.exec(line);
+    if (open) {
+      fence = { char: open[1]![0]!, length: open[1]!.length };
+      return;
+    }
+    const heading = atxHeading(line);
+    if (heading) headings.push({ line: i + 1, ...heading });
+  });
+
+  /** Lines `from` to `to` (from 1) without the blank lines at either end. */
+  const span = (from: number, to: number) => {
+    while (from <= to && !lines[from - 1]!.trim()) from += 1;
+    while (to >= from && !lines[to - 1]!.trim()) to -= 1;
+    return { from, to };
+  };
+  const out: Section[] = [];
+  const firstHeading = headings[0]?.line ?? lines.length + 1;
+  const before = span(1, firstHeading - 1);
+  if (before.from <= before.to)
+    out.push({
+      position: 0,
+      level: 0,
+      title: untitled,
+      text: lines.slice(before.from - 1, before.to).join("\n"),
+      headingPath: [],
+      parent: null,
+      from: before.from,
+      to: before.to,
+      quote: lines[before.from - 1]!.trim(),
+    });
+
+  const open: Section[] = [];
+  headings.forEach((heading, n) => {
+    const end = (headings[n + 1]?.line ?? lines.length + 1) - 1;
+    const body = span(heading.line + 1, end);
+    while (open.length && open[open.length - 1]!.level >= heading.level) open.pop();
+    const parent = open[open.length - 1] ?? null;
+    const section: Section = {
+      position: n + 1,
+      level: heading.level,
+      title: heading.title,
+      text: body.from <= body.to ? lines.slice(body.from - 1, body.to).join("\n") : "",
+      headingPath: [...(parent?.headingPath ?? []), heading.title],
+      parent: parent?.position ?? null,
+      from: heading.line,
+      to: body.from <= body.to ? body.to : heading.line,
+      quote: lines[heading.line - 1]!.trim(),
+    };
+    open.push(section);
+    out.push(section);
+  });
+  return out;
+}
+
+/** The RFC 5147 fragment of a section's lines: `line=<from - 1>,<to>`. */
+export function lineFragment(section: Pick<Section, "from" | "to">): string {
+  return `line=${section.from - 1},${section.to}`;
+}

package/templates/sections/samples/expected.json

@@ -0,0 +1,82 @@
+{
+  "support-handbook.md": {
+    "sections": [
+      {
+        "position": 0,
+        "level": 0,
+        "title": "support-handbook",
+        "parent": null,
+        "lines": "line=0,2",
+        "why": "The text before the first heading, named after the file.",
+        "text": "This handbook says how the support team answers requests.\nThe team reviews it every quarter."
+      },
+      {
+        "position": 1,
+        "level": 1,
+        "title": "Support handbook",
+        "parent": null,
+        "lines": "line=3,6",
+        "heading_path": "Support handbook",
+        "text": "The team answers requests from customers by email and by chat."
+      },
+      {
+        "position": 2,
+        "level": 2,
+        "title": "Opening hours",
+        "parent": 1,
+        "lines": "line=7,11",
+        "heading_path": "Support handbook > Opening hours",
+        "text": "Requests are answered from 08:00 to 18:00 on weekdays.\nOn public holidays the chat is closed."
+      },
+      {
+        "position": 3,
+        "level": 2,
+        "title": "Response times",
+        "parent": 1,
+        "lines": "line=12,13",
+        "heading_path": "Support handbook > Response times",
+        "why": "A heading without text of its own: its lines are its heading line.",
+        "text": null
+      },
+      {
+        "position": 4,
+        "level": 3,
+        "title": "Email",
+        "parent": 3,
+        "lines": "line=14,17",
+        "heading_path": "Support handbook > Response times > Email",
+        "text": "Every email gets a first answer within one working day."
+      },
+      {
+        "position": 5,
+        "level": 3,
+        "title": "Chat",
+        "parent": 3,
+        "lines": "line=18,25",
+        "heading_path": "Support handbook > Response times > Chat",
+        "why": "The line with a hash in the fenced code block is text.",
+        "text": "A chat gets an answer within five minutes.\n\n```text\n# This line is in a code block: no heading.\n```"
+      },
+      {
+        "position": 6,
+        "level": 2,
+        "title": "Escalation",
+        "parent": 1,
+        "lines": "line=26,30",
+        "heading_path": "Support handbook > Escalation",
+        "why": "The closing hashes of the heading are dropped. A hash without a space after it starts no heading.",
+        "text": "A request that needs an engineer goes to the escalation queue.\n#This line is no heading: the hash has no space after it."
+      },
+      {
+        "position": 7,
+        "level": 1,
+        "title": "Glossary",
+        "parent": null,
+        "lines": "line=31,34",
+        "heading_path": "Glossary",
+        "text": "A ticket is one customer request, with every message about it."
+      }
+    ],
+    "represents": "Support handbook"
+  }
+}

package/templates/sections/samples/guides.json

@@ -0,0 +1,4 @@
+[
+  { "name": "Support handbook", "text": "How the support team answers requests." },
+  { "name": "Billing guide", "text": "How invoices and refunds work." }
+]

package/templates/sections/samples/support-handbook.md

@@ -0,0 +1,34 @@
+This handbook says how the support team answers requests.
+The team reviews it every quarter.
+
+# Support handbook
+
+The team answers requests from customers by email and by chat.
+
+## Opening hours
+
+Requests are answered from 08:00 to 18:00 on weekdays.
+On public holidays the chat is closed.
+
+## Response times
+
+### Email
+
+Every email gets a first answer within one working day.
+
+### Chat
+
+A chat gets an answer within five minutes.
+
+```text
+# This line is in a code block: no heading.
+```
+
+## Escalation ##
+
+A request that needs an engineer goes to the escalation queue.
+#This line is no heading: the hash has no space after it.
+
+# Glossary
+
+A ticket is one customer request, with every message about it.

package/templates/sections/samples.ts

@@ -0,0 +1,41 @@
+// The samples of sections, for `npm run try`: an invented support handbook
+// (samples/support-handbook.md) and two guides (samples/guides.json) that a
+// file's name can name.
+import { readFile } from "node:fs/promises";
+import { join } from "node:path";
+import type { LbbClient } from "@littlebigbrain/client";
+
+/** The class of the guides; set `represents: { class: GUIDE_CLASS }` to link the handbook to its guide. */
+export const GUIDE_CLASS = "Guide";
+
+/** Commit the guides, then upload the sample handbook to the client's graph. */
+export async function loadSamples(client: LbbClient, dir: string): Promise<void> {
+  const guides = JSON.parse(await readFile(join(dir, "guides.json"), "utf8")) as {
+    name: string;
+    text: string;
+  }[];
+  await client.ontology.evolve({
+    ops: [
+      { op: "add_entity_type", name: GUIDE_CLASS },
+      { op: "add_property", name: "text", value_type: "text" },
+    ],
+  });
+  await client.commit(
+    {
+      entity_properties: guides.map((guide) => ({
+        type: GUIDE_CLASS,
+        name: guide.name,
+        properties: { text: guide.text },
+      })),
+    },
+    { idempotencyKey: "sections-samples" },
+  );
+  console.log(`Committed ${guides.length} ${GUIDE_CLASS} records`);
+  const name = "support-handbook.md";
+  const bytes = await readFile(join(dir, name));
+  const uploaded = await client.files.upload(new Uint8Array(bytes), {
+    name,
+    contentType: "text/markdown",
+  });
+  console.log(`Uploaded ${name} (${uploaded.file.sha256.slice(0, 12)})`);
+}

package/templates/sections/setup.ts

@@ -0,0 +1,18 @@
+// Create the workflow on your graph: add the class, the properties and the
+// relation it writes, then create the named workflow whose content trigger
+// sends it each Markdown upload. Each call is safe to repeat.
+//
+//   LBB_URL=... LBB_API_KEY=... LBB_GRAPH=... npm run setup
+import { LbbClient, lbbActivities } from "@littlebigbrain/client";
+import { ontology, trigger } from "./workflow.js";
+
+const graph = process.env.LBB_GRAPH ?? "main";
+const client = new LbbClient({
+  baseUrl: process.env.LBB_URL ?? "http://127.0.0.1:7400",
+  apiKey: process.env.LBB_API_KEY,
+  graph,
+});
+await lbbActivities(client).evolve(ontology());
+const created = await client.workflows.named.put(trigger());
+console.log(`Created ${created.name} on the graph ${graph}.`);
+console.log(`Serve it with LBB_GRAPH=${graph} npm start.`);

package/templates/sections/try.ts

@@ -0,0 +1,32 @@
+// Try the workflow on its samples: create a graph (default try-sections),
+// add the sample guides, upload the sample handbook, and create the workflow
+// there. Then serve it with npm start and the same LBB_GRAPH: the content
+// trigger reads the files uploaded before it too.
+//
+//   LBB_URL=... LBB_API_KEY=... LBB_GRAPH=try-sections npm run try
+import { fileURLToPath } from "node:url";
+import { LbbClient, LbbError, lbbActivities } from "@littlebigbrain/client";
+import { loadSamples } from "./samples.js";
+import { ontology, trigger } from "./workflow.js";
+
+const graph = process.env.LBB_GRAPH ?? "try-sections";
+const client = new LbbClient({
+  baseUrl: process.env.LBB_URL ?? "http://127.0.0.1:7400",
+  apiKey: process.env.LBB_API_KEY,
+  graph,
+});
+try {
+  await client.graph(graph).create();
+  console.log(`Created the graph ${graph}`);
+} catch (error) {
+  // 409: the graph exists already.
+  if (!(error instanceof LbbError && error.status === 409)) throw error;
+}
+await lbbActivities(client).evolve(ontology());
+await loadSamples(client, fileURLToPath(new URL("../samples/", import.meta.url)));
+const created = await client.workflows.named.put(trigger());
+console.log(`Created ${created.name} on the graph ${graph}.`);
+console.log(`Serve it with LBB_GRAPH=${graph} npm start, and follow it with:`);
+console.log(
+  `  curl "$LBB_URL/v1/workflows/named/queue?graph=${graph}&name=${created.name}" -H "Authorization: Bearer $LBB_API_KEY"`,
+);

package/templates/sections/worker.ts

@@ -0,0 +1,24 @@
+// Serve the workflow: the worker claims the turns of its content trigger's
+// messages and runs them, one at a time. Run more workers for more at once.
+//
+//   LBB_URL=... LBB_API_KEY=... LBB_GRAPH=... npm start
+import { LbbClient, WorkflowWorker, lbbActivities } from "@littlebigbrain/client";
+import { createWorkflow } from "./workflow.js";
+
+const graph = process.env.LBB_GRAPH ?? "main";
+const client = new LbbClient({
+  baseUrl: process.env.LBB_URL ?? "http://127.0.0.1:7400",
+  apiKey: process.env.LBB_API_KEY,
+  graph,
+});
+// The client reads the files' bytes and the derivations.
+const workflow = createWorkflow(lbbActivities(client), client);
+const stop = new AbortController();
+process.once("SIGINT", () => stop.abort());
+process.once("SIGTERM", () => stop.abort());
+console.log(
+  `Serving ${workflow.name}@${workflow.version} for the graph ${graph}. Stop with Ctrl-C.`,
+);
+await new WorkflowWorker(client, [workflow], {
+  worker: process.env.LBB_WORKER_ID ?? `${workflow.name}-1`,
+}).run({ signal: stop.signal });