semiotic 3.7.4 → 3.8.0

package/spec/README.md

@@ -0,0 +1,152 @@
+# The IDID Portability Spec
+
+**Version 0.1 — library-neutral JSON Schemas for three chart-metadata primitives.**
+
+These schemas describe three pieces of metadata that make a chart *legible to an agent* and
+*receivable by a real audience*, independent of any rendering library:
+
+| Primitive | Schema | Answers |
+|---|---|---|
+| **Chart Capability** | [`v0.1/chart-capability.schema.json`](./v0.1/chart-capability.schema.json) | *What is this chart good at, and how does it survive mobile?* — declaratively, so a heuristic or an LLM can rank it against a dataset, goal, audience, and phone viewport without running a chart library. |
+| **Audience Profile** | [`v0.1/audience-profile.schema.json`](./v0.1/audience-profile.schema.json) | *Who is reading, and what is the org trying to grow?* — so a suggestion is calibrated to a real audience, not a generic baseline. |
+| **Annotation Provenance & Lifecycle** | [`v0.1/annotation-provenance.schema.json`](./v0.1/annotation-provenance.schema.json) | *Where did this note come from, how much do we trust it, and how does it age?* — so a claim *on* a chart carries its own evidence and expiry. |
+
+## Why a spec, and not just a library API
+
+A chart that an AI can pick correctly, that a screen-reader user can receive, and that carries
+its own provenance is more useful than one that only looks right. Those three properties are
+*metadata*, not pixels — and metadata is portable in a way a rendering engine is not. The value
+of writing them down as **library-neutral schemas** is that the ideas can travel: a Python
+notebook, a Vega-Lite spec, a BI tool, or a competing chart library can attach a capability
+descriptor, calibrate to an audience profile, or stamp an annotation with provenance *without
+depending on any particular renderer*. The schema is the contract; the renderer is one
+implementation of it.
+
+A practical consequence: a format adapter (Vega-Lite → chart, Mermaid → chart, dbt test →
+annotation) becomes an *export of these ideas* rather than just a parser. It does not merely
+reproduce a source format's appearance — it carries the capability/audience/provenance metadata
+the source never had. That is the difference between a chart that renders and a chart that
+*communicates*.
+
+## The `x-idid-status` field convention
+
+Public domain properties carry an `x-idid-status` annotation so a reader can tell what is real
+today from what the spec reserves for the future:
+
+- **`shipped`** — the field exists in a reference implementation's runtime types today
+  (Semiotic ships all v0.1 fields). Safe to depend on.
+- **`proposed`** — reserved by the program but not yet a runtime field. May change before it
+  ships. *(No v0.1 fields carry this status yet; the marker exists so additions are honestly
+  labeled rather than silently mixed in.)*
+- **`spec`** — a spec-housekeeping field (e.g. `specVersion`), not a domain field.
+
+Open string unions in the source types (e.g. `provenance.source`, which recognizes
+`user`/`ai`/`agent`/… but accepts any string) are expressed as `{ "type": "string" }` with an
+`x-idid-recognized-values` list and matching `examples`, rather than a closed `enum`. This is
+faithful to the runtime contract — recognized values are documented, but a consumer may pass its
+own label and it is preserved. Genuinely closed unions (`lifecycle.freshness`, `lifecycle.status`,
+`lifecycle.anchor`) use a strict `enum`.
+
+Chart capabilities may also include a `mobile` block. This is intentionally part of capability,
+not rendering props: it lets a custom chart, Vega-Lite binding, or recipe adapter state its mobile
+strategy, breakpoints, mark budget, touch target, label plan, and custom scene semantics without
+requiring the receiving library to execute the renderer.
+
+## Versioning
+
+The spec is versioned by directory (`v0.1/`). A `0.x` line may add `proposed` fields and tighten
+descriptions, but will not remove or repurpose a `shipped` field within the line. Breaking
+changes bump the minor (pre-1.0) or major (post-1.0) version into a new directory; old versions
+remain resolvable by their `$id`. A document may declare the version it targets via the optional
+`specVersion: "0.1"` property.
+
+The `$id` URIs (`https://semiotic.dev/spec/v0.1/…`) are stable identifiers, not fetch targets.
+The canonical files ship in the `semiotic` npm tarball and are exposed as
+`semiotic/spec/v0.1/<schema-name>` resource paths; they also remain available in this directory
+for repository consumers.
+
+## Validation
+
+These are plain [JSON Schema 2020-12](https://json-schema.org/draft/2020-12) documents. Validate
+with any compliant validator (ajv, `jsonschema`, etc.) — no library dependency is required, which
+is the point. A reference implementation also ships small dependency-free structural validators
+(see *The Vega-Lite binding*, below) for hosts that do not want to pull in a full schema engine.
+
+## The Vega-Lite binding (carrying IDID metadata on a portable spec)
+
+Vega-Lite is the closest thing the ecosystem has to a neutral chart-interchange format, but it has
+no place for capability, audience, or provenance metadata. The binding convention defines where
+IDID metadata rides on a Vega-Lite spec so the two travel together:
+
+### Capability and audience → `usermeta.idid`
+
+Vega-Lite reserves [`usermeta`](https://vega.github.io/vega-lite/docs/spec.html#top-level) for
+arbitrary application metadata that validators ignore. IDID metadata lives under a single
+namespaced key:
+
+```jsonc
+{
+  "mark": "bar",
+  "encoding": { /* … */ },
+  "usermeta": {
+    "idid": {
+      "specVersion": "0.1",
+      "capability": { "component": "BarChart", "rubric": { "familiarity": 5, "accuracy": 5, "precision": 4 }, "intentScores": { "compare-categories": 5, "rank": 4 } },
+      "audience": { "name": "Exec review", "receptionModality": "visual", "familiarity": { "BoxPlot": 2 } }
+    }
+  }
+}
+```
+
+A consumer reads `spec.usermeta.idid.capability` and routes it straight into a chart-suggestion
+engine; the spec still renders as ordinary Vega-Lite everywhere else.
+
+### Provenanced annotations → `usermeta.idid.annotations`
+
+Annotations ride verbatim — with their provenance/lifecycle blocks — under the same
+`usermeta.idid` key, composing with capability/audience. As a courtesy, any annotation with a
+representable shape (a threshold, a labelled callout) also emits a best-effort `rule`/`text` mark
+in an appended layer, so a plain Vega-Lite renderer still draws something:
+
+```jsonc
+{
+  "usermeta": {
+    "idid": {
+      "specVersion": "0.1",
+      "annotations": [
+        {
+          "type": "y-threshold",
+          "value": 1000,
+          "label": "SLA floor",
+          "provenance": { "source": "ai", "basis": "statistical-test", "confidence": 0.7, "createdAt": "2026-06-20T14:00:00Z" },
+          "lifecycle": { "ttlHint": "P7D", "status": "proposed", "anchor": "semantic" }
+        }
+      ]
+    }
+  },
+  "layer": [
+    { "mark": "line", "encoding": { /* the data */ } },
+    { "mark": "rule", "encoding": { "y": { "datum": 1000 } }, "usermeta": { "idid": { "role": "annotation-layer" } } }
+  ]
+}
+```
+
+A non-IDID Vega-Lite renderer draws the rule and ignores the metadata; an IDID-aware host reads
+the provenance/lifecycle off `usermeta.idid.annotations` and can dim a stale note, badge an
+AI-authored one with its confidence, or surface it in an accessible navigation tree. When no
+annotation is representable as a mark, the metadata still rides on `usermeta` and no empty layer
+is added.
+
+### Round-tripping
+
+The binding is designed for a tested supported single-view subset: a chart → Vega-Lite spec (with
+IDID metadata in `usermeta`) → back to a chart preserves the supported mark, data, and encoding
+semantics. Unsupported composition or chart semantics must return a typed refusal rather than a
+plausible fallback. IDID-enriched annotation layers recover their one base view, while arbitrary
+Vega-Lite layer composition remains deliberately unsupported.
+
+---
+
+*This directory is the canonical published copy of the schemas and is packed unchanged into npm.
+A reference implementation may embed copies for its own validators; those copies are kept
+byte-for-byte in sync with these files by a test, so this directory is always authoritative.*

package/spec/bindings/README.md

@@ -0,0 +1,35 @@
+# IDID spec bindings
+
+Reference implementations of the [IDID portability spec](../README.md) for
+specific host formats. **These are zero-dependency, library-neutral, and free to
+copy.** They exist to prove the spec is implementable without the reference
+library — the leading indicator that a standard is actually portable is someone
+implementing it in a stack that isn't the original.
+
+## `vega-lite.mjs`
+
+The IDID-over-Vega-Lite binding: carries chart capability, audience profile, and
+provenanced/lifecycled annotations on an ordinary Vega-Lite spec under
+`usermeta.idid` (which every Vega-Lite renderer ignores), so the spec and its
+meaning travel together.
+
+```js
+import {
+  attachIdid, readIdid, attachIdidAnnotations, readIdidAnnotations,
+  suggestionInputFromSpec,
+} from "./vega-lite.mjs"
+
+let spec = { mark: "bar", encoding: { /* … */ } }
+spec = attachIdid(spec, { capability, audience })          // ride under usermeta.idid
+spec = attachIdidAnnotations(spec, [provenancedNote])       // + a note with its evidence
+
+// An IDID-aware host reads it back and acts on it — e.g. routes the carried
+// capability + audience through any chart-suggestion engine:
+const input = suggestionInputFromSpec(spec)                 // { capability, audience } | null
+```
+
+The output is byte-compatible with Semiotic's TypeScript binding
+(`semiotic/experimental`: `unstable_attachIDID`, `unstable_readIDID`,
+`unstable_attachIDIDAnnotations`, …), so a spec enriched in one can be read in
+the other. Validate the carried metadata against the published
+[JSON Schemas](../v0.1).

package/spec/bindings/vega-lite.mjs

@@ -0,0 +1,112 @@
+/**
+ * IDID-over-Vega-Lite binding — standalone reference implementation.
+ *
+ * Zero dependencies, zero Semiotic imports. This is the proof that the IDID
+ * portability spec (../README.md, ../v0.1/*.schema.json) is implementable in
+ * any stack: it carries the three IDID primitives — chart capability, audience
+ * profile, and provenanced/lifecycled annotations — on an ordinary Vega-Lite
+ * spec, so the spec and its meaning travel together.
+ *
+ * Copy this file into your codebase and use it directly; it does not depend on
+ * Semiotic and never will. Semiotic ships a TypeScript equivalent
+ * (`semiotic/experimental`) that produces byte-compatible output, so a spec
+ * enriched here is read correctly there and vice versa.
+ *
+ * Convention (see ../README.md):
+ *   - capability + audience ride under `usermeta.idid.{capability,audience}`
+ *   - annotations ride verbatim under `usermeta.idid.annotations`, with
+ *     best-effort `rule`/`text` courtesy marks appended so a plain Vega-Lite
+ *     renderer still draws something.
+ */
+
+export const IDID_SPEC_VERSION = "0.1"
+const ANNOTATION_LAYER_ROLE = "annotation-layer"
+
+/** Read the IDID metadata block off a Vega-Lite spec, or undefined. */
+export function readIdid(spec) {
+  return spec && spec.usermeta ? spec.usermeta.idid : undefined
+}
+
+/**
+ * Attach a chart capability and/or an audience profile under `usermeta.idid`.
+ * Returns a new spec; the input is not mutated.
+ */
+export function attachIdid(spec, { capability, audience } = {}) {
+  const idid = { ...readIdid(spec), specVersion: IDID_SPEC_VERSION }
+  if (capability) idid.capability = capability
+  if (audience) idid.audience = audience
+  return { ...spec, usermeta: { ...(spec && spec.usermeta), idid } }
+}
+
+/**
+ * Carry provenanced/lifecycled annotations on the spec. They are stored
+ * verbatim under `usermeta.idid.annotations`; any annotation with a
+ * representable shape also emits a best-effort mark in an appended layer.
+ * Returns a new spec; the input is not mutated.
+ */
+export function attachIdidAnnotations(spec, annotations = []) {
+  const idid = { ...readIdid(spec), specVersion: IDID_SPEC_VERSION, annotations: [...annotations] }
+
+  const courtesy = annotations.map(annotationToMark).filter(Boolean)
+  if (courtesy.length === 0) {
+    return { ...spec, usermeta: { ...(spec && spec.usermeta), idid } }
+  }
+
+  const baseLayers = Array.isArray(spec && spec.layer) ? spec.layer : [viewOf(spec)]
+  const { mark, encoding, data, transform, usermeta, ...top } = spec || {}
+  return {
+    ...top,
+    usermeta: { ...(spec && spec.usermeta), idid },
+    layer: [...baseLayers, ...courtesy],
+  }
+}
+
+/** Read provenanced annotations back off a Vega-Lite spec. */
+export function readIdidAnnotations(spec) {
+  const idid = readIdid(spec)
+  return idid && Array.isArray(idid.annotations) ? idid.annotations : []
+}
+
+/**
+ * Extract the inputs an IDID-aware host routes through a chart-suggestion
+ * engine. Library-neutral: pass the result to whatever ranker you use
+ * (Semiotic's `suggestCharts`, your own heuristic, an LLM call). Returns null
+ * when the spec carries no IDID metadata.
+ */
+export function suggestionInputFromSpec(spec) {
+  const idid = readIdid(spec)
+  if (!idid) return null
+  return { capability: idid.capability, audience: idid.audience }
+}
+
+// ── helpers ──────────────────────────────────────────────────────────────────
+
+/** The view-level fields of a single-view spec, for use as a layer entry. */
+function viewOf(spec) {
+  const out = {}
+  if (!spec) return out
+  if (spec.mark) out.mark = spec.mark
+  if (spec.encoding) out.encoding = spec.encoding
+  if (spec.data) out.data = spec.data
+  if (spec.transform) out.transform = spec.transform
+  return out
+}
+
+/** Best-effort Vega-Lite mark for a representable annotation, else null. */
+function annotationToMark(annotation) {
+  const tag = { usermeta: { idid: { role: ANNOTATION_LAYER_ROLE } } }
+  switch (annotation && annotation.type) {
+    case "y-threshold":
+      return { mark: "rule", encoding: { y: { datum: annotation.value } }, ...tag }
+    case "x-threshold":
+      return { mark: "rule", encoding: { x: { datum: annotation.value } }, ...tag }
+    case "callout":
+    case "label":
+    case "text":
+      return typeof annotation.label === "string"
+        ? { mark: { type: "text", text: annotation.label }, ...tag }
+        : null
+    default:
+      return null
+  }
+}

package/spec/v0.1/annotation-provenance.schema.json

@@ -0,0 +1,127 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://semiotic.dev/spec/v0.1/annotation-provenance.schema.json",
+  "title": "IDID Annotation Provenance & Lifecycle",
+  "description": "Two orthogonal metadata blocks that attach to any chart annotation: WHERE it came from and how much to trust it (provenance), and HOW it ages and stands editorially (lifecycle). This is the primitive that distinguishes 'insight text beside a chart' from a provenance-tracked claim ON the chart — and the one that must survive being carried into another tool's annotation model. See ./README.md for the x-idid-status convention. Open string unions are expressed as type:string with a recognized-values list rather than a closed enum, faithful to the runtime types; closed unions use enum.",
+  "type": "object",
+  "$ref": "#/$defs/annotated",
+  "$defs": {
+    "annotated": {
+      "title": "Annotated",
+      "type": "object",
+      "description": "Wrapper concept: any annotation MAY carry an optional provenance block and an optional lifecycle block. Attaching neither leaves an annotation unchanged.",
+      "additionalProperties": true,
+      "properties": {
+        "provenance": { "$ref": "#/$defs/provenance" },
+        "lifecycle": { "$ref": "#/$defs/lifecycle" }
+      }
+    },
+    "provenance": {
+      "title": "Annotation Provenance",
+      "type": "object",
+      "description": "Where an annotation came from and how confident we should be in it.",
+      "additionalProperties": true,
+      "properties": {
+        "author": {
+          "type": "string",
+          "description": "Display name or stable id of who created the annotation — a person, agent, or watcher. Together with authorKind this expresses a structured author { kind, id }.",
+          "x-idid-status": "shipped"
+        },
+        "authorKind": {
+          "type": "string",
+          "description": "Actor category — WHO created the annotation. Distinct from basis (HOW it was derived) and the coarser source. Open string; recognized values below.",
+          "x-idid-recognized-values": ["human", "agent", "watcher", "system"],
+          "examples": ["human", "agent", "watcher", "system"],
+          "x-idid-status": "shipped"
+        },
+        "source": {
+          "type": "string",
+          "description": "Coarse origin label, kept as a single convenience field when the finer authorKind/basis split is not needed. Open string; recognized values below.",
+          "x-idid-recognized-values": ["user", "ai", "agent", "import", "computed", "system"],
+          "examples": ["user", "ai", "agent", "import", "computed", "system"],
+          "x-idid-status": "shipped"
+        },
+        "basis": {
+          "type": "string",
+          "description": "Evidence type — HOW the claim was derived (a hand note, a statistical test, a rule, an LLM inference, an external source). Distinct from authorKind: a human author can relay a statistical-test basis. Lets a reader weight a note by the strength of its evidence, not just who left it. Open string; recognized values below.",
+          "x-idid-recognized-values": ["human-note", "statistical-test", "rule", "llm-inference", "external-source", "computed"],
+          "examples": ["human-note", "statistical-test", "rule", "llm-inference", "external-source", "computed"],
+          "x-idid-status": "shipped"
+        },
+        "confidence": {
+          "type": "number",
+          "minimum": 0,
+          "maximum": 1,
+          "description": "Confidence in the assertion, 0–1. 1 is a hand-placed user annotation; LLM-suggested annotations typically land below 0.8.",
+          "x-idid-status": "shipped"
+        },
+        "createdAt": {
+          "type": "string",
+          "format": "date-time",
+          "description": "ISO 8601 timestamp marking when the annotation was created. Drives lifecycle.freshness together with lifecycle.ttlHint.",
+          "x-idid-status": "shipped"
+        },
+        "dataVersion": {
+          "type": "string",
+          "description": "Identifier of the data snapshot the annotation was made against. Lets a consumer distinguish 'written about last week's data' from 'still tracks current data', independent of wall-clock freshness.",
+          "x-idid-status": "shipped"
+        },
+        "stableId": {
+          "type": "string",
+          "description": "Stable, opaque identifier that survives data refresh and chart recreation. Used to re-locate 'the Q3 spike' after new data arrives (semantic anchoring), and as the target of another annotation's lifecycle.supersedes.",
+          "x-idid-status": "shipped"
+        }
+      }
+    },
+    "lifecycle": {
+      "title": "Annotation Lifecycle",
+      "type": "object",
+      "description": "Two orthogonal axes: the TEMPORAL band (freshness, derived from createdAt + ttlHint) and the EDITORIAL state (status, driven by an accept/dispute/retract flow). A note can be fresh-but-disputed or stale-but-accepted.",
+      "additionalProperties": true,
+      "properties": {
+        "freshness": {
+          "type": "string",
+          "enum": ["fresh", "aging", "stale", "expired"],
+          "description": "Temporal band. When omitted, a host derives it from ttlHint and the data's current temporal extent. Closed union.",
+          "x-idid-status": "shipped"
+        },
+        "status": {
+          "type": "string",
+          "enum": ["proposed", "accepted", "disputed", "retracted"],
+          "description": "Editorial standing, orthogonal to freshness. proposed = placed but unreviewed; accepted = confirmed; disputed = contested; retracted = withdrawn (treat like an expired note). Closed union.",
+          "x-idid-status": "shipped"
+        },
+        "supersedes": {
+          "type": "string",
+          "description": "provenance.stableId of the annotation this one replaces. Forms a revision chain; the superseded note is typically hidden once its replacement is accepted.",
+          "x-idid-status": "shipped"
+        },
+        "ttlHint": {
+          "description": "How long this annotation should be considered fresh. Either an ISO 8601 duration string (\"PT24H\", \"P7D\") or a number of milliseconds. Freshness walks fresh → aging → stale → expired as 'now' advances past createdAt + ttlHint.",
+          "oneOf": [
+            { "type": "string", "description": "ISO 8601 duration" },
+            { "type": "number", "description": "milliseconds" }
+          ],
+          "x-idid-status": "shipped"
+        },
+        "anchor": {
+          "type": "string",
+          "enum": ["fixed", "latest", "sticky", "semantic"],
+          "description": "Anchor resolution strategy across renders. fixed (default) = pinned to datum coordinates; latest = re-pins to the most recent datum each frame; sticky = stays at last known pixel position after the target scrolls off; semantic = re-resolves via provenance.stableId when new data arrives. Closed union.",
+          "x-idid-status": "shipped"
+        }
+      }
+    },
+    "lifecycleBandThresholds": {
+      "title": "Lifecycle Band Thresholds",
+      "type": "object",
+      "description": "Multipliers of ttl that mark the upper edge of each band. Defaults: fresh < 1.0×ttl, aging < 1.5×ttl, stale < 3.0×ttl, expired ≥ 3.0×ttl. The shared age→band classifier (bandFromAge) reads these.",
+      "additionalProperties": false,
+      "properties": {
+        "fresh": { "type": "number", "description": "Upper edge of fresh, ×ttl. Default 1.0." },
+        "aging": { "type": "number", "description": "Upper edge of aging, ×ttl. Default 1.5." },
+        "stale": { "type": "number", "description": "Upper edge of stale, ×ttl. Default 3.0." }
+      }
+    }
+  }
+}

package/spec/v0.1/audience-profile.schema.json

@@ -0,0 +1,67 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://semiotic.dev/spec/v0.1/audience-profile.schema.json",
+  "title": "IDID Audience Profile",
+  "description": "A serializable description of WHO is reading the charts and what the organization is trying to grow. The library never measures familiarity — it consumes a profile produced however the org sees fit (surveys, telemetry, manager judgment) and biases its ranking accordingly. Carrying this profile across tools is what lets a suggestion be calibrated to a real audience instead of a generic data-literate baseline. See ./README.md for the x-idid-status convention.",
+  "type": "object",
+  "additionalProperties": true,
+  "properties": {
+    "specVersion": {
+      "type": "string",
+      "const": "0.1",
+      "x-idid-status": "spec"
+    },
+    "name": {
+      "type": "string",
+      "description": "Display name. A ranking engine may surface this in its reasons so a reader can see whose policy shaped the result.",
+      "x-idid-status": "shipped"
+    },
+    "familiarity": {
+      "type": "object",
+      "description": "Per-chart familiarity override, 1–5, keyed by component name. Replaces the chart capability's rubric.familiarity for listed charts; unlisted charts fall back to the descriptor.",
+      "propertyNames": { "type": "string" },
+      "additionalProperties": { "type": "integer", "minimum": 1, "maximum": 5 },
+      "x-idid-status": "shipped"
+    },
+    "targets": {
+      "type": "object",
+      "description": "Adoption targets — which charts the org wants to grow or reduce. A ranking engine applies a score bias so growth targets win close calls and reduce targets fall back unless they are the only fit.",
+      "propertyNames": { "type": "string" },
+      "additionalProperties": { "$ref": "#/$defs/target" },
+      "x-idid-status": "shipped"
+    },
+    "exposureLevel": {
+      "type": "integer",
+      "enum": [0, 1, 2],
+      "description": "Controls visibility of stretch picks (unfamiliar-but-relevant charts). 0 = never surface stretches; 1 = surface in a separate stretch list (default when an audience is set); 2 = same as 1 but lowers the familiarity threshold for what counts as a stretch, widening the menu.",
+      "x-idid-status": "shipped"
+    },
+    "receptionModality": {
+      "type": "string",
+      "enum": ["visual", "screen-reader", "sonified", "agent"],
+      "description": "The channel this audience receives charts through. Orthogonal to familiarity: a reader can be highly familiar with a chart yet unable to RECEIVE it in their channel (an 8-slice pie is familiar but illegible to a screen reader). A non-visual modality instructs a host to down-rank charts whose meaning does not survive that channel. Defaults to \"visual\" (no receivability bias).",
+      "x-idid-status": "shipped"
+    }
+  },
+  "$defs": {
+    "target": {
+      "type": "object",
+      "title": "Audience Target",
+      "required": ["direction"],
+      "additionalProperties": false,
+      "properties": {
+        "direction": { "type": "string", "enum": ["increase", "decrease"] },
+        "weight": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 3,
+          "description": "Controls bias magnitude. Default 1."
+        },
+        "reason": {
+          "type": "string",
+          "description": "Human-readable rationale; a ranking engine may surface it in its reasons when the target fires."
+        }
+      }
+    }
+  }
+}