@volter/twin-segment 0.1.0

package/README.md

@@ -0,0 +1,154 @@
+# @volter/twin-segment
+
+> **Legacy connector helpers:** this package still has callable helpers using the retired v1
+> `syncPull` API. Those paths require migration before use on the current kernel;
+> older helper descriptions below do not establish current compatibility. Check the
+> [generated index](../../../generated/INDEX.md) for protocol standing and use the
+> [shared model](../../../docs/concepts/the-model.md) for current state semantics.
+
+A local, stateful replica of **Segment's HTTP Tracking API**. Point an unmodified
+`@segment/analytics-node` client at it with nothing but the SDK's own `host` setting and your
+events fold into durable twin state — offline, deterministic, with no write key that means
+anything and no traffic leaving the machine.
+
+```bash
+bunx world-segment serve --port 8787     # then: new Analytics({ writeKey, host: 'http://localhost:8787' })
+bunx world-segment ops                   # the ratified surface, one line per operation
+bunx world-segment conformance           # vendor-property checks over the modeled plane
+```
+
+## Coverage
+
+**The denominator is AUTHORED, not compiled.** Segment publishes no machine-readable spec for its
+ingestion surface, so this pack's sixteen operations were ratified one at a time from evidence at
+`var/line/segment/SURFACE.json`, each entry carrying the source it stands on:
+
+| tier | ops | grounding |
+|---|---|---|
+| SDK wire literal | 2 | `POST /v1/batch` (the single URL `@segment/analytics-node@2.3.0`'s Publisher ever posts to) and `POST /token` on the OAuth authorization server |
+| Docs only, not verified offline | 14 | the six direct `POST /v1/{track,identify,page,screen,group,alias}` routes, the mobile batch route `POST /v1/b`, analytics.js's `POST /v1/t`, and the six literal Pixel Routes `GET /v1/pixel/{...}` |
+
+Docs grounding comes from **`github.com/segmentio/segment-docs`, the vendor's own documentation
+source repository** — `segment.com` answers 403 to non-browser clients, so the rendered pages were
+not read. Rulings and the ratified vendor facts live in `var/line/segment/RULINGS.json` and
+`FACTS.json`.
+
+Coverage reads honestly **partial**: the ingestion half of the tracking plane is modeled, and
+everything else is a filed `todo`. That is the intended shape — a broad honest denominator beats a
+thin self-portrait.
+
+### What is modeled
+
+- **`POST /v1/batch`** — the envelope every official server SDK sends, `{batch, writeKey, sentAt}`,
+  dispatched **per message** by each message's own `type`. Batch-level `context` and
+  `integrations` merge into every message, message-level keys winning, as the vendor documents.
+- **The direct routes** `POST /v1/{track,identify,page,group,alias}`, which fold through the
+  *same* code path as their batched equivalents, so the two can never disagree.
+- **All three documented auth schemes**: `writeKey` in the body with no header (the SDK's own
+  default), HTTP Basic with the write key as the username and an empty password (analytics-node
+  v1's shape), and OAuth `Bearer` alongside a payload write key.
+- **The vendor's accept-and-drop semantics.** Segment "returns a `200` response for all API
+  requests except errors caused by large payloads and JSON errors", and then silently rejects
+  events with no `userId`/`anonymousId` (its own `no_user_anon_id` error), Tracks with no `event`
+  name, and batch members past 2,500 events or 32KB. The twin reproduces that exactly and records
+  each drop in a local `dropped` projection so the outcome stays inspectable.
+- **`messageId` dedupe**, the vendor's own idempotency key — across requests and within one batch.
+- The documented **400s** (invalid JSON, oversize payload) with the `{code, message}` envelope
+  `analytics-python` parses off every non-200.
+
+### Deliberate deviation: unmodeled operations fail LOUDLY
+
+The real vendor answers `200` to almost everything. This twin answers a `404` `[twin gap]` naming
+the operation for anything it has ratified but not modeled — `/v1/b`, `/v1/t`, `/v1/screen`, the
+six pixel routes, `/token`, and the `screen` message type inside an otherwise-modeled batch. That
+is a knowing departure from fidelity, and the right one: a `200` that stores nothing is
+indistinguishable from success, which is the one thing a twin may never do. It is ruled in
+`var/line/segment/RULINGS.json` and pinned by `segment.api.fail_loudly`.
+
+### What this denominator excludes, and why
+
+Each of these is a **ruling**, not an omission:
+
+- **The Public API** (`api.segmentapis.com`) — workspaces, sources, destinations, warehouses,
+  tracking plans. A different product on a different host behind a workspace token. Its host is
+  **not** claimed in the injector map, so a Public API call from a world still reaches the real
+  vendor: a named exposure, and the first thing a follow-up article should close.
+- **The Profile API** (`profiles.segment.com`) — a different host, credential and product tier,
+  returning resolved profiles rather than ingested events.
+- **`cdn.segment.com/v1/projects/{writeKey}/settings`** — real and first-party, but that host also
+  serves the analytics.js bundle, so intercepting it would break loading the real library.
+- **Inbound webhooks** — this vendor has none on the tracking plane. Segment's "Webhooks (Actions)"
+  is a *destination* (Segment → you).
+- **The Objects APIs** (`objects.segment.com/v1/set` and the four `/v0` routes of
+  `objects-bulk-api.segmentapis.com`) — real, first-party, on the same server-source catalog
+  branch and behind the same source write key, but beta, warehouse-object loading rather than
+  event ingestion, and unreachable by any `analytics-<language>` client, so this article's
+  SDK-anchored method has nothing to verify against. Neither host is claimed in the injector map —
+  the same named exposure the Public API carries, and the same follow-up
+  (`denominator:object-apis-plane-named-but-not-claimed`, filed by A3).
+
+### Connector
+
+**Push** replays locally-ingested flushes onto the real `POST /v1/batch` through an injected
+client, rebuilding the SDK's own envelope. **Pull mirrors nothing, and that is a vendor fact**:
+every one of the sixteen ratified operations is a *write*. Segment's own answer to "did my event
+land" is the browser Source Debugger, a live websocket view — not an API. Rather than fake an
+empty account over real observed state, the gap is filed as `segment.connector.pull`.
+
+### UI mirror — the Source Debugger
+
+Segment **is owed a mirror**. Its core browser job — wiring sources to destinations, watching the
+Source Debugger, editing tracking plans — is exactly the "the UI is the product" case, so
+`ui-scope.json` rules `needsUi: true`.
+
+The **Source Debugger** is built (`segment.ui.debugger`): a React screen showing the two live
+streams a source has, the **accepted** messages with their type, label, subject and properties, and
+the **dropped** ones with the REASON they were refused. It cannot BE an API capability: all sixteen
+ratified operations are ingests and none of them reads a message back, so the vendor's own answer to
+"my event returned 200 and never arrived" is a screen rather than a call.
+
+**The dropped pane is the twin's, not the vendor's** — a deliberate superset, labelled as one on the
+screen itself. Segment's real Source Debugger shows what a source RECEIVED; it has no
+accepted-then-silently-dropped view, because the vendor answers 200 and discards. This twin records
+the refusal instead, which is the only way that failure becomes observable at all. Of the seven drop
+reasons it shows, only `no_user_anon_id` is a string Segment itself prints (its Errors page); the
+other six are this pack's own descriptive labels and the row marks each of them **twin label**.
+
+```bash
+bun run packages/twin/segment/src/cli.ts mirror --port 4100   # the debugger + the tracking API, one origin
+```
+
+The screen reads the twin's own **store door** (`GET /twin/store/events`, `…/dropped`) on the same
+origin the tracking routes are served on — one serving code path, so the screen and its data cannot
+drift. A real headless-Chromium journey (`segment-journey.uitest.ts`,
+`segment.journey.debugger`) drives it end to end.
+
+Two screens remain filed as live todos rather than fabricated: `segment.ui.connections` (the
+Connections graph, which needs the Public API plane this pack does not twin) and
+`segment.ui.profiles` (a user/group explorer over the identity projections). Cite those two as a
+**tracked gap**, never as precedent for omitting a mirror.
+
+### The store doors
+
+The HTTP Tracking API is write-only: every route is an ingest and none of them reads a message
+back. What was accepted — and what was accepted-and-dropped, which Segment answers `200` to and
+never reports — is read through the twin's own named projections, `GET /twin/store/{events,
+identities,groups,dropped}`: the read-only doors the programming model gives a twin whose vendor
+has no listing endpoint (the `mailgun` precedent). They are deliberately **out of the capability
+manifest**: counting scaffolding the vendor does not have would pad the denominator. `GET /twin`
+names them under `stores`.
+
+### Rate budget
+
+The vendor publishes `1,000 requests per second` per workspace, and this pack deliberately does
+**not** adopt it: that is a workspace-wide production-throughput recommendation, not a
+per-connector allowance, and this pack's only live caller is a replay path. The declaration sits
+at exactly the kernel fallback's shape (60 weighted units / 60s at `defaultWeight` 2). See
+`src/segment-budget.ts` for the full reasoning, the figures, their source and the access date.
+
+### Fidelity test
+
+`src/segment-sdk.integration.test.ts` drives the **real, unmodified** `@segment/analytics-node`
+(pinned `^2.3.0`) against an in-process twin server, configured only through the SDK's own public
+`host`/`flushAt`/`maxRetries` settings — configuration, not modification. It asserts the SDK's own
+**delivery verdict**, so a twin answering the wrong status cannot pass.

package/client/segment-mirror.css

@@ -0,0 +1,45 @@
+:root {
+  --bg: #0f1115;
+  --panel: #171a21;
+  --line: #262b35;
+  --text: #e6e9ef;
+  --muted: #9aa4b2;
+  --accent: #292929;
+  --track: #52bd94;
+  --identity: #6aa9f0;
+  --screenish: #b98cf0;
+  --dropped: #d24b4b;
+}
+* { box-sizing: border-box; }
+body { margin: 0; font-family: -apple-system, system-ui, sans-serif; background: var(--bg); color: var(--text); }
+.app { max-width: 1040px; margin: 0 auto; padding: 0 20px 32px; }
+.app-header { display: flex; align-items: center; justify-content: space-between; height: 60px; border-bottom: 1px solid var(--line); }
+.app-title { font-size: 18px; font-weight: 700; margin: 0; letter-spacing: 0.02em; }
+.counts { display: flex; gap: 14px; font-size: 13px; color: var(--muted); font-variant-numeric: tabular-nums; }
+.count-dropped { color: var(--dropped); }
+.app-nav { display: flex; gap: 6px; padding: 14px 0; }
+.nav-item { background: none; border: 1px solid transparent; color: var(--muted); padding: 7px 14px; border-radius: 999px; cursor: pointer; font-size: 14px; }
+.nav-item:hover { color: var(--text); }
+.nav-item-active { background: var(--accent); border-color: var(--line); color: var(--text); }
+.event-stream, .dropped-stream { list-style: none; margin: 0; padding: 0; background: var(--panel); border: 1px solid var(--line); border-radius: 10px; overflow: hidden; }
+.event-row, .dropped-row { padding: 12px 14px; border-bottom: 1px solid var(--line); }
+.event-row:last-child, .dropped-row:last-child { border-bottom: none; }
+.event-head { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
+.event-clock { color: var(--muted); font-size: 12px; font-variant-numeric: tabular-nums; }
+.event-label { font-weight: 600; font-size: 14px; }
+.event-subject, .drop-endpoint { color: var(--muted); font-size: 12px; margin-left: auto; }
+.drop-reason { color: var(--dropped); font-size: 13px; }
+.drop-reason-twin { font-size: 11px; color: var(--muted); border: 1px dashed var(--line); border-radius: 4px; padding: 1px 6px; }
+.stream-note { color: var(--muted); font-size: 12px; margin: 0 0 10px; }
+.stream-note code { color: var(--text); }
+.pill { padding: 3px 10px; border-radius: 999px; font-size: 12px; white-space: nowrap; }
+.pill-track { background: rgba(82,189,148,0.18); color: var(--track); }
+.pill-identity { background: rgba(106,169,240,0.18); color: var(--identity); }
+.pill-screenish { background: rgba(185,140,240,0.18); color: var(--screenish); }
+.pill-dropped { background: rgba(210,75,75,0.18); color: var(--dropped); }
+.chips { display: flex; flex-wrap: wrap; gap: 6px; margin-top: 8px; }
+.chips-empty { color: var(--muted); font-size: 12px; }
+.chip { display: inline-flex; gap: 6px; background: var(--bg); border: 1px solid var(--line); border-radius: 6px; padding: 3px 8px; font-size: 12px; }
+.chip-key { color: var(--muted); }
+.chip-value { font-variant-numeric: tabular-nums; }
+.empty { color: var(--muted); padding: 24px; background: var(--panel); border: 1px solid var(--line); border-radius: 10px; }

package/client/segment-mirror.tsx

@@ -0,0 +1,154 @@
+// Segment mirror — the SOURCE DEBUGGER, as a React/TSX client. Every byte on screen is fetched
+// from the twin's OWN store door on the same origin (`/twin/store/events`, `/twin/store/dropped`)
+// — the kernel's read-only named-projection door, and the sanctioned way a mirror reads twin
+// state. API↔UI parity: this is the projection the ingest routes fold into, not a second copy.
+import { createRoot } from 'react-dom/client';
+import { useEffect, useState } from 'react';
+import {
+  PILL_CLASS,
+  clockLabel,
+  messageLabel,
+  propertyChips,
+  subjectOf,
+  typeTone,
+  type SegmentRow,
+} from '../src/segment-mirror-ui.ts';
+
+// WHERE THIS MIRROR LIVES. Served at a vendor root the base is '' (fetches are root-relative, as
+// before); served under a path prefix with a <base> tag every read and write resolves inside that
+// prefix instead of escaping it.
+const WIRE_BASE = typeof document === 'undefined' || document.querySelector('base[href]') === null ? '' : new URL('.', document.baseURI).pathname.replace(/\/$/, '');
+
+const STREAMS = [
+  { key: 'events', label: 'Accepted' },
+  { key: 'dropped', label: 'Dropped' },
+] as const;
+type StreamKey = (typeof STREAMS)[number]['key'];
+
+/** Read one named projection off the twin's store door. */
+export async function store(name: string): Promise<SegmentRow[]> {
+  const res = await fetch(`${WIRE_BASE}/twin/store/${name}`);
+  if (!res.ok) return [];
+  const body = (await res.json()) as unknown;
+  return Array.isArray(body) ? (body as SegmentRow[]) : [];
+}
+
+/** The property/trait chips on an expanded row — what a debugger reader is actually looking for
+ *  when an event "arrived but is wrong". */
+export function PropertyChips({ row }: { row: SegmentRow }) {
+  const chips = [...propertyChips(row.properties), ...propertyChips(row.traits)];
+  if (chips.length === 0) return <div className="chips chips-empty">no properties</div>;
+  return (
+    <div className="chips">
+      {chips.map((c) => (
+        <span className="chip" key={c.key}>
+          <span className="chip-key">{c.key}</span>
+          <span className="chip-value">{c.value}</span>
+        </span>
+      ))}
+    </div>
+  );
+}
+
+/** One row of the accepted stream. */
+export function EventRow({ row }: { row: SegmentRow }) {
+  const tone = typeTone(row.messageType);
+  return (
+    <li className="event-row" data-message-id={String(row.messageId ?? '')}>
+      <div className="event-head">
+        <span className="event-clock">{clockLabel(row.receivedAt)}</span>
+        <span className={PILL_CLASS[tone]}>{String(row.messageType ?? '')}</span>
+        <span className="event-label">{messageLabel(row)}</span>
+        <span className="event-subject">{subjectOf(row)}</span>
+      </div>
+      <PropertyChips row={row} />
+    </li>
+  );
+}
+
+/**
+ * One row of the dropped stream.
+ *
+ * THIS PANE IS THE TWIN'S, NOT THE VENDOR'S, and the screen says so. Segment's real Source
+ * Debugger shows what a source RECEIVED; it has no "accepted-then-silently-dropped, with a reason"
+ * view, because the vendor answers 200 and discards. This twin records the refusal instead, which
+ * is the only way "returned 200 and never arrived" becomes observable — a superset of the vendor's
+ * screen, and the honest label for it. Only `no_user_anon_id` is a reason Segment itself prints
+ * (its Errors page); the other six are this pack's own descriptive strings, which is why the row
+ * marks them.
+ */
+export function DroppedRow({ row }: { row: SegmentRow }) {
+  return (
+    <li className="dropped-row" data-message-id={String(row.messageId ?? '')}>
+      <div className="event-head">
+        <span className={PILL_CLASS.dropped}>dropped</span>
+        <span className="event-label">{String(row.messageType ?? '')}</span>
+        <span className="drop-reason">{String(row.reason ?? 'unknown')}</span>
+        {String(row.reason ?? '') === 'no_user_anon_id'
+          ? null
+          : <span className="drop-reason-twin" title="This reason is a twin-authored label. Segment prints only `no_user_anon_id`.">twin label</span>}
+        <span className="drop-endpoint">{String(row.endpoint ?? '')}</span>
+      </div>
+    </li>
+  );
+}
+
+/** The accepted stream, newest last (the order the source received them). */
+export function EventStream({ rows }: { rows: SegmentRow[] }) {
+  if (rows.length === 0) return <div className="empty">No events yet. Send one and it appears here.</div>;
+  return <ul className="event-stream">{rows.map((r) => <EventRow row={r} key={String(r.messageId)} />)}</ul>;
+}
+
+/** The dropped stream. */
+export function DroppedStream({ rows }: { rows: SegmentRow[] }) {
+  if (rows.length === 0) return <div className="empty">Nothing dropped. Every message this source received was attributed.</div>;
+  return (
+    <>
+      <p className="stream-note">
+        Segment answers 200 and discards these; the twin records them. Only <code>no_user_anon_id</code> is
+        a reason Segment itself prints — the rest carry a <span className="drop-reason-twin">twin label</span>.
+      </p>
+      <ul className="dropped-stream">{rows.map((r) => <DroppedRow row={r} key={String(r.messageId)} />)}</ul>
+    </>
+  );
+}
+
+function App() {
+  const [stream, setStream] = useState<StreamKey>('events');
+  const [events, setEvents] = useState<SegmentRow[]>([]);
+  const [dropped, setDropped] = useState<SegmentRow[]>([]);
+
+  useEffect(() => {
+    void (async () => {
+      setEvents(await store('events'));
+      setDropped(await store('dropped'));
+    })();
+  }, []);
+
+  return (
+    <main className="app">
+      <header className="app-header">
+        <h1 className="app-title">Source Debugger</h1>
+        <span className="counts">
+          <span className="count-accepted" data-count={events.length}>{events.length} accepted</span>
+          <span className="count-dropped" data-count={dropped.length}>{dropped.length} dropped</span>
+        </span>
+      </header>
+      <nav className="app-nav">
+        {STREAMS.map((s) => (
+          <button
+            key={s.key}
+            className={stream === s.key ? 'nav-item nav-item-active' : 'nav-item'}
+            onClick={() => setStream(s.key)}
+          >
+            {s.label}
+          </button>
+        ))}
+      </nav>
+      {stream === 'events' ? <EventStream rows={events} /> : <DroppedStream rows={dropped} />}
+    </main>
+  );
+}
+
+const mount = typeof document === 'undefined' ? null : document.getElementById('root');
+if (mount) createRoot(mount).render(<App />);