@wardx/core 0.1.0

package/README.md

@@ -0,0 +1,289 @@
+# @wardx/core
+
+`@wardx/core` is the runtime-agnostic engine of Wardx.
+
+The engine records metrics, events, and logs in memory. The engine also stores Remote Config and assigns experiment variants.
+
+The engine does not send HTTP. A runtime package, for example `wardx`, sends the frames.
+
+Node.js 20 or later is required.
+
+Install this package from npm when you write a custom runtime. If you use Node.js, install `wardx`. The `wardx` package depends on `@wardx/core`.
+
+## Install
+
+```bash
+npm install @wardx/core
+```
+
+```js
+import { WardxCore, assignVariant, loadSdkDefaults } from '@wardx/core';
+```
+
+## Design rules
+
+- A measure call changes local memory only.
+- A measure call does not send network data.
+- A measure call does not wait for a Promise.
+- If a buffer is full, the engine discards data. The engine does not block the application.
+- Counters in a frame are window deltas. Counters are not lifetime totals.
+
+## Settings
+
+`WardxCore` needs a settings object. Use `loadSdkDefaults()` and add the identity fields.
+
+| Key | Description |
+| --- | --- |
+| `endpoint` | Sync URL. The core does not use this key. Runtimes use this key. |
+| `projectKey` | Project credential. Runtimes send this key. If `privacySalt` is empty, the core uses this key as the salt. |
+| `project` | Project name. |
+| `role` | Runtime identity inside the project. Runtimes send this name. The core does not use this key. |
+| `appVersion` | Application version. |
+| `environment` | Environment name, for example `production`. |
+| `privacySalt` | Salt for the hashed subject. If you omit this key, the core uses `projectKey`. |
+| `aggregateIntervalMs` | Default `1000`. Interval to snapshot dirty data. |
+| `syncIntervalMs` | Default `15000`. Interval for the runtime sync. |
+| `maxBufferedEvents` | Default `5000`. |
+| `maxBufferedLogs` | Default `2000`. |
+| `maxFrameBytes` | Default `524288`. |
+| `maxSeriesPerMetric` | Default `1000`. |
+| `maxDimensionKeys` | Default `8`. |
+| `maxDimensionValueLength` | Default `64`. |
+| `histogramBuckets` | Default `[10, 25, 50, 100, 250, 500, 1000]`. |
+| `tracer` | Optional. Duck-typed local hook with any of `measure`, `event`, `log`, `frame`. The core does not print. Runtimes may also call `sync`. |
+
+The defaults live in `defaults.json`. Do not omit a required key. The loader does not add a fallback for a missing key. `tracer` is not a default key. Omit it to keep the measure path unchanged.
+
+## Use case 1: Record metrics in a custom runtime
+
+**When:** You write a runtime that is not Node.js, or you test the engine without HTTP.
+
+**Objective:** Record counters, gauges, histograms, and timers. Then make a frame.
+
+```js
+import { WardxCore, loadSdkDefaults } from '@wardx/core';
+
+const settings = {
+  ...loadSdkDefaults(),
+  endpoint: 'http://127.0.0.1:8787',
+  projectKey: 'dev_project_key',
+  project: 'demo',
+  role: 'client',
+  appVersion: '0.1.0',
+  environment: 'development',
+  privacySalt: 'dev_project_key'
+};
+
+const core = new WardxCore(settings);
+
+core.counter('match.completed', { mode: 'ranked' }).inc();
+core.counter('coins.awarded').add(25);
+core.gauge('players.online').set(12921);
+core.histogram('request.duration').observe(42);
+
+const endTimer = core.timer('matchmaking.duration');
+endTimer({ result: 'success' });
+
+const fitted = core.snapshotFrame();
+const frames = core.takePendingFrames();
+```
+
+### Procedure
+
+1. Load the SDK defaults.
+2. Add the identity fields.
+3. Construct `WardxCore`.
+4. Call `counter`, `gauge`, `histogram`, or `timer`.
+5. Call `snapshotFrame` when you need a frame.
+6. Call `takePendingFrames` to get the pending frames.
+
+`counter(name, dims).inc()` adds `1`. `add(n)` adds a finite number `n`.
+
+`gauge(name, dims).set(value)` stores the last finite value and a timestamp.
+
+`histogram(name).observe(value)` records a finite value into buckets. You can set buckets:
+
+```js
+core.histogram('request.duration', { buckets: [10, 25, 50, 100] }).observe(42);
+core.histogram('coins.award_size').observe(80, { grantId: 'g-80' });
+```
+
+`observe(value, attrs)` keeps `attrs` only when `value` is the window max. The frame stores that pair as `exemplar`. Attrs use the same key and value limits as dimensions. Do not change the buckets of an existing series. The engine throws an error.
+
+`timer(name, dims)` starts a timer. The returned function records the duration in milliseconds into a histogram. You can add dimensions when you stop the timer.
+
+If a series is above `maxSeriesPerMetric`, or a dimension is not valid, the engine returns a no-op object. The engine increments `wardx.internal.cardinality_dropped`.
+
+A dimension value must be a string, a number, or a boolean.
+
+## Use case 2: Record product events and logs
+
+**When:** You need discrete product events or structured logs in the same frame as metrics.
+
+**Objective:** Buffer events and logs until the next snapshot.
+
+```js
+core.event('purchase', { product: 'premium' });
+core.log.info('match_started', { mode: 'ranked', players: 4 });
+core.log.error('payment_failed', { code: 'timeout' });
+```
+
+Log levels: `debug`, `info`, `warn`, `error`.
+
+### Buffer limits
+
+- If the event buffer is full, the engine discards the new event.
+- If the log buffer is full, the engine replaces a log with a lower severity when possible.
+- If the engine cannot replace a log, the engine discards the new log.
+
+Dropped items increment `wardx.internal.events_dropped` or `wardx.internal.logs_dropped`.
+
+Use counters and histograms for rates and latency. Use events for rare product facts: a purchase, an experiment exposure or goal, a named screen. Use logs for failures. Do not put user ids on metric dimensions.
+
+## Use case 3: Get Remote Config and assign an experiment
+
+**When:** The server sends a config snapshot. You need a value for a subject.
+
+**Objective:** Get a config value. If an experiment applies, get the variant value.
+
+```js
+core.applyConfig(13, {
+  values: {
+    'message.delayMs': 1000,
+    'chat.enabled': true
+  },
+  experiments: [
+    {
+      id: 'message-delay-v1',
+      enabled: true,
+      allocation: 1,
+      salt: '3ad8f9',
+      primaryMetric: 'message.sent',
+      variants: [
+        { key: 'control', weight: 50, values: { 'message.delayMs': 1000 } },
+        { key: 'fast', weight: 50, values: { 'message.delayMs': 400 } }
+      ]
+    }
+  ]
+});
+
+const fallback = 1000;
+const delay = core.configGet('message.delayMs', fallback, { subjectId: 'user-1' });
+core.experimentGoal('message.sent', { subjectId: 'user-1', value: 1 });
+```
+
+### Resolution order
+
+1. If the key is not in the snapshot, return `fallback`.
+2. If `subjectId` is missing, return the Remote Config value.
+3. If no enabled experiment contains the key, return the Remote Config value.
+4. If the subject is not in the allocation, return the Remote Config value.
+5. If the subject is in the allocation, return the variant value.
+
+The assignment is deterministic. The same `experimentId`, `subjectId`, and `salt` always give the same variant.
+
+The first resolve for a subject in a session emits event `experiment.exposure`. The payload contains a hashed subject. The payload does not contain the raw `subjectId`.
+
+`experimentGoal` emits event `experiment.goal`. You must supply `subjectId`. You can supply `value`.
+
+## Use case 4: Assign a variant without WardxCore
+
+**When:** You verify experiment math, or you assign a variant in a test.
+
+**Objective:** Use the same FNV-1a 32-bit function as the SDK.
+
+```js
+import { assignVariant, assignmentHash, hashToUnitInterval, subjectHash } from '@wardx/core';
+
+const experiment = {
+  id: 'message-delay-v1',
+  enabled: true,
+  allocation: 0.5,
+  salt: '3ad8f9',
+  variants: [
+    { key: 'control', weight: 50, values: { 'message.delayMs': 1000 } },
+    { key: 'fast', weight: 50, values: { 'message.delayMs': 400 } }
+  ]
+};
+
+const variant = assignVariant(experiment, 'user-1');
+const hash = assignmentHash(experiment.id, 'user-1', experiment.salt);
+const bucket = hashToUnitInterval(hash);
+const hashedSubject = subjectHash('dev_project_key', 'user-1');
+```
+
+Hash input:
+
+```text
+hash = fnv1a32(experimentId + ':' + subjectId + ':' + salt)
+bucket = hash / 2^32
+```
+
+If `bucket >= allocation`, `assignVariant` returns `null`.
+
+`subjectHash` returns 8 lowercase hex digits of `fnv1a32(privacySalt + ':' + subjectId)`.
+
+## Use case 5: Build a frame for a custom transport
+
+**When:** You send frames with your transport. You do not use the Node SDK.
+
+**Objective:** Snapshot dirty data, then take the pending frames.
+
+```js
+import { FrameBuilder, PROTOCOL_VERSION, SDK_NAME, PLATFORM } from '@wardx/core';
+
+core.counter('match.completed').inc();
+const fitted = core.snapshotIfDirty();
+if (fitted) {
+  const frames = core.takePendingFrames();
+  const envelope = {
+    protocol: PROTOCOL_VERSION,
+    project: settings.project,
+    sdk: { name: SDK_NAME, version: '0.1.0' },
+    client: {
+      instanceId: '01…',
+      sessionId: '01…',
+      role: settings.role,
+      appVersion: settings.appVersion,
+      environment: settings.environment,
+      platform: PLATFORM
+    },
+    configVersion: core.configStore.version,
+    frames
+  };
+}
+```
+
+`snapshotIfDirty` returns `null` when there is no new data.
+
+If a frame is larger than `maxFrameBytes`, `FrameBuilder.fitToMaxBytes` discards data in this order:
+
+1. Logs with the lowest severity.
+2. Events from the end of the buffer.
+3. Application histograms.
+4. Application gauges. Internal gauges stay.
+
+Internal series use the prefix `wardx.internal.`.
+
+## Exports
+
+| Export | Function |
+| --- | --- |
+| `WardxCore` | Engine. |
+| `ConfigStore` | Stores one Remote Config snapshot. |
+| `ExperimentResolver` | Assigns variants and records exposure. |
+| `assignVariant` | Assigns one variant. |
+| `fnv1a32`, `assignmentHash`, `hashToUnitInterval`, `subjectHash` | Hash helpers. |
+| `Counter`, `Gauge`, `Histogram`, `MetricsRegistry` | Metric types. |
+| `EventBuffer`, `LogBuffer` | In-memory buffers. |
+| `FrameBuilder` | Builds and trims frames. |
+| `resolveSettings`, `loadSdkDefaults`, `nextSyncDelayMs` | Settings helpers. |
+| `PROTOCOL_VERSION`, `SDK_NAME`, `PLATFORM`, `INTERNAL` | Protocol constants. |
+| `ulid` | Identifier helper. |
+
+## Related packages
+
+- Node.js SDK: `wardx`
+- Ingest server: `@wardx/server`
+
+The wire contract is protocol version 1. A runtime sends `POST /v1/sync` with JSON and gzip. The request header is `X-Wardx-Key`.

package/defaults.json

@@ -0,0 +1,14 @@
+{
+  "aggregateIntervalMs": 1000,
+  "syncIntervalMs": 15000,
+  "syncJitterMin": 0.85,
+  "syncJitterMax": 1.15,
+  "maxBufferedEvents": 5000,
+  "maxBufferedLogs": 2000,
+  "maxFrameBytes": 524288,
+  "maxSeriesPerMetric": 1000,
+  "maxDimensionKeys": 8,
+  "maxDimensionValueLength": 64,
+  "httpTimeoutMs": 10000,
+  "histogramBuckets": [10, 25, 50, 100, 250, 500, 1000]
+}

package/package.json

@@ -0,0 +1,19 @@
+{
+  "name": "@wardx/core",
+  "version": "0.1.0",
+  "description": "Runtime-agnostic Wardx engine for metrics, events, logs, Remote Config, and experiments.",
+  "type": "module",
+  "engines": {
+    "node": ">=20"
+  },
+  "exports": {
+    ".": "./src/index.js"
+  },
+  "files": [
+    "src",
+    "defaults.json"
+  ],
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/src/WardxCore.js

@@ -0,0 +1,198 @@
+import { MetricsRegistry } from './metrics/MetricsRegistry.js';
+import { EventBuffer } from './buffers/EventBuffer.js';
+import { LogBuffer } from './buffers/LogBuffer.js';
+import { ConfigStore } from './config/ConfigStore.js';
+import { ExperimentResolver } from './config/ExperimentResolver.js';
+import { FrameBuilder } from './frame/FrameBuilder.js';
+import { InternalMetrics } from './internal/InternalMetrics.js';
+import { NOOP_COUNTER } from './metrics/Counter.js';
+import { NOOP_GAUGE } from './metrics/Gauge.js';
+import { NOOP_HISTOGRAM } from './metrics/Histogram.js';
+import { startTimer } from './metrics/Timer.js';
+import { emit } from './trace/emit.js';
+import { wrapCounter, wrapGauge, wrapHistogram } from './trace/wrap.js';
+
+export class WardxCore {
+  constructor(settings) {
+    this.settings = settings;
+    this.stopped = false;
+    this._tracer = settings.tracer ?? null;
+    this._wrappers = this._tracer ? new WeakMap() : null;
+    this.internal = new InternalMetrics();
+    this.metrics = new MetricsRegistry({
+      maxSeriesPerMetric: settings.maxSeriesPerMetric,
+      maxDimensionKeys: settings.maxDimensionKeys,
+      maxDimensionValueLength: settings.maxDimensionValueLength,
+      defaultHistogramBuckets: settings.histogramBuckets,
+      onCardinalityDropped: () => {
+        this.internal.cardinalityDropped += 1;
+      }
+    });
+    this.events = new EventBuffer(settings.maxBufferedEvents);
+    this.logs = new LogBuffer(settings.maxBufferedLogs);
+    this.configStore = new ConfigStore();
+    this.experiments = new ExperimentResolver({
+      privacySalt: settings.privacySalt,
+      onExposure: (payload) => {
+        this.event('experiment.exposure', payload);
+      }
+    });
+    this.seq = 0;
+    this.pendingFrames = [];
+    this.windowStart = Date.now();
+    this.log = {
+      debug: (message, attrs) => this._log('debug', message, attrs),
+      info: (message, attrs) => this._log('info', message, attrs),
+      warn: (message, attrs) => this._log('warn', message, attrs),
+      error: (message, attrs) => this._log('error', message, attrs)
+    };
+  }
+
+  counter(name, dims) {
+    return this._wrap(this.metrics.counter(name, dims), NOOP_COUNTER, (series, noop) =>
+      wrapCounter(this._tracer, series, noop, name, dims)
+    );
+  }
+
+  gauge(name, dims) {
+    return this._wrap(this.metrics.gauge(name, dims), NOOP_GAUGE, (series, noop) =>
+      wrapGauge(this._tracer, series, noop, name, dims)
+    );
+  }
+
+  histogram(name, a, b) {
+    return this._wrap(this.metrics.histogram(name, a, b), NOOP_HISTOGRAM, (series, noop) =>
+      wrapHistogram(this._tracer, series, noop, name)
+    );
+  }
+
+  timer(name, dims) {
+    if (this._tracer === null) return this.metrics.timer(name, dims);
+    return startTimer((duration, endDims) => {
+      const merged = endDims ? { ...(dims || {}), ...endDims } : dims;
+      this.histogram(name, merged).observe(duration);
+    });
+  }
+
+  event(name, attrs) {
+    const dropped = !this.events.push(name, attrs);
+    if (dropped) this.internal.eventsDropped += 1;
+    emit(this._tracer, 'event', { name, attrs: attrs ?? null, dropped });
+  }
+
+  _log(level, message, attrs) {
+    const dropped = !this.logs.push(level, message, attrs);
+    if (dropped) this.internal.logsDropped += 1;
+    emit(this._tracer, 'log', { level, message, attrs: attrs ?? null, dropped });
+  }
+
+  _wrap(series, noopSentinel, factory) {
+    if (this._tracer === null) return series;
+    if (series === noopSentinel) return factory(series, true);
+    let wrapped = this._wrappers.get(series);
+    if (wrapped) return wrapped;
+    wrapped = factory(series, false);
+    this._wrappers.set(series, wrapped);
+    return wrapped;
+  }
+
+  configGet(key, fallback, context) {
+    if (!this.configStore.has(key)) return fallback;
+    const remote = this.configStore.getRaw(key);
+    if (!context || context.subjectId === undefined || context.subjectId === null) {
+      return remote;
+    }
+    return this.experiments.resolve(
+      key,
+      remote,
+      context.subjectId,
+      this.configStore.experimentsByKey
+    );
+  }
+
+  experimentGoal(name, context) {
+    if (!context || context.subjectId === undefined || context.subjectId === null) {
+      throw new Error('experiment.goal requires subjectId');
+    }
+    if (typeof name !== 'string' || name.length === 0) {
+      throw new Error('experiment.goal requires a metric name');
+    }
+    const subject = this.experiments.hashSubject(context.subjectId);
+    const experiments = this.experiments.relevantExperiments(
+      context.subjectId,
+      this.configStore.experiments
+    );
+    const payload = {
+      metric: name,
+      subject,
+      experiments
+    };
+    if (context.value !== undefined) payload.value = context.value;
+    this.event('experiment.goal', payload);
+  }
+
+  applyConfig(version, config) {
+    this.configStore.applySnapshot({
+      version,
+      values: config.values,
+      experiments: config.experiments
+    });
+    this.internal.configVersion = version;
+  }
+
+  snapshotIfDirty() {
+    if (
+      !this.metrics.isDirty() &&
+      this.events.length === 0 &&
+      this.logs.length === 0 &&
+      !this.internal.hasCounterActivity()
+    ) {
+      return null;
+    }
+    return this.snapshotFrame();
+  }
+
+  snapshotFrame() {
+    const to = Date.now();
+    const from = this.windowStart;
+    this.windowStart = to;
+    this.internal.eventsBuffered = this.events.length;
+    this.internal.logsBuffered = this.logs.length;
+    const metrics = this.metrics.snapshotAndReset();
+    const events = this.events.swap();
+    const logs = this.logs.swap();
+    const internal = this.internal.snapshotAndReset();
+    const frame = FrameBuilder.build({
+      seq: ++this.seq,
+      from,
+      to,
+      metrics,
+      events,
+      logs,
+      internal
+    });
+    const fitted = FrameBuilder.fitToMaxBytes(frame, this.settings.maxFrameBytes);
+    this.internal.logsDropped += fitted.droppedLogs;
+    this.internal.eventsDropped += fitted.droppedEvents;
+    this.pendingFrames.push(fitted.frame);
+    emit(this._tracer, 'frame', {
+      seq: fitted.frame.seq,
+      from: fitted.frame.from,
+      to: fitted.frame.to,
+      counters: fitted.frame.metrics.counters.length,
+      gauges: fitted.frame.metrics.gauges.length,
+      histograms: fitted.frame.metrics.histograms.length,
+      events: fitted.frame.events.length,
+      logs: fitted.frame.logs.length,
+      droppedLogs: fitted.droppedLogs,
+      droppedEvents: fitted.droppedEvents
+    });
+    return fitted;
+  }
+
+  takePendingFrames() {
+    const frames = this.pendingFrames;
+    this.pendingFrames = [];
+    return frames;
+  }
+}

package/src/buffers/EventBuffer.js

@@ -0,0 +1,28 @@
+export class EventBuffer {
+  constructor(maxBufferedEvents) {
+    if (!Number.isFinite(maxBufferedEvents) || maxBufferedEvents < 1) {
+      throw new Error('maxBufferedEvents must be >= 1');
+    }
+    this.max = maxBufferedEvents;
+    this.buf = [];
+  }
+
+  get length() {
+    return this.buf.length;
+  }
+
+  push(name, attrs) {
+    if (typeof name !== 'string' || name.length === 0) {
+      throw new Error('event name must be a non-empty string');
+    }
+    if (this.buf.length >= this.max) return false;
+    this.buf.push([Date.now(), name, attrs ?? null]);
+    return true;
+  }
+
+  swap() {
+    const sealed = this.buf;
+    this.buf = [];
+    return sealed;
+  }
+}

package/src/buffers/LogBuffer.js

@@ -0,0 +1,48 @@
+import { LOG_RANK } from '../protocol.js';
+
+export class LogBuffer {
+  constructor(maxBufferedLogs) {
+    if (!Number.isFinite(maxBufferedLogs) || maxBufferedLogs < 1) {
+      throw new Error('maxBufferedLogs must be >= 1');
+    }
+    this.max = maxBufferedLogs;
+    this.buf = [];
+  }
+
+  get length() {
+    return this.buf.length;
+  }
+
+  push(level, message, attrs) {
+    if (!Object.prototype.hasOwnProperty.call(LOG_RANK, level)) {
+      throw new Error(`invalid log level: ${level}`);
+    }
+    if (typeof message !== 'string' || message.length === 0) {
+      throw new Error('log message must be a non-empty string');
+    }
+    const entry = [Date.now(), level, message, attrs ?? null];
+    if (this.buf.length < this.max) {
+      this.buf.push(entry);
+      return true;
+    }
+    const incomingRank = LOG_RANK[level];
+    let victim = -1;
+    let victimRank = incomingRank;
+    for (let i = 0; i < this.buf.length; i++) {
+      const rank = LOG_RANK[this.buf[i][1]];
+      if (rank < victimRank) {
+        victimRank = rank;
+        victim = i;
+      }
+    }
+    if (victim === -1) return false;
+    this.buf[victim] = entry;
+    return false;
+  }
+
+  swap() {
+    const sealed = this.buf;
+    this.buf = [];
+    return sealed;
+  }
+}

package/src/config/ConfigStore.js

@@ -0,0 +1,31 @@
+import { indexExperimentsByKey } from './ExperimentResolver.js';
+
+export class ConfigStore {
+  constructor() {
+    this.version = 0;
+    this.values = Object.create(null);
+    this.experiments = [];
+    this.experimentsByKey = new Map();
+  }
+
+  applySnapshot(snapshot) {
+    if (!snapshot || typeof snapshot !== 'object') {
+      throw new Error('config snapshot must be an object');
+    }
+    if (typeof snapshot.version !== 'number' || !Number.isFinite(snapshot.version)) {
+      throw new Error('config snapshot version must be a finite number');
+    }
+    this.version = snapshot.version;
+    this.values = snapshot.values && typeof snapshot.values === 'object' ? snapshot.values : Object.create(null);
+    this.experiments = Array.isArray(snapshot.experiments) ? snapshot.experiments : [];
+    this.experimentsByKey = indexExperimentsByKey(this.experiments);
+  }
+
+  has(key) {
+    return Object.prototype.hasOwnProperty.call(this.values, key);
+  }
+
+  getRaw(key) {
+    return this.values[key];
+  }
+}