@dataworks-technology/data 0.1.3 → 0.1.5

package/README.md

@@ -3,6 +3,7 @@
 [![npm version](https://img.shields.io/npm/v/@dataworks-technology/data)](https://www.npmjs.com/package/@dataworks-technology/data)
 [![license](https://img.shields.io/npm/l/@dataworks-technology/data)](./LICENSE)
 [![bundle size](https://img.shields.io/bundlephobia/minzip/@dataworks-technology/data)](https://bundlephobia.com/package/@dataworks-technology/data)
+[![docs](https://img.shields.io/badge/docs-data--sdk--docs.dataworks.live-blue)](https://data-sdk-docs.dataworks.live)
 
 Official SDK for the Dataworks Data Engine — authenticate, ingest live athlete metrics, subscribe to real-time data streams, and report errors.
 
@@ -51,7 +52,7 @@ bun add @dataworks-technology/data
 ```typescript
 import { DataClient } from "@dataworks-technology/data";
 
-const client = new DataClient({
+const dataworks = new DataClient({
   cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
   clientId: "your-client-id",
   ingestUrl: "https://your-ingest-endpoint.dataworks.live",
@@ -60,14 +61,14 @@ const client = new DataClient({
 });
 
 // Authenticate
-await client.login("username", "password");
+await dataworks.login("username", "password");
 
 // Ingest metrics
-await client.ingest(
+await dataworks.ingest(
   [
     {
       athleteId: "athlete-1",
-      metric: "heartrate",
+      metric: "heartrate_calculated",
       value: 172,
       timestamp: Math.floor(Date.now() / 1000),
     },
@@ -84,7 +85,7 @@ await client.ingest(
 Create a new client instance.
 
 ```typescript
-const client = new DataClient({
+const dataworks = new DataClient({
   cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
   clientId: "abc123",
   ingestUrl: "https://ingest.dataworks.live",
@@ -93,37 +94,39 @@ const client = new DataClient({
 });
 ```
 
-### `client.login(username, password)`
+### `dataworks.login(username, password)`
 
 Authenticate with Cognito. Must be called before any other operation.
 
 ```typescript
-const result = await client.login("username", "password");
+const result = await dataworks.login("username", "password");
 // result: { accessToken, idToken, refreshToken, tenant }
 ```
 
-### `client.ingest(metrics, eventId, datasetDatasourceId)`
+### `dataworks.ingest(metrics, eventId, datasetDatasourceId)`
 
 Send metric data points to the Data Engine. Invalid metrics are automatically filtered out.
 
 ```typescript
-await client.ingest(
+await dataworks.ingest(
   [
-    { athleteId: "1", metric: "heartrate", value: 172, timestamp: 1700000000 },
-    { athleteId: "1", metric: "speed", value: 4.2, timestamp: 1700000000 },
+    { athleteId: "1", metric: "heartrate_calculated", value: 172, timestamp: 1700000000 },
+    { athleteId: "1", metric: "speed_calculated", value: 4.2, timestamp: 1700000000 },
   ],
   "event-123",
   "ds-456",
 );
 ```
 
-### `client.subscribe(channel, onEvent)`
+### `dataworks.subscribe(channel, onEvent)`
 
 Subscribe to real-time data events via WebSocket.
 
+Tip: start with `dataworks/1/1/*` to inspect all metrics for a dataset-datasource/event pair, then narrow to a specific metric channel such as `dataworks/1/1/heartrate`.
+
 ```typescript
-const subscription = client.subscribe(
-  "/default/events/event-123",
+const subscription = dataworks.subscribe(
+  "dataworks/1/1/heartrate",
   (event) => {
     console.log("Received:", event);
   },
@@ -133,36 +136,36 @@ const subscription = client.subscribe(
 subscription.close();
 ```
 
-### `client.reportError(error)`
+### `dataworks.reportError(error)`
 
 Report an error to the Data Engine for monitoring and alerting.
 
 ```typescript
-await client.reportError({
+await dataworks.reportError({
   datasetDatasourceId: "ds-456",
-  eventId: "event-123",
+  athleteId: "athlete-123",
   clientId: 1,
   errorTitle: "Sensor disconnected",
   errorDescription: "BLE heart rate sensor lost connection at 00:42:15",
 });
 ```
 
-### `client.isAuthenticated`
+### `dataworks.isAuthenticated`
 
 Check if the client has valid credentials.
 
 ```typescript
-if (client.isAuthenticated) {
-  await client.ingest(metrics, eventId, dsId);
+if (dataworks.isAuthenticated) {
+  await dataworks.ingest(metrics, eventId, dsId);
 }
 ```
 
-### `client.tenant`
+### `dataworks.tenant`
 
 Get the tenant from the authenticated session (or `null`).
 
 ```typescript
-console.log(`Logged in as tenant: ${client.tenant}`);
+console.log(`Logged in as tenant: ${dataworks.tenant}`);
 ```
 
 ## Validation Utilities
@@ -237,7 +240,7 @@ interface Metric {
 
 interface ErrorReport {
   datasetDatasourceId: string;
-  eventId: string;
+  athleteId: string;
   clientId: number;
   errorTitle: string;
   errorDescription: string;
@@ -257,13 +260,13 @@ All async methods throw on failure. Wrap calls in try/catch:
 
 ```typescript
 try {
-  await client.login("user", "pass");
+  await dataworks.login("user", "pass");
 } catch (err) {
   // Authentication failed (invalid credentials, network error, etc.)
 }
 
 try {
-  await client.ingest(metrics, eventId, dsId);
+  await dataworks.ingest(metrics, eventId, dsId);
 } catch (err) {
   // Ingestion failed (401 expired token, 5xx server error, etc.)
 }
@@ -278,11 +281,277 @@ Calling `ingest()`, `reportError()`, or `subscribe()` before `login()` throws im
 - **TypeScript** ≥ 5.0 (optional — works with plain JavaScript too)
 - **Browser** — compatible with any environment that has `fetch` and `WebSocket`
 
+## Derived Metrics — Round-Trip Example
+
+A core use case: subscribe to live metrics, apply your own calculations externally, and push **derived metrics** back into the Data Engine. This creates a feedback loop where the platform stores both raw and enriched data.
+
+### Concept
+
+```
+Data Engine → subscribe("dataworks/{datasetDatasourceId}/{eventId}/heartrate") → Your App → calculate rolling avg → ingest("heartrate_rolling_avg_calculated") → Data Engine
+```
+
+Any metric you ingest is treated the same as raw sensor data — it flows through the enrichment pipeline (avg/min/max/zones), gets stored, and is available via subscriptions and the GraphQL API.
+
+### TypeScript Example
+
+```typescript
+import { DataClient } from "@dataworks-technology/data";
+
+const dataworks = new DataClient({
+  cognitoEndpoint: "https://cognito-idp.eu-west-1.amazonaws.com/",
+  clientId: "your-client-id",
+  ingestUrl: "https://your-ingest-endpoint.dataworks.live",
+  errorUrl: "https://your-error-endpoint.dataworks.live",
+  realtimeUrl: "https://your-realtime-endpoint.dataworks.live",
+});
+
+await dataworks.login("developer", "password");
+
+// 1. Subscribe to raw heart rate data
+const buffer: number[] = [];
+const datasetDatasourceId = "1";
+const eventId = "1";
+const heartrateChannel = `dataworks/${datasetDatasourceId}/${eventId}/heartrate`;
+
+dataworks.subscribe(heartrateChannel, (event) => {
+  // No metric guard needed here: the channel already filters to heartrate.
+
+  // 2. Calculate a rolling 3-point average
+  buffer.push(event.value);
+  if (buffer.length > 3) buffer.shift();
+  const avg = buffer.reduce((a, b) => a + b, 0) / buffer.length;
+
+  // 3. Push the derived metric back into the engine
+  dataworks.ingest(
+    [
+      {
+        athleteId: event.athleteId,
+        metric: "heartrate_rolling_avg_calculated",
+        value: Math.round(avg * 10) / 10,
+        timestamp: event.timestamp,
+      },
+    ],
+    event.eventId,
+    event.datasetDatasourceId,
+  );
+});
+```
+
+### Python Example (Cross-Platform)
+
+The same flow works from any language. A full working Python demo is included at `test/demo/round-trip.py` — it proves every SDK capability end-to-end:
+
+| Phase | What it does | SDK equivalent |
+|-------|-------------|----------------|
+| 1. Authenticate | Cognito USER_PASSWORD_AUTH | `dataworks.login()` |
+| 2. Subscribe | WebSocket → AppSync Events API | `dataworks.subscribe()` |
+| 3. Receive | Catch events on the subscriber | `onEvent` callback |
+| 4. Calculate | Write to CSV/Google Sheets, compute rolling avg + HR zones | Your business logic |
+| 5. Re-ingest | POST enriched metrics back to the Data Engine | `dataworks.ingest()` |
+| 6. Report Error | Send error through the error pipeline | `dataworks.reportError()` |
+
+#### Run the demo
+
+```bash
+# Install Python dependencies
+pip install websocket-client requests
+
+# Run from the repo root (reads test.env for credentials)
+python test/demo/round-trip.py
+```
+
+#### Output
+
+```
+PHASE 4: WRITE DATA + CALCULATE
+  Enriched data:
+    HR=130 → avg=130.0, zone=Zone 2 (Aerobic), delta=—
+    HR=140 → avg=135.0, zone=Zone 3 (Tempo), delta=+10
+    HR=150 → avg=140.0, zone=Zone 3 (Tempo), delta=+10
+    HR=160 → avg=150.0, zone=Zone 4 (Threshold), delta=+10
+    HR=170 → avg=160.0, zone=Zone 4 (Threshold), delta=+10
+
+PHASE 5: RE-INGEST ENRICHED METRICS
+    → heartrate_rolling_avg=130.0
+    → heartrate_rolling_avg=135.0
+    → heartrate_rolling_avg=140.0
+    → heartrate_rolling_avg=150.0
+    → heartrate_rolling_avg=160.0
+  ✓ Ingested (200)
+```
+
+The raw `heartrate` values (130–170 bpm) are transformed into `heartrate_rolling_avg_calculated` derived metrics and pushed back into the platform. These derived metrics are then available to all consumers (Console, Visual, Moments triggers) just like any other metric.
+
+#### Google Sheets (optional)
+
+The demo can write to Google Sheets instead of CSV — formulas compute rolling averages, HR zone classification, and deltas in the spreadsheet before re-ingesting. See `test/demo/README.md` for setup.
+
+### Ideas for Derived Metrics
+
+| Derived metric | Calculation | Use case |
+|---|---|---|
+| `heartrate_rolling_avg_calculated` | 3-point moving average | Smooth noisy sensor data |
+| `heartrate_zone` | Zone classification (1–5) | Training load analysis |
+| `speed_efficiency` | Speed / heart rate ratio | Fatigue detection |
+| `power_to_weight` | Power / athlete weight | Normalised performance |
+| `pace_delta` | Current pace − target pace | Real-time coaching alerts |
+| `fatigue_index` | Custom formula over time window | Trigger warnings via Moments engine |
+
 ## Documentation
 
 Full documentation with guides, examples, and detailed API reference:
 
-**[https://data-docs.dataworks.live](https://data-docs.dataworks.live)**
+**[https://data-sdk-docs.dataworks.live](https://data-sdk-docs.dataworks.live)**
+
+## Development
+
+> Internal contributors only — this section covers building, testing, and publishing the package.
+
+This package publishes to **public npm** (`registry.npmjs.org`), not to the internal CodeArtifact registry used by `@dataworks/sdk`. The `packages/sdk/.npmrc` and `publishConfig` in `package.json` enforce this.
+
+### Build
+
+```bash
+bun run build    # tsup → dist/ (ESM + CJS + DTS)
+bun run dev      # tsup --watch (rebuild on save)
+bun run test     # vitest unit tests (mocked — no infra needed)
+```
+
+### Local Development
+
+To test SDK changes locally before publishing:
+
+**Option A — `npm pack` (recommended, simulates real install):**
+
+```bash
+# In packages/sdk/
+bun run build
+npm pack                    # creates dataworks-technology-data-0.1.4.tgz
+
+# In your consumer project (e.g. test/demo/)
+npm install ../../packages/sdk/dataworks-technology-data-0.1.4.tgz
+```
+
+**Option B — `bun link` (faster iteration):**
+
+```bash
+# In packages/sdk/
+bun run build
+bun link
+
+# In your consumer project
+bun link @dataworks-technology/data
+```
+
+**Option C — watch mode + link (live reload):**
+
+```bash
+# Terminal 1: packages/sdk/
+bun run dev                 # rebuilds dist/ on every save
+
+# Terminal 2: consumer project
+bun link @dataworks-technology/data
+npx tsx demo.ts             # picks up latest build automatically
+```
+
+> **Tip:** After local testing, remember to `bun unlink` or reinstall from npm before committing.
+
+### Publish
+
+Login to npm first (one-time):
+```bash
+npm login --registry https://registry.npmjs.org/
+```
+
+Then from `packages/sdk/`:
+```bash
+bun run publish:patch   # bump patch + publish
+bun run publish:minor   # bump minor + publish
+bun run publish:major   # bump major + publish
+```
+
+### E2E Tests
+
+The e2e test suite (`test/e2e-public-sdk.test.ts`) validates the full external developer flow against deployed infrastructure. It runs against live AWS services and proves the public SDK works end-to-end across all three stacks (Shared, Data, SDK).
+
+#### Prerequisites
+
+Deploy all three stacks first — `make deploy` generates `test.env` with the required env vars:
+
+| Env var | Source | Purpose |
+|---|---|---|
+| `FRONTEND_CLIENT_ID` | Shared stack → Cognito | App client for USER_PASSWORD_AUTH |
+| `API_URL` | Data stack → API Gateway | Metric ingestion endpoint |
+| `ERROR_URL` | Data stack → API Gateway | Error reporting endpoint |
+| `REALTIME_URL` | Data stack → AppSync Events API | WebSocket subscription endpoint |
+| `COGNITO_BASE_DOMAIN` | Shared stack → Cognito | IDP endpoint for auth |
+| `DEVELOPER_USERNAME` | Shared stack → Cognito | Test user in the `developer` group |
+| `DEVELOPER_PASSWORD` | Shared stack → Cognito | Test user password |
+
+#### Run
+
+```bash
+bun x vitest run test/e2e-public-sdk.test.ts
+```
+
+#### What each test proves
+
+| # | Area | Test | What it validates | How to verify manually |
+|---|---|---|---|---|
+| 1 | **Login** | `should authenticate testdeveloper` | USER_PASSWORD_AUTH works with frontend client | AWS Console → Cognito → User Pool → Users → `testdeveloper` exists in `developer` group |
+| 2 | **Login** | `should extract tenant from JWT` | `custom:tenant` claim is present in ID token | Decode the JWT at [jwt.io](https://jwt.io) — look for `custom:tenant` |
+| 3 | **Login** | `should include resource server scopes` | Pre-token Lambda injects `dataworks.live/read` and `/write` | AWS Console → Cognito → User Pool → Lambda triggers → Pre token generation V2 is set |
+| 4 | **Login** | `should reject invalid credentials` | Bad password returns error, not a token | Try `dataworks.login("testdeveloper", "wrong")` — should throw |
+| 5 | **Ingest** | `should ingest valid metrics` | JWT passes API Gateway authoriser, Lambda processes payload | AWS Console → CloudWatch → Log group for the ingest Lambda — look for the metric |
+| 6 | **Ingest** | `should filter invalid metrics` | Client-side validation drops bad metrics before sending | The `stderr` output shows `Dropping invalid metric` — local validation, no server call for bad data |
+| 7 | **Ingest** | `should throw when not authenticated` | `ingest()` before `login()` throws immediately | Client-side guard — no network call made |
+| 8 | **Error** | `should reach error endpoint` | Auth + API Gateway routing work (500 = Lambda ran, FK failed) | AWS Console → CloudWatch → Log group for error processor Lambda |
+| 9 | **Error** | `should throw when not authenticated` | `reportError()` before `login()` throws immediately | Client-side guard |
+| 10 | **Subscribe** | `should throw when not authenticated` | `subscribe()` before `login()` throws immediately | Client-side guard |
+| 11 | **Subscribe** | `should create a subscription` | **Skipped in Node.js** (no `WebSocket` global) — see below | Use the Python verification script |
+| 12–14 | **Validation** | `validateMetric` / `filterValidMetrics` | Static validation utilities work correctly | Pure functions — no infra dependency |
+
+#### Verifying WebSocket subscribe (test 11)
+
+The subscribe test is skipped in the Node.js Vitest environment because `globalThis.WebSocket` is not available. To prove subscribe works end-to-end, use the standalone Python script:
+
+```bash
+pip install websocket-client requests
+python test/verify-subscribe.py
+```
+
+This script:
+1. Authenticates via Cognito USER_PASSWORD_AUTH (same as the SDK)
+2. Opens a WebSocket to the AppSync Events API with base64URL-encoded JWT auth subprotocols
+3. Sends `connection_init` and receives `connection_ack` from AppSync
+4. Sends a `subscribe` message and receives `subscribe_success`
+5. Closes cleanly
+
+Expected output:
+```
+Loading config from test.env
+
+1. Authenticating as 'testdeveloper'...
+   ✓ Login successful
+   Scopes: aws.cognito.signin.user.admin https://dataworks.live/write https://dataworks.live/read
+   Tenant: 2x9AOzBvDBgllLB0CXcCrPjK03C
+
+2. Connecting to wss://event-api-chris-dev.dataworks.live/event/realtime...
+   Protocol: aws-appsync-event-ws (manual)
+   Received: connection_ack
+   ✓ WebSocket connection established
+
+3. Subscribing to channel '/dataworks/e2e-verify'...
+   Received: subscribe_success
+   ✓ Subscribed to /dataworks/e2e-verify
+
+4. WebSocket closed cleanly
+
+============================================================
+ALL CHECKS PASSED — subscribe flow works end-to-end
+============================================================
+```
 
 ## License