@mnemom/aip-otel-exporter 0.1.0

package/README.md

@@ -0,0 +1,172 @@
+# @mnemom/aip-otel-exporter
+
+[![npm](https://img.shields.io/npm/v/@mnemom/aip-otel-exporter)](https://www.npmjs.com/package/@mnemom/aip-otel-exporter)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](../../LICENSE)
+
+OpenTelemetry exporter for [AIP](https://github.com/mnemom/aip) integrity checkpoints and [AAP](https://github.com/mnemom/aap) verification results.
+
+Send AIP/AAP telemetry to any OTel-compatible observability platform (Langfuse, Arize Phoenix, Datadog, Grafana) with zero custom code.
+
+## Three Layers
+
+| Layer | Import | OTel SDK Required | Use Case |
+|---|---|---|---|
+| **Manual API** | `@mnemom/aip-otel-exporter` | Yes | Works everywhere with OTel SDK |
+| **Auto-instrumentation** | `@mnemom/aip-otel-exporter/auto` | Yes | Wraps AIP/AAP calls automatically (Node.js) |
+| **CF Workers adapter** | `@mnemom/aip-otel-exporter/workers` | No | Cloudflare Workers (no OTel SDK needed) |
+
+## Installation
+
+```bash
+npm install @mnemom/aip-otel-exporter
+# Peer dependency (optional — required for Manual API and Auto-instrumentation):
+npm install @opentelemetry/api
+```
+
+## Quick Start
+
+### Manual API
+
+```typescript
+import { NodeTracerProvider } from "@opentelemetry/sdk-trace-base";
+import { createAIPOTelRecorder } from "@mnemom/aip-otel-exporter";
+
+const provider = new NodeTracerProvider();
+// ... configure your OTLP exporter (Langfuse, Datadog, etc.)
+provider.register();
+
+const recorder = createAIPOTelRecorder({ tracerProvider: provider });
+
+// After an AIP integrity check:
+recorder.recordIntegrityCheck(signal);
+
+// After an AAP verification:
+recorder.recordVerification(result);
+
+// After an AAP coherence check:
+recorder.recordCoherence(result);
+
+// After AAP drift detection:
+recorder.recordDrift(alerts, tracesAnalyzed);
+```
+
+### Auto-instrumentation (Node.js)
+
+```typescript
+import { instrument } from "@mnemom/aip-otel-exporter/auto";
+
+// Automatically wraps AIPClient.check() and verifyTrace()
+instrument();
+```
+
+### Cloudflare Workers
+
+```typescript
+import { createWorkersExporter } from "@mnemom/aip-otel-exporter/workers";
+
+export default {
+  async fetch(request, env, ctx) {
+    const exporter = createWorkersExporter({
+      endpoint: env.OTLP_ENDPOINT,
+      authorization: `Bearer ${env.OTLP_TOKEN}`,
+    });
+
+    // ... your handler logic ...
+    exporter.recordIntegrityCheck(signal);
+
+    ctx.waitUntil(exporter.flush());
+    return new Response("OK");
+  },
+};
+```
+
+## Span Hierarchy
+
+Integrity/alignment spans are children of the current active span:
+
+```
+invoke_agent (existing)
+  └── aip.integrity_check (our span)
+       ├── event: aip.concern (one per concern)
+       └── event: aip.drift_alert (conditional)
+  └── aap.verify_trace (our span)
+       └── event: aap.violation (one per violation)
+```
+
+## Attributes
+
+### `aip.integrity_check` span
+
+| Attribute | Type | Description |
+|---|---|---|
+| `aip.integrity.checkpoint_id` | string | Checkpoint ID |
+| `aip.integrity.verdict` | string | clear / review_needed / boundary_violation |
+| `aip.integrity.proceed` | boolean | Whether to proceed |
+| `aip.integrity.recommended_action` | string | Recommended action |
+| `aip.integrity.concerns_count` | int | Number of concerns |
+| `aip.integrity.agent_id` | string | Agent ID |
+| `aip.integrity.card_id` | string | Alignment Card ID |
+| `aip.integrity.session_id` | string | Session ID |
+| `aip.integrity.thinking_hash` | string | SHA-256 of thinking block |
+| `aip.integrity.analysis_model` | string | Analysis LLM model |
+| `aip.integrity.analysis_duration_ms` | float | Analysis duration |
+| `aip.integrity.thinking_tokens` | int | Original thinking tokens |
+| `aip.integrity.truncated` | boolean | Whether thinking was truncated |
+| `aip.integrity.extraction_confidence` | float | Extraction confidence |
+| `aip.conscience.consultation_depth` | string | surface / standard / deep |
+| `aip.conscience.values_checked_count` | int | Values checked count |
+| `aip.conscience.conflicts_count` | int | Conflicts count |
+| `aip.window.size` | int | Window size |
+| `aip.window.integrity_ratio` | float | Integrity ratio (0.0-1.0) |
+| `aip.window.drift_alert_active` | boolean | Drift alert active |
+| `gen_ai.evaluation.verdict` | string | GenAI SIG alias |
+| `gen_ai.evaluation.score` | float | GenAI SIG alias |
+
+### `aap.verify_trace` span
+
+| Attribute | Type |
+|---|---|
+| `aap.verification.result` | boolean |
+| `aap.verification.similarity_score` | float |
+| `aap.verification.violations_count` | int |
+| `aap.verification.warnings_count` | int |
+| `aap.verification.trace_id` | string |
+| `aap.verification.card_id` | string |
+| `aap.verification.duration_ms` | float |
+| `aap.verification.checks_performed` | string |
+
+## Metrics
+
+| Metric | Type | Labels |
+|---|---|---|
+| `aip.integrity_checks.total` | Counter | verdict, agent_id |
+| `aip.concerns.total` | Counter | category, severity |
+| `aip.analysis.duration_ms` | Histogram | verdict |
+| `aip.window.integrity_ratio` | Histogram | — |
+| `aip.drift_alerts.total` | Counter | — |
+| `aap.verifications.total` | Counter | verified |
+| `aap.violations.total` | Counter | type, severity |
+| `aap.verification.duration_ms` | Histogram | — |
+| `aap.coherence.score` | Histogram | compatible |
+
+## Dashboard Templates
+
+Pre-built dashboards are included in the `dashboards/` directory:
+
+- `grafana-aip-overview.json` — Grafana system overview
+- `grafana-aip-detail.json` — Grafana per-agent deep-dive
+- `datadog-aip-overview.json` — Datadog importable dashboard
+
+See `dashboards/README.md` for import instructions.
+
+## Performance
+
+| Operation | Target | Actual |
+|---|---|---|
+| `recordIntegrityCheck()` | <1ms | Run `npm run bench` |
+| `recordVerification()` | <1ms | Run `npm run bench` |
+| Workers OTLP serialize | <0.5ms | Run `npm run bench` |
+
+## License
+
+[Apache 2.0](../../LICENSE)

package/dashboards/README.md

@@ -0,0 +1,92 @@
+# Dashboard Templates
+
+Pre-built dashboard templates for visualizing AIP integrity checks and AAP verification results exported via `@mnemom/aip-otel-exporter`.
+
+## Available Dashboards
+
+| File | Platform | Description |
+|------|----------|-------------|
+| `grafana-aip-overview.json` | Grafana | Fleet-wide overview: verdict distribution, check rate, integrity ratio, concerns, AAP pass rate |
+| `grafana-aip-detail.json` | Grafana | Per-agent deep dive: integrity ratio timeline, concern breakdown, verdict history, top concerns |
+| `datadog-aip-overview.json` | Datadog | Fleet-wide overview using Datadog widget types: sunburst, timeseries, toplist, query_value |
+
+## Importing into Grafana
+
+### Via the Grafana UI
+
+1. Open your Grafana instance.
+2. Navigate to **Dashboards > New > Import**.
+3. Click **Upload dashboard JSON file** and select the desired `.json` file.
+4. Select your Prometheus data source when prompted (the dashboards use a `DS_PROMETHEUS` input variable).
+5. Click **Import**.
+
+### Via the Grafana API
+
+```bash
+curl -X POST \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <your-grafana-api-key>" \
+  -d @grafana-aip-overview.json \
+  https://<your-grafana-host>/api/dashboards/import
+```
+
+Wrap the dashboard JSON in the import envelope if your Grafana version requires it:
+
+```bash
+jq '{ dashboard: ., overwrite: true, inputs: [{ name: "DS_PROMETHEUS", type: "datasource", pluginId: "prometheus", value: "your-prometheus-uid" }] }' \
+  grafana-aip-overview.json | \
+curl -X POST \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <your-grafana-api-key>" \
+  -d @- \
+  https://<your-grafana-host>/api/dashboards/import
+```
+
+## Importing into Datadog
+
+### Via the Datadog API
+
+```bash
+curl -X POST \
+  -H "Content-Type: application/json" \
+  -H "DD-API-KEY: <your-datadog-api-key>" \
+  -H "DD-APPLICATION-KEY: <your-datadog-app-key>" \
+  -d @datadog-aip-overview.json \
+  https://api.datadoghq.com/api/v1/dashboard
+```
+
+For EU-region accounts, use `api.datadoghq.eu` instead of `api.datadoghq.com`.
+
+## Metric Name Mapping
+
+The `@mnemom/aip-otel-exporter` package emits metrics with dotted names (e.g., `aip.integrity_checks.total`). How these appear in your observability backend depends on the backend:
+
+| Backend | Metric name format | Example |
+|---------|-------------------|---------|
+| **Prometheus** / **Grafana** | Dots replaced with underscores | `aip_integrity_checks_total` |
+| **Datadog** | Dots preserved | `aip.integrity_checks.total` |
+| **Langfuse** | Trace attributes only (no metrics) | `aip.integrity.verdict` span attribute |
+
+The Grafana dashboards use underscore-separated names (`aip_integrity_checks_total`) to match Prometheus conventions. The Datadog dashboard uses dot-separated names (`aip.integrity_checks.total`) to match Datadog conventions.
+
+## Metrics Reference
+
+| Metric | Type | Labels | Description |
+|--------|------|--------|-------------|
+| `aip.integrity_checks.total` | Counter | `verdict`, `agent_id` | Total AIP integrity checks |
+| `aip.concerns.total` | Counter | `category`, `severity` | Total integrity concerns raised |
+| `aip.analysis.duration_ms` | Histogram | | Analysis duration in milliseconds |
+| `aip.window.integrity_ratio` | Histogram | | Sliding window integrity ratio (0-1) |
+| `aip.drift_alerts.total` | Counter | `severity` | AIP drift alerts |
+| `aap.verifications.total` | Counter | `verified` | Total AAP trace verifications |
+| `aap.violations.total` | Counter | `type`, `severity` | Total AAP violations |
+| `aap.verification.duration_ms` | Histogram | | AAP verification duration in milliseconds |
+| `aap.coherence.score` | Histogram | | AAP coherence scores |
+
+## Customization
+
+These dashboards are starting points. Common modifications:
+
+- **Add alerting rules**: Create Grafana alerts on integrity ratio dropping below a threshold, or Datadog monitors on violation spikes.
+- **Filter by environment**: Add a `$environment` template variable and filter metrics by an `environment` label if you tag your OTel resources with deployment environment.
+- **Add additional agents**: The Grafana detail dashboard uses a single `$agent_id` selector. Switch it to multi-select if you want to compare agents side by side.

package/dashboards/datadog-aip-overview.json

@@ -0,0 +1,311 @@
+{
+  "title": "AIP Integrity Overview",
+  "description": "Overview dashboard for AIP integrity checks and AAP verification results exported via @mnemom/aip-otel-exporter.",
+  "layout_type": "ordered",
+  "is_read_only": false,
+  "notify_list": [],
+  "reflow_type": "fixed",
+  "tags": ["aip", "integrity", "otel"],
+  "widgets": [
+    {
+      "definition": {
+        "title": "Integrity Checks",
+        "type": "group",
+        "layout_type": "ordered",
+        "widgets": [
+          {
+            "definition": {
+              "title": "Integrity Verdict Distribution",
+              "type": "sunburst",
+              "requests": [
+                {
+                  "formulas": [{ "formula": "query1" }],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aip.integrity_checks.total{*} by {verdict}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ]
+            },
+            "layout": { "x": 0, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Integrity Checks per Minute",
+              "type": "timeseries",
+              "requests": [
+                {
+                  "formulas": [{ "formula": "query1" }],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aip.integrity_checks.total{*} by {verdict}.as_rate()",
+                      "aggregator": "avg"
+                    }
+                  ],
+                  "response_format": "timeseries",
+                  "display_type": "line"
+                }
+              ],
+              "yaxis": { "label": "checks/min", "include_zero": true }
+            },
+            "layout": { "x": 4, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Current Integrity Ratio",
+              "type": "query_value",
+              "requests": [
+                {
+                  "formulas": [
+                    {
+                      "formula": "query1 / (query1 + query2 + query3)",
+                      "limit": { "count": 1, "order": "desc" }
+                    }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aip.integrity_checks.total{verdict:clear}",
+                      "aggregator": "sum"
+                    },
+                    {
+                      "data_source": "metrics",
+                      "name": "query2",
+                      "query": "sum:aip.integrity_checks.total{verdict:review_needed}",
+                      "aggregator": "sum"
+                    },
+                    {
+                      "data_source": "metrics",
+                      "name": "query3",
+                      "query": "sum:aip.integrity_checks.total{verdict:boundary_violation}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ],
+              "autoscale": false,
+              "precision": 2,
+              "custom_unit": "%",
+              "conditional_formats": [
+                { "comparator": ">=", "value": 0.95, "palette": "white_on_green" },
+                { "comparator": ">=", "value": 0.85, "palette": "white_on_yellow" },
+                { "comparator": "<", "value": 0.85, "palette": "white_on_red" }
+              ]
+            },
+            "layout": { "x": 8, "y": 0, "width": 4, "height": 3 }
+          }
+        ]
+      },
+      "layout": { "x": 0, "y": 0, "width": 12, "height": 4 }
+    },
+    {
+      "definition": {
+        "title": "Concerns & Analysis",
+        "type": "group",
+        "layout_type": "ordered",
+        "widgets": [
+          {
+            "definition": {
+              "title": "Concerns by Category",
+              "type": "toplist",
+              "requests": [
+                {
+                  "formulas": [
+                    {
+                      "formula": "query1",
+                      "limit": { "count": 10, "order": "desc" }
+                    }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aip.concerns.total{*} by {category}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ]
+            },
+            "layout": { "x": 0, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Concerns by Severity",
+              "type": "toplist",
+              "requests": [
+                {
+                  "formulas": [
+                    {
+                      "formula": "query1",
+                      "limit": { "count": 10, "order": "desc" }
+                    }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aip.concerns.total{*} by {severity}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ]
+            },
+            "layout": { "x": 4, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Analysis Duration (p50 / p95 / p99)",
+              "type": "timeseries",
+              "requests": [
+                {
+                  "formulas": [
+                    { "formula": "query1", "alias": "p50" },
+                    { "formula": "query2", "alias": "p95" },
+                    { "formula": "query3", "alias": "p99" }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "p50:aip.analysis.duration_ms{*}",
+                      "aggregator": "avg"
+                    },
+                    {
+                      "data_source": "metrics",
+                      "name": "query2",
+                      "query": "p95:aip.analysis.duration_ms{*}",
+                      "aggregator": "avg"
+                    },
+                    {
+                      "data_source": "metrics",
+                      "name": "query3",
+                      "query": "p99:aip.analysis.duration_ms{*}",
+                      "aggregator": "avg"
+                    }
+                  ],
+                  "response_format": "timeseries",
+                  "display_type": "line"
+                }
+              ],
+              "yaxis": { "label": "ms", "include_zero": true }
+            },
+            "layout": { "x": 8, "y": 0, "width": 4, "height": 3 }
+          }
+        ]
+      },
+      "layout": { "x": 0, "y": 4, "width": 12, "height": 4 }
+    },
+    {
+      "definition": {
+        "title": "AAP Verification",
+        "type": "group",
+        "layout_type": "ordered",
+        "widgets": [
+          {
+            "definition": {
+              "title": "Verification Pass Rate",
+              "type": "query_value",
+              "requests": [
+                {
+                  "formulas": [
+                    {
+                      "formula": "query1 / (query1 + query2)",
+                      "limit": { "count": 1, "order": "desc" }
+                    }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aap.verifications.total{verified:true}",
+                      "aggregator": "sum"
+                    },
+                    {
+                      "data_source": "metrics",
+                      "name": "query2",
+                      "query": "sum:aap.verifications.total{verified:false}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ],
+              "autoscale": false,
+              "precision": 2,
+              "custom_unit": "%",
+              "conditional_formats": [
+                { "comparator": ">=", "value": 0.98, "palette": "white_on_green" },
+                { "comparator": ">=", "value": 0.9, "palette": "white_on_yellow" },
+                { "comparator": "<", "value": 0.9, "palette": "white_on_red" }
+              ]
+            },
+            "layout": { "x": 0, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Violations by Type",
+              "type": "toplist",
+              "requests": [
+                {
+                  "formulas": [
+                    {
+                      "formula": "query1",
+                      "limit": { "count": 10, "order": "desc" }
+                    }
+                  ],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aap.violations.total{*} by {type}",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "scalar"
+                }
+              ]
+            },
+            "layout": { "x": 4, "y": 0, "width": 4, "height": 3 }
+          },
+          {
+            "definition": {
+              "title": "Violations Over Time by Severity",
+              "type": "timeseries",
+              "requests": [
+                {
+                  "formulas": [{ "formula": "query1" }],
+                  "queries": [
+                    {
+                      "data_source": "metrics",
+                      "name": "query1",
+                      "query": "sum:aap.violations.total{*} by {severity}.as_count()",
+                      "aggregator": "sum"
+                    }
+                  ],
+                  "response_format": "timeseries",
+                  "display_type": "bars"
+                }
+              ],
+              "yaxis": { "label": "violations", "include_zero": true }
+            },
+            "layout": { "x": 8, "y": 0, "width": 4, "height": 3 }
+          }
+        ]
+      },
+      "layout": { "x": 0, "y": 8, "width": 12, "height": 4 }
+    }
+  ]
+}