@manifest-cyber/observability-ts 0.2.0 → 0.2.2

package/TRACING_IMPLEMENTATION_PLAN.md

@@ -1,758 +0,0 @@
-# Distributed Tracing Implementation Plan for Manifest Cyber
-
-**Status:** Implementation Ready  
-**Date:** November 11, 2025  
-**Scope:** End-to-end distributed tracing across web-app, manifest-api, message queues, and job-\* services
-
----
-
-## Executive Summary
-
-This document outlines the comprehensive approach to implement distributed tracing across Manifest Cyber's event-driven architecture using OpenTelemetry (OTEL) standards, supporting both SaaS (ECS) and on-premise (K8s) deployments.
-
-### Architecture Overview
-
-```
-┌──────────┐    HTTP     ┌──────────────┐    SQS/RabbitMQ    ┌────────────┐
-│ web-app  │ ────────►   │ manifest-api │ ──────────────────► │  job-*     │
-│ (React)  │             │  (Express)   │                     │ (Workers)  │
-└──────────┘             └──────────────┘                     └────────────┘
-     │                          │                                    │
-     │ W3C Traceparent          │ W3C Traceparent                    │
-     │ via HTTP Headers         │ via Message Attributes             │
-     │                          │                                    │
-     └──────────────────────────┴────────────────────────────────────┘
-                                 │
-                                 ▼
-                         ┌────────────────┐
-                         │  Vector        │
-                         │  (Sidecar)     │
-                         └────────────────┘
-                                 │
-                                 ▼
-                         ┌────────────────┐
-                         │ VictoriaTraces │
-                         │ (OTLP 4317/18) │
-                         └────────────────┘
-                                 │
-                                 ▼
-                         ┌────────────────┐
-                         │    Grafana     │
-                         │   (Tempo UI)   │
-                         └────────────────┘
-```
-
----
-
-## Current State Analysis
-
-### ✅ What's Already Working
-
-1. **Logger-ts with Trace Support**
-   - `TraceContext` class with trace_id, span_id, parent_id
-   - `OtelLogger` with automatic trace extraction
-   - W3C traceparent parsing in messaging layer
-   - AsyncLocalStorage for context propagation
-
-2. **Messaging Infrastructure (job-common)**
-   - W3C traceparent creation/parsing utilities
-   - Trace context propagation via message attributes
-   - Integration with logger-ts for trace-aware logging
-
-3. **Observability Stack**
-   - Vector deployed as ECS sidecar
-   - VictoriaTraces endpoint: `https://victoria-traces.development.manifestcyber.dev`
-   - OTLP ports: 4317 (gRPC), 4318 (HTTP)
-   - Grafana for visualization
-
-4. **Metrics Infrastructure**
-   - Prometheus metrics collection via Vector
-   - VictoriaMetrics storage
-   - Service-level instrumentation pattern established
-
-### 🔴 What's Missing
-
-1. **OpenTelemetry Instrumentation**
-   - No OTEL SDK integration in any service
-   - No automatic span creation for HTTP requests
-   - No automatic span creation for queue operations
-   - No OTLP exporter configured
-
-2. **Trace Propagation Standards**
-   - No standardized baggage/tracestate handling
-   - No sampling strategy defined
-   - No trace ID generation at entry points
-
-3. **Service-Level Integration**
-   - web-app: No trace injection in HTTP calls
-   - manifest-api: No auto-instrumentation
-   - job-\*: No trace continuation from messages
-
-4. **Infrastructure Configuration**
-   - Vector not configured for OTLP ingestion
-   - No trace collection pipeline defined
-   - K8s on-prem configuration missing
-
----
-
-## Technical Architecture
-
-### 1. Trace Flow: Web App → API → Queue → Job
-
-#### Entry Point: User-Initiated (web-app)
-
-```typescript
-// web-app/src/api/client.ts
-import { trace, context } from '@opentelemetry/api';
-
-const span = tracer.startSpan('http.request', {
-  attributes: {
-    'http.method': 'POST',
-    'http.url': '/api/sboms/upload',
-  }
-});
-
-context.with(trace.setSpan(context.active(), span), () => {
-  axios.post('/api/sboms/upload', data, {
-    headers: {
-      traceparent: // Auto-injected by OTEL instrumentation
-    }
-  });
-});
-```
-
-#### Ingress: API Receives Request (manifest-api)
-
-```typescript
-// manifest-api auto-instrumentation extracts traceparent
-// Express middleware creates span automatically
-// Pass to logger:
-const logger = OtelLogger.create(baseLogger, { autoTrace: true });
-logger.info('Processing SBOM upload', { userId, organizationId });
-```
-
-#### Egress: API → Queue (manifest-api)
-
-```typescript
-// Use job-common messaging with trace context
-import { createTraceparentFromLogger } from '@manifest-cyber/job-common';
-
-const traceparent = createTraceparentFromLogger(logger);
-await messageClient.send('sbom-process-queue', {
-  sbomId,
-  organizationId,
-  traceContext: { traceparent },
-});
-```
-
-#### Ingress: Job Receives Message (job-sbom-process)
-
-```typescript
-// job-common automatically extracts trace context
-import { createLoggerWithMessageTrace } from '@manifest-cyber/job-common';
-
-const logger = createLoggerWithMessageTrace(message, baseLogger);
-// Logger now has trace context from message
-```
-
-#### Entry Point: System-Initiated (cron job)
-
-```typescript
-// job-daily-vuln-match/src/invoke.ts
-import { TraceContext } from '@manifest-cyber/logger-ts';
-
-const traceContext = TraceContext.generateRandom();
-const logger = OtelLogger.create(baseLogger, {
-  fields: { trace_id: traceContext.trace_id, span_id: traceContext.span_id },
-});
-```
-
-### 2. OpenTelemetry SDK Integration
-
-#### Core Dependencies
-
-```json
-{
-  "@opentelemetry/sdk-node": "^0.54.0",
-  "@opentelemetry/auto-instrumentations-node": "^0.51.0",
-  "@opentelemetry/exporter-trace-otlp-grpc": "^0.54.0",
-  "@opentelemetry/resources": "^1.28.0",
-  "@opentelemetry/semantic-conventions": "^1.28.0"
-}
-```
-
-#### Initialization Pattern (Backend Services)
-
-```typescript
-// src/tracing.ts
-import { NodeSDK } from '@opentelemetry/sdk-node';
-import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
-import { Resource } from '@opentelemetry/resources';
-import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
-import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
-
-const sdk = new NodeSDK({
-  resource: new Resource({
-    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || 'unknown-service',
-    environment: process.env.ENV || 'development',
-    'service.version': process.env.VERSION || '0.0.0',
-  }),
-  traceExporter: new OTLPTraceExporter({
-    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317',
-  }),
-  instrumentations: [
-    getNodeAutoInstrumentations({
-      '@opentelemetry/instrumentation-http': { enabled: true },
-      '@opentelemetry/instrumentation-express': { enabled: true },
-      '@opentelemetry/instrumentation-mongodb': { enabled: true },
-      '@opentelemetry/instrumentation-aws-sdk': { enabled: true },
-    }),
-  ],
-});
-
-sdk.start();
-
-process.on('SIGTERM', () => {
-  sdk.shutdown().finally(() => process.exit(0));
-});
-```
-
-### 3. Message Queue Trace Propagation
-
-#### W3C Traceparent Format
-
-```
-00-{trace-id:32-hex}-{span-id:16-hex}-{flags:2-hex}
-Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
-```
-
-#### SQS Message Attributes
-
-```typescript
-// Producer (manifest-api)
-await sqs.sendMessage({
-  QueueUrl: queueUrl,
-  MessageBody: JSON.stringify(payload),
-  MessageAttributes: {
-    traceparent: {
-      DataType: 'String',
-      StringValue: traceparent, // W3C format
-    },
-    // Optional: Add tracestate for baggage
-    tracestate: {
-      DataType: 'String',
-      StringValue: 'key1=value1,key2=value2',
-    },
-  },
-});
-```
-
-#### RabbitMQ Message Properties
-
-```typescript
-// Producer (manifest-api)
-await channel.sendToQueue(queueName, Buffer.from(JSON.stringify(payload)), {
-  headers: {
-    traceparent: traceparent, // W3C format
-    tracestate: 'key1=value1,key2=value2', // Optional
-  },
-  persistent: true,
-});
-```
-
-### 4. Sampling Strategy
-
-#### Development/Staging
-
-- **Rate:** 100% (always sample)
-- **Reason:** Full visibility for debugging
-
-#### Production
-
-- **Rate:** 10% (probabilistic sampling)
-- **Head-based sampling:** Decision at trace root
-- **Override:** Always sample errors (500 status, exceptions)
-
-```typescript
-import {
-  ParentBasedSampler,
-  TraceIdRatioBasedSampler,
-} from '@opentelemetry/sdk-trace-base';
-
-const sampler = new ParentBasedSampler({
-  root: new TraceIdRatioBasedSampler(process.env.ENV === 'production' ? 0.1 : 1.0),
-});
-```
-
----
-
-## Implementation Roadmap
-
-### Phase 1: Core Library Support (Week 1) ✅ COMPLETE
-
-**Goal:** Create unified observability library with metrics + tracing
-
-- [x] Rename package to `@manifest-cyber/observability-ts`
-- [x] Add OpenTelemetry dependencies
-- [x] Create src/tracing/ module with SDK initialization helpers
-- [x] Create span management utilities (spans.ts)
-- [x] Create context propagation helpers (context.ts)
-- [x] Reorganize metrics into src/metrics/ for modularity
-- [x] Configure subpath exports for tree-shaking
-- [x] Update documentation with tracing examples
-
-**Deliverables:**
-
-- `@manifest-cyber/observability-ts` v0.2.0 published
-- Deprecated `@manifest-cyber/metrics` on NPM
-- [TRACING_GUIDE.md](./TRACING_GUIDE.md) - Complete tracing documentation
-- [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md) - Migration instructions
-- Backward-compatible metrics API
-
-### Phase 2: Pilot Service (manifest-api) (Week 2)
-
-**Goal:** End-to-end tracing for one critical service
-
-**Tasks:**
-
-1. Initialize OTEL SDK in manifest-api
-2. Configure OTLP exporter (Vector/VictoriaTraces)
-3. Enable auto-instrumentation (HTTP, Express, MongoDB)
-4. Add custom spans for business logic
-5. Test trace propagation to downstream jobs
-
-**Success Criteria:**
-
-- Traces visible in Grafana
-- HTTP spans include request metadata
-- Traces propagate through SQS to job-sbom-process
-- Logs include trace_id/span_id
-
-### Phase 3: Message Queue Jobs (Week 3)
-
-**Goal:** Instrument 3 high-volume jobs
-
-**Services:**
-
-- job-sbom-process
-- job-vulnerability-match
-- job-component-process
-
-**Tasks:**
-
-1. Add OTEL SDK initialization to each job
-2. Update message consumers to extract trace context
-3. Create spans for processing stages
-4. Link spans to parent API traces
-5. Add error tracking with span status
-
-**Success Criteria:**
-
-- Full trace visibility from API → Queue → Job
-- Job spans linked to API spans
-- Error traces captured
-- Average trace latency <500ms overhead
-
-### Phase 4: Web App (Week 4)
-
-**Goal:** Frontend trace initiation
-
-**Tasks:**
-
-1. Add @opentelemetry/web SDK
-2. Configure trace initiation on user actions
-3. Inject traceparent in HTTP requests
-4. Add resource timing spans
-5. Configure sampling (1% production)
-
-**Success Criteria:**
-
-- User action → API trace correlation
-- Performance metrics captured
-- Minimal bundle size impact (<50KB)
-
-### Phase 5: Infrastructure & Rollout (Week 5-6)
-
-**Goal:** Production-ready infrastructure
-
-**Tasks:**
-
-1. Configure Vector for OTLP ingestion (ECS)
-2. Deploy OTEL collector for K8s
-3. Configure Grafana Tempo datasource
-4. Create trace dashboards
-5. Instrument remaining 20+ services
-6. Production rollout with 10% sampling
-
-**Success Criteria:**
-
-- 100% service coverage
-- <5% trace data loss
-- Query latency <3s in Grafana
-- Cost <$200/month trace storage
-
----
-
-## Deployment Considerations
-
-### ECS (SaaS) Configuration
-
-#### Vector Sidecar Update
-
-```yaml
-# infrastructure/vector/vector-config.yaml
-sources:
-  otlp_grpc:
-    type: 'opentelemetry'
-    address: '0.0.0.0:4317'
-    grpc:
-      enabled: true
-
-  otlp_http:
-    type: 'opentelemetry'
-    address: '0.0.0.0:4318'
-    http:
-      enabled: true
-
-sinks:
-  victoria_traces:
-    type: 'opentelemetry'
-    inputs: ['otlp_grpc', 'otlp_http']
-    endpoint: 'https://victoria-traces.development.manifestcyber.dev'
-    compression: 'gzip'
-    batch:
-      max_events: 1000
-      timeout_secs: 10
-```
-
-#### ECS Task Environment Variables
-
-```bash
-OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
-OTEL_SERVICE_NAME=${SERVICE_NAME}
-OTEL_RESOURCE_ATTRIBUTES=environment=${ENV},cluster=${CLUSTER_NAME}
-OTEL_TRACES_SAMPLER=parentbased_traceidratio
-OTEL_TRACES_SAMPLER_ARG=0.1  # 10% in production
-```
-
-### Kubernetes (On-Prem) Configuration
-
-#### OTEL Collector DaemonSet
-
-```yaml
-# k3s-on-prem/kustomize/apps/otel-collector/daemonset.yaml
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: otel-collector
-  namespace: observability
-spec:
-  selector:
-    matchLabels:
-      app: otel-collector
-  template:
-    metadata:
-      labels:
-        app: otel-collector
-    spec:
-      containers:
-        - name: otel-collector
-          image: otel/opentelemetry-collector-contrib:latest
-          ports:
-            - containerPort: 4317 # OTLP gRPC
-              hostPort: 4317
-            - containerPort: 4318 # OTLP HTTP
-              hostPort: 4318
-          volumeMounts:
-            - name: config
-              mountPath: /etc/otel
-          env:
-            - name: VICTORIA_TRACES_ENDPOINT
-              value: 'http://victoria-traces.observability.svc.cluster.local:4317'
-      volumes:
-        - name: config
-          configMap:
-            name: otel-collector-config
-```
-
-#### OTEL Collector ConfigMap
-
-```yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: otel-collector-config
-  namespace: observability
-data:
-  config.yaml: |
-    receivers:
-      otlp:
-        protocols:
-          grpc:
-            endpoint: 0.0.0.0:4317
-          http:
-            endpoint: 0.0.0.0:4318
-
-    processors:
-      batch:
-        timeout: 10s
-        send_batch_size: 1024
-      
-      resource:
-        attributes:
-        - key: deployment.environment
-          value: ${env:ENVIRONMENT}
-          action: insert
-
-    exporters:
-      otlp:
-        endpoint: victoria-traces.observability.svc.cluster.local:4317
-        tls:
-          insecure: true
-
-    service:
-      pipelines:
-        traces:
-          receivers: [otlp]
-          processors: [batch, resource]
-          exporters: [otlp]
-```
-
----
-
-## Best Practices & Patterns
-
-### 1. Span Naming Conventions
-
-Follow OpenTelemetry semantic conventions:
-
-```typescript
-// ✅ Good
-tracer.startSpan('http.client.request', {
-  attributes: {
-    'http.method': 'POST',
-    'http.url': '/api/users',
-    'http.status_code': 200,
-  },
-});
-
-// ❌ Bad
-tracer.startSpan('API Call', {
-  attributes: { url: '/api/users' },
-});
-```
-
-### 2. Error Handling
-
-Always set span status on errors:
-
-```typescript
-try {
-  await processData();
-  span.setStatus({ code: SpanStatusCode.OK });
-} catch (error) {
-  span.setStatus({
-    code: SpanStatusCode.ERROR,
-    message: error.message,
-  });
-  span.recordException(error);
-  throw error;
-} finally {
-  span.end();
-}
-```
-
-### 3. Cardinality Management
-
-```typescript
-// ✅ Good - bounded cardinality
-span.setAttribute('http.status_code', 200);
-span.setAttribute('user.role', 'admin');
-
-// ❌ Bad - unbounded cardinality
-span.setAttribute('user.id', 'user-12345-abcde-...'); // Too unique
-span.setAttribute('request.body', JSON.stringify(body)); // Too large
-```
-
-### 4. Context Propagation in Async Operations
-
-```typescript
-import { context } from '@opentelemetry/api';
-
-async function processInBackground(data: any) {
-  // Capture current context
-  const currentContext = context.active();
-
-  // Run in background with context
-  setTimeout(() => {
-    context.with(currentContext, () => {
-      const span = tracer.startSpan('background.process');
-      // Process data with trace context
-      span.end();
-    });
-  }, 1000);
-}
-```
-
----
-
-## Monitoring & Alerting
-
-### Key Metrics to Track
-
-1. **Trace Volume**
-   - `traces_received_total`
-   - `traces_dropped_total`
-   - `trace_export_errors_total`
-
-2. **Trace Latency**
-   - `trace_export_duration_seconds`
-   - `span_processing_duration_seconds`
-
-3. **Sampling Rates**
-   - `traces_sampled_ratio`
-   - `traces_head_sampled_total`
-
-### Alerts
-
-```yaml
-# vmalert/alerts/tracing.yaml
-groups:
-  - name: tracing
-    interval: 1m
-    rules:
-      - alert: HighTraceDropRate
-        expr: |
-          rate(traces_dropped_total[5m]) / rate(traces_received_total[5m]) > 0.05
-        for: 5m
-        annotations:
-          summary: 'High trace drop rate detected'
-          description: '{{ $value }}% of traces are being dropped'
-
-      - alert: TraceExportFailures
-        expr: rate(trace_export_errors_total[5m]) > 10
-        for: 5m
-        annotations:
-          summary: 'Trace export failures detected'
-```
-
----
-
-## Cost Analysis
-
-### Expected Costs (Production at Scale)
-
-| Component               | Monthly Cost | Notes                              |
-| ----------------------- | ------------ | ---------------------------------- |
-| VictoriaTraces Storage  | $50-100      | 7 days retention, 10% sampling     |
-| Vector CPU/Memory (ECS) | $30-50       | Sidecar overhead across tasks      |
-| OTEL Collector (K8s)    | $20-30       | DaemonSet across nodes             |
-| Network Transfer        | $10-20       | OTLP data to VictoriaTraces        |
-| **Total**               | **$110-200** | **Compared to $800/mo CloudWatch** |
-
-### Cost Optimization Strategies
-
-1. **Sampling:** 10% production, 100% dev/staging
-2. **Retention:** 7 days hot, 30 days cold (S3)
-3. **Filtering:** Drop health check traces
-4. **Compression:** gzip for OTLP payloads
-
----
-
-## Testing Strategy
-
-### Unit Tests
-
-```typescript
-import { trace, context } from '@opentelemetry/api';
-import { InMemorySpanExporter } from '@opentelemetry/sdk-trace-base';
-
-describe('Tracing', () => {
-  let exporter: InMemorySpanExporter;
-
-  beforeEach(() => {
-    exporter = new InMemorySpanExporter();
-    // Configure test tracer with in-memory exporter
-  });
-
-  it('should create span with correct attributes', () => {
-    const span = tracer.startSpan('test.operation');
-    span.setAttribute('test.key', 'value');
-    span.end();
-
-    const spans = exporter.getFinishedSpans();
-    expect(spans).toHaveLength(1);
-    expect(spans[0].attributes['test.key']).toBe('value');
-  });
-});
-```
-
-### Integration Tests
-
-```typescript
-describe('Trace Propagation', () => {
-  it('should propagate trace from API to job', async () => {
-    // 1. Create trace in API
-    const response = await request(app).post('/api/sboms/upload').send(sbomData);
-
-    const traceparent = response.headers['traceparent'];
-
-    // 2. Verify message has trace context
-    const message = await queueClient.receiveMessage();
-    expect(message.traceContext.traceparent).toBe(traceparent);
-
-    // 3. Verify job creates child span
-    const jobSpans = await getSpansForTrace(traceparent);
-    expect(jobSpans).toContainEqual(
-      expect.objectContaining({
-        name: 'job.process.sbom',
-        parentSpanId: extractSpanId(traceparent),
-      }),
-    );
-  });
-});
-```
-
----
-
-## Migration Checklist
-
-- [ ] **Week 1:** Implement tracing in observability-ts library
-- [ ] **Week 1:** Update job-common with OTEL integration
-- [ ] **Week 2:** Deploy to manifest-api (pilot)
-- [ ] **Week 2:** Verify traces in Grafana
-- [ ] **Week 3:** Instrument 3 priority jobs
-- [ ] **Week 3:** Test end-to-end trace propagation
-- [ ] **Week 4:** Add web-app frontend tracing
-- [ ] **Week 4:** Create trace dashboards in Grafana
-- [ ] **Week 5:** Configure Vector/OTEL collector
-- [ ] **Week 5:** Deploy to K8s staging
-- [ ] **Week 6:** Rollout to remaining services
-- [ ] **Week 6:** Production validation (30 days)
-
----
-
-## Success Criteria
-
-1. ✅ **100% Service Coverage:** All services emit traces
-2. ✅ **End-to-End Visibility:** Traces from web-app → job completion
-3. ✅ **Performance:** <500ms tracing overhead per request
-4. ✅ **Reliability:** <1% trace data loss
-5. ✅ **Cost:** <$200/month trace infrastructure
-6. ✅ **Adoption:** Team uses traces for 80% of incidents
-
----
-
-## References
-
-- [OpenTelemetry Docs](https://opentelemetry.io/docs/)
-- [W3C Trace Context](https://www.w3.org/TR/trace-context/)
-- [OTLP Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md)
-- [VictoriaMetrics Tracing](https://docs.victoriametrics.com/victorialogs/)
-- [Vector OTLP Source](https://vector.dev/docs/reference/configuration/sources/opentelemetry/)
-
----
-
-**Document Owner:** Platform Team  
-**Last Updated:** November 11, 2025  
-**Status:** Ready for Implementation