hatch3r 1.4.0 → 1.5.0

package/rules/hatch3r-observability-tracing-detail.md

@@ -0,0 +1,160 @@
+---
+id: hatch3r-observability-tracing-detail
+type: rule
+description: Extended tracing reference -- AI agent instrumentation, tool call audit trails, LLM request tracing, and correlation ID patterns
+scope: conditional
+globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/*agent*,**/observability/**"
+tags: [devops]
+quality_charter: agents/shared/quality-charter.md
+---
+# Observability -- Tracing Extended Reference
+
+On-demand companion to `hatch3r-observability-tracing`. Load when instrumenting AI agent systems, implementing tool call audit trails, or setting up correlation IDs for multi-agent workflows.
+
+## GenAI Span Attributes
+
+Use these attributes on all spans representing interactions with generative AI models:
+
+| Attribute | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `gen_ai.system` | string | GenAI provider system name | `openai`, `anthropic`, `azure_openai` |
+| `gen_ai.request.model` | string | Model name as specified in the request | `gpt-4o`, `claude-sonnet-4-20250514` |
+| `gen_ai.response.model` | string | Model name as returned in the response | `gpt-4o-2024-08-06` |
+| `gen_ai.request.max_tokens` | int | Maximum tokens requested for generation | `4096` |
+| `gen_ai.request.temperature` | float | Temperature parameter | `0.7` |
+| `gen_ai.response.finish_reasons` | string[] | Reasons the model stopped generating | `["stop"]`, `["length"]` |
+| `gen_ai.usage.input_tokens` | int | Tokens in the input/prompt | `1250` |
+| `gen_ai.usage.output_tokens` | int | Tokens in the generated output | `530` |
+
+- Always set `gen_ai.system` and `gen_ai.request.model` on every GenAI span.
+- Record `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` from the API response for cost dashboards.
+- Use `gen_ai.response.finish_reasons` to detect truncated outputs (`length`) and trigger re-prompting.
+
+## Agent Invocation Spans
+
+Instrument the full lifecycle of an agent invocation with a dedicated span. This span is the parent for all LLM calls, tool executions, and sub-agent delegations.
+
+- **Span name pattern:** `agent.{agent_name}.invoke`
+- **Required attributes:** `agent.id`, `agent.name`, `agent.parent_id`, `agent.task`, `agent.framework`
+- **Span events for state transitions:** `agent.planning`, `agent.tool_selection`, `agent.awaiting_human`, `agent.delegating`, `agent.completed`, `agent.error`
+
+```typescript
+const agentSpan = tracer.startSpan('agent.code_reviewer.invoke', {
+  attributes: {
+    'agent.id': invocationId,
+    'agent.name': 'code_reviewer',
+    'agent.parent_id': parentAgentId ?? '',
+    'agent.task': `review PR #${prNumber}`,
+    'agent.framework': 'custom',
+  },
+});
+agentSpan.addEvent('agent.planning');
+// ... agent reasoning and tool calls happen as child spans ...
+agentSpan.addEvent('agent.completed');
+agentSpan.end();
+```
+
+## Tool Call Spans
+
+Every tool invocation by an agent creates a child span of the agent invocation span.
+
+- **Span name pattern:** `tool.{tool_name}.execute`
+- **Required attributes:** `tool.name`, `tool.input_hash` (SHA-256), `tool.output_status`, `tool.duration_ms`, `tool.parameters_count`
+- Tool spans must be children of the invoking agent span. Set span status to `ERROR` when `tool.output_status` is `error` or `timeout`.
+- For tools performing I/O, create nested child spans using appropriate semantic conventions (`http.*`, `db.*`).
+
+```typescript
+const toolSpan = tracer.startSpan(
+  'tool.git_diff.execute',
+  { attributes: { 'tool.name': 'git_diff' } },
+  trace.setSpan(context.active(), agentSpan),
+);
+try {
+  const result = await tools.gitDiff(params);
+  toolSpan.setAttributes({
+    'tool.output_status': 'success',
+    'tool.duration_ms': performance.now() - startTime,
+    'tool.input_hash': hashInput(params),
+  });
+} catch (err) {
+  toolSpan.setAttributes({ 'tool.output_status': 'error' });
+  toolSpan.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
+  toolSpan.recordException(err);
+  throw err;
+} finally {
+  toolSpan.end();
+}
+```
+
+## LLM Request/Response Tracing
+
+- **Span name pattern:** `gen_ai.{operation}` (e.g., `gen_ai.chat`, `gen_ai.completion`)
+- **Token tracking:** Capture `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Aggregate in metrics: Counter `gen_ai.tokens_total` with labels `{direction, model, agent_name}`, Histogram `gen_ai.request_duration_ms`.
+- **Model version tracking:** Record both `gen_ai.request.model` and `gen_ai.response.model` for drift detection.
+- **Retry spans:** Each retry attempt is a separate child span. Set `gen_ai.request.retries` on the final span. Record `http.response.status_code` on failed spans (429 vs 500+).
+- Never log raw prompt content or full model responses as span attributes. Use token counts for cost tracking and correlated logs for prompt debugging in non-production environments.
+- Sample GenAI spans at 50-100% in production (higher than general spans) because each call is expensive and low volume.
+
+## Tool Call Audit Trail
+
+Maintain a structured audit log for every tool invocation in agentic workflows, separate from tracing spans.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `tool.name` | string | Name of the tool invoked |
+| `tool.input_hash` | string | SHA-256 hash of tool input (never log raw input) |
+| `tool.output_status` | string | `success`, `error`, `timeout`, or `denied` |
+| `tool.duration_ms` | float | Execution time in milliseconds |
+| `agent.id` | string | ID of the invoking agent |
+| `agent.name` | string | Human-readable agent name |
+| `correlation.id` | string | Trace correlation ID |
+| `timestamp` | string | ISO 8601 timestamp |
+| `session.id` | string | Session identifier |
+
+- Log tool invocations at `info` level, failures at `error` level with `error.type` and `error.message`.
+- Aggregate tool call counts per agent per session for anomaly detection.
+- Retain audit logs for a minimum of 90 days.
+
+## Correlation IDs for Agent Workflows
+
+- Use UUIDv4 with workflow-type prefix: `{workflow-type}-{uuid}` (e.g., `agent-run-550e8400-...`).
+- Generate at the workflow entry point. Propagate to all sub-agents and tool calls.
+- Every log entry, span, and metric must include `correlation.id`.
+- Cross-process: propagate via `X-Correlation-ID` header alongside W3C Trace Context.
+- Use OpenTelemetry `SpanLink` for cross-workflow references (e.g., agent run triggered by CI event).
+
+```typescript
+import { randomUUID } from 'node:crypto';
+import { context, trace, SpanStatusCode } from '@opentelemetry/api';
+
+function generateCorrelationId(workflowType: string): string {
+  return `${workflowType}-${randomUUID()}`;
+}
+
+async function runAgentWorkflow(task: string): Promise<void> {
+  const correlationId = generateCorrelationId('agent-run');
+  const tracer = trace.getTracer('agent-orchestrator');
+  const rootSpan = tracer.startSpan('agent.orchestrator.invoke', {
+    attributes: {
+      'correlation.id': correlationId,
+      'agent.name': 'orchestrator',
+      'agent.task': task,
+    },
+  });
+  try {
+    await context.with(trace.setSpan(context.active(), rootSpan), async () => {
+      await delegateToSubAgent('code_reviewer', {
+        correlationId,
+        parentSpanId: rootSpan.spanContext().spanId,
+        task: 'review changes',
+      });
+    });
+  } catch (err) {
+    rootSpan.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
+    rootSpan.recordException(err as Error);
+    throw err;
+  } finally {
+    rootSpan.end();
+  }
+}
+```

package/rules/hatch3r-observability-tracing-detail.mdc

@@ -0,0 +1,63 @@
+---
+description: Extended tracing reference -- AI agent instrumentation, tool call audit trails, LLM request tracing, and correlation ID patterns
+globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/*agent*", "**/observability/**"]
+alwaysApply: false
+---
+# Observability -- Tracing Extended Reference
+
+On-demand companion to `hatch3r-observability-tracing`. Load when instrumenting AI agent systems, implementing tool call audit trails, or setting up correlation IDs for multi-agent workflows.
+
+## GenAI Span Attributes
+
+Use these attributes on all spans representing interactions with generative AI models:
+
+| Attribute | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `gen_ai.system` | string | GenAI provider system name | `openai`, `anthropic`, `azure_openai` |
+| `gen_ai.request.model` | string | Model name as specified in the request | `gpt-4o`, `claude-sonnet-4-20250514` |
+| `gen_ai.response.model` | string | Model name as returned in the response | `gpt-4o-2024-08-06` |
+| `gen_ai.request.max_tokens` | int | Maximum tokens requested for generation | `4096` |
+| `gen_ai.request.temperature` | float | Temperature parameter | `0.7` |
+| `gen_ai.response.finish_reasons` | string[] | Reasons the model stopped generating | `["stop"]`, `["length"]` |
+| `gen_ai.usage.input_tokens` | int | Tokens in the input/prompt | `1250` |
+| `gen_ai.usage.output_tokens` | int | Tokens in the generated output | `530` |
+
+- Always set `gen_ai.system` and `gen_ai.request.model` on every GenAI span.
+- Record `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` from the API response for cost dashboards.
+- Use `gen_ai.response.finish_reasons` to detect truncated outputs (`length`) and trigger re-prompting.
+
+## Agent Invocation Spans
+
+Instrument the full lifecycle of an agent invocation with a dedicated span. This span is the parent for all LLM calls, tool executions, and sub-agent delegations.
+
+- **Span name pattern:** `agent.{agent_name}.invoke`
+- **Required attributes:** `agent.id`, `agent.name`, `agent.parent_id`, `agent.task`, `agent.framework`
+- **Span events for state transitions:** `agent.planning`, `agent.tool_selection`, `agent.awaiting_human`, `agent.delegating`, `agent.completed`, `agent.error`
+
+## Tool Call Spans
+
+Every tool invocation by an agent creates a child span of the agent invocation span.
+
+- **Span name pattern:** `tool.{tool_name}.execute`
+- **Required attributes:** `tool.name`, `tool.input_hash` (SHA-256), `tool.output_status`, `tool.duration_ms`, `tool.parameters_count`
+- Tool spans must be children of the invoking agent span. Set span status to `ERROR` when `tool.output_status` is `error` or `timeout`.
+
+## LLM Request/Response Tracing
+
+- **Span name pattern:** `gen_ai.{operation}` (e.g., `gen_ai.chat`, `gen_ai.completion`)
+- **Token tracking:** Capture `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Aggregate in metrics.
+- **Model version tracking:** Record both `gen_ai.request.model` and `gen_ai.response.model` for drift detection.
+- **Retry spans:** Each retry attempt is a separate child span. Set `gen_ai.request.retries` on the final span.
+- Never log raw prompt content or full model responses as span attributes.
+
+## Tool Call Audit Trail
+
+Maintain a structured audit log for every tool invocation in agentic workflows. Key fields: `tool.name`, `tool.input_hash`, `tool.output_status`, `tool.duration_ms`, `agent.id`, `agent.name`, `correlation.id`, `timestamp`, `session.id`. Retain for 90 days minimum.
+
+## Correlation IDs for Agent Workflows
+
+- Use UUIDv4 with workflow-type prefix: `{workflow-type}-{uuid}`.
+- Generate at the workflow entry point. Propagate to all sub-agents and tool calls.
+- Every log entry, span, and metric must include `correlation.id`.
+- Cross-process: propagate via `X-Correlation-ID` header alongside W3C Trace Context.
+- Use OpenTelemetry `SpanLink` for cross-workflow references.

package/rules/hatch3r-observability-tracing.md

@@ -0,0 +1,86 @@
+---
+id: hatch3r-observability-tracing
+type: rule
+description: Distributed tracing and OpenTelemetry core conventions for the project
+scope: conditional
+globs: "**/*trac*,**/*span*,**/*telemetry*,**/*otel*,**/observability/**"
+tags: [devops]
+quality_charter: agents/shared/quality-charter.md
+---
+# Observability -- Distributed Tracing & OpenTelemetry
+
+Core distributed tracing and OpenTelemetry conventions. For structured logging see `hatch3r-observability-logging`. For metrics, SLOs, alerting, and dashboards see `hatch3r-observability-metrics`. For AI agent instrumentation, tool call audit trails, and correlation ID patterns see `hatch3r-observability-tracing-detail`.
+
+## Distributed Tracing
+
+- Use OpenTelemetry SDK for all tracing instrumentation. Initialize the TracerProvider once at application startup before any instrumented libraries load.
+- Propagate trace context via W3C Trace Context headers (`traceparent`, `tracestate`) across all service boundaries, queues, and async workflows.
+- Span naming conventions:
+
+| Span Type   | Pattern                        | Example                     |
+| ----------- | ------------------------------ | --------------------------- |
+| HTTP server | `HTTP {method} {route}`       | `HTTP GET /api/users/:id`   |
+| HTTP client | `HTTP {method} {host}{path}`  | `HTTP POST api.stripe.com/` |
+| DB query    | `{db.system} {operation}`     | `firestore getDoc`          |
+| Queue       | `{queue} {operation}`         | `tasks-queue publish`       |
+| Internal    | `{module}.{function}`         | `auth.verifyToken`          |
+
+- Required span attributes: `service.name`, `service.version`, `deployment.environment`. Add domain-specific attributes (e.g., `user.id`, `tenant.id`) where relevant.
+- Parent-child span relationships: every outbound call (HTTP, DB, queue) creates a child span of the current context. Never create orphan spans.
+- Sampling strategies: use `ParentBased(TraceIdRatioBased(0.1))` in production (10% sample rate). Always sample errors and slow requests (> p95 latency) at 100%.
+- Use the OpenTelemetry Collector as a gateway between applications and backends to enable batching, retrying, and vendor-neutral export.
+- Keep span event count low (< 32 per span). For high-volume events, use correlated logs or `SpanLink` instead.
+
+## OpenTelemetry Semantic Conventions
+
+Follow the [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) (v1.29+) for consistent attribute naming across all telemetry signals.
+
+### Standard Attribute Namespaces
+
+| Namespace | Scope | Key Attributes |
+|-----------|-------|----------------|
+| `http.*` | HTTP client and server spans | `http.request.method`, `http.response.status_code`, `http.route`, `url.full`, `url.scheme` |
+| `db.*` | Database client spans | `db.system`, `db.operation.name`, `db.collection.name`, `db.query.text` (sanitized) |
+| `rpc.*` | RPC client and server spans | `rpc.system`, `rpc.service`, `rpc.method`, `rpc.grpc.status_code` |
+| `messaging.*` | Message queue spans | `messaging.system`, `messaging.operation.type`, `messaging.destination.name` |
+| `faas.*` | Serverless/FaaS invocations | `faas.trigger`, `faas.invoked_name`, `faas.coldstart` |
+| `cloud.*` | Cloud provider context | `cloud.provider`, `cloud.region`, `cloud.availability_zone` |
+| `k8s.*` | Kubernetes context | `k8s.namespace.name`, `k8s.pod.name`, `k8s.deployment.name` |
+
+- Use semantic convention attribute names exactly as specified. Do not invent custom alternatives for concepts already covered.
+- When semantic conventions are marked "Experimental," prefer them over project-specific names to ease future migration.
+
+### Resource Semantic Conventions
+
+Every telemetry-producing service must declare resource attributes at startup:
+
+| Attribute | Requirement | Description |
+|-----------|-------------|-------------|
+| `service.name` | Required | Logical name of the service |
+| `service.version` | Recommended | Semantic version of the service |
+| `deployment.environment.name` | Recommended | Deployment environment (production, staging, development) |
+| `service.instance.id` | Recommended | Unique instance identifier (pod name, container ID) |
+
+- Configure via environment variables (`OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`) or programmatically at SDK initialization.
+- Do not use the default `unknown_service` value in any deployed environment.
+
+### Span Status Codes
+
+| Code | When to Set |
+|------|-------------|
+| `UNSET` | Default. Span completed without error indication. |
+| `OK` | Set only when the application explicitly considers the operation successful and wants to override lower-level error signals. Use sparingly. |
+| `ERROR` | Operation failed: exception caught, HTTP 5xx, or business-logic error visible in error rate metrics. |
+
+- Set `ERROR` for server-side errors (5xx) and unhandled exceptions. Do not set `ERROR` for client errors (4xx) on the server span.
+- Attach exceptions as span events (`exception.type`, `exception.message`, `exception.stacktrace`) when setting `ERROR`.
+
+### Attribute Naming Guidelines
+
+- Use dot-separated namespaces: `http.request.method`, not `httpRequestMethod`.
+- Attribute values should be low-cardinality. Never use unbounded values (full URLs with query params, raw SQL) as attribute values.
+- Prefer semantic convention attributes over custom attributes. Prefix custom attributes with your project namespace (e.g., `myapp.feature.flag_key`).
+
+### AI Agent Semantic Conventions (Summary)
+
+Follow the [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for AI/LLM agent instrumentation. Key attributes: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`. For full attribute tables, code examples, tool call audit trails, and correlation ID patterns, see `hatch3r-observability-tracing-detail`.

package/rules/hatch3r-observability-tracing.mdc

@@ -0,0 +1,77 @@
+---
+description: Distributed tracing and OpenTelemetry core conventions for the project
+globs: ["**/*trac*", "**/*span*", "**/*telemetry*", "**/*otel*", "**/observability/**"]
+alwaysApply: false
+---
+# Observability -- Distributed Tracing & OpenTelemetry
+
+Core distributed tracing and OpenTelemetry conventions. For structured logging see `hatch3r-observability-logging`. For metrics, SLOs, alerting, and dashboards see `hatch3r-observability-metrics`. For AI agent instrumentation, tool call audit trails, and correlation ID patterns see `hatch3r-observability-tracing-detail`.
+
+## Distributed Tracing
+
+- Use OpenTelemetry SDK for all tracing instrumentation. Initialize the TracerProvider once at application startup before any instrumented libraries load.
+- Propagate trace context via W3C Trace Context headers (`traceparent`, `tracestate`) across all service boundaries, queues, and async workflows.
+- Span naming conventions:
+
+| Span Type   | Pattern                        | Example                     |
+| ----------- | ------------------------------ | --------------------------- |
+| HTTP server | `HTTP {method} {route}`       | `HTTP GET /api/users/:id`   |
+| HTTP client | `HTTP {method} {host}{path}`  | `HTTP POST api.stripe.com/` |
+| DB query    | `{db.system} {operation}`     | `firestore getDoc`          |
+| Queue       | `{queue} {operation}`         | `tasks-queue publish`       |
+| Internal    | `{module}.{function}`         | `auth.verifyToken`          |
+
+- Required span attributes: `service.name`, `service.version`, `deployment.environment`. Add domain-specific attributes where relevant.
+- Parent-child span relationships: every outbound call (HTTP, DB, queue) creates a child span of the current context. Never create orphan spans.
+- Sampling strategies: use `ParentBased(TraceIdRatioBased(0.1))` in production (10% sample rate). Always sample errors and slow requests (> p95 latency) at 100%.
+- Use the OpenTelemetry Collector as a gateway between applications and backends to enable batching, retrying, and vendor-neutral export.
+- Keep span event count low (< 32 per span). For high-volume events, use correlated logs or `SpanLink` instead.
+
+## OpenTelemetry Semantic Conventions
+
+Follow the [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) (v1.29+) for consistent attribute naming.
+
+### Standard Attribute Namespaces
+
+| Namespace | Scope | Key Attributes |
+|-----------|-------|----------------|
+| `http.*` | HTTP client and server spans | `http.request.method`, `http.response.status_code`, `http.route` |
+| `db.*` | Database client spans | `db.system`, `db.operation.name`, `db.collection.name` |
+| `rpc.*` | RPC client and server spans | `rpc.system`, `rpc.service`, `rpc.method` |
+| `messaging.*` | Message queue spans | `messaging.system`, `messaging.operation.type`, `messaging.destination.name` |
+| `faas.*` | Serverless/FaaS invocations | `faas.trigger`, `faas.invoked_name`, `faas.coldstart` |
+| `cloud.*` | Cloud provider context | `cloud.provider`, `cloud.region`, `cloud.availability_zone` |
+| `k8s.*` | Kubernetes context | `k8s.namespace.name`, `k8s.pod.name`, `k8s.deployment.name` |
+
+- Use semantic convention attribute names exactly as specified.
+- Prefer experimental conventions over project-specific names for future migration.
+
+### Resource Semantic Conventions
+
+| Attribute | Requirement | Description |
+|-----------|-------------|-------------|
+| `service.name` | Required | Logical name of the service |
+| `service.version` | Recommended | Semantic version of the service |
+| `deployment.environment.name` | Recommended | Deployment environment |
+| `service.instance.id` | Recommended | Unique instance identifier |
+
+### Span Status Codes
+
+| Code | When to Set |
+|------|-------------|
+| `UNSET` | Default. Span completed without error indication. |
+| `OK` | Explicitly override lower-level error signals. Use sparingly. |
+| `ERROR` | Exception caught, HTTP 5xx, or business-logic error. |
+
+- Set `ERROR` for server-side errors (5xx) and unhandled exceptions. Do not set `ERROR` for client errors (4xx).
+- Attach exceptions as span events when setting `ERROR`.
+
+### Attribute Naming Guidelines
+
+- Use dot-separated namespaces: `http.request.method`, not `httpRequestMethod`.
+- Attribute values should be low-cardinality. Never use unbounded values as attribute values.
+- Prefix custom attributes with your project namespace.
+
+### AI Agent Semantic Conventions (Summary)
+
+Follow the [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for AI/LLM agent instrumentation. Key attributes: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`. For full attribute tables, code examples, tool call audit trails, and correlation ID patterns, see `hatch3r-observability-tracing-detail`.