@opengroundplan/core 0.1.1

package/templates/layers/observability-otel/files/docs/observability.md

@@ -0,0 +1,224 @@
+# Observability
+
+OpenTelemetry traces and metrics over OTLP, plus structured logs that carry `trace_id`
+and `span_id`. Everything is configured through standard `OTEL_*` environment variables;
+no endpoint, sampling ratio, or interval appears at a call site.
+
+## The one thing to know first
+
+**"OTel logs are GA" is only true for .NET, Java, PHP and C++.** Per
+[opentelemetry.io/status](https://opentelemetry.io/status/):
+
+| Language | Traces | Metrics | **Logs** |
+|---|---|---|---|
+| .NET, Java, PHP, C++ | Stable | Stable | **Stable** |
+| Go | Stable | Stable | **Beta** |
+| Python | Stable | Stable | **Development** |
+| JS / Node | Stable | Stable | **Development** |
+| Rust | Beta | Beta | Beta |
+
+So this layer never makes OTLP the *only* log path. **Stdout JSON is the durable path on
+every stack**; OTLP log export is additive and off by default (`OTEL_LOGS_EXPORTER=none`).
+Turn it on with `OTEL_LOGS_EXPORTER=otlp` when your collector and language are ready —
+on .NET that is today.
+
+## Getting a local backend
+
+```bash
+docker compose -f docker-compose.otel.yml up -d
+```
+
+That starts an OpenTelemetry Collector on `4318` (OTLP/HTTP) and `4317` (OTLP/gRPC) with a
+`debug` exporter, so you can see spans immediately. Replace `debug` in
+`otel-collector-config.yaml` with a real backend exporter — the application never learns
+the backend changed, which is the whole point of exporting to a Collector rather than to a
+vendor endpoint.
+
+## Configuration
+
+Copy what you need from `.env.otel.example`. Each key has one typed default in the
+telemetry config module; the environment only overrides keys that already exist there.
+
+| Variable | Meaning |
+|---|---|
+| `OTEL_SDK_DISABLED` | `true` turns the SDK off entirely (tests, CI, offline dev) |
+| `OTEL_SERVICE_NAME` | Service identity on every span, metric and log record |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | **Base** URL; the signal path (`/v1/traces`) is appended |
+| `OTEL_EXPORTER_OTLP_HEADERS` | `k=v,k2=v2` — auth for a hosted collector |
+| `OTEL_TRACES_SAMPLER_ARG` | Head-sampling ratio: `1` in dev, `0.05`–`0.1` in production |
+| `OTEL_METRIC_EXPORT_INTERVAL` | Metric push interval, milliseconds |
+| `OTEL_LOGS_EXPORTER` | `none` (default) or `otlp` — see the table above |
+| `LOG_LEVEL` | `debug` / `info` / `warn` / `error` |
+
+Sampling uses `parentbased_traceidratio`, so once an upstream service decides to sample a
+trace every downstream service keeps it. A per-service ratio would shred traces in half.
+
+Declarative SDK configuration (`otel-config.yaml`) became stable in 2026 and is the
+direction of travel, but support is uneven across SDKs. This layer uses environment
+variables so the same repo works on every runtime today.
+
+## Wiring it up
+
+### Node / TypeScript
+
+Instrumentation has to patch modules **before** the application requires them, so the SDK
+is started from an entry point, never from application code.
+
+*Any Node server (NestJS, Hono, …)* — `src/otel/register.ts` is preloaded:
+
+```jsonc
+// package.json
+"scripts": {
+  "start": "node --import ./dist/otel/register.js dist/main.js",
+  "dev":   "node --import ./src/otel/register.ts src/main.ts"
+}
+```
+
+*Next.js* — `src/instrumentation.ts` exports `register()`, which Next calls once per
+server runtime. Nothing else to do. If the build complains about bundling the SDK, add it
+to `serverExternalPackages` in `next.config.ts`:
+
+```ts
+serverExternalPackages: ['@opentelemetry/sdk-node', '@opentelemetry/auto-instrumentations-node'],
+```
+
+Then log through `src/otel/logger.ts` instead of `console`:
+
+```ts
+import { logger } from './otel/logger.js';
+
+logger.info({ orderId }, 'order accepted');
+```
+
+Install the dependencies the layer added to `package.json`:
+
+```bash
+npm install    # or pnpm install / yarn / bun install
+```
+
+`getNodeAutoInstrumentations` covers HTTP, fetch, Express/Nest/Fastify, and the common
+database and queue clients. `@opentelemetry/instrumentation-fs` is disabled — filesystem
+spans bury every span worth reading.
+
+### Python
+
+Call `setup_telemetry()` before the app serves its first request, and instrument the app:
+
+```python
+from contextlib import asynccontextmanager
+
+from fastapi import FastAPI
+
+from {{ solutionName | snake }}.otel import instrument_fastapi, setup_telemetry, shutdown_telemetry
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    setup_telemetry()
+    instrument_fastapi(app)
+    yield
+    shutdown_telemetry()
+
+
+app = FastAPI(lifespan=lifespan)
+```
+
+Log through structlog, which renders JSON to stdout with the trace ids already attached:
+
+```python
+from {{ solutionName | snake }}.otel import get_logger
+
+log = get_logger(__name__)
+log.info("order accepted", order_id=order_id)
+```
+
+The OTLP log path imports `opentelemetry.sdk._logs`. The underscore is not an accident —
+it matches the *Development* status in the table above, which is why it is opt-in.
+
+### Go
+
+```go
+cfg, err := telemetry.LoadConfig()
+if err != nil {
+    return err
+}
+shutdown, err := telemetry.Setup(ctx, cfg)
+if err != nil {
+    return err
+}
+defer func() { _ = shutdown(context.Background()) }()
+
+logger := telemetry.NewLogger(cfg, os.Stdout)
+slog.SetDefault(logger)
+
+handler := telemetry.Middleware(router)
+```
+
+`telemetry.Middleware` wraps the whole router, producing server spans and RED metrics for
+every route at once. Use `telemetry.Client(nil)` for outbound calls so the downstream
+service joins the same trace.
+
+Span names come from `http.Request.Pattern` (stdlib `ServeMux`, Go 1.23+). A third-party
+router that does not populate it falls back to the method — pass your own
+`otelhttp.WithSpanNameFormatter` if you need the route pattern from that router.
+
+Go logs go to stdout as JSON only. To also ship them over OTLP once the Beta signal is
+acceptable to you:
+
+```bash
+go get go.opentelemetry.io/otel/log go.opentelemetry.io/otel/sdk/log
+go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
+go get go.opentelemetry.io/contrib/bridges/otelslog
+```
+
+…then build the `otelslog` handler alongside — not instead of — the JSON handler.
+
+### .NET
+
+Add the project to the solution and reference it from the API project:
+
+```bash
+dotnet sln add src/{{ solutionName | dot }}.Observability
+dotnet add src/{{ solutionName | dot }}.Api reference src/{{ solutionName | dot }}.Observability
+```
+
+One line in `Program.cs`:
+
+```csharp
+builder.AddObservability();
+```
+
+Settings bind from the `Observability` section, so `appsettings.Production.json` can
+override any default:
+
+```json
+{
+  "Observability": {
+    "Endpoint": "http://otel-collector:4318",
+    "SamplingRatio": 0.1,
+    "LogsExporter": "otlp"
+  }
+}
+```
+
+.NET is one of the four languages where the log signal is Stable, so `"LogsExporter":
+"otlp"` is a safe production setting here. JSON on stdout stays enabled regardless.
+
+## What to actually measure
+
+- **RED** for every service — Rate, Errors, Duration. The HTTP instrumentation emits all
+  three without any application code.
+- **USE** for every resource — Utilization, Saturation, Errors.
+- Alert on **SLO burn rate**, not on raw thresholds. For a 99.9% SLO the canonical pairs
+  are 14.4x over 1h/5m and 6x over 6h/30m (page), 1x over 3d/6h (ticket). Generate the
+  rules with [Sloth](https://sloth.dev) rather than hand-writing them.
+
+Health endpoints are a separate concern from telemetry: use the Kubernetes convention
+`/livez` and `/readyz` (`/healthz` has been deprecated since Kubernetes v1.16), and keep
+**liveness free of dependency checks** — a liveness probe that pings the database turns a
+database blip into a cluster-wide restart storm.
+
+## Turning it off
+
+`OTEL_SDK_DISABLED=true` short-circuits provider setup on every stack. Structured logging
+stays on, because logs are the one signal that must never depend on a collector.

package/templates/layers/observability-otel/files/go.mod

@@ -0,0 +1,7 @@
+require go.opentelemetry.io/otel v1.44.0
+require go.opentelemetry.io/otel/trace v1.44.0
+require go.opentelemetry.io/otel/metric v1.44.0
+require go.opentelemetry.io/otel/sdk v1.44.0
+require go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0
+require go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0
+require go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0

package/templates/layers/observability-otel/files/internal/telemetry/config.go

@@ -0,0 +1,122 @@
+// Package telemetry wires OpenTelemetry traces and metrics, and the structured logger
+// that carries trace ids. Named telemetry, not otel, so it can import go.opentelemetry.io/otel.
+package telemetry
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// Config is the fully resolved telemetry settings for one process.
+type Config struct {
+	Enabled              bool
+	ServiceName          string
+	ServiceVersion       string
+	Environment          string
+	Endpoint             string
+	Headers              map[string]string
+	ExportTimeout        time.Duration
+	SamplingRatio        float64
+	MetricExportInterval time.Duration
+	// Go's log signal is Beta; stdout JSON stays the durable path either way.
+	LogsViaOTLP bool
+	LogLevel    string
+}
+
+// defaults is the single source of truth for every tunable. Standard OTEL_* names, so
+// the SDK, a Collector and an operator all read the same keys.
+var defaults = map[string]string{
+	"OTEL_SDK_DISABLED":           "false",
+	"OTEL_SERVICE_NAME":           "{{ solutionName | kebab }}",
+	"OTEL_SERVICE_VERSION":        "0.1.0",
+	"OTEL_DEPLOYMENT_ENVIRONMENT": "development",
+	"OTEL_EXPORTER_OTLP_ENDPOINT": "{{ otelExporterEndpoint }}",
+	"OTEL_EXPORTER_OTLP_HEADERS":  "",
+	"OTEL_EXPORTER_OTLP_TIMEOUT":  "10000",
+	"OTEL_TRACES_SAMPLER_ARG":     "{{ otelSamplingRatio }}",
+	"OTEL_METRIC_EXPORT_INTERVAL": "{{ otelMetricExportIntervalMs }}",
+	"OTEL_LOGS_EXPORTER":          "{{ otelLogsExporter }}",
+	"LOG_LEVEL":                   "{{ otelLogLevel }}",
+}
+
+// LoadConfig reads the environment once and reports every bad value at once.
+func LoadConfig() (Config, error) {
+	r := &reader{}
+	cfg := Config{
+		Enabled:              !r.boolean("OTEL_SDK_DISABLED"),
+		ServiceName:          r.str("OTEL_SERVICE_NAME"),
+		ServiceVersion:       r.str("OTEL_SERVICE_VERSION"),
+		Environment:          r.str("OTEL_DEPLOYMENT_ENVIRONMENT"),
+		Endpoint:             strings.TrimRight(r.str("OTEL_EXPORTER_OTLP_ENDPOINT"), "/"),
+		Headers:              parseHeaders(r.str("OTEL_EXPORTER_OTLP_HEADERS")),
+		ExportTimeout:        time.Duration(r.integer("OTEL_EXPORTER_OTLP_TIMEOUT")) * time.Millisecond,
+		SamplingRatio:        r.float("OTEL_TRACES_SAMPLER_ARG"),
+		MetricExportInterval: time.Duration(r.integer("OTEL_METRIC_EXPORT_INTERVAL")) * time.Millisecond,
+		LogsViaOTLP:          r.str("OTEL_LOGS_EXPORTER") == "otlp",
+		LogLevel:             r.str("LOG_LEVEL"),
+	}
+	if len(r.errs) > 0 {
+		return Config{}, errors.Join(r.errs...)
+	}
+	return cfg, nil
+}
+
+// SignalURL is the full OTLP/HTTP URL for one signal; exporters do not take the base.
+func (c Config) SignalURL(signal string) string {
+	return c.Endpoint + "/v1/" + signal
+}
+
+// OTEL_EXPORTER_OTLP_HEADERS is `k=v,k2=v2` per the OTLP exporter spec.
+func parseHeaders(raw string) map[string]string {
+	headers := map[string]string{}
+	for _, pair := range strings.Split(raw, ",") {
+		key, value, found := strings.Cut(pair, "=")
+		if key = strings.TrimSpace(key); found && key != "" {
+			headers[key] = strings.TrimSpace(value)
+		}
+	}
+	return headers
+}
+
+type reader struct {
+	errs []error
+}
+
+func (r *reader) str(key string) string {
+	def, declared := defaults[key]
+	if !declared {
+		r.errs = append(r.errs, fmt.Errorf("telemetry: no default declared for %q", key))
+		return ""
+	}
+	if v, found := os.LookupEnv(key); found && strings.TrimSpace(v) != "" {
+		return strings.TrimSpace(v)
+	}
+	return def
+}
+
+func (r *reader) integer(key string) int {
+	raw := r.str(key)
+	v, err := strconv.Atoi(raw)
+	if err != nil {
+		r.errs = append(r.errs, fmt.Errorf("telemetry: %s must be an integer, got %q", key, raw))
+	}
+	return v
+}
+
+func (r *reader) float(key string) float64 {
+	raw := r.str(key)
+	v, err := strconv.ParseFloat(raw, 64)
+	if err != nil {
+		r.errs = append(r.errs, fmt.Errorf("telemetry: %s must be a number, got %q", key, raw))
+	}
+	return v
+}
+
+func (r *reader) boolean(key string) bool {
+	raw := strings.ToLower(r.str(key))
+	return raw == "1" || raw == "true" || raw == "yes"
+}

package/templates/layers/observability-otel/files/internal/telemetry/http.go

@@ -0,0 +1,33 @@
+package telemetry
+
+import (
+	"net/http"
+
+	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
+)
+
+// Middleware produces server spans and the RED metrics (rate, errors, duration) for
+// every route at once. Wrap the router, never individual handlers.
+func Middleware(next http.Handler) http.Handler {
+	return otelhttp.NewHandler(next, "",
+		// Without this the span name is the empty operation above; the route pattern
+		// keeps cardinality bounded where the raw path would not.
+		otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string {
+			if pattern := r.Pattern; pattern != "" {
+				return pattern
+			}
+			return r.Method
+		}),
+	)
+}
+
+// Client is an http.Client that propagates W3C traceparent on every outbound request,
+// so a downstream service joins the same trace.
+func Client(base *http.Client) *http.Client {
+	if base == nil {
+		base = &http.Client{}
+	}
+	instrumented := *base
+	instrumented.Transport = otelhttp.NewTransport(base.Transport)
+	return &instrumented
+}

package/templates/layers/observability-otel/files/internal/telemetry/logging.go

@@ -0,0 +1,84 @@
+package telemetry
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"os"
+
+	oteltrace "go.opentelemetry.io/otel/trace"
+)
+
+// OTel SeverityNumber per slog level, so a Collector needs no mapping processor.
+var severityNumber = map[slog.Level]int{
+	slog.LevelDebug: 5,
+	slog.LevelInfo:  9,
+	slog.LevelWarn:  13,
+	slog.LevelError: 17,
+}
+
+// NewLogger returns the process logger. Stdout JSON is the durable path: Go's OTel log
+// signal is Beta, so nothing that matters may depend on it (see docs/observability.md).
+func NewLogger(cfg Config, out io.Writer) *slog.Logger {
+	if out == nil {
+		out = os.Stdout
+	}
+
+	var level slog.Level
+	if err := level.UnmarshalText([]byte(cfg.LogLevel)); err != nil {
+		level = slog.LevelInfo
+	}
+
+	handler := slog.NewJSONHandler(out, &slog.HandlerOptions{
+		Level:       level,
+		ReplaceAttr: toOTelFields,
+	})
+	return slog.New(&traceHandler{Handler: handler}).With(
+		slog.String("service.name", cfg.ServiceName),
+		slog.String("service.version", cfg.ServiceVersion),
+	)
+}
+
+// toOTelFields renames slog's built-ins to OTel log-record field names, so stdout JSON
+// and an OTLP record carry the same shape.
+func toOTelFields(groups []string, a slog.Attr) slog.Attr {
+	if len(groups) > 0 {
+		return a
+	}
+	switch a.Key {
+	case slog.MessageKey:
+		a.Key = "body"
+	case slog.LevelKey:
+		a.Key = "severity_text"
+	case slog.TimeKey:
+		a.Key = "timestamp"
+	}
+	return a
+}
+
+// traceHandler stamps every record with the active span so a log line is worth keeping
+// next to a trace.
+type traceHandler struct {
+	slog.Handler
+}
+
+func (h *traceHandler) Handle(ctx context.Context, record slog.Record) error {
+	record.AddAttrs(slog.Int("severity_number", severityNumber[record.Level]))
+	if sc := oteltrace.SpanContextFromContext(ctx); sc.IsValid() {
+		record.AddAttrs(
+			slog.String("trace_id", sc.TraceID().String()),
+			slog.String("span_id", sc.SpanID().String()),
+		)
+	}
+	return h.Handler.Handle(ctx, record)
+}
+
+// Both wrappers are required: embedding alone would return the inner handler and drop
+// trace stamping the first time a caller uses With or WithGroup.
+func (h *traceHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
+	return &traceHandler{Handler: h.Handler.WithAttrs(attrs)}
+}
+
+func (h *traceHandler) WithGroup(name string) slog.Handler {
+	return &traceHandler{Handler: h.Handler.WithGroup(name)}
+}

package/templates/layers/observability-otel/files/internal/telemetry/provider.go

@@ -0,0 +1,77 @@
+package telemetry
+
+import (
+	"context"
+	"errors"
+
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
+	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
+	"go.opentelemetry.io/otel/propagation"
+	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
+	"go.opentelemetry.io/otel/sdk/resource"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+)
+
+// Shutdown flushes what is still batched. A hard exit drops it.
+type Shutdown func(context.Context) error
+
+// Setup installs the global tracer and meter providers and the W3C propagator.
+// Returns a no-op Shutdown when telemetry is disabled, so callers never branch.
+func Setup(ctx context.Context, cfg Config) (Shutdown, error) {
+	if !cfg.Enabled {
+		return func(context.Context) error { return nil }, nil
+	}
+
+	// Schemaless: the attribute keys below are semantic-convention literals and do not
+	// need a schema URL that must then be kept in step with the SDK version.
+	res := resource.NewSchemaless(
+		attribute.String("service.name", cfg.ServiceName),
+		attribute.String("service.version", cfg.ServiceVersion),
+		attribute.String("deployment.environment.name", cfg.Environment),
+	)
+
+	traceExporter, err := otlptracehttp.New(ctx,
+		otlptracehttp.WithEndpointURL(cfg.SignalURL("traces")),
+		otlptracehttp.WithHeaders(cfg.Headers),
+		otlptracehttp.WithTimeout(cfg.ExportTimeout),
+	)
+	if err != nil {
+		return nil, err
+	}
+
+	metricExporter, err := otlpmetrichttp.New(ctx,
+		otlpmetrichttp.WithEndpointURL(cfg.SignalURL("metrics")),
+		otlpmetrichttp.WithHeaders(cfg.Headers),
+		otlpmetrichttp.WithTimeout(cfg.ExportTimeout),
+	)
+	if err != nil {
+		return nil, errors.Join(err, traceExporter.Shutdown(ctx))
+	}
+
+	// Parent-based keeps a sampled trace whole once an upstream service decided.
+	tracerProvider := sdktrace.NewTracerProvider(
+		sdktrace.WithResource(res),
+		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(cfg.SamplingRatio))),
+		sdktrace.WithBatcher(traceExporter),
+	)
+	meterProvider := sdkmetric.NewMeterProvider(
+		sdkmetric.WithResource(res),
+		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter,
+			sdkmetric.WithInterval(cfg.MetricExportInterval),
+		)),
+	)
+
+	otel.SetTracerProvider(tracerProvider)
+	otel.SetMeterProvider(meterProvider)
+	// W3C traceparent + baggage: the only propagation every other SDK agrees on.
+	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
+		propagation.TraceContext{},
+		propagation.Baggage{},
+	))
+
+	return func(ctx context.Context) error {
+		return errors.Join(tracerProvider.Shutdown(ctx), meterProvider.Shutdown(ctx))
+	}, nil
+}

package/templates/layers/observability-otel/files/otel-collector-config.yaml

@@ -0,0 +1,47 @@
+# Local Collector. Applications always speak OTLP to this process; swapping `debug`
+# for a vendor exporter changes the backend without touching a line of app code.
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  # First in every pipeline: refuses data before the process OOMs rather than after.
+  memory_limiter:
+    check_interval: 1s
+    limit_percentage: 75
+    spike_limit_percentage: 15
+  batch:
+    timeout: 5s
+    send_batch_size: 512
+
+exporters:
+  debug:
+    verbosity: normal
+
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+
+service:
+  extensions: [health_check]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [debug]
+  telemetry:
+    logs:
+      level: info

package/templates/layers/observability-otel/files/package.json

@@ -0,0 +1,15 @@
+{
+  "dependencies": {
+    "@opentelemetry/api": "^1.9.0",
+    "@opentelemetry/sdk-node": "^0.208.0",
+    "@opentelemetry/sdk-trace-base": "^2.2.0",
+    "@opentelemetry/sdk-metrics": "^2.2.0",
+    "@opentelemetry/sdk-logs": "^0.208.0",
+    "@opentelemetry/resources": "^2.2.0",
+    "@opentelemetry/auto-instrumentations-node": "^0.68.0",
+    "@opentelemetry/exporter-trace-otlp-proto": "^0.208.0",
+    "@opentelemetry/exporter-metrics-otlp-proto": "^0.208.0",
+    "@opentelemetry/exporter-logs-otlp-proto": "^0.208.0",
+    "pino": "^10.3.1"
+  }
+}

package/templates/layers/observability-otel/files/pyproject.toml

@@ -0,0 +1,2 @@
+[project]
+dependencies = ["opentelemetry-sdk>=1.44.0", "opentelemetry-exporter-otlp>=1.44.0", "opentelemetry-instrumentation-fastapi>=0.65b0", "structlog>=26.1.0"]