claude-code-kit 0.42.0__tar.gz → 0.43.0__tar.gz

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
       "name": "claude-kit",
       "source": "./",
       "description": "Cookiecutter-style scaffolder for an autonomous Claude Code SDLC config (no app code, no Docker): install CLAUDE.md + .claude/ (rules, the profile's agents/skills, hooks, artifact templates) + optional .mcp.json, then run /sdlc to drive spec → review → build → test → security → ship through profile-aware quality gates, working memory, and a self-improving learnings loop.",
-      "version": "0.42.0",
+      "version": "0.43.0",
       "license": "MIT",
       "keywords": ["sdlc", "agents", "orchestration", "quality-gates", "workflow", "scaffold", "cookiecutter"]
     }

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-kit",
-  "version": "0.42.0",
+  "version": "0.43.0",
   "description": "Cookiecutter-style scaffolder for an autonomous Claude Code SDLC config (no app code, no Docker). `claude-kit init` asks ordered questions and installs CLAUDE.md + .claude/ (rules, the profile's agents/skills, hooks, artifact templates) + optional .mcp.json; run /sdlc to drive spec → review → build → test → security → ship through profile-aware quality gates with working memory and a self-improving learnings loop.",
   "author": {
     "name": "Arjunsingh Yadav",

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/CHANGELOG.md

@@ -4,6 +4,27 @@ All notable changes to claude-kit are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/), and the project uses
 [semantic versioning](https://semver.org/).
 
+## [0.43.0] — 2026-06-26
+
+**New `otel-tracing` collection skill** — vendor-neutral distributed tracing with OpenTelemetry and
+Grafana Tempo. Re-derived from public OpenTelemetry, W3C Trace Context, and Grafana Tempo
+documentation; nothing vendored, no proprietary content.
+
+- **`skills/otel-tracing/`** (new collection skill) — the language-neutral tracing **pipeline and
+  backend**, complementary to the app-side `observability-and-logging`. `SKILL.md` covers the OTel
+  trace data model, the standard OTLP env-var contract (incl. the 4317-gRPC / 4318-HTTP +
+  `/v1/traces` port/path traps and the missing-`service.name` trap), the signal pipeline, env-gated /
+  fail-safe / idempotent instrumentation (auto + manual + framework, init order, span processors),
+  head-plus-tail sampling, Tempo essentials (distributor/ingester/querier/compactor, retention, the
+  metrics-generator), and trace↔logs↔metrics correlation. Six references go deep:
+  `instrumentation.md`, `collector.md`, `sampling.md`, `tempo.md`, `correlation.md`, and `gotchas.md`.
+- **Clean auto-trigger boundaries** — the `description:` routes app-side observability code to
+  `observability-and-logging` and LLM/model tracing to `langfuse-llm-tracing`; both siblings carry a
+  reciprocal pointer back (the `observability-and-logging` OTel section now points to `otel-tracing`
+  for the pipeline/backend layer). The neutral policy stays in `.claude/rules/devops-observability.md`
+  and the role stays the `observability-engineer` agent.
+- Counts: **102 → 103** total skills (**46 → 47** stack-collection; core unchanged at 56).
+
 ## [0.42.0] — 2026-06-26
 
 **Doc hygiene + workflow polish** (PR 4 of 4 — the final adoption from the MIT

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: claude-code-kit
-Version: 0.42.0
+Version: 0.43.0
 Summary: Cookiecutter-style scaffolder for an autonomous Claude Code SDLC configuration (no app code, no Docker). Asks ordered questions and installs CLAUDE.md + .claude/ (rules, the chosen profile's agents/skills, hooks, artifact templates) + optional .mcp.json; run /sdlc to drive spec → review → build → test → security → ship through profile-aware quality gates, working memory, and a self-improving learnings loop.
 Project-URL: Homepage, https://github.com/ajyadav013/claude-kit
 Project-URL: Repository, https://github.com/ajyadav013/claude-kit
@@ -84,7 +84,7 @@ Quick start) for the full breakdown of any row.
 | 🔁 **Pipeline & quality gates** | Gate-enforced progression (zero open Critical/High/Medium to advance), per-profile gate sets, a fast-track for small changes, and an anti-sycophancy `devils-advocate` pass |
 | 🤖 **Agent roster** | **28** tiered agents + per-database overlays + **6** org personas, led by an Orchestrator that never writes code |
 | 🔍 **Self-verification & review** | RARV green-Verify (real commands, not imagined), blind parallel review, and read-only risk classification |
-| 📐 **Rules & skills** | **23** stack-agnostic core rules (incl. 8 agent-operation rules) + **102** context-activated skills (56 core + 46 stack-collection) |
+| 📐 **Rules & skills** | **23** stack-agnostic core rules (incl. 8 agent-operation rules) + **103** context-activated skills (56 core + 47 stack-collection) |
 | 🧱 **Stacks & overlays** | Stack-agnostic core + **10** overlay rule sets (React · FastAPI · Go · Postgres · Mongo) wired to your exact commands, incl. a full React design system |
 | 🎚️ **Profiles, scopes & org** | **3** rigor profiles · **3** scopes · **5** autonomy levels · **7** org packs + **10** policy rules |
 | 🧠 **Memory & learning** | Working memory across context compaction + a cost-aware learnings loop (`capture_mode`) so the same mistake isn't repeated |
@@ -281,7 +281,7 @@ need.
 - **23 stack-agnostic core rules** in [`rules/`](rules/) — `mandatory-workflow`, `quality-gates`,
   `rarv-cycle`, `continuity`, plus eight agent-operation rules and `autonomy-levels` +
   `risk-classification` (see [`docs/agentic-patterns.md`](docs/agentic-patterns.md)).
-- **102 on-demand skills** in [`skills/`](skills/) (56 core + 46 stack-collection) — spec-driven dev, planning, TDD, debugging, code
+- **103 on-demand skills** in [`skills/`](skills/) (56 core + 47 stack-collection) — spec-driven dev, planning, TDD, debugging, code
   review, threat modeling, and more — activated by context and led by the `sdlc` entrypoint.
 - **Profile-subset install** — each profile installs a strict subset of agents, skills, hooks, and
   rules (`lean ⊂ standard ⊂ enterprise`), from fast track to full audit.
@@ -437,7 +437,7 @@ owned gates, and the rule that **no gate passes on an unproven verdict**.
 | **[wshobson/agents](https://github.com/wshobson/agents)** & similar agent collections | Large libraries of individual subagent prompts you pick from | A **smaller, opinionated set wired into a sequenced pipeline with owned quality gates** — agents aren't a menu, they're stages that hand off and block on each other. Adopt-by-reuse, not by accumulation. |
 | **[GitHub spec-kit](https://github.com/github/spec-kit)** | A spec-driven workflow (constitution → spec → tasks → analyze) | The same coverage-gate idea (the `story-planner` 1f gate + `task-tracker-sync`) **absorbed into a broader** lifecycle that also covers review, security, build, test, release, and observability gates. Complementary, wider scope. |
 | **claude-flow / multi-agent runtimes** | Runtime orchestrators that *execute* swarms of agents | **Portable configuration**, not a running process — the orchestration is described in rules the host (Claude Code) executes. No daemon, no lock-in, no app code. |
-| **dotfiles / `CLAUDE.md` starters** | A single rules file or settings snippet | A **catalog-driven generator**: resolves your stack/profile/scope into the right subset of 23 rules, 28 agents, 102 skills, gates, and hooks, kept **upgradeable** (`claude-kit upgrade` preserves your edits via owner + checksum). |
+| **dotfiles / `CLAUDE.md` starters** | A single rules file or settings snippet | A **catalog-driven generator**: resolves your stack/profile/scope into the right subset of 23 rules, 28 agents, 103 skills, gates, and hooks, kept **upgradeable** (`claude-kit upgrade` preserves your edits via owner + checksum). |
 
 **Choose claude-kit when** you want a consistent, reviewable, **gate-enforced** autonomous-SDLC setup
 that's the same across every repo and stack, installs in seconds, ships nothing you have to run, and

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/README.md

@@ -54,7 +54,7 @@ Quick start) for the full breakdown of any row.
 | 🔁 **Pipeline & quality gates** | Gate-enforced progression (zero open Critical/High/Medium to advance), per-profile gate sets, a fast-track for small changes, and an anti-sycophancy `devils-advocate` pass |
 | 🤖 **Agent roster** | **28** tiered agents + per-database overlays + **6** org personas, led by an Orchestrator that never writes code |
 | 🔍 **Self-verification & review** | RARV green-Verify (real commands, not imagined), blind parallel review, and read-only risk classification |
-| 📐 **Rules & skills** | **23** stack-agnostic core rules (incl. 8 agent-operation rules) + **102** context-activated skills (56 core + 46 stack-collection) |
+| 📐 **Rules & skills** | **23** stack-agnostic core rules (incl. 8 agent-operation rules) + **103** context-activated skills (56 core + 47 stack-collection) |
 | 🧱 **Stacks & overlays** | Stack-agnostic core + **10** overlay rule sets (React · FastAPI · Go · Postgres · Mongo) wired to your exact commands, incl. a full React design system |
 | 🎚️ **Profiles, scopes & org** | **3** rigor profiles · **3** scopes · **5** autonomy levels · **7** org packs + **10** policy rules |
 | 🧠 **Memory & learning** | Working memory across context compaction + a cost-aware learnings loop (`capture_mode`) so the same mistake isn't repeated |
@@ -251,7 +251,7 @@ need.
 - **23 stack-agnostic core rules** in [`rules/`](rules/) — `mandatory-workflow`, `quality-gates`,
   `rarv-cycle`, `continuity`, plus eight agent-operation rules and `autonomy-levels` +
   `risk-classification` (see [`docs/agentic-patterns.md`](docs/agentic-patterns.md)).
-- **102 on-demand skills** in [`skills/`](skills/) (56 core + 46 stack-collection) — spec-driven dev, planning, TDD, debugging, code
+- **103 on-demand skills** in [`skills/`](skills/) (56 core + 47 stack-collection) — spec-driven dev, planning, TDD, debugging, code
   review, threat modeling, and more — activated by context and led by the `sdlc` entrypoint.
 - **Profile-subset install** — each profile installs a strict subset of agents, skills, hooks, and
   rules (`lean ⊂ standard ⊂ enterprise`), from fast track to full audit.
@@ -407,7 +407,7 @@ owned gates, and the rule that **no gate passes on an unproven verdict**.
 | **[wshobson/agents](https://github.com/wshobson/agents)** & similar agent collections | Large libraries of individual subagent prompts you pick from | A **smaller, opinionated set wired into a sequenced pipeline with owned quality gates** — agents aren't a menu, they're stages that hand off and block on each other. Adopt-by-reuse, not by accumulation. |
 | **[GitHub spec-kit](https://github.com/github/spec-kit)** | A spec-driven workflow (constitution → spec → tasks → analyze) | The same coverage-gate idea (the `story-planner` 1f gate + `task-tracker-sync`) **absorbed into a broader** lifecycle that also covers review, security, build, test, release, and observability gates. Complementary, wider scope. |
 | **claude-flow / multi-agent runtimes** | Runtime orchestrators that *execute* swarms of agents | **Portable configuration**, not a running process — the orchestration is described in rules the host (Claude Code) executes. No daemon, no lock-in, no app code. |
-| **dotfiles / `CLAUDE.md` starters** | A single rules file or settings snippet | A **catalog-driven generator**: resolves your stack/profile/scope into the right subset of 23 rules, 28 agents, 102 skills, gates, and hooks, kept **upgradeable** (`claude-kit upgrade` preserves your edits via owner + checksum). |
+| **dotfiles / `CLAUDE.md` starters** | A single rules file or settings snippet | A **catalog-driven generator**: resolves your stack/profile/scope into the right subset of 23 rules, 28 agents, 103 skills, gates, and hooks, kept **upgradeable** (`claude-kit upgrade` preserves your edits via owner + checksum). |
 
 **Choose claude-kit when** you want a consistent, reviewable, **gate-enforced** autonomous-SDLC setup
 that's the same across every repo and stack, installs in seconds, ships nothing you have to run, and

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/docs/stack-skills/README.md

@@ -1,6 +1,6 @@
 # Production Engineering Skills Collection
 
-These forty-six Claude Code skills encode production-proven engineering conventions derived from real-world Python/FastAPI, Node/Express, and React services. Each skill was extracted from analyzing patterns across multiple production backend and frontend services, capturing canonical patterns for FastAPI microservices, async Python, Kafka/Temporal integration, multi-tenancy, database access, layered configuration, Redis caching, GCS file handling, LLM integration (Vertex AI / Langfuse), Node/Express services, React frontends (state, data-fetching, forms, design systems, testing), observability (including Grafana dashboards & unified alerting), authentication, containerization (with granular Dockerfile/compose deep-dives), testing, data pipelines, security scanning (OWASP ZAP VAPT/DAST and autonomous AI pentesting), security architecture (edge-to-service signed-identity trust boundary, Kubernetes workload hardening), scheduled & recurring jobs (Kubernetes CronJob + Temporal Schedule), kubectl day-2 operations, and modernization strategies. Use them to align new projects with established patterns, audit existing services, or onboard engineers to modern Python, Node, and React development practices.
+These forty-seven Claude Code skills encode production-proven engineering conventions derived from real-world Python/FastAPI, Node/Express, and React services. Each skill was extracted from analyzing patterns across multiple production backend and frontend services, capturing canonical patterns for FastAPI microservices, async Python, Kafka/Temporal integration, multi-tenancy, database access, layered configuration, Redis caching, GCS file handling, LLM integration (Vertex AI / Langfuse), Node/Express services, React frontends (state, data-fetching, forms, design systems, testing), observability (including Grafana dashboards & unified alerting), authentication, containerization (with granular Dockerfile/compose deep-dives), testing, data pipelines, security scanning (OWASP ZAP VAPT/DAST and autonomous AI pentesting), security architecture (edge-to-service signed-identity trust boundary, Kubernetes workload hardening), scheduled & recurring jobs (Kubernetes CronJob + Temporal Schedule), kubectl day-2 operations, and modernization strategies. Use them to align new projects with established patterns, audit existing services, or onboard engineers to modern Python, Node, and React development practices.
 
 > **Where these live in this repo:** the skill folders are under [`skills/`](../../skills/) (auto-discovered by the claude-kit plugin). The links below point there.
 
@@ -20,6 +20,7 @@ These forty-six Claude Code skills encode production-proven engineering conventi
 | **design-patterns-and-conventions** | Design conventions across repos — naming (classes/files/routes), error handling (ResponseData envelope), config hierarchy, logging structured-log patterns, code organization | [SKILL.md](../../skills/design-patterns-and-conventions/SKILL.md) |
 | **multi-tenancy-patterns** | Multi-tenant isolation strategies — tenant resolution (header/JWT/session), Postgres RLS, multi-pool databases, lazy connectors, data-layer org_id tenancy | [SKILL.md](../../skills/multi-tenancy-patterns/SKILL.md) |
 | **observability-and-logging** | Structured logging, OpenTelemetry tracing, Sentry error tracking, Prometheus metrics, NewRelic, request/correlation IDs, /health and /readiness endpoints | [SKILL.md](../../skills/observability-and-logging/SKILL.md) |
+| **otel-tracing** | Vendor-neutral distributed tracing *pipeline + backend* (language-agnostic; complement to observability-and-logging) — the OTel trace data model, the standard OTLP env-var contract, the OTLP→Collector→Tempo path, head-plus-tail sampling (loadbalancing exporter, `tail_sampling`), W3C context propagation across async/queue boundaries, the Grafana Tempo backend (distributor/ingester/querier/compactor, retention/compaction, OTLP ports/paths), TraceQL, the metrics-generator (span metrics + service graphs), and trace↔logs↔metrics correlation (exemplars); re-derived from public OpenTelemetry, W3C Trace Context, and Grafana Tempo docs | [SKILL.md](../../skills/otel-tracing/SKILL.md) |
 | **grafana-dashboards-and-alerts** | Grafana dashboards + unified alerting — dashboard JSON model, `$datasource`/cascading `label_values` template variables, RED-metric PromQL (NGINX ingress, OTel span-metrics, pod utilization), multi-stage alert rules with label-based routing (slack_0/pagerduty_0/webhook_0) + dashboard/panel deep-link annotations, Tempo service graphs, and dashboards-as-code provisioning | [SKILL.md](../../skills/grafana-dashboards-and-alerts/SKILL.md) |
 | **auth-and-rbac** | Authentication dependency chain (get_current_client / x-user-data header), JWT verification, RBAC role/permission enforcement, password hashing (argon2), OTP (pyotp), tenant-scoped authorization | [SKILL.md](../../skills/auth-and-rbac/SKILL.md) |
 | **containerization-and-deployment** | Multi-stage Dockerfiles, docker-compose for local infra, one-image-many-roles pattern (entrypoint.py MODE dispatch), cert/keytab writing from env + kinit bootstrap, k8s health/readiness probes, CI pipelines, secrets/env hygiene | [SKILL.md](../../skills/containerization-and-deployment/SKILL.md) |
@@ -71,7 +72,8 @@ These forty-six Claude Code skills encode production-proven engineering conventi
    - "build multi-tenant data isolation" → **multi-tenancy-patterns**
    - "set up React frontend" → **frontend-repo-architecture**
    - "migrate Pydantic v1 to v2" → **pydantic-schema-patterns**
-   - "add observability / structured logging / tracing" → **observability-and-logging**
+   - "add observability / structured logging / app-side tracing instrumentation" → **observability-and-logging**
+   - "design the OTLP/Collector/Tempo tracing pipeline / sample traces (head+tail) / write TraceQL / configure the Tempo backend / correlate traces↔logs↔metrics" → **otel-tracing**
    - "build a Grafana dashboard / write dashboard JSON or PromQL / RED dashboard / Grafana alert rule / contact points / provision dashboards as code" → **grafana-dashboards-and-alerts**
    - "implement authentication / RBAC / JWT verification" → **auth-and-rbac**
    - "create Dockerfile / docker-compose / k8s probes" → **containerization-and-deployment**

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "claude-code-kit"
-version = "0.42.0"
+version = "0.43.0"
 description = "Cookiecutter-style scaffolder for an autonomous Claude Code SDLC configuration (no app code, no Docker). Asks ordered questions and installs CLAUDE.md + .claude/ (rules, the chosen profile's agents/skills, hooks, artifact templates) + optional .mcp.json; run /sdlc to drive spec → review → build → test → security → ship through profile-aware quality gates, working memory, and a self-improving learnings loop."
 readme = "README.md"
 requires-python = ">=3.9"

{claude_code_kit-0.42.0 → claude_code_kit-0.43.0}/skills/observability-and-logging/SKILL.md

@@ -76,6 +76,8 @@ and blow up log volume. _(reference service pattern)_
 
 ### OpenTelemetry Distributed Tracing
 
+> This section is the **app-side** wiring (SDK init, FastAPI/asyncpg/Redis/Kafka auto-instrumentation, the `OTEL_ENABLED` switch). For the language-neutral tracing **pipeline and backend** — the OTLP/Collector topology, head-plus-tail sampling, the Grafana Tempo backend, TraceQL, and trace↔logs↔metrics correlation — see the sibling `otel-tracing` skill.
+
 **Initialize SDK first**: Call `initialize_telemetry()` in the app startup hook; create `TracerProvider(resource=Resource({SERVICE_NAME: ...}))`, add `BatchSpanProcessor(OTLPSpanExporter(endpoint=...))`, set global provider with `trace.set_tracer_provider()`. _(reference service pattern)_
 
 **Auto-instrumentation**: After SDK init, call `FastAPIInstrumentor.instrument_app(app)`, `AsyncPGInstrumentor().instrument()`, `RedisInstrumentor().instrument()`. Kafka consumers: `KafkaInstrumentor().instrument()`. _(production services pattern)_

claude_code_kit-0.43.0/skills/otel-tracing/README.md

@@ -0,0 +1,49 @@
+# otel-tracing
+
+Language-neutral distributed tracing: model your system with the OpenTelemetry data model, ship spans through the OTLP/Collector pipeline into Grafana Tempo, sample smartly (head plus tail), propagate W3C context across services and queues, query with TraceQL, and correlate traces with logs and metrics. This skill is the concrete OpenTelemetry-plus-Tempo how-to for turning isolated spans into debuggable distributed request flows, independent of any single language or framework.
+
+> **Boundary.** This skill owns the vendor-neutral tracing *pipeline* and *backend*: the OTel data model, OTLP protocol, Collector topology, sampling, propagation, the Tempo backend, and TraceQL. It does NOT cover app-side observability setup (structured logging, PII/secret redaction, error tracking, application RED metrics, health probes, framework auto-instrumentation wiring) — that is the sibling **observability-and-logging** — nor LLM/model-call tracing (prompts, tokens, cost) — that is **langfuse-llm-tracing**. The neutral policy lives in `.claude/rules/devops-observability.md` and the role is the **observability-engineer** agent; this skill is their concrete OTel+Tempo reference.
+
+## What's inside
+
+| File | Covers |
+|------|--------|
+| `SKILL.md` | The overview: OTel data model, the standard env-var contract, the signal pipeline, instrumentation principles (env-gated/fail-safe init, auto vs manual vs framework interceptors, one-trace-tree propagation), sampling (head + tail), Tempo essentials, TraceQL, and trace/logs/metrics correlation. |
+| `references/instrumentation.md` | The data model and resource semantic conventions, SDK init order (Resource → Exporter → SpanProcessor → TracerProvider), the env-gated/fail-safe/idempotent/shutdown-flush pattern, the full env-var contract, auto vs manual vs framework-interceptor instrumentation, and W3C context propagation across async/queue boundaries. |
+| `references/collector.md` | When to run the OTel Collector (and when to export app→Tempo directly), the receiver→processor→exporter pipeline, agent vs gateway topology, the core processors (`memory_limiter`, `batch`, `resource`/`attributes`, `tail_sampling`), the OTLP exporter to Tempo, and the loadbalancing exporter that tail sampling requires. |
+| `references/sampling.md` | Head sampling (parent-based + trace-id-ratio in the SDK) vs tail sampling (`tail_sampling` in the Collector), the hybrid production pattern (keep all errors + slow traces + a baseline %), consistent sampling-decision propagation, and the cost of 100% sampling. |
+| `references/tempo.md` | Tempo as an object-storage trace backend: native OTLP ingest (4317/4318) and the `/v1/traces` path nuance, the distributor/ingester/querier/compactor roles, deployment modes, retention/compaction, TraceQL query patterns, and the metrics-generator (span metrics + service graphs → Prometheus). |
+| `references/correlation.md` | Binding the three signals: `trace_id`/`span_id` on every log line (logs→traces), exemplars (metrics→traces), service graphs and span metrics (traces→metrics), consistent resource attributes so they join, and the error-tracker/OTel coexistence rule (avoid double-tracing). |
+| `references/gotchas.md` | The highest-leverage failure modes and fixes: endpoint/port/path traps, a legacy exporter overriding OTLP, fail-safe init, lost context propagation across queues, missing `service.name`, over-sampling cost, batch-queue overflow, cardinality blowups, and double-tracing. |
+
+## Provenance
+
+Re-derived (not vendored) from public, authoritative documentation: OpenTelemetry.io (SDK configuration, OTLP exporter, Collector architecture, semantic conventions, context propagation, sampling), the W3C Trace Context specification, and Grafana Tempo documentation (architecture, TraceQL, metrics-generator, datasource configuration). Vendor-neutral; no proprietary content. When this skill and the upstream docs disagree, trust upstream.
+
+## When to use
+
+- **Designing the pipeline** — where to run the Collector, agent vs gateway topology, how to order receivers/processors/exporters.
+- **Debugging missing or fragmented traces** — TracerProvider init order, context propagation across async/queue boundaries, `tail_sampling` without a loadbalancing exporter, endpoint/port/path mismatches.
+- **Choosing sampling** — head-based in the SDK, tail-based (error/latency-aware) in the Collector, and the hybrid production pattern: SDKs use `parentbased_always_on` to preserve context, the Collector applies `tail_sampling` (100% errors, 100% latency outliers, low-rate probabilistic for the rest).
+- **Configuring Tempo** — OTLP receivers, trace-ID sharding, retention/compaction, object storage, and the metrics-generator (RED metrics, service graphs).
+- **Writing TraceQL** — filtering by service, span attribute, duration, status; navigating structure with descendant/ancestor operators; preferring trace-level intrinsics.
+- **Correlating signals** — stamping `trace_id`/`span_id` onto log lines, configuring trace-to-logs in Grafana, enabling exemplars, and querying across traces, logs, and metrics.
+
+Do NOT reach here for app-side observability code (that is **observability-and-logging**) or LLM tracing (that is **langfuse-llm-tracing**).
+
+## Boundary vs siblings
+
+| Skill | Owns | Scope |
+|-------|------|-------|
+| **otel-tracing** (this) | OTel data model, OTLP/Collector/Tempo pipeline, sampling, W3C propagation, TraceQL, signal correlation | Language-neutral, backend-focused |
+| **observability-and-logging** | App-side instrumentation: framework/library auto-instrumentation wiring, structured logging, PII/secret redaction, error tracking, application RED metrics, health checks | Stack-specific, app-focused |
+| **langfuse-llm-tracing** | LLM observability: prompt/completion/token/cost tracking, model-call hierarchies | Model-call-focused |
+
+When a task crosses the line — for example "add tracing to a service" — **observability-and-logging** wires the app-side auto-instrumentation and sets `OTEL_EXPORTER_OTLP_ENDPOINT`; **otel-tracing** designs the Collector pipeline and Tempo backend that receive the spans. The policy governing all three lives in `.claude/rules/devops-observability.md` (honesty, fail-safe init, reuse-first cluster integration, PII redaction).
+
+## Related
+
+- **Agent:** `observability-engineer` (owns the tracing pipeline, sampling, Tempo, correlation).
+- **Rules:** `.claude/rules/devops-observability.md` (policy), `.claude/rules/agent-resilience.md` (fail-safe init).
+- **Sibling skills:** `observability-and-logging` (app-side), `langfuse-llm-tracing` (LLM-specific).
+- **Deep dive:** `SKILL.md` — the overview, with the six `references/` files above.

claude_code_kit-0.43.0/skills/otel-tracing/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: otel-tracing
+description: >-
+  Vendor-neutral distributed tracing with OpenTelemetry exported to Grafana Tempo. Covers the
+  OTel trace data model, the OTLP/Collector/Tempo pipeline, the standard env-var contract,
+  context propagation, sampling, TraceQL, service graphs, trace-to-logs/metrics correlation.
+  Do NOT use for app-side observability setup (structured logging, PII redaction, error tracking,
+  RED metrics, liveness/readiness, framework auto-instrumentation wiring) — that is
+  'observability-and-logging'. Do NOT use for LLM call tracing (prompts, tokens, cost) — that is
+  'langfuse-llm-tracing'. The neutral policy lives in .claude/rules/devops-observability.md; the
+  role is the observability-engineer agent.
+---
+
+# Distributed Tracing with OpenTelemetry and Grafana Tempo
+
+Vendor-neutral distributed tracing instruments requests across service boundaries, correlates them with logs and metrics, and exports trace data to Grafana Tempo for querying with TraceQL. The OpenTelemetry SDK captures spans in your service, exports OTLP to an optional Collector for sampling and enrichment, and lands traces in Tempo's object-storage backend. Tempo derives service-graph metrics, links traces to logs via shared identifiers, and surfaces exemplars that pivot from metric spikes to representative traces. This skill covers the OTel data model, the standard env-var contract, the pipeline topology, instrumentation principles, sampling, TraceQL, and the correlation patterns that bind traces to logs and metrics.
+
+> **Sibling boundary:** This skill covers the language-neutral OTLP/Collector/Tempo pipeline and the OTel data model. For app-side observability (structured logging, PII/secret redaction, error tracking, RED metrics, liveness/readiness probes, framework auto-instrumentation wiring), see `observability-and-logging`. For LLM call tracing (prompts, tokens, cost), see `langfuse-llm-tracing`.
+
+## When to use
+
+- You need to trace requests across multiple services and see the full call graph.
+- You want to correlate logs with traces by stamping trace IDs onto structured log lines.
+- You are adopting OpenTelemetry and need to configure the SDK, Collector, and Tempo backend.
+- You need to derive RED metrics (service graphs) from trace spans without instrumenting metrics in application code.
+- You want to sample traces intelligently (retain all errors and slow requests, probabilistic sampling for normal traffic) to control ingestion cost.
+- You need to query traces by service name, span attributes, duration, or status using TraceQL.
+- You are joining a framework or orchestration-engine OTel interceptor with HTTP/RPC auto-instrumentation to build a single trace tree.
+
+## The signal pipeline
+
+```
+┌────────────────┐
+│  Instrumented  │  service.name, deployment.environment
+│    Service     │  (OTel SDK: auto + manual instrumentation)
+└────────┬───────┘
+         │ OTLP (gRPC :4317 or HTTP :4318)
+         ▼
+┌────────────────┐  optional but recommended in production
+│ OTel Collector │  (batch, tail_sampling, enrichment, redaction)
+└────────┬───────┘
+         │ OTLP
+         ▼
+┌────────────────┐  native OTLP receiver (:4317/:4318)
+│ Grafana Tempo  │  (distributor → ingester → object storage)
+│  (Backend)     │  (metrics-generator → Prometheus, querier, compactor)
+└────────┬───────┘
+         │ HTTP/gRPC API + TraceQL
+         ▼
+┌────────────────┐
+│     Grafana    │  (Explore, dashboards, trace-to-logs, trace-to-metrics)
+└────────────────┘
+```
+
+The service exports OTLP either directly to Tempo's distributor or through a Collector. The Collector is optional for simple setups but recommended for buffering, tail sampling, attribute redaction, and fan-out to multiple backends. Tempo's distributor validates spans and shards a trace by hashing its TraceId; the ingester batches spans into blocks and flushes them to object storage (S3/GCS/Azure); the compactor merges blocks and enforces retention; the querier (behind a query-frontend) serves TraceQL. The optional metrics-generator derives RED metrics, and Grafana queries it all.
+
+## The OTel trace data model
+
+| Concept | Definition |
+|---------|-----------|
+| **Trace** | The full lifecycle of a request as it flows through a distributed system, identified by a 16-byte TraceId. |
+| **Span** | A single operation within a trace (an HTTP request, a database query, a function call), identified by an 8-byte SpanId. Child spans record their parent's SpanId to form hierarchies. |
+| **SpanContext** | The propagated, immutable identifiers: TraceId, SpanId, and TraceFlags (a 1-byte bitmap carrying the sampling decision). W3C Trace Context carries it in the `traceparent` header. |
+| **Attributes** | Key-value pairs describing the span (`http.request.method`, `db.system`, `rpc.service`). Use semantic conventions for cross-vendor portability. |
+| **Events** | Timestamped annotations attached to a span (exceptions, state transitions). Capped by `OTEL_SPAN_EVENT_COUNT_LIMIT` (default 128). |
+| **Status** | Span outcome: Unset, Ok, or Error. Tail sampling uses status to retain all errors. |
+| **Resource** | The entity producing telemetry (service, host, container). Must include `service.name`; if unset, SDKs fall back to `unknown_service:<executable>`. The value should be identical across all instances of a horizontally-scaled service. |
+| **Semantic Conventions** | Standard attribute names (`http.response.status_code`, `db.query.text`, `server.address`) that enable cross-backend queries and automatic dashboards. |
+
+## The standard OTel env-var contract
+
+Configure the SDK via environment variables to keep application code endpoint-agnostic. The same binary points at a local collector, a cluster collector, or Tempo's OTLP receiver just by changing the environment.
+
+| Variable | Purpose | Default / Notes |
+|----------|---------|-----------------|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base endpoint URL for all signals (e.g. `http://collector:4318`). OTLP/HTTP appends `/v1/traces`. | `http://localhost:4317` (gRPC) |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Signal-specific traces endpoint; overrides the base endpoint for traces. Use when routing signals to different backends. | Unset |
+| `OTEL_EXPORTER_OTLP_PROTOCOL` | Transport: `grpc`, `http/protobuf` (recommended), `http/json`. Verify backend support. | SDK-dependent |
+| `OTEL_TRACES_EXPORTER` | Exporter to use: `otlp` (default), `jaeger`, `zipkin`, `none`. Set `none` to disable trace export. | `otlp` |
+| `OTEL_SERVICE_NAME` | Sets the `service.name` resource attribute. **Required to find traces in Tempo.** Takes precedence over `OTEL_RESOURCE_ATTRIBUTES`. Use the same value for all instances of one service. | `unknown_service:<executable>` |
+| `OTEL_RESOURCE_ATTRIBUTES` | Comma-separated key=value pairs (e.g. `service.namespace=checkout,deployment.environment=production,service.version=1.2.3`). | Empty |
+| `OTEL_EXPORTER_OTLP_HEADERS` | Headers sent with OTLP requests (e.g. `Authorization=Bearer token`). For signal-specific headers, use `OTEL_EXPORTER_OTLP_TRACES_HEADERS`. | Empty |
+| `OTEL_EXPORTER_OTLP_COMPRESSION` | Compression: `gzip`, `none`. Use `gzip` over public networks to reduce egress. | `none` |
+| `OTEL_PROPAGATORS` | Propagators: `tracecontext`, `baggage`, `b3`, `b3multi`, `jaeger`, `xray`, `ottrace`. Comma-separated. | `tracecontext,baggage` (W3C) |
+| `OTEL_TRACES_SAMPLER` | Sampler: `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`. | `parentbased_always_on` |
+| `OTEL_TRACES_SAMPLER_ARG` | For ratio samplers, a decimal 0.0–1.0 sampling probability (e.g. `0.1` for 10%). | Unset |
+| `OTEL_LOG_LEVEL` | SDK log level: `debug`, `info`, `warn`, `error`. Use `debug` to troubleshoot init or export. | `info` |
+| `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | Max length of span attribute values (e.g. `4096`). Prevents unbounded attribute sizes. | Unlimited |
+| `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` | Max attributes per span. | `128` |
+
+**Endpoint pitfall:** Port 4317 is gRPC, 4318 is HTTP. OTLP/HTTP requires `/v1/traces` appended to the endpoint (some libraries append it, some require it in `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`); OTLP/gRPC uses no path suffix. Verify protocol-port alignment — 4317 with `http/protobuf`, or 4318 with `grpc`, is a common misconfiguration.
+
+**Resource pitfall:** Missing `service.name` is the most common "I can't find my traces" cause. Always set `OTEL_SERVICE_NAME` or include `service.name=your-service` in `OTEL_RESOURCE_ATTRIBUTES`. Setting both can confuse debugging, since `OTEL_SERVICE_NAME` wins.
+
+## Instrumentation principle
+
+**Env-gated + fail-safe + idempotent:**
+
+- Enable tracing only when a standard OTLP endpoint/exporter env var is present (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, or `OTEL_TRACES_EXPORTER` set to a non-`none` value). Otherwise, no-op and log.
+- Tracing setup must never crash or block service startup. Wrap initialization in the language's error guard and continue untraced on failure (`.claude/rules/agent-resilience.md`).
+- Make initialization idempotent. Flush and shut down the tracer provider on graceful shutdown, or you lose the last batch.
+
+**Auto + manual + framework interceptors:**
+
+- **Auto-instrumentation** uses language mechanisms (Java bytecode manipulation, runtime monkey patching, .NET runtime hooks) to intercept library/framework calls (HTTP, database, RPC). It gives immediate visibility; its limitation is no business-logic awareness.
+- **Manual instrumentation** provides precision for custom spans, business operations, and enriched attributes. Production pattern: auto-instrumentation baseline plus manual spans for business-critical flows.
+- **Framework interceptors** (web-framework and workflow/orchestration-engine OTel interceptors) emit deeper semantic spans than generic HTTP instrumentation and propagate context to child operations.
+
+**One trace tree via propagation:**
+
+- W3C Trace Context propagation uses the `traceparent` header (version, trace-id, parent span-id, trace-flags). Configure a composite propagator with the W3C trace-context and baggage propagators.
+- Context propagation across async boundaries requires the language's context mechanism (e.g. `AsyncLocalStorage` in Node.js, `contextvars` in Python). At queue boundaries, manually inject context into message headers before enqueue and extract it after dequeue to maintain trace continuity.
+- Baggage propagates key-value pairs across service boundaries alongside trace context. It is visible to the entire request path; validate at system boundaries, keep it small, and use it only for non-sensitive identifiers (never PII or credentials).
+
+**Initialization order:**
+
+Configure Resource → Exporter → SpanProcessor → TracerProvider, then register the global provider **before** any instrumentation or application code runs. Instrumentation libraries cache the global TracerProvider at import/load time, so late initialization yields no-op tracers and zero spans.
+
+**Span processors:**
+
+- Production deployments use `BatchSpanProcessor` (batches spans before export; OTel spec defaults: 5s schedule delay, 512 max export batch size, 2048 max queue size, 30s export timeout). High-throughput services raise the queue and batch sizes so spans are not dropped under load.
+- `SimpleSpanProcessor` (synchronous export per span) is for testing only; it blocks application threads and degrades latency.
+
+See `references/instrumentation.md` for auto-instrumentation setup, manual span creation, and context propagation patterns. See `references/collector.md` for Collector topology and processor pipelines.
+
+## Sampling
+
+**Head-based sampling** makes the decision at span creation time. It is stateless and gives predictable cost control but cannot guarantee capture of errors or latency outliers. SDKs use `parentbased_always_on` (preserve trace context) or `parentbased_traceidratio` (probabilistic).
+
+**Tail-based sampling** defers the decision until the trace completes, enabling outcome-driven selection (errors, high latency, specific attributes). It requires buffering spans in Collector memory, routing all spans of a trace to one instance via the loadbalancing exporter (keyed by TraceId), and careful capacity planning. A missing or misconfigured loadbalancing exporter fragments traces and produces wrong decisions.
+
+**Hybrid sampling pattern (production standard):**
+
+- SDKs use `parentbased_always_on` to preserve trace context across services.
+- The Collector applies `tail_sampling` with policies: 100% errors, 100% latency outliers, a low probabilistic rate (e.g. ~5%) for normal traffic.
+- Result: every critical trace retained for debugging, normal traffic sampled for cost control.
+
+**Tail sampling pitfalls:** Set `decision_wait` to 2–3× p99 trace duration; if it is shorter, the sampler decides on incomplete traces. Tail sampling buffers whole traces in memory, so monitor Collector memory and always pair `tail_sampling` with the `memory_limiter` processor (first in the pipeline, `batch` last).
+
+See `references/sampling.md` for sampler configuration, tail-sampling policies, and the loadbalancing exporter topology.
+
+## Tempo essentials
+
+Grafana Tempo is a cost-efficient, object-storage-backed trace backend that natively accepts OTLP on the default ports 4317 (gRPC) and 4318 (HTTP) — no special exporter is needed, since the app and Collector already speak OTLP. The **distributor** validates spans and shards a trace by hashing its TraceId. The **ingester** batches spans into blocks and flushes them to object storage (S3/GCS/Azure); recent, not-yet-flushed traces are served from the ingesters. The **querier** (behind a **query-frontend**) reads bloom filters and indexes from object storage to locate traces for TraceQL. The **compactor** merges smaller blocks, deduplicates, and expires data after the retention period (the `compactor.block_retention` default is 336h / 14 days). Tempo runs as a single binary (monolithic) or as independently-scaled microservices; newer versions add an optional Kafka-based ingest path — check your version's docs before assuming it.
+
+**TraceQL:** Tempo's trace query language. It addresses resource attributes (`resource.service.name`, `resource.deployment.environment`), span attributes (`span.http.request.method`, `span.db.system`), and intrinsics (`span:duration`, `span:status`, `span:kind`, `trace:duration`, `trace:rootService`). It supports comparison operators (`=`, `!=`, `>`, `>=`, `<`, `<=`, `=~` regex, `!~` negated regex), logical operators (`&&`, `||`), aggregators (`count`, `avg`, `max`, `min`, `sum`), and structural operators (`>>` descendant, `<<` ancestor, `>` child, `<` parent, `~` sibling). Regex operators are anchored by default — use `.*pattern.*` for unanchored matching. **Trace-level intrinsics** (`trace:duration`, `trace:rootService`) are generally more performant than span-level scans; prefer them when possible.
+
+**Metrics-generator:** Derives RED metrics from spans via the `span-metrics` processor (counter `traces_spanmetrics_calls_total`, histogram `traces_spanmetrics_latency`) and the `service-graphs` processor (request/failed counters and latency histograms per client-server pair). It writes to a Prometheus-compatible `remote_write` endpoint. Enabling many span-metric dimensions sharply increases cardinality and cost — keep the dimension set bounded.
+
+See `references/tempo.md` for deployment modes, retention configuration, and TraceQL query patterns.
+
+## Correlation
+
+**Trace-to-logs:** Stamp the active `trace_id` and `span_id` onto every structured log line. In Grafana, configure the Tempo datasource's trace-to-logs feature pointing at your log store (Loki, Elasticsearch, and others). This lets you pivot a log line to its full trace and back. Use consistent resource attributes (`service.name`, `deployment.environment`) across logs, metrics, and traces so they join.
+
+**Trace-to-metrics:** Grafana's trace-to-metrics links spans to metric queries using span/trace variables (e.g. span name, duration, trace ID), so you can jump from a span to the RED metrics around it.
+
+**Metrics-to-traces (exemplars):** The metrics-generator's span metrics can embed trace IDs as exemplars on Prometheus histograms. In Grafana, clicking an exemplar on a graph jumps to the corresponding trace — the reverse link from a metric spike to a representative trace.
+
+**Service graphs:** The `service-graphs` processor analyzes traces for edges (parent-child span relationships representing requests between services) and produces metrics labelled by `client`, `server`, and connection type. Grafana's Service Graph view renders these edges as a directed graph with RED metrics per edge.
+
+**Cardinality pitfall:** Raw IDs and paths used as span-derived metric labels explode cardinality. Normalize (replace IDs/UUIDs with placeholders) before they become labels. Span attributes in Tempo are cheaper than metric labels, but keep them bounded too.
+
+See `references/correlation.md` for trace_id log injection, exemplar configuration, and service-graph setup.
+
+## References
+
+- `references/instrumentation.md` — Auto-instrumentation setup, manual span creation, context propagation across async/queue boundaries, the standard env-var contract, and TracerProvider initialization order.
+- `references/collector.md` — Collector agent-gateway topology, processor pipelines (memory_limiter, batch, tail_sampling, attributes, filter), OTLP exporter configuration, and the loadbalancing exporter for tail sampling.
+- `references/sampling.md` — Head-based vs tail-based sampling, sampler configuration (SDK and Collector), tail-sampling policies (error, latency, probabilistic), and the hybrid sampling pattern.
+- `references/tempo.md` — Tempo architecture (distributor, ingester, querier, compactor), deployment modes (monolithic vs microservices), retention/compaction configuration, TraceQL query patterns, and metrics-generator setup.
+- `references/correlation.md` — Trace-to-logs (trace_id injection, Grafana datasource config), trace-to-metrics, metrics-to-traces (exemplars), and service graphs.
+- `references/gotchas.md` — The highest-leverage failure modes and fixes: endpoint/port/path traps, lost context propagation, missing `service.name`, over-sampling cost, batch-queue overflow, cardinality blowups, and double-tracing.
+
+## Provenance
+
+Re-derived from public OpenTelemetry documentation (opentelemetry.io), the W3C Trace Context specification (w3.org/TR/trace-context), and Grafana Tempo documentation (grafana.com/docs/tempo). When upstream documentation disagrees with this skill, trust upstream and update the skill.
+
+## Related
+
+- `.claude/rules/devops-observability.md` — The neutral policy for observability, logging, metrics, and tracing across all stacks.
+- `observability-engineer` agent — The role responsible for instrumenting services, configuring the Collector and Tempo, and maintaining correlation patterns.
+- `.claude/rules/agent-resilience.md` — Fail-safe principles for instrumentation (never crash, degrade gracefully).
+- `observability-and-logging` — App-side observability: structured logging, PII/secret redaction, error tracking, RED metrics, liveness/readiness probes, and framework auto-instrumentation wiring.
+- `langfuse-llm-tracing` — LLM call tracing (prompts, tokens, cost) for model observability.