llm-otel-kit 0.1.0__tar.gz

llm_otel_kit-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+# Environment / secrets
+.env
+
+# Python
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+
+# IDE
+.vscode/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Docker
+open-webui-data/

llm_otel_kit-0.1.0/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## 0.1.0 (2025-07-12)
+
+### Added
+
+- Initial release
+- Provider abstraction: `LLMProvider` ABC with `complete()`, `stream()`, `list_models()`
+- Providers: Ollama, OpenAI-compatible (OpenAI, vLLM, llama.cpp, LM Studio, Groq, Together, Fireworks, Azure OpenAI, LiteLLM), Anthropic
+- OTel bootstrap: `init_observability()` with Dynatrace-compatible temporality
+- GenAI metrics: 10 instruments following OTel GenAI semantic conventions
+- Span helpers: `set_genai_span()`, `set_genai_response()`, `classify_request()`
+- Config: `AppConfig.from_env()` with legacy `OLLAMA_BASE_URL` fallback

llm_otel_kit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 theharithsa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_otel_kit-0.1.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.4
+Name: llm-otel-kit
+Version: 0.1.0
+Summary: Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud.
+Project-URL: Homepage, https://github.com/theharithsa/Local-LLM-Application-with-OpenLLMetry
+Project-URL: Issues, https://github.com/theharithsa/Local-LLM-Application-with-OpenLLMetry/issues
+Author: Vishruth Harithsa
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dynatrace,genai,llm,observability,ollama,openai,opentelemetry
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: opentelemetry-api>=1.25
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.25
+Requires-Dist: opentelemetry-sdk>=1.25
+Requires-Dist: traceloop-sdk>=0.59
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.25; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-asyncio; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# llm-otel-kit
+
+Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud.
+
+## What it does
+
+`llm-otel-kit` gives you **full OTel GenAI semantic convention coverage** for any LLM provider in ~10 lines of code:
+
+- **Traces** with `gen_ai.*` span attributes (model, tokens, latency, streaming mode)
+- **Metrics** — 10 instruments: operation duration, token usage, TTFT, TPOT, throughput, error rate, active requests
+- **Logs** exported via OTLP with structured context (model, duration, token counts)
+- **Dynatrace-ready** — correct temporality (DELTA for counters/histograms, CUMULATIVE for UpDownCounters)
+
+## Supported Providers
+
+| Provider | Type | Config name |
+|----------|------|-------------|
+| Ollama | Local | `ollama` |
+| OpenAI | Cloud | `openai` |
+| Anthropic | Cloud | `anthropic` |
+| vLLM | Local | `vllm` |
+| llama.cpp | Local | `llamacpp` |
+| LM Studio | Local | `lmstudio` |
+| Groq | Cloud | `groq` |
+| Together | Cloud | `together` |
+| Fireworks | Cloud | `fireworks` |
+| Azure OpenAI | Cloud | `azure_openai` |
+| LiteLLM | Proxy | `litellm` |
+
+## Quick Start
+
+```python
+from llm_otel_kit import AppConfig, GenAIMetrics, init_observability, create_provider
+
+config = AppConfig.from_env()
+otel = init_observability(config.app_name, config.otlp_endpoint, config.otlp_token)
+provider = create_provider(config.provider)
+m = GenAIMetrics(otel.meter)
+
+# Use provider.complete() / provider.stream() for instrumented LLM calls
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LLM_PROVIDER` | `ollama` | Provider name (see table above) |
+| `LLM_BASE_URL` | `http://localhost:11434` | Provider API base URL |
+| `LLM_API_KEY` | (empty) | API key for cloud providers |
+| `DEFAULT_MODEL` | (empty) | Fallback model name |
+| `APP_NAME` | `llm-backend` | OTel service name |
+| `TRACELOOP_BASE_URL` | (empty) | OTLP endpoint URL |
+| `DT_OTLP_TOKEN` | (empty) | Dynatrace API token |
+
+## Install
+
+```bash
+pip install llm-otel-kit
+```
+
+For Anthropic support:
+
+```bash
+pip install llm-otel-kit[anthropic]
+```
+
+## License
+
+MIT

llm_otel_kit-0.1.0/README.md

@@ -0,0 +1,69 @@
+# llm-otel-kit
+
+Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud.
+
+## What it does
+
+`llm-otel-kit` gives you **full OTel GenAI semantic convention coverage** for any LLM provider in ~10 lines of code:
+
+- **Traces** with `gen_ai.*` span attributes (model, tokens, latency, streaming mode)
+- **Metrics** — 10 instruments: operation duration, token usage, TTFT, TPOT, throughput, error rate, active requests
+- **Logs** exported via OTLP with structured context (model, duration, token counts)
+- **Dynatrace-ready** — correct temporality (DELTA for counters/histograms, CUMULATIVE for UpDownCounters)
+
+## Supported Providers
+
+| Provider | Type | Config name |
+|----------|------|-------------|
+| Ollama | Local | `ollama` |
+| OpenAI | Cloud | `openai` |
+| Anthropic | Cloud | `anthropic` |
+| vLLM | Local | `vllm` |
+| llama.cpp | Local | `llamacpp` |
+| LM Studio | Local | `lmstudio` |
+| Groq | Cloud | `groq` |
+| Together | Cloud | `together` |
+| Fireworks | Cloud | `fireworks` |
+| Azure OpenAI | Cloud | `azure_openai` |
+| LiteLLM | Proxy | `litellm` |
+
+## Quick Start
+
+```python
+from llm_otel_kit import AppConfig, GenAIMetrics, init_observability, create_provider
+
+config = AppConfig.from_env()
+otel = init_observability(config.app_name, config.otlp_endpoint, config.otlp_token)
+provider = create_provider(config.provider)
+m = GenAIMetrics(otel.meter)
+
+# Use provider.complete() / provider.stream() for instrumented LLM calls
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LLM_PROVIDER` | `ollama` | Provider name (see table above) |
+| `LLM_BASE_URL` | `http://localhost:11434` | Provider API base URL |
+| `LLM_API_KEY` | (empty) | API key for cloud providers |
+| `DEFAULT_MODEL` | (empty) | Fallback model name |
+| `APP_NAME` | `llm-backend` | OTel service name |
+| `TRACELOOP_BASE_URL` | (empty) | OTLP endpoint URL |
+| `DT_OTLP_TOKEN` | (empty) | Dynatrace API token |
+
+## Install
+
+```bash
+pip install llm-otel-kit
+```
+
+For Anthropic support:
+
+```bash
+pip install llm-otel-kit[anthropic]
+```
+
+## License
+
+MIT

llm_otel_kit-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "llm-otel-kit"
+version = "0.1.0"
+description = "Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.11"
+authors = [{ name = "Vishruth Harithsa" }]
+keywords = ["opentelemetry", "llm", "observability", "genai", "dynatrace", "ollama", "openai"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Framework :: FastAPI",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: System :: Monitoring",
+]
+dependencies = [
+    "httpx>=0.27",
+    "opentelemetry-api>=1.25",
+    "opentelemetry-sdk>=1.25",
+    "opentelemetry-exporter-otlp-proto-http>=1.25",
+    "traceloop-sdk>=0.59",
+]
+
+[project.optional-dependencies]
+anthropic = ["anthropic>=0.25"]
+dev = ["pytest", "pytest-asyncio", "ruff"]
+
+[project.urls]
+Homepage = "https://github.com/theharithsa/Local-LLM-Application-with-OpenLLMetry"
+Issues = "https://github.com/theharithsa/Local-LLM-Application-with-OpenLLMetry/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/llm_otel_kit"]

llm_otel_kit-0.1.0/src/llm_otel_kit/__init__.py

@@ -0,0 +1,27 @@
+"""llm-otel-kit — Drop-in OTel GenAI observability for any LLM backend."""
+
+from llm_otel_kit.bootstrap import OTelComponents, init_observability
+from llm_otel_kit.config import AppConfig, ProviderConfig
+from llm_otel_kit.metrics import GenAIMetrics
+from llm_otel_kit.providers import create_provider
+from llm_otel_kit.spans import (
+    classify_request,
+    detect_provider,
+    record_metrics,
+    set_genai_response,
+    set_genai_span,
+)
+
+__all__ = [
+    "AppConfig",
+    "GenAIMetrics",
+    "OTelComponents",
+    "ProviderConfig",
+    "classify_request",
+    "create_provider",
+    "detect_provider",
+    "init_observability",
+    "record_metrics",
+    "set_genai_response",
+    "set_genai_span",
+]

llm_otel_kit-0.1.0/src/llm_otel_kit/bootstrap.py

@@ -0,0 +1,137 @@
+"""
+Dynatrace-compatible OpenTelemetry bootstrap for GenAI applications.
+
+Handles the critical init order: MeterProvider → Logs → Traceloop.
+"""
+
+import logging
+import os
+from typing import NamedTuple
+
+from opentelemetry import metrics, trace
+from opentelemetry._logs import set_logger_provider
+from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
+from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
+from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
+from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+from opentelemetry.sdk.metrics import (
+    Counter,
+    Histogram,
+    MeterProvider,
+    UpDownCounter,
+)
+from opentelemetry.sdk.metrics.export import (
+    AggregationTemporality,
+    PeriodicExportingMetricReader,
+)
+from opentelemetry.sdk.metrics.view import (
+    ExplicitBucketHistogramAggregation,
+    View,
+)
+from traceloop.sdk import Traceloop
+
+# ---------------------------------------------------------------------------
+# OTel GenAI semantic-convention histogram bucket boundaries
+# ---------------------------------------------------------------------------
+DURATION_BUCKETS = [
+    0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56,
+    5.12, 10.24, 20.48, 40.96, 81.92,
+]
+TOKEN_BUCKETS = [
+    1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144,
+    1048576, 4194304, 16777216, 67108864,
+]
+TTFT_BUCKETS = [
+    0.001, 0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.25,
+    0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0,
+]
+TPOT_BUCKETS = [
+    0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5,
+    0.75, 1.0, 2.5,
+]
+
+
+class OTelComponents(NamedTuple):
+    """Tuple returned by init_observability()."""
+    meter: metrics.Meter
+    tracer: trace.Tracer
+    logger: logging.Logger
+
+
+def _init_metrics(app_name: str, otlp_endpoint: str, otlp_token: str) -> metrics.Meter:
+    headers: dict[str, str] = {}
+    if otlp_token:
+        headers["Authorization"] = f"Api-Token {otlp_token}"
+
+    exporter = OTLPMetricExporter(
+        endpoint=f"{otlp_endpoint}/v1/metrics",
+        headers=headers,
+        preferred_temporality={
+            Counter: AggregationTemporality.DELTA,
+            Histogram: AggregationTemporality.DELTA,
+            UpDownCounter: AggregationTemporality.CUMULATIVE,
+        },
+    )
+    provider = MeterProvider(
+        metric_readers=[
+            PeriodicExportingMetricReader(exporter, export_interval_millis=30_000),
+        ],
+        views=[
+            View(instrument_name="gen_ai.client.operation.duration",
+                 aggregation=ExplicitBucketHistogramAggregation(boundaries=DURATION_BUCKETS)),
+            View(instrument_name="gen_ai.client.token.usage",
+                 aggregation=ExplicitBucketHistogramAggregation(boundaries=TOKEN_BUCKETS)),
+            View(instrument_name="gen_ai.server.time_to_first_token",
+                 aggregation=ExplicitBucketHistogramAggregation(boundaries=TTFT_BUCKETS)),
+            View(instrument_name="gen_ai.server.time_per_output_token",
+                 aggregation=ExplicitBucketHistogramAggregation(boundaries=TPOT_BUCKETS)),
+        ],
+    )
+    metrics.set_meter_provider(provider)
+    return metrics.get_meter(app_name, "1.0.0")
+
+
+def _init_logs(otlp_endpoint: str, otlp_token: str) -> None:
+    headers: dict[str, str] = {}
+    if otlp_token:
+        headers["Authorization"] = f"Api-Token {otlp_token}"
+
+    exporter = OTLPLogExporter(
+        endpoint=f"{otlp_endpoint}/v1/logs",
+        headers=headers,
+    )
+    provider = LoggerProvider()
+    provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
+    set_logger_provider(provider)
+    handler = LoggingHandler(level=logging.INFO, logger_provider=provider)
+    logging.getLogger().addHandler(handler)
+    logging.getLogger().setLevel(logging.INFO)
+
+
+def init_observability(
+    app_name: str,
+    otlp_endpoint: str = "",
+    otlp_token: str = "",
+) -> OTelComponents:
+    """
+    One-call OTel bootstrap: metrics → logs → tracing (order matters).
+
+    If *otlp_endpoint* / *otlp_token* are empty, falls back to env vars
+    ``TRACELOOP_BASE_URL`` and ``DT_OTLP_TOKEN``.
+    """
+    otlp_endpoint = otlp_endpoint or os.getenv("TRACELOOP_BASE_URL", "")
+    otlp_token = otlp_token or os.getenv("DT_OTLP_TOKEN", "")
+
+    if otlp_endpoint:
+        meter = _init_metrics(app_name, otlp_endpoint, otlp_token)
+        _init_logs(otlp_endpoint, otlp_token)
+    else:
+        meter = metrics.get_meter(app_name, "1.0.0")
+
+    # Traceloop MUST init after MeterProvider to avoid conflicts
+    Traceloop.init(app_name=app_name, disable_batch=False)
+
+    tracer = trace.get_tracer(app_name, "1.0.0")
+    logger = logging.getLogger(app_name)
+
+    return OTelComponents(meter=meter, tracer=tracer, logger=logger)

llm_otel_kit-0.1.0/src/llm_otel_kit/config.py

@@ -0,0 +1,80 @@
+"""Configuration models for llm-otel-kit."""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+
+
+@dataclass
+class ProviderConfig:
+    """LLM provider connection settings.
+
+    Attributes:
+        name: Provider identifier — ``"ollama"``, ``"openai"``, or ``"anthropic"``.
+        base_url: API base URL (e.g. ``http://localhost:11434``,
+                  ``https://api.openai.com``).
+        api_key: API key for cloud providers.  Leave empty for local providers.
+        default_model: Fallback model when the request doesn't specify one.
+    """
+
+    name: str = "ollama"
+    base_url: str = "http://localhost:11434"
+    api_key: str = ""
+    default_model: str = ""
+
+
+@dataclass
+class AppConfig:
+    """Full application configuration — provider + observability.
+
+    Attributes:
+        app_name: OTel service name.
+        provider: LLM provider settings.
+        otlp_endpoint: OTLP base URL (e.g. Dynatrace OTLP endpoint).
+        otlp_token: Auth token for the OTLP exporter.
+    """
+
+    app_name: str = "llm-backend"
+    provider: ProviderConfig = field(default_factory=ProviderConfig)
+    otlp_endpoint: str = ""
+    otlp_token: str = ""
+
+    @classmethod
+    def from_env(cls) -> AppConfig:
+        """Build config from environment variables.
+
+        Env vars:
+            ``APP_NAME``              — OTel service name (default: ``llm-backend``)
+            ``LLM_PROVIDER``          — ``ollama`` | ``openai`` | ``anthropic``
+            ``LLM_BASE_URL``          — Provider API base URL
+            ``LLM_API_KEY``           — API key for cloud providers
+            ``DEFAULT_MODEL``         — Fallback model name
+            ``TRACELOOP_BASE_URL``    — OTLP endpoint
+            ``DT_OTLP_TOKEN``         — Dynatrace API token
+
+        Legacy env vars (``OLLAMA_BASE_URL``) are supported as fallbacks.
+        """
+        provider_name = os.getenv("LLM_PROVIDER", "ollama").lower()
+
+        # Resolve base URL with legacy fallback
+        base_url = os.getenv("LLM_BASE_URL", "")
+        if not base_url:
+            if provider_name == "ollama":
+                base_url = os.getenv("OLLAMA_BASE_URL", "http://host.docker.internal:11434")
+            elif provider_name == "openai":
+                base_url = "https://api.openai.com"
+            elif provider_name == "anthropic":
+                base_url = "https://api.anthropic.com"
+
+        return cls(
+            app_name=os.getenv("APP_NAME", "llm-backend"),
+            provider=ProviderConfig(
+                name=provider_name,
+                base_url=base_url,
+                api_key=os.getenv("LLM_API_KEY", ""),
+                default_model=os.getenv("DEFAULT_MODEL", ""),
+            ),
+            otlp_endpoint=os.getenv("TRACELOOP_BASE_URL", ""),
+            otlp_token=os.getenv("DT_OTLP_TOKEN", ""),
+        )

llm_otel_kit-0.1.0/src/llm_otel_kit/metrics.py

@@ -0,0 +1,58 @@
+"""GenAI semantic-convention + operational metric instruments."""
+
+from dataclasses import dataclass, field
+
+from opentelemetry.metrics import Counter, Histogram, Meter, UpDownCounter
+
+
+@dataclass
+class GenAIMetrics:
+    """Pre-created OTel metric instruments for LLM observability.
+
+    Usage::
+
+        from llm_otel_kit import GenAIMetrics, init_observability
+
+        otel = init_observability("my-app")
+        m = GenAIMetrics(otel.meter)
+        m.request_count.add(1, {"model": "gpt-4o"})
+    """
+
+    _meter: Meter = field(repr=False)
+
+    # GenAI semconv
+    operation_duration: Histogram = field(init=False)
+    token_usage: Histogram = field(init=False)
+    ttft: Histogram = field(init=False)
+    tpot: Histogram = field(init=False)
+
+    # Operational
+    request_count: Counter = field(init=False)
+    error_count: Counter = field(init=False)
+    active_requests: UpDownCounter = field(init=False)
+    stream_chunks: Counter = field(init=False)
+    token_throughput: Histogram = field(init=False)
+    message_count: Histogram = field(init=False)
+
+    def __post_init__(self) -> None:
+        m = self._meter
+        self.operation_duration = m.create_histogram(
+            "gen_ai.client.operation.duration", "GenAI operation duration", "s")
+        self.token_usage = m.create_histogram(
+            "gen_ai.client.token.usage", "Input and output token counts", "{token}")
+        self.ttft = m.create_histogram(
+            "gen_ai.server.time_to_first_token", "Time to first token", "s")
+        self.tpot = m.create_histogram(
+            "gen_ai.server.time_per_output_token", "Time per output token", "s")
+        self.request_count = m.create_counter(
+            "llm.request.count", "Total LLM requests", "1")
+        self.error_count = m.create_counter(
+            "llm.request.errors", "Failed LLM requests", "1")
+        self.active_requests = m.create_up_down_counter(
+            "llm.request.active", "In-flight LLM requests", "1")
+        self.stream_chunks = m.create_counter(
+            "llm.stream.chunks", "Streaming chunks sent", "1")
+        self.token_throughput = m.create_histogram(
+            "llm.token.throughput", "Output token throughput", "{token}/s")
+        self.message_count = m.create_histogram(
+            "llm.request.message_count", "Messages in prompt", "1")

llm_otel_kit-0.1.0/src/llm_otel_kit/providers/__init__.py

@@ -0,0 +1,36 @@
+"""Provider registry — factory for LLM backend providers."""
+
+from llm_otel_kit.config import ProviderConfig
+from llm_otel_kit.providers.base import LLMProvider
+
+
+def create_provider(config: ProviderConfig) -> LLMProvider:
+    """Instantiate the correct provider from config.
+
+    Supported providers:
+        - ``ollama`` — Ollama native API (``/api/chat``)
+        - ``openai``  — OpenAI-compatible (works with OpenAI, vLLM, llama.cpp,
+          LM Studio, Groq, Together, Fireworks, Azure OpenAI, LiteLLM, etc.)
+        - ``anthropic`` — Anthropic Messages API (``/v1/messages``)
+    """
+    name = config.name.lower()
+
+    if name == "ollama":
+        from llm_otel_kit.providers.ollama import OllamaProvider
+        return OllamaProvider(config)
+    if name in ("openai", "vllm", "llamacpp", "lmstudio", "groq", "together",
+                "fireworks", "azure_openai", "litellm"):
+        from llm_otel_kit.providers.openai_compat import OpenAICompatProvider
+        return OpenAICompatProvider(config)
+    if name == "anthropic":
+        from llm_otel_kit.providers.anthropic import AnthropicProvider
+        return AnthropicProvider(config)
+
+    raise ValueError(
+        f"Unknown provider '{config.name}'. "
+        "Supported: ollama, openai, anthropic, vllm, llamacpp, lmstudio, "
+        "groq, together, fireworks, azure_openai, litellm"
+    )
+
+
+__all__ = ["LLMProvider", "create_provider"]

llm_otel_kit-0.1.0/src/llm_otel_kit/providers/anthropic.py

@@ -0,0 +1,142 @@
+"""Anthropic Messages API provider (``/v1/messages``)."""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import AsyncIterator
+
+import httpx
+
+from llm_otel_kit.providers.base import (
+    CompletionResult,
+    LLMProvider,
+    StreamChunk,
+    TimingInfo,
+)
+
+
+class AnthropicProvider(LLMProvider):
+    """Provider for the Anthropic Claude API."""
+
+    @property
+    def system_name(self) -> str:
+        return "anthropic"
+
+    def _headers(self) -> dict[str, str]:
+        return {
+            "Content-Type": "application/json",
+            "x-api-key": self.config.api_key,
+            "anthropic-version": "2023-06-01",
+        }
+
+    def build_payload(
+        self,
+        model: str,
+        messages: list[dict],
+        stream: bool,
+        **kwargs,
+    ) -> dict:
+        # Anthropic separates system message from the messages array
+        system_parts: list[str] = []
+        user_messages: list[dict] = []
+        for msg in messages:
+            if msg["role"] == "system":
+                system_parts.append(msg["content"])
+            else:
+                user_messages.append({"role": msg["role"], "content": msg["content"]})
+
+        payload: dict = {
+            "model": model,
+            "messages": user_messages,
+            "max_tokens": kwargs.get("max_tokens", 4096),
+            "stream": stream,
+        }
+        if system_parts:
+            payload["system"] = "\n".join(system_parts)
+        if kwargs.get("temperature") is not None:
+            payload["temperature"] = kwargs["temperature"]
+        if kwargs.get("top_p") is not None:
+            payload["top_p"] = kwargs["top_p"]
+        return payload
+
+    async def complete(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> CompletionResult:
+        response = await client.post(
+            f"{self.base_url}/v1/messages",
+            json=payload,
+            headers=self._headers(),
+        )
+        response.raise_for_status()
+        data = response.json()
+
+        content_blocks = data.get("content", [])
+        text = "".join(b.get("text", "") for b in content_blocks if b.get("type") == "text")
+        usage = data.get("usage", {})
+
+        return CompletionResult(
+            content=text,
+            model=data.get("model", payload.get("model", "")),
+            prompt_tokens=usage.get("input_tokens", 0),
+            completion_tokens=usage.get("output_tokens", 0),
+            finish_reason=data.get("stop_reason", "end_turn"),
+            response_id=data.get("id", ""),
+            timing=TimingInfo(),
+        )
+
+    async def stream(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> AsyncIterator[StreamChunk]:
+        prompt_tokens = 0
+        completion_tokens = 0
+
+        async with client.stream(
+            "POST",
+            f"{self.base_url}/v1/messages",
+            json=payload,
+            headers=self._headers(),
+        ) as resp:
+            resp.raise_for_status()
+            async for line in resp.aiter_lines():
+                if not line.startswith("data: "):
+                    continue
+                try:
+                    event = json.loads(line[6:])
+                except json.JSONDecodeError:
+                    continue
+
+                event_type = event.get("type", "")
+
+                if event_type == "message_start":
+                    usage = event.get("message", {}).get("usage", {})
+                    prompt_tokens = usage.get("input_tokens", 0)
+
+                elif event_type == "content_block_delta":
+                    delta = event.get("delta", {})
+                    yield StreamChunk(content=delta.get("text", ""))
+
+                elif event_type == "message_delta":
+                    usage = event.get("usage", {})
+                    completion_tokens = usage.get("output_tokens", 0)
+                    stop_reason = event.get("delta", {}).get("stop_reason", "end_turn")
+                    yield StreamChunk(
+                        done=True,
+                        prompt_tokens=prompt_tokens,
+                        completion_tokens=completion_tokens,
+                        finish_reason=stop_reason,
+                        timing=TimingInfo(),
+                    )
+
+    async def list_models(self, client: httpx.AsyncClient) -> list[dict]:
+        # Anthropic doesn't have a models endpoint; return a static list
+        models = ["claude-sonnet-4-20250514", "claude-3-5-haiku-20241022", "claude-3-opus-20240229"]
+        return [
+            {"id": m, "object": "model",
+             "created": int(time.time()), "owned_by": "anthropic"}
+            for m in models
+        ]

llm_otel_kit-0.1.0/src/llm_otel_kit/providers/base.py

@@ -0,0 +1,88 @@
+"""Abstract base class for LLM providers."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from typing import AsyncIterator
+from urllib.parse import urlparse
+
+import httpx
+
+from llm_otel_kit.config import ProviderConfig
+
+
+@dataclass
+class TimingInfo:
+    """TTFT / TPOT extracted from the provider response."""
+    ttft: float | None = None
+    tpot: float | None = None
+
+
+@dataclass
+class CompletionResult:
+    """Normalised result of a non-streaming chat completion."""
+    content: str = ""
+    model: str = ""
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    finish_reason: str = "stop"
+    response_id: str = ""
+    timing: TimingInfo = field(default_factory=TimingInfo)
+
+
+@dataclass
+class StreamChunk:
+    """One chunk from a streaming completion."""
+    content: str = ""
+    done: bool = False
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    finish_reason: str = ""
+    timing: TimingInfo = field(default_factory=TimingInfo)
+
+
+class LLMProvider(ABC):
+    """Interface that every LLM backend must implement."""
+
+    def __init__(self, config: ProviderConfig) -> None:
+        self.config = config
+        self.base_url = config.base_url.rstrip("/")
+        parsed = urlparse(self.base_url)
+        self.host = parsed.hostname or "localhost"
+        self.port = parsed.port or (443 if parsed.scheme == "https" else 80)
+
+    @property
+    @abstractmethod
+    def system_name(self) -> str:
+        """OTel ``gen_ai.system`` value (e.g. ``"ollama"``, ``"openai"``)."""
+
+    @abstractmethod
+    def build_payload(
+        self,
+        model: str,
+        messages: list[dict],
+        stream: bool,
+        **kwargs,
+    ) -> dict:
+        """Translate OpenAI-format request into provider-native payload."""
+
+    @abstractmethod
+    async def complete(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> CompletionResult:
+        """Non-streaming chat completion."""
+
+    @abstractmethod
+    async def stream(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> AsyncIterator[StreamChunk]:
+        """Streaming chat completion — yields ``StreamChunk``s."""
+
+    @abstractmethod
+    async def list_models(self, client: httpx.AsyncClient) -> list[dict]:
+        """Return models in OpenAI list format ``[{"id": ..., "object": "model", ...}]``."""

llm_otel_kit-0.1.0/src/llm_otel_kit/providers/ollama.py

@@ -0,0 +1,113 @@
+"""Ollama native API provider (``/api/chat``)."""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import AsyncIterator
+
+import httpx
+
+from llm_otel_kit.providers.base import (
+    CompletionResult,
+    LLMProvider,
+    StreamChunk,
+    TimingInfo,
+)
+
+
+class OllamaProvider(LLMProvider):
+    """Provider for Ollama running locally or on a remote host."""
+
+    @property
+    def system_name(self) -> str:
+        return "ollama"
+
+    def build_payload(
+        self,
+        model: str,
+        messages: list[dict],
+        stream: bool,
+        **kwargs,
+    ) -> dict:
+        payload: dict = {
+            "model": model,
+            "messages": messages,
+            "stream": stream,
+            "options": {},
+        }
+        if kwargs.get("temperature") is not None:
+            payload["options"]["temperature"] = kwargs["temperature"]
+        if kwargs.get("top_p") is not None:
+            payload["options"]["top_p"] = kwargs["top_p"]
+        if kwargs.get("max_tokens") is not None:
+            payload["options"]["num_predict"] = kwargs["max_tokens"]
+        return payload
+
+    async def complete(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> CompletionResult:
+        response = await client.post(f"{self.base_url}/api/chat", json=payload)
+        response.raise_for_status()
+        data = response.json()
+
+        prompt_eval_ns = data.get("prompt_eval_duration", 0)
+        eval_ns = data.get("eval_duration", 0)
+        completion_tokens = data.get("eval_count", 0)
+
+        ttft = (prompt_eval_ns / 1e9) if prompt_eval_ns > 0 else None
+        tpot = None
+        if eval_ns > 0 and completion_tokens > 1:
+            tpot = (eval_ns / 1e9) / (completion_tokens - 1)
+
+        return CompletionResult(
+            content=data.get("message", {}).get("content", ""),
+            model=data.get("model", payload.get("model", "")),
+            prompt_tokens=data.get("prompt_eval_count", 0),
+            completion_tokens=completion_tokens,
+            finish_reason="stop",
+            timing=TimingInfo(ttft=ttft, tpot=tpot),
+        )
+
+    async def stream(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> AsyncIterator[StreamChunk]:
+        async with client.stream("POST", f"{self.base_url}/api/chat", json=payload) as resp:
+            resp.raise_for_status()
+            async for line in resp.aiter_lines():
+                if not line:
+                    continue
+                try:
+                    chunk = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+
+                content = chunk.get("message", {}).get("content", "")
+                done = chunk.get("done", False)
+
+                sc = StreamChunk(content=content, done=done)
+                if done:
+                    prompt_eval_ns = chunk.get("prompt_eval_duration", 0)
+                    eval_ns = chunk.get("eval_duration", 0)
+                    sc.prompt_tokens = chunk.get("prompt_eval_count", 0)
+                    sc.completion_tokens = chunk.get("eval_count", 0)
+                    sc.finish_reason = "stop"
+                    sc.timing = TimingInfo(
+                        ttft=(prompt_eval_ns / 1e9) if prompt_eval_ns > 0 else None,
+                        tpot=((eval_ns / 1e9) / (sc.completion_tokens - 1)
+                              if eval_ns > 0 and sc.completion_tokens > 1 else None),
+                    )
+                yield sc
+
+    async def list_models(self, client: httpx.AsyncClient) -> list[dict]:
+        response = await client.get(f"{self.base_url}/api/tags")
+        response.raise_for_status()
+        return [
+            {"id": m["name"], "object": "model",
+             "created": int(time.time()), "owned_by": "ollama"}
+            for m in response.json().get("models", [])
+        ]

llm_otel_kit-0.1.0/src/llm_otel_kit/providers/openai_compat.py

@@ -0,0 +1,137 @@
+"""OpenAI-compatible API provider.
+
+Works with: OpenAI, Azure OpenAI, vLLM, llama.cpp (server mode),
+LM Studio, Groq, Together.ai, Fireworks.ai, LiteLLM, LocalAI, etc.
+"""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import AsyncIterator
+
+import httpx
+
+from llm_otel_kit.providers.base import (
+    CompletionResult,
+    LLMProvider,
+    StreamChunk,
+    TimingInfo,
+)
+
+
+class OpenAICompatProvider(LLMProvider):
+    """Provider for any OpenAI-compatible API endpoint."""
+
+    @property
+    def system_name(self) -> str:
+        return self.config.name if self.config.name != "openai" else "openai"
+
+    def _headers(self) -> dict[str, str]:
+        h: dict[str, str] = {"Content-Type": "application/json"}
+        if self.config.api_key:
+            h["Authorization"] = f"Bearer {self.config.api_key}"
+        return h
+
+    def build_payload(
+        self,
+        model: str,
+        messages: list[dict],
+        stream: bool,
+        **kwargs,
+    ) -> dict:
+        payload: dict = {
+            "model": model,
+            "messages": messages,
+            "stream": stream,
+        }
+        if kwargs.get("temperature") is not None:
+            payload["temperature"] = kwargs["temperature"]
+        if kwargs.get("top_p") is not None:
+            payload["top_p"] = kwargs["top_p"]
+        if kwargs.get("max_tokens") is not None:
+            payload["max_tokens"] = kwargs["max_tokens"]
+        if stream:
+            payload["stream_options"] = {"include_usage": True}
+        return payload
+
+    async def complete(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> CompletionResult:
+        response = await client.post(
+            f"{self.base_url}/v1/chat/completions",
+            json=payload,
+            headers=self._headers(),
+        )
+        response.raise_for_status()
+        data = response.json()
+
+        choice = data.get("choices", [{}])[0]
+        usage = data.get("usage", {})
+
+        return CompletionResult(
+            content=choice.get("message", {}).get("content", ""),
+            model=data.get("model", payload.get("model", "")),
+            prompt_tokens=usage.get("prompt_tokens", 0),
+            completion_tokens=usage.get("completion_tokens", 0),
+            finish_reason=choice.get("finish_reason", "stop"),
+            response_id=data.get("id", ""),
+            timing=TimingInfo(),  # OpenAI API doesn't expose server-side timing
+        )
+
+    async def stream(
+        self,
+        client: httpx.AsyncClient,
+        payload: dict,
+    ) -> AsyncIterator[StreamChunk]:
+        async with client.stream(
+            "POST",
+            f"{self.base_url}/v1/chat/completions",
+            json=payload,
+            headers=self._headers(),
+        ) as resp:
+            resp.raise_for_status()
+            async for line in resp.aiter_lines():
+                if not line.startswith("data: "):
+                    continue
+                data_str = line[6:]
+                if data_str.strip() == "[DONE]":
+                    return
+                try:
+                    chunk = json.loads(data_str)
+                except json.JSONDecodeError:
+                    continue
+
+                choice = chunk.get("choices", [{}])[0]
+                delta = choice.get("delta", {})
+                content = delta.get("content", "")
+                finish_reason = choice.get("finish_reason")
+
+                # Usage arrives in the final chunk when stream_options.include_usage is set
+                usage = chunk.get("usage") or {}
+
+                sc = StreamChunk(
+                    content=content,
+                    done=finish_reason is not None,
+                    finish_reason=finish_reason or "",
+                    prompt_tokens=usage.get("prompt_tokens", 0),
+                    completion_tokens=usage.get("completion_tokens", 0),
+                    timing=TimingInfo(),
+                )
+                yield sc
+
+    async def list_models(self, client: httpx.AsyncClient) -> list[dict]:
+        response = await client.get(
+            f"{self.base_url}/v1/models",
+            headers=self._headers(),
+        )
+        response.raise_for_status()
+        data = response.json()
+        return [
+            {"id": m["id"], "object": "model",
+             "created": m.get("created", int(time.time())),
+             "owned_by": m.get("owned_by", self.system_name)}
+            for m in data.get("data", [])
+        ]

llm_otel_kit-0.1.0/src/llm_otel_kit/spans.py

@@ -0,0 +1,185 @@
+"""GenAI span attribute helpers and request classification."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+
+from opentelemetry.trace import Span
+
+from llm_otel_kit.metrics import GenAIMetrics
+
+# ---------------------------------------------------------------------------
+# Provider detection from model name
+# ---------------------------------------------------------------------------
+_PROVIDER_PATTERNS: list[tuple[list[str], str]] = [
+    (["gpt-", "o1-", "o3-", "o4-", "dall-e", "text-embedding"], "openai"),
+    (["claude-"], "anthropic"),
+    (["gemini-"], "google"),
+    (["copilot-", "github/"], "github.copilot"),
+    (["mistral-", "mixtral-", "codestral-"], "mistral"),
+    (["command-", "embed-"], "cohere"),
+    (["deepseek-"], "deepseek"),
+]
+
+_MAX_CONTENT_LEN = 500
+
+
+def detect_provider(model: str) -> str:
+    """Infer ``gen_ai.system`` from model name prefix."""
+    model_lower = model.lower()
+    for prefixes, provider in _PROVIDER_PATTERNS:
+        if any(model_lower.startswith(p) for p in prefixes):
+            return provider
+    return "ollama"
+
+
+def classify_request(messages: list[dict]) -> str:
+    """Classify an OpenAI-format message list into a purpose label."""
+    last_content = (messages[-1].get("content", "") if messages else "").lower()
+    if "generate a concise" in last_content and "title" in last_content:
+        return "Title Generation"
+    if ("generate tags" in last_content or "categorize" in last_content
+            or "tag the conversation" in last_content):
+        return "Tag Generation"
+    if "follow-up" in last_content or ("suggest" in last_content and "question" in last_content):
+        return "Suggestion Generation"
+    if messages and all(msg.get("role") == "system" for msg in messages):
+        return "System Prompt"
+    return "User Chat"
+
+
+def _truncate(text: str) -> str:
+    return text[:_MAX_CONTENT_LEN] + "..." if len(text) > _MAX_CONTENT_LEN else text
+
+
+# ---------------------------------------------------------------------------
+# Span attribute setters (OTel GenAI semconv)
+# ---------------------------------------------------------------------------
+
+def semconv_attrs(model: str, server_host: str, server_port: int) -> dict:
+    """Build the standard GenAI metric attribute dict."""
+    return {
+        "gen_ai.operation.name": "chat",
+        "gen_ai.system": detect_provider(model),
+        "gen_ai.request.model": model,
+        "gen_ai.response.model": model,
+        "server.address": server_host,
+        "server.port": server_port,
+    }
+
+
+def set_genai_span(
+    span: Span,
+    model: str,
+    request_type: str,
+    stream: bool,
+    messages: list[dict],
+    server_host: str,
+    server_port: int,
+    *,
+    temperature: float | None = None,
+    top_p: float | None = None,
+    max_tokens: int | None = None,
+    auth_header: str = "",
+) -> None:
+    """Set all gen_ai.* request attributes and input span event."""
+    provider = detect_provider(model)
+
+    span.update_name(f"{request_type} · {model}")
+
+    span.set_attribute("gen_ai.system", provider)
+    span.set_attribute("gen_ai.provider.name", provider)
+    span.set_attribute("gen_ai.operation.name", "chat")
+    span.set_attribute("gen_ai.request.model", model)
+    span.set_attribute("llm.request.type", "chat")
+    span.set_attribute("llm.is_streaming", stream)
+    span.set_attribute("llm.request.purpose", request_type)
+    span.set_attribute("server.address", server_host)
+    span.set_attribute("server.port", server_port)
+
+    if temperature is not None:
+        span.set_attribute("gen_ai.request.temperature", temperature)
+    if top_p is not None:
+        span.set_attribute("gen_ai.request.top_p", top_p)
+    if max_tokens is not None:
+        span.set_attribute("gen_ai.request.max_tokens", max_tokens)
+
+    # Indexed prompt attribute (last user message)
+    for msg in reversed(messages):
+        if msg.get("role") == "user":
+            span.set_attribute("gen_ai.prompt.0.role", "user")
+            span.set_attribute("gen_ai.prompt.0.content", msg.get("content", ""))
+            break
+
+    # Span event
+    for msg in reversed(messages):
+        if msg.get("role") == "user":
+            span.add_event("gen_ai.user.message", {
+                "gen_ai.prompt.role": "user",
+                "gen_ai.prompt.content": _truncate(msg.get("content", "")),
+            })
+            break
+
+    # Conversation fingerprint
+    user_msgs = [msg["content"] for msg in messages if msg["role"] == "user"]
+    fp_input = user_msgs[0][:200] if (request_type != "User Chat" and user_msgs) else "|".join(user_msgs)
+    span.set_attribute("conversation.fingerprint",
+                       hashlib.sha256(fp_input.encode()).hexdigest()[:12])
+
+    if auth_header:
+        span.set_attribute("enduser.id",
+                           hashlib.sha256(auth_header.encode()).hexdigest()[:8])
+
+
+def set_genai_response(
+    span: Span,
+    content: str,
+    model: str,
+    prompt_tokens: int,
+    completion_tokens: int,
+    finish_reason: str = "stop",
+    response_id: str = "",
+) -> None:
+    """Set response attributes and assistant span event."""
+    span.set_attribute("gen_ai.completion.0.role", "assistant")
+    span.set_attribute("gen_ai.completion.0.content", content)
+    span.set_attribute("gen_ai.completion.0.finish_reason", finish_reason)
+
+    span.set_attribute("gen_ai.response.model", model)
+    span.set_attribute("gen_ai.response.finish_reasons", json.dumps([finish_reason]))
+    span.set_attribute("gen_ai.usage.input_tokens", prompt_tokens)
+    span.set_attribute("gen_ai.usage.output_tokens", completion_tokens)
+    span.set_attribute("gen_ai.usage.prompt_tokens", prompt_tokens)
+    span.set_attribute("gen_ai.usage.completion_tokens", completion_tokens)
+    if response_id:
+        span.set_attribute("gen_ai.response.id", response_id)
+
+    span.add_event("gen_ai.assistant.message", {
+        "gen_ai.completion.role": "assistant",
+        "gen_ai.completion.content": _truncate(content),
+        "gen_ai.completion.finish_reason": finish_reason,
+    })
+
+
+def record_metrics(
+    m: GenAIMetrics,
+    attrs: dict,
+    model: str,
+    duration: float,
+    prompt_tokens: int,
+    completion_tokens: int,
+    ttft: float | None = None,
+    tpot: float | None = None,
+) -> None:
+    """Record all GenAI + operational metrics for one completed request."""
+    m.operation_duration.record(duration, attrs)
+    m.token_usage.record(prompt_tokens, {**attrs, "gen_ai.token.type": "input"})
+    m.token_usage.record(completion_tokens, {**attrs, "gen_ai.token.type": "output"})
+    if ttft is not None:
+        m.ttft.record(ttft, attrs)
+    if tpot is not None:
+        m.tpot.record(tpot, attrs)
+    if duration > 0 and completion_tokens > 0:
+        m.token_throughput.record(completion_tokens / duration, attrs)
+    m.active_requests.add(-1, {"model": model})