ModelMetre 0.1.0__tar.gz

modelmetre-0.1.0/PKG-INFO

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: ModelMetre
+Version: 0.1.0
+Summary: Multi-provider LLM cost observability SDK
+Author-email: Jyotir Nandan <jyotirnandan92129@gmail.com>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/Kakashi15-pix/ModelMetre
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.31.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: httpx>=0.24.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.11.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: ruff>=0.0.280; extra == "dev"
+Requires-Dist: mypy>=1.4.0; extra == "dev"
+
+# LLM Cost Observability SDK
+
+Client-side SDK for provider response interception, usage extraction, request buffering, and optional telemetry flush to a backend cost pipeline.
+## What This SDK Does
+
+- Wraps provider SDK methods without changing normal response behavior.
+- Extracts usage metadata from provider responses locally.
+- Buffers request details in-memory.
+- Flushes buffered batches on size/time thresholds.
+- Optionally sends flushed batches to backend telemetry endpoint.
+- Supports lazy API-key verification for direct server API calls via `CostAnalyticsClient`.
+
+## What This SDK Does Not Do On Client
+
+- Does not perform authoritative upstream pricing sync on the client.
+- Does not do final backend-grade cost computation in the main intercept-and-flush path.
+- Does not require provider credentials to leave the client environment.
+
+## Architecture (Current)
+
+1. Your app calls provider SDK (Anthropic/OpenAI/custom).
+2. Wrapped method runs, returns original provider response.
+3. Interceptor extracts usage/model/stop reason.
+4. `RequestDetailsBuffer` stores request details locally.
+5. Buffer flushes when:
+   - `FLUSH_BATCH_SIZE` reached (default `50`), or
+   - timer hits `FLUSH_INTERVAL_SECONDS` (default `30`).
+6. If telemetry is configured, flushed batch is POSTed to backend (`/v1/telemetry/flush` by default).
+7. Backend resolves pricing and computes final costs.
+
+## Package Layout (Relevant)
+
+- `my-sdk/src/sdk.py`: `CostAnalyticsSDK` facade.
+- `my-sdk/src/pricing/interceptor.py`: wrapping and extraction pipeline.
+- `my-sdk/src/pricing/extractors.py`: provider extractors.
+- `my-sdk/src/pricing/aggregator.py`: request buffer + flush triggers.
+- `my-sdk/src/api/telemetry.py`: telemetry sender with failed flush retention/retry behavior.
+- `my-sdk/src/client.py`: authenticated API client (`CostAnalyticsClient`) with lazy key verification.
+- `my-sdk/src/auth/Config.py`: env-based API key loading (`CA_API_KEY`).
+
+## Install
+
+```bash
+pip install -e .
+```
+
+## Quick Start
+
+### 1) Wrap a Provider Client
+
+```python
+from anthropic import Anthropic
+from sdk import CostAnalyticsSDK
+
+sdk = CostAnalyticsSDK(server_url="https://telemetry.example.com")
+
+client = Anthropic()
+client = sdk.wrap_client(
+    client=client,
+    provider="anthropic",
+    method_path="messages.create",
+)
+
+response = client.messages.create(
+    model="claude-3-haiku-20240307",
+    max_tokens=128,
+    messages=[{"role": "user", "content": "Hello"}],
+)
+
+print(sdk.get_metrics())
+```
+
+### 2) Wrap Other Client Shapes
+
+```python
+from sdk import CostAnalyticsSDK
+
+sdk = CostAnalyticsSDK(server_url="https://telemetry.example.com")
+
+client = sdk.wrap_client(
+    client=client,
+    provider="custom-provider",
+    method_path="responses.create",
+)
+```
+
+### 3) Manual Flush
+
+```python
+from pricing import get_request_buffer
+
+buffer = get_request_buffer()
+buffer.flush()
+```
+
+## Core APIs
+
+### `CostAnalyticsSDK` (`my-sdk/src/sdk.py`)
+
+- `CostAnalyticsSDK(server_url: Optional[str] = None)`
+- `wrap_client(client, provider, method_path, response_to_dict=None, metadata=None)`
+- `process_response(response, provider, request_id=None, metadata=None)`
+- `get_metrics()`
+- `get_pending_requests()`
+- `flush_buffer()`
+
+`get_metrics()` currently returns buffer-oriented metrics:
+
+```json
+{
+  "buffer_size": 12,
+  "pending_requests": 12
+}
+```
+
+### Buffer (`my-sdk/src/pricing/aggregator.py`)
+
+- `FLUSH_BATCH_SIZE = 50`
+- `FLUSH_INTERVAL_SECONDS = 30`
+- `get_request_buffer()` / `get_cost_aggregator()`
+- `RequestDetailsBuffer.record_request(...)`
+- `RequestDetailsBuffer.flush()`
+- `RequestDetailsBuffer.get_pending_requests()`
+
+### Telemetry (`my-sdk/src/api/telemetry.py`)
+
+`TelemetryClient` behavior:
+
+- Sends flush payload to `POST {server_url}/v1/telemetry/flush` (default endpoint path).
+- Includes headers:
+  - `Content-Type: application/json`
+  - `X-Client-ID: <uuid>`
+  - optional `Authorization: Bearer <api_key>`
+- On flush failure, retains up to last 5 failed batches in-memory.
+- If failure looks like "request not received", retries once immediately.
+- Failed batches are retried on subsequent flush attempts.
+
+## Auth for SDK-to-Server API Calls
+
+`CostAnalyticsClient` (`my-sdk/src/client.py`) supports direct authenticated calls to server APIs.
+
+- Reads `CA_API_KEY` when `api_key` not passed.
+- Performs lazy auth verification against `GET /v1/auth/verify` (default path).
+- Sends request metadata headers:
+  - `Authorization`
+  - `X-CA-Key-Id`
+  - `X-CA-User-Id`
+  - `X-Request-Id`
+  - `X-CA-Provider`
+  - `X-CA-Model`
+
+Example:
+
+```python
+from client import CostAnalyticsClient
+
+client = CostAnalyticsClient(
+    api_key="ca_live_...",
+    base_url="https://api.example.com",
+)
+
+resp = client.request("GET", "/v1/costs")
+print(resp.status_code)
+```
+
+## Environment Variables
+
+- `CA_API_KEY`: client API key for `CostAnalyticsClient`.
+- `CA_API_BASE_URL`: optional base URL override for `CostAnalyticsClient`.
+
+Note: backend services may require additional variables (for example server HMAC secret) that are configured in the server repository.
+
+## Backend Contract Expectations
+
+Telemetry flush endpoint should accept payload in this shape:
+
+```json
+{
+  "client_id": "uuid",
+  "batch": [
+    {
+      "timestamp": "2026-01-01T00:00:00.000000",
+      "request_id": "req-123",
+      "model": "claude-3-haiku-20240307",
+      "provider": "anthropic",
+      "input_tokens": 100,
+      "output_tokens": 50,
+      "cache_read_tokens": 0,
+      "cache_creation_tokens": 0,
+      "stop_reason": "end_turn",
+      "metadata": {"method": "messages.create"}
+    }
+  ]
+}
+```
+
+Auth verification endpoint expected by `CostAnalyticsClient`:
+
+- `GET /v1/auth/verify`
+- Response:
+
+```json
+{
+  "user_id": "...",
+  "api_key_id": "..."
+}
+```
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## Notes on Compatibility Surface
+
+The SDK exposes one generic extractor/interceptor path. Provider names are metadata values passed to `wrap_client`; authoritative pricing resolution lives on the backend.

modelmetre-0.1.0/README.md

@@ -0,0 +1,218 @@
+# LLM Cost Observability SDK
+
+Client-side SDK for provider response interception, usage extraction, request buffering, and optional telemetry flush to a backend cost pipeline.
+## What This SDK Does
+
+- Wraps provider SDK methods without changing normal response behavior.
+- Extracts usage metadata from provider responses locally.
+- Buffers request details in-memory.
+- Flushes buffered batches on size/time thresholds.
+- Optionally sends flushed batches to backend telemetry endpoint.
+- Supports lazy API-key verification for direct server API calls via `CostAnalyticsClient`.
+
+## What This SDK Does Not Do On Client
+
+- Does not perform authoritative upstream pricing sync on the client.
+- Does not do final backend-grade cost computation in the main intercept-and-flush path.
+- Does not require provider credentials to leave the client environment.
+
+## Architecture (Current)
+
+1. Your app calls provider SDK (Anthropic/OpenAI/custom).
+2. Wrapped method runs, returns original provider response.
+3. Interceptor extracts usage/model/stop reason.
+4. `RequestDetailsBuffer` stores request details locally.
+5. Buffer flushes when:
+   - `FLUSH_BATCH_SIZE` reached (default `50`), or
+   - timer hits `FLUSH_INTERVAL_SECONDS` (default `30`).
+6. If telemetry is configured, flushed batch is POSTed to backend (`/v1/telemetry/flush` by default).
+7. Backend resolves pricing and computes final costs.
+
+## Package Layout (Relevant)
+
+- `my-sdk/src/sdk.py`: `CostAnalyticsSDK` facade.
+- `my-sdk/src/pricing/interceptor.py`: wrapping and extraction pipeline.
+- `my-sdk/src/pricing/extractors.py`: provider extractors.
+- `my-sdk/src/pricing/aggregator.py`: request buffer + flush triggers.
+- `my-sdk/src/api/telemetry.py`: telemetry sender with failed flush retention/retry behavior.
+- `my-sdk/src/client.py`: authenticated API client (`CostAnalyticsClient`) with lazy key verification.
+- `my-sdk/src/auth/Config.py`: env-based API key loading (`CA_API_KEY`).
+
+## Install
+
+```bash
+pip install -e .
+```
+
+## Quick Start
+
+### 1) Wrap a Provider Client
+
+```python
+from anthropic import Anthropic
+from sdk import CostAnalyticsSDK
+
+sdk = CostAnalyticsSDK(server_url="https://telemetry.example.com")
+
+client = Anthropic()
+client = sdk.wrap_client(
+    client=client,
+    provider="anthropic",
+    method_path="messages.create",
+)
+
+response = client.messages.create(
+    model="claude-3-haiku-20240307",
+    max_tokens=128,
+    messages=[{"role": "user", "content": "Hello"}],
+)
+
+print(sdk.get_metrics())
+```
+
+### 2) Wrap Other Client Shapes
+
+```python
+from sdk import CostAnalyticsSDK
+
+sdk = CostAnalyticsSDK(server_url="https://telemetry.example.com")
+
+client = sdk.wrap_client(
+    client=client,
+    provider="custom-provider",
+    method_path="responses.create",
+)
+```
+
+### 3) Manual Flush
+
+```python
+from pricing import get_request_buffer
+
+buffer = get_request_buffer()
+buffer.flush()
+```
+
+## Core APIs
+
+### `CostAnalyticsSDK` (`my-sdk/src/sdk.py`)
+
+- `CostAnalyticsSDK(server_url: Optional[str] = None)`
+- `wrap_client(client, provider, method_path, response_to_dict=None, metadata=None)`
+- `process_response(response, provider, request_id=None, metadata=None)`
+- `get_metrics()`
+- `get_pending_requests()`
+- `flush_buffer()`
+
+`get_metrics()` currently returns buffer-oriented metrics:
+
+```json
+{
+  "buffer_size": 12,
+  "pending_requests": 12
+}
+```
+
+### Buffer (`my-sdk/src/pricing/aggregator.py`)
+
+- `FLUSH_BATCH_SIZE = 50`
+- `FLUSH_INTERVAL_SECONDS = 30`
+- `get_request_buffer()` / `get_cost_aggregator()`
+- `RequestDetailsBuffer.record_request(...)`
+- `RequestDetailsBuffer.flush()`
+- `RequestDetailsBuffer.get_pending_requests()`
+
+### Telemetry (`my-sdk/src/api/telemetry.py`)
+
+`TelemetryClient` behavior:
+
+- Sends flush payload to `POST {server_url}/v1/telemetry/flush` (default endpoint path).
+- Includes headers:
+  - `Content-Type: application/json`
+  - `X-Client-ID: <uuid>`
+  - optional `Authorization: Bearer <api_key>`
+- On flush failure, retains up to last 5 failed batches in-memory.
+- If failure looks like "request not received", retries once immediately.
+- Failed batches are retried on subsequent flush attempts.
+
+## Auth for SDK-to-Server API Calls
+
+`CostAnalyticsClient` (`my-sdk/src/client.py`) supports direct authenticated calls to server APIs.
+
+- Reads `CA_API_KEY` when `api_key` not passed.
+- Performs lazy auth verification against `GET /v1/auth/verify` (default path).
+- Sends request metadata headers:
+  - `Authorization`
+  - `X-CA-Key-Id`
+  - `X-CA-User-Id`
+  - `X-Request-Id`
+  - `X-CA-Provider`
+  - `X-CA-Model`
+
+Example:
+
+```python
+from client import CostAnalyticsClient
+
+client = CostAnalyticsClient(
+    api_key="ca_live_...",
+    base_url="https://api.example.com",
+)
+
+resp = client.request("GET", "/v1/costs")
+print(resp.status_code)
+```
+
+## Environment Variables
+
+- `CA_API_KEY`: client API key for `CostAnalyticsClient`.
+- `CA_API_BASE_URL`: optional base URL override for `CostAnalyticsClient`.
+
+Note: backend services may require additional variables (for example server HMAC secret) that are configured in the server repository.
+
+## Backend Contract Expectations
+
+Telemetry flush endpoint should accept payload in this shape:
+
+```json
+{
+  "client_id": "uuid",
+  "batch": [
+    {
+      "timestamp": "2026-01-01T00:00:00.000000",
+      "request_id": "req-123",
+      "model": "claude-3-haiku-20240307",
+      "provider": "anthropic",
+      "input_tokens": 100,
+      "output_tokens": 50,
+      "cache_read_tokens": 0,
+      "cache_creation_tokens": 0,
+      "stop_reason": "end_turn",
+      "metadata": {"method": "messages.create"}
+    }
+  ]
+}
+```
+
+Auth verification endpoint expected by `CostAnalyticsClient`:
+
+- `GET /v1/auth/verify`
+- Response:
+
+```json
+{
+  "user_id": "...",
+  "api_key_id": "..."
+}
+```
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## Notes on Compatibility Surface
+
+The SDK exposes one generic extractor/interceptor path. Provider names are metadata values passed to `wrap_client`; authoritative pricing resolution lives on the backend.

modelmetre-0.1.0/pyproject.toml

@@ -0,0 +1,57 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ModelMetre"
+version = "0.1.0"
+description = "Multi-provider LLM cost observability SDK"
+readme = "README.md"
+requires-python = ">=3.9"
+license ="MIT"
+authors = [
+    {name = "Jyotir Nandan", email = "jyotirnandan92129@gmail.com"}
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+
+dependencies = [
+    "requests>=2.31.0",
+    "pydantic>=2.0.0",
+    "httpx>=0.24.0",
+]
+[project.urls]
+Repository = "https://github.com/Kakashi15-pix/ModelMetre"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.4.0",
+    "pytest-cov>=4.1.0",
+    "pytest-mock>=3.11.0",
+    "black>=23.0.0",
+    "ruff>=0.0.280",
+    "mypy>=1.4.0",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.black]
+line-length = 88
+target-version = ['py39']
+
+[tool.ruff]
+line-length = 88
+target-version = "py39"
+
+[tool.mypy]
+python_version = "3.9"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = false

modelmetre-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+