frugal-sdk-python 0.4.0__tar.gz → 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/PKG-INFO +75 -20
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/README.md +74 -19
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/docs/metrics-reference.md +79 -11
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/docs/quickstart.md +155 -25
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/dashboards/frugal-metrics.json +1 -1
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/block_hash_lru.py +22 -6
- frugal_sdk_python-0.6.0/src/frugal_metrics/config.py +739 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/debug.py +6 -2
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/instrument.py +238 -87
- frugal_sdk_python-0.6.0/src/frugal_metrics/model_cap.py +103 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/otel.py +280 -35
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/providers.py +45 -21
- frugal_sdk_python-0.6.0/src/frugal_metrics/safe.py +58 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/semconv.py +18 -4
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/streams.py +5 -4
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/track.py +13 -6
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/_batch.py +20 -12
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/anthropic.py +4 -4
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/bedrock.py +4 -4
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/openai.py +4 -4
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/conftest.py +33 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_block_hash_lru.py +31 -2
- frugal_sdk_python-0.6.0/tests/test_config.py +741 -0
- frugal_sdk_python-0.6.0/tests/test_model_cap.py +413 -0
- frugal_sdk_python-0.6.0/tests/test_never_impact_host.py +270 -0
- frugal_sdk_python-0.6.0/tests/test_node_parity.py +791 -0
- frugal_sdk_python-0.6.0/tests/test_otel_overflow_detection.py +439 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_providers.py +31 -0
- frugal_sdk_python-0.6.0/tests/test_sibling_parity.py +79 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_track_ai_call.py +130 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_anthropic_batches.py +52 -0
- frugal_sdk_python-0.4.0/src/frugal_metrics/config.py +0 -448
- frugal_sdk_python-0.4.0/src/frugal_metrics/safe.py +0 -45
- frugal_sdk_python-0.4.0/tests/test_config.py +0 -357
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/.github/workflows/cd.yml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/.gitignore +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/LICENSES/Apache-2.0.txt +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/REUSE.toml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/README.md +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/collector/otel-collector-config.yaml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/docker-compose.yml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/provisioning/dashboards/frugal.yml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/provisioning/datasources/prometheus.yml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/Dockerfile +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/package.json +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/src/server.js +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/index.html.example +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/models.html +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/overview.html +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/sites.html +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/prometheus.yml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/Dockerfile +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/clients.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/anon.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/classify.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/deferred.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/fail.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/pipeline.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/summarize.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/translate.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/via_blocked_lib.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/via_unblocked_lib.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/internal_lib/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/internal_lib/ai_helper.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/main.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/sibling_lib/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/sibling_lib/ai_helper.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/pyproject.toml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/requirements.txt +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/scripts/verify.sh +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/pyproject.toml +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/caller.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/duplicate_filter.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/prompt_analyzer.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/shutdown.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/_patching.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/bedrock_models.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/__init__.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_bedrock_models.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_caller.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_caller_edge_cases.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_debug.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_duplicate_filter.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_prompt_analyzer.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_shutdown.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_anthropic.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_bedrock.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai_analyzer.py +0 -0
- {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai_batches.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: frugal-sdk-python
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.6.0
|
|
4
4
|
Summary: Instrumentation for Cost-to-Code attribution.
|
|
5
5
|
Project-URL: Homepage, https://frugal.co
|
|
6
6
|
Author: Frugal AI Inc.
|
|
@@ -62,7 +62,7 @@ All configuration is via environment variables. No `init()` call, no configurati
|
|
|
62
62
|
|
|
63
63
|
### Required
|
|
64
64
|
|
|
65
|
-
|
|
65
|
+
Console mode (`FRUGAL_METRICS_EXPORTER=console`) needs only `FRUGAL_PROJECT_ID`; every other mode needs both. If a required variable is unset, the wrapper silently returns the client unmodified:
|
|
66
66
|
|
|
67
67
|
| Variable | Purpose |
|
|
68
68
|
| ------------------- | ---------------------------------------------------------- |
|
|
@@ -71,21 +71,27 @@ Set both or the wrapper silently returns the client unmodified:
|
|
|
71
71
|
|
|
72
72
|
### Optional
|
|
73
73
|
|
|
74
|
+
The export endpoint and authentication come only from `FRUGAL_`-prefixed variables, so the SDK never crosses wires with the host's own OTel pipeline: the generic `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are never read. (The OTel exporter library still reads its own generic TLS, compression, timeout and temporality variables.)
|
|
75
|
+
|
|
74
76
|
| Variable | Purpose |
|
|
75
77
|
| ---------------------------------------- | --------------------------------------------------------------------------------------- |
|
|
76
78
|
| `FRUGAL_CUSTOMER_ID` | Customer ID issued by Frugal. When unset, the tenant label (auto-injected server-side by VMAuth from the bearer token) identifies the customer and `frugal.customer_id` is omitted. |
|
|
77
79
|
| `FRUGAL_REPO_ROOT` | Absolute path to the source root; the strip prefix for `caller_file`. Defaults to the process working directory. |
|
|
78
|
-
| `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics
|
|
79
|
-
| `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`).
|
|
80
|
-
| `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. |
|
|
80
|
+
| `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics` (also when set but empty). |
|
|
81
|
+
| `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Precedence: this variable wins; otherwise the header built from `FRUGAL_API_KEY` (`FRUGAL_METRICS_API_KEY` first), which is sent verbatim. Values set here are percent-decoded (`Bearer%20tok` → `Bearer tok`). A value with no `key=value` pair is treated as missing auth (the API key is not used instead); a blank (whitespace-only) value is treated as unset. |
|
|
82
|
+
| `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. Deduped, and only the first **10** entries are honored (hard cap, not configurable); the rest are dropped at startup with one debug warning. |
|
|
81
83
|
| `FRUGAL_CUSTOM_PROVIDERS` | Comma-separated allowlist of provider names beyond the built-in registry (e.g. `together_ai,fireworks`) that should pass through verbatim on `gen_ai.system` / `gen_ai.provider.name` instead of being coerced to `custom`. Capped at 10 (extras are coerced) to keep label cardinality bounded. |
|
|
82
|
-
| `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label
|
|
84
|
+
| `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`. The vocabulary is the first **20** comma-separated entries by raw position (hard cap, not configurable); empty entries are ignored and duplicates collapsed. `other` is reserved (the coercion sentinel) and is ignored with one warning. A value outside the allowlist is coerced to `other` with a bounded once-per-value warning; when the variable is unset, every value is `other`. One warning fires only when non-empty entries past the 20th are actually dropped (a trailing comma does not warn). |
|
|
85
|
+
| `FRUGAL_METRICS_EXPORTER` | `otlp` (default) or `console` (print metrics to stdout; console mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables are ignored). Case-insensitive; any other value warns once and uses `otlp`. |
|
|
86
|
+
| `FRUGAL_METRICS_PROJECT_ID` / `FRUGAL_METRICS_API_KEY` | Namespaced overrides read before `FRUGAL_PROJECT_ID` / `FRUGAL_API_KEY`, for apps that already use those names. An empty value falls through to the bare name. |
|
|
83
87
|
| `FRUGAL_PATH_BLOCK_LIST` | Comma-separated repo-relative paths to skip for caller attribution (see below). |
|
|
84
|
-
| `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds
|
|
85
|
-
| `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute. Default `500`. Set to `0` to disable. |
|
|
86
|
-
| `
|
|
88
|
+
| `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds, `1`-`2147483647`. Default `60000`; a value outside the range warns once and uses the default. |
|
|
89
|
+
| `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute; past it `frugal.caller_file` / `frugal.caller_function` become `<overflow>`. Default `500`. Set to `0` to disable. |
|
|
90
|
+
| `FRUGAL_MAX_MODELS` | Cardinality cap — max distinct model strings to attribute; past it `gen_ai.request.model` / `gen_ai.response.model` become `<overflow>`. One registry shared by both labels, so a request alias and its dated response snapshot each take a slot. Default `100`. Set to `0` to disable. |
|
|
91
|
+
| `FRUGAL_MAX_SERIES_PER_INSTRUMENT` | OpenTelemetry's per-instrument limit on distinct attribute **combinations**, raised from the SDK default. Default `50000`. Set to `0` for effectively unlimited (a 1,000,000 ceiling). Applied **only when the installed `opentelemetry-sdk` accepts a per-instrument cardinality limit** — current releases (through 1.42) do not implement one, so nothing is capped and setting this variable emits one `warn` telling you it has no effect. |
|
|
92
|
+
| `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls that run the lightweight prompt analyzer: a decimal in [0, 1], plain or scientific notation (e.g. `0.25`, `1e-1`); anything else warns once and uses `0.1`. Default `0.1`. Set to `0` to disable, `1` for every call. |
|
|
87
93
|
| `FRUGAL_DUPLICATE_REQUEST_DETECTION` | Duplicate-request detection (the `frugal.gen_ai.duplicate_requests` metric) is **on by default** and hashes each chat request in-process (the hash never leaves the SDK). Set to `0`/`false`/`no`/`off` to disable — no hashing, no filter memory, no metric. |
|
|
88
|
-
| `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `
|
|
94
|
+
| `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `86400` (1 day). |
|
|
89
95
|
| `FRUGAL_DUPLICATE_CAPACITY` | Expected distinct request signatures per window (sizes the in-memory filter). Default `10000`, max `1000000`. Raise it if the SDK warns the filter is saturated. |
|
|
90
96
|
| `FRUGAL_DUPLICATE_FP_RATE` | Target false-positive rate of the duplicate filter. Default `0.01`, clamped to `[1e-6, 0.5]`. |
|
|
91
97
|
| `FRUGAL_DUPLICATE_GENERATIONS` | Rotation granularity of the duplicate filter's time window. Default `2`, max `16`. |
|
|
@@ -103,8 +109,9 @@ Every wrapped call emits OTel metrics tagged with the caller's source file and f
|
|
|
103
109
|
| `frugal.gen_ai.cache_read_tokens` | histogram | Prompt cache hits | — |
|
|
104
110
|
| `frugal.gen_ai.cache_write_tokens` | histogram | Prompt cache writes | — |
|
|
105
111
|
| `frugal.gen_ai.reasoning_tokens` | histogram | OpenAI reasoning tokens (o-series), Anthropic thinking tokens| — |
|
|
112
|
+
| `frugal.gen_ai.agent.turns` | histogram | Model-response turns per successful `invoke_agent` call (manual only) | — |
|
|
106
113
|
| `frugal.gen_ai.errors` | counter | Errors by exception type | `frugal.error.type` |
|
|
107
|
-
| `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality
|
|
114
|
+
| `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality caps, bugs) | `frugal.internal_error.stage`, `frugal.internal_error.type`, `frugal.instrument` |
|
|
108
115
|
| `frugal.gen_ai.output_cap_utilization` | histogram | `output_tokens / max_tokens` ratio when `max_tokens` is set | `frugal.structured_output_mode` |
|
|
109
116
|
| `frugal.gen_ai.history_depth` | histogram | `messages.length` — every chat call (100%, not sampled) | — |
|
|
110
117
|
| `frugal.gen_ai.duplicate_requests` | counter | Repeat-request detection — every chat call (100%, not sampled; disable via `FRUGAL_DUPLICATE_REQUEST_DETECTION`) | `frugal.dup_state` |
|
|
@@ -121,15 +128,17 @@ The analyzer-sampled metrics fire on a fraction of calls
|
|
|
121
128
|
Every metric carries this base attribute set (except
|
|
122
129
|
`frugal.gen_ai.internal_errors`, which is emitted from failure paths where
|
|
123
130
|
these attributes may not exist yet and carries only `frugal.project_id`, the
|
|
124
|
-
stage/type,
|
|
131
|
+
stage/type, `frugal.instrument` on the `otel_cardinality_overflow` stage, and
|
|
132
|
+
`frugal.customer_id` / `frugal.custom_label` when set; `frugal.custom_label`
|
|
133
|
+
never on the setup_failed / emission stages):
|
|
125
134
|
|
|
126
135
|
- `frugal.project_id` — from your env vars (`frugal.customer_id` too when set); `frugal.component` is added on the per-component series when `FRUGAL_COMPONENTS` is set
|
|
127
136
|
- `frugal.caller_file`, `frugal.caller_function` — auto-detected from the call stack
|
|
128
137
|
- `gen_ai.system` + `gen_ai.provider.name` — the provider, dual-emitted under both the legacy and current OTel attribute (`openai`, `anthropic`, `aws.bedrock`, `vertex_ai`/`gcp.vertex_ai`, …, or `custom`). Set via the `system` kwarg; an unrecognized value is coerced to `custom` with a one-time warning. See [`docs/metrics-reference.md`](docs/metrics-reference.md#attributes).
|
|
129
|
-
- `gen_ai.request.model`, `gen_ai.response.model` — the model used
|
|
138
|
+
- `gen_ai.request.model`, `gen_ai.response.model` — the model used; both draw on one shared registry and become `<overflow>` past `FRUGAL_MAX_MODELS` distinct model strings
|
|
130
139
|
- `gen_ai.operation.name` — `chat`, `text_completion`, `embeddings`, `generate_content`, `execute_tool`, `create_agent`, `invoke_agent`, or `other` (the sentinel an unrecognized operation coerces to)
|
|
131
140
|
- `frugal.batched` — `true` for batch-API paths (OpenAI Batch API submissions; Anthropic Message Batches submissions and results), `false` otherwise
|
|
132
|
-
- `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other`
|
|
141
|
+
- `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other` (with a once-per-value warning). `custom_label=None` means no label; a non-string value is coerced to `other`
|
|
133
142
|
|
|
134
143
|
The "Additional attributes" column above lists what each metric carries on top of the base set. See [`docs/metrics-reference.md`](docs/metrics-reference.md) for the values each attribute takes, sentinel values for caller fields, and the full internal-error-stage list.
|
|
135
144
|
|
|
@@ -169,6 +178,39 @@ with track_ai_call(system="anthropic", request_model="claude-sonnet-4-20250514")
|
|
|
169
178
|
)
|
|
170
179
|
```
|
|
171
180
|
|
|
181
|
+
For an enclosing agent invocation, set `operation="invoke_agent"` and report
|
|
182
|
+
the number of model-response steps performed inside it:
|
|
183
|
+
|
|
184
|
+
```python
|
|
185
|
+
with track_ai_call(
|
|
186
|
+
system="anthropic",
|
|
187
|
+
operation="invoke_agent",
|
|
188
|
+
request_model="claude-sonnet-4-20250514",
|
|
189
|
+
) as span:
|
|
190
|
+
result = my_agent_framework_call(prompt)
|
|
191
|
+
span.set_usage(
|
|
192
|
+
input_tokens=result.usage.input_tokens,
|
|
193
|
+
output_tokens=result.usage.output_tokens,
|
|
194
|
+
agent_turns=result.turn_count,
|
|
195
|
+
)
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
- Provider wrappers still measure individual model calls and never set
|
|
199
|
+
`agent_turns`.
|
|
200
|
+
- Agent frameworks such as the Claude Agent SDK, OpenAI Agents SDK, and
|
|
201
|
+
LangGraph do not have dedicated wrappers yet; track them manually as shown.
|
|
202
|
+
- `frugal.gen_ai.history_depth` is the request message count
|
|
203
|
+
(`messages.length`), not a turn count.
|
|
204
|
+
- Missing, non-integer, non-positive, or otherwise invalid `agent_turns`
|
|
205
|
+
values are silently dropped. They never raise or affect customer execution.
|
|
206
|
+
- Turn data is emitted only on success, like token usage.
|
|
207
|
+
- **Avoid double-counting tokens.** Only report aggregate `input_tokens` /
|
|
208
|
+
`output_tokens` on the `invoke_agent` call when the agent's inner
|
|
209
|
+
model-client calls are **not** already wrapped by a supported provider
|
|
210
|
+
wrapper. If they are wrapped, those wrappers already emit token usage per
|
|
211
|
+
inner call — report `agent_turns` only on the `invoke_agent` call in that
|
|
212
|
+
case, or the same tokens are double-counted.
|
|
213
|
+
|
|
172
214
|
The `with` block times the call and emits metrics on exit — or an error metric if the block raises (the exception always propagates). Works the same around an `await` in async code. Call `span.set_usage(...)` once the response is back; if you never call it, you still get a call-count and latency point.
|
|
173
215
|
|
|
174
216
|
For calls that span function boundaries — where a single `with` block won't fit — use the manual handle:
|
|
@@ -191,9 +233,9 @@ except BaseException as exc:
|
|
|
191
233
|
|
|
192
234
|
`success()` and `error()` finalize the call exactly once and are idempotent — extra calls are ignored.
|
|
193
235
|
|
|
194
|
-
**Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
|
|
236
|
+
**Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call except the `frugal.gen_ai.internal_errors` setup_failed / emission points; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
|
|
195
237
|
|
|
196
|
-
**Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `response_model`.
|
|
238
|
+
**Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `agent_turns`, `response_model`.
|
|
197
239
|
|
|
198
240
|
The same env-var configuration, safety guarantees, and exported metrics apply. See [`docs/quickstart.md`](docs/quickstart.md#custom-call-sites) for the full custom call-site walkthrough.
|
|
199
241
|
|
|
@@ -290,14 +332,27 @@ It is **opt-in** because installing signal handlers is intrusive. If you run you
|
|
|
290
332
|
|
|
291
333
|
## Verifying the setup
|
|
292
334
|
|
|
293
|
-
Set
|
|
335
|
+
Set `FRUGAL_METRICS_EXPORTER=console` to print the metrics to stdout instead
|
|
336
|
+
of exporting them. Console mode needs only `FRUGAL_PROJECT_ID`; the API key,
|
|
337
|
+
endpoint and header variables are ignored. An empty
|
|
338
|
+
endpoint does not select the console exporter: it falls through to the default
|
|
339
|
+
Frugal endpoint.
|
|
340
|
+
|
|
341
|
+
```bash
|
|
342
|
+
export FRUGAL_PROJECT_ID=my-project
|
|
343
|
+
export FRUGAL_METRICS_EXPORTER=console
|
|
344
|
+
export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
|
|
345
|
+
python my_script.py
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
Or point the library at a local OTLP/HTTP collector to see what it exports:
|
|
294
349
|
|
|
295
350
|
```bash
|
|
296
|
-
export FRUGAL_API_KEY=
|
|
351
|
+
export FRUGAL_API_KEY=test-key
|
|
297
352
|
export FRUGAL_PROJECT_ID=my-project
|
|
298
|
-
export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=
|
|
353
|
+
export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/metrics
|
|
299
354
|
export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
|
|
300
355
|
python my_script.py
|
|
301
356
|
```
|
|
302
357
|
|
|
303
|
-
|
|
358
|
+
The collector should receive metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
|
|
@@ -33,7 +33,7 @@ All configuration is via environment variables. No `init()` call, no configurati
|
|
|
33
33
|
|
|
34
34
|
### Required
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
Console mode (`FRUGAL_METRICS_EXPORTER=console`) needs only `FRUGAL_PROJECT_ID`; every other mode needs both. If a required variable is unset, the wrapper silently returns the client unmodified:
|
|
37
37
|
|
|
38
38
|
| Variable | Purpose |
|
|
39
39
|
| ------------------- | ---------------------------------------------------------- |
|
|
@@ -42,21 +42,27 @@ Set both or the wrapper silently returns the client unmodified:
|
|
|
42
42
|
|
|
43
43
|
### Optional
|
|
44
44
|
|
|
45
|
+
The export endpoint and authentication come only from `FRUGAL_`-prefixed variables, so the SDK never crosses wires with the host's own OTel pipeline: the generic `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are never read. (The OTel exporter library still reads its own generic TLS, compression, timeout and temporality variables.)
|
|
46
|
+
|
|
45
47
|
| Variable | Purpose |
|
|
46
48
|
| ---------------------------------------- | --------------------------------------------------------------------------------------- |
|
|
47
49
|
| `FRUGAL_CUSTOMER_ID` | Customer ID issued by Frugal. When unset, the tenant label (auto-injected server-side by VMAuth from the bearer token) identifies the customer and `frugal.customer_id` is omitted. |
|
|
48
50
|
| `FRUGAL_REPO_ROOT` | Absolute path to the source root; the strip prefix for `caller_file`. Defaults to the process working directory. |
|
|
49
|
-
| `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics
|
|
50
|
-
| `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`).
|
|
51
|
-
| `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. |
|
|
51
|
+
| `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics` (also when set but empty). |
|
|
52
|
+
| `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Precedence: this variable wins; otherwise the header built from `FRUGAL_API_KEY` (`FRUGAL_METRICS_API_KEY` first), which is sent verbatim. Values set here are percent-decoded (`Bearer%20tok` → `Bearer tok`). A value with no `key=value` pair is treated as missing auth (the API key is not used instead); a blank (whitespace-only) value is treated as unset. |
|
|
53
|
+
| `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. Deduped, and only the first **10** entries are honored (hard cap, not configurable); the rest are dropped at startup with one debug warning. |
|
|
52
54
|
| `FRUGAL_CUSTOM_PROVIDERS` | Comma-separated allowlist of provider names beyond the built-in registry (e.g. `together_ai,fireworks`) that should pass through verbatim on `gen_ai.system` / `gen_ai.provider.name` instead of being coerced to `custom`. Capped at 10 (extras are coerced) to keep label cardinality bounded. |
|
|
53
|
-
| `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label
|
|
55
|
+
| `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`. The vocabulary is the first **20** comma-separated entries by raw position (hard cap, not configurable); empty entries are ignored and duplicates collapsed. `other` is reserved (the coercion sentinel) and is ignored with one warning. A value outside the allowlist is coerced to `other` with a bounded once-per-value warning; when the variable is unset, every value is `other`. One warning fires only when non-empty entries past the 20th are actually dropped (a trailing comma does not warn). |
|
|
56
|
+
| `FRUGAL_METRICS_EXPORTER` | `otlp` (default) or `console` (print metrics to stdout; console mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables are ignored). Case-insensitive; any other value warns once and uses `otlp`. |
|
|
57
|
+
| `FRUGAL_METRICS_PROJECT_ID` / `FRUGAL_METRICS_API_KEY` | Namespaced overrides read before `FRUGAL_PROJECT_ID` / `FRUGAL_API_KEY`, for apps that already use those names. An empty value falls through to the bare name. |
|
|
54
58
|
| `FRUGAL_PATH_BLOCK_LIST` | Comma-separated repo-relative paths to skip for caller attribution (see below). |
|
|
55
|
-
| `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds
|
|
56
|
-
| `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute. Default `500`. Set to `0` to disable. |
|
|
57
|
-
| `
|
|
59
|
+
| `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds, `1`-`2147483647`. Default `60000`; a value outside the range warns once and uses the default. |
|
|
60
|
+
| `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute; past it `frugal.caller_file` / `frugal.caller_function` become `<overflow>`. Default `500`. Set to `0` to disable. |
|
|
61
|
+
| `FRUGAL_MAX_MODELS` | Cardinality cap — max distinct model strings to attribute; past it `gen_ai.request.model` / `gen_ai.response.model` become `<overflow>`. One registry shared by both labels, so a request alias and its dated response snapshot each take a slot. Default `100`. Set to `0` to disable. |
|
|
62
|
+
| `FRUGAL_MAX_SERIES_PER_INSTRUMENT` | OpenTelemetry's per-instrument limit on distinct attribute **combinations**, raised from the SDK default. Default `50000`. Set to `0` for effectively unlimited (a 1,000,000 ceiling). Applied **only when the installed `opentelemetry-sdk` accepts a per-instrument cardinality limit** — current releases (through 1.42) do not implement one, so nothing is capped and setting this variable emits one `warn` telling you it has no effect. |
|
|
63
|
+
| `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls that run the lightweight prompt analyzer: a decimal in [0, 1], plain or scientific notation (e.g. `0.25`, `1e-1`); anything else warns once and uses `0.1`. Default `0.1`. Set to `0` to disable, `1` for every call. |
|
|
58
64
|
| `FRUGAL_DUPLICATE_REQUEST_DETECTION` | Duplicate-request detection (the `frugal.gen_ai.duplicate_requests` metric) is **on by default** and hashes each chat request in-process (the hash never leaves the SDK). Set to `0`/`false`/`no`/`off` to disable — no hashing, no filter memory, no metric. |
|
|
59
|
-
| `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `
|
|
65
|
+
| `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `86400` (1 day). |
|
|
60
66
|
| `FRUGAL_DUPLICATE_CAPACITY` | Expected distinct request signatures per window (sizes the in-memory filter). Default `10000`, max `1000000`. Raise it if the SDK warns the filter is saturated. |
|
|
61
67
|
| `FRUGAL_DUPLICATE_FP_RATE` | Target false-positive rate of the duplicate filter. Default `0.01`, clamped to `[1e-6, 0.5]`. |
|
|
62
68
|
| `FRUGAL_DUPLICATE_GENERATIONS` | Rotation granularity of the duplicate filter's time window. Default `2`, max `16`. |
|
|
@@ -74,8 +80,9 @@ Every wrapped call emits OTel metrics tagged with the caller's source file and f
|
|
|
74
80
|
| `frugal.gen_ai.cache_read_tokens` | histogram | Prompt cache hits | — |
|
|
75
81
|
| `frugal.gen_ai.cache_write_tokens` | histogram | Prompt cache writes | — |
|
|
76
82
|
| `frugal.gen_ai.reasoning_tokens` | histogram | OpenAI reasoning tokens (o-series), Anthropic thinking tokens| — |
|
|
83
|
+
| `frugal.gen_ai.agent.turns` | histogram | Model-response turns per successful `invoke_agent` call (manual only) | — |
|
|
77
84
|
| `frugal.gen_ai.errors` | counter | Errors by exception type | `frugal.error.type` |
|
|
78
|
-
| `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality
|
|
85
|
+
| `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality caps, bugs) | `frugal.internal_error.stage`, `frugal.internal_error.type`, `frugal.instrument` |
|
|
79
86
|
| `frugal.gen_ai.output_cap_utilization` | histogram | `output_tokens / max_tokens` ratio when `max_tokens` is set | `frugal.structured_output_mode` |
|
|
80
87
|
| `frugal.gen_ai.history_depth` | histogram | `messages.length` — every chat call (100%, not sampled) | — |
|
|
81
88
|
| `frugal.gen_ai.duplicate_requests` | counter | Repeat-request detection — every chat call (100%, not sampled; disable via `FRUGAL_DUPLICATE_REQUEST_DETECTION`) | `frugal.dup_state` |
|
|
@@ -92,15 +99,17 @@ The analyzer-sampled metrics fire on a fraction of calls
|
|
|
92
99
|
Every metric carries this base attribute set (except
|
|
93
100
|
`frugal.gen_ai.internal_errors`, which is emitted from failure paths where
|
|
94
101
|
these attributes may not exist yet and carries only `frugal.project_id`, the
|
|
95
|
-
stage/type,
|
|
102
|
+
stage/type, `frugal.instrument` on the `otel_cardinality_overflow` stage, and
|
|
103
|
+
`frugal.customer_id` / `frugal.custom_label` when set; `frugal.custom_label`
|
|
104
|
+
never on the setup_failed / emission stages):
|
|
96
105
|
|
|
97
106
|
- `frugal.project_id` — from your env vars (`frugal.customer_id` too when set); `frugal.component` is added on the per-component series when `FRUGAL_COMPONENTS` is set
|
|
98
107
|
- `frugal.caller_file`, `frugal.caller_function` — auto-detected from the call stack
|
|
99
108
|
- `gen_ai.system` + `gen_ai.provider.name` — the provider, dual-emitted under both the legacy and current OTel attribute (`openai`, `anthropic`, `aws.bedrock`, `vertex_ai`/`gcp.vertex_ai`, …, or `custom`). Set via the `system` kwarg; an unrecognized value is coerced to `custom` with a one-time warning. See [`docs/metrics-reference.md`](docs/metrics-reference.md#attributes).
|
|
100
|
-
- `gen_ai.request.model`, `gen_ai.response.model` — the model used
|
|
109
|
+
- `gen_ai.request.model`, `gen_ai.response.model` — the model used; both draw on one shared registry and become `<overflow>` past `FRUGAL_MAX_MODELS` distinct model strings
|
|
101
110
|
- `gen_ai.operation.name` — `chat`, `text_completion`, `embeddings`, `generate_content`, `execute_tool`, `create_agent`, `invoke_agent`, or `other` (the sentinel an unrecognized operation coerces to)
|
|
102
111
|
- `frugal.batched` — `true` for batch-API paths (OpenAI Batch API submissions; Anthropic Message Batches submissions and results), `false` otherwise
|
|
103
|
-
- `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other`
|
|
112
|
+
- `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other` (with a once-per-value warning). `custom_label=None` means no label; a non-string value is coerced to `other`
|
|
104
113
|
|
|
105
114
|
The "Additional attributes" column above lists what each metric carries on top of the base set. See [`docs/metrics-reference.md`](docs/metrics-reference.md) for the values each attribute takes, sentinel values for caller fields, and the full internal-error-stage list.
|
|
106
115
|
|
|
@@ -140,6 +149,39 @@ with track_ai_call(system="anthropic", request_model="claude-sonnet-4-20250514")
|
|
|
140
149
|
)
|
|
141
150
|
```
|
|
142
151
|
|
|
152
|
+
For an enclosing agent invocation, set `operation="invoke_agent"` and report
|
|
153
|
+
the number of model-response steps performed inside it:
|
|
154
|
+
|
|
155
|
+
```python
|
|
156
|
+
with track_ai_call(
|
|
157
|
+
system="anthropic",
|
|
158
|
+
operation="invoke_agent",
|
|
159
|
+
request_model="claude-sonnet-4-20250514",
|
|
160
|
+
) as span:
|
|
161
|
+
result = my_agent_framework_call(prompt)
|
|
162
|
+
span.set_usage(
|
|
163
|
+
input_tokens=result.usage.input_tokens,
|
|
164
|
+
output_tokens=result.usage.output_tokens,
|
|
165
|
+
agent_turns=result.turn_count,
|
|
166
|
+
)
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
- Provider wrappers still measure individual model calls and never set
|
|
170
|
+
`agent_turns`.
|
|
171
|
+
- Agent frameworks such as the Claude Agent SDK, OpenAI Agents SDK, and
|
|
172
|
+
LangGraph do not have dedicated wrappers yet; track them manually as shown.
|
|
173
|
+
- `frugal.gen_ai.history_depth` is the request message count
|
|
174
|
+
(`messages.length`), not a turn count.
|
|
175
|
+
- Missing, non-integer, non-positive, or otherwise invalid `agent_turns`
|
|
176
|
+
values are silently dropped. They never raise or affect customer execution.
|
|
177
|
+
- Turn data is emitted only on success, like token usage.
|
|
178
|
+
- **Avoid double-counting tokens.** Only report aggregate `input_tokens` /
|
|
179
|
+
`output_tokens` on the `invoke_agent` call when the agent's inner
|
|
180
|
+
model-client calls are **not** already wrapped by a supported provider
|
|
181
|
+
wrapper. If they are wrapped, those wrappers already emit token usage per
|
|
182
|
+
inner call — report `agent_turns` only on the `invoke_agent` call in that
|
|
183
|
+
case, or the same tokens are double-counted.
|
|
184
|
+
|
|
143
185
|
The `with` block times the call and emits metrics on exit — or an error metric if the block raises (the exception always propagates). Works the same around an `await` in async code. Call `span.set_usage(...)` once the response is back; if you never call it, you still get a call-count and latency point.
|
|
144
186
|
|
|
145
187
|
For calls that span function boundaries — where a single `with` block won't fit — use the manual handle:
|
|
@@ -162,9 +204,9 @@ except BaseException as exc:
|
|
|
162
204
|
|
|
163
205
|
`success()` and `error()` finalize the call exactly once and are idempotent — extra calls are ignored.
|
|
164
206
|
|
|
165
|
-
**Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
|
|
207
|
+
**Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call except the `frugal.gen_ai.internal_errors` setup_failed / emission points; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
|
|
166
208
|
|
|
167
|
-
**Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `response_model`.
|
|
209
|
+
**Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `agent_turns`, `response_model`.
|
|
168
210
|
|
|
169
211
|
The same env-var configuration, safety guarantees, and exported metrics apply. See [`docs/quickstart.md`](docs/quickstart.md#custom-call-sites) for the full custom call-site walkthrough.
|
|
170
212
|
|
|
@@ -261,14 +303,27 @@ It is **opt-in** because installing signal handlers is intrusive. If you run you
|
|
|
261
303
|
|
|
262
304
|
## Verifying the setup
|
|
263
305
|
|
|
264
|
-
Set
|
|
306
|
+
Set `FRUGAL_METRICS_EXPORTER=console` to print the metrics to stdout instead
|
|
307
|
+
of exporting them. Console mode needs only `FRUGAL_PROJECT_ID`; the API key,
|
|
308
|
+
endpoint and header variables are ignored. An empty
|
|
309
|
+
endpoint does not select the console exporter: it falls through to the default
|
|
310
|
+
Frugal endpoint.
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
export FRUGAL_PROJECT_ID=my-project
|
|
314
|
+
export FRUGAL_METRICS_EXPORTER=console
|
|
315
|
+
export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
|
|
316
|
+
python my_script.py
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
Or point the library at a local OTLP/HTTP collector to see what it exports:
|
|
265
320
|
|
|
266
321
|
```bash
|
|
267
|
-
export FRUGAL_API_KEY=
|
|
322
|
+
export FRUGAL_API_KEY=test-key
|
|
268
323
|
export FRUGAL_PROJECT_ID=my-project
|
|
269
|
-
export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=
|
|
324
|
+
export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/metrics
|
|
270
325
|
export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
|
|
271
326
|
python my_script.py
|
|
272
327
|
```
|
|
273
328
|
|
|
274
|
-
|
|
329
|
+
The collector should receive metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
|
|
@@ -20,6 +20,7 @@ metric may carry the extra attributes shown in the rightmost column.
|
|
|
20
20
|
| `frugal.gen_ai.cache_read_tokens` | histogram | `{token}` | when provider reports cache reads | — |
|
|
21
21
|
| `frugal.gen_ai.cache_write_tokens` | histogram | `{token}` | when provider reports cache writes | — |
|
|
22
22
|
| `frugal.gen_ai.reasoning_tokens` | histogram | `{token}` | OpenAI o-series reasoning models | — |
|
|
23
|
+
| `frugal.gen_ai.agent.turns` | histogram | `{turn}` | successful `invoke_agent` calls with a valid `agent_turns` count (manual tracking only) | — |
|
|
23
24
|
| `frugal.gen_ai.errors` | counter | `{error}` | when the wrapped call raises | `frugal.error.type` |
|
|
24
25
|
| `frugal.gen_ai.internal_errors` | counter | `{error}` | frugal-metrics internal failures (cardinality cap, instrumentation bugs) | `frugal.internal_error.stage` |
|
|
25
26
|
| `frugal.gen_ai.output_cap_utilization`| histogram | `1` | when request had `max_tokens` (ratio 0..1) | `frugal.structured_output_mode` |
|
|
@@ -34,6 +35,37 @@ metric may carry the extra attributes shown in the rightmost column.
|
|
|
34
35
|
`gen_ai.client.token.usage` is split across input and output via the
|
|
35
36
|
`gen_ai.token.type` attribute (`input` | `output`).
|
|
36
37
|
|
|
38
|
+
### Agent turns
|
|
39
|
+
|
|
40
|
+
Manual `track_ai_call` and `start_ai_call` instrumentation accepts the
|
|
41
|
+
`agent_turns` usage kwarg through `AICallSpan.set_usage` and
|
|
42
|
+
`AICallSpan.success`. A sample is emitted only when the call succeeds, the
|
|
43
|
+
normalized `gen_ai.operation.name` is exactly `invoke_agent`, and the value is
|
|
44
|
+
a positive integer. Booleans, missing values, non-integers, zero, and negative
|
|
45
|
+
values are silently ignored; validation never affects customer execution.
|
|
46
|
+
Provider wrappers never set this field, so agent frameworks remain
|
|
47
|
+
manual-tracking call sites in this release. `frugal.gen_ai.history_depth`
|
|
48
|
+
continues to mean request message count, not agent turns. The histogram uses
|
|
49
|
+
explicit bucket boundaries
|
|
50
|
+
`(1, 2, 3, 4, 5, 6, 8, 10, 12, 15, 20, 25, 30, 40, 50, 75, 100)` (mirroring
|
|
51
|
+
`gen_ai.client.operation.duration`'s view) so `histogram_quantile()` resolves
|
|
52
|
+
meaningful spread across realistic agent-run turn counts instead of
|
|
53
|
+
collapsing into OTel's duration-tuned default buckets.
|
|
54
|
+
|
|
55
|
+
Only report aggregate token usage on the `invoke_agent` call itself when the
|
|
56
|
+
agent's inner model-client calls are **not** already wrapped by a supported
|
|
57
|
+
provider wrapper; if they are wrapped, report `agent_turns` only on the
|
|
58
|
+
`invoke_agent` call, since the inner wrappers already emit token usage and
|
|
59
|
+
combining both would double-count tokens.
|
|
60
|
+
|
|
61
|
+
To derive average turns per reporting agent invocation in MetricsQL, use:
|
|
62
|
+
|
|
63
|
+
```promql
|
|
64
|
+
increase(frugal.gen_ai.agent.turns_sum[...])
|
|
65
|
+
/
|
|
66
|
+
increase(frugal.gen_ai.agent.turns_count[...])
|
|
67
|
+
```
|
|
68
|
+
|
|
37
69
|
The analyzer-sampled metrics are gated by `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE`
|
|
38
70
|
(default `0.1`). Sampling is deterministic per `(caller_file, caller_function,
|
|
39
71
|
minute)` — within a minute a given call site is either fully sampled or fully
|
|
@@ -88,8 +120,8 @@ Controls:
|
|
|
88
120
|
`false`/`0`/`no`/`off`, no filter is allocated, no signature is hashed, and the
|
|
89
121
|
metric is not emitted. Any other value keeps detection enabled and emits a
|
|
90
122
|
one-time `warn` diagnostic naming the unrecognized value.
|
|
91
|
-
- `FRUGAL_DUPLICATE_WINDOW_SECONDS` (default `3600`, clamped to `
|
|
92
|
-
|
|
123
|
+
- `FRUGAL_DUPLICATE_WINDOW_SECONDS` (default `3600`, clamped to `86400` — 1
|
|
124
|
+
day) — retention window. Because the filter rotates in `generations` slots,
|
|
93
125
|
a signature is remembered for **at least**
|
|
94
126
|
`window × (generations−1)/generations` and **at most** `window` seconds
|
|
95
127
|
(30–60 min at defaults) — treat the counterfactual cache TTL as that range,
|
|
@@ -101,6 +133,15 @@ Controls:
|
|
|
101
133
|
- `FRUGAL_DUPLICATE_GENERATIONS` (default `2`, clamped to `16`) — rotating-Bloom
|
|
102
134
|
generation count.
|
|
103
135
|
|
|
136
|
+
Exporter selection: `FRUGAL_METRICS_EXPORTER` (`otlp` default, or `console` to
|
|
137
|
+
print every metric to stdout via the OpenTelemetry console exporter; console
|
|
138
|
+
mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables
|
|
139
|
+
are ignored, and any other value warns once and uses `otlp`). An empty endpoint variable falls through to the next source and
|
|
140
|
+
never selects console.
|
|
141
|
+
|
|
142
|
+
`FRUGAL_METRIC_EXPORT_INTERVAL_MS` accepts `1` to `2147483647` (default `60000`);
|
|
143
|
+
a value outside that range warns once and uses the default.
|
|
144
|
+
|
|
104
145
|
Out-of-range or non-finite values for any of these fall back to the default (or
|
|
105
146
|
clamp to the bound) with a one-time named `warn` through the debug callback.
|
|
106
147
|
|
|
@@ -159,18 +200,18 @@ Reading `duplicate_requests` correctly:
|
|
|
159
200
|
|
|
160
201
|
| Attribute | Value |
|
|
161
202
|
| ------------------------- | ---------------------------------------------------------- |
|
|
162
|
-
| `frugal.customer_id` | From `FRUGAL_CUSTOMER_ID`
|
|
163
|
-
| `frugal.project_id` | From `FRUGAL_PROJECT_ID`
|
|
203
|
+
| `frugal.customer_id` | From `FRUGAL_CUSTOMER_ID` — present only when set |
|
|
204
|
+
| `frugal.project_id` | From `FRUGAL_METRICS_PROJECT_ID`, else `FRUGAL_PROJECT_ID` |
|
|
164
205
|
| `frugal.component` | From each entry in `FRUGAL_COMPONENTS` — present only on the per-component series, absent on the base project series |
|
|
165
206
|
| `frugal.caller_file` | Source file repo-relative to `FRUGAL_REPO_ROOT` |
|
|
166
207
|
| `frugal.caller_function` | Function/method name at the call site |
|
|
167
|
-
| `frugal.error.type` | Exception class name (error-path metrics only)
|
|
168
|
-
| `frugal.batched` | `true` when the metric came from a provider batch-API path, `false` otherwise. Present on every metric. Batch submissions (OpenAI `batches.create`, Anthropic `messages.batches.create`) emit `frugal.gen_ai.calls` with `batched="true"` — one point per request for Anthropic (the requests array is in hand, grouped by model), one per submission for OpenAI (per-request counts are not reported at submission time, so OpenAI batched call volume is a floor, never an overcount). Iterating Anthropic `messages.batches.results()` additionally emits per-entry `gen_ai.client.token.usage` (+ cache metrics) with `batched="true"`, attributed to the call site that submitted the batch when the submission happened in the same process (bounded in-process memory; falls back to the results call site). OpenAI batch token usage lives in the output file read through the generic files API and is not intercepted. The manual track API accepts a `batched` flag for hand-instrumented batch paths. An app on an SDK predating batch support reads `false` on genuinely batched traffic — the label reflects what the deployed SDK can see. |
|
|
208
|
+
| `frugal.error.type` | Exception class name (error-path metrics only); stripped from `frugal.gen_ai.history_depth` and `frugal.gen_ai.duplicate_requests` so success and error points stay additive |
|
|
209
|
+
| `frugal.batched` | `true` when the metric came from a provider batch-API path, `false` otherwise. Present on every metric of the call except the `frugal.gen_ai.internal_errors` points. Batch submissions (OpenAI `batches.create`, Anthropic `messages.batches.create`) emit `frugal.gen_ai.calls` with `batched="true"` — one point per request for Anthropic (the requests array is in hand, grouped by model), one per submission for OpenAI (per-request counts are not reported at submission time, so OpenAI batched call volume is a floor, never an overcount). Iterating Anthropic `messages.batches.results()` additionally emits per-entry `gen_ai.client.token.usage` (+ cache metrics) with `batched="true"`, attributed to the call site that submitted the batch when the submission happened in the same process (bounded in-process memory; falls back to the results call site). OpenAI batch token usage lives in the output file read through the generic files API and is not intercepted. The manual track API accepts a `batched` flag for hand-instrumented batch paths. An app on an SDK predating batch support reads `false` on genuinely batched traffic — the label reflects what the deployed SDK can see. |
|
|
169
210
|
| `frugal.structured_output_mode` | `none` \| `json_object` \| `json_schema` \| `tools`. Derived from request params by precedence: non-empty `tools` → `tools`; else `response_format.type` `json_object` / `json_schema` → the matching value; else `none`. On `frugal.gen_ai.output_cap_utilization` (when `max_tokens` set), and on `frugal.gen_ai.calls` for every call with a request payload (omitted when there is no payload). |
|
|
170
211
|
| `frugal.block` | `system` \| `user` \| `tools` \| `schema`. Only on `frugal.gen_ai.cache_prefix_stable`. |
|
|
171
212
|
| `frugal.stable` | `true` when the block hash matches the previous call at this site. Only on `frugal.gen_ai.cache_prefix_stable`. |
|
|
172
213
|
| `frugal.history_bucket` | Bounded bucket of `messages.length`: `0` \| `1-2` \| `3-5` \| `6-10` \| `11-20` \| `21-50` \| `51+`. On `frugal.gen_ai.calls` and `gen_ai.client.token.usage`, and only for chat calls (payload has a `messages` array); omitted otherwise. 100% (not analyzer-sampled). |
|
|
173
|
-
| `frugal.custom_label` | Custom label (e.g. a business unit) passed as `custom_label=` to `track_ai_call` / `start_ai_call` — never set by the provider wrappers. When passed, it rides
|
|
214
|
+
| `frugal.custom_label` | Custom label (e.g. a business unit) passed as `custom_label=` to `track_ai_call` / `start_ai_call` — never set by the provider wrappers. When passed, it rides every metric of that call (success and error paths alike), except the `frugal.gen_ai.internal_errors` `setup_failed`/`emission` points, which carry only project, customer and error type. The value is trimmed of surrounding whitespace (matching the Node SDK), then must match the closed vocabulary from the first 20 comma-separated entries of `FRUGAL_CUSTOM_LABEL` by raw position (hard cap, not configurable; empties ignored, duplicates collapsed) or it is coerced to `other` with a bounded once-per-value warning. `other` is reserved as the coercion sentinel: an `other` entry in the env var is ignored with a warning. With the variable unset, every value is `other`. A non-string value is coerced to `other` with a type warning; `None` means no label. A truncation warning fires only when non-empty entries past the 20th are dropped. Omitted when not passed. |
|
|
174
215
|
| `frugal.dup_state` | `first_seen` \| `repeat_seen` — whether this request's canonical signature was already seen within the duplicate-detection window. Only on `frugal.gen_ai.duplicate_requests`. |
|
|
175
216
|
|
|
176
217
|
### Sentinel values for `frugal.caller_file` / `frugal.caller_function`
|
|
@@ -182,6 +223,11 @@ Reading `duplicate_requests` correctly:
|
|
|
182
223
|
| `<lambda>` | Call originated from a `lambda` expression. This is Python's native `co_name` for lambdas. Only appears in `frugal.caller_function`. |
|
|
183
224
|
| `<overflow>` | Cardinality cap (`FRUGAL_MAX_CALL_SITES`) exceeded. See below. |
|
|
184
225
|
|
|
226
|
+
> `<overflow>` is the shared overflow sentinel, not a caller-only value: it also
|
|
227
|
+
> appears on `gen_ai.request.model` / `gen_ai.response.model` once
|
|
228
|
+
> `FRUGAL_MAX_MODELS` distinct model strings have been observed. See
|
|
229
|
+
> [Model cap behavior](#model-cap-behavior).
|
|
230
|
+
|
|
185
231
|
### Cardinality cap behavior
|
|
186
232
|
|
|
187
233
|
Once `FRUGAL_MAX_CALL_SITES` distinct `(file, function)` pairs have been
|
|
@@ -196,6 +242,23 @@ The first time the cap is hit per process, the library emits a single
|
|
|
196
242
|
`frugal.gen_ai.internal_errors{stage=cardinality_cap_hit}` point so you can
|
|
197
243
|
see the transition in your dashboard.
|
|
198
244
|
|
|
245
|
+
### Model cap behavior
|
|
246
|
+
|
|
247
|
+
Model ids are free-form provider strings, so `FRUGAL_MAX_MODELS` (default `100`,
|
|
248
|
+
`0` disables) bounds them the same way. The first N distinct model strings a
|
|
249
|
+
process observes keep their real names; every later one is reported as
|
|
250
|
+
`<overflow>` on whichever of `gen_ai.request.model` / `gen_ai.response.model`
|
|
251
|
+
carries it. The registry is process-wide and **shared by both labels** — a model
|
|
252
|
+
first seen in a response is free to use as a request model, and vice versa — so
|
|
253
|
+
a model already in the registry is never re-charged. The default is generous
|
|
254
|
+
because one logical model normally occupies **two** slots: providers answer a
|
|
255
|
+
request alias (`claude-sonnet-4-5`) with a dated response snapshot
|
|
256
|
+
(`claude-sonnet-4-5-20250929`). The SDK's own sentinels — `<unknown>` and
|
|
257
|
+
`<overflow>` — pass through without consuming a slot. The first time the cap is
|
|
258
|
+
hit per process the library emits a single
|
|
259
|
+
`frugal.gen_ai.internal_errors{stage=model_cap_hit}` point plus one `warn`
|
|
260
|
+
through the debug callback.
|
|
261
|
+
|
|
199
262
|
### Internal-error stages
|
|
200
263
|
|
|
201
264
|
`frugal.gen_ai.internal_errors` carries a `frugal.internal_error.stage`
|
|
@@ -203,18 +266,23 @@ attribute describing where our own code failed. Non-exhaustive list:
|
|
|
203
266
|
|
|
204
267
|
| Stage | When it fires |
|
|
205
268
|
| ----------------------- | ---------------------------------------------------------------------- |
|
|
206
|
-
| `setup_failed` | Our wrapper setup (OTel init, caller resolution) threw; original call continued normally |
|
|
269
|
+
| `setup_failed` | Our wrapper setup (OTel init, caller resolution) threw; original call continued normally. Emitted by the provider wrappers and the manual track API (`track_ai_call` / `start_ai_call`); carries `frugal.project_id`, `frugal.customer_id` (when set) and `frugal.internal_error.type` |
|
|
207
270
|
| `stream_proxy_failed` | Constructing the stream wrapper threw; original response returned unwrapped |
|
|
208
|
-
| `emission` | Emitting a success/error metric threw
|
|
271
|
+
| `emission` | Emitting a success/error metric threw. Emitted by the provider wrappers and the manual track API; carries the same project/customer identity and `frugal.internal_error.type` as `setup_failed` |
|
|
272
|
+
| `call_shape` | Deriving history_depth / history_bucket / structured_output_mode from the request payload threw; carries `frugal.internal_error.type`; core metrics still emitted |
|
|
209
273
|
| `duplicate_detection` | The duplicate-request path (signature, filter, or emission) threw — including the one-time filter allocation failure; carries `frugal.internal_error.type` with the exception class |
|
|
210
274
|
| `cardinality_cap_hit` | `FRUGAL_MAX_CALL_SITES` was just reached (once per process) |
|
|
275
|
+
| `model_cap_hit` | `FRUGAL_MAX_MODELS` was just reached (once per process); further distinct models are reported as `<overflow>` |
|
|
276
|
+
| `otel_cardinality_overflow` | OpenTelemetry merged an instrument's attribute sets into its `otel.metric.overflow` series (once per instrument name); carries `frugal.instrument` with the affected instrument |
|
|
211
277
|
|
|
212
278
|
Unlike every other metric, `frugal.gen_ai.internal_errors` does **not** carry
|
|
213
279
|
the full base attribute set — it is emitted from failure paths where the
|
|
214
280
|
caller/provider/model attributes may not exist yet. Its points carry only
|
|
215
281
|
`frugal.project_id`, `frugal.internal_error.stage` (plus
|
|
216
|
-
`frugal.internal_error.type` where noted
|
|
217
|
-
`
|
|
282
|
+
`frugal.internal_error.type` where noted, and `frugal.instrument` on the
|
|
283
|
+
`otel_cardinality_overflow` stage), and — when set — `frugal.customer_id` and
|
|
284
|
+
`frugal.custom_label` (`setup_failed` and `emission` never carry the custom
|
|
285
|
+
label).
|
|
218
286
|
|
|
219
287
|
## Provider coverage
|
|
220
288
|
|