frugal-sdk-python 0.4.0__tar.gz → 0.6.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/PKG-INFO +75 -20
  2. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/README.md +74 -19
  3. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/docs/metrics-reference.md +79 -11
  4. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/docs/quickstart.md +155 -25
  5. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/dashboards/frugal-metrics.json +1 -1
  6. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/block_hash_lru.py +22 -6
  7. frugal_sdk_python-0.6.0/src/frugal_metrics/config.py +739 -0
  8. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/debug.py +6 -2
  9. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/instrument.py +238 -87
  10. frugal_sdk_python-0.6.0/src/frugal_metrics/model_cap.py +103 -0
  11. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/otel.py +280 -35
  12. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/providers.py +45 -21
  13. frugal_sdk_python-0.6.0/src/frugal_metrics/safe.py +58 -0
  14. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/semconv.py +18 -4
  15. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/streams.py +5 -4
  16. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/track.py +13 -6
  17. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/_batch.py +20 -12
  18. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/anthropic.py +4 -4
  19. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/bedrock.py +4 -4
  20. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/openai.py +4 -4
  21. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/conftest.py +33 -0
  22. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_block_hash_lru.py +31 -2
  23. frugal_sdk_python-0.6.0/tests/test_config.py +741 -0
  24. frugal_sdk_python-0.6.0/tests/test_model_cap.py +413 -0
  25. frugal_sdk_python-0.6.0/tests/test_never_impact_host.py +270 -0
  26. frugal_sdk_python-0.6.0/tests/test_node_parity.py +791 -0
  27. frugal_sdk_python-0.6.0/tests/test_otel_overflow_detection.py +439 -0
  28. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_providers.py +31 -0
  29. frugal_sdk_python-0.6.0/tests/test_sibling_parity.py +79 -0
  30. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_track_ai_call.py +130 -0
  31. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_anthropic_batches.py +52 -0
  32. frugal_sdk_python-0.4.0/src/frugal_metrics/config.py +0 -448
  33. frugal_sdk_python-0.4.0/src/frugal_metrics/safe.py +0 -45
  34. frugal_sdk_python-0.4.0/tests/test_config.py +0 -357
  35. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/.github/workflows/cd.yml +0 -0
  36. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/.gitignore +0 -0
  37. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/LICENSES/Apache-2.0.txt +0 -0
  38. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/REUSE.toml +0 -0
  39. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/README.md +0 -0
  40. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/collector/otel-collector-config.yaml +0 -0
  41. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/docker-compose.yml +0 -0
  42. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/provisioning/dashboards/frugal.yml +0 -0
  43. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/grafana/provisioning/datasources/prometheus.yml +0 -0
  44. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/Dockerfile +0 -0
  45. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/package.json +0 -0
  46. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/mockllm/src/server.js +0 -0
  47. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/index.html.example +0 -0
  48. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/models.html +0 -0
  49. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/overview.html +0 -0
  50. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/consoles/sites.html +0 -0
  51. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/prometheus/prometheus.yml +0 -0
  52. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/Dockerfile +0 -0
  53. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/__init__.py +0 -0
  54. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/clients.py +0 -0
  55. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/__init__.py +0 -0
  56. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/anon.py +0 -0
  57. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/classify.py +0 -0
  58. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/deferred.py +0 -0
  59. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/fail.py +0 -0
  60. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/pipeline.py +0 -0
  61. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/summarize.py +0 -0
  62. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/translate.py +0 -0
  63. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/via_blocked_lib.py +0 -0
  64. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/handlers/via_unblocked_lib.py +0 -0
  65. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/internal_lib/__init__.py +0 -0
  66. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/internal_lib/ai_helper.py +0 -0
  67. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/main.py +0 -0
  68. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/sibling_lib/__init__.py +0 -0
  69. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/app/sibling_lib/ai_helper.py +0 -0
  70. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/pyproject.toml +0 -0
  71. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/sample-app/requirements.txt +0 -0
  72. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/examples/test-env/scripts/verify.sh +0 -0
  73. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/pyproject.toml +0 -0
  74. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/__init__.py +0 -0
  75. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/caller.py +0 -0
  76. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/duplicate_filter.py +0 -0
  77. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/prompt_analyzer.py +0 -0
  78. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/shutdown.py +0 -0
  79. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/__init__.py +0 -0
  80. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/_patching.py +0 -0
  81. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/src/frugal_metrics/wrappers/bedrock_models.py +0 -0
  82. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/__init__.py +0 -0
  83. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_bedrock_models.py +0 -0
  84. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_caller.py +0 -0
  85. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_caller_edge_cases.py +0 -0
  86. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_debug.py +0 -0
  87. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_duplicate_filter.py +0 -0
  88. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_prompt_analyzer.py +0 -0
  89. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_shutdown.py +0 -0
  90. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_anthropic.py +0 -0
  91. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_bedrock.py +0 -0
  92. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai.py +0 -0
  93. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai_analyzer.py +0 -0
  94. {frugal_sdk_python-0.4.0 → frugal_sdk_python-0.6.0}/tests/test_wrap_openai_batches.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: frugal-sdk-python
3
- Version: 0.4.0
3
+ Version: 0.6.0
4
4
  Summary: Instrumentation for Cost-to-Code attribution.
5
5
  Project-URL: Homepage, https://frugal.co
6
6
  Author: Frugal AI Inc.
@@ -62,7 +62,7 @@ All configuration is via environment variables. No `init()` call, no configurati
62
62
 
63
63
  ### Required
64
64
 
65
- Set both or the wrapper silently returns the client unmodified:
65
+ Console mode (`FRUGAL_METRICS_EXPORTER=console`) needs only `FRUGAL_PROJECT_ID`; every other mode needs both. If a required variable is unset, the wrapper silently returns the client unmodified:
66
66
 
67
67
  | Variable | Purpose |
68
68
  | ------------------- | ---------------------------------------------------------- |
@@ -71,21 +71,27 @@ Set both or the wrapper silently returns the client unmodified:
71
71
 
72
72
  ### Optional
73
73
 
74
+ The export endpoint and authentication come only from `FRUGAL_`-prefixed variables, so the SDK never crosses wires with the host's own OTel pipeline: the generic `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are never read. (The OTel exporter library still reads its own generic TLS, compression, timeout and temporality variables.)
75
+
74
76
  | Variable | Purpose |
75
77
  | ---------------------------------------- | --------------------------------------------------------------------------------------- |
76
78
  | `FRUGAL_CUSTOMER_ID` | Customer ID issued by Frugal. When unset, the tenant label (auto-injected server-side by VMAuth from the bearer token) identifies the customer and `frugal.customer_id` is omitted. |
77
79
  | `FRUGAL_REPO_ROOT` | Absolute path to the source root; the strip prefix for `caller_file`. Defaults to the process working directory. |
78
- | `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics`. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`. |
79
- | `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Overrides the header built from `FRUGAL_API_KEY`. Falls back to `OTEL_EXPORTER_OTLP_HEADERS`. |
80
- | `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. |
80
+ | `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics` (also when set but empty). |
81
+ | `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Precedence: this variable wins; otherwise the header built from `FRUGAL_API_KEY` (`FRUGAL_METRICS_API_KEY` first), which is sent verbatim. Values set here are percent-decoded (`Bearer%20tok` → `Bearer tok`). A value with no `key=value` pair is treated as missing auth (the API key is not used instead); a blank (whitespace-only) value is treated as unset. |
82
+ | `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. Deduped, and only the first **10** entries are honored (hard cap, not configurable); the rest are dropped at startup with one debug warning. |
81
83
  | `FRUGAL_CUSTOM_PROVIDERS` | Comma-separated allowlist of provider names beyond the built-in registry (e.g. `together_ai,fireworks`) that should pass through verbatim on `gen_ai.system` / `gen_ai.provider.name` instead of being coerced to `custom`. Capped at 10 (extras are coerced) to keep label cardinality bounded. |
82
- | `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; a value outside the first 20 entries (hard cap, not configurable) is coerced to `other`. |
84
+ | `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`. The vocabulary is the first **20** comma-separated entries by raw position (hard cap, not configurable); empty entries are ignored and duplicates collapsed. `other` is reserved (the coercion sentinel) and is ignored with one warning. A value outside the allowlist is coerced to `other` with a bounded once-per-value warning; when the variable is unset, every value is `other`. One warning fires only when non-empty entries past the 20th are actually dropped (a trailing comma does not warn). |
85
+ | `FRUGAL_METRICS_EXPORTER` | `otlp` (default) or `console` (print metrics to stdout; console mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables are ignored). Case-insensitive; any other value warns once and uses `otlp`. |
86
+ | `FRUGAL_METRICS_PROJECT_ID` / `FRUGAL_METRICS_API_KEY` | Namespaced overrides read before `FRUGAL_PROJECT_ID` / `FRUGAL_API_KEY`, for apps that already use those names. An empty value falls through to the bare name. |
83
87
  | `FRUGAL_PATH_BLOCK_LIST` | Comma-separated repo-relative paths to skip for caller attribution (see below). |
84
- | `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds. Default `60000`. |
85
- | `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute. Default `500`. Set to `0` to disable. |
86
- | `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls (0..1) that run the lightweight prompt analyzer. Default `0.1`. Set to `0` to disable, `1` for every call. |
88
+ | `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds, `1`-`2147483647`. Default `60000`; a value outside the range warns once and uses the default. |
89
+ | `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute; past it `frugal.caller_file` / `frugal.caller_function` become `<overflow>`. Default `500`. Set to `0` to disable. |
90
+ | `FRUGAL_MAX_MODELS` | Cardinality cap — max distinct model strings to attribute; past it `gen_ai.request.model` / `gen_ai.response.model` become `<overflow>`. One registry shared by both labels, so a request alias and its dated response snapshot each take a slot. Default `100`. Set to `0` to disable. |
91
+ | `FRUGAL_MAX_SERIES_PER_INSTRUMENT` | OpenTelemetry's per-instrument limit on distinct attribute **combinations**, raised from the SDK default. Default `50000`. Set to `0` for effectively unlimited (a 1,000,000 ceiling). Applied **only when the installed `opentelemetry-sdk` accepts a per-instrument cardinality limit** — current releases (through 1.42) do not implement one, so nothing is capped and setting this variable emits one `warn` telling you it has no effect. |
92
+ | `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls that run the lightweight prompt analyzer: a decimal in [0, 1], plain or scientific notation (e.g. `0.25`, `1e-1`); anything else warns once and uses `0.1`. Default `0.1`. Set to `0` to disable, `1` for every call. |
87
93
  | `FRUGAL_DUPLICATE_REQUEST_DETECTION` | Duplicate-request detection (the `frugal.gen_ai.duplicate_requests` metric) is **on by default** and hashes each chat request in-process (the hash never leaves the SDK). Set to `0`/`false`/`no`/`off` to disable — no hashing, no filter memory, no metric. |
88
- | `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `604800` (7 days). |
94
+ | `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `86400` (1 day). |
89
95
  | `FRUGAL_DUPLICATE_CAPACITY` | Expected distinct request signatures per window (sizes the in-memory filter). Default `10000`, max `1000000`. Raise it if the SDK warns the filter is saturated. |
90
96
  | `FRUGAL_DUPLICATE_FP_RATE` | Target false-positive rate of the duplicate filter. Default `0.01`, clamped to `[1e-6, 0.5]`. |
91
97
  | `FRUGAL_DUPLICATE_GENERATIONS` | Rotation granularity of the duplicate filter's time window. Default `2`, max `16`. |
@@ -103,8 +109,9 @@ Every wrapped call emits OTel metrics tagged with the caller's source file and f
103
109
  | `frugal.gen_ai.cache_read_tokens` | histogram | Prompt cache hits | — |
104
110
  | `frugal.gen_ai.cache_write_tokens` | histogram | Prompt cache writes | — |
105
111
  | `frugal.gen_ai.reasoning_tokens` | histogram | OpenAI reasoning tokens (o-series), Anthropic thinking tokens| — |
112
+ | `frugal.gen_ai.agent.turns` | histogram | Model-response turns per successful `invoke_agent` call (manual only) | — |
106
113
  | `frugal.gen_ai.errors` | counter | Errors by exception type | `frugal.error.type` |
107
- | `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality cap, bugs) | `frugal.internal_error.stage` |
114
+ | `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality caps, bugs) | `frugal.internal_error.stage`, `frugal.internal_error.type`, `frugal.instrument` |
108
115
  | `frugal.gen_ai.output_cap_utilization` | histogram | `output_tokens / max_tokens` ratio when `max_tokens` is set | `frugal.structured_output_mode` |
109
116
  | `frugal.gen_ai.history_depth` | histogram | `messages.length` — every chat call (100%, not sampled) | — |
110
117
  | `frugal.gen_ai.duplicate_requests` | counter | Repeat-request detection — every chat call (100%, not sampled; disable via `FRUGAL_DUPLICATE_REQUEST_DETECTION`) | `frugal.dup_state` |
@@ -121,15 +128,17 @@ The analyzer-sampled metrics fire on a fraction of calls
121
128
  Every metric carries this base attribute set (except
122
129
  `frugal.gen_ai.internal_errors`, which is emitted from failure paths where
123
130
  these attributes may not exist yet and carries only `frugal.project_id`, the
124
- stage/type, and `frugal.customer_id` / `frugal.custom_label` when set):
131
+ stage/type, `frugal.instrument` on the `otel_cardinality_overflow` stage, and
132
+ `frugal.customer_id` / `frugal.custom_label` when set; `frugal.custom_label`
133
+ never on the setup_failed / emission stages):
125
134
 
126
135
  - `frugal.project_id` — from your env vars (`frugal.customer_id` too when set); `frugal.component` is added on the per-component series when `FRUGAL_COMPONENTS` is set
127
136
  - `frugal.caller_file`, `frugal.caller_function` — auto-detected from the call stack
128
137
  - `gen_ai.system` + `gen_ai.provider.name` — the provider, dual-emitted under both the legacy and current OTel attribute (`openai`, `anthropic`, `aws.bedrock`, `vertex_ai`/`gcp.vertex_ai`, …, or `custom`). Set via the `system` kwarg; an unrecognized value is coerced to `custom` with a one-time warning. See [`docs/metrics-reference.md`](docs/metrics-reference.md#attributes).
129
- - `gen_ai.request.model`, `gen_ai.response.model` — the model used
138
+ - `gen_ai.request.model`, `gen_ai.response.model` — the model used; both draw on one shared registry and become `<overflow>` past `FRUGAL_MAX_MODELS` distinct model strings
130
139
  - `gen_ai.operation.name` — `chat`, `text_completion`, `embeddings`, `generate_content`, `execute_tool`, `create_agent`, `invoke_agent`, or `other` (the sentinel an unrecognized operation coerces to)
131
140
  - `frugal.batched` — `true` for batch-API paths (OpenAI Batch API submissions; Anthropic Message Batches submissions and results), `false` otherwise
132
- - `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other`
141
+ - `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other` (with a once-per-value warning). `custom_label=None` means no label; a non-string value is coerced to `other`
133
142
 
134
143
  The "Additional attributes" column above lists what each metric carries on top of the base set. See [`docs/metrics-reference.md`](docs/metrics-reference.md) for the values each attribute takes, sentinel values for caller fields, and the full internal-error-stage list.
135
144
 
@@ -169,6 +178,39 @@ with track_ai_call(system="anthropic", request_model="claude-sonnet-4-20250514")
169
178
  )
170
179
  ```
171
180
 
181
+ For an enclosing agent invocation, set `operation="invoke_agent"` and report
182
+ the number of model-response steps performed inside it:
183
+
184
+ ```python
185
+ with track_ai_call(
186
+ system="anthropic",
187
+ operation="invoke_agent",
188
+ request_model="claude-sonnet-4-20250514",
189
+ ) as span:
190
+ result = my_agent_framework_call(prompt)
191
+ span.set_usage(
192
+ input_tokens=result.usage.input_tokens,
193
+ output_tokens=result.usage.output_tokens,
194
+ agent_turns=result.turn_count,
195
+ )
196
+ ```
197
+
198
+ - Provider wrappers still measure individual model calls and never set
199
+ `agent_turns`.
200
+ - Agent frameworks such as the Claude Agent SDK, OpenAI Agents SDK, and
201
+ LangGraph do not have dedicated wrappers yet; track them manually as shown.
202
+ - `frugal.gen_ai.history_depth` is the request message count
203
+ (`messages.length`), not a turn count.
204
+ - Missing, non-integer, non-positive, or otherwise invalid `agent_turns`
205
+ values are silently dropped. They never raise or affect customer execution.
206
+ - Turn data is emitted only on success, like token usage.
207
+ - **Avoid double-counting tokens.** Only report aggregate `input_tokens` /
208
+ `output_tokens` on the `invoke_agent` call when the agent's inner
209
+ model-client calls are **not** already wrapped by a supported provider
210
+ wrapper. If they are wrapped, those wrappers already emit token usage per
211
+ inner call — report `agent_turns` only on the `invoke_agent` call in that
212
+ case, or the same tokens are double-counted.
213
+
172
214
  The `with` block times the call and emits metrics on exit — or an error metric if the block raises (the exception always propagates). Works the same around an `await` in async code. Call `span.set_usage(...)` once the response is back; if you never call it, you still get a call-count and latency point.
173
215
 
174
216
  For calls that span function boundaries — where a single `with` block won't fit — use the manual handle:
@@ -191,9 +233,9 @@ except BaseException as exc:
191
233
 
192
234
  `success()` and `error()` finalize the call exactly once and are idempotent — extra calls are ignored.
193
235
 
194
- **Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
236
+ **Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call except the `frugal.gen_ai.internal_errors` setup_failed / emission points; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
195
237
 
196
- **Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `response_model`.
238
+ **Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `agent_turns`, `response_model`.
197
239
 
198
240
  The same env-var configuration, safety guarantees, and exported metrics apply. See [`docs/quickstart.md`](docs/quickstart.md#custom-call-sites) for the full custom call-site walkthrough.
199
241
 
@@ -290,14 +332,27 @@ It is **opt-in** because installing signal handlers is intrusive. If you run you
290
332
 
291
333
  ## Verifying the setup
292
334
 
293
- Set the OTLP endpoint to empty to print metrics to the console instead of exporting them:
335
+ Set `FRUGAL_METRICS_EXPORTER=console` to print the metrics to stdout instead
336
+ of exporting them. Console mode needs only `FRUGAL_PROJECT_ID`; the API key,
337
+ endpoint and header variables are ignored. An empty
338
+ endpoint does not select the console exporter: it falls through to the default
339
+ Frugal endpoint.
340
+
341
+ ```bash
342
+ export FRUGAL_PROJECT_ID=my-project
343
+ export FRUGAL_METRICS_EXPORTER=console
344
+ export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
345
+ python my_script.py
346
+ ```
347
+
348
+ Or point the library at a local OTLP/HTTP collector to see what it exports:
294
349
 
295
350
  ```bash
296
- export FRUGAL_API_KEY=local
351
+ export FRUGAL_API_KEY=test-key
297
352
  export FRUGAL_PROJECT_ID=my-project
298
- export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT= # empty → ConsoleMetricExporter
353
+ export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/metrics
299
354
  export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
300
355
  python my_script.py
301
356
  ```
302
357
 
303
- You should see JSON metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
358
+ The collector should receive metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
@@ -33,7 +33,7 @@ All configuration is via environment variables. No `init()` call, no configurati
33
33
 
34
34
  ### Required
35
35
 
36
- Set both or the wrapper silently returns the client unmodified:
36
+ Console mode (`FRUGAL_METRICS_EXPORTER=console`) needs only `FRUGAL_PROJECT_ID`; every other mode needs both. If a required variable is unset, the wrapper silently returns the client unmodified:
37
37
 
38
38
  | Variable | Purpose |
39
39
  | ------------------- | ---------------------------------------------------------- |
@@ -42,21 +42,27 @@ Set both or the wrapper silently returns the client unmodified:
42
42
 
43
43
  ### Optional
44
44
 
45
+ The export endpoint and authentication come only from `FRUGAL_`-prefixed variables, so the SDK never crosses wires with the host's own OTel pipeline: the generic `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are never read. (The OTel exporter library still reads its own generic TLS, compression, timeout and temporality variables.)
46
+
45
47
  | Variable | Purpose |
46
48
  | ---------------------------------------- | --------------------------------------------------------------------------------------- |
47
49
  | `FRUGAL_CUSTOMER_ID` | Customer ID issued by Frugal. When unset, the tenant label (auto-injected server-side by VMAuth from the bearer token) identifies the customer and `frugal.customer_id` is omitted. |
48
50
  | `FRUGAL_REPO_ROOT` | Absolute path to the source root; the strip prefix for `caller_file`. Defaults to the process working directory. |
49
- | `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics`. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`. |
50
- | `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Overrides the header built from `FRUGAL_API_KEY`. Falls back to `OTEL_EXPORTER_OTLP_HEADERS`. |
51
- | `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. |
51
+ | `FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector URL for Frugal metrics. Defaults to `https://metrics.frugal.co/opentelemetry/v1/metrics` (also when set but empty). |
52
+ | `FRUGAL_OTEL_EXPORTER_OTLP_HEADERS` | Export auth headers (e.g. `Authorization=Bearer ...`). Precedence: this variable wins; otherwise the header built from `FRUGAL_API_KEY` (`FRUGAL_METRICS_API_KEY` first), which is sent verbatim. Values set here are percent-decoded (`Bearer%20tok` → `Bearer tok`). A value with no `key=value` pair is treated as missing auth (the API key is not used instead); a blank (whitespace-only) value is treated as unset. |
53
+ | `FRUGAL_COMPONENTS` | Comma-separated logical components (e.g. `ai-handler,reco`). When set, each metric is ALSO emitted once per component (tagged `frugal.component`) on top of the base project series, so usage can be viewed for the whole project or split by component. Deduped, and only the first **10** entries are honored (hard cap, not configurable); the rest are dropped at startup with one debug warning. |
52
54
  | `FRUGAL_CUSTOM_PROVIDERS` | Comma-separated allowlist of provider names beyond the built-in registry (e.g. `together_ai,fireworks`) that should pass through verbatim on `gen_ai.system` / `gen_ai.provider.name` instead of being coerced to `custom`. Capped at 10 (extras are coerced) to keep label cardinality bounded. |
53
- | `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; a value outside the first 20 entries (hard cap, not configurable) is coerced to `other`. |
55
+ | `FRUGAL_CUSTOM_LABEL` | Comma-separated allowlist of values for the `frugal.custom_label` label (e.g. `billing,search`). Applies only to `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`. The vocabulary is the first **20** comma-separated entries by raw position (hard cap, not configurable); empty entries are ignored and duplicates collapsed. `other` is reserved (the coercion sentinel) and is ignored with one warning. A value outside the allowlist is coerced to `other` with a bounded once-per-value warning; when the variable is unset, every value is `other`. One warning fires only when non-empty entries past the 20th are actually dropped (a trailing comma does not warn). |
56
+ | `FRUGAL_METRICS_EXPORTER` | `otlp` (default) or `console` (print metrics to stdout; console mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables are ignored). Case-insensitive; any other value warns once and uses `otlp`. |
57
+ | `FRUGAL_METRICS_PROJECT_ID` / `FRUGAL_METRICS_API_KEY` | Namespaced overrides read before `FRUGAL_PROJECT_ID` / `FRUGAL_API_KEY`, for apps that already use those names. An empty value falls through to the bare name. |
54
58
  | `FRUGAL_PATH_BLOCK_LIST` | Comma-separated repo-relative paths to skip for caller attribution (see below). |
55
- | `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds. Default `60000`. |
56
- | `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute. Default `500`. Set to `0` to disable. |
57
- | `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls (0..1) that run the lightweight prompt analyzer. Default `0.1`. Set to `0` to disable, `1` for every call. |
59
+ | `FRUGAL_METRIC_EXPORT_INTERVAL_MS` | Export cadence in milliseconds, `1`-`2147483647`. Default `60000`; a value outside the range warns once and uses the default. |
60
+ | `FRUGAL_MAX_CALL_SITES` | Cardinality cap — max distinct `(caller_file, caller_function)` pairs to attribute; past it `frugal.caller_file` / `frugal.caller_function` become `<overflow>`. Default `500`. Set to `0` to disable. |
61
+ | `FRUGAL_MAX_MODELS` | Cardinality cap — max distinct model strings to attribute; past it `gen_ai.request.model` / `gen_ai.response.model` become `<overflow>`. One registry shared by both labels, so a request alias and its dated response snapshot each take a slot. Default `100`. Set to `0` to disable. |
62
+ | `FRUGAL_MAX_SERIES_PER_INSTRUMENT` | OpenTelemetry's per-instrument limit on distinct attribute **combinations**, raised from the SDK default. Default `50000`. Set to `0` for effectively unlimited (a 1,000,000 ceiling). Applied **only when the installed `opentelemetry-sdk` accepts a per-instrument cardinality limit** — current releases (through 1.42) do not implement one, so nothing is capped and setting this variable emits one `warn` telling you it has no effect. |
63
+ | `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE` | Fraction of calls that run the lightweight prompt analyzer: a decimal in [0, 1], plain or scientific notation (e.g. `0.25`, `1e-1`); anything else warns once and uses `0.1`. Default `0.1`. Set to `0` to disable, `1` for every call. |
58
64
  | `FRUGAL_DUPLICATE_REQUEST_DETECTION` | Duplicate-request detection (the `frugal.gen_ai.duplicate_requests` metric) is **on by default** and hashes each chat request in-process (the hash never leaves the SDK). Set to `0`/`false`/`no`/`off` to disable — no hashing, no filter memory, no metric. |
59
- | `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `604800` (7 days). |
65
+ | `FRUGAL_DUPLICATE_WINDOW_SECONDS` | How long a request signature is remembered for duplicate detection. Default `3600`, max `86400` (1 day). |
60
66
  | `FRUGAL_DUPLICATE_CAPACITY` | Expected distinct request signatures per window (sizes the in-memory filter). Default `10000`, max `1000000`. Raise it if the SDK warns the filter is saturated. |
61
67
  | `FRUGAL_DUPLICATE_FP_RATE` | Target false-positive rate of the duplicate filter. Default `0.01`, clamped to `[1e-6, 0.5]`. |
62
68
  | `FRUGAL_DUPLICATE_GENERATIONS` | Rotation granularity of the duplicate filter's time window. Default `2`, max `16`. |
@@ -74,8 +80,9 @@ Every wrapped call emits OTel metrics tagged with the caller's source file and f
74
80
  | `frugal.gen_ai.cache_read_tokens` | histogram | Prompt cache hits | — |
75
81
  | `frugal.gen_ai.cache_write_tokens` | histogram | Prompt cache writes | — |
76
82
  | `frugal.gen_ai.reasoning_tokens` | histogram | OpenAI reasoning tokens (o-series), Anthropic thinking tokens| — |
83
+ | `frugal.gen_ai.agent.turns` | histogram | Model-response turns per successful `invoke_agent` call (manual only) | — |
77
84
  | `frugal.gen_ai.errors` | counter | Errors by exception type | `frugal.error.type` |
78
- | `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality cap, bugs) | `frugal.internal_error.stage` |
85
+ | `frugal.gen_ai.internal_errors` | counter | frugal-metrics internal failures (cardinality caps, bugs) | `frugal.internal_error.stage`, `frugal.internal_error.type`, `frugal.instrument` |
79
86
  | `frugal.gen_ai.output_cap_utilization` | histogram | `output_tokens / max_tokens` ratio when `max_tokens` is set | `frugal.structured_output_mode` |
80
87
  | `frugal.gen_ai.history_depth` | histogram | `messages.length` — every chat call (100%, not sampled) | — |
81
88
  | `frugal.gen_ai.duplicate_requests` | counter | Repeat-request detection — every chat call (100%, not sampled; disable via `FRUGAL_DUPLICATE_REQUEST_DETECTION`) | `frugal.dup_state` |
@@ -92,15 +99,17 @@ The analyzer-sampled metrics fire on a fraction of calls
92
99
  Every metric carries this base attribute set (except
93
100
  `frugal.gen_ai.internal_errors`, which is emitted from failure paths where
94
101
  these attributes may not exist yet and carries only `frugal.project_id`, the
95
- stage/type, and `frugal.customer_id` / `frugal.custom_label` when set):
102
+ stage/type, `frugal.instrument` on the `otel_cardinality_overflow` stage, and
103
+ `frugal.customer_id` / `frugal.custom_label` when set; `frugal.custom_label`
104
+ never on the setup_failed / emission stages):
96
105
 
97
106
  - `frugal.project_id` — from your env vars (`frugal.customer_id` too when set); `frugal.component` is added on the per-component series when `FRUGAL_COMPONENTS` is set
98
107
  - `frugal.caller_file`, `frugal.caller_function` — auto-detected from the call stack
99
108
  - `gen_ai.system` + `gen_ai.provider.name` — the provider, dual-emitted under both the legacy and current OTel attribute (`openai`, `anthropic`, `aws.bedrock`, `vertex_ai`/`gcp.vertex_ai`, …, or `custom`). Set via the `system` kwarg; an unrecognized value is coerced to `custom` with a one-time warning. See [`docs/metrics-reference.md`](docs/metrics-reference.md#attributes).
100
- - `gen_ai.request.model`, `gen_ai.response.model` — the model used
109
+ - `gen_ai.request.model`, `gen_ai.response.model` — the model used; both draw on one shared registry and become `<overflow>` past `FRUGAL_MAX_MODELS` distinct model strings
101
110
  - `gen_ai.operation.name` — `chat`, `text_completion`, `embeddings`, `generate_content`, `execute_tool`, `create_agent`, `invoke_agent`, or `other` (the sentinel an unrecognized operation coerces to)
102
111
  - `frugal.batched` — `true` for batch-API paths (OpenAI Batch API submissions; Anthropic Message Batches submissions and results), `false` otherwise
103
- - `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other`
112
+ - `frugal.custom_label` — only on `track_ai_call` / `start_ai_call` calls that pass `custom_label=...`; the value must be in the `FRUGAL_CUSTOM_LABEL` allowlist or it is coerced to `other` (with a once-per-value warning). `custom_label=None` means no label; a non-string value is coerced to `other`
104
113
 
105
114
  The "Additional attributes" column above lists what each metric carries on top of the base set. See [`docs/metrics-reference.md`](docs/metrics-reference.md) for the values each attribute takes, sentinel values for caller fields, and the full internal-error-stage list.
106
115
 
@@ -140,6 +149,39 @@ with track_ai_call(system="anthropic", request_model="claude-sonnet-4-20250514")
140
149
  )
141
150
  ```
142
151
 
152
+ For an enclosing agent invocation, set `operation="invoke_agent"` and report
153
+ the number of model-response steps performed inside it:
154
+
155
+ ```python
156
+ with track_ai_call(
157
+ system="anthropic",
158
+ operation="invoke_agent",
159
+ request_model="claude-sonnet-4-20250514",
160
+ ) as span:
161
+ result = my_agent_framework_call(prompt)
162
+ span.set_usage(
163
+ input_tokens=result.usage.input_tokens,
164
+ output_tokens=result.usage.output_tokens,
165
+ agent_turns=result.turn_count,
166
+ )
167
+ ```
168
+
169
+ - Provider wrappers still measure individual model calls and never set
170
+ `agent_turns`.
171
+ - Agent frameworks such as the Claude Agent SDK, OpenAI Agents SDK, and
172
+ LangGraph do not have dedicated wrappers yet; track them manually as shown.
173
+ - `frugal.gen_ai.history_depth` is the request message count
174
+ (`messages.length`), not a turn count.
175
+ - Missing, non-integer, non-positive, or otherwise invalid `agent_turns`
176
+ values are silently dropped. They never raise or affect customer execution.
177
+ - Turn data is emitted only on success, like token usage.
178
+ - **Avoid double-counting tokens.** Only report aggregate `input_tokens` /
179
+ `output_tokens` on the `invoke_agent` call when the agent's inner
180
+ model-client calls are **not** already wrapped by a supported provider
181
+ wrapper. If they are wrapped, those wrappers already emit token usage per
182
+ inner call — report `agent_turns` only on the `invoke_agent` call in that
183
+ case, or the same tokens are double-counted.
184
+
143
185
  The `with` block times the call and emits metrics on exit — or an error metric if the block raises (the exception always propagates). Works the same around an `await` in async code. Call `span.set_usage(...)` once the response is back; if you never call it, you still get a call-count and latency point.
144
186
 
145
187
  For calls that span function boundaries — where a single `with` block won't fit — use the manual handle:
@@ -162,9 +204,9 @@ except BaseException as exc:
162
204
 
163
205
  `success()` and `error()` finalize the call exactly once and are idempotent — extra calls are ignored.
164
206
 
165
- **Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
207
+ **Metadata** — `track_ai_call` / `start_ai_call` kwargs (all optional): `system` (→ `gen_ai.system`, default `"custom"`), `operation` (default `"chat"`), `request_model` (default `"<unknown>"`), `custom_label` (→ the `frugal.custom_label` label on every metric of the call except the `frugal.gen_ai.internal_errors` setup_failed / emission points; trimmed, and coerced to `other` unless in the `FRUGAL_CUSTOM_LABEL` allowlist). To also run the sampled prompt analyzer, pass any of `messages` / `system_prompt` / `tools` / `response_format` / `max_tokens`.
166
208
 
167
- **Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `response_model`.
209
+ **Usage** — `set_usage(...)` / `success(...)` kwargs (all optional): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `agent_turns`, `response_model`.
168
210
 
169
211
  The same env-var configuration, safety guarantees, and exported metrics apply. See [`docs/quickstart.md`](docs/quickstart.md#custom-call-sites) for the full custom call-site walkthrough.
170
212
 
@@ -261,14 +303,27 @@ It is **opt-in** because installing signal handlers is intrusive. If you run you
261
303
 
262
304
  ## Verifying the setup
263
305
 
264
- Set the OTLP endpoint to empty to print metrics to the console instead of exporting them:
306
+ Set `FRUGAL_METRICS_EXPORTER=console` to print the metrics to stdout instead
307
+ of exporting them. Console mode needs only `FRUGAL_PROJECT_ID`; the API key,
308
+ endpoint and header variables are ignored. An empty
309
+ endpoint does not select the console exporter: it falls through to the default
310
+ Frugal endpoint.
311
+
312
+ ```bash
313
+ export FRUGAL_PROJECT_ID=my-project
314
+ export FRUGAL_METRICS_EXPORTER=console
315
+ export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
316
+ python my_script.py
317
+ ```
318
+
319
+ Or point the library at a local OTLP/HTTP collector to see what it exports:
265
320
 
266
321
  ```bash
267
- export FRUGAL_API_KEY=local
322
+ export FRUGAL_API_KEY=test-key
268
323
  export FRUGAL_PROJECT_ID=my-project
269
- export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT= # empty → ConsoleMetricExporter
324
+ export FRUGAL_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/metrics
270
325
  export FRUGAL_METRIC_EXPORT_INTERVAL_MS=5000
271
326
  python my_script.py
272
327
  ```
273
328
 
274
- You should see JSON metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
329
+ The collector should receive metric records with `gen_ai.*` and `frugal.*` attributes every 5 seconds.
@@ -20,6 +20,7 @@ metric may carry the extra attributes shown in the rightmost column.
20
20
  | `frugal.gen_ai.cache_read_tokens` | histogram | `{token}` | when provider reports cache reads | — |
21
21
  | `frugal.gen_ai.cache_write_tokens` | histogram | `{token}` | when provider reports cache writes | — |
22
22
  | `frugal.gen_ai.reasoning_tokens` | histogram | `{token}` | OpenAI o-series reasoning models | — |
23
+ | `frugal.gen_ai.agent.turns` | histogram | `{turn}` | successful `invoke_agent` calls with a valid `agent_turns` count (manual tracking only) | — |
23
24
  | `frugal.gen_ai.errors` | counter | `{error}` | when the wrapped call raises | `frugal.error.type` |
24
25
  | `frugal.gen_ai.internal_errors` | counter | `{error}` | frugal-metrics internal failures (cardinality cap, instrumentation bugs) | `frugal.internal_error.stage` |
25
26
  | `frugal.gen_ai.output_cap_utilization`| histogram | `1` | when request had `max_tokens` (ratio 0..1) | `frugal.structured_output_mode` |
@@ -34,6 +35,37 @@ metric may carry the extra attributes shown in the rightmost column.
34
35
  `gen_ai.client.token.usage` is split across input and output via the
35
36
  `gen_ai.token.type` attribute (`input` | `output`).
36
37
 
38
+ ### Agent turns
39
+
40
+ Manual `track_ai_call` and `start_ai_call` instrumentation accepts the
41
+ `agent_turns` usage kwarg through `AICallSpan.set_usage` and
42
+ `AICallSpan.success`. A sample is emitted only when the call succeeds, the
43
+ normalized `gen_ai.operation.name` is exactly `invoke_agent`, and the value is
44
+ a positive integer. Booleans, missing values, non-integers, zero, and negative
45
+ values are silently ignored; validation never affects customer execution.
46
+ Provider wrappers never set this field, so agent frameworks remain
47
+ manual-tracking call sites in this release. `frugal.gen_ai.history_depth`
48
+ continues to mean request message count, not agent turns. The histogram uses
49
+ explicit bucket boundaries
50
+ `(1, 2, 3, 4, 5, 6, 8, 10, 12, 15, 20, 25, 30, 40, 50, 75, 100)` (mirroring
51
+ `gen_ai.client.operation.duration`'s view) so `histogram_quantile()` resolves
52
+ meaningful spread across realistic agent-run turn counts instead of
53
+ collapsing into OTel's duration-tuned default buckets.
54
+
55
+ Only report aggregate token usage on the `invoke_agent` call itself when the
56
+ agent's inner model-client calls are **not** already wrapped by a supported
57
+ provider wrapper; if they are wrapped, report `agent_turns` only on the
58
+ `invoke_agent` call, since the inner wrappers already emit token usage and
59
+ combining both would double-count tokens.
60
+
61
+ To derive average turns per reporting agent invocation in MetricsQL, use:
62
+
63
+ ```promql
64
+ increase(frugal.gen_ai.agent.turns_sum[...])
65
+ /
66
+ increase(frugal.gen_ai.agent.turns_count[...])
67
+ ```
68
+
37
69
  The analyzer-sampled metrics are gated by `FRUGAL_PROMPT_ANALYSIS_SAMPLE_RATE`
38
70
  (default `0.1`). Sampling is deterministic per `(caller_file, caller_function,
39
71
  minute)` — within a minute a given call site is either fully sampled or fully
@@ -88,8 +120,8 @@ Controls:
88
120
  `false`/`0`/`no`/`off`, no filter is allocated, no signature is hashed, and the
89
121
  metric is not emitted. Any other value keeps detection enabled and emits a
90
122
  one-time `warn` diagnostic naming the unrecognized value.
91
- - `FRUGAL_DUPLICATE_WINDOW_SECONDS` (default `3600`, clamped to `604800` — 7
92
- days) — retention window. Because the filter rotates in `generations` slots,
123
+ - `FRUGAL_DUPLICATE_WINDOW_SECONDS` (default `3600`, clamped to `86400` — 1
124
+ day) — retention window. Because the filter rotates in `generations` slots,
93
125
  a signature is remembered for **at least**
94
126
  `window × (generations−1)/generations` and **at most** `window` seconds
95
127
  (30–60 min at defaults) — treat the counterfactual cache TTL as that range,
@@ -101,6 +133,15 @@ Controls:
101
133
  - `FRUGAL_DUPLICATE_GENERATIONS` (default `2`, clamped to `16`) — rotating-Bloom
102
134
  generation count.
103
135
 
136
+ Exporter selection: `FRUGAL_METRICS_EXPORTER` (`otlp` default, or `console` to
137
+ print every metric to stdout via the OpenTelemetry console exporter; console
138
+ mode needs only `FRUGAL_PROJECT_ID`; the API key, endpoint and header variables
139
+ are ignored, and any other value warns once and uses `otlp`). An empty endpoint variable falls through to the next source and
140
+ never selects console.
141
+
142
+ `FRUGAL_METRIC_EXPORT_INTERVAL_MS` accepts `1` to `2147483647` (default `60000`);
143
+ a value outside that range warns once and uses the default.
144
+
104
145
  Out-of-range or non-finite values for any of these fall back to the default (or
105
146
  clamp to the bound) with a one-time named `warn` through the debug callback.
106
147
 
@@ -159,18 +200,18 @@ Reading `duplicate_requests` correctly:
159
200
 
160
201
  | Attribute | Value |
161
202
  | ------------------------- | ---------------------------------------------------------- |
162
- | `frugal.customer_id` | From `FRUGAL_CUSTOMER_ID` |
163
- | `frugal.project_id` | From `FRUGAL_PROJECT_ID` |
203
+ | `frugal.customer_id` | From `FRUGAL_CUSTOMER_ID` — present only when set |
204
+ | `frugal.project_id` | From `FRUGAL_METRICS_PROJECT_ID`, else `FRUGAL_PROJECT_ID` |
164
205
  | `frugal.component` | From each entry in `FRUGAL_COMPONENTS` — present only on the per-component series, absent on the base project series |
165
206
  | `frugal.caller_file` | Source file repo-relative to `FRUGAL_REPO_ROOT` |
166
207
  | `frugal.caller_function` | Function/method name at the call site |
167
- | `frugal.error.type` | Exception class name (error-path metrics only) |
168
- | `frugal.batched` | `true` when the metric came from a provider batch-API path, `false` otherwise. Present on every metric. Batch submissions (OpenAI `batches.create`, Anthropic `messages.batches.create`) emit `frugal.gen_ai.calls` with `batched="true"` — one point per request for Anthropic (the requests array is in hand, grouped by model), one per submission for OpenAI (per-request counts are not reported at submission time, so OpenAI batched call volume is a floor, never an overcount). Iterating Anthropic `messages.batches.results()` additionally emits per-entry `gen_ai.client.token.usage` (+ cache metrics) with `batched="true"`, attributed to the call site that submitted the batch when the submission happened in the same process (bounded in-process memory; falls back to the results call site). OpenAI batch token usage lives in the output file read through the generic files API and is not intercepted. The manual track API accepts a `batched` flag for hand-instrumented batch paths. An app on an SDK predating batch support reads `false` on genuinely batched traffic — the label reflects what the deployed SDK can see. |
208
+ | `frugal.error.type` | Exception class name (error-path metrics only); stripped from `frugal.gen_ai.history_depth` and `frugal.gen_ai.duplicate_requests` so success and error points stay additive |
209
+ | `frugal.batched` | `true` when the metric came from a provider batch-API path, `false` otherwise. Present on every metric of the call except the `frugal.gen_ai.internal_errors` points. Batch submissions (OpenAI `batches.create`, Anthropic `messages.batches.create`) emit `frugal.gen_ai.calls` with `batched="true"` — one point per request for Anthropic (the requests array is in hand, grouped by model), one per submission for OpenAI (per-request counts are not reported at submission time, so OpenAI batched call volume is a floor, never an overcount). Iterating Anthropic `messages.batches.results()` additionally emits per-entry `gen_ai.client.token.usage` (+ cache metrics) with `batched="true"`, attributed to the call site that submitted the batch when the submission happened in the same process (bounded in-process memory; falls back to the results call site). OpenAI batch token usage lives in the output file read through the generic files API and is not intercepted. The manual track API accepts a `batched` flag for hand-instrumented batch paths. An app on an SDK predating batch support reads `false` on genuinely batched traffic — the label reflects what the deployed SDK can see. |
169
210
  | `frugal.structured_output_mode` | `none` \| `json_object` \| `json_schema` \| `tools`. Derived from request params by precedence: non-empty `tools` → `tools`; else `response_format.type` `json_object` / `json_schema` → the matching value; else `none`. On `frugal.gen_ai.output_cap_utilization` (when `max_tokens` set), and on `frugal.gen_ai.calls` for every call with a request payload (omitted when there is no payload). |
170
211
  | `frugal.block` | `system` \| `user` \| `tools` \| `schema`. Only on `frugal.gen_ai.cache_prefix_stable`. |
171
212
  | `frugal.stable` | `true` when the block hash matches the previous call at this site. Only on `frugal.gen_ai.cache_prefix_stable`. |
172
213
  | `frugal.history_bucket` | Bounded bucket of `messages.length`: `0` \| `1-2` \| `3-5` \| `6-10` \| `11-20` \| `21-50` \| `51+`. On `frugal.gen_ai.calls` and `gen_ai.client.token.usage`, and only for chat calls (payload has a `messages` array); omitted otherwise. 100% (not analyzer-sampled). |
173
- | `frugal.custom_label` | Custom label (e.g. a business unit) passed as `custom_label=` to `track_ai_call` / `start_ai_call` — never set by the provider wrappers. When passed, it rides EVERY metric of that call (success and error paths alike). The value is trimmed of surrounding whitespace (matching the Node SDK), then must match one of the first 20 entries of `FRUGAL_CUSTOM_LABEL` (hard cap, not configurable) or it is coerced to `other`. Omitted when not passed. |
214
+ | `frugal.custom_label` | Custom label (e.g. a business unit) passed as `custom_label=` to `track_ai_call` / `start_ai_call` — never set by the provider wrappers. When passed, it rides every metric of that call (success and error paths alike), except the `frugal.gen_ai.internal_errors` `setup_failed`/`emission` points, which carry only project, customer and error type. The value is trimmed of surrounding whitespace (matching the Node SDK), then must match the closed vocabulary from the first 20 comma-separated entries of `FRUGAL_CUSTOM_LABEL` by raw position (hard cap, not configurable; empties ignored, duplicates collapsed) or it is coerced to `other` with a bounded once-per-value warning. `other` is reserved as the coercion sentinel: an `other` entry in the env var is ignored with a warning. With the variable unset, every value is `other`. A non-string value is coerced to `other` with a type warning; `None` means no label. A truncation warning fires only when non-empty entries past the 20th are dropped. Omitted when not passed. |
174
215
  | `frugal.dup_state` | `first_seen` \| `repeat_seen` — whether this request's canonical signature was already seen within the duplicate-detection window. Only on `frugal.gen_ai.duplicate_requests`. |
175
216
 
176
217
  ### Sentinel values for `frugal.caller_file` / `frugal.caller_function`
@@ -182,6 +223,11 @@ Reading `duplicate_requests` correctly:
182
223
  | `<lambda>` | Call originated from a `lambda` expression. This is Python's native `co_name` for lambdas. Only appears in `frugal.caller_function`. |
183
224
  | `<overflow>` | Cardinality cap (`FRUGAL_MAX_CALL_SITES`) exceeded. See below. |
184
225
 
226
+ > `<overflow>` is the shared overflow sentinel, not a caller-only value: it also
227
+ > appears on `gen_ai.request.model` / `gen_ai.response.model` once
228
+ > `FRUGAL_MAX_MODELS` distinct model strings have been observed. See
229
+ > [Model cap behavior](#model-cap-behavior).
230
+
185
231
  ### Cardinality cap behavior
186
232
 
187
233
  Once `FRUGAL_MAX_CALL_SITES` distinct `(file, function)` pairs have been
@@ -196,6 +242,23 @@ The first time the cap is hit per process, the library emits a single
196
242
  `frugal.gen_ai.internal_errors{stage=cardinality_cap_hit}` point so you can
197
243
  see the transition in your dashboard.
198
244
 
245
+ ### Model cap behavior
246
+
247
+ Model ids are free-form provider strings, so `FRUGAL_MAX_MODELS` (default `100`,
248
+ `0` disables) bounds them the same way. The first N distinct model strings a
249
+ process observes keep their real names; every later one is reported as
250
+ `<overflow>` on whichever of `gen_ai.request.model` / `gen_ai.response.model`
251
+ carries it. The registry is process-wide and **shared by both labels** — a model
252
+ first seen in a response is free to use as a request model, and vice versa — so
253
+ a model already in the registry is never re-charged. The default is generous
254
+ because one logical model normally occupies **two** slots: providers answer a
255
+ request alias (`claude-sonnet-4-5`) with a dated response snapshot
256
+ (`claude-sonnet-4-5-20250929`). The SDK's own sentinels — `<unknown>` and
257
+ `<overflow>` — pass through without consuming a slot. The first time the cap is
258
+ hit per process the library emits a single
259
+ `frugal.gen_ai.internal_errors{stage=model_cap_hit}` point plus one `warn`
260
+ through the debug callback.
261
+
199
262
  ### Internal-error stages
200
263
 
201
264
  `frugal.gen_ai.internal_errors` carries a `frugal.internal_error.stage`
@@ -203,18 +266,23 @@ attribute describing where our own code failed. Non-exhaustive list:
203
266
 
204
267
  | Stage | When it fires |
205
268
  | ----------------------- | ---------------------------------------------------------------------- |
206
- | `setup_failed` | Our wrapper setup (OTel init, caller resolution) threw; original call continued normally |
269
+ | `setup_failed` | Our wrapper setup (OTel init, caller resolution) threw; original call continued normally. Emitted by the provider wrappers and the manual track API (`track_ai_call` / `start_ai_call`); carries `frugal.project_id`, `frugal.customer_id` (when set) and `frugal.internal_error.type` |
207
270
  | `stream_proxy_failed` | Constructing the stream wrapper threw; original response returned unwrapped |
208
- | `emission` | Emitting a success/error metric threw |
271
+ | `emission` | Emitting a success/error metric threw. Emitted by the provider wrappers and the manual track API; carries the same project/customer identity and `frugal.internal_error.type` as `setup_failed` |
272
+ | `call_shape` | Deriving history_depth / history_bucket / structured_output_mode from the request payload threw; carries `frugal.internal_error.type`; core metrics still emitted |
209
273
  | `duplicate_detection` | The duplicate-request path (signature, filter, or emission) threw — including the one-time filter allocation failure; carries `frugal.internal_error.type` with the exception class |
210
274
  | `cardinality_cap_hit` | `FRUGAL_MAX_CALL_SITES` was just reached (once per process) |
275
+ | `model_cap_hit` | `FRUGAL_MAX_MODELS` was just reached (once per process); further distinct models are reported as `<overflow>` |
276
+ | `otel_cardinality_overflow` | OpenTelemetry merged an instrument's attribute sets into its `otel.metric.overflow` series (once per instrument name); carries `frugal.instrument` with the affected instrument |
211
277
 
212
278
  Unlike every other metric, `frugal.gen_ai.internal_errors` does **not** carry
213
279
  the full base attribute set — it is emitted from failure paths where the
214
280
  caller/provider/model attributes may not exist yet. Its points carry only
215
281
  `frugal.project_id`, `frugal.internal_error.stage` (plus
216
- `frugal.internal_error.type` where noted), and — when set —
217
- `frugal.customer_id` and `frugal.custom_label`.
282
+ `frugal.internal_error.type` where noted, and `frugal.instrument` on the
283
+ `otel_cardinality_overflow` stage), and — when set — `frugal.customer_id` and
284
+ `frugal.custom_label` (`setup_failed` and `emission` never carry the custom
285
+ label).
218
286
 
219
287
  ## Provider coverage
220
288