ehs-llm-client 0.1.14__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ehs_llm_client-0.2.0/PKG-INFO +445 -0
- ehs_llm_client-0.2.0/README.md +431 -0
- ehs_llm_client-0.2.0/ehs_llm_client/__init__.py +12 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client/client.py +122 -21
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client/config.py +1 -0
- ehs_llm_client-0.2.0/ehs_llm_client/decisions.py +254 -0
- ehs_llm_client-0.2.0/ehs_llm_client/exceptions.py +18 -0
- ehs_llm_client-0.2.0/ehs_llm_client/py.typed +0 -0
- ehs_llm_client-0.2.0/ehs_llm_client.egg-info/PKG-INFO +445 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client.egg-info/SOURCES.txt +4 -1
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client.egg-info/requires.txt +1 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/pyproject.toml +7 -2
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/tests/test_client.py +129 -6
- ehs_llm_client-0.2.0/tests/test_decisions.py +323 -0
- ehs_llm_client-0.1.14/PKG-INFO +0 -286
- ehs_llm_client-0.1.14/README.md +0 -274
- ehs_llm_client-0.1.14/ehs_llm_client/__init__.py +0 -3
- ehs_llm_client-0.1.14/ehs_llm_client/exceptions.py +0 -9
- ehs_llm_client-0.1.14/ehs_llm_client.egg-info/PKG-INFO +0 -286
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client/utils.py +0 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client.egg-info/dependency_links.txt +0 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/ehs_llm_client.egg-info/top_level.txt +0 -0
- {ehs_llm_client-0.1.14 → ehs_llm_client-0.2.0}/setup.cfg +0 -0
|
@@ -0,0 +1,445 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ehs-llm-client
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Async client for OpenAI, Azure OpenAI, Microsoft Foundry, Google Gemini and TypeSafe decision models
|
|
5
|
+
Author-email: Andersen Huang <andersen.huang@ehsanalytics.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Requires-Python: >=3.10
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
Requires-Dist: openai>=1.97.0
|
|
10
|
+
Requires-Dist: httpx[http2]
|
|
11
|
+
Requires-Dist: google-genai
|
|
12
|
+
Requires-Dist: python-dotenv
|
|
13
|
+
Requires-Dist: python-dateutil
|
|
14
|
+
|
|
15
|
+
# llm-client
|
|
16
|
+
|
|
17
|
+
[](https://pypi.org/project/ehs-llm-client/)
|
|
18
|
+
[](https://www.python.org/downloads/)
|
|
19
|
+
|
|
20
|
+
`ehs-llm-client` is a unified **async Python client** for **OpenAI**, **Azure OpenAI**,
|
|
21
|
+
**Microsoft Foundry (AIF)**, **Google Gemini**, and **TypeSafe/Jev**.
|
|
22
|
+
It supports generative responses and native typed decisions. Provider batch APIs
|
|
23
|
+
remain available for supported generative deployments.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Features
|
|
28
|
+
|
|
29
|
+
- Async support for multiple LLM providers
|
|
30
|
+
- Unified interface across OpenAI, Azure, and Google Gemini
|
|
31
|
+
- Structured JSON responses via schemas
|
|
32
|
+
- Native `choice`, `noul` and `score` decisions via Foundry and TypeSafe
|
|
33
|
+
- Retry and timeout handling
|
|
34
|
+
- Batch processing support
|
|
35
|
+
- Environment variable configuration for API keys
|
|
36
|
+
- Easy integration into existing projects or monorepos
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## Installation
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
# Install from PyPI
|
|
44
|
+
pip install ehs-llm-client
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Or install editable version during development:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
# From project root
|
|
51
|
+
pip install -e .
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## Configuration
|
|
57
|
+
|
|
58
|
+
`ehs-llm-client` supports **three configuration modes**:
|
|
59
|
+
|
|
60
|
+
### 1️⃣ Config file (`.cfg`)
|
|
61
|
+
|
|
62
|
+
Example: `llmconfig.cfg`
|
|
63
|
+
|
|
64
|
+
```ini
|
|
65
|
+
[default_settings]
|
|
66
|
+
provider = openai
|
|
67
|
+
model = gpt-4.1-mini
|
|
68
|
+
|
|
69
|
+
[prod]
|
|
70
|
+
provider = openai
|
|
71
|
+
model = gpt-4.1
|
|
72
|
+
model_batch = gpt-4.1-mini
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Usage:
|
|
76
|
+
|
|
77
|
+
```python
|
|
78
|
+
from ehs_llm_client import LLM
|
|
79
|
+
|
|
80
|
+
llm = LLM("prod", config_file_path="llmconfig.cfg")
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
### 2️⃣ Config dictionary (for tests / CI)
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
from ehs_llm_client import LLM
|
|
89
|
+
|
|
90
|
+
llm = LLM(
|
|
91
|
+
"default",
|
|
92
|
+
config={
|
|
93
|
+
"default": {
|
|
94
|
+
"provider": "openai",
|
|
95
|
+
"model": "gpt-4.1-mini"
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
### 3️⃣ Environment variable fallback
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
export LLM_PROVIDER=openai
|
|
107
|
+
export LLM_MODEL=gpt-4.1-mini
|
|
108
|
+
export OPENAI_API_KEY=sk-...
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
llm = LLM("default")
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Azure OpenAI with GPT-6
|
|
116
|
+
|
|
117
|
+
Deploy your chosen GPT-6 model in Azure, then use its **deployment name** as `model`:
|
|
118
|
+
|
|
119
|
+
```ini
|
|
120
|
+
[azure]
|
|
121
|
+
provider = azure_openai
|
|
122
|
+
model = my-gpt6-deployment
|
|
123
|
+
base_url = https://YOUR-RESOURCE.openai.azure.com/
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Set `AZURE_OPENAI_API_KEY` and initialize the client:
|
|
127
|
+
|
|
128
|
+
```python
|
|
129
|
+
llm = LLM("azure", config_file_path="llmconfig.cfg")
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
You can also configure everything with environment variables:
|
|
133
|
+
|
|
134
|
+
```powershell
|
|
135
|
+
$env:LLM_PROVIDER = "azure_openai"
|
|
136
|
+
$env:LLM_MODEL = "my-gpt6-deployment"
|
|
137
|
+
$env:AZURE_OPENAI_ENDPOINT = "https://YOUR-RESOURCE.openai.azure.com/"
|
|
138
|
+
$env:AZURE_OPENAI_API_KEY = "YOUR-AZURE-API-KEY"
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Then use `LLM("default")`. `LLM_BASE_URL` or a configured `base_url` takes precedence
|
|
142
|
+
over `AZURE_OPENAI_ENDPOINT`. Both a resource URL and a full `/openai/v1/` URL are
|
|
143
|
+
accepted, including endpoints on `services.ai.azure.com`. The client uses Azure's
|
|
144
|
+
v1 API; existing `api_version` settings are retained for compatibility and ignored.
|
|
145
|
+
See [Azure Responses API documentation](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses)
|
|
146
|
+
for supported models and regions.
|
|
147
|
+
|
|
148
|
+
### Microsoft Foundry (AIF) and TypeSafe
|
|
149
|
+
|
|
150
|
+
Version 0.2.0 adds two providers. Existing providers, credential precedence and
|
|
151
|
+
the `get_response_async()` positional arguments and return tuple are preserved.
|
|
152
|
+
|
|
153
|
+
| Provider | `api` | Method | Endpoint |
|
|
154
|
+
| --- | --- | --- | --- |
|
|
155
|
+
| `azure_foundry` | `responses` (default) | `get_response_async()` | `/openai/v1/responses` |
|
|
156
|
+
| `azure_foundry` | `systemone` | `decide_async()` | `/providers/microsoft/v1/systemone` |
|
|
157
|
+
| `typesafe` | `systemone` (default) | `decide_async()` | `https://api.typesafe.ai/v1/systemone` |
|
|
158
|
+
|
|
159
|
+
Select the API explicitly for Foundry decision deployments; deployment names do
|
|
160
|
+
not determine the protocol. This release supports these two APIs, rather than
|
|
161
|
+
every model/API in the Foundry catalog.
|
|
162
|
+
|
|
163
|
+
```ini
|
|
164
|
+
[foundry_gpt]
|
|
165
|
+
provider = azure_foundry
|
|
166
|
+
api = responses
|
|
167
|
+
model = gpt-6-luna
|
|
168
|
+
base_url = https://YOUR-RESOURCE.services.ai.azure.com
|
|
169
|
+
default_reasoning_effort = low
|
|
170
|
+
|
|
171
|
+
[foundry_decision]
|
|
172
|
+
provider = azure_foundry
|
|
173
|
+
api = systemone
|
|
174
|
+
model = decision-1
|
|
175
|
+
base_url = https://YOUR-RESOURCE.services.ai.azure.com
|
|
176
|
+
|
|
177
|
+
[typesafe_jev]
|
|
178
|
+
provider = typesafe
|
|
179
|
+
model = jev-latest
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
For Foundry, `model` is the **deployment name**. For TypeSafe, it is the model
|
|
183
|
+
name or alias. Foundry uses its own `AZURE_FOUNDRY_API_KEY` and
|
|
184
|
+
`AZURE_FOUNDRY_ENDPOINT`; it never falls back to Azure OpenAI credentials.
|
|
185
|
+
A configured `base_url` takes precedence over `LLM_BASE_URL`, then the
|
|
186
|
+
provider-specific endpoint variable. TypeSafe defaults to `https://api.typesafe.ai`;
|
|
187
|
+
an explicit `base_url` or environment-only `LLM_BASE_URL` can override it.
|
|
188
|
+
|
|
189
|
+
Foundry expects a resource endpoint, not an `/api/projects/...` URL. Responses
|
|
190
|
+
also accepts a full `/openai/v1/` endpoint. System One accepts the resource root,
|
|
191
|
+
its corresponding versioned API prefix, or its full System One endpoint.
|
|
192
|
+
These endpoints must use HTTPS and contain no query string, fragment or embedded
|
|
193
|
+
credentials. Selecting `azure_foundry` does not itself establish project billing
|
|
194
|
+
attribution; that depends on the Azure endpoint and service capabilities.
|
|
195
|
+
|
|
196
|
+
For environment-only configuration, set `LLM_PROVIDER`, `LLM_MODEL`, and
|
|
197
|
+
`LLM_API=systemone` for Foundry decision models, together with its endpoint/key.
|
|
198
|
+
TypeSafe needs `LLM_PROVIDER=typesafe`, `LLM_MODEL=jev-latest`, and
|
|
199
|
+
`TYPESAFE_API_KEY`.
|
|
200
|
+
|
|
201
|
+
See the [Microsoft Decision API](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision)
|
|
202
|
+
and [TypeSafe System One API](https://docs.typesafe.ai/api/system-one).
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## Usage
|
|
207
|
+
|
|
208
|
+
### Single async call
|
|
209
|
+
|
|
210
|
+
```python
|
|
211
|
+
import asyncio
|
|
212
|
+
from ehs_llm_client import LLM
|
|
213
|
+
|
|
214
|
+
async def main():
|
|
215
|
+
llm = LLM("prod", config_file_path="llmconfig.cfg")
|
|
216
|
+
messages = [{"role": "user", "content": "Say hello in JSON"}]
|
|
217
|
+
|
|
218
|
+
response, in_tokens, out_tokens = await llm.get_response_async(
|
|
219
|
+
messages, schema="return json"
|
|
220
|
+
)
|
|
221
|
+
|
|
222
|
+
print(response)
|
|
223
|
+
|
|
224
|
+
asyncio.run(main())
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
One `LLM` can be used from several event loops in turn, for example a Celery task
|
|
228
|
+
that calls `asyncio.run()` for each group of work. Since 0.1.13, `llm.client` is
|
|
229
|
+
replaced with a new provider client the first time it is used from a new event
|
|
230
|
+
loop. Before that, calls in a later loop reused connections opened by the closed
|
|
231
|
+
one; they failed, and the OpenAI SDK retried them ("Retrying request to
|
|
232
|
+
/responses"). Assigning `llm.client` yourself still works; the next event loop
|
|
233
|
+
to use it adopts it.
|
|
234
|
+
|
|
235
|
+
### Connections and concurrency
|
|
236
|
+
|
|
237
|
+
Hosts such as Azure App Service give each instance a limited number of outbound
|
|
238
|
+
(SNAT) ports per destination, about 128. Every open connection uses one. When
|
|
239
|
+
they run out, new connections time out (`httpx.ConnectTimeout`, then
|
|
240
|
+
`openai.APITimeoutError`) while existing ones keep working. Three settings,
|
|
241
|
+
read from the environment, control how many connections the client opens:
|
|
242
|
+
|
|
243
|
+
| Variable | Default | Effect |
|
|
244
|
+
|---|---|---|
|
|
245
|
+
| `LLM_MAX_CONCURRENCY` | `0` (no limit) | Most `get_response_async` requests in flight at once, shared by every `LLM` in the process (per event loop). Further calls wait for a slot. Waiting does not count toward `timeout`, and a call holds no slot while it sleeps between retries. Since 0.1.15. |
|
|
246
|
+
| `LLM_HTTP2` | off | `true` sends OpenAI and Azure OpenAI requests over HTTP/2, so concurrent requests share connections instead of opening one each. Servers without HTTP/2 are spoken to over HTTP/1.1. Since 0.1.15. |
|
|
247
|
+
| `LLM_KEEPALIVE_EXPIRY` | `60` | Seconds an idle connection is kept for reuse (the OpenAI SDK default is 5), so calls that come in groups reuse connections instead of opening new ones. Must be more than 0 and less than 240: Azure drops outbound connections idle for longer without telling either end. Since 0.1.14. |
|
|
248
|
+
|
|
249
|
+
The ports a process can use at once are about `LLM_MAX_CONCURRENCY` over
|
|
250
|
+
HTTP/1.1, and far fewer over HTTP/2. Several processes on one instance share
|
|
251
|
+
its ports, so size the limit as the instance's budget divided by the number of
|
|
252
|
+
processes that make calls. A lower limit means calls in a large burst wait for
|
|
253
|
+
each other; a burst that is never larger than the limit is not slowed.
|
|
254
|
+
|
|
255
|
+
HTTP/2 trade-offs: a connection that drops or loses packets affects every
|
|
256
|
+
request on it at once, and httpx's HTTP/2 support has seen less use than its
|
|
257
|
+
HTTP/1.1 support. Turn it on in a test environment first.
|
|
258
|
+
|
|
259
|
+
Other SDK defaults are unchanged: connect timeout 5 s, read timeout 600 s, at
|
|
260
|
+
most 1000 connections per client. Batch file and job calls are not limited by
|
|
261
|
+
`LLM_MAX_CONCURRENCY`.
|
|
262
|
+
|
|
263
|
+
Blocked, failed or incomplete OpenAI/Azure responses raise
|
|
264
|
+
`ehs_llm_client.exceptions.LLMResponseError` (a `RuntimeError`) instead of returning
|
|
265
|
+
partial or empty text as success. Content-filter errors are not retried with the same prompt.
|
|
266
|
+
|
|
267
|
+
### Reasoning effort
|
|
268
|
+
|
|
269
|
+
The reasoning controls from version 0.1.10 are supported:
|
|
270
|
+
|
|
271
|
+
```ini
|
|
272
|
+
[reasoning]
|
|
273
|
+
provider = openai
|
|
274
|
+
model = gpt-6-astra
|
|
275
|
+
default_reasoning_effort = medium
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Set `default_reasoning_effort` in a config section/dictionary, or set
|
|
279
|
+
`LLM_DEFAULT_REASONING_EFFORT` when using environment configuration. Override it
|
|
280
|
+
for a single call with `reasoning_effort`:
|
|
281
|
+
|
|
282
|
+
```python
|
|
283
|
+
response, in_tokens, out_tokens = await llm.get_response_async(
|
|
284
|
+
messages, schema="return json", reasoning_effort="high"
|
|
285
|
+
)
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
`call_async` also accepts `reasoning_effort`. OpenAI uses the model support check
|
|
289
|
+
from 0.1.10. Azure forwards an explicitly configured effort because deployment
|
|
290
|
+
names can be arbitrary; only set it for a reasoning-capable deployment. Supported
|
|
291
|
+
effort values depend on the model. If you omit the setting, the API's default applies.
|
|
292
|
+
See [OpenAI reasoning guidance](https://developers.openai.com/api/docs/guides/reasoning).
|
|
293
|
+
The setting applies to single calls; batch behavior remains as in 0.1.10.
|
|
294
|
+
|
|
295
|
+
The 0.1.10 method signatures are preserved, including positional reasoning,
|
|
296
|
+
retry and timeout arguments. Prefer keyword arguments for clarity.
|
|
297
|
+
|
|
298
|
+
OpenAI/Azure JSON mode (`schema="return json"`) requires string message contents,
|
|
299
|
+
as in 0.1.10. Regular calls and schema-based output retain support for content arrays.
|
|
300
|
+
|
|
301
|
+
### Native typed decisions
|
|
302
|
+
|
|
303
|
+
Use the same questions with either a Foundry `systemone` section or a TypeSafe
|
|
304
|
+
section. State can be text, a JSON object or an array. Instructions and criteria
|
|
305
|
+
descriptions may also contain JSON structures.
|
|
306
|
+
|
|
307
|
+
```python
|
|
308
|
+
import asyncio
|
|
309
|
+
from ehs_llm_client import LLM, ChoiceAnswer
|
|
310
|
+
|
|
311
|
+
async def main():
|
|
312
|
+
async with LLM("typesafe_jev", config_file_path="llmconfig.cfg") as model:
|
|
313
|
+
result = await model.decide_async(
|
|
314
|
+
state="A worker suffered a minor cut and received first aid.",
|
|
315
|
+
questions={
|
|
316
|
+
"category": {
|
|
317
|
+
"type": "choice",
|
|
318
|
+
"instructions": "Classify the reported incident.",
|
|
319
|
+
"criteria": {
|
|
320
|
+
"injury": "An injury occurred.",
|
|
321
|
+
"near_miss": "An event occurred without injury.",
|
|
322
|
+
"unknown": "There is insufficient information.",
|
|
323
|
+
},
|
|
324
|
+
},
|
|
325
|
+
"injured": {"type": "noul", "instructions": "Was anyone injured?"},
|
|
326
|
+
"severity": {
|
|
327
|
+
"type": "score",
|
|
328
|
+
"instructions": "Rate the reported outcome.",
|
|
329
|
+
"criteria": ["No injury", "Minor injury", "Serious injury"],
|
|
330
|
+
},
|
|
331
|
+
},
|
|
332
|
+
max_retries=3,
|
|
333
|
+
timeout=30,
|
|
334
|
+
)
|
|
335
|
+
answer = result.answers["category"]
|
|
336
|
+
if isinstance(answer, ChoiceAnswer):
|
|
337
|
+
print(answer.choice, answer.probabilities, answer.confidence)
|
|
338
|
+
print(result.model, result.usage.input_tokens, result.usage.output_tokens)
|
|
339
|
+
|
|
340
|
+
asyncio.run(main())
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
`DecisionResult` contains typed `ChoiceAnswer`, `NoulAnswer` and `ScoreAnswer`
|
|
344
|
+
objects, a resolved `model`, `DecisionUsage`, an optional `request_id`, and the
|
|
345
|
+
complete provider JSON in `raw_response`. Choice/score probabilities, confidence
|
|
346
|
+
and score legends are preserved when supplied; absent optional fields are `None`.
|
|
347
|
+
Noul is a probability from 0 to 1, not a boolean. Score is a number between zero
|
|
348
|
+
and the last criterion's index and can fall between levels. The library does not
|
|
349
|
+
choose thresholds, turn uncertainty into labels, or translate answers for EIDA.
|
|
350
|
+
|
|
351
|
+
Input and answer shapes, allowed labels, numeric ranges and token usage are
|
|
352
|
+
validated. Malformed successful responses raise `LLMResponseError`. HTTP failures
|
|
353
|
+
raise `LLMAPIError` (a subclass) with `status_code` and optional `request_id`;
|
|
354
|
+
error messages exclude provider response bodies. Inspect `raw_response` carefully
|
|
355
|
+
before logging it because provider extensions can contain sensitive information.
|
|
356
|
+
|
|
357
|
+
Decision calls retry transport failures, timeouts and HTTP 408, 429, 500, 502,
|
|
358
|
+
503, 504 and 529, honoring `Retry-After` and `retry-after-ms` when valid. Other HTTP
|
|
359
|
+
failures and invalid answers are not retried. As with the existing library API,
|
|
360
|
+
`max_retries` means **total attempts**, including the first request; `1` disables
|
|
361
|
+
retries. `timeout` is the limit for each attempt, so retries and backoff can make
|
|
362
|
+
the complete call longer. Cancellation propagates to the caller.
|
|
363
|
+
|
|
364
|
+
System One clients reject generative calls, provider batch operations,
|
|
365
|
+
`model_batch`, and `default_reasoning_effort`. Use concurrent `decide_async()`
|
|
366
|
+
calls for application-level parallelism. Generative clients reject `decide_async()`.
|
|
367
|
+
|
|
368
|
+
All providers support `async with LLM(...)` and `await llm.aclose()` to release
|
|
369
|
+
their clients. Keep a client open across repeated calls to reuse connections.
|
|
370
|
+
|
|
371
|
+
---
|
|
372
|
+
|
|
373
|
+
### Batch request
|
|
374
|
+
|
|
375
|
+
```python
|
|
376
|
+
batch_line = llm.create_batch_request(
|
|
377
|
+
custom_id="test1",
|
|
378
|
+
messages=[{"role": "user", "content": "Give me a JSON object"}],
|
|
379
|
+
schema="return json"
|
|
380
|
+
)
|
|
381
|
+
|
|
382
|
+
batch_id = await llm.run_batch_process([batch_line], submission_id="batch123")
|
|
383
|
+
status = await llm.get_batch_status(batch_id)
|
|
384
|
+
print(status)
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
`create_batch_request` returns a dictionary. `run_batch_process` accepts these
|
|
388
|
+
dictionaries or JSONL strings that you already serialized. When `model_batch` is
|
|
389
|
+
omitted, OpenAI/Azure batches use `model`.
|
|
390
|
+
|
|
391
|
+
For Azure, set `model_batch` to a deployment that supports batch processing
|
|
392
|
+
(`GlobalBatch` or `DataZoneBatch`). Model and region availability can differ from
|
|
393
|
+
single-call deployments; see [Azure batch documentation](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/batch).
|
|
394
|
+
Azure supports submission, status checking, cancellation and result downloads
|
|
395
|
+
through the same client methods.
|
|
396
|
+
|
|
397
|
+
---
|
|
398
|
+
|
|
399
|
+
## API Key Management
|
|
400
|
+
|
|
401
|
+
Keep API keys **out of code**. Recommended environment variables:
|
|
402
|
+
|
|
403
|
+
| Provider | Standard env var | Existing env vars (still supported) |
|
|
404
|
+
|----------|------------------|-----------------------------------|
|
|
405
|
+
| OpenAI | `OPENAI_API_KEY` | `openaiAPIkey`, `openaiAPIKey` |
|
|
406
|
+
| Azure OpenAI | `AZURE_OPENAI_API_KEY` | `azure_openaiAPIkey`, `azure_openaiAPIKey` |
|
|
407
|
+
| Google Gemini | `GOOGLE_API_KEY` | `googleAPIkey`, `googleAPIKey` |
|
|
408
|
+
| Microsoft Foundry | `AZURE_FOUNDRY_API_KEY` | `azure_foundryAPIkey`, `azure_foundryAPIKey` |
|
|
409
|
+
| TypeSafe | `TYPESAFE_API_KEY` | `typesafeAPIkey`, `typesafeAPIKey` |
|
|
410
|
+
|
|
411
|
+
The 0.1.10 `APIKey` spelling takes precedence, followed by the older `APIkey`
|
|
412
|
+
spelling and then the standard variable. Azure credentials must come from your Azure resource.
|
|
413
|
+
|
|
414
|
+
You can also inject an in-memory key using the new keyword-only argument
|
|
415
|
+
`LLM(..., api_key=secret_value)`. This takes precedence over all environment
|
|
416
|
+
variables and works for existing providers too. Retrieve Key Vault secrets in
|
|
417
|
+
your application or notebook and pass the value; this package has no dependency
|
|
418
|
+
on a particular vault, Azure identity configuration or DataTeam filesystem path.
|
|
419
|
+
Do not store keys in configuration files or commit them to source control.
|
|
420
|
+
|
|
421
|
+
---
|
|
422
|
+
|
|
423
|
+
## Development
|
|
424
|
+
|
|
425
|
+
```bash
|
|
426
|
+
# Create a virtual environment
|
|
427
|
+
python -m venv .venv
|
|
428
|
+
source .venv/bin/activate # macOS/Linux
|
|
429
|
+
.venv\Scripts\activate # Windows
|
|
430
|
+
|
|
431
|
+
# Install dependencies
|
|
432
|
+
pip install -e .
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
Run the regression checks (simulated HTTP responses; no API credentials needed):
|
|
436
|
+
|
|
437
|
+
```bash
|
|
438
|
+
python -m unittest discover -s tests -v
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
---
|
|
442
|
+
|
|
443
|
+
## License
|
|
444
|
+
|
|
445
|
+
MIT License © 2026 Your Name
|