dot-inference 2.0.2__py3-none-any.whl

dot_inference-2.0.2.dist-info/METADATA

@@ -0,0 +1,351 @@
+Metadata-Version: 2.5
+Name: dot-inference
+Version: 2.0.2
+Summary: Unified async Python library for LLM inference and embeddings across multiple providers
+Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
+Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
+Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/issues
+Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
+License-Expression: AGPL-3.0-or-later
+License-File: LICENSE.md
+Keywords: embeddings,inference,llm,providers
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: <3.14,>=3.12
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typing-extensions>=4.16
+Provides-Extra: mistral
+Requires-Dist: mistralai>=2.0.0; extra == 'mistral'
+Description-Content-Type: text/markdown
+
+# dot-inference
+
+[![PyPI](https://img.shields.io/pypi/v/dot-inference)](https://pypi.org/project/dot-inference/)
+![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
+[![Licence: AGPL v3](https://img.shields.io/badge/licence-AGPL--3.0--or--later-blue)](LICENSE.md)
+[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/pipelines)
+
+**One async interface for LLM inference and embeddings, across every provider.**
+
+```python
+from dot_inference import InferenceSettings, Message, Role, get_llm_client
+
+client = get_llm_client(InferenceSettings())  # provider and model come from env
+response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
+print(response.content)
+```
+
+## Why dot-inference
+
+Switching LLM provider usually means rewriting call sites: each SDK has its own
+client, its own message shape, its own way of doing structured output, streaming
+and tool calls. Retries, cost accounting and rate limiting end up reinvented in
+every project.
+
+dot-inference puts one async interface in front of OpenAI, Mistral, OpenRouter
+and any OpenAI-compatible server. Changing provider is an environment variable,
+not a refactor. Retries honouring `Retry-After`, USD cost tracking, rate limiting
+and structured output come with it, and a model registry records what each model
+can actually do — so an unsupported `reasoning_effort` fails with a clear error
+instead of silently doing nothing.
+
+## Features
+
+- Multi-provider: OpenAI, Mistral, OpenRouter, and any OpenAI-compatible server (vLLM, SGLang, Ollama)
+- Structured outputs backed by Pydantic models
+- Rich event streaming: text, reasoning, incremental tool-call arguments, tool calls, usage and cost
+- Native tool calling with `tool_choice` control (`auto` / `required` / `none`)
+- Embeddings with automatic batching
+- Retries on transient errors, honouring `Retry-After`, with exponential backoff
+- Cost tracking in tokens and USD
+- Rate limiting through an async sliding window
+- YAML model registry carrying pricing and per-model capabilities
+- Per-model default provider resolution — pick a model, let the registry choose the provider
+- Configurable HTTP timeouts (`DOTI_GENERATION__TIMEOUT`, `DOTI_EMBEDDING__TIMEOUT`)
+- Mock clients for testing without API calls
+- Sync wrappers (`call_sync`, `embed_sync`) for non-async codebases
+
+## Installation
+
+```bash
+pip install dot-inference
+
+# With the native Mistral SDK:
+pip install 'dot-inference[mistral]'
+```
+
+Requires Python 3.12+.
+
+## Configuration
+
+Settings come from `DOTI_`-prefixed environment variables, using `__` as the
+nesting delimiter — or from an env file, modelled on `.env.template`:
+
+```bash
+DOTI_GENERATION__PROVIDER=openai
+DOTI_GENERATION__OPENAI__MODEL_NAME=gpt-5-mini
+DOTI_GENERATION__OPENAI__API_KEY=sk-...
+```
+
+API keys are held as `SecretStr`, so a settings object caught in a traceback or
+an error report never carries the key in clear text.
+
+## Quick start
+
+### From settings
+
+```python
+import asyncio
+from dot_inference import InferenceSettings, Message, Role, get_llm_client
+
+settings = InferenceSettings()  # reads from DOTI_* env vars
+client = get_llm_client(settings)
+
+messages = [Message(role=Role.USER, content="What is the height of the Eiffel Tower?")]
+response = asyncio.run(client.call(messages))
+# or synchronously:
+response = client.call_sync(messages)
+print(response.content)
+```
+
+### Explicit provider and model
+
+```python
+from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client
+
+settings = InferenceSettings()
+client = get_llm_client(settings, provider=Provider.OPENAI, model="gpt-5-mini")
+response = asyncio.run(client.call([Message(role=Role.USER, content="Hello!")]))
+```
+
+### Reasoning models
+
+Use `ReasoningEffort` to control thinking budget on reasoning models (e.g. `gpt-5*`, `o-series`):
+
+```python
+from dot_inference import (
+    InferenceSettings,
+    Message,
+    Provider,
+    ReasoningEffort,
+    Role,
+    get_llm_client,
+)
+
+settings = InferenceSettings()
+client = get_llm_client(
+    settings,
+    provider=Provider.OPENAI,
+    model="gpt-5-mini",
+    reasoning_effort=ReasoningEffort.HIGH,
+)
+response = asyncio.run(client.call([Message(role=Role.USER, content="Solve this step by step...")]))
+```
+
+### Structured output
+
+```python
+from pydantic import BaseModel
+from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client
+
+
+class Answer(BaseModel):
+    answer: str
+    confidence: float
+
+
+settings = InferenceSettings()
+client = get_llm_client(settings, provider=Provider.OPENAI, model="gpt-5-mini")
+response = asyncio.run(
+    client.call(
+        [Message(role=Role.USER, content="What is 2+2?")],
+        response_format=Answer,
+    )
+)
+print(response.structured_output)  # Answer(answer='4', confidence=1.0)
+```
+
+### Streaming
+
+`stream()` yields typed `StreamEvent` objects:
+
+| Event | Description |
+| ----- | ----------- |
+| `TextDelta` | A chunk of text content (`content: str`) |
+| `ReasoningDelta` | A chunk of reasoning text, when the provider exposes it |
+| `InputJsonDelta` | A chunk of tool-call argument JSON (`tool_call_index`, `tool_name`, `partial_json`) — emitted incrementally before `ToolCallDone` |
+| `ToolCallDone` | A fully assembled tool call (`tool_call: ToolCall`) |
+| `StreamUsage` | Token counts and cost (`prompt_tokens`, `completion_tokens`, `cost`) |
+| `StreamDone` | Terminal event carrying the final `LLMResponse` — identical to what `call()` returns |
+
+**Simple text streaming** — use `stream_text()` for plain `str` chunks:
+
+```python
+from dot_inference import InferenceSettings, Message, Role, get_llm_client
+
+settings = InferenceSettings()
+client = get_llm_client(settings)
+
+
+async def main():
+    async for chunk in client.stream_text([Message(role=Role.USER, content="Tell me a story")]):
+        print(chunk, end="", flush=True)
+
+
+asyncio.run(main())
+```
+
+**Rich event streaming** — consume all events for text + usage + tool calls:
+
+```python
+from dot_inference import (
+    InferenceSettings,
+    Message,
+    Role,
+    get_llm_client,
+    ToolDef,
+    TextDelta,
+    InputJsonDelta,
+    ToolCallDone,
+    StreamUsage,
+    StreamDone,
+)
+
+
+async def main():
+    settings = InferenceSettings()
+    client = get_llm_client(settings)
+
+    async for event in client.stream(
+        [Message(role=Role.USER, content="What's the weather in Paris?")],
+        tools=[ToolDef(name="get_weather", description="Get weather", parameters={...})],
+    ):
+        if isinstance(event, TextDelta):
+            print(event.content, end="", flush=True)
+        elif isinstance(event, InputJsonDelta):
+            print(event.partial_json, end="", flush=True)
+        elif isinstance(event, ToolCallDone):
+            print(f"Tool call: {event.tool_call.name}({event.tool_call.arguments})")
+        elif isinstance(event, StreamUsage):
+            print(f"Tokens: {event.prompt_tokens} in / {event.completion_tokens} out")
+        elif isinstance(event, StreamDone):
+            # event.response is a full LLMResponse, works with CostTracker
+            tracker.record(event.response)
+
+
+asyncio.run(main())
+```
+
+### Embeddings
+
+```python
+from dot_inference import InferenceSettings, get_embedding_client
+
+settings = InferenceSettings()
+client = get_embedding_client(settings)
+vectors = asyncio.run(client.embed(["Hello world", "Bonjour le monde"]))
+print(len(vectors), len(vectors[0]))  # 2, 1536
+```
+
+### Cost tracking
+
+```python
+from dot_inference import CostTracker, InferenceSettings, Message, Provider, Role, get_llm_client
+
+settings = InferenceSettings()
+tracker = CostTracker()
+client = get_llm_client(settings, provider=Provider.OPENROUTER, model="openai/gpt-5-mini")
+
+response = asyncio.run(client.call([Message(role=Role.USER, content="Hi")]))
+tracker.record(response)
+print(f"Tokens: {tracker.total_tokens}, Cost: ${tracker.total_cost:.4f}")
+```
+
+### Retries
+
+Transient provider errors (5xx, rate limits, timeouts) are retried automatically with exponential backoff. When the provider returns a `Retry-After` header, the wait is honored (clamped to 60s). Client-side errors (400, 401, 403, 404, 422) are not retried.
+
+Attach `on_retry` to observe retry events — useful for UI feedback:
+
+```python
+client = get_llm_client(settings)
+client.on_retry = lambda attempt, max_attempts, err: print(f"Retry {attempt}/{max_attempts}: {err}")
+```
+
+## On-premise inference (vLLM, SGLang, Ollama)
+
+Point the `local` provider at any OpenAI-compatible endpoint:
+
+```shell
+export DOTI_GENERATION__PROVIDER=local
+export DOTI_GENERATION__LOCAL__MODEL_NAME=openai/gpt-oss-120b
+export DOTI_GENERATION__LOCAL__BASE_URL=http://localhost:8000/v1
+```
+
+```python
+from dot_inference import InferenceSettings, Message, Role, get_llm_client
+
+client = get_llm_client(InferenceSettings())
+response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
+print(response.content)
+```
+
+## Testing without API calls
+
+```python
+from dot_inference.mock import MockLLMClient, MockEmbeddingClient
+```
+
+Both return deterministic responses — including structured output built from
+your Pydantic schema — so tests never reach a provider.
+
+## Stability
+
+`dot-inference` follows semantic versioning: everything exported from the top-level
+package is covered, anything underscore-prefixed is internal and may change in
+any release. Public names are never removed without a deprecation period.
+
+```toml
+dependencies = ["dot-inference>=1.0,<2"]
+```
+
+See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.
+
+## Roadmap
+
+- [ ] Azure OpenAI provider
+- [ ] Logging integration (dot-logger)
+- [ ] Jinja templates for prompt management
+
+## Documentation
+
+| Document | Contents |
+|---|---|
+| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Internal design and module layout |
+| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
+| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
+| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
+| [CHANGELOG.md](CHANGELOG.md) | Release history |
+
+## Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the DCO
+sign-off requirement, the licensing terms that apply to contributions, and how to
+submit a change.
+
+## Licence
+
+Copyright (C) 2026 Kannon For Deep Tech (deepika)
+
+This software is distributed under the GNU Affero General Public License,
+version 3 or later — see [LICENSE.md](LICENSE.md).
+
+A commercial licence is available for use in proprietary environments.
+Contact: louis.letarnec@deepika.ai

dot_inference-2.0.2.dist-info/RECORD

@@ -0,0 +1,19 @@
+dot_inference/README.md,sha256=2kGOQCZE9pTmaU3Xy0WAKR2duCAGVVXD4VU0g8rzP5o,3361
+dot_inference/__init__.py,sha256=VyKDfo4A50ub_4AcJBK3Iuv6dbnXblvzEJiwHzqF2pM,1949
+dot_inference/_client.py,sha256=sDvleCbipeoEpNPDoMrR0vylPeTjn0VThuQWvkeY_fY,19537
+dot_inference/_cost_tracker.py,sha256=YWRaTa5vBvsIH5e0tDxB4wsowEgcqMTJyjzRSuL6R3M,2401
+dot_inference/_llm_registry.py,sha256=OzlMaw-Oum3-_J7zzSQQQiJgx9cgdEskWnY17cvKHu0,11928
+dot_inference/_local_check.py,sha256=dPEhChhmRTqS6GXJsqCTOfIoMJU6TJIGKmOBzuIc3dI,2807
+dot_inference/_mistral_compat.py,sha256=spYmHwvG61ILh-8TpxuIUKQpcyzC1o_54pluDGKDLkg,19636
+dot_inference/_openai_compat.py,sha256=BNTsMixgDdIKZDOJ-mY3j35Eybf9F5jfRhetYWHuul4,21576
+dot_inference/_rate_limit.py,sha256=RSJtBdBYw1wFI2WdpPi14mIAHXLm4aXyF1nwo5G81n8,2081
+dot_inference/_retry.py,sha256=npyEmZqZFScxoDlJDf_c9JsoGERMjRIv0eq8e2MXvQY,2355
+dot_inference/_sync.py,sha256=AGiGc_Wbqiga75oYqtgjv24WUlTD_B1ugCV8horYK4I,1554
+dot_inference/_types.py,sha256=B_IS8CRFB0eI9TRRkEw1CszRW6_9kuSLzHSR2G_HZ3E,7001
+dot_inference/mock.py,sha256=7ksDCjYZ7RrZf1kEtI_gcox7_cGdPFPUJ0pEjIY8_uU,5120
+dot_inference/models.yaml,sha256=-dugBbjbErEZ6PCvQcRGDaIiaUeS08ahvyEt1n4Xw9E,16250
+dot_inference/settings.py,sha256=_biGkq-CSFEQXuA_X9ELjED__qjI_PC15fOWrjEMOzw,4641
+dot_inference-2.0.2.dist-info/METADATA,sha256=EiHVFEULCWIymQY6H78xqkg0BWegUBiuJm37w1IeBWg,11900
+dot_inference-2.0.2.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+dot_inference-2.0.2.dist-info/licenses/LICENSE.md,sha256=ADUqsZhl4juwq34PRTMiBqumpm11s_PMli_dZQjWPqQ,34260
+dot_inference-2.0.2.dist-info/RECORD,,

dot_inference-2.0.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any