lm-deluge 0.0.4__tar.gz → 0.0.6__tar.gz

lm_deluge-0.0.6/PKG-INFO

@@ -0,0 +1,170 @@
+Metadata-Version: 2.4
+Name: lm_deluge
+Version: 0.0.6
+Summary: Python utility for using LLM API models.
+Author-email: Benjamin Anderson <ben@trytaylor.ai>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: python-dotenv
+Requires-Dist: json5
+Requires-Dist: PyYAML
+Requires-Dist: pandas
+Requires-Dist: aiohttp
+Requires-Dist: tiktoken
+Requires-Dist: xxhash
+Requires-Dist: tqdm
+Requires-Dist: google-auth
+Requires-Dist: requests-aws4auth
+Requires-Dist: pydantic
+Requires-Dist: bs4
+Requires-Dist: lxml
+Requires-Dist: pdf2image
+Requires-Dist: pillow
+Provides-Extra: full
+Requires-Dist: pymupdf; extra == "full"
+Requires-Dist: fasttext-wheel; extra == "full"
+Requires-Dist: fasttext-langdetect; extra == "full"
+
+# lm_deluge
+
+`lm_deluge` is a lightweight helper library for maxing out your rate limits with LLM providers. It provides the following:
+
+- **Unified client** – Send prompts to all relevant models with a single client.
+- **Massive concurrency with throttling** – Set `max_tokens_per_minute` and `max_requests_per_minute` and let it fly. The client will process as many requests as possible while respecting rate limits and retrying failures.
+- **Spray across models/providers** – Configure a client with multiple models from any provider(s), and sampling weights. The client samples a model for each request.
+- **Caching** – Save completions in a local or distributed cache to avoid repeated LLM calls to process the same input.
+- **Convenient message constructor** – No more looking up how to build an Anthropic messages list with images. Our `Conversation` and `Message` classes work great with our client or with the `openai` and `anthropic` packages.
+- **Sync and async APIs** – Use the client from sync or async code.
+
+**STREAMING IS NOT IN SCOPE.** There are plenty of packages that let you stream chat completions across providers. The sole purpose of this package is to do very fast batch inference using APIs. Sorry!
+
+## Installation
+
+```bash
+pip install lm-deluge
+```
+
+There are optional goodies. If you want support for PDFs and language-detection via FastText:
+```bash
+pip install "lm-deluge[full]"
+```
+
+The package relies on environment variables for API keys. Typical variables include `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`, `META_API_KEY`, and `GOOGLE_API_KEY`. `LLMClient` will automatically load the `.env` file when imported; we recommend using that to set the environment variables.
+
+## Quickstart
+
+The easiest way to get started is with the `.basic` constructor. This uses sensible default arguments for rate limits and sampling parameters so that you don't have to provide a ton of arguments.
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic("gpt-4o-mini")
+resps = client.process_prompts_sync(["Hello, world!"])
+print(resp[0].completion)
+```
+
+## Spraying Across Models
+
+To distribute your requests across models, just provide a list of more than one model to the constructor. The rate limits for the client apply to the client as a whole, not per-model, so you may want to increase them:
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic(
+    ["gpt-4o-mini", "claude-haiku-anthropic"],
+    max_requests_per_minute=10_000
+)
+resps = client.process_prompts_sync(
+    ["Hello, ChatGPT!", "Hello, Claude!"]
+)
+print(resp[0].completion)
+```
+
+## Configuration
+
+API calls can be customized in a few ways.
+
+1. **Sampling Parameters.** This determines things like structured outputs, maximum completion tokens, nucleus sampling, etc. Provide a custom `SamplingParams` to the `LLMClient` to set temperature, top_p, json_mode, max_new_tokens, and/or reasoning_effort.
+
+You can pass 1 `SamplingParams` to use for all models, or a list of `SamplingParams` that's the same length as the list of models. You can also pass many of these arguments directly to `LLMClient.basic` so you don't have to construct an entire `SamplingParams` object.
+
+
+2. **Arguments to LLMClient.** This is where you set request timeout, rate limits, model name(s), model weight(s) for distributing requests across models, retries, and caching.
+3. **Arguments to process_prompts.** Per-call, you can set verbosity, whether to display progress, and whether to return just completions (rather than the full APIResponse object).
+
+Putting it all together:
+
+```python
+from lm_deluge import LLMClient, SamplingParams
+
+client = LLMClient(
+    "gpt-4",
+    max_requests_per_minute=100,
+    max_tokens_per_minute=100_000,
+    max_concurrent_requests=500,
+    sampling_params=SamplingParams(temperature=0.5, max_new_tokens=30)
+)
+
+await client.process_prompts_async(
+    ["What is the capital of Mars?"],
+    show_progress=False,
+    return_completions_only=True
+)
+```
+
+## Multi-Turn Conversations
+
+Constructing conversations to pass to models is notoriously annoying. Each provider has a slightly different way of defining a list of messages, and with the introduction of images/multi-part messages it's only gotten worse. We provide convenience constructors so you don't have to remember all that stuff.
+
+```python
+from lm_deluge import Message, Conversation
+
+prompt = Conversation.system("You are a helpful assistant.").add(
+    Message.user("What's in this image?").add_image("tests/image.jpg")
+)
+
+client = LLMClient.basic("gpt-4.1-mini")
+resps = client.process_prompts_sync([prompt])
+```
+
+This just works. Images can be local images on disk, URLs, bytes, base64 data URLs... go wild. You can use `Conversation.to_openai` or `Conversation.to_anthropic` to format your messages for the OpenAI or Anthropic clients directly.
+
+## Caching
+
+`lm_deluge.cache` includes LevelDB, SQLite and custom dictionary based caches.  Pass an instance via `LLMClient(..., cache=my_cache)` and previously seen prompts will not be re‑sent across different `process_prompts_[...]` calls.
+
+**IMPORTANT:** Caching does not currently work for prompts in the SAME batch. That is, if you call `process_prompts_sync` with the same prompt 100 times, there will be 0 cache hits. If you call `process_prompts_sync` a *second* time with those same 100 prompts, all 100 will be cache hits. The cache is intended to be persistent and help you save costs across many invocations, but it can't help with a single batch-inference session (yet!).
+
+## Asynchronous Client
+Use this in asynchronous code, or in a Jupyter notebook. If you try to use the sync client in a Jupyter notebook, you'll have to use `nest-asyncio`, because internally the sync client uses async code. Don't do it! Just use the async client!
+
+```python
+import asyncio
+
+async def main():
+    responses = await client.process_prompts_async(
+        ["an async call"],
+        return_completions_only=True,
+    )
+    print(responses[0])
+
+asyncio.run(main())
+```
+
+## Available Models
+
+We support all models in `src/lm_deluge/models.py`. An older version of this client supported Bedrock and Vertex. We plan to re-implement Bedrock support (our previous support was spotty and we need to figure out cross-region inference in order to support the newest Claude models). Vertex support is not currently planned, since Google allows you to connect your Vertex account to AI Studio, and Vertex authentication is a huge pain (requires service account credentials, etc.)
+
+## Feature Support
+
+We support structured outputs via `json_mode` parameter provided to `SamplingParams`. Structured outputs with a schema are planned. Reasoning models are supported via the `reasoning_effort` parameter, which is translated to a thinking budget for Claude/Gemini. Image models are supported. We don't support tool use yet, but support is planned (keep an eye out for a unified tool definition spec that works for all models!). We support logprobs for OpenAI models that return them via the `logprobs` argument to the `LLMClient`.
+
+## Built‑in tools
+
+The `lm_deluge.llm_tools` package exposes a few helper functions:
+
+- `extract` – structure text or images into a Pydantic model based on a schema.
+- `translate` – translate a list of strings to English.
+- `score_llm` – simple yes/no style scoring with optional log probability output.
+
+Experimental embeddings (`embed.embed_parallel_async`) and document reranking (`rerank.rerank_parallel_async`) clients are also provided.

lm_deluge-0.0.6/README.md

@@ -0,0 +1,143 @@
+# lm_deluge
+
+`lm_deluge` is a lightweight helper library for maxing out your rate limits with LLM providers. It provides the following:
+
+- **Unified client** – Send prompts to all relevant models with a single client.
+- **Massive concurrency with throttling** – Set `max_tokens_per_minute` and `max_requests_per_minute` and let it fly. The client will process as many requests as possible while respecting rate limits and retrying failures.
+- **Spray across models/providers** – Configure a client with multiple models from any provider(s), and sampling weights. The client samples a model for each request.
+- **Caching** – Save completions in a local or distributed cache to avoid repeated LLM calls to process the same input.
+- **Convenient message constructor** – No more looking up how to build an Anthropic messages list with images. Our `Conversation` and `Message` classes work great with our client or with the `openai` and `anthropic` packages.
+- **Sync and async APIs** – Use the client from sync or async code.
+
+**STREAMING IS NOT IN SCOPE.** There are plenty of packages that let you stream chat completions across providers. The sole purpose of this package is to do very fast batch inference using APIs. Sorry!
+
+## Installation
+
+```bash
+pip install lm-deluge
+```
+
+There are optional goodies. If you want support for PDFs and language-detection via FastText:
+```bash
+pip install "lm-deluge[full]"
+```
+
+The package relies on environment variables for API keys. Typical variables include `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`, `META_API_KEY`, and `GOOGLE_API_KEY`. `LLMClient` will automatically load the `.env` file when imported; we recommend using that to set the environment variables.
+
+## Quickstart
+
+The easiest way to get started is with the `.basic` constructor. This uses sensible default arguments for rate limits and sampling parameters so that you don't have to provide a ton of arguments.
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic("gpt-4o-mini")
+resps = client.process_prompts_sync(["Hello, world!"])
+print(resp[0].completion)
+```
+
+## Spraying Across Models
+
+To distribute your requests across models, just provide a list of more than one model to the constructor. The rate limits for the client apply to the client as a whole, not per-model, so you may want to increase them:
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic(
+    ["gpt-4o-mini", "claude-haiku-anthropic"],
+    max_requests_per_minute=10_000
+)
+resps = client.process_prompts_sync(
+    ["Hello, ChatGPT!", "Hello, Claude!"]
+)
+print(resp[0].completion)
+```
+
+## Configuration
+
+API calls can be customized in a few ways.
+
+1. **Sampling Parameters.** This determines things like structured outputs, maximum completion tokens, nucleus sampling, etc. Provide a custom `SamplingParams` to the `LLMClient` to set temperature, top_p, json_mode, max_new_tokens, and/or reasoning_effort.
+
+You can pass 1 `SamplingParams` to use for all models, or a list of `SamplingParams` that's the same length as the list of models. You can also pass many of these arguments directly to `LLMClient.basic` so you don't have to construct an entire `SamplingParams` object.
+
+
+2. **Arguments to LLMClient.** This is where you set request timeout, rate limits, model name(s), model weight(s) for distributing requests across models, retries, and caching.
+3. **Arguments to process_prompts.** Per-call, you can set verbosity, whether to display progress, and whether to return just completions (rather than the full APIResponse object).
+
+Putting it all together:
+
+```python
+from lm_deluge import LLMClient, SamplingParams
+
+client = LLMClient(
+    "gpt-4",
+    max_requests_per_minute=100,
+    max_tokens_per_minute=100_000,
+    max_concurrent_requests=500,
+    sampling_params=SamplingParams(temperature=0.5, max_new_tokens=30)
+)
+
+await client.process_prompts_async(
+    ["What is the capital of Mars?"],
+    show_progress=False,
+    return_completions_only=True
+)
+```
+
+## Multi-Turn Conversations
+
+Constructing conversations to pass to models is notoriously annoying. Each provider has a slightly different way of defining a list of messages, and with the introduction of images/multi-part messages it's only gotten worse. We provide convenience constructors so you don't have to remember all that stuff.
+
+```python
+from lm_deluge import Message, Conversation
+
+prompt = Conversation.system("You are a helpful assistant.").add(
+    Message.user("What's in this image?").add_image("tests/image.jpg")
+)
+
+client = LLMClient.basic("gpt-4.1-mini")
+resps = client.process_prompts_sync([prompt])
+```
+
+This just works. Images can be local images on disk, URLs, bytes, base64 data URLs... go wild. You can use `Conversation.to_openai` or `Conversation.to_anthropic` to format your messages for the OpenAI or Anthropic clients directly.
+
+## Caching
+
+`lm_deluge.cache` includes LevelDB, SQLite and custom dictionary based caches.  Pass an instance via `LLMClient(..., cache=my_cache)` and previously seen prompts will not be re‑sent across different `process_prompts_[...]` calls.
+
+**IMPORTANT:** Caching does not currently work for prompts in the SAME batch. That is, if you call `process_prompts_sync` with the same prompt 100 times, there will be 0 cache hits. If you call `process_prompts_sync` a *second* time with those same 100 prompts, all 100 will be cache hits. The cache is intended to be persistent and help you save costs across many invocations, but it can't help with a single batch-inference session (yet!).
+
+## Asynchronous Client
+Use this in asynchronous code, or in a Jupyter notebook. If you try to use the sync client in a Jupyter notebook, you'll have to use `nest-asyncio`, because internally the sync client uses async code. Don't do it! Just use the async client!
+
+```python
+import asyncio
+
+async def main():
+    responses = await client.process_prompts_async(
+        ["an async call"],
+        return_completions_only=True,
+    )
+    print(responses[0])
+
+asyncio.run(main())
+```
+
+## Available Models
+
+We support all models in `src/lm_deluge/models.py`. An older version of this client supported Bedrock and Vertex. We plan to re-implement Bedrock support (our previous support was spotty and we need to figure out cross-region inference in order to support the newest Claude models). Vertex support is not currently planned, since Google allows you to connect your Vertex account to AI Studio, and Vertex authentication is a huge pain (requires service account credentials, etc.)
+
+## Feature Support
+
+We support structured outputs via `json_mode` parameter provided to `SamplingParams`. Structured outputs with a schema are planned. Reasoning models are supported via the `reasoning_effort` parameter, which is translated to a thinking budget for Claude/Gemini. Image models are supported. We don't support tool use yet, but support is planned (keep an eye out for a unified tool definition spec that works for all models!). We support logprobs for OpenAI models that return them via the `logprobs` argument to the `LLMClient`.
+
+## Built‑in tools
+
+The `lm_deluge.llm_tools` package exposes a few helper functions:
+
+- `extract` – structure text or images into a Pydantic model based on a schema.
+- `translate` – translate a list of strings to English.
+- `score_llm` – simple yes/no style scoring with optional log probability output.
+
+Experimental embeddings (`embed.embed_parallel_async`) and document reranking (`rerank.rerank_parallel_async`) clients are also provided.

{lm_deluge-0.0.4 → lm_deluge-0.0.6}/pyproject.toml

@@ -3,7 +3,7 @@ requires = ["setuptools", "wheel"]
 
 [project]
 name = "lm_deluge"
-version = "0.0.4"
+version = "0.0.6"
 authors = [{ name = "Benjamin Anderson", email = "ben@trytaylor.ai" }]
 description = "Python utility for using LLM API models."
 readme = "README.md"
@@ -25,15 +25,12 @@ dependencies = [
     "pydantic",
     "bs4",
     "lxml",
+    "pdf2image",
+    "pillow"
 ]
 
 [project.optional-dependencies]
-image = ["pdf2image", "pillow"]
-pdf = ["pdf2image", "pymupdf"]
-translate = ["fasttext-wheel", "fasttext-langdetect"]
 full = [
-    "pillow",
-    "pdf2image",
     "pymupdf",
     "fasttext-wheel",
     "fasttext-langdetect",

lm_deluge-0.0.6/src/lm_deluge/__init__.py

@@ -0,0 +1,7 @@
+from .client import LLMClient, SamplingParams, APIResponse
+from .prompt import Conversation, Message
+import dotenv
+
+dotenv.load_dotenv()
+
+__all__ = ["LLMClient", "SamplingParams", "APIResponse", "Conversation", "Message"]

{lm_deluge-0.0.4 → lm_deluge-0.0.6}/src/lm_deluge/api_requests/anthropic.py

@@ -119,9 +119,7 @@ class AnthropicRequest(APIRequestBase):
         if status_code >= 200 and status_code < 300:
             try:
                 data = await http_response.json()
-                print("response data:", data)
                 content = data["content"]  # [0]["text"]
-                print("content is length", len(content))
                 for item in content:
                     if item["type"] == "text":
                         completion = item["text"]

{lm_deluge-0.0.4 → lm_deluge-0.0.6}/src/lm_deluge/api_requests/base.py

@@ -41,6 +41,7 @@ class APIResponse:
     logprobs: list | None = None
     finish_reason: str | None = None  # make required later
     cost: float | None = None  # calculated automatically
+    cache_hit: bool = False  # manually set if true
     # set to true if is_error and should be retried with a different model
     retry_with_different_model: bool | None = False
     # set to true if should NOT retry with the same model (unrecoverable error)

lm_deluge-0.0.6/src/lm_deluge/api_requests/common.py

@@ -0,0 +1,9 @@
+from .openai import OpenAIRequest
+from .anthropic import AnthropicRequest
+from .mistral import MistralRequest
+
+CLASSES = {
+    "openai": OpenAIRequest,
+    "anthropic": AnthropicRequest,
+    "mistral": MistralRequest,
+}

lm_deluge-0.0.6/src/lm_deluge/api_requests/deprecated/cohere.py

@@ -0,0 +1,132 @@
+# # https://docs.cohere.com/reference/chat
+# # https://cohere.com/pricing
+# import asyncio
+# from aiohttp import ClientResponse
+# import json
+# import os
+# from tqdm import tqdm
+# from typing import Callable
+# from lm_deluge.prompt import Conversation
+# from .base import APIRequestBase, APIResponse
+
+# from ..tracker import StatusTracker
+# from ..sampling_params import SamplingParams
+# from ..models import APIModel
+
+
+# class CohereRequest(APIRequestBase):
+#     def __init__(
+#         self,
+#         task_id: int,
+#         # should always be 'role', 'content' keys.
+#         # internal logic should handle translating to specific API format
+#         model_name: str,  # must correspond to registry
+#         prompt: Conversation,
+#         attempts_left: int,
+#         status_tracker: StatusTracker,
+#         results_arr: list,
+#         retry_queue: asyncio.Queue,
+#         request_timeout: int = 30,
+#         sampling_params: SamplingParams = SamplingParams(),
+#         pbar: tqdm | None = None,
+#         callback: Callable | None = None,
+#         debug: bool = False,
+#         all_model_names: list[str] | None = None,
+#         all_sampling_params: list[SamplingParams] | None = None,
+#     ):
+#         super().__init__(
+#             task_id=task_id,
+#             model_name=model_name,
+#             prompt=prompt,
+#             attempts_left=attempts_left,
+#             status_tracker=status_tracker,
+#             retry_queue=retry_queue,
+#             results_arr=results_arr,
+#             request_timeout=request_timeout,
+#             sampling_params=sampling_params,
+#             pbar=pbar,
+#             callback=callback,
+#             debug=debug,
+#             all_model_names=all_model_names,
+#             all_sampling_params=all_sampling_params,
+#         )
+#         self.system_message = None
+#         self.last_user_message = None
+
+#         self.model = APIModel.from_registry(model_name)
+#         self.url = f"{self.model.api_base}/chat"
+#         messages = prompt.to_cohere()
+
+#         self.request_header = {
+#             "Authorization": f"bearer {os.getenv(self.model.api_key_env_var)}",
+#             "content-type": "application/json",
+#             "accept": "application/json",
+#         }
+
+#         self.request_json = {
+#             "model": self.model.name,
+#             "messages": messages,
+#             "temperature": sampling_params.temperature,
+#             "top_p": sampling_params.top_p,
+#             "max_tokens": sampling_params.max_new_tokens,
+#         }
+
+#     async def handle_response(self, http_response: ClientResponse) -> APIResponse:
+#         is_error = False
+#         error_message = None
+#         completion = None
+#         input_tokens = None
+#         output_tokens = None
+#         status_code = http_response.status
+#         mimetype = http_response.headers.get("Content-Type", None)
+#         if status_code >= 200 and status_code < 300:
+#             try:
+#                 data = await http_response.json()
+#             except Exception:
+#                 data = None
+#                 is_error = True
+#                 error_message = (
+#                     f"Error calling .json() on response w/ status {status_code}"
+#                 )
+#             if not is_error and isinstance(data, dict):
+#                 try:
+#                     completion = data["text"]
+#                     input_tokens = data["meta"]["billed_units"]["input_tokens"]
+#                     output_tokens = data["meta"]["billed_units"]["input_tokens"]
+#                 except Exception:
+#                     is_error = True
+#                     error_message = f"Error getting 'text' or 'meta' from {self.model.name} response."
+#         elif mimetype is not None and "json" in mimetype.lower():
+#             is_error = True  # expected status is 200, otherwise it's an error
+#             data = await http_response.json()
+#             error_message = json.dumps(data)
+
+#         else:
+#             is_error = True
+#             text = await http_response.text()
+#             error_message = text
+
+#         # handle special kinds of errors. TODO: make sure these are correct for anthropic
+#         if is_error and error_message is not None:
+#             if (
+#                 "rate limit" in error_message.lower()
+#                 or "overloaded" in error_message.lower()
+#             ):
+#                 error_message += " (Rate limit error, triggering cooldown.)"
+#                 self.status_tracker.rate_limit_exceeded()
+#             if "context length" in error_message:
+#                 error_message += " (Context length exceeded, set retries to 0.)"
+#                 self.attempts_left = 0
+
+#         return APIResponse(
+#             id=self.task_id,
+#             status_code=status_code,
+#             is_error=is_error,
+#             error_message=error_message,
+#             prompt=self.prompt,
+#             completion=completion,
+#             model_internal=self.model_name,
+#             sampling_params=self.sampling_params,
+#             input_tokens=input_tokens,
+#             output_tokens=output_tokens,
+#         )