lm-deluge 0.0.3__tar.gz

lm_deluge-0.0.3/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: lm_deluge
+Version: 0.0.3
+Summary: Python utility for using LLM API models.
+Author-email: Benjamin Anderson <ben@trytaylor.ai>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: python-dotenv
+Requires-Dist: json5
+Requires-Dist: PyYAML
+Requires-Dist: pandas
+Requires-Dist: aiohttp
+Requires-Dist: tiktoken
+Requires-Dist: xxhash
+Requires-Dist: tqdm
+Requires-Dist: google-auth
+Requires-Dist: requests-aws4auth
+Requires-Dist: pydantic
+Requires-Dist: bs4
+Requires-Dist: lxml
+Provides-Extra: image
+Requires-Dist: pdf2image; extra == "image"
+Requires-Dist: pillow; extra == "image"
+Provides-Extra: pdf
+Requires-Dist: pdf2image; extra == "pdf"
+Requires-Dist: pymupdf; extra == "pdf"
+Provides-Extra: translate
+Requires-Dist: fasttext-wheel; extra == "translate"
+Requires-Dist: fasttext-langdetect; extra == "translate"
+Provides-Extra: full
+Requires-Dist: pillow; extra == "full"
+Requires-Dist: pdf2image; extra == "full"
+Requires-Dist: pymupdf; extra == "full"
+Requires-Dist: fasttext-wheel; extra == "full"
+Requires-Dist: fasttext-langdetect; extra == "full"
+
+# lm_deluge
+
+`lm_deluge` is a lightweight helper library for talking to large language model APIs.  It wraps several providers under a single interface, handles rate limiting, and exposes a few useful utilities for common NLP tasks.
+
+## Features
+
+- **Unified client** – send prompts to OpenAI‑compatible models, Anthropic, Cohere and Vertex hosted Claude models using the same API.
+- **Async or sync** – process prompts concurrently with `process_prompts_async` or run them synchronously with `process_prompts_sync`.
+- **Spray across providers** – configure multiple model names with weighting so requests are distributed across different providers.
+- **Caching** – optional LevelDB, SQLite or custom caches to avoid duplicate calls.
+- **Embeddings and reranking** – helper functions for embedding text and reranking documents via Cohere/OpenAI endpoints.
+- **Built‑in tools** – simple `extract`, `translate` and `score_llm` helpers for common patterns.
+
+## Installation
+
+```bash
+pip install lm_deluge
+```
+
+The package relies on environment variables for API keys.  Typical variables include `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`, `META_API_KEY` (for Llama) and `GOOGLE_APPLICATION_CREDENTIALS` for Vertex.
+
+## Quickstart
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic(
+    model=["gpt-4o-mini"],    # any model id from lm_deluge.models.registry
+    temperature=0.2,
+    max_new_tokens=256,
+)
+
+resp = client.process_prompts_sync(["Hello, world!"])  # returns list[APIResponse]
+print(resp[0].completion)
+```
+
+### Asynchronous usage
+
+```python
+import asyncio
+
+async def main():
+    responses = await client.process_prompts_async(
+        ["an async call"],
+        return_completions_only=True,
+    )
+    print(responses[0])
+
+asyncio.run(main())
+```
+
+### Distributing requests across models
+
+You can provide multiple `model_names` and optional `model_weights` when creating an `LLMClient`.  Each prompt will be sent to one of the models based on those weights.
+
+```python
+client = LLMClient(
+    model_names=["gpt-4o-mini", "claude-haiku-anthropic"],
+    model_weights="rate_limit",        # or a list like [0.7, 0.3]
+    max_requests_per_minute=5000,
+    max_tokens_per_minute=1_000_000,
+    max_concurrent_requests=100,
+)
+```
+
+### Provider specific notes
+
+- **OpenAI and compatible providers** – set `OPENAI_API_KEY`.  Model ids in the registry include OpenAI models as well as Meta Llama, Grok and many others that expose OpenAI style APIs.
+- **Anthropic** – set `ANTHROPIC_API_KEY`.  Use model ids such as `claude-haiku-anthropic` or `claude-sonnet-anthropic`.
+- **Cohere** – set `COHERE_API_KEY`.  Models like `command-r` are available.
+- **Vertex Claude** – set `GOOGLE_APPLICATION_CREDENTIALS` and `PROJECT_ID`.  Use a model id such as `claude-sonnet-vertex`.
+
+The [models.py](src/lm_deluge/models.py) file lists every supported model and the required environment variable.
+
+## Built‑in tools
+
+The `lm_deluge.llm_tools` package exposes a few helper functions:
+
+- `extract` – structure text or images into a Pydantic model based on a schema.
+- `translate` – translate a list of strings to English if needed.
+- `score_llm` – simple yes/no style scoring with optional log probability output.
+
+Embeddings (`embed.embed_parallel_async`) and document reranking (`rerank.rerank_parallel_async`) are also provided.
+
+## Caching results
+
+`lm_deluge.cache` includes LevelDB, SQLite and custom dictionary based caches.  Pass an instance via `LLMClient(..., cache=my_cache)` and previously seen prompts will not be re‑sent.
+
+## Development notes
+
+Models and costs are defined in [src/lm_deluge/models.py](src/lm_deluge/models.py).  Conversations are built using the `Conversation` and `Message` helpers in [src/lm_deluge/prompt.py](src/lm_deluge/prompt.py), which also support images.

lm_deluge-0.0.3/README.md

@@ -0,0 +1,91 @@
+# lm_deluge
+
+`lm_deluge` is a lightweight helper library for talking to large language model APIs.  It wraps several providers under a single interface, handles rate limiting, and exposes a few useful utilities for common NLP tasks.
+
+## Features
+
+- **Unified client** – send prompts to OpenAI‑compatible models, Anthropic, Cohere and Vertex hosted Claude models using the same API.
+- **Async or sync** – process prompts concurrently with `process_prompts_async` or run them synchronously with `process_prompts_sync`.
+- **Spray across providers** – configure multiple model names with weighting so requests are distributed across different providers.
+- **Caching** – optional LevelDB, SQLite or custom caches to avoid duplicate calls.
+- **Embeddings and reranking** – helper functions for embedding text and reranking documents via Cohere/OpenAI endpoints.
+- **Built‑in tools** – simple `extract`, `translate` and `score_llm` helpers for common patterns.
+
+## Installation
+
+```bash
+pip install lm_deluge
+```
+
+The package relies on environment variables for API keys.  Typical variables include `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `COHERE_API_KEY`, `META_API_KEY` (for Llama) and `GOOGLE_APPLICATION_CREDENTIALS` for Vertex.
+
+## Quickstart
+
+```python
+from lm_deluge import LLMClient
+
+client = LLMClient.basic(
+    model=["gpt-4o-mini"],    # any model id from lm_deluge.models.registry
+    temperature=0.2,
+    max_new_tokens=256,
+)
+
+resp = client.process_prompts_sync(["Hello, world!"])  # returns list[APIResponse]
+print(resp[0].completion)
+```
+
+### Asynchronous usage
+
+```python
+import asyncio
+
+async def main():
+    responses = await client.process_prompts_async(
+        ["an async call"],
+        return_completions_only=True,
+    )
+    print(responses[0])
+
+asyncio.run(main())
+```
+
+### Distributing requests across models
+
+You can provide multiple `model_names` and optional `model_weights` when creating an `LLMClient`.  Each prompt will be sent to one of the models based on those weights.
+
+```python
+client = LLMClient(
+    model_names=["gpt-4o-mini", "claude-haiku-anthropic"],
+    model_weights="rate_limit",        # or a list like [0.7, 0.3]
+    max_requests_per_minute=5000,
+    max_tokens_per_minute=1_000_000,
+    max_concurrent_requests=100,
+)
+```
+
+### Provider specific notes
+
+- **OpenAI and compatible providers** – set `OPENAI_API_KEY`.  Model ids in the registry include OpenAI models as well as Meta Llama, Grok and many others that expose OpenAI style APIs.
+- **Anthropic** – set `ANTHROPIC_API_KEY`.  Use model ids such as `claude-haiku-anthropic` or `claude-sonnet-anthropic`.
+- **Cohere** – set `COHERE_API_KEY`.  Models like `command-r` are available.
+- **Vertex Claude** – set `GOOGLE_APPLICATION_CREDENTIALS` and `PROJECT_ID`.  Use a model id such as `claude-sonnet-vertex`.
+
+The [models.py](src/lm_deluge/models.py) file lists every supported model and the required environment variable.
+
+## Built‑in tools
+
+The `lm_deluge.llm_tools` package exposes a few helper functions:
+
+- `extract` – structure text or images into a Pydantic model based on a schema.
+- `translate` – translate a list of strings to English if needed.
+- `score_llm` – simple yes/no style scoring with optional log probability output.
+
+Embeddings (`embed.embed_parallel_async`) and document reranking (`rerank.rerank_parallel_async`) are also provided.
+
+## Caching results
+
+`lm_deluge.cache` includes LevelDB, SQLite and custom dictionary based caches.  Pass an instance via `LLMClient(..., cache=my_cache)` and previously seen prompts will not be re‑sent.
+
+## Development notes
+
+Models and costs are defined in [src/lm_deluge/models.py](src/lm_deluge/models.py).  Conversations are built using the `Conversation` and `Message` helpers in [src/lm_deluge/prompt.py](src/lm_deluge/prompt.py), which also support images.

lm_deluge-0.0.3/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools", "wheel"]
+
+[project]
+name = "lm_deluge"
+version = "0.0.3"
+authors = [{ name = "Benjamin Anderson", email = "ben@trytaylor.ai" }]
+description = "Python utility for using LLM API models."
+readme = "README.md"
+requires-python = ">=3.9"
+keywords = []
+license = { text = "" }
+classifiers = []
+dependencies = [
+    "python-dotenv",
+    "json5",
+    "PyYAML",
+    "pandas",
+    "aiohttp",
+    "tiktoken",
+    "xxhash",
+    "tqdm",
+    "google-auth",
+    "requests-aws4auth",
+    "pydantic",
+    "bs4",
+    "lxml",
+]
+
+[project.optional-dependencies]
+image = ["pdf2image", "pillow"]
+pdf = ["pdf2image", "pymupdf"]
+translate = ["fasttext-wheel", "fasttext-langdetect"]
+full = [
+    "pillow",
+    "pdf2image",
+    "pymupdf",
+    "fasttext-wheel",
+    "fasttext-langdetect",
+]

lm_deluge-0.0.3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lm_deluge-0.0.3/src/lm_deluge/__init__.py

@@ -0,0 +1,6 @@
+from .client import LLMClient, SamplingParams, APIResponse
+import dotenv
+
+dotenv.load_dotenv()
+
+__all__ = ["LLMClient", "SamplingParams", "APIResponse"]

lm_deluge-0.0.3/src/lm_deluge/api_requests/__init__.py

@@ -0,0 +1,3 @@
+from .base import create_api_request
+
+__all__ = ["create_api_request"]

lm_deluge-0.0.3/src/lm_deluge/api_requests/anthropic.py

@@ -0,0 +1,177 @@
+import asyncio
+from aiohttp import ClientResponse
+import json
+import os
+import warnings
+import time
+from tqdm import tqdm
+from typing import Optional, Callable
+
+from lm_deluge.prompt import Conversation
+from .base import APIRequestBase, APIResponse
+
+from ..tracker import StatusTracker
+from ..sampling_params import SamplingParams
+from ..models import APIModel
+
+
+class AnthropicRequest(APIRequestBase):
+    def __init__(
+        self,
+        task_id: int,
+        # should always be 'role', 'content' keys.
+        # internal logic should handle translating to specific API format
+        model_name: str,  # must correspond to registry
+        prompt: Conversation,
+        attempts_left: int,
+        status_tracker: StatusTracker,
+        retry_queue: asyncio.Queue,
+        results_arr: list,
+        request_timeout: int = 30,
+        sampling_params: SamplingParams = SamplingParams(),
+        pbar: Optional[tqdm] = None,
+        callback: Optional[Callable] = None,
+        debug: bool = False,
+        # for retries
+        all_model_names: list[str] | None = None,
+        all_sampling_params: list[SamplingParams] | None = None,
+    ):
+        super().__init__(
+            task_id=task_id,
+            model_name=model_name,
+            prompt=prompt,
+            attempts_left=attempts_left,
+            status_tracker=status_tracker,
+            retry_queue=retry_queue,
+            results_arr=results_arr,
+            request_timeout=request_timeout,
+            sampling_params=sampling_params,
+            pbar=pbar,
+            callback=callback,
+            debug=debug,
+            all_model_names=all_model_names,
+            all_sampling_params=all_sampling_params,
+        )
+        self.model = APIModel.from_registry(model_name)
+        self.url = f"{self.model.api_base}/messages"
+
+        self.system_message, messages = prompt.to_anthropic()
+        self.request_header = {
+            "x-api-key": os.getenv(self.model.api_key_env_var),
+            "anthropic-version": "2023-06-01",
+            "content-type": "application/json",
+        }
+
+        self.request_json = {
+            "model": self.model.name,
+            "messages": messages,
+            "temperature": self.sampling_params.temperature,
+            "top_p": self.sampling_params.top_p,
+            "max_tokens": self.sampling_params.max_new_tokens,
+        }
+        # handle thinking
+        if self.model.reasoning_model:
+            if sampling_params.reasoning_effort:
+                # translate reasoning effort of low, medium, high to budget tokens
+                budget = {"low": 1024, "medium": 4096, "high": 16384}.get(
+                    sampling_params.reasoning_effort
+                )
+                self.request_json["thinking"] = {
+                    "type": "enabled",
+                    "budget_tokens": budget,
+                }
+                self.request_json.pop("top_p")
+                self.request_json["temperature"] = 1.0
+                self.request_json["max_tokens"] += (
+                    budget  # assume max tokens is max completion tokens
+                )
+            else:
+                # no thinking
+                self.request_json["thinking"] = {"type": "disabled"}
+        else:
+            if sampling_params.reasoning_effort:
+                warnings.warn(
+                    f"Ignoring reasoning_effort param for non-reasoning model: {model_name}"
+                )
+        if self.system_message is not None:
+            self.request_json["system"] = self.system_message
+
+        # print("request data:", self.request_json)
+
+    async def handle_response(self, http_response: ClientResponse) -> APIResponse:
+        is_error = False
+        error_message = None
+        thinking = None
+        completion = None
+        input_tokens = None
+        output_tokens = None
+        status_code = http_response.status
+        mimetype = http_response.headers.get("Content-Type", None)
+        rate_limits = {}
+        for header in [
+            "anthropic-ratelimit-requests-limit",
+            "anthropic-ratelimit-requests-remaining",
+            "anthropic-ratelimit-requests-reset",
+            "anthropic-ratelimit-tokens-limit",
+            "anthropic-ratelimit-tokens-remaining",
+            "anthropic-ratelimit-tokens-reset",
+        ]:
+            rate_limits[header] = http_response.headers.get(header, None)
+        if self.debug:
+            print(f"Rate limits: {rate_limits}")
+        if status_code >= 200 and status_code < 300:
+            try:
+                data = await http_response.json()
+                print("response data:", data)
+                content = data["content"]  # [0]["text"]
+                print("content is length", len(content))
+                for item in content:
+                    if item["type"] == "text":
+                        completion = item["text"]
+                    elif item["type"] == "thinking":
+                        thinking = item["thinking"]
+                    elif item["type"] == "tool_use":
+                        continue  # TODO: implement and report tool use
+                input_tokens = data["usage"]["input_tokens"]
+                output_tokens = data["usage"]["output_tokens"]
+            except Exception as e:
+                is_error = True
+                error_message = (
+                    f"Error calling .json() on response w/ status {status_code}: {e}"
+                )
+        elif mimetype and "json" in mimetype.lower():
+            is_error = True  # expected status is 200, otherwise it's an error
+            data = await http_response.json()
+            error_message = json.dumps(data)
+
+        else:
+            is_error = True
+            text = await http_response.text()
+            error_message = text
+
+        # handle special kinds of errors. TODO: make sure these are correct for anthropic
+        if is_error and error_message is not None:
+            if (
+                "rate limit" in error_message.lower()
+                or "overloaded" in error_message.lower()
+            ):
+                error_message += " (Rate limit error, triggering cooldown.)"
+                self.status_tracker.time_of_last_rate_limit_error = time.time()
+                self.status_tracker.num_rate_limit_errors += 1
+            if "context length" in error_message:
+                error_message += " (Context length exceeded, set retries to 0.)"
+                self.attempts_left = 0
+
+        return APIResponse(
+            id=self.task_id,
+            status_code=status_code,
+            is_error=is_error,
+            error_message=error_message,
+            prompt=self.prompt,
+            completion=completion,
+            thinking=thinking,
+            model_internal=self.model_name,
+            sampling_params=self.sampling_params,
+            input_tokens=input_tokens,
+            output_tokens=output_tokens,
+        )