lmnr 0.3.7__tar.gz → 0.4.1__tar.gz

lmnr-0.4.1/PKG-INFO

@@ -0,0 +1,214 @@
+Metadata-Version: 2.1
+Name: lmnr
+Version: 0.4.1
+Summary: Python SDK for Laminar AI
+License: Apache-2.0
+Author: lmnr.ai
+Requires-Python: >=3.9,<4.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: asyncio (>=3.4.3,<4.0.0)
+Requires-Dist: backoff (>=2.2.1,<3.0.0)
+Requires-Dist: pydantic (>=2.7.4,<3.0.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Requires-Dist: traceloop-sdk (>=0.29.2,<0.30.0)
+Description-Content-Type: text/markdown
+
+# Laminar Python
+
+OpenTelemetry log sender for [Laminar](https://github.com/lmnr-ai/lmnr) for Python code.
+
+ <a href="https://pypi.org/project/lmnr/"> ![PyPI - Version](https://img.shields.io/pypi/v/lmnr?label=lmnr&logo=pypi&logoColor=3775A9) </a>
+![PyPI - Downloads](https://img.shields.io/pypi/dm/lmnr)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/lmnr)
+
+
+
+## Quickstart
+```sh
+python3 -m venv .myenv
+source .myenv/bin/activate  # or use your favorite env management tool
+
+pip install lmnr
+```
+
+And the in your main Python file
+
+```python
+from lmnr import Laminar as L
+
+L.initialize(project_api_key="<LMNR_PROJECT_API_KEY>")
+```
+
+This will automatically instrument most of the LLM, Vector DB, and related
+calls with OpenTelemetry-compatible instrumentation.
+
+We rely on the amazing [OpenLLMetry](https://github.com/traceloop/openllmetry), open-source package
+by TraceLoop, to achieve that.
+
+### Project API key
+
+Get the key from the settings page of your Laminar project ([Learn more](https://docs.lmnr.ai/api-reference/introduction#authentication)).
+You can either pass it to `.initialize()` or set it to `.env` at the root of your package with the key `LMNR_PROJECT_API_KEY`.
+
+## Instrumentation
+
+In addition to automatic instrumentation, we provide a simple `@observe()` decorator, if you want more fine-grained tracing
+or to trace other functions.
+
+### Example
+
+```python
+import os
+from openai import OpenAI
+
+
+from lmnr import observe, Laminar as L
+L.initialize(project_api_key="<LMNR_PROJECT_API_KEY>")
+
+client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
+
+@observe()  # annotate all functions you want to trace
+def poem_writer(topic="turbulence"):
+    prompt = f"write a poem about {topic}"
+    response = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": prompt},
+        ],
+    )
+    poem = response.choices[0].message.content
+    return poem
+
+print(poem_writer(topic="laminar flow"))
+```
+
+
+## Sending events
+
+You can send events in two ways:
+- `.event(name, value)` – for a pre-defined event with one of possible values.
+- `.evaluate_event(name, evaluator, data)` – for an event that is evaluated by evaluator pipeline based on the data.
+
+Note that to run an evaluate event, you need to crate an evaluator pipeline and create a target version for it.
+
+Read our [docs](https://docs.lmnr.ai) to learn more about event types and how they are created and evaluated.
+
+### Example
+
+```python
+from lmnr import Laminar as L
+# ...
+poem = response.choices[0].message.content
+
+# this will register True or False value with Laminar
+L.event("topic alignment", topic in poem)
+
+# this will run the pipeline `check_wordy` with `poem` set as the value
+# of `text_input` node, and write the result as an event with name
+# "excessive_wordiness"
+L.evaluate_event("excessive_wordiness", "check_wordy", {"text_input": poem})
+```
+
+## Laminar pipelines as prompt chain managers
+
+You can create Laminar pipelines in the UI and manage chains of LLM calls there.
+
+After you are ready to use your pipeline in your code, deploy it in Laminar by selecting the target version for the pipeline.
+
+Once your pipeline target is set, you can call it from Python in just a few lines.
+
+Example use:
+
+```python
+from lmnr import Laminar as L
+
+L.initialize('<YOUR_PROJECT_API_KEY>')
+
+result = l.run(
+    pipeline = 'my_pipeline_name',
+    inputs = {'input_node_name': 'some_value'},
+    # all environment variables
+    env = {'OPENAI_API_KEY': 'sk-some-key'},
+)
+```
+
+Resulting in:
+
+```python
+>>> result
+PipelineRunResponse(
+    outputs={'output': {'value': [ChatMessage(role='user', content='hello')]}},
+    # useful to locate your trace
+    run_id='53b012d5-5759-48a6-a9c5-0011610e3669'
+)
+```
+
+## Running offline evaluations on your data
+
+You can evaluate your code with your own data and send it to Laminar using the `Evaluation` class.
+
+Evaluation takes in the following parameters:
+- `name` – the name of your evaluation. If no such evaluation exists in the project, it will be created. Otherwise, data will be pushed to the existing evaluation
+- `data` – an array of `EvaluationDatapoint` objects, where each `EvaluationDatapoint` has two keys: `target` and `data`, each containing a key-value object. Alternatively, you can pass in dictionaries, and we will instantiate `EvaluationDatapoint`s with pydantic if possible
+- `executor` – the logic you want to evaluate. This function must take `data` as the first argument, and produce any output. *
+- `evaluators` – evaluaton logic. List of functions that take output of executor as the first argument, `target` as the second argument and produce a numeric scores. Each function can produce either a single number or `dict[str, int|float]` of scores.
+
+\* If you already have the outputs of executors you want to evaluate, you can specify the executor as an identity function, that takes in `data` and returns only needed value(s) from it.
+
+### Example
+
+```python
+from openai import AsyncOpenAI
+import asyncio
+import os
+
+openai_client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
+
+async def get_capital(data):
+    country = data["country"]
+    response = await openai_client.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[
+            {"role": "system", "content": "You are a helpful assistant."},
+            {
+                "role": "user",
+                "content": f"What is the capital of {country}? Just name the "
+                "city and nothing else",
+            },
+        ],
+    )
+    return response.choices[0].message.content.strip()
+
+
+# Evaluation data
+data = [
+    {"data": {"country": "Canada"}, "target": {"capital": "Ottawa"}},
+    {"data": {"country": "Germany"}, "target": {"capital": "Berlin"}},
+    {"data": {"country": "Tanzania"}, "target": {"capital": "Dodoma"}},
+]
+
+
+def evaluator_A(output, target):
+    return 1 if output == target["capital"] else 0
+
+
+# Create an Evaluation instance
+e = Evaluation(
+    name="py-evaluation-async",
+    data=data,
+    executor=get_capital,
+    evaluators=[evaluator_A],
+    project_api_key=os.environ["LMNR_PROJECT_API_KEY"],
+)
+
+# Run the evaluation
+asyncio.run(e.run())
+```
+

lmnr-0.4.1/README.md

@@ -0,0 +1,192 @@
+# Laminar Python
+
+OpenTelemetry log sender for [Laminar](https://github.com/lmnr-ai/lmnr) for Python code.
+
+ <a href="https://pypi.org/project/lmnr/"> ![PyPI - Version](https://img.shields.io/pypi/v/lmnr?label=lmnr&logo=pypi&logoColor=3775A9) </a>
+![PyPI - Downloads](https://img.shields.io/pypi/dm/lmnr)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/lmnr)
+
+
+
+## Quickstart
+```sh
+python3 -m venv .myenv
+source .myenv/bin/activate  # or use your favorite env management tool
+
+pip install lmnr
+```
+
+And the in your main Python file
+
+```python
+from lmnr import Laminar as L
+
+L.initialize(project_api_key="<LMNR_PROJECT_API_KEY>")
+```
+
+This will automatically instrument most of the LLM, Vector DB, and related
+calls with OpenTelemetry-compatible instrumentation.
+
+We rely on the amazing [OpenLLMetry](https://github.com/traceloop/openllmetry), open-source package
+by TraceLoop, to achieve that.
+
+### Project API key
+
+Get the key from the settings page of your Laminar project ([Learn more](https://docs.lmnr.ai/api-reference/introduction#authentication)).
+You can either pass it to `.initialize()` or set it to `.env` at the root of your package with the key `LMNR_PROJECT_API_KEY`.
+
+## Instrumentation
+
+In addition to automatic instrumentation, we provide a simple `@observe()` decorator, if you want more fine-grained tracing
+or to trace other functions.
+
+### Example
+
+```python
+import os
+from openai import OpenAI
+
+
+from lmnr import observe, Laminar as L
+L.initialize(project_api_key="<LMNR_PROJECT_API_KEY>")
+
+client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
+
+@observe()  # annotate all functions you want to trace
+def poem_writer(topic="turbulence"):
+    prompt = f"write a poem about {topic}"
+    response = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": prompt},
+        ],
+    )
+    poem = response.choices[0].message.content
+    return poem
+
+print(poem_writer(topic="laminar flow"))
+```
+
+
+## Sending events
+
+You can send events in two ways:
+- `.event(name, value)` – for a pre-defined event with one of possible values.
+- `.evaluate_event(name, evaluator, data)` – for an event that is evaluated by evaluator pipeline based on the data.
+
+Note that to run an evaluate event, you need to crate an evaluator pipeline and create a target version for it.
+
+Read our [docs](https://docs.lmnr.ai) to learn more about event types and how they are created and evaluated.
+
+### Example
+
+```python
+from lmnr import Laminar as L
+# ...
+poem = response.choices[0].message.content
+
+# this will register True or False value with Laminar
+L.event("topic alignment", topic in poem)
+
+# this will run the pipeline `check_wordy` with `poem` set as the value
+# of `text_input` node, and write the result as an event with name
+# "excessive_wordiness"
+L.evaluate_event("excessive_wordiness", "check_wordy", {"text_input": poem})
+```
+
+## Laminar pipelines as prompt chain managers
+
+You can create Laminar pipelines in the UI and manage chains of LLM calls there.
+
+After you are ready to use your pipeline in your code, deploy it in Laminar by selecting the target version for the pipeline.
+
+Once your pipeline target is set, you can call it from Python in just a few lines.
+
+Example use:
+
+```python
+from lmnr import Laminar as L
+
+L.initialize('<YOUR_PROJECT_API_KEY>')
+
+result = l.run(
+    pipeline = 'my_pipeline_name',
+    inputs = {'input_node_name': 'some_value'},
+    # all environment variables
+    env = {'OPENAI_API_KEY': 'sk-some-key'},
+)
+```
+
+Resulting in:
+
+```python
+>>> result
+PipelineRunResponse(
+    outputs={'output': {'value': [ChatMessage(role='user', content='hello')]}},
+    # useful to locate your trace
+    run_id='53b012d5-5759-48a6-a9c5-0011610e3669'
+)
+```
+
+## Running offline evaluations on your data
+
+You can evaluate your code with your own data and send it to Laminar using the `Evaluation` class.
+
+Evaluation takes in the following parameters:
+- `name` – the name of your evaluation. If no such evaluation exists in the project, it will be created. Otherwise, data will be pushed to the existing evaluation
+- `data` – an array of `EvaluationDatapoint` objects, where each `EvaluationDatapoint` has two keys: `target` and `data`, each containing a key-value object. Alternatively, you can pass in dictionaries, and we will instantiate `EvaluationDatapoint`s with pydantic if possible
+- `executor` – the logic you want to evaluate. This function must take `data` as the first argument, and produce any output. *
+- `evaluators` – evaluaton logic. List of functions that take output of executor as the first argument, `target` as the second argument and produce a numeric scores. Each function can produce either a single number or `dict[str, int|float]` of scores.
+
+\* If you already have the outputs of executors you want to evaluate, you can specify the executor as an identity function, that takes in `data` and returns only needed value(s) from it.
+
+### Example
+
+```python
+from openai import AsyncOpenAI
+import asyncio
+import os
+
+openai_client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
+
+async def get_capital(data):
+    country = data["country"]
+    response = await openai_client.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[
+            {"role": "system", "content": "You are a helpful assistant."},
+            {
+                "role": "user",
+                "content": f"What is the capital of {country}? Just name the "
+                "city and nothing else",
+            },
+        ],
+    )
+    return response.choices[0].message.content.strip()
+
+
+# Evaluation data
+data = [
+    {"data": {"country": "Canada"}, "target": {"capital": "Ottawa"}},
+    {"data": {"country": "Germany"}, "target": {"capital": "Berlin"}},
+    {"data": {"country": "Tanzania"}, "target": {"capital": "Dodoma"}},
+]
+
+
+def evaluator_A(output, target):
+    return 1 if output == target["capital"] else 0
+
+
+# Create an Evaluation instance
+e = Evaluation(
+    name="py-evaluation-async",
+    data=data,
+    executor=get_capital,
+    evaluators=[evaluator_A],
+    project_api_key=os.environ["LMNR_PROJECT_API_KEY"],
+)
+
+# Run the evaluation
+asyncio.run(e.run())
+```

{lmnr-0.3.7 → lmnr-0.4.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "lmnr"
-version = "0.3.7"
+version = "0.4.1"
 description = "Python SDK for Laminar AI"
 authors = [
   { name = "lmnr.ai", email = "founders@lmnr.ai" }
@@ -11,7 +11,7 @@ license = "Apache-2.0"
 
 [tool.poetry]
 name = "lmnr"
-version = "0.3.7"
+version = "0.4.1"
 description = "Python SDK for Laminar AI"
 authors = ["lmnr.ai"]
 readme = "README.md"
@@ -22,9 +22,9 @@ python = "^3.9"
 pydantic = "^2.7.4"
 requests = "^2.32.3"
 python-dotenv = "^1.0.1"
-
-openai = "^1.41.1"
 backoff = "^2.2.1"
+traceloop-sdk = "^0.29.2"
+asyncio = "^3.4.3"
 
 [tool.poetry.group.dev.dependencies]
 black = "^24.8.0"

lmnr-0.4.1/src/lmnr/__init__.py

@@ -0,0 +1,4 @@
+from .sdk.evaluations import Evaluation
+from .sdk.laminar import Laminar
+from .sdk.types import ChatMessage, PipelineRunError, PipelineRunResponse, NodeInput
+from .sdk.decorators import observe

lmnr-0.4.1/src/lmnr/sdk/decorators.py

@@ -0,0 +1,72 @@
+from traceloop.sdk.decorators.base import (
+    entity_method,
+    aentity_method,
+)
+from opentelemetry.trace import INVALID_SPAN, get_current_span
+from traceloop.sdk import Traceloop
+
+from typing import Callable, Optional, ParamSpec, TypeVar, cast
+
+from .laminar import Laminar as L
+from .utils import is_async
+
+P = ParamSpec("P")
+R = TypeVar("R")
+
+
+def observe(
+    *,
+    name: Optional[str] = None,
+    user_id: Optional[str] = None,
+    session_id: Optional[str] = None,
+) -> Callable[[Callable[P, R]], Callable[P, R]]:
+    """The main decorator entrypoint for Laminar. This is used to wrap
+    functions and methods to create spans.
+
+    Args:
+        name (Optional[str], optional): Name of the span. Function
+                        name is used if not specified.
+                        Defaults to None.
+        user_id (Optional[str], optional): User ID to associate
+                        with the span and the following context.
+                        Defaults to None.
+        session_id (Optional[str], optional): Session ID to associate with the
+                        span and the following context. Defaults to None.
+
+    Raises:
+        Exception: re-raises the exception if the wrapped function raises
+                   an exception
+
+    Returns:
+        R: Returns the result of the wrapped function
+    """
+
+    def decorator(func: Callable[P, R]) -> Callable[P, R]:
+        if not L.is_initialized():
+            raise Exception(
+                "Laminar is not initialized. Please "
+                + "call Laminar.initialize() first."
+            )
+        current_span = get_current_span()
+        if current_span != INVALID_SPAN:
+            if session_id is not None:
+                current_span.set_attribute(
+                    "traceloop.association.properties.session_id", session_id
+                )
+            if user_id is not None:
+                current_span.set_attribute(
+                    "traceloop.association.properties.user_id", user_id
+                )
+        association_properties = {}
+        if session_id is not None:
+            association_properties["session_id"] = session_id
+        if user_id is not None:
+            association_properties["user_id"] = user_id
+        Traceloop.set_association_properties(association_properties)
+        return (
+            aentity_method(name=name)(func)
+            if is_async(func)
+            else entity_method(name=name)(func)
+        )
+
+    return cast(Callable[P, R], decorator)

lmnr-0.4.1/src/lmnr/sdk/evaluations.py

@@ -0,0 +1,163 @@
+from typing import Union
+
+from .utils import is_async
+from .types import EvaluatorFunction, ExecutorFunction, EvaluationDatapoint, Numeric
+from .laminar import Laminar as L
+import asyncio
+
+from abc import ABC, abstractmethod
+
+DEFAULT_BATCH_SIZE = 5
+
+
+class EvaluationDataset(ABC):
+    @abstractmethod
+    def __init__(self, *args, **kwargs):
+        pass
+
+    @abstractmethod
+    def __len__(self) -> int:
+        pass
+
+    @abstractmethod
+    def __getitem__(self, idx) -> EvaluationDatapoint:
+        pass
+
+    def slice(self, start: int, end: int):
+        return [self[i] for i in range(max(start, 0), min(end, len(self)))]
+
+
+class Evaluation:
+    def __init__(
+        self,
+        name,
+        data: Union[EvaluationDataset, list[Union[EvaluationDatapoint, dict]]],
+        executor: ExecutorFunction,
+        evaluators: list[EvaluatorFunction],
+        batch_size: int = DEFAULT_BATCH_SIZE,
+        project_api_key: str = "",
+        base_url: str = "https://api.lmnr.ai",
+    ):
+        """
+        Initializes an instance of the Evaluations class.
+        Parameters:
+            name (str): The name of the evaluation.
+            data (Union[List[Union[EvaluationDatapoint, dict]], EvaluationDataset]): List of data points to evaluate or an evaluation dataset.
+                            `data` is the input to the executor function,
+                            `target` is the input to the evaluator function.
+            executor (Callable[..., Any]): The executor function.
+                            Takes the data point + any additional arguments
+                            and returns the output to evaluate.
+            evaluators (List[Callable[..., Any]]): List of evaluator functions.
+                Each evaluator function takes the output of the executor _and_
+                the target data, and returns a score. The score can be a
+                single number or a record of string keys and number values.
+                If the score is a single number, it will be named after the
+                evaluator function. If the function is anonymous, it will be
+                named `evaluator_${index}`, where index is the index of the
+                evaluator function in the list starting from 1.
+            batch_size (int, optional): The batch size for evaluation.
+                            Defaults to DEFAULT_BATCH_SIZE.
+            project_api_key (str, optional): The project API key.
+                            Defaults to an empty string.
+            base_url (str, optional): The base URL for the LMNR API.
+                            Useful if self-hosted elsewhere.
+                            Defaults to "https://api.lmnr.ai".
+        """
+
+        self.name = name
+        self.executor = executor
+        self.evaluators = dict(
+            zip(
+                [
+                    (
+                        e.__name__
+                        if e.__name__ and e.__name__ != "<lambda>"
+                        else f"evaluator_{i+1}"
+                    )
+                    for i, e in enumerate(evaluators)
+                ],
+                evaluators,
+            )
+        )
+        self.evaluator_names = list(self.evaluators.keys())
+        if isinstance(data, list):
+            self.data = [
+                (
+                    EvaluationDatapoint.model_validate(point)
+                    if isinstance(point, dict)
+                    else point
+                )
+                for point in data
+            ]
+        else:
+            self.data = data
+        self.batch_size = batch_size
+        L.initialize(project_api_key=project_api_key, base_url=base_url)
+
+    async def run(self):
+        """Runs the evaluation.
+
+        Creates a new evaluation if no evaluation with such name exists, or
+        adds data to an existing one otherwise. Evaluates data points in
+        batches of `self.batch_size`. The executor
+        function is called on each data point to get the output,
+        and then evaluate it by each evaluator function.
+        """
+        response = L.create_evaluation(self.name)
+        batch_promises = []
+
+        for i in range(0, len(self.data), self.batch_size):
+            batch = (
+                self.data[i : i + self.batch_size]
+                if isinstance(self.data, list)
+                else self.data.slice(i, i + self.batch_size)
+            )
+            batch_promises.append(self._evaluate_batch(batch))
+
+        try:
+            await asyncio.gather(*batch_promises)
+            L.update_evaluation_status(response.name, "Finished")
+            print(f"Evaluation {response.id} complete")
+        except Exception as e:
+            print(f"Error evaluating batch: {e}")
+
+    async def _evaluate_batch(self, batch: list[EvaluationDatapoint]):
+        results = []
+        for datapoint in batch:
+            output = (
+                await self.executor(datapoint.data)
+                if is_async(self.executor)
+                else self.executor(datapoint.data)
+            )
+            target = datapoint.target
+
+            # iterate in order of evaluators
+            scores = {}
+            for evaluator_name in self.evaluator_names:
+                evaluator = self.evaluators[evaluator_name]
+                value = (
+                    await evaluator(output, target)
+                    if is_async(evaluator)
+                    else evaluator(output, target)
+                )
+
+                # if the evaluator returns a single number,
+                # use the evaluator name as the key
+                if isinstance(value, Numeric):
+                    scores[evaluator_name] = value
+                else:
+                    # if the evaluator returns an object,
+                    # use the object keys as the keys
+                    scores.update(value)
+
+            results.append(
+                {
+                    "executorOutput": output,
+                    "data": datapoint.data,
+                    "target": target,
+                    "scores": scores,
+                }
+            )
+
+        return L.post_evaluation_results(self.name, results)