bir-sdk 0.1.0__tar.gz

bir_sdk-0.1.0/LICENSE

@@ -0,0 +1,100 @@
+# Functional Source License, Version 1.1, ALv2 Future License
+
+## Abbreviation
+
+FSL-1.1-ALv2
+
+## Notice
+
+Copyright 2026 mskayacioglu
+
+## Terms and Conditions
+
+### Licensor ("We")
+
+The party offering the Software under these Terms and Conditions.
+
+### The Software
+
+The "Software" is each version of the software that we make available under
+these Terms and Conditions, as indicated by our inclusion of these Terms and
+Conditions with the Software.
+
+### License Grant
+
+Subject to your compliance with this License Grant and the Patents,
+Redistribution and Trademark clauses below, we hereby grant you the right to
+use, copy, modify, create derivative works, publicly perform, publicly display
+and redistribute the Software for any Permitted Purpose identified below.
+
+### Permitted Purpose
+
+A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
+means making the Software available to others in a commercial product or service
+that:
+
+1. substitutes for the Software;
+2. substitutes for any other product or service we offer using the Software that
+   exists as of the date we make the Software available; or
+3. offers the same or substantially similar functionality as the Software.
+
+Permitted Purposes specifically include using the Software:
+
+1. for your internal use and access;
+2. for non-commercial education;
+3. for non-commercial research; and
+4. in connection with professional services that you provide to a licensee using
+   the Software in accordance with these Terms and Conditions.
+
+### Patents
+
+To the extent your use for a Permitted Purpose would necessarily infringe our
+patents, the license grant above includes a license under our patents. If you
+make a claim against any party that the Software infringes or contributes to the
+infringement of any patent, then your patent license to the Software ends
+immediately.
+
+### Redistribution
+
+The Terms and Conditions apply to all copies, modifications and derivatives of
+the Software.
+
+If you redistribute any copies, modifications or derivatives of the Software,
+you must include a copy of or a link to these Terms and Conditions and not
+remove any copyright notices provided in or with the Software.
+
+### Disclaimer
+
+THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
+PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
+
+IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
+SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES, EVEN
+IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
+
+### Trademarks
+
+Except for displaying the License Details and identifying us as the origin of
+the Software, you have no right under these Terms and Conditions to use our
+trademarks, trade names, service marks or product names.
+
+## Grant of Future License
+
+We hereby irrevocably grant you an additional license to use the Software under
+the Apache License, Version 2.0 that is effective on the second anniversary of
+the date we make the Software available. On or after that date, you may use the
+Software under the Apache License, Version 2.0, in which case the following
+will apply:
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License.
+
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

bir_sdk-0.1.0/PKG-INFO

@@ -0,0 +1,584 @@
+Metadata-Version: 2.4
+Name: bir-sdk
+Version: 0.1.0
+Summary: Minimal local tracing SDK for LLM applications.
+Author: mskayacioglu
+Project-URL: Homepage, https://github.com/mskayacioglu/bir
+Project-URL: Documentation, https://github.com/mskayacioglu/bir/tree/main/docs
+Project-URL: Source, https://github.com/mskayacioglu/bir
+Project-URL: Issues, https://github.com/mskayacioglu/bir/issues
+Keywords: ai,evals,llm,observability,tracing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Dynamic: license-file
+
+# Bir Python SDK
+
+Minimal local tracing SDK for Python LLM applications.
+
+Bir records traces, spans, generations, tool calls, and scores to local JSONL
+without requiring a server. Start locally, then send events to the Bir FastAPI
+server when you want to inspect them in the dashboard.
+
+## Installation
+
+After the first package release:
+
+```bash
+python -m pip install bir-sdk
+```
+
+The distribution is published on PyPI as `bir-sdk`; the import name is `bir`
+(e.g. `from bir import observe`).
+
+For local development from this repository:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+## Quickstart
+
+```python
+from bir import generation, observe, retrieval, score, span
+
+
+@observe()
+def answer_question(question: str) -> str:
+    with span("retrieve_context"):
+        with retrieval("search_docs", query=question) as result:
+            result.add_document(id="doc-1", text="local context")
+            documents = ["local context"]
+
+    with generation("local.llm", model="demo-model") as gen:
+        response = f"{documents[0]}: {question}"
+        gen.set_output(response)
+        gen.set_usage(input_tokens=12, output_tokens=24)
+        gen.set_cost(input_cost=0.000012, output_cost=0.000048)
+
+    score("helpfulness", 0.82)
+    return response
+```
+
+Trace, span, tool call, generation, and score events are written as JSONL to:
+
+```text
+.bir/traces.jsonl
+```
+
+Writes to the local trace file are serialized within the SDK process so
+multi-threaded sync applications keep the JSONL file line-delimited and
+parseable.
+
+## Manual Trace Contexts
+
+Use `trace()` when your workflow is easier to wrap with a context manager than a
+decorator:
+
+```python
+from bir import generation, score, span, trace
+
+with trace("answer_question", metadata={"kind": "manual"}):
+    with span("draft_answer"):
+        with generation("local.llm", model="demo-model") as gen:
+            response = "ok"
+            gen.set_output(response)
+    score("helpfulness", 0.82)
+```
+
+## Read Local Traces
+
+You can also read local traces back from the same file:
+
+```python
+from bir import load_traces
+
+for trace in load_traces():
+    print(trace.name, trace.status, trace.duration_ms)
+    for event in trace.events:
+        print(event.type, event.name)
+```
+
+## Send Events To The Server
+
+To send local events to a running Bir server:
+
+```python
+from bir import send_events
+
+result = send_events("http://127.0.0.1:8000")
+print(result.accepted, result.attempted, result.skipped)
+```
+
+`send_events()` posts local JSONL events to the Bir server, batching them into a
+single request when the server supports it and otherwise posting them to
+`/v1/events` one at a time. It uses the Python standard library, reports how many
+local events were attempted, how many were newly accepted, and how many were
+skipped by an idempotent server response,
+raises `RuntimeError` when the server rejects an event or cannot be reached, and
+does not remove local events after sending. Re-sending the same file is safe
+against the Bir server because duplicate event IDs are treated as already
+ingested.
+Complete traces are sent root-first so the server receives the trace event before
+its spans, tool calls, generations, and scores.
+
+## Privacy And Capture
+
+Input and output capture is disabled by default. Enable it globally with `configure()`
+or for a single function with `@observe(capture_inputs=True, capture_outputs=True)`.
+Common secret-like fields such as `api_key`, `authorization`, `password`, `secret`,
+and `token` are redacted before events are written.
+Captured strings, fallback object representations, and captured error messages
+are also scanned for common secret-like text patterns before events are written.
+Redaction is best-effort, so keep capture opt-in for sensitive payloads and review
+what your application records.
+
+```python
+from bir import configure
+
+configure(capture_inputs=True, capture_outputs=True)
+```
+
+Captured values are normalized to JSON-compatible data before writing. Non-finite
+floats such as `NaN` and `Infinity` are stored as strings, and deeply nested
+values are truncated. `score()` requires a finite numeric value and accepts
+optional `metadata` (for example `score("faithfulness", 0.4, metadata={"reason":
+"answer cites no context"})`) that is redacted with the same rules before it is
+written to the score event. Generation token usage and generation cost require at
+least one field, and all provided values must be non-negative finite numbers.
+
+Generation cost is user-provided. Bir records explicit cost values and defaults
+the currency to `USD`; it does not calculate provider pricing automatically.
+
+## Service Metadata
+
+Use `configure()` to tag traces with the service and environment that produced
+them. Both values are optional, must be non-empty strings, and are recorded on
+trace root events under `metadata.service` so the server and dashboard can
+filter traces by service and environment.
+
+```python
+from bir import configure
+
+configure(service_name="rag-api", environment="production")
+```
+
+## Sampling
+
+Use `configure(sample_rate=...)` to keep local trace volume bounded under load.
+`sample_rate` is the probability (`0.0` to `1.0`) that a trace is recorded and
+defaults to `1.0`, which records every trace. The decision is made once per
+trace root and inherited by every event under it, so a sampled-out trace and all
+of its spans, generations, tool calls, retrievals, and scores write nothing.
+
+Sampling never changes control flow: a sampled-out function still runs and still
+raises its own exceptions; only the local JSONL writes are skipped.
+
+```python
+from bir import configure
+
+configure(sample_rate=0.1)  # record about 10% of traces
+```
+
+## Retrieval
+
+Use `retrieval()` to record RAG lookups with the existing `tool_call` event
+contract. It sets `metadata.kind` to `retrieval`, stores the query at
+`input.query` when input capture is enabled, and stores retrieved records at
+`output.documents` when output capture is enabled.
+Document `rank` values must be non-negative integers, and document `score`
+values must be non-negative finite numbers.
+
+```python
+from bir import retrieval
+
+with retrieval("vector_search", query=question) as result:
+    result.add_document(
+        id="doc-1",
+        rank=1,
+        score=0.82,
+        source="docs",
+        text="Bir records local traces with JSONL.",
+    )
+```
+
+## Prompt Versions
+
+Use `prompt()` to attach prompt identity and version metadata to a generation.
+Prompt template text, variables, and rendered prompts are not captured unless
+you opt in.
+
+```python
+from bir import generation, prompt
+
+answer_prompt = prompt(
+    "answer_question",
+    version="v1",
+    template="Answer using this context: {context}",
+    variables={"context": "local context"},
+)
+
+with generation("local.llm", model="demo-model", prompt=answer_prompt) as gen:
+    gen.set_output("ok")
+```
+
+The generation event records `metadata.prompt.name`, `metadata.prompt.version`,
+and a `metadata.prompt.template_sha256` when a template is provided. To inspect
+the actual prompt payload locally, opt in explicitly:
+
+```python
+answer_prompt = prompt(
+    "answer_question",
+    version="v1",
+    template="Answer using this context: {context}",
+    variables={"context": "local context"},
+    capture_template=True,
+    capture_variables=True,
+    capture_rendered=True,
+)
+```
+
+Captured prompt fields use the same best-effort redaction as other captured
+payloads. After sending traces to the local server, the dashboard shows prompt
+metadata on generation details without requiring you to inspect the raw event
+JSON.
+
+## LangChain Callback
+
+Use `BirCallbackHandler` to record LangChain callback events as Bir traces
+without adding LangChain as a Bir dependency:
+
+```python
+from bir import configure
+from bir.integrations.langchain import BirCallbackHandler
+
+configure(capture_inputs=True, capture_outputs=True)
+
+result = chain.invoke(
+    {"question": "What is Bir?"},
+    config={"callbacks": [BirCallbackHandler()]},
+)
+```
+
+Root chains become trace events, nested chains become spans, LLM/chat model
+callbacks become generation events, retrievers become retrieval tool calls, and
+tools become tool call events. Direct model calls without an active chain create
+a small implicit trace root. The handler records token usage from common
+LangChain response shapes including `llm_output.token_usage`,
+`usage_metadata`, and `response_metadata.token_usage`.
+
+## Mistral
+
+Use `trace_chat()` to record Mistral chat completions without adding `mistralai`
+as a Bir dependency:
+
+```python
+from bir import trace
+from bir.integrations.mistral import trace_chat
+
+with trace("chat"):
+    response = trace_chat(
+        client.chat.complete,
+        model="mistral-small-latest",
+        messages=[{"role": "user", "content": "What is Bir?"}],
+    )
+```
+
+The wrapper forwards positional and keyword arguments unchanged, returns the
+Mistral response untouched, and records the response model, token usage, and
+`model_dump()` output when capture settings allow it.
+
+## Cohere
+
+Use `trace_chat()` to record Cohere v2 chat calls without adding `cohere` as a
+Bir dependency:
+
+```python
+from bir import trace
+from bir.integrations.cohere import trace_chat
+
+with trace("chat"):
+    response = trace_chat(
+        client.chat,
+        model="command-a-03-2025",
+        messages=[{"role": "user", "content": "What is Bir?"}],
+    )
+```
+
+The wrapper forwards positional and keyword arguments unchanged, returns the
+Cohere response untouched, records the request model, and reads token usage from
+`response.usage.tokens` when present.
+
+## Local Evals And Experiments
+
+Bir includes a small deterministic evaluation layer for local regression checks.
+It does not require a server or an LLM judge.
+
+```python
+from bir.evals import Dataset, DatasetExample, contains, exact_match, latency_under, run_experiment
+
+
+dataset = Dataset(
+    [
+        DatasetExample(
+            id="q1",
+            input={"question": "What is Bir?"},
+            expected="An observability SDK",
+        )
+    ]
+)
+
+
+def answer_question(question: str) -> str:
+    return "Bir is an observability SDK."
+
+
+result = run_experiment(
+    "quickstart",
+    dataset=dataset,
+    task=answer_question,
+    evaluators=[
+        contains(),
+        exact_match("Bir is an observability SDK."),
+        latency_under(1000),
+    ],
+)
+
+print(result.aggregate_scores)
+```
+
+Datasets can be stored as JSONL:
+
+```json
+{"id":"q1","input":{"question":"What is Bir?"},"expected":"An observability SDK"}
+```
+
+Load and run them locally:
+
+```python
+from bir.evals import Dataset, contains, list_experiments, load_experiment, run_experiment, send_experiment
+
+dataset = Dataset.from_jsonl("questions.jsonl")
+result = run_experiment("prompt-v1", dataset=dataset, task=answer_question, evaluators=[contains()])
+loaded = load_experiment(result.path)
+summaries = list_experiments()
+```
+
+`Dataset.to_jsonl()` redacts common secret-like values by default when exporting
+examples. If you intentionally need to preserve raw dataset payloads, opt out
+explicitly:
+
+```python
+dataset.to_jsonl("questions.jsonl", redact=False)
+```
+
+Experiment results are written to `.bir/experiments/*.jsonl` by default, with
+one result row per example. Bir also writes a sibling `.summary.json` file with
+the experiment id, status, example count, error count, aggregate scores, and
+result path so local runs can be listed without scanning every result row.
+Available deterministic evaluators are `exact_match()`, `contains()`,
+`regex_match()`, `json_valid()`, `field_equals()`, `field_contains()`,
+`latency_under()`, `cost_under()`, `numeric_between()`,
+`retrieved_context_contains()`, `answer_context_overlap()`,
+`answer_contains_citation()`, and `custom_evaluator()`.
+
+Upload a completed experiment to a running Bir server so the dashboard can show
+the experiment list and per-example result detail:
+
+```python
+send_experiment(result.path, "http://127.0.0.1:8000")
+```
+
+To connect local experiment rows back to traces, opt in when running the
+experiment:
+
+```python
+result = run_experiment(
+    "prompt-v1",
+    dataset=dataset,
+    task=answer_question,
+    evaluators=[contains()],
+    record_traces=True,
+)
+```
+
+`record_traces=True` writes one trace per dataset example and records evaluator
+outputs as score events on that trace.
+
+Use threshold evaluators for local gates:
+
+```python
+from bir.evals import cost_under, latency_under, numeric_between
+
+evaluators = [
+    latency_under(1000),
+    cost_under(0.05),
+    numeric_between(min_value=0.0, max_value=1.0),
+]
+```
+
+`latency_under()` uses measured task duration from `run_experiment()`.
+`cost_under()` checks explicit cost fields returned by your task, either
+`{"total_cost": 0.01}` or `{"cost": {"total_cost": 0.01}}`. Bir records only the
+cost values you provide and does not calculate provider pricing.
+`numeric_between()` checks numeric task outputs, or a numeric field when
+`field=` is provided.
+
+Use structured output evaluators for JSON-like task results:
+
+```python
+from bir.evals import field_contains, field_equals, numeric_between
+
+evaluators = [
+    field_contains("answer", "observability"),
+    field_equals("citations[0].id", "doc-1"),
+    numeric_between(min_value=0.7, max_value=1.0, field="confidence"),
+]
+```
+
+Field paths support dot paths and list indexes, such as `answer`,
+`usage.total_tokens`, and `items[0].name`. Missing paths produce a `0.0` score
+with failure metadata instead of failing the experiment.
+
+Use `custom_evaluator()` for local checks that are specific to your task:
+
+```python
+from bir.evals import EvalResult, custom_evaluator
+
+has_citation = custom_evaluator(
+    "has_citation",
+    lambda output, expected: "[1]" in str(output),
+)
+
+debuggable = custom_evaluator(
+    "debuggable",
+    lambda output, expected: EvalResult(
+        name="debuggable",
+        value=1.0,
+        metadata={"expected": expected},
+    ),
+)
+```
+
+Custom evaluators may return `bool`, `int`, `float`, or `EvalResult`. Exceptions
+from custom evaluator functions surface normally during development.
+
+Use `retrieved_context_contains()` to check retrieval quality without an LLM
+judge:
+
+```python
+from bir.evals import retrieved_context_contains
+
+evaluators = [
+    retrieved_context_contains("observability"),
+]
+```
+
+`retrieved_context_contains()` reads the `contexts` list from a structured RAG
+output such as `{"answer": "...", "contexts": ["doc text", ...]}` and scores
+`1.0` when `expected` appears in one of the retrieved strings. Missing or empty
+`contexts` produce a `0.0` score with failure metadata instead of failing the
+experiment. Pass `case_sensitive=False` for case-insensitive matching. This is a
+deterministic retrieval check, not proof that the answer used the context.
+
+Use `answer_context_overlap()` to flag answers that may not be grounded in the
+retrieved context, also without an LLM judge:
+
+```python
+from bir.evals import answer_context_overlap
+
+evaluators = [
+    answer_context_overlap(0.5),
+]
+```
+
+`answer_context_overlap()` reads the same structured RAG output
+(`{"answer": "...", "contexts": ["doc text", ...]}`) and scores `1.0` when at
+least `min_ratio` of the answer's word tokens also appear in the retrieved
+contexts. It is a deterministic faithfulness heuristic, not proof of
+faithfulness: paraphrased but faithful answers can score low, and unfaithful
+answers that reuse context words can score high. Missing answers or contexts
+produce a `0.0` score with failure metadata instead of failing the experiment.
+
+Use `answer_contains_citation()` to check that an answer cites a source, also
+without an LLM judge:
+
+```python
+from bir.evals import answer_contains_citation
+
+evaluators = [
+    answer_contains_citation(),
+]
+```
+
+`answer_contains_citation()` reads a plain answer string or the `answer` field
+of a structured RAG output (`{"answer": "...", "contexts": [...]}`) and scores
+`1.0` when the answer contains a citation marker. By default any bracketed
+marker such as `[1]` or `[doc-1]` counts; pass `pattern` to require a custom
+citation format such as `pattern=r"\(\d+\)"` for markers like `(1)`. This is a
+deterministic format check, not proof that the citation is correct or that the
+cited source supports the answer. Non-text output or a missing `answer` produces
+a `0.0` score with failure metadata instead of failing the experiment.
+
+## Event Loading
+
+`load_events()` validates JSONL records against the current event schema and
+raises `ValueError` for malformed rows, unsupported event types, invalid
+timestamps, or unsupported schema versions.
+
+To write traces somewhere else:
+
+```python
+configure(trace_path="tmp/bir-traces.jsonl")
+```
+
+## Development
+
+Run the SDK unit tests from this directory:
+
+```bash
+PYTHONPATH=src ../../.venv/bin/python -m unittest discover -s tests
+```
+
+Or install the package with test dependencies and run pytest:
+
+```bash
+python3 -m pip install -e ".[dev]"
+pytest
+```
+
+Run repository type checking from the repository root:
+
+```bash
+./.venv/bin/pyright
+```
+
+Run the release verification script from the repository root before publishing:
+
+```bash
+./.venv/bin/python scripts/verify_release.py
+```
+
+The script builds a temporary SDK wheel, installs it in a fresh temporary
+virtual environment, and smoke-tests local tracing and retrieval without writing
+build artifacts into the repository.
+
+Release planning lives in `CHANGELOG.md` and `docs/SDK_RELEASE_CHECKLIST.md`.
+
+## License
+
+Bir is source-available under the Functional Source License 1.1 with Apache 2.0
+as the future license (`FSL-1.1-ALv2`). FSL is not an OSI-approved open source
+license.