truss 0.11.12rc508__tar.gz → 0.11.12rc509__tar.gz

truss-0.11.12rc509/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.4
+Name: truss
+Version: 0.11.12rc509
+Summary: A seamless bridge from model development to model delivery
+Project-URL: Repository, https://github.com/basetenlabs/truss
+Project-URL: Homepage, https://truss.baseten.co
+Project-URL: Bug Reports, https://github.com/basetenlabs/truss/issues
+Project-URL: Documentation, https://truss.baseten.co
+Project-URL: Baseten, https://baseten.co
+Author-email: Pankaj Gupta <no-reply@baseten.co>, Phil Howes <no-reply@baseten.co>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: AI,MLOps,Machine Learning,Model Deployment,Model Serving
+Requires-Python: <3.14,>=3.9
+Requires-Dist: aiofiles<25,>=24.1.0
+Requires-Dist: blake3<2,>=1.0.4
+Requires-Dist: boto3<2,>=1.34.85
+Requires-Dist: click<9,>=8.0.3
+Requires-Dist: google-cloud-storage>=2.10.0
+Requires-Dist: httpx-ws<0.8,>=0.7.1
+Requires-Dist: httpx>=0.24.1
+Requires-Dist: huggingface-hub>=0.25.0
+Requires-Dist: inquirerpy<0.4,>=0.3.4
+Requires-Dist: jinja2<4,>=3.1.2
+Requires-Dist: libcst>=1.1.2
+Requires-Dist: loguru>=0.7.2
+Requires-Dist: packaging>=20.9
+Requires-Dist: pathspec>=0.9.0
+Requires-Dist: psutil>=5.9.4
+Requires-Dist: pydantic>=2.10.0
+Requires-Dist: python-json-logger>=2.0.2
+Requires-Dist: python-on-whales>=0.68.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31
+Requires-Dist: rich-click<2,>=1.6.1
+Requires-Dist: rich<14,>=13.4.2
+Requires-Dist: ruff>=0.4.8
+Requires-Dist: tenacity>=8.0.1
+Requires-Dist: tomlkit>=0.13.2
+Requires-Dist: truss-transfer<0.0.37,>=0.0.32
+Requires-Dist: watchfiles<0.20,>=0.19.0
+Description-Content-Type: text/markdown
+
+# Truss
+
+**The simplest way to serve AI/ML models in production**
+
+[![PyPI version](https://badge.fury.io/py/truss.svg)](https://badge.fury.io/py/truss)
+[![ci_status](https://github.com/basetenlabs/truss/actions/workflows/release.yml/badge.svg)](https://github.com/basetenlabs/truss/actions/workflows/release.yml)
+
+## Why Truss?
+
+* **Write once, run anywhere:** Package and test model code, weights, and dependencies with a model server that behaves the same in development and production.
+* **Fast developer loop:** Implement your model with fast feedback from a live reload server, and skip Docker and Kubernetes configuration with a batteries-included model serving environment.
+* **Support for all Python frameworks**: From `transformers` and `diffusers` to `PyTorch` and `TensorFlow` to `TensorRT` and `Triton`, Truss supports models created and served with any framework.
+
+See Trusses for popular models including:
+
+* 🦙 [Llama 2 7B](https://github.com/basetenlabs/truss-examples/tree/main/llama/llama-2-7b-chat) ([13B](https://github.com/basetenlabs/truss-examples/tree/main/llama/llama-2-13b-chat)) ([70B](https://github.com/basetenlabs/truss-examples/tree/main/llama/llama-2-70b-chat))
+* 🎨 [Stable Diffusion XL](https://github.com/basetenlabs/truss-examples/tree/main/stable-diffusion/stable-diffusion-xl-1.0)
+* 🗣 [Whisper](https://github.com/basetenlabs/truss-examples/tree/main/whisper/whisper-truss)
+
+and [dozens more examples](https://github.com/basetenlabs/truss-examples/).
+
+## Installation
+
+Install Truss with:
+
+```
+pip install --upgrade truss
+```
+
+## Quickstart
+
+As a quick example, we'll package a [text classification pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines) from the open-source [`transformers` package](https://github.com/huggingface/transformers).
+
+### Create a Truss
+
+To get started, create a Truss with the following terminal command:
+
+```sh
+truss init text-classification
+```
+
+When prompted, give your Truss a name like `Text classification`.
+
+Then, navigate to the newly created directory:
+
+```sh
+cd text-classification
+```
+
+### Implement the model
+
+One of the two essential files in a Truss is `model/model.py`. In this file, you write a `Model` class: an interface between the ML model that you're packaging and the model server that you're running it on.
+
+There are two member functions that you must implement in the `Model` class:
+
+* `load()` loads the model onto the model server. It runs exactly once when the model server is spun up or patched.
+* `predict()` handles model inference. It runs every time the model server is called.
+
+Here's the complete `model/model.py` for the text classification model:
+
+```python
+from transformers import pipeline
+
+
+class Model:
+    def __init__(self, **kwargs):
+        self._model = None
+
+    def load(self):
+        self._model = pipeline("text-classification")
+
+    def predict(self, model_input):
+        return self._model(model_input)
+```
+
+### Add model dependencies
+
+The other essential file in a Truss is `config.yaml`, which configures the model serving environment. For a complete list of the config options, see [the config reference](https://truss.baseten.co/reference/config).
+
+The pipeline model relies on [Transformers](https://huggingface.co/docs/transformers/index) and [PyTorch](https://pytorch.org/). These dependencies must be specified in the Truss config.
+
+In `config.yaml`, find the line `requirements`. Replace the empty list with:
+
+```yaml
+requirements:
+  - torch==2.0.1
+  - transformers==4.30.0
+```
+
+No other configuration is needed.
+
+## Deployment
+
+Truss is maintained by [Baseten](https://baseten.co), which provides infrastructure for running ML models in production. We'll use Baseten as the remote host for your model.
+
+Other remotes are coming soon, starting with AWS SageMaker.
+
+### Get an API key
+
+To set up the Baseten remote, you'll need a [Baseten API key](https://app.baseten.co/settings/account/api_keys). If you don't have a Baseten account, no worries, just [sign up for an account](https://app.baseten.co/signup/) and you'll be issued plenty of free credits to get you started.
+
+### Run `truss push`
+
+With your Baseten API key ready to paste when prompted, you can deploy your model:
+
+```sh
+truss push
+```
+
+You can monitor your model deployment from [your model dashboard on Baseten](https://app.baseten.co/models/).
+
+### Invoke the model
+
+After the model has finished deploying, you can invoke it from the terminal.
+
+**Invocation**
+
+```sh
+truss predict -d '"Truss is awesome!"'
+```
+
+**Response**
+
+```json
+[
+  {
+    "label": "POSITIVE",
+    "score": 0.999873161315918
+  }
+]
+```
+
+## Truss contributors
+
+Truss is backed by Baseten and built in collaboration with ML engineers worldwide. Special thanks to [Stephan Auerhahn](https://github.com/palp) @ [stability.ai](https://stability.ai/) and [Daniel Sarfati](https://github.com/dsarfati) @ [Salad Technologies](https://salad.com/) for their contributions.
+
+We enthusiastically welcome contributions in accordance with our [contributors' guide](CONTRIBUTING.md) and [code of conduct](CODE_OF_CONDUCT.md).

truss-0.11.12rc509/baseten/.github/PULL_REQUEST_TEMPLATE/prod_deploy.md

@@ -0,0 +1,3 @@
+## BEFORE DEPLOY DO THESE THINGS
+- ensure the commit is **merge commit**
+- look at whats in the PR and make sure appropriate people are tagged for review

truss-0.11.12rc509/baseten/.github/pull_request_template.md

@@ -0,0 +1,34 @@
+<!--
+  What does this PR add, remove, and/or change?
+-->
+## :rocket: What
+
+<!--
+  How was the change described above implemented?
+-->
+## :computer: How
+
+<!--
+  How have I ensured release and ongoing quality of this change?
+-->
+## :microscope: Testing
+
+<!--
+  For each of the categories of requirements, please specify what is required or write "none".
+
+  Pre-release requirements - Are there any changes need to be made or or rolled out before release of this PR?
+  - In most cases, before feature flags, Django fields, or internal APIs are deleted, all usages of them should be removed.
+  - Link to any PRs that need to be merged and fully deployed before this PR can be merged.
+
+  Post-release requirements - Does anything need to be changed or configured after this PR is merged for it to be fully rolled out?
+  - E.g. does a feature flag need to be enabled, a constance value updated, or a management command run?
+  - Please include *when* this should happen (e.g. immediately after a deploy, or are there other dependencies or tests that need to take place).
+  - Does this change require an image tag update?
+
+  External communication - Does this PR require a docs update, a changelog post, or reaching out to impacted customers?
+-->
+## :ship: Release requirements
+
+- **Pre-release:** Specify here
+- **Post-release:** Specify here
+- **External communication:** Specify here

truss-0.11.12rc509/baseten/.proto/plugins/README.md

@@ -0,0 +1,3 @@
+## Proto plugins
+
+By placing all [plugins](https://moonrepo.dev/docs/proto/plugins) in the monorepo, we ensure improved security and easier maintenance for everyone involved.

truss-0.11.12rc509/baseten/3rd-party-apps/README.md

@@ -0,0 +1,12 @@
+# Purpose
+
+We need a way to build images off 3rd party repos but not necessarily forking their repos.
+Use this folder to host the configuration of the jobs and build/push from the GH action workflows.
+
+## How
+- Clone the 3rd party repo
+- Duplicate the files need to be modified
+- Modify the files
+- Run `diff -u` to generate patch file and place it under the `3rd-party-apps/<app-name>` folder (combine all patches into one file)
+- Create a `config.yaml` file in the folder
+- Create a patch file, make sure the patch level should be `-p0` 

truss-0.11.12rc509/baseten/README.md

@@ -0,0 +1,109 @@
+# Baseten Server
+
+## Initial setup
+
+See [instructions in the wiki](/docs/local-dev/Dev-setup.md)
+
+## Running everything
+
+To start all services, run the following script from the project root:
+
+```sh
+bin/run_everything.sh
+```
+
+You should now be able to access the baseten application at http://localhost:9090
+
+## Prediction with Beefeater and Django
+
+Django does not support calling model prediction directly. Instead, Beefeater acts as an intermediary:
+
+- It sends all non-predict requests to Django
+- When it receives a predict request, it calls Djangos `authz` endpoint to check if the user is allowed to make the request and if so calls the model in the minikube cluster.
+
+Here's a flowchart illustrating the process:
+
+```mermaid
+flowchart LR
+    client[Client]
+    beefeater[Beefeater]
+    django[Django]
+    minikube[Minikube Cluster]
+
+    client -->|:9090| beefeater
+    beefeater -->|proxies to :8000| django
+    beefeater -->|predict path| minikube
+    django -->|communicates with| minikube
+```
+
+## Running individual services
+See [bin/run_everything.sh](bin/run_everything.sh) for how we run each service in the project.
+
+### Running beefeater
+
+```sh
+cd ./go/beefeater
+make run-cs
+```
+
+### Running the frontend
+
+```sh
+npm run start
+```
+
+### Running the Django backend
+Run the backend, overriding the `HOST_URL` to point to the beefeater server:
+
+```sh
+HOST_URL=http://localhost:9090 uv run python manage.py runserver
+```
+
+You should now be able to access the baseten application at http://localhost:9090
+
+### Starting the operator
+
+See [operator/README.md](operator/README.md) for instructions on running the operator server locally.
+
+### Running celery
+
+See [docs/local-dev/how-to-run-celery-locally.md](docs/local-dev/how-to-run-celery-locally.md) for instructions on running celery locally.
+
+### Running the async request service
+
+See [async-request-service/README.md](async-request-service/README.md) for instructions on running the async request service locally.
+
+### Starting the kube event watcher
+
+To start the **kube-event-watcher**, run:
+
+```sh
+cd go/kube-watcher
+make build
+API_KEY=[your api key] HOST_URL=http://localhost:8000 ./event-watcher
+```
+
+The API used should be associated to a service account user. You can either change the super user to be one in billip or you create a service account user and an associated API key in `shell_plus`
+
+```
+service_user = User.objects.create(username="watcher_service_account", is_service_account=True, password="any password")
+UserAPIKey.objects.create_key(user=service_user, name="kube-watcher-api-key")
+```
+
+The kube-event-watcher does not yet support live reload. It must be restarted when the code changes
+
+## Testing
+In the project root, run:
+```
+pytest backend/path/to/test_file.py
+```
+
+Note: many tests use fixtures that are defined in `backend/conftest.py`
+
+## Pre-commit
+
+Install the pre-commit hooks by running `uv run pre-commit install` on project root.
+
+## More docs
+
+Find them under [/docs](docs).

truss-0.11.12rc509/baseten/access-training-buckets-lambda/README.md

@@ -0,0 +1,9 @@
+# Access Training Buckets
+
+NOTE: this is not a python package. The contents of this folder should not be imported in other python projects.
+
+The `lambda_function.py` is the function that is deployed to AWS Lambda that is used to enable creation and access of training buckets.
+
+It is deployed to lambda via zip, by running `zip -r lambda_function.zip lambda_function.py` and uploading to AWS Lambda.
+
+It relies on the role `TrainingCheckpointsLambda` which has the `LambdaGetOrCreateTrainingBucketAndRole` policy. 

truss-0.11.12rc509/baseten/async-request-service/README.md

@@ -0,0 +1,194 @@
+# Async request service
+
+Run the async request service. It runs on port 9001 by default:
+```
+make run-server
+```
+
+Run the request processor. It runs on port 9002 by default (for metrics and liveness probes):
+```
+make run-processor
+```
+
+Test:
+```
+moon run test
+```
+
+## Database
+This project uses [sqlalchemy](https://www.sqlalchemy.org/) as its ORM and [alembic](https://alembic.sqlalchemy.org/en/latest/) to manage DB migrations
+
+To setup the dev and test DBs. It will create the databases and apply the alembic migrations
+
+```
+make dev-db
+```
+
+### Make a new migrations
+If you change the tables definition, you will need to generate a new migration and apply it
+```
+alembic revision --autogenerate -m "describe you change here"
+
+alembic upgrade head
+```
+
+You can revert a migration by running
+```
+alembic downgrade -1
+```
+
+## Run async request service locally
+
+If running the async request service from VS Code (including in a codespace), you should add the `./async-request-service` folder as a separate VS Code workspace root. Open the command palette (Cmd + Shift + P) and select "Add folder to workspace", then navigate to the "async-request-service" folder.
+
+When you open a new VS Code terminal, you'll be asked which folder to use. To run the async request service, make sure you select the "async-request-service" folder. Then run:
+
+```
+uv sync
+```
+
+### Async request service API:
+```
+uv run python core/server.py --reload
+
+OR
+
+export DB_HOST=$(minikube ip)
+export DB_PORT=$(kubectl get service -n baseten baseten-postgres-np -o jsonpath={.spec.ports[0].nodePort})
+make run-server
+```
+
+### Queue processor:
+
+You'll first need a `DJANGO_API_KEY` to properly interface with your local Django server. You can get this by running the following command in your Django project:
+```
+uv run manage.py create_async_request_service_account --global
+```
+
+```
+uv run python core/processor_server.py
+
+OR
+
+export DB_HOST=$(minikube ip)
+export DB_PORT=$(kubectl get service -n baseten baseten-postgres-np -o jsonpath={.spec.ports[0].nodePort})
+export DJANGO_API_KEY=<api key outputted from running the above command>
+make run-processor
+```
+
+### Webhook server:
+In its own terminal, start the local_webhook_server:
+```
+make run-local-webhook
+```
+
+
+### Running tests locally:
+To run the tests locally, you'll need to set the `DB_HOST` and `DB_PORT` environment variables.
+```
+export DB_HOST=$(minikube ip)
+export DB_PORT=$(kubectl get service -n baseten baseten-postgres-np -o jsonpath={.spec.ports[0].nodePort})
+make test
+```
+After setting the environment variables, you can also run the tests manually:
+```
+uv run pytest tests/... # specify the test file or directory
+```
+
+### Sample request
+Run this in a different terminal using curl or any REST client you like. 
+Don't forget to use the localtunnel url in the 'webhook_endpoint' field.
+
+#### Running an async inference on a model deployment
+```
+curl --request POST \
+  --url http://localhost:9090/environments/production/async_predict \
+  --header "Authorization: Api-Key $BASETEN_API_KEY" \
+  --header 'Content-Type: application/json' \
+  --header 'Host: model-n4q95w5d.api.dev.baseten.co' \
+  --data '{
+	"model_input": "THIS IS SOME TEXT IN UPPER CASE SO I SEE IT IN LOGS EASILY",
+	"webhook_endpoint": "http://0.0.0.0:9003/webhook",
+	"max_time_in_queue_seconds": 100,
+	"inference_retry_config": {
+		"max_attempts": 3,
+		"initial_delay_ms": 1000,
+		"max_delay_ms": 5000
+	}
+}'
+```
+Getting async request status:
+```
+curl --request GET \
+  --url http://localhost:9090/async_request/653335653076454c8c76dc48f75d526d \
+  --header "Authorization: Api-Key $BASETEN_API_KEY" \
+  --header 'Host: model-n4q95w5d.api.dev.baseten.co'
+```
+
+#### Running an async inference on a chain deployment
+```
+curl --request POST \
+  --url http://localhost:9090/environments/production/async_run_remote \
+  --header "Authorization: Api-Key $BASETEN_API_KEY" \
+  --header 'Content-Type: application/json' \
+  --header 'Host: chain-n4q95w5d.api.dev.baseten.co' \
+  --data '{
+	"model_input": "THIS IS SOME TEXT IN UPPER CASE SO I SEE IT IN LOGS EASILY",
+	"webhook_endpoint": "http://0.0.0.0:9003/webhook"
+}'
+```
+Getting async request status:
+```
+curl --request GET \
+  --url http://localhost:9090/async_request/653335653076454c8c76dc48f75d526d \
+  --header "Authorization: Api-Key $BASETEN_API_KEY" \
+  --header 'Host: chain-n4q95w5d.api.dev.baseten.co'
+```
+
+## Testing on staging
+To test the end to end async request feature in staging, you'll need something to receive the webhook. This project comes with a very lightweight fastapi server to do just that. See `tests/local_webhook_server.py` for the details.
+
+You will also need something to forward the webhook requests from the queue-processor deployed on staging (or dev or prod) to you local machine. We recommend using [localtunnel](https://theboroer.github.io/localtunnel-www/), but you can use any tool you like to do this.
+
+Here is an example using `localtunnel` and the `local_webhook_server`
+```
+# install the localtunnel client
+> npm install -g localtunnel
+
+# In its own terminal, start localtunnel
+> lt --port 9003
+your url is: https://two-hairs-bake.loca.lt
+
+# In its own terminal, start the local_webhook_server
+> make run-local-webhook
+
+# In a different terminal using curl or any REST client you like
+# Don't forget to use the localtunnel url in the 'webhook_endpoint' field
+curl --request POST \
+  --url https://model-5wo8m73y.api.staging.baseten.co/environments/production/async_predict \
+  --header "Authorization: Api-Key $BASETEN_API_KEY" \
+  --header 'Content-Type: application/json' \
+  --data '{
+	"model_input": {"prompt":"cowboy riding of in the sunset"},
+	"webhook_endpoint": "https://two-hairs-bake.loca.lt/webhook",
+	"max_time_in_queue_seconds": 100,
+	"inference_retry_config": {
+		"max_attempts": 3,
+		"initial_delay_ms": 1000,
+		"max_delay_ms": 5000
+	}
+}'
+```
+
+## Tracing async requests
+Right now the async request service only supports propagating request traces to Beefeater's `/predict` endpoint. We do this so we can reuse the same async request ID as a trace ID throughout the async request lifecycle. This is important for debugging and monitoring purposes.
+
+This works as follows:
+1. When `/async_predict` is called, Beefeater creates a `request_id` and passes it to the async request service.
+2. When the async request service is ready to run the inference, it sends a request to Beefeater's `/predict` endpoint with the `request_id` included as the `trace_id` in the `traceparent` header. This allows Beefeater to associate the inference request with the original async request ID and trace it throughout its lifecycle.
+
+We use a couple of [OpenTelemetry-specific concepts](https://opentelemetry.io/docs/languages/python/cookbook/#manually-setting-span-context) to achieve this:
+1. A [`TracerProvider`](https://opentelemetry.io/docs/specs/otel/trace/api/#tracerprovider) is initialized once in the processor server, this will create a globally accessible `TracerProvider` object that can be used throughout the service to obtain tracers.
+2. A [`Tracer`](https://opentelemetry.io/docs/specs/otel/trace/api/#tracer) object is used to create spans for a request, right now we create one in the `beefeater_client`. This is where we create a span for the request to Beefeater's `/predict` endpoint.
+3. A [`SpanContext`](https://opentelemetry.io/docs/specs/otel/trace/api/#spancontext) object is created to specify the trace ID and span ID for a request. This is used to create the `traceparent` header that is propagated to Beefeater. We explicitly set the `trace_id` to the `request_id` so that we can trace the request throughout its lifecycle.
+4. We create a [`TraceContextTextMapPropagator` propagator](https://opentelemetry.io/docs/specs/otel/context/api-propagators/#propagator-types) that is used to create and inject a [`traceparent` header](https://www.w3.org/TR/trace-context/#traceparent-header) into the request. This is used to propagate the trace ID from the async request service to Beefeater.

truss-0.11.12rc509/baseten/backend/.venv.old/lib/python3.9/site-packages/Brotli-1.1.0.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+_brotli
+brotli

truss-0.11.12rc509/baseten/backend/.venv.old/lib/python3.9/site-packages/ConfigArgParse-1.7.dist-info/top_level.txt

@@ -0,0 +1 @@
+configargparse

truss-0.11.12rc509/baseten/backend/.venv.old/lib/python3.9/site-packages/Cython-0.29.37.dist-info/COPYING.txt

@@ -0,0 +1,19 @@
+The original Pyrex code as of 2006-04 is licensed under the following
+license: "Copyright stuff: Pyrex is free of restrictions. You may use,
+redistribute, modify and distribute modified versions."
+
+------------------
+
+Cython, which derives from Pyrex, is licensed under the Apache 2.0
+Software License.  More precisely, all modifications and new code
+made to go from Pyrex to Cython are so licensed.
+
+See LICENSE.txt for more details.
+
+------------------
+
+The output of a Cython compilation is NOT considered a derivative
+work of Cython.  Specifically, though the compilation process may
+embed snippets of varying lengths into the final output, these
+snippets, as embedded in the output, do not encumber the resulting
+output with any license restrictions.