PyPI - litserve - Versions diffs - 0.2.7__tar.gz → 0.2.8.dev0__tar.gz - Mend

litserve 0.2.7tar.gz → 0.2.8.dev0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (52) hide show

{litserve-0.2.7/src/litserve.egg-info → litserve-0.2.8.dev0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: litserve
-Version: 0.2.7
+Version: 0.2.8.dev0
 Summary: Lightweight AI server.
 Home-page: https://github.com/Lightning-AI/litserve
 Download-URL: https://github.com/Lightning-AI/litserve
@@ -30,7 +30,7 @@ License-File: LICENSE
 Requires-Dist: fastapi>=0.100
 Requires-Dist: uvicorn[standard]>=0.29.0
 Requires-Dist: pyzmq>=22.0.0
-Requires-Dist: starlette<0.46.0
+Requires-Dist: starlette
 Provides-Extra: test
 Requires-Dist: asgi-lifespan; extra == "test"
 Requires-Dist: coverage[toml]>=7.5.3; extra == "test"
@@ -62,6 +62,7 @@ Dynamic: download-url
 Dynamic: home-page
 Dynamic: keywords
 Dynamic: license
+Dynamic: license-file
 Dynamic: project-url
 Dynamic: provides-extra
 Dynamic: requires-dist
@@ -89,10 +90,11 @@ LitServe is at least [2x faster](#performance) than plain FastAPI due to AI-spec
 <div align='center'>
 <pre>
-✅ (2x)+ faster serving  ✅ Easy to use          ✅ LLMs, non LLMs and more
-✅ Bring your own model  ✅ PyTorch/JAX/TF/...   ✅ Built on FastAPI
-✅ GPU autoscaling       ✅ Batching, Streaming  ✅ Self-host or ⚡️ managed
-✅ Compound AI           ✅ Integrate with vLLM and more
+✅ (2x)+ faster serving  ✅ Easy to use               ✅ LLMs, non LLMs and more
+✅ Bring your own model  ✅ PyTorch/JAX/TF/...        ✅ Built on FastAPI
+✅ GPU autoscaling       ✅ Batching, Streaming       ✅ Self-host or ⚡️ managed
+✅ Compound AI           ✅ Integrate with vLLM, etc  ✅ Serverless
 </pre>
 <div align='center'>
@@ -169,12 +171,23 @@ if __name__ == "__main__":
     server.run(port=8000)
 ```
-Now run the server via the command-line
+Now run the server anywhere (local or cloud) via the command-line.
 ```bash
+# Deploy to the cloud of your choice via Lightning AI (serverless, autoscaling, etc.)
+lightning serve server.py
+# Or run locally (self host anywhere)
+lightning serve server.py --local
+```
+Learn more about managed hosting on [Lightning AI](#hosting-options).
+You can also run the server manually:
+```bash
 python server.py
 ```
 ### Test the server
 Run the auto-generated test client:
 ```bash
@@ -198,7 +211,7 @@ litgpt serve microsoft/phi-2
 - LitAPI lets you easily build complex AI systems with one or more models ([docs](https://lightning.ai/docs/litserve/api-reference/litapi)).
 - Use the setup method for one-time tasks like connecting models, DBs, and loading data ([docs](https://lightning.ai/docs/litserve/api-reference/litapi#setup)).
 - LitServer handles optimizations like batching, GPU autoscaling, streaming, etc... ([docs](https://lightning.ai/docs/litserve/api-reference/litserver)).
-- Self host on your own machines or use Lightning Studios for a fully managed deployment ([learn more](#hosting-options)).
+- Self host on your machines or create a fully managed deployment with Lightning ([learn more](https://lightning.ai/docs/litserve/features/deploy-on-cloud)).
 [Learn how to make this server 200x faster](https://lightning.ai/docs/litserve/home/speed-up-serving-by-200x).
@@ -206,14 +219,7 @@ litgpt serve microsoft/phi-2
 # Featured examples
 Use LitServe to deploy any model or AI service: (Compound AI, Gen AI, classic ML, embeddings, LLMs, vision, audio, etc...)
-<div align='center'>
-  <div width='200px'>
-        <video src="https://github.com/user-attachments/assets/5e73549a-bc0f-47a9-9d9c-5b54389be5de" width='200px' controls></video>
-  </div>
-</div>
-## Examples
 <pre>
 <strong>Toy model:</strong>      <a target="_blank" href="#define-a-server">Hello world</a>
 <strong>LLMs:</strong>           <a target="_blank" href="https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-vision-with-litserve">Llama 3.2</a>, <a target="_blank" href="https://lightning.ai/lightning-ai/studios/openai-fault-tolerant-proxy-server">LLM Proxy server</a>, <a target="_blank" href="https://lightning.ai/lightning-ai/studios/deploy-ai-agent-with-tool-use">Agent with tool use</a>
@@ -233,31 +239,59 @@ Use LitServe to deploy any model or AI service: (Compound AI, Gen AI, classic ML
 &nbsp;
-# Features
-State-of-the-art features:
-✅ [(2x)+ faster than plain FastAPI](#performance)
-✅ [Bring your own model](https://lightning.ai/docs/litserve/features/full-control)
-✅ [Build compound systems (1+ models)](https://lightning.ai/docs/litserve/home)
-✅ [GPU autoscaling](https://lightning.ai/docs/litserve/features/gpu-inference)
-✅ [Batching](https://lightning.ai/docs/litserve/features/batching)
-✅ [Streaming](https://lightning.ai/docs/litserve/features/streaming)
-✅ [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)
-✅ [Self-host on your machines](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-your-own)
-✅ [Host fully managed on Lightning AI](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-lightning-studios)
-✅ [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)
-✅ [Scale to zero (serverless)](https://lightning.ai/docs/litserve/features/streaming)
-✅ [Supports PyTorch, JAX, TF, etc...](https://lightning.ai/docs/litserve/features/full-control)
-✅ [OpenAPI compliant](https://www.openapis.org/)
-✅ [Open AI compatibility](https://lightning.ai/docs/litserve/features/open-ai-spec)
-✅ [Authentication](https://lightning.ai/docs/litserve/features/authentication)
-✅ [Dockerization](https://lightning.ai/docs/litserve/features/dockerization-deployment)
+# Hosting options
+Self host LitServe anywhere or deploy to your favorite cloud via [Lightning AI](http://lightning.ai/deploy).
+https://github.com/user-attachments/assets/ff83dab9-0c9f-4453-8dcb-fb9526726344
+Self-hosting is ideal for hackers, students, and DIY developers while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
+To host on [Lightning AI](https://lightning.ai/deploy), simply add the `--cloud` arg, login and choose the cloud of your choice.
+```bash
+lightning serve api server.py --cloud
+```
+&nbsp;
+<div align='center'>
-[10+ features...](https://lightning.ai/docs/litserve/features)
+| [Feature](https://lightning.ai/docs/litserve/features)               | Self Managed                      | [Fully Managed on Lightning](https://lightning.ai/deploy)         |
+|----------------------------------------------------------------------|-----------------------------------|------------------------------------|
+| Docker-first deployment          | ✅ DIY                             | ✅ One-click deploy                |
+| Cost                             | ✅ Free (DIY)                      | ✅ Generous [free tier](https://lightning.ai/pricing) with pay as you go                |
+| Full control                     | ✅                                 | ✅                                 |
+| Use any engine (vLLM, etc.)      | ✅                                 | ✅ vLLM, Ollama, LitServe, etc.    |
+| Own VPC                          | ✅ (manual setup)                  | ✅ Connect your own VPC            |
+| [(2x)+ faster than plain FastAPI](#performance)                                               | ✅       | ✅                                 |
+| [Bring your own model](https://lightning.ai/docs/litserve/features/full-control)              | ✅       | ✅                                 |
+| [Build compound systems (1+ models)](https://lightning.ai/docs/litserve/home)                 | ✅       | ✅                                 |
+| [GPU autoscaling](https://lightning.ai/docs/litserve/features/gpu-inference)                  | ✅       | ✅                                 |
+| [Batching](https://lightning.ai/docs/litserve/features/batching)                              | ✅       | ✅                                 |
+| [Streaming](https://lightning.ai/docs/litserve/features/streaming)                            | ✅       | ✅                                 |
+| [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)                 | ✅       | ✅                                 |
+| [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)         | ✅       | ✅                                 |
+| [Supports PyTorch, JAX, TF, etc...](https://lightning.ai/docs/litserve/features/full-control) | ✅       | ✅                                 |
+| [OpenAPI compliant](https://www.openapis.org/)                                                | ✅       | ✅                                 |
+| [Open AI compatibility](https://lightning.ai/docs/litserve/features/open-ai-spec)             | ✅       | ✅                                 |
+| [Authentication](https://lightning.ai/docs/litserve/features/authentication)                  | ❌ DIY   | ✅ Token, password, custom         |
+| GPUs                             | ❌ DIY                             | ✅ 8+ GPU types, H100s from $1.75  |
+| Load balancing                   | ❌                                 | ✅ Built-in                        |
+| Scale to zero (serverless)       | ❌                                 | ✅ No machine runs when idle       |
+| Autoscale up on demand           | ❌                                 | ✅ Auto scale up/down              |
+| Multi-node inference             | ❌                                 | ✅ Distribute across nodes         |
+| Use AWS/GCP credits              | ❌                                 | ✅ Use existing cloud commits      |
+| Versioning                       | ❌                                 | ✅ Make and roll back releases     |
+| Enterprise-grade uptime (99.95%) | ❌                                 | ✅ SLA-backed                      |
+| SOC2 / HIPAA compliance          | ❌                                 | ✅ Certified & secure              |
+| Observability                    | ❌                                 | ✅ Built-in, connect 3rd party tools|
+| CI/CD ready                      | ❌                                 | ✅ Lightning SDK                   |
+| 24/7 enterprise support          | ❌                                 | ✅ Dedicated support               |
+| Cost controls & audit logs       | ❌                                 | ✅ Budgets, breakdowns, logs       |
+| Debug on GPUs                    | ❌                                 | ✅ Studio integration              |
+| [20+ features](https://lightning.ai/docs/litserve/features)                    | -                                 | -                                  |
-**Note:** We prioritize scalable, enterprise-level features over hype.
+</div>
 &nbsp;
@@ -276,40 +310,8 @@ These results are for image and text classification ML tasks. The performance re
 ***💡 Note on LLM serving:*** For high-performance LLM serving (like Ollama/vLLM), integrate [vLLM with LitServe](https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api), use [LitGPT](https://github.com/Lightning-AI/litgpt?tab=readme-ov-file#deploy-an-llm), or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.
-&nbsp;
-# Hosting options
-LitServe can be hosted independently on your own machines or fully managed via Lightning Studios.
-Self-hosting is ideal for hackers, students, and DIY developers, while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
 &nbsp;
-<div align="center">
-<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litserve-hello-world">
-  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/host-on-lightning.svg" alt="Host on Lightning"/>
-</a>
-</div>
-&nbsp;
-<div align='center'>
-| Feature                          | Self Managed                      | Fully Managed on Studios            |
-|----------------------------------|-----------------------------------|-------------------------------------|
-| Deployment                       | ✅ Do it yourself deployment      | ✅ One-button cloud deploy          |
-| Load balancing                   | ❌                                | ✅                                  |
-| Autoscaling                      | ❌                                | ✅                                  |
-| Scale to zero                    | ❌                                | ✅                                  |
-| Multi-machine inference          | ❌                                | ✅                                  |
-| Authentication                   | ❌                                | ✅                                  |
-| Own VPC                          | ❌                                | ✅                                  |
-| AWS, GCP                         | ❌                                | ✅                                  |
-| Use your own cloud commits       | ❌                                | ✅                                  |
-</div>
-&nbsp;
 # Community
 LitServe is a [community project accepting contributions](https://lightning.ai/docs/litserve/community) - Let's make the world's most advanced AI inference engine.

{litserve-0.2.7 → litserve-0.2.8.dev0}/README.md RENAMED Viewed

@@ -19,10 +19,11 @@ LitServe is at least [2x faster](#performance) than plain FastAPI due to AI-spec
 <div align='center'>
 <pre>
-✅ (2x)+ faster serving  ✅ Easy to use          ✅ LLMs, non LLMs and more
-✅ Bring your own model  ✅ PyTorch/JAX/TF/...   ✅ Built on FastAPI
-✅ GPU autoscaling       ✅ Batching, Streaming  ✅ Self-host or ⚡️ managed
-✅ Compound AI           ✅ Integrate with vLLM and more
+✅ (2x)+ faster serving  ✅ Easy to use               ✅ LLMs, non LLMs and more
+✅ Bring your own model  ✅ PyTorch/JAX/TF/...        ✅ Built on FastAPI
+✅ GPU autoscaling       ✅ Batching, Streaming       ✅ Self-host or ⚡️ managed
+✅ Compound AI           ✅ Integrate with vLLM, etc  ✅ Serverless
 </pre>
 <div align='center'>
@@ -99,12 +100,23 @@ if __name__ == "__main__":
     server.run(port=8000)
 ```
-Now run the server via the command-line
+Now run the server anywhere (local or cloud) via the command-line.
 ```bash
+# Deploy to the cloud of your choice via Lightning AI (serverless, autoscaling, etc.)
+lightning serve server.py
+# Or run locally (self host anywhere)
+lightning serve server.py --local
+```
+Learn more about managed hosting on [Lightning AI](#hosting-options).
+You can also run the server manually:
+```bash
 python server.py
 ```
 ### Test the server
 Run the auto-generated test client:
 ```bash
@@ -128,7 +140,7 @@ litgpt serve microsoft/phi-2
 - LitAPI lets you easily build complex AI systems with one or more models ([docs](https://lightning.ai/docs/litserve/api-reference/litapi)).
 - Use the setup method for one-time tasks like connecting models, DBs, and loading data ([docs](https://lightning.ai/docs/litserve/api-reference/litapi#setup)).
 - LitServer handles optimizations like batching, GPU autoscaling, streaming, etc... ([docs](https://lightning.ai/docs/litserve/api-reference/litserver)).
-- Self host on your own machines or use Lightning Studios for a fully managed deployment ([learn more](#hosting-options)).
+- Self host on your machines or create a fully managed deployment with Lightning ([learn more](https://lightning.ai/docs/litserve/features/deploy-on-cloud)).
 [Learn how to make this server 200x faster](https://lightning.ai/docs/litserve/home/speed-up-serving-by-200x).
@@ -136,14 +148,7 @@ litgpt serve microsoft/phi-2
 # Featured examples
 Use LitServe to deploy any model or AI service: (Compound AI, Gen AI, classic ML, embeddings, LLMs, vision, audio, etc...)
-<div align='center'>
-  <div width='200px'>
-        <video src="https://github.com/user-attachments/assets/5e73549a-bc0f-47a9-9d9c-5b54389be5de" width='200px' controls></video>
-  </div>
-</div>
-## Examples
 <pre>
 <strong>Toy model:</strong>      <a target="_blank" href="#define-a-server">Hello world</a>
 <strong>LLMs:</strong>           <a target="_blank" href="https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-vision-with-litserve">Llama 3.2</a>, <a target="_blank" href="https://lightning.ai/lightning-ai/studios/openai-fault-tolerant-proxy-server">LLM Proxy server</a>, <a target="_blank" href="https://lightning.ai/lightning-ai/studios/deploy-ai-agent-with-tool-use">Agent with tool use</a>
@@ -163,31 +168,59 @@ Use LitServe to deploy any model or AI service: (Compound AI, Gen AI, classic ML
 &nbsp;
-# Features
-State-of-the-art features:
-✅ [(2x)+ faster than plain FastAPI](#performance)
-✅ [Bring your own model](https://lightning.ai/docs/litserve/features/full-control)
-✅ [Build compound systems (1+ models)](https://lightning.ai/docs/litserve/home)
-✅ [GPU autoscaling](https://lightning.ai/docs/litserve/features/gpu-inference)
-✅ [Batching](https://lightning.ai/docs/litserve/features/batching)
-✅ [Streaming](https://lightning.ai/docs/litserve/features/streaming)
-✅ [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)
-✅ [Self-host on your machines](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-your-own)
-✅ [Host fully managed on Lightning AI](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-lightning-studios)
-✅ [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)
-✅ [Scale to zero (serverless)](https://lightning.ai/docs/litserve/features/streaming)
-✅ [Supports PyTorch, JAX, TF, etc...](https://lightning.ai/docs/litserve/features/full-control)
-✅ [OpenAPI compliant](https://www.openapis.org/)
-✅ [Open AI compatibility](https://lightning.ai/docs/litserve/features/open-ai-spec)
-✅ [Authentication](https://lightning.ai/docs/litserve/features/authentication)
-✅ [Dockerization](https://lightning.ai/docs/litserve/features/dockerization-deployment)
+# Hosting options
+Self host LitServe anywhere or deploy to your favorite cloud via [Lightning AI](http://lightning.ai/deploy).
+https://github.com/user-attachments/assets/ff83dab9-0c9f-4453-8dcb-fb9526726344
+Self-hosting is ideal for hackers, students, and DIY developers while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
+To host on [Lightning AI](https://lightning.ai/deploy), simply add the `--cloud` arg, login and choose the cloud of your choice.
+```bash
+lightning serve api server.py --cloud
+```
+&nbsp;
+<div align='center'>
-[10+ features...](https://lightning.ai/docs/litserve/features)
+| [Feature](https://lightning.ai/docs/litserve/features)               | Self Managed                      | [Fully Managed on Lightning](https://lightning.ai/deploy)         |
+|----------------------------------------------------------------------|-----------------------------------|------------------------------------|
+| Docker-first deployment          | ✅ DIY                             | ✅ One-click deploy                |
+| Cost                             | ✅ Free (DIY)                      | ✅ Generous [free tier](https://lightning.ai/pricing) with pay as you go                |
+| Full control                     | ✅                                 | ✅                                 |
+| Use any engine (vLLM, etc.)      | ✅                                 | ✅ vLLM, Ollama, LitServe, etc.    |
+| Own VPC                          | ✅ (manual setup)                  | ✅ Connect your own VPC            |
+| [(2x)+ faster than plain FastAPI](#performance)                                               | ✅       | ✅                                 |
+| [Bring your own model](https://lightning.ai/docs/litserve/features/full-control)              | ✅       | ✅                                 |
+| [Build compound systems (1+ models)](https://lightning.ai/docs/litserve/home)                 | ✅       | ✅                                 |
+| [GPU autoscaling](https://lightning.ai/docs/litserve/features/gpu-inference)                  | ✅       | ✅                                 |
+| [Batching](https://lightning.ai/docs/litserve/features/batching)                              | ✅       | ✅                                 |
+| [Streaming](https://lightning.ai/docs/litserve/features/streaming)                            | ✅       | ✅                                 |
+| [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)                 | ✅       | ✅                                 |
+| [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)         | ✅       | ✅                                 |
+| [Supports PyTorch, JAX, TF, etc...](https://lightning.ai/docs/litserve/features/full-control) | ✅       | ✅                                 |
+| [OpenAPI compliant](https://www.openapis.org/)                                                | ✅       | ✅                                 |
+| [Open AI compatibility](https://lightning.ai/docs/litserve/features/open-ai-spec)             | ✅       | ✅                                 |
+| [Authentication](https://lightning.ai/docs/litserve/features/authentication)                  | ❌ DIY   | ✅ Token, password, custom         |
+| GPUs                             | ❌ DIY                             | ✅ 8+ GPU types, H100s from $1.75  |
+| Load balancing                   | ❌                                 | ✅ Built-in                        |
+| Scale to zero (serverless)       | ❌                                 | ✅ No machine runs when idle       |
+| Autoscale up on demand           | ❌                                 | ✅ Auto scale up/down              |
+| Multi-node inference             | ❌                                 | ✅ Distribute across nodes         |
+| Use AWS/GCP credits              | ❌                                 | ✅ Use existing cloud commits      |
+| Versioning                       | ❌                                 | ✅ Make and roll back releases     |
+| Enterprise-grade uptime (99.95%) | ❌                                 | ✅ SLA-backed                      |
+| SOC2 / HIPAA compliance          | ❌                                 | ✅ Certified & secure              |
+| Observability                    | ❌                                 | ✅ Built-in, connect 3rd party tools|
+| CI/CD ready                      | ❌                                 | ✅ Lightning SDK                   |
+| 24/7 enterprise support          | ❌                                 | ✅ Dedicated support               |
+| Cost controls & audit logs       | ❌                                 | ✅ Budgets, breakdowns, logs       |
+| Debug on GPUs                    | ❌                                 | ✅ Studio integration              |
+| [20+ features](https://lightning.ai/docs/litserve/features)                    | -                                 | -                                  |
-**Note:** We prioritize scalable, enterprise-level features over hype.
+</div>
 &nbsp;
@@ -206,40 +239,8 @@ These results are for image and text classification ML tasks. The performance re
 ***💡 Note on LLM serving:*** For high-performance LLM serving (like Ollama/vLLM), integrate [vLLM with LitServe](https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api), use [LitGPT](https://github.com/Lightning-AI/litgpt?tab=readme-ov-file#deploy-an-llm), or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.
-&nbsp;
-# Hosting options
-LitServe can be hosted independently on your own machines or fully managed via Lightning Studios.
-Self-hosting is ideal for hackers, students, and DIY developers, while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
 &nbsp;
-<div align="center">
-<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litserve-hello-world">
-  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/host-on-lightning.svg" alt="Host on Lightning"/>
-</a>
-</div>
-&nbsp;
-<div align='center'>
-| Feature                          | Self Managed                      | Fully Managed on Studios            |
-|----------------------------------|-----------------------------------|-------------------------------------|
-| Deployment                       | ✅ Do it yourself deployment      | ✅ One-button cloud deploy          |
-| Load balancing                   | ❌                                | ✅                                  |
-| Autoscaling                      | ❌                                | ✅                                  |
-| Scale to zero                    | ❌                                | ✅                                  |
-| Multi-machine inference          | ❌                                | ✅                                  |
-| Authentication                   | ❌                                | ✅                                  |
-| Own VPC                          | ❌                                | ✅                                  |
-| AWS, GCP                         | ❌                                | ✅                                  |
-| Use your own cloud commits       | ❌                                | ✅                                  |
-</div>
-&nbsp;
 # Community
 LitServe is a [community project accepting contributions](https://lightning.ai/docs/litserve/community) - Let's make the world's most advanced AI inference engine.

{litserve-0.2.7 → litserve-0.2.8.dev0}/requirements.txt RENAMED Viewed

@@ -1,4 +1,4 @@
 fastapi >=0.100
 uvicorn[standard] >=0.29.0
 pyzmq >=22.0.0
-starlette<0.46.0  # TODO: fix root cause for buffer error
+starlette

{litserve-0.2.7 → litserve-0.2.8.dev0}/src/litserve/__about__.py RENAMED Viewed

@@ -11,7 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-__version__ = "0.2.7"
+__version__ = "0.2.8.dev0"
 __author__ = "Lightning-AI et al."
 __author_email__ = "community@lightning.ai"
 __license__ = "Apache-2.0"

{litserve-0.2.7 → litserve-0.2.8.dev0}/src/litserve/loops/base.py RENAMED Viewed

@@ -15,7 +15,6 @@ import asyncio
 import inspect
 import logging
 import pickle
-import signal
 import sys
 import time
 from abc import ABC
@@ -27,12 +26,14 @@ from starlette.formparsers import MultiPartParser
 from litserve import LitAPI
 from litserve.callbacks import CallbackRunner
 from litserve.specs.base import LitSpec
+from litserve.transport.base import MessageTransport
 from litserve.utils import LitAPIStatus
-from litserve.zmq_queue import Producer
 logger = logging.getLogger(__name__)
 # FastAPI writes form files to disk over 1MB by default, which prevents serialization by multiprocessing
 MultiPartParser.max_file_size = sys.maxsize
+# renamed in PR: https://github.com/encode/starlette/pull/2780
+MultiPartParser.spool_max_size = sys.maxsize
 def _inject_context(context: Union[List[dict], dict], func, *args, **kwargs):
@@ -152,7 +153,7 @@ class _BaseLoop(ABC):
         device: str,
         worker_id: int,
         request_queue: Queue,
-        response_queues: List[Queue],
+        transport: MessageTransport,
         max_batch_size: int,
         batch_timeout: float,
         stream: bool,
@@ -164,9 +165,7 @@ class _BaseLoop(ABC):
             async def _wrapper():
                 logger.info("Running LitLoop in a asyncio event loop")
-                future = self.schedule_task(
-                    lit_api, lit_spec, request_queue, max_batch_size, batch_timeout, response_queues
-                )
+                future = self.schedule_task(lit_api, lit_spec, request_queue, max_batch_size, batch_timeout, transport)
                 _ = event_loop.create_task(future)
                 while True:
                     try:
@@ -176,7 +175,7 @@ class _BaseLoop(ABC):
                             device,
                             worker_id,
                             request_queue,
-                            response_queues,
+                            transport,
                             max_batch_size,
                             batch_timeout,
                             stream,
@@ -196,7 +195,7 @@ class _BaseLoop(ABC):
                     device,
                     worker_id,
                     request_queue,
-                    response_queues,
+                    transport,
                     max_batch_size,
                     batch_timeout,
                     stream,
@@ -211,7 +210,7 @@ class _BaseLoop(ABC):
         device: str,
         worker_id: int,
         request_queue: Queue,
-        response_queues: List[Queue],
+        transport: MessageTransport,
         max_batch_size: int,
         batch_timeout: float,
         stream: bool,
@@ -223,9 +222,7 @@ class _BaseLoop(ABC):
 class LitLoop(_BaseLoop):
     def __init__(self):
-        self.producer: Optional[Producer] = None
         self._context = {}
-        self._setup_signal_handlers()
     def get_batch_requests(self, lit_api: LitAPI, request_queue: Queue, max_batch_size: int, batch_timeout: float):
         batches, timed_out_uids = collate_requests(
@@ -247,32 +244,15 @@ class LitLoop(_BaseLoop):
             lit_spec.populate_context(self._context, request)
     def put_response(
-        self, response_queues: List[Queue], response_queue_id: int, uid: str, response_data: Any, status: LitAPIStatus
+        self, transport: MessageTransport, response_queue_id: int, uid: str, response_data: Any, status: LitAPIStatus
     ) -> None:
-        if self.producer:
-            self.producer.put((uid, (response_data, status)), consumer_id=response_queue_id)
-        else:
-            response_queues[response_queue_id].put((uid, (response_data, status)), block=False)
+        transport.send((uid, (response_data, status)), consumer_id=response_queue_id)
     def put_error_response(
-        self, response_queues: List[Queue], response_queue_id: int, uid: str, error: Exception
+        self, transport: MessageTransport, response_queue_id: int, uid: str, error: Exception
     ) -> None:
         error = pickle.dumps(error)
-        self.put_response(response_queues, response_queue_id, uid, error, LitAPIStatus.ERROR)
-    def __del__(self):
-        if self.producer:
-            self.producer.close()
-    def _setup_signal_handlers(self):
-        def cleanup_handler(signum=None, frame=None):
-            logging.debug("Worker process received shutdown signal")
-            if self.producer:
-                self.producer.close()
-            sys.exit(0)
-        signal.signal(signal.SIGINT, cleanup_handler)
-        signal.signal(signal.SIGTERM, cleanup_handler)
+        self.put_response(transport, response_queue_id, uid, error, LitAPIStatus.ERROR)
 class DefaultLoop(LitLoop):

{litserve-0.2.7 → litserve-0.2.8.dev0}/src/litserve/loops/continuous_batching_loop.py RENAMED Viewed

@@ -23,6 +23,7 @@ from litserve import LitAPI
 from litserve.callbacks import CallbackRunner
 from litserve.loops.base import LitLoop
 from litserve.specs.base import LitSpec
+from litserve.transport.base import MessageTransport
 from litserve.utils import LitAPIStatus
 logger = logging.getLogger(__name__)
@@ -185,7 +186,7 @@ requires the lit_api to have a has_finished method. Please implement the has_fin
         device: str,
         worker_id: int,
         request_queue: Queue,
-        response_queues: List[Queue],
+        transport: MessageTransport,
         max_batch_size: int,
         batch_timeout: float,
         stream: bool,
@@ -214,19 +215,19 @@ requires the lit_api to have a has_finished method. Please implement the has_fin
                     response_data = lit_api.format_encoded_response(response_data)
                     if status == LitAPIStatus.ERROR:
-                        self.put_error_response(response_queues, response_queue_id, uid, response_data)
+                        self.put_error_response(transport, response_queue_id, uid, response_data)
                         self.mark_completed(uid)
                     elif status == LitAPIStatus.FINISH_STREAMING:
-                        self.put_response(response_queues, response_queue_id, uid, response_data, status)
+                        self.put_response(transport, response_queue_id, uid, response_data, status)
                         self.mark_completed(uid)
                     else:
-                        self.put_response(response_queues, response_queue_id, uid, response_data, status)
+                        self.put_response(transport, response_queue_id, uid, response_data, status)
         except Exception as e:
             logger.exception(f"Error in continuous batching loop: {e}")
             # Handle any errors by sending error responses for all tracked requests
             for uid, response_queue_id in self.response_queue_ids.items():
-                self.put_error_response(response_queues, response_queue_id, uid, e)
+                self.put_error_response(transport, response_queue_id, uid, e)
             self.response_queue_ids.clear()
             self.active_sequences.clear()

{litserve-0.2.7 → litserve-0.2.8.dev0}/src/litserve/loops/loops.py RENAMED Viewed

@@ -13,7 +13,7 @@
 # limitations under the License.
 import logging
 from queue import Queue
-from typing import Dict, List, Optional, Union
+from typing import Dict, Optional, Union
 from litserve import LitAPI
 from litserve.callbacks import CallbackRunner, EventTypes
@@ -21,8 +21,8 @@ from litserve.loops.base import _BaseLoop
 from litserve.loops.simple_loops import BatchedLoop, SingleLoop
 from litserve.loops.streaming_loops import BatchedStreamingLoop, StreamingLoop
 from litserve.specs.base import LitSpec
+from litserve.transport.base import MessageTransport
 from litserve.utils import WorkerSetupStatus
-from litserve.zmq_queue import Producer
 logger = logging.getLogger(__name__)
@@ -45,15 +45,13 @@ def inference_worker(
     device: str,
     worker_id: int,
     request_queue: Queue,
-    response_queues: List[Queue],
+    transport: MessageTransport,
     max_batch_size: int,
     batch_timeout: float,
     stream: bool,
     workers_setup_status: Dict[int, str],
     callback_runner: CallbackRunner,
     loop: Union[str, _BaseLoop],
-    use_zmq: bool,
-    zmq_addr: Optional[str],
 ):
     callback_runner.trigger_event(EventTypes.BEFORE_SETUP, lit_api=lit_api)
     try:
@@ -76,18 +74,13 @@ def inference_worker(
     if loop == "auto":
         loop = get_default_loop(stream, max_batch_size)
-    if use_zmq:
-        producer = Producer(address=zmq_addr)
-        producer.wait_for_subscribers(timeout=5)
-        loop.producer = producer
     loop(
         lit_api,
         lit_spec,
         device,
         worker_id,
         request_queue,
-        response_queues,
+        transport,
         max_batch_size,
         batch_timeout,
         stream,

litserve 0.2.7__tar.gz → 0.2.8.dev0__tar.gz

litserve 0.2.7tar.gz → 0.2.8.dev0tar.gz