litgpt 0.4.0.dev0__tar.gz → 0.4.2__tar.gz

{litgpt-0.4.0.dev0 → litgpt-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: litgpt
-Version: 0.4.0.dev0
+Version: 0.4.2
 Summary: Hackable implementation of state-of-the-art open-source LLMs
 Author-email: Lightning AI <contact@lightning.ai>
 License:                                  Apache License
@@ -220,15 +220,16 @@ Requires-Dist: pytest-dependency>=0.6.0; extra == "test"
 Requires-Dist: transformers>=4.38.0; extra == "test"
 Requires-Dist: einops>=0.7.0; extra == "test"
 Requires-Dist: protobuf>=4.23.4; extra == "test"
-Requires-Dist: lightning-thunder==0.2.0.dev20240505; python_version >= "3.10" and extra == "test"
+Requires-Dist: lightning-thunder==0.2.0.dev20240623; python_version >= "3.10" and extra == "test"
 Provides-Extra: all
 Requires-Dist: bitsandbytes==0.42.0; extra == "all"
 Requires-Dist: sentencepiece>=0.2.0; extra == "all"
 Requires-Dist: tokenizers>=0.15.2; extra == "all"
 Requires-Dist: requests>=2.31.0; extra == "all"
 Requires-Dist: litdata==0.2.6; extra == "all"
-Requires-Dist: litserve==0.1.1dev0; extra == "all"
+Requires-Dist: litserve>=0.1.2; extra == "all"
 Requires-Dist: zstandard>=0.22.0; extra == "all"
+Requires-Dist: numpy<2.0.0; extra == "all"
 Requires-Dist: pandas>=1.9.0; extra == "all"
 Requires-Dist: pyarrow>=15.0.2; extra == "all"
 Requires-Dist: tensorboard>=2.14.0; extra == "all"
@@ -244,13 +245,13 @@ Requires-Dist: huggingface_hub[hf_transfer]>=0.21.0; extra == "all"
 
 # ⚡ LitGPT
 
-**Pretrain, finetune, evaluate, and deploy 20+ LLMs on your own data**
+**Load, finetune, pretrain, evaluate, and deploy 20+ LLMs on your own data**
 
 Uses the latest state-of-the-art techniques:
 
 <pre>
-✅ flash attention    ✅ fp4/8/16/32        ✅ LoRA, QLoRA, Adapter
-✅ FSDP               ✅ 1-1000+ GPUs/TPUs  ✅ 20+ LLMs            
+✅ Scratch implementations  ✅ flash attention  ✅ fp4/8/16/32        ✅ LoRA, QLoRA, Adapter
+✅ No abstractions          ✅ FSDP             ✅ 1-1000+ GPUs/TPUs  ✅ 20+ LLMs            
 </pre>
 
 
@@ -262,14 +263,13 @@ Uses the latest state-of-the-art techniques:
 
 <p align="center">
   <a href="https://lightning.ai/">Lightning AI</a> •
-  <a href="#choose-from-20-llms">Models</a> •
   <a href="#quick-start">Quick start</a> •
-  <a href="#use-an-llm-for-inference">Inference</a> •
-  <a href="#finetune-an-llm">Finetune</a> •
-  <a href="#finetune-an-llm">Pretrain</a> •
+  <a href="#choose-from-20-llms">Models</a> •
+  <a href="#finetune-an-llm">Finetune/pretrain</a> •   
     <a href="#deploy-an-llm">Deploy</a> •
   <a href="#state-of-the-art-features">Features</a> •
-  <a href="#training-recipes">Training recipes (YAML)</a>
+  <a href="#training-recipes">Training recipes (YAML)</a> •
+    <a href="#tutorials">Tutorials</a>
 </p>
 
 </div>
@@ -278,15 +278,63 @@ Uses the latest state-of-the-art techniques:
 <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/GithubLitGPTDAG2.png" alt="LitGPT steps" width="auto"/>
 &nbsp;
 
-# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡
-LitGPT is a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and [deploy](#deploy-an-llm) [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large language models (LLMs).
+# Load, finetune, pretrain, deploy LLMs Lightning fast ⚡⚡
+LitGPT is a library of **lightning-fast** large language model (LLMs) **implemented from scratch** (Apache 2.0) with **no abstractions**.   
 
 We reimplemented all model architectures and training recipes from scratch for 4 reasons:
 
-1. Remove all abstraction layers and have single file implementations.
-2. Guarantee Apache 2.0 compliance to enable enterprise use without limits.
-3. Optimized each model's architectural detail to maximize performance, reduce costs, and speed up training.
-4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale.
+✅ Apache 2.0 compliance to enable unlimited enterprise use.    
+✅ Easy debugging/hacking with no abstraction layers and single file implementations.    
+✅ Optimized model architectures to maximize performance, reduce costs, and speed up training.    
+✅ Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale.    
+
+In addition to a simple Python API, it offers a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and [deploy](#deploy-an-llm) [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large language models (LLMs).
+
+&nbsp;
+
+# Quick start
+Install LitGPT
+```
+pip install 'litgpt[all]'
+```
+
+Load and use any of the [20+ LLMs](#choose-from-20-llms):   
+```python
+from litgpt import LLM
+
+llm = LLM.load("microsoft/phi-2")
+text = llm.generate("Correct the spelling: Every summer, the familly enjoys a trip to the mountains.")
+print(text)
+# Corrected Sentence: Every summer, the family enjoys a vacation to the mountains.       
+```
+
+> [!NOTE]
+> **[Explore the Python API options](tutorials/python-api.md)**.
+
+&nbsp;
+&nbsp;
+
+✅ Optimized for fast inference    
+✅ Quantization    
+✅ Runs on low-memory GPUs    
+✅ No layers of internal abstractions    
+✅ Optimized for production scale   
+
+&nbsp;
+
+<details>
+  <summary>Advanced install options</summary>
+
+&nbsp;
+
+Install from source:
+
+```bash
+git clone https://github.com/Lightning-AI/litgpt
+cd litgpt
+pip install -e '.[all]'
+```
+</details>
 
 ---
 
@@ -345,40 +393,21 @@ LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials
 
 </details>
 
-&nbsp;
-
-## Install LitGPT
-
-Install LitGPT with all dependencies (including CLI, quantization, tokenizers for all models, etc.):
-
-```bash
-pip install 'litgpt[all]'
-```
-
-<details>
-  <summary>Advanced install options</summary>
+---
 
 &nbsp;
 
-Install from source:
-
-```bash
-git clone https://github.com/Lightning-AI/litgpt
-cd litgpt
-pip install -e '.[all]'
-```
-</details>
-
----
+# Advanced workflows
+Use the command line interface to run advanced workflows such as pretraining or finetuning on your own data.   
 
-&nbsp;
-# Quick start
+## All commands   
 After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...):
 
 ```bash
 # ligpt [action] [model]
 litgpt  download  meta-llama/Meta-Llama-3-8B-Instruct
 litgpt  chat      meta-llama/Meta-Llama-3-8B-Instruct
+litgpt  evaluate  meta-llama/Meta-Llama-3-8B-Instruct
 litgpt  finetune  meta-llama/Meta-Llama-3-8B-Instruct
 litgpt  pretrain  meta-llama/Meta-Llama-3-8B-Instruct
 litgpt  serve     meta-llama/Meta-Llama-3-8B-Instruct
@@ -386,34 +415,6 @@ litgpt  serve     meta-llama/Meta-Llama-3-8B-Instruct
 
 &nbsp;
 
-###  Use an LLM for inference
-Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc.
-Here's an example showing how to use the Phi-2 LLM.
-
-<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-chat">
-  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
-</a>
-
-&nbsp;
-
-```bash
-# 1) List all available models in litgpt
-litgpt download list
-
-# 2) Download a pretrained model
-litgpt download microsoft/phi-2
-
-# 3) Chat with the model
-litgpt chat microsoft/phi-2
-
->> Prompt: What do Llamas eat?
-```
-
-The download of certain models requires an additional access token. You can read more about this in the [download](tutorials/download_model_weights.md#specific-models-and-access-tokens) documentation. 
-For more information on the different inference options, refer to the [inference](tutorials/inference.md) tutorial.
-
-&nbsp;
-
 ### Finetune an LLM
 [Finetune](tutorials/finetune.md) a model to specialize it on your own custom dataset:
 
@@ -443,7 +444,8 @@ litgpt chat out/custom-model/final
 &nbsp;
 
 ### Pretrain an LLM
-Train an LLM from scratch on your own data via pretraining:
+
+[Train an LLM from scratch](tutorials/pretrain.md) on your own data via pretraining:
 
 <a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-pretrain">
 <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg"; alt="Open In Studio"/>
@@ -475,7 +477,8 @@ litgpt chat out/custom-model/final
 &nbsp;
 
 ### Continue pretraining an LLM
-This is another way of finetuning that specializes an already pretrained model by training on custom data:
+
+[Continued pretraining](tutorials/pretrain.md#continued-pretraining-on-custom-data) is another way of finetuning that specializes an already pretrained model by training on custom data:
 
 
 <a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-continue-pretraining">
@@ -507,8 +510,21 @@ litgpt chat out/custom-model/final
 
 &nbsp;
 
+### Evaluate an LLM
+
+If you want to [evaluate](tutorials/evaluation.md) a downloaded, finetuned, or pretrained LLM on popular benchmark tasks, such as MMLU and Truthful QA, run the following command:
+
+```bash
+litgpt evaluate microsoft/phi-2 --tasks 'truthfulqa_mc2,mmlu'
+```
+
+> [!NOTE]
+> **[Read the evaluation docs](tutorials/evaluation.md)** for more options.
+
+&nbsp;
+
 ### Deploy an LLM
-Once you're ready to deploy a finetuned LLM, run this command:
+Once you're ready to [deploy](tutorials/deploy.md) a finetuned LLM, run this command:
 
 <a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-serve">
   <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
@@ -527,25 +543,55 @@ litgpt serve microsoft/phi-2
 
 Test the server in a separate terminal and integrate the model API into your AI product:
 ```python
-# 3) Use the server (in a separate session)
+# 3) Use the server (in a separate Python session)
 import requests, json
- response = requests.post(
-     "http://127.0.0.1:8000/predict",
-     json={"prompt": "Fix typos in the following sentence: Exampel input"}
+response = requests.post(
+    "http://127.0.0.1:8000/predict",
+    json={"prompt": "Fix typos in the following sentence: Exampel input"}
 )
 print(response.json()["output"])
 ```
 
-&nbsp;
 
 > [!NOTE]
-> **[Read the full docs](tutorials/0_to_litgpt.md)**.
+> **[Read the full docs](tutorials/deploy.md)**.
 
 &nbsp;
 
 ----
 
+###  Use an LLM for inference
+
+Use LLMs for [inference](tutorials/deploy.md) to test its chatting capabilities, run evaluations, or extract embeddings, etc.
+Here's an example showing how to use the Phi-2 LLM.
+
+<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litgpt-chat">
+  <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
+</a>
+
+&nbsp;
+
+```bash
+# 1) List all available models in litgpt
+litgpt download list
+
+# 2) Download a pretrained model
+litgpt download microsoft/phi-2
+
+# 3) Chat with the model
+litgpt chat microsoft/phi-2
+
+>> Prompt: What do Llamas eat?
+```
+
+The download of certain models requires an additional access token. You can read more about this in the [download](tutorials/download_model_weights.md#specific-models-and-access-tokens) documentation. 
+For more information on the different inference options, refer to the [inference](tutorials/inference.md) tutorial.
+
+----
+&nbsp;
+
 # State-of-the-art features
+
 ✅ &nbsp;State-of-the-art optimizations: Flash Attention v2, multi-GPU support via fully-sharded data parallelism, [optional CPU offloading](tutorials/oom.md#do-sharding-across-multiple-gpus), and [TPU and XLA support](extensions/xla).
 
 ✅ &nbsp;[Pretrain](tutorials/pretrain.md), [finetune](tutorials/finetune.md), and [deploy](tutorials/inference.md)
@@ -580,8 +626,9 @@ Browse all training recipes [here](config_hub).
 litgpt finetune \
   --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml
 ```
-
-### What is a config
+<details>
+  <summary>✅ Use configs to customize training</summary>
+  
 Configs let you customize training for all granular parameters like:
 
 ```yaml
@@ -596,9 +643,10 @@ precision: bf16-true
 
 ...
 ```
+</details>
 
 <details>
-  <summary>Example: LoRA finetuning config</summary>
+  <summary>✅ Example: LoRA finetuning config</summary>
 
 &nbsp;
 
@@ -728,82 +776,51 @@ seed: 1337
 ```
 </details>
 
-### Override config params via CLI
-Override any parameter in the CLI:
+<details>
+  <summary>✅ Override any parameter in the CLI:</summary>
 
 ```bash
 litgpt finetune \
   --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml \
   --lora_r 4
 ```
+</details>
 
 &nbsp;
 
 # Community
 
-## Get involved!
-
-We appreciate your feedback and contributions. If you have feature requests, questions, or want to contribute code or config files, please don't hesitate to use the [GitHub Issue](https://github.com/Lightning-AI/litgpt/issues) tracker.
-
 We welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.
 
-&nbsp;
-
-> [!TIP]
-> Unsure about contributing? Check out our [How to Contribute to LitGPT](https://lightning.ai/pages/community/tutorial/how-to-contribute-to-litgpt/) guide.
-
-If you have general questions about building with LitGPT, please [join our Discord](https://discord.gg/VptPCZkGNa).
-
+- [Request a feature](https://github.com/Lightning-AI/litgpt/issues)    
+- [Submit your first contribution](https://lightning.ai/pages/community/tutorial/how-to-contribute-to-litgpt/)    
+- [Join our Discord](https://discord.gg/VptPCZkGNa)    
 
 &nbsp;
 
-## Tutorials, how-to guides, and docs
-
-
-> [!NOTE]
-> We recommend starting with the **[Zero to LitGPT: Getting Started with Pretraining, Finetuning, and Using LLMs](tutorials/0_to_litgpt.md)** if you are looking to get started with using LitGPT.
-
-Tutorials and in-depth feature documentation can be found below:
+# Tutorials   
 
--  Finetuning, incl. LoRA, QLoRA, and Adapters ([tutorials/finetune.md](tutorials/finetune.md))
--  Pretraining ([tutorials/pretrain.md](tutorials/pretrain.md))
--  Model evaluation ([tutorials/evaluation.md](tutorials/evaluation.md))
--  Supported and custom datasets ([tutorials/prepare_dataset.md](tutorials/prepare_dataset.md))
--  Quantization ([tutorials/quantize.md](tutorials/quantize.md))
--  Tips for dealing with out-of-memory (OOM) errors ([tutorials/oom.md](tutorials/oom.md))
+🚀 [Get started](tutorials/0_to_litgpt.md)    
+⚡️  [Finetuning, incl. LoRA, QLoRA, and Adapters](tutorials/finetune.md)    
+🤖 [Pretraining](tutorials/pretrain.md)    
+💬 [Model evaluation](tutorials/evaluation.md)    
+📘 [Supported and custom datasets](tutorials/prepare_dataset.md)    
+🧹 [Quantization](tutorials/quantize.md)    
+🤯 [Tips for dealing with out-of-memory (OOM) errors](tutorials/oom.md)   
+🧑🏽‍💻 [Using cloud TPUs](extensions/xla)
 
 &nbsp;
 
-## XLA
-
-Lightning AI has partnered with Google to add first-class support for [Cloud TPUs](https://cloud.google.com/tpu) in [Lightning's frameworks](https://github.com/Lightning-AI/lightning) and LitGPT,
-helping democratize AI for millions of developers and researchers worldwide.
-
-Using TPUs with Lightning is as straightforward as changing one line of code.
-
-We provide scripts fully optimized for TPUs in the [XLA directory](extensions/xla).
 
+## Projects using LitGPT
 
+Check out the projects below that use and build on LitGPT. If you have a project you'd like to add to this section, please don't hesitate to open a pull request.
 
 &nbsp;
 
-## Acknowledgements
-
-This implementation extends on [Lit-LLaMA](https://github.com/lightning-AI/lit-llama) and [nanoGPT](https://github.com/karpathy/nanoGPT), and it's **powered by [Lightning Fabric](https://lightning.ai/docs/fabric/stable/) ⚡**.
-
-- [@karpathy](https://github.com/karpathy) for [nanoGPT](https://github.com/karpathy/nanoGPT)
-- [@EleutherAI](https://github.com/EleutherAI) for [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) and the [Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
-- [@TimDettmers](https://github.com/TimDettmers) for [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
-- [@Microsoft](https://github.com/microsoft) for [LoRA](https://github.com/microsoft/LoRA)
-- [@tridao](https://github.com/tridao) for [Flash Attention 2](https://github.com/Dao-AILab/flash-attention)
+📊 **SAMBA: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling**
 
-
-&nbsp;
-
-
-## Community showcase
-
-Check out the projects below that use and build on LitGPT. If you have a project you'd like to add to this section, please don't hesitate to open a pull request.
+The [Samba](https://github.com/microsoft/Samba) project by researchers at Microsoft is built on top of the LitGPT code base and combines state space models with sliding window attention, which outperforms pure state space models.
 
 &nbsp;
 
@@ -831,6 +848,23 @@ The research paper ["Pre-training Small Base LMs with Fewer Tokens"](https://arx
 
 &nbsp;
 
+## Acknowledgements
+
+This implementation extends on [Lit-LLaMA](https://github.com/lightning-AI/lit-llama) and [nanoGPT](https://github.com/karpathy/nanoGPT), and it's **powered by [Lightning Fabric](https://lightning.ai/docs/fabric/stable/) ⚡**.
+
+- [@karpathy](https://github.com/karpathy) for [nanoGPT](https://github.com/karpathy/nanoGPT)
+- [@EleutherAI](https://github.com/EleutherAI) for [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) and the [Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
+- [@TimDettmers](https://github.com/TimDettmers) for [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
+- [@Microsoft](https://github.com/microsoft) for [LoRA](https://github.com/microsoft/LoRA)
+- [@tridao](https://github.com/tridao) for [Flash Attention 2](https://github.com/Dao-AILab/flash-attention)
+
+&nbsp;
+
+## License
+
+LitGPT is released under the [Apache 2.0](https://github.com/Lightning-AI/litgpt/blob/main/LICENSE) license.
+
+
 ## Citation
 
 If you use LitGPT in your research, please cite the following work:
@@ -845,7 +879,3 @@ If you use LitGPT in your research, please cite the following work:
 ```
 
 &nbsp;
-
-## License
-
-LitGPT is released under the [Apache 2.0](https://github.com/Lightning-AI/litgpt/blob/main/LICENSE) license.