llm-ie 0.1.5__py3-none-any.whl → 0.1.6__py3-none-any.whl

llm_ie/engines.py

@@ -246,4 +246,4 @@ class OpenAIInferenceEngine(InferenceEngine):
                     print(chunk.choices[0].delta.content, end="")
             return res
         
-        return response.choices[0].delta.content
+        return response.choices[0].message.content

{llm_ie-0.1.5.dist-info → llm_ie-0.1.6.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: llm-ie
-Version: 0.1.5
+Version: 0.1.6
 Summary: An LLM-powered tool that transforms everyday language into robust information extraction pipelines.
 License: MIT
 Author: Enshuo (David) Hsu
@@ -37,7 +37,7 @@ LLM-IE is a toolkit that provides robust information extraction utilities for fr
 <div align="center"><img src="doc_asset/readme_img/LLM-IE flowchart.png" width=800 ></div>
 
 ## Prerequisite
-At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), and <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction). For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
+At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction), and <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM. For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
 
 ## Installation
 The Python package is available on PyPI. 
@@ -92,6 +92,26 @@ from llm_ie.engines import OpenAIInferenceEngine
 llm = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+</details>
+
+<details>
+<summary><img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM</summary>
+
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct
+```
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="EMPTY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+
+
 </details>
 
 In this quick start demo, we use Llama-cpp-python to run Llama-3.1-8B with int8 quantization ([bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF](https://huggingface.co/bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF)). 
@@ -244,6 +264,24 @@ from llm_ie.engines import OpenAIInferenceEngine
 openai_engine = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+#### <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+CUDA_VISIBLE_DEVICES=<GPU#> vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --api-key MY_API_KEY --tensor-parallel-size <# of GPUs to use>
+```
+Use ```CUDA_VISIBLE_DEVICES``` to specify GPUs to use. The ```--tensor-parallel-size``` should be set accordingly. The ```--api-key``` is optional. 
+the default port is 8000. ```--port``` sets the port. 
+
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="MY_API_KEY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+The ```model``` must match the repo name specified in the server.
 
 #### Test inference engine configuration
 To test the inference engine, use the ```chat()``` method. 

{llm_ie-0.1.5.dist-info → llm_ie-0.1.6.dist-info}/RECORD

@@ -5,9 +5,9 @@ llm_ie/asset/prompt_guide/BasicFrameExtractor_prompt_guide.txt,sha256=XbnU8byLGG
 llm_ie/asset/prompt_guide/ReviewFrameExtractor_prompt_guide.txt,sha256=XbnU8byLGGUA3A3lT0bb2Hw-ggzhcqD3ZuKzduod2ww,1944
 llm_ie/asset/prompt_guide/SentenceFrameExtractor_prompt_guide.txt,sha256=8nj9OLPJMtr9Soi5JU3Xk-HC7pKNoI54xA_A4u7I5j4,2620
 llm_ie/data_types.py,sha256=MnpyXFviFWhxeC5mqbaPdAxGx6vV_PhnUIFfUamq3D8,6687
-llm_ie/engines.py,sha256=TuxM56_u6-dsAAuNdfuKSH23nb9UfFbg6T60e-OXEA8,9294
+llm_ie/engines.py,sha256=m9ytGUX61jEy9SmVHbb90mrfGMAwC6dV-v7Jke1U7Ho,9296
 llm_ie/extractors.py,sha256=rpHJhlV3A9-9nldIutxd8rtgf7903Ke6QkwbCIVdUdY,22546
 llm_ie/prompt_editor.py,sha256=dbu7A3O7O7Iw2v-xCgrTFH1-wTLAGf4SHDqdeS-He2Q,1869
-llm_ie-0.1.5.dist-info/METADATA,sha256=2z2VDBlBq7GLws7aEcyjRmw0NC3gvKJIXS4LB6AaeI4,28028
-llm_ie-0.1.5.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
-llm_ie-0.1.5.dist-info/RECORD,,
+llm_ie-0.1.6.dist-info/METADATA,sha256=xD_BHcUAirE7BZJ2wQEaTQUlmzDRo4Yz8Ztr-Gpfivk,29712
+llm_ie-0.1.6.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
+llm_ie-0.1.6.dist-info/RECORD,,