llm-ie 0.1.4__tar.gz → 0.1.6__tar.gz

{llm_ie-0.1.4 → llm_ie-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: llm-ie
-Version: 0.1.4
+Version: 0.1.6
 Summary: An LLM-powered tool that transforms everyday language into robust information extraction pipelines.
 License: MIT
 Author: Enshuo (David) Hsu
@@ -37,7 +37,7 @@ LLM-IE is a toolkit that provides robust information extraction utilities for fr
 <div align="center"><img src="doc_asset/readme_img/LLM-IE flowchart.png" width=800 ></div>
 
 ## Prerequisite
-At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), and <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction). For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
+At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction), and <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM. For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
 
 ## Installation
 The Python package is available on PyPI. 
@@ -92,6 +92,26 @@ from llm_ie.engines import OpenAIInferenceEngine
 llm = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+</details>
+
+<details>
+<summary><img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM</summary>
+
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct
+```
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="EMPTY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+
+
 </details>
 
 In this quick start demo, we use Llama-cpp-python to run Llama-3.1-8B with int8 quantization ([bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF](https://huggingface.co/bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF)). 
@@ -244,6 +264,24 @@ from llm_ie.engines import OpenAIInferenceEngine
 openai_engine = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+#### <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+CUDA_VISIBLE_DEVICES=<GPU#> vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --api-key MY_API_KEY --tensor-parallel-size <# of GPUs to use>
+```
+Use ```CUDA_VISIBLE_DEVICES``` to specify GPUs to use. The ```--tensor-parallel-size``` should be set accordingly. The ```--api-key``` is optional. 
+the default port is 8000. ```--port``` sets the port. 
+
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="MY_API_KEY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+The ```model``` must match the repo name specified in the server.
 
 #### Test inference engine configuration
 To test the inference engine, use the ```chat()``` method. 

{llm_ie-0.1.4 → llm_ie-0.1.6}/README.md

@@ -23,7 +23,7 @@ LLM-IE is a toolkit that provides robust information extraction utilities for fr
 <div align="center"><img src="doc_asset/readme_img/LLM-IE flowchart.png" width=800 ></div>
 
 ## Prerequisite
-At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), and <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction). For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
+At least one LLM inference engine is required. There are built-in supports for 🦙 [Llama-cpp-python](https://github.com/abetlen/llama-cpp-python), <img src="https://avatars.githubusercontent.com/u/151674099?s=48&v=4" alt="Icon" width="20"/> [Ollama](https://github.com/ollama/ollama), 🤗 [Huggingface_hub](https://github.com/huggingface/huggingface_hub), <img src=doc_asset/readme_img/openai-logomark.png width=16 /> [OpenAI API](https://platform.openai.com/docs/api-reference/introduction), and <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM. For installation guides, please refer to those projects. Other inference engines can be configured through the [InferenceEngine](src/llm_ie/engines.py) abstract class. See [LLM Inference Engine](#llm-inference-engine) section below.
 
 ## Installation
 The Python package is available on PyPI. 
@@ -78,6 +78,26 @@ from llm_ie.engines import OpenAIInferenceEngine
 llm = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+</details>
+
+<details>
+<summary><img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM</summary>
+
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct
+```
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="EMPTY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+
+
 </details>
 
 In this quick start demo, we use Llama-cpp-python to run Llama-3.1-8B with int8 quantization ([bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF](https://huggingface.co/bullerwins/Meta-Llama-3.1-8B-Instruct-GGUF)). 
@@ -230,6 +250,24 @@ from llm_ie.engines import OpenAIInferenceEngine
 openai_engine = OpenAIInferenceEngine(model="gpt-4o-mini")
 ```
 
+#### <img src=doc_asset/readme_img/vllm-logo.png width=20 /> vLLM
+The vLLM support follows the [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). For more parameters, please refer to the documentation.
+
+Start the server
+```cmd
+CUDA_VISIBLE_DEVICES=<GPU#> vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --api-key MY_API_KEY --tensor-parallel-size <# of GPUs to use>
+```
+Use ```CUDA_VISIBLE_DEVICES``` to specify GPUs to use. The ```--tensor-parallel-size``` should be set accordingly. The ```--api-key``` is optional. 
+the default port is 8000. ```--port``` sets the port. 
+
+Define inference engine
+```python
+from llm_ie.engines import OpenAIInferenceEngine
+engine = OpenAIInferenceEngine(base_url="http://localhost:8000/v1",
+                               api_key="MY_API_KEY",
+                               model="meta-llama/Meta-Llama-3.1-8B-Instruct")
+```
+The ```model``` must match the repo name specified in the server.
 
 #### Test inference engine configuration
 To test the inference engine, use the ```chat()``` method. 

{llm_ie-0.1.4 → llm_ie-0.1.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "llm-ie"
-version = "0.1.4"
+version = "0.1.6"
 description = "An LLM-powered tool that transforms everyday language into robust information extraction pipelines."
 authors = ["Enshuo (David) Hsu"]
 license = "MIT"

{llm_ie-0.1.4 → llm_ie-0.1.6}/src/llm_ie/data_types.py

@@ -3,7 +3,7 @@ import yaml
 
 
 class LLMInformationExtractionFrame:
-    def __init__(self, frame_id:str, start:int, end:int, entity_text:str, attr:Dict[str,str]):
+    def __init__(self, frame_id:str, start:int, end:int, entity_text:str, attr:Dict[str,str]=None):
         """
         This class holds a frame (entity) extracted by LLM. 
         A frame contains the span (start and end character positions), a entity text, and 
@@ -19,7 +19,7 @@ class LLMInformationExtractionFrame:
             entity end character position
         entity_text : str
             entity string. Should be the exact string by [start:end]
-        attr : Dict[str,str]
+        attr : Dict[str,str], Optional
             dict of attributes
         """
         assert isinstance(frame_id, str), "frame_id must be a string."
@@ -27,7 +27,10 @@ class LLMInformationExtractionFrame:
         self.start = start
         self.end = end
         self.entity_text = entity_text
-        self.attr = attr.copy()
+        if attr:
+            self.attr = attr.copy()
+        else:
+            self.attr = None
 
     def is_equal(self, frame:"LLMInformationExtractionFrame") -> bool:
         """ 

{llm_ie-0.1.4 → llm_ie-0.1.6}/src/llm_ie/engines.py

@@ -246,4 +246,4 @@ class OpenAIInferenceEngine(InferenceEngine):
                     print(chunk.choices[0].delta.content, end="")
             return res
         
-        return response.choices[0].delta.content
+        return response.choices[0].message.content

{llm_ie-0.1.4 → llm_ie-0.1.6}/src/llm_ie/extractors.py

@@ -197,9 +197,13 @@ class BasicFrameExtractor(FrameExtractor):
         Return : str
             the output from LLM. Need post-processing.
         """
+        messages = []
+        if self.system_prompt:
+            messages.append({'role': 'system', 'content': self.system_prompt})
+
+        messages.append({'role': 'user', 'content': self._get_user_prompt(text_content)})
         response = self.inference_engine.chat(
-                    messages=[{'role': 'system', 'content': self.system_prompt},
-                              {'role': 'user', 'content': self._get_user_prompt(text_content)}],
+                    messages=messages,
                     max_new_tokens=max_new_tokens, 
                     temperature=temperature,
                     stream=stream,
@@ -312,9 +316,11 @@ class ReviewFrameExtractor(BasicFrameExtractor):
             the output from LLM. Need post-processing.
         """
         # Pormpt extraction
-        messages=[{'role': 'system', 'content': self.system_prompt},
-                  {'role': 'user', 'content': self._get_user_prompt(text_content)}]
-        
+        messages = []
+        if self.system_prompt:
+            messages.append({'role': 'system', 'content': self.system_prompt})
+
+        messages.append({'role': 'user', 'content': self._get_user_prompt(text_content)})
         initial = self.inference_engine.chat(
                         messages=messages,
                         max_new_tokens=max_new_tokens,