langroid 0.1.133__tar.gz → 0.1.135__tar.gz

{langroid-0.1.133 → langroid-0.1.135}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: langroid
-Version: 0.1.133
+Version: 0.1.135
 Summary: Harness LLMs with Multi-Agent Programming
 License: MIT
 Author: Prasad Chalasani
@@ -100,7 +100,7 @@ Description-Content-Type: text/markdown
 [![Multi-Architecture DockerHub](https://github.com/langroid/langroid/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/langroid/langroid/actions/workflows/docker-publish.yml)
 
 [![Static Badge](https://img.shields.io/badge/Documentation-blue?link=https%3A%2F%2Flangroid.github.io%2Flangroid%2F&link=https%3A%2F%2Flangroid.github.io%2Flangroid%2F)](https://langroid.github.io/langroid)
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb)
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
 [![Discord](https://img.shields.io/badge/Discord-%235865F2.svg?style=flat&logo=discord&logoColor=white)](https://discord.gg/ZU36McDgDs)
 [![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?style=flat&logo=substack&logoColor=FF6719)](https://langroid.substack.com/p/langroid-harness-llms-with-multi-agent-programming)
 </div>
@@ -138,7 +138,7 @@ We welcome contributions -- See the [contributions](./CONTRIBUTING.md) document
 for ideas on what to contribute.
 
 
-Building LLM Applications? Prasad Chalasani is available for consulting
+Building LLM Applications? [Prasad Chalasani](https://www.linkedin.com/in/pchalasani/) is available for consulting
 (advisory/development): pchalasani at gmail dot com.
 
 Sponsorship is also accepted via [GitHub Sponsors](https://github.com/sponsors/langroid)
@@ -148,12 +148,12 @@ Sponsorship is also accepted via [GitHub Sponsors](https://github.com/sponsors/l
 # Quick glimpse of coding with Langroid
 This is just a teaser; there's much more, like function-calling/tools, 
 Multi-Agent Collaboration, Structured Information Extraction, DocChatAgent 
-(RAG), SQLChatAgent, etc. Scroll down or see docs for more.
-
-:fire: Just released! See this [Colab](https://colab.research.google.com/drive/190Tk7t4AdY1P9F_NlZ33-YEoGnHweQQ0) 
-for a walk-through of the new `OpenAIAssistant` class (with near-complete support for the new OpenAI Assistants API) 
-in a multi-agent setting.
+(RAG), SQLChatAgent, non-OpenAI local/remote LLMs, etc. Scroll down or see docs for more.
 
+:fire: Just released! Updated Langroid Quick-Start [Colab](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
+that builds up to a 2-agent chat example using the OpenAI ChatCompletion API. 
+See also this [version](https://colab.research.google.com/drive/190Tk7t4AdY1P9F_NlZ33-YEoGnHweQQ0) 
+that uses the OpenAI Assistants API instead.
 
 ```python
 from langroid.language_models import  OpenAIGPTConfig, OpenAIChatModel, OpenAIGPT
@@ -264,7 +264,7 @@ See [this test](tests/main/test_recipient_tool.py) for example usage.
   - **Example:** [Answer questions](examples/docqa/chat-search.py) using Google Search + vecdb-retrieval from URL contents. 
   - **0.1.39:** [`GoogleSearchTool`](langroid/agent/tools/google_search_tool.py) to enable Agents (their LLM) to do Google searches via function-calling/tools.
     See [this chat example](examples/basic/chat-search.py) for how easy it is to add this tool to an agent.
-  - **Colab notebook** to try the quick-start examples: [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb) 
+  - **Colab notebook** to try the quick-start examples: [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb) 
   - **0.1.37:** Added [`SQLChatAgent`](langroid/agent/special/sql_chat_agent.py) -- thanks to our latest contributor [Rithwik Babu](https://github.com/rithwikbabu)!
   - Multi-agent Example: [Autocorrect chat](examples/basic/autocorrect.py)
 - **July 2023:** 
@@ -515,7 +515,7 @@ for a detailed tutorial.
 
 Click to expand any of the code examples below.
 All of these can be run in a Colab notebook:
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb)
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
 
 <details>
 <summary> <b> Direct interaction with OpenAI LLM </b> </summary>

{langroid-0.1.133 → langroid-0.1.135}/README.md

@@ -11,7 +11,7 @@
 [![Multi-Architecture DockerHub](https://github.com/langroid/langroid/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/langroid/langroid/actions/workflows/docker-publish.yml)
 
 [![Static Badge](https://img.shields.io/badge/Documentation-blue?link=https%3A%2F%2Flangroid.github.io%2Flangroid%2F&link=https%3A%2F%2Flangroid.github.io%2Flangroid%2F)](https://langroid.github.io/langroid)
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb)
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
 [![Discord](https://img.shields.io/badge/Discord-%235865F2.svg?style=flat&logo=discord&logoColor=white)](https://discord.gg/ZU36McDgDs)
 [![Substack](https://img.shields.io/badge/Substack-%23006f5c.svg?style=flat&logo=substack&logoColor=FF6719)](https://langroid.substack.com/p/langroid-harness-llms-with-multi-agent-programming)
 </div>
@@ -49,7 +49,7 @@ We welcome contributions -- See the [contributions](./CONTRIBUTING.md) document
 for ideas on what to contribute.
 
 
-Building LLM Applications? Prasad Chalasani is available for consulting
+Building LLM Applications? [Prasad Chalasani](https://www.linkedin.com/in/pchalasani/) is available for consulting
 (advisory/development): pchalasani at gmail dot com.
 
 Sponsorship is also accepted via [GitHub Sponsors](https://github.com/sponsors/langroid)
@@ -59,12 +59,12 @@ Sponsorship is also accepted via [GitHub Sponsors](https://github.com/sponsors/l
 # Quick glimpse of coding with Langroid
 This is just a teaser; there's much more, like function-calling/tools, 
 Multi-Agent Collaboration, Structured Information Extraction, DocChatAgent 
-(RAG), SQLChatAgent, etc. Scroll down or see docs for more.
-
-:fire: Just released! See this [Colab](https://colab.research.google.com/drive/190Tk7t4AdY1P9F_NlZ33-YEoGnHweQQ0) 
-for a walk-through of the new `OpenAIAssistant` class (with near-complete support for the new OpenAI Assistants API) 
-in a multi-agent setting.
+(RAG), SQLChatAgent, non-OpenAI local/remote LLMs, etc. Scroll down or see docs for more.
 
+:fire: Just released! Updated Langroid Quick-Start [Colab](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
+that builds up to a 2-agent chat example using the OpenAI ChatCompletion API. 
+See also this [version](https://colab.research.google.com/drive/190Tk7t4AdY1P9F_NlZ33-YEoGnHweQQ0) 
+that uses the OpenAI Assistants API instead.
 
 ```python
 from langroid.language_models import  OpenAIGPTConfig, OpenAIChatModel, OpenAIGPT
@@ -175,7 +175,7 @@ See [this test](tests/main/test_recipient_tool.py) for example usage.
   - **Example:** [Answer questions](examples/docqa/chat-search.py) using Google Search + vecdb-retrieval from URL contents. 
   - **0.1.39:** [`GoogleSearchTool`](langroid/agent/tools/google_search_tool.py) to enable Agents (their LLM) to do Google searches via function-calling/tools.
     See [this chat example](examples/basic/chat-search.py) for how easy it is to add this tool to an agent.
-  - **Colab notebook** to try the quick-start examples: [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb) 
+  - **Colab notebook** to try the quick-start examples: [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb) 
   - **0.1.37:** Added [`SQLChatAgent`](langroid/agent/special/sql_chat_agent.py) -- thanks to our latest contributor [Rithwik Babu](https://github.com/rithwikbabu)!
   - Multi-agent Example: [Autocorrect chat](examples/basic/autocorrect.py)
 - **July 2023:** 
@@ -426,7 +426,7 @@ for a detailed tutorial.
 
 Click to expand any of the code examples below.
 All of these can be run in a Colab notebook:
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/langroid_quick_examples.ipynb)
+[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb)
 
 <details>
 <summary> <b> Direct interaction with OpenAI LLM </b> </summary>

{langroid-0.1.133 → langroid-0.1.135}/langroid/agent/base.py

@@ -143,6 +143,9 @@ class Agent(ABC):
     def get_dialog(self) -> List[Tuple[str, str]]:
         return self.dialog
 
+    def clear_dialog(self) -> None:
+        self.dialog = []
+
     def _get_tool_list(
         self, message_class: Optional[Type[ToolMessage]] = None
     ) -> List[str]:

{langroid-0.1.133 → langroid-0.1.135}/langroid/agent/task.py

@@ -100,6 +100,7 @@ class Task:
         if isinstance(agent, ChatAgent) and len(agent.message_history) == 0 or restart:
             agent = cast(ChatAgent, agent)
             agent.clear_history(0)
+            agent.clear_dialog()
             # possibly change the system and user messages
             if system_message:
                 # we always have at least 1 task_message

{langroid-0.1.133 → langroid-0.1.135}/langroid/parsing/parser.py

@@ -1,6 +1,6 @@
 import logging
 from enum import Enum
-from typing import List
+from typing import Dict, List
 
 import tiktoken
 from pydantic import BaseSettings
@@ -52,27 +52,42 @@ class Parser:
         return len(tokens)
 
     def add_window_ids(self, chunks: List[Document]) -> None:
-        """Chunks are consecutive parts of a single original document.
-        Add window_ids in metadata"""
+        """Chunks may belong to multiple docs, but for each doc,
+        they appear consecutively. Add window_ids in metadata"""
 
         # The original metadata.id (if any) is ignored since it will be same for all
         # chunks and is useless. We want a distinct id for each chunk.
+        orig_ids = [c.metadata.id for c in chunks]
         ids = [Document.hash_id(str(c)) for c in chunks]
+        id2chunk = {id: c for id, c in zip(ids, chunks)}
+
+        # group the ids by orig_id
+        orig_id_to_ids: Dict[str, List[str]] = {}
+        for orig_id, id in zip(orig_ids, ids):
+            if orig_id not in orig_id_to_ids:
+                orig_id_to_ids[orig_id] = []  # type: ignore
+            orig_id_to_ids[orig_id].append(id)  # type: ignore
+
+        # now each orig_id maps to a sequence of ids within a single doc
 
         k = self.config.n_neighbor_ids
-        n = len(ids)
-        window_ids = [ids[max(0, i - k) : min(n, i + k + 1)] for i in range(n)]
-        for i, c in enumerate(chunks):
-            if c.content.strip() == "":
-                continue
-            c.metadata.window_ids = window_ids[i]
-            c.metadata.id = ids[i]
-            c.metadata.is_chunk = True
+        for orig, ids in orig_id_to_ids.items():
+            # ids are consecutive chunks in a single doc
+            n = len(ids)
+            window_ids = [ids[max(0, i - k) : min(n, i + k + 1)] for i in range(n)]
+            for i, _ in enumerate(ids):
+                c = id2chunk[ids[i]]
+                if c.content.strip() == "":
+                    continue
+                c.metadata.window_ids = window_ids[i]
+                c.metadata.id = ids[i]
+                c.metadata.is_chunk = True
 
     def split_simple(self, docs: List[Document]) -> List[Document]:
         if len(self.config.separators) == 0:
             raise ValueError("Must have at least one separator")
         final_docs = []
+
         for d in docs:
             if d.content.strip() == "":
                 continue
@@ -89,35 +104,35 @@ class Parser:
         return final_docs
 
     def split_para_sentence(self, docs: List[Document]) -> List[Document]:
-        final_chunks = []
         chunks = docs
         while True:
-            long_chunks = [
-                p
-                for p in chunks
-                if self.num_tokens(p.content) > 1.3 * self.config.chunk_size
-            ]
-            if len(long_chunks) == 0:
-                break
-            short_chunks = [
-                p
-                for p in chunks
-                if self.num_tokens(p.content) <= 1.3 * self.config.chunk_size
-            ]
-            final_chunks += short_chunks
-            chunks = self._split_para_sentence_once(long_chunks)
-            if len(chunks) == len(long_chunks):
-                max_len = max([self.num_tokens(p.content) for p in long_chunks])
-                logger.warning(
-                    f"""
-                    Unable to split {len(long_chunks)} long chunks
-                    using chunk_size = {self.config.chunk_size}.
-                    Max chunk size is {max_len} tokens.
-                    """
-                )
+            un_splittables = 0
+            split_chunks = []
+            for c in chunks:
+                if c.content.strip() == "":
+                    continue
+                if self.num_tokens(c.content) <= 1.3 * self.config.chunk_size:
+                    # small chunk: no need to split
+                    split_chunks.append(c)
+                    continue
+                splits = self._split_para_sentence_once([c])
+                un_splittables += len(splits) == 1
+                split_chunks += splits
+            if len(split_chunks) == len(chunks):
+                if un_splittables > 0:
+                    max_len = max([self.num_tokens(p.content) for p in chunks])
+                    logger.warning(
+                        f"""
+                        Unable to split {un_splittables} chunks
+                        using chunk_size = {self.config.chunk_size}.
+                        Max chunk size is {max_len} tokens.
+                        """
+                    )
                 break  # we won't be able to shorten them with current settings
+            chunks = split_chunks.copy()
 
-        return final_chunks + chunks
+        self.add_window_ids(chunks)
+        return chunks
 
     def _split_para_sentence_once(self, docs: List[Document]) -> List[Document]:
         final_chunks = []
@@ -132,7 +147,6 @@ class Parser:
                 for c in chunks
                 if c.strip() != ""
             ]
-            self.add_window_ids(chunk_docs)
             final_chunks += chunk_docs
 
         return final_chunks
@@ -240,6 +254,11 @@ class Parser:
     def split(self, docs: List[Document]) -> List[Document]:
         if len(docs) == 0:
             return []
+        # create ids in metadata of docs if absent:
+        # we need this to distinguish docs later in add_window_ids
+        for d in docs:
+            if d.metadata.id is None:
+                d.metadata.id = d.id()
         # some docs are already splits, so don't split them further!
         chunked_docs = [d for d in docs if d.metadata.is_chunk]
         big_docs = [d for d in docs if not d.metadata.is_chunk]

{langroid-0.1.133 → langroid-0.1.135}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "langroid"
-version = "0.1.133"
+version = "0.1.135"
 description = "Harness LLMs with Multi-Agent Programming"
 authors = ["Prasad Chalasani <pchalasani@gmail.com>"]
 readme = "README.md"