langroid 0.43.1__tar.gz → 0.45.0__tar.gz

{langroid-0.43.1 → langroid-0.45.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langroid
-Version: 0.43.1
+Version: 0.45.0
 Summary: Harness LLMs with Multi-Agent Programming
 Author-email: Prasad Chalasani <pchalasani@gmail.com>
 License: MIT
@@ -63,6 +63,7 @@ Requires-Dist: docling<3.0.0,>=2.16.0; extra == 'all'
 Requires-Dist: fastembed<0.4.0,>=0.3.1; extra == 'all'
 Requires-Dist: huggingface-hub<1.0.0,>=0.21.2; extra == 'all'
 Requires-Dist: litellm<2.0.0,>=1.30.1; extra == 'all'
+Requires-Dist: marker-pdf; extra == 'all'
 Requires-Dist: metaphor-python<0.2.0,>=0.1.23; extra == 'all'
 Requires-Dist: neo4j<6.0.0,>=5.14.1; extra == 'all'
 Requires-Dist: pdf2image<2.0.0,>=1.17.0; extra == 'all'
@@ -99,6 +100,7 @@ Requires-Dist: pymysql<2.0.0,>=1.1.0; extra == 'db'
 Requires-Dist: sqlalchemy<3.0.0,>=2.0.19; extra == 'db'
 Provides-Extra: doc-chat
 Requires-Dist: docling<3.0.0,>=2.20.0; extra == 'doc-chat'
+Requires-Dist: marker-pdf; extra == 'doc-chat'
 Requires-Dist: pdf2image<2.0.0,>=1.17.0; extra == 'doc-chat'
 Requires-Dist: pymupdf4llm<0.1.0,>=0.0.17; extra == 'doc-chat'
 Requires-Dist: pymupdf<2.0.0,>=1.23.3; extra == 'doc-chat'
@@ -138,6 +140,9 @@ Requires-Dist: pyarrow<16.0.0,>=15.0.0; extra == 'lancedb'
 Requires-Dist: tantivy<0.22.0,>=0.21.0; extra == 'lancedb'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.30.1; extra == 'litellm'
+Provides-Extra: marker-pdf
+Requires-Dist: marker-pdf[full]>=1.6.0; (sys_platform != 'darwin' or platform_machine != 'x86_64') and extra == 'marker-pdf'
+Requires-Dist: opencv-python>=4.11.0.86; extra == 'marker-pdf'
 Provides-Extra: meilisearch
 Requires-Dist: meilisearch-python-sdk<3.0.0,>=2.2.3; extra == 'meilisearch'
 Provides-Extra: metaphor
@@ -150,6 +155,7 @@ Provides-Extra: neo4j
 Requires-Dist: neo4j<6.0.0,>=5.14.1; extra == 'neo4j'
 Provides-Extra: pdf-parsers
 Requires-Dist: docling<3.0.0,>=2.16.0; extra == 'pdf-parsers'
+Requires-Dist: marker-pdf; extra == 'pdf-parsers'
 Requires-Dist: markitdown>=0.0.1a3; extra == 'pdf-parsers'
 Requires-Dist: pdf2image<2.0.0,>=1.17.0; extra == 'pdf-parsers'
 Requires-Dist: pymupdf4llm<0.1.0,>=0.0.17; extra == 'pdf-parsers'
@@ -237,9 +243,11 @@ This Multi-Agent paradigm is inspired by the
 
 `Langroid` is a fresh take on LLM app-development, where considerable thought has gone 
 into simplifying the developer experience; 
-it does not use `Langchain`, or any other LLM framework.
+it does not use `Langchain`, or any other LLM framework, 
+and works with [practically any LLM](https://langroid.github.io/langroid/tutorials/supported-models/).
 
-:fire: Read the (WIP) [overview of the langroid architecture](https://langroid.github.io/langroid/blog/2024/08/15/overview-of-langroids-multi-agent-architecture-prelim/) and a [quick tour of Langroid](https://langroid.github.io/langroid/tutorials/langroid-tour/)
+:fire: Read the (WIP) [overview of the langroid architecture](https://langroid.github.io/langroid/blog/2024/08/15/overview-of-langroids-multi-agent-architecture-prelim/), 
+ and a [quick tour of Langroid](https://langroid.github.io/langroid/tutorials/langroid-tour/).
 
 📢 Companies are using/adapting Langroid in **production**. Here is a quote:
 
@@ -327,6 +335,18 @@ teacher_task.run()
 <details>
 <summary> <b>Click to expand</b></summary>
 
+- **Feb 2025:**
+  - [0.43.0](https://github.com/langroid/langroid/releases/tag/0.43.0): `GeminiPdfParser` for parsing PDF using 
+    Gemini LLMs - Thanks @abab-dev.
+  - [0.42.0](https://github.com/langroid/langroid/releases/tag/0.42.0): `markitdown` parser for `pptx,xlsx,xls` files 
+    Thanks @abab-dev.
+  - [0.41.0](https://github.com/langroid/langroid/releases/tag/0.41.0): `pinecone` vector-db (Thanks @coretado), 
+    `Tavily` web-search (Thanks @Sozhan308), `Exa` web-search (Thanks @MuddyHope).
+  - [0.40.0](https://github.com/langroid/langroid/releases/tag/0.40.0): `pgvector` vector-db. Thanks @abab-dev.
+  - [0.39.0](https://github.com/langroid/langroid/releases/tag/0.39.0): `ChatAgentConfig.handle_llm_no_tool` for 
+    handling LLM "forgetting" to use a tool.
+  - [0.38.0](https://github.com/langroid/langroid/releases/tag/0.38.0): Gemini embeddings - Thanks @abab-dev)
+  - [0.37.0](https://github.com/langroid/langroid/releases/tag/0.37.0): New PDF Parsers: `docling`, `pymupdf4llm`
 - **Jan 2025:**
   - [0.36.0](https://github.com/langroid/langroid/releases/tag/0.36.0): Weaviate vector-db support (thanks @abab-dev).
   - [0.35.0](https://github.com/langroid/langroid/releases/tag/0.35.0): Capture/Stream reasoning content from 
@@ -591,7 +611,8 @@ section above)
   Agents with specific skills, wrap them in Tasks, and combine tasks in a flexible way.
 - **LLM Support**: Langroid supports OpenAI LLMs as well as LLMs from hundreds of 
 providers ([local/open](https://langroid.github.io/langroid/tutorials/local-llm-setup/) or [remote/commercial](https://langroid.github.io/langroid/tutorials/non-openai-llms/)) via proxy libraries and local model servers
-such as [ollama](https://github.com/ollama), [oobabooga](https://github.com/oobabooga/text-generation-webui), [LiteLLM](https://docs.litellm.ai/docs/providers) that in effect mimic the OpenAI API. 
+such as [ollama](https://github.com/ollama), [oobabooga](https://github.com/oobabooga/text-generation-webui), 
+  [LiteLLM](https://docs.litellm.ai/docs/providers) that in effect mimic the OpenAI API. See the [supported LLMs](https://langroid.github.io/langroid/tutorials/supported-models/). 
 - **Caching of LLM responses:** Langroid supports [Redis](https://redis.com/try-free/) and 
   [Momento](https://www.gomomento.com/) to cache LLM responses.
 - **Vector-stores**: [LanceDB](https://github.com/lancedb/lancedb), [Qdrant](https://qdrant.tech/), [Chroma](https://www.trychroma.com/) are currently supported.
@@ -776,8 +797,8 @@ wget -O .env https://raw.githubusercontent.com/langroid/langroid/main/.env-templ
 # Edit the .env file with your favorite editor (here nano), and remove any un-used settings. E.g. there are "dummy" values like "your-redis-port" etc -- if you are not using them, you MUST remove them.
 nano .env
 
-# launch the container
-docker run -it --rm  -v ./.env:/langroid/.env langroid/langroid
+# launch the container (the appropriate image for your architecture will be pulled automatically)
+docker run -it --rm  -v ./.env:/langroid/.env langroid/langroid:latest
 
 # Use this command to run any of the scripts in the `examples` directory
 python examples/<Path/To/Example.py> 

{langroid-0.43.1 → langroid-0.45.0}/README.md

@@ -45,9 +45,11 @@ This Multi-Agent paradigm is inspired by the
 
 `Langroid` is a fresh take on LLM app-development, where considerable thought has gone 
 into simplifying the developer experience; 
-it does not use `Langchain`, or any other LLM framework.
+it does not use `Langchain`, or any other LLM framework, 
+and works with [practically any LLM](https://langroid.github.io/langroid/tutorials/supported-models/).
 
-:fire: Read the (WIP) [overview of the langroid architecture](https://langroid.github.io/langroid/blog/2024/08/15/overview-of-langroids-multi-agent-architecture-prelim/) and a [quick tour of Langroid](https://langroid.github.io/langroid/tutorials/langroid-tour/)
+:fire: Read the (WIP) [overview of the langroid architecture](https://langroid.github.io/langroid/blog/2024/08/15/overview-of-langroids-multi-agent-architecture-prelim/), 
+ and a [quick tour of Langroid](https://langroid.github.io/langroid/tutorials/langroid-tour/).
 
 📢 Companies are using/adapting Langroid in **production**. Here is a quote:
 
@@ -135,6 +137,18 @@ teacher_task.run()
 <details>
 <summary> <b>Click to expand</b></summary>
 
+- **Feb 2025:**
+  - [0.43.0](https://github.com/langroid/langroid/releases/tag/0.43.0): `GeminiPdfParser` for parsing PDF using 
+    Gemini LLMs - Thanks @abab-dev.
+  - [0.42.0](https://github.com/langroid/langroid/releases/tag/0.42.0): `markitdown` parser for `pptx,xlsx,xls` files 
+    Thanks @abab-dev.
+  - [0.41.0](https://github.com/langroid/langroid/releases/tag/0.41.0): `pinecone` vector-db (Thanks @coretado), 
+    `Tavily` web-search (Thanks @Sozhan308), `Exa` web-search (Thanks @MuddyHope).
+  - [0.40.0](https://github.com/langroid/langroid/releases/tag/0.40.0): `pgvector` vector-db. Thanks @abab-dev.
+  - [0.39.0](https://github.com/langroid/langroid/releases/tag/0.39.0): `ChatAgentConfig.handle_llm_no_tool` for 
+    handling LLM "forgetting" to use a tool.
+  - [0.38.0](https://github.com/langroid/langroid/releases/tag/0.38.0): Gemini embeddings - Thanks @abab-dev)
+  - [0.37.0](https://github.com/langroid/langroid/releases/tag/0.37.0): New PDF Parsers: `docling`, `pymupdf4llm`
 - **Jan 2025:**
   - [0.36.0](https://github.com/langroid/langroid/releases/tag/0.36.0): Weaviate vector-db support (thanks @abab-dev).
   - [0.35.0](https://github.com/langroid/langroid/releases/tag/0.35.0): Capture/Stream reasoning content from 
@@ -399,7 +413,8 @@ section above)
   Agents with specific skills, wrap them in Tasks, and combine tasks in a flexible way.
 - **LLM Support**: Langroid supports OpenAI LLMs as well as LLMs from hundreds of 
 providers ([local/open](https://langroid.github.io/langroid/tutorials/local-llm-setup/) or [remote/commercial](https://langroid.github.io/langroid/tutorials/non-openai-llms/)) via proxy libraries and local model servers
-such as [ollama](https://github.com/ollama), [oobabooga](https://github.com/oobabooga/text-generation-webui), [LiteLLM](https://docs.litellm.ai/docs/providers) that in effect mimic the OpenAI API. 
+such as [ollama](https://github.com/ollama), [oobabooga](https://github.com/oobabooga/text-generation-webui), 
+  [LiteLLM](https://docs.litellm.ai/docs/providers) that in effect mimic the OpenAI API. See the [supported LLMs](https://langroid.github.io/langroid/tutorials/supported-models/). 
 - **Caching of LLM responses:** Langroid supports [Redis](https://redis.com/try-free/) and 
   [Momento](https://www.gomomento.com/) to cache LLM responses.
 - **Vector-stores**: [LanceDB](https://github.com/lancedb/lancedb), [Qdrant](https://qdrant.tech/), [Chroma](https://www.trychroma.com/) are currently supported.
@@ -584,8 +599,8 @@ wget -O .env https://raw.githubusercontent.com/langroid/langroid/main/.env-templ
 # Edit the .env file with your favorite editor (here nano), and remove any un-used settings. E.g. there are "dummy" values like "your-redis-port" etc -- if you are not using them, you MUST remove them.
 nano .env
 
-# launch the container
-docker run -it --rm  -v ./.env:/langroid/.env langroid/langroid
+# launch the container (the appropriate image for your architecture will be pulled automatically)
+docker run -it --rm  -v ./.env:/langroid/.env langroid/langroid:latest
 
 # Use this command to run any of the scripts in the `examples` directory
 python examples/<Path/To/Example.py> 

{langroid-0.43.1 → langroid-0.45.0}/langroid/agent/base.py

@@ -1016,7 +1016,7 @@ class Agent(ABC):
             # we would have already displayed the msg "live" ONLY if
             # streaming was enabled, AND we did not find a cached response
             # If we are here, it means the response has not yet been displayed.
-            cached = f"[red]{self.indent}(cached)[/red]" if response.cached else ""
+            cached = "[red](cached)[/red]" if response.cached else ""
             console.print(f"[green]{self.indent}", end="")
             print(cached + "[green]" + escape(response.message))
         self.update_token_usage(

{langroid-0.43.1 → langroid-0.45.0}/langroid/agent/callbacks/chainlit.py

@@ -5,7 +5,16 @@ Callbacks for Chainlit integration.
 import json
 import logging
 import textwrap
-from typing import Any, Callable, Dict, List, Literal, Optional, no_type_check
+from typing import (
+    TYPE_CHECKING,
+    Any,
+    Callable,
+    Dict,
+    List,
+    Literal,
+    Optional,
+    no_type_check,
+)
 
 from langroid.exceptions import LangroidImportError
 from langroid.pydantic_v1 import BaseSettings
@@ -18,7 +27,8 @@ except ImportError:
 from chainlit import run_sync
 from chainlit.logger import logger
 
-import langroid as lr
+if TYPE_CHECKING:
+    from langroid import Agent, Task
 import langroid.language_models as lm
 from langroid.language_models import StreamEventType
 from langroid.utils.configuration import settings
@@ -222,11 +232,11 @@ class ChainlitAgentCallbacks:
     last_step: Optional[cl.Step] = None  # used to display sub-steps under this
     curr_step: Optional[cl.Step] = None  # used to update an initiated step
     stream: Optional[cl.Step] = None  # pushed into openai_gpt.py to stream tokens
-    parent_agent: Optional[lr.Agent] = None  # used to get parent id, for step nesting
+    parent_agent: Optional["Agent"] = None  # used to get parent id, for step nesting
 
     def __init__(
         self,
-        agent: lr.Agent,
+        agent: "Agent",
         config: ChainlitCallbackConfig = ChainlitCallbackConfig(),
     ):
         """Add callbacks to the agent, and save the initial message,
@@ -245,7 +255,7 @@ class ChainlitAgentCallbacks:
         agent.callbacks.show_error_message = self.show_error_message
         agent.callbacks.show_start_response = self.show_start_response
         self.config = config
-        self.agent: lr.Agent = agent
+        self.agent: "Agent" = agent
         if self.agent.llm is not None:
             # We don't want to suppress LLM output in async + streaming,
             # since we often use chainlit async callbacks to display LLM output
@@ -271,7 +281,7 @@ class ChainlitAgentCallbacks:
         )
         return last_step.id  # type: ignore
 
-    def set_parent_agent(self, parent: lr.Agent) -> None:
+    def set_parent_agent(self, parent: "Agent") -> None:
         self.parent_agent = parent
 
     def get_last_step(self) -> Optional[cl.Step]:
@@ -559,7 +569,7 @@ class ChainlitTaskCallbacks(ChainlitAgentCallbacks):
 
     def __init__(
         self,
-        task: lr.Task,
+        task: "Task",
         config: ChainlitCallbackConfig = ChainlitCallbackConfig(),
     ):
         """Inject callbacks recursively, ensuring msg is passed to the
@@ -573,7 +583,7 @@ class ChainlitTaskCallbacks(ChainlitAgentCallbacks):
 
     @classmethod
     def _inject_callbacks(
-        cls, task: lr.Task, config: ChainlitCallbackConfig = ChainlitCallbackConfig()
+        cls, task: "Task", config: ChainlitCallbackConfig = ChainlitCallbackConfig()
     ) -> None:
         # recursively apply ChainlitAgentCallbacks to agents of sub-tasks
         for t in task.sub_tasks:
@@ -581,7 +591,7 @@ class ChainlitTaskCallbacks(ChainlitAgentCallbacks):
             # ChainlitTaskCallbacks(t, config=config)
 
     def show_subtask_response(
-        self, task: lr.Task, content: str, is_tool: bool = False
+        self, task: "Task", content: str, is_tool: bool = False
     ) -> None:
         """Show sub-task response as a step, nested at the right level."""
 

{langroid-0.43.1 → langroid-0.45.0}/langroid/agent/special/doc_chat_agent.py

@@ -14,6 +14,7 @@ pip install "langroid[hf-embeddings]"
 
 """
 
+import importlib
 import logging
 from collections import OrderedDict
 from functools import cache
@@ -82,14 +83,13 @@ about them, or summarize them into coherent answers.
 """
 
 CHUNK_ENRICHMENT_DELIMITER = "\n<##-##-##>\n"
-
-has_sentence_transformers = False
 try:
-    from sentence_transformers import SentenceTransformer  # noqa: F401
-
-    has_sentence_transformers = True
-except ImportError:
-    pass
+    # Check if  module exists in sys.path
+    spec = importlib.util.find_spec("sentence_transformers")
+    has_sentence_transformers = spec is not None
+except Exception as e:
+    logger.warning(f"Error checking sentence_transformers: {e}")
+    has_sentence_transformers = False
 
 
 hf_embed_config = SentenceTransformerEmbeddingsConfig(
@@ -236,6 +236,7 @@ class DocChatAgent(ChatAgent):
         self.chunked_docs: List[Document] = []
         self.chunked_docs_clean: List[Document] = []
         self.response: None | Document = None
+
         if len(config.doc_paths) > 0:
             self.ingest()
 

{langroid-0.43.1 → langroid-0.45.0}/langroid/parsing/document_parser.py

@@ -16,28 +16,11 @@ from dotenv import load_dotenv
 from langroid.exceptions import LangroidImportError
 from langroid.utils.object_registry import ObjectRegistry
 
-try:
+if TYPE_CHECKING:
+    import docling  # noqa
     import fitz
-except ImportError:
-    if not TYPE_CHECKING:
-        fitz = None
-try:
-    import pymupdf4llm
-except ImportError:
-    if not TYPE_CHECKING:
-        pymupdf4llm = None
-
-try:
-    import docling
-except ImportError:
-    if not TYPE_CHECKING:
-        docling = None
-
-try:
+    import pymupdf4llm  # noqa
     import pypdf
-except ImportError:
-    if not TYPE_CHECKING:
-        pypdf = None
 
 
 import requests
@@ -167,6 +150,8 @@ class DocumentParser(Parser):
                 return ImagePdfParser(source, config)
             elif config.pdf.library == "gemini":
                 return GeminiPdfParser(source, config)
+            elif config.pdf.library == "marker":
+                return MarkerPdfParser(source, config)
             else:
                 raise ValueError(
                     f"Unsupported PDF library specified: {config.pdf.library}"
@@ -469,8 +454,10 @@ class FitzPDFParser(DocumentParser):
         Returns:
             Generator[fitz.Page]: Generator yielding each page.
         """
-        if fitz is None:
-            raise LangroidImportError("fitz", "pdf-parsers")
+        try:
+            import fitz
+        except ImportError:
+            LangroidImportError("fitz", "doc-chat")
         doc = fitz.open(stream=self.doc_bytes, filetype="pdf")
         for i, page in enumerate(doc):
             yield i, page
@@ -504,7 +491,10 @@ class PyMuPDF4LLMParser(DocumentParser):
         Returns:
             Generator[fitz.Page]: Generator yielding each page.
         """
-        if fitz is None:
+        try:
+            import pymupdf4llm  # noqa
+            import fitz
+        except ImportError:
             raise LangroidImportError(
                 "pymupdf4llm", ["pymupdf4llm", "all", "pdf-parsers", "doc-chat"]
             )
@@ -548,7 +538,9 @@ class DoclingParser(DocumentParser):
         Returns:
             Generator[docling.Page]: Generator yielding each page.
         """
-        if docling is None:
+        try:
+            import docling  # noqa
+        except ImportError:
             raise LangroidImportError(
                 "docling", ["docling", "pdf-parsers", "all", "doc-chat"]
             )
@@ -637,7 +629,9 @@ class PyPDFParser(DocumentParser):
         Returns:
             Generator[pypdf.pdf.PageObject]: Generator yielding each page.
         """
-        if pypdf is None:
+        try:
+            import pypdf
+        except ImportError:
             raise LangroidImportError("pypdf", "pdf-parsers")
         reader = pypdf.PdfReader(self.doc_bytes)
         for i, page in enumerate(reader.pages):
@@ -1364,3 +1358,85 @@ class GeminiPdfParser(DocumentParser):
             content=page,
             metadata=DocMetaData(source=self.source),
         )
+
+
+class MarkerPdfParser(DocumentParser):
+    DEFAULT_CONFIG = {"paginate_output": True, "output_format": "markdown"}
+
+    def __init__(self, source: Union[str, bytes], config: ParsingConfig):
+        super().__init__(source, config)
+        user_config = (
+            config.pdf.marker_config.config_dict if config.pdf.marker_config else {}
+        )
+
+        self.config_dict = {**MarkerPdfParser.DEFAULT_CONFIG, **user_config}
+
+    def iterate_pages(self) -> Generator[Tuple[int, Any], None, None]:
+        """
+        Yield each page in the PDF using `marker`.
+        """
+        try:
+            import marker  # noqa
+        except ImportError:
+            raise LangroidImportError(
+                "marker-pdf", ["marker-pdf", "pdf-parsers", "all", "doc-chat"]
+            )
+
+        import re
+
+        from marker.config.parser import ConfigParser
+        from marker.converters.pdf import PdfConverter
+        from marker.models import create_model_dict
+        from marker.output import save_output
+
+        config_parser = ConfigParser(self.config_dict)
+        converter = PdfConverter(
+            config=config_parser.generate_config_dict(),
+            artifact_dict=create_model_dict(),
+            processor_list=config_parser.get_processors(),
+            renderer=config_parser.get_renderer(),
+            llm_service=config_parser.get_llm_service(),
+        )
+        doc_path = self.source
+        if doc_path == "bytes":
+            # write to tmp file, then use that path
+            with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as temp_file:
+                temp_file.write(self.doc_bytes.getvalue())
+                doc_path = temp_file.name
+
+        output_dir = Path(str(Path(doc_path).with_suffix("")) + "-pages")
+        os.makedirs(output_dir, exist_ok=True)
+        filename = Path(doc_path).stem + "_converted"
+
+        rendered = converter(doc_path)
+        save_output(rendered, output_dir=output_dir, fname_base=filename)
+        file_path = output_dir / f"{filename}.md"
+
+        with open(file_path, "r", encoding="utf-8") as f:
+            full_markdown = f.read()
+
+        # Regex for splitting pages
+        pages = re.split(r"\{\d+\}----+", full_markdown)
+
+        page_no = 0
+        for page in pages:
+            if page.strip():
+                yield page_no, page
+            page_no += 1
+
+    def get_document_from_page(self, page: str) -> Document:
+        """
+        Get Document object from a given 1-page markdown file,
+        possibly containing image refs.
+
+        Args:
+            page (str): The page we get by splitting large md file from
+            marker
+
+        Returns:
+            Document: Document object, with content and possible metadata.
+        """
+        return Document(
+            content=self.fix_text(page),
+            metadata=DocMetaData(source=self.source),
+        )

{langroid-0.43.1 → langroid-0.45.0}/langroid/parsing/parser.py

@@ -38,8 +38,13 @@ class GeminiConfig(BaseSettings):
     requests_per_minute: Optional[int] = 5
 
 
-class PdfParsingConfig(BaseParsingConfig):
+class MarkerConfig(BaseSettings):
+    """Configuration for Markitdown-based parsing."""
+
+    config_dict: Dict[str, Any] = {}
 
+
+class PdfParsingConfig(BaseParsingConfig):
     library: Literal[
         "fitz",
         "pymupdf4llm",
@@ -49,16 +54,26 @@ class PdfParsingConfig(BaseParsingConfig):
         "pdf2image",
         "markitdown",
         "gemini",
+        "marker",
     ] = "pymupdf4llm"
     gemini_config: Optional[GeminiConfig] = None
+    marker_config: Optional[MarkerConfig] = None
 
     @root_validator(pre=True)
-    def enable_gemini_config(cls, values: Dict[str, Any]) -> Dict[str, Any]:
-        """Ensure GeminiConfig is set only when library is 'gemini'."""
-        if values.get("library") == "gemini":
-            values["gemini_config"] = values.get("gemini_config") or GeminiConfig()
+    def enable_configs(cls, values: Dict[str, Any]) -> Dict[str, Any]:
+        """Ensure correct config is set based on library selection."""
+        library = values.get("library")
+
+        if library == "gemini":
+            values.setdefault("gemini_config", GeminiConfig())
         else:
             values["gemini_config"] = None
+
+        if library == "marker":
+            values.setdefault("marker_config", MarkerConfig())
+        else:
+            values["marker_config"] = None
+
         return values
 
 

{langroid-0.43.1 → langroid-0.45.0}/langroid/parsing/repo_loader.py

@@ -7,14 +7,16 @@ import tempfile
 import time
 from collections import deque
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
 from urllib.parse import urlparse
 
 from dotenv import load_dotenv
-from github import Github
-from github.ContentFile import ContentFile
-from github.Label import Label
-from github.Repository import Repository
+
+if TYPE_CHECKING:
+    from github import Github
+    from github.ContentFile import ContentFile
+    from github.Label import Label
+    from github.Repository import Repository
 
 from langroid.mytypes import DocMetaData, Document
 from langroid.parsing.document_parser import DocumentParser, DocumentType
@@ -24,7 +26,7 @@ from langroid.pydantic_v1 import BaseModel, BaseSettings, Field
 logger = logging.getLogger(__name__)
 
 
-def _get_decoded_content(content_file: ContentFile) -> str:
+def _get_decoded_content(content_file: "ContentFile") -> str:
     if content_file.encoding == "base64":
         return content_file.decoded_content.decode("utf-8") or ""
     elif content_file.encoding == "none":
@@ -54,7 +56,7 @@ class IssueData(BaseModel):
     text: str = Field(..., description="Text of issue, i.e. description body")
 
 
-def get_issue_size(labels: List[Label]) -> str | None:
+def get_issue_size(labels: List["Label"]) -> str | None:
     sizes = ["XS", "S", "M", "L", "XL", "XXL"]
     return next((label.name for label in labels if label.name in sizes), None)
 
@@ -117,6 +119,8 @@ class RepoLoader:
         self.config = config
         self.clone_path: Optional[str] = None
         self.log_file = ".logs/repo_loader/download_log.json"
+        self.repo: Optional["Repository"] = None  # Initialize repo as Optional
+
         os.makedirs(os.path.dirname(self.log_file), exist_ok=True)
         if not os.path.exists(self.log_file):
             with open(self.log_file, "w") as f:
@@ -127,20 +131,25 @@ class RepoLoader:
             logger.info(f"Repo Already downloaded in {log[self.url]}")
             self.clone_path = log[self.url]
 
+        # it's a core dependency, so we don't need to enclose in try/except
+        from github import Github  # Late import
+
+        load_dotenv()
+        # authenticated calls to github api have higher rate limit
+        token = os.getenv("GITHUB_ACCESS_TOKEN")
+
         if "github.com" in self.url:
             repo_name = self.url.split("github.com/")[1]
         else:
             repo_name = self.url
-        load_dotenv()
-        # authenticated calls to github api have higher rate limit
-        token = os.getenv("GITHUB_ACCESS_TOKEN")
+
         g = Github(token)
         self.repo = self._get_repo_with_retry(g, repo_name)
 
     @staticmethod
     def _get_repo_with_retry(
-        g: Github, repo_name: str, max_retries: int = 5
-    ) -> Repository:
+        g: "Github", repo_name: str, max_retries: int = 5
+    ) -> "Repository":
         """
         Get a repo from the GitHub API, retrying if the request fails,
         with exponential backoff.
@@ -173,6 +182,10 @@ class RepoLoader:
 
     def get_issues(self, k: int | None = 100) -> List[IssueData]:
         """Get up to k issues from the GitHub repo."""
+        if self.repo is None:
+            logger.warning("No repo found. Ensure the URL is correct.")
+            return []  # Return an empty list rather than raise an error in this case
+
         if k is None:
             issues = self.repo.get_issues(state="all")
         else:
@@ -224,7 +237,7 @@ class RepoLoader:
         """
         return file_type not in self.config.non_code_types
 
-    def _is_allowed(self, content: ContentFile) -> bool:
+    def _is_allowed(self, content: "ContentFile") -> bool:
         """
         Check if a file or directory content is allowed to be included.
 
@@ -301,6 +314,10 @@ class RepoLoader:
             Dict[str, Union[str, List[Dict]]]:
             A dictionary containing file and directory names, with file contents.
         """
+        if self.repo is None:
+            logger.warning("No repo found. Ensure the URL is correct.")
+            return {}  # Return an empty dict rather than raise an error in this case
+
         root_contents = self.repo.get_contents("")
         if not isinstance(root_contents, list):
             root_contents = [root_contents]
@@ -519,8 +536,7 @@ class RepoLoader:
                 which includes all depths.
             lines (int, optional): Number of lines to read from each file.
                 Defaults to None, which reads all lines.
-            doc_type (str|DocumentType, optional): The type of document to parse.
-
+            doc_type (str|DocumentType | None, optional): The type of document to parse.
         Returns:
             List[Document]: List of Document objects representing files.
 
@@ -584,6 +600,10 @@ class RepoLoader:
             list of Document objects, each has fields `content` and `metadata`,
             and `metadata` has fields `url`, `filename`, `extension`, `language`
         """
+        if self.repo is None:
+            logger.warning("No repo found. Ensure the URL is correct.")
+            return []  # Return an empty list rather than raise an error
+
         contents = self.repo.get_contents("")
         if not isinstance(contents, list):
             contents = [contents]

{langroid-0.43.1 → langroid-0.45.0}/langroid/parsing/search.py

@@ -10,9 +10,6 @@ import difflib
 import re
 from typing import List, Tuple
 
-from nltk.corpus import stopwords
-from nltk.stem import WordNetLemmatizer
-from nltk.tokenize import RegexpTokenizer
 from rank_bm25 import BM25Okapi
 from thefuzz import fuzz, process
 
@@ -120,6 +117,9 @@ def preprocess_text(text: str) -> str:
     # Ensure the NLTK resources are available
     for resource in ["tokenizers/punkt", "corpora/wordnet", "corpora/stopwords"]:
         download_nltk_resource(resource)
+    from nltk.corpus import stopwords
+    from nltk.stem import WordNetLemmatizer
+    from nltk.tokenize import RegexpTokenizer
 
     # Lowercase the text
     text = text.lower()