local-deep-research 0.1.0__tar.gz → 0.1.12__tar.gz

{local_deep_research-0.1.0/src/local_deep_research.egg-info → local_deep_research-0.1.12}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: local-deep-research
-Version: 0.1.0
+Version: 0.1.12
 Summary: AI-powered research assistant with deep, iterative analysis using LLMs and web searches
 Author-email: LearningCircuit <185559241+LearningCircuit@users.noreply.github.com>, HashedViking <6432677+HashedViking@users.noreply.github.com>
 License: MIT License
@@ -51,7 +51,7 @@ Requires-Dist: flask-socketio>=5.1.1
 Requires-Dist: sqlalchemy>=1.4.23
 Requires-Dist: wikipedia
 Requires-Dist: arxiv>=1.4.3
-Requires-Dist: PyPDF2>=2.0.0
+Requires-Dist: pypdf
 Requires-Dist: sentence-transformers
 Requires-Dist: faiss-cpu
 Requires-Dist: pydantic>=2.0.0
@@ -59,6 +59,7 @@ Requires-Dist: pydantic-settings>=2.0.0
 Requires-Dist: toml>=0.10.2
 Requires-Dist: platformdirs>=3.0.0
 Requires-Dist: dynaconf
+Dynamic: license-file
 
 # Local Deep Research
 
@@ -91,12 +92,13 @@ A powerful AI-powered research assistant that performs deep, iterative analysis
 
 - 🌐 **Enhanced Search Integration**
   - **Auto-selection of search sources**: The "auto" search engine intelligently analyzes your query and selects the most appropriate search engine based on the query content
+  - **SearXNG** integration for local web-search engine, great for privacy, no API key required (requires a searxng server)
   - Wikipedia integration for factual knowledge
   - arXiv integration for scientific papers and academic research
   - PubMed integration for biomedical literature and medical research
   - DuckDuckGo integration for web searches (may experience rate limiting)
   - SerpAPI integration for Google search results (requires API key)
-  - **Google Programmable Search Engine** integration for custom search experiences (requires API key)
+  - Google Programmable Search Engine integration for custom search experiences (requires API key)
   - The Guardian integration for news articles and journalism (requires API key)
   - **Local RAG search for private documents** - search your own documents with vector embeddings
   - Full webpage content retrieval
@@ -127,10 +129,10 @@ This example showcases the system's ability to perform multiple research iterati
 
 1. Clone the repository:
 ```bash
-git clone https://github.com/yourusername/local-deep-research.git
+git clone https://github.com/LearningCircuit/local-deep-research.git
 cd local-deep-research
 ```
-
+(experimental pip install with new features (but not so well tested yet): **pip install local-deep-research** )
 2. Install dependencies:
 ```bash
 pip install -r requirements.txt
@@ -147,6 +149,20 @@ ollama pull mistral  # Default model - many work really well choose best for you
 ```bash
 # Copy the template
 cp .env.template .env
+```
+
+## Experimental install
+```bash
+#experimental pip install with new features (but not so well tested yet):
+pip install local-deep-research
+playwright install
+ollama pull mistral 
+```
+## Community & Support
+
+We've just launched our [Discord server](https://discord.gg/2E6gYU2Z) for this project!
+
+Our Discord server can help to exchange ideas about research approaches, discuss advanced usage patterns, and share other ideas.
 
 # Edit .env with your API keys (if using cloud LLMs)
 ANTHROPIC_API_KEY=your-api-key-here  # For Claude
@@ -276,6 +292,7 @@ You can use local search in several ways:
 The system supports multiple search engines that can be selected by changing the `search_tool` variable in `config.py`:
 
 - **Auto** (`auto`): Intelligent search engine selector that analyzes your query and chooses the most appropriate source (Wikipedia, arXiv, local collections, etc.)
+- **SearXNG** (`searxng`): Local web-search engine, great for privacy, no API key required (requires a searxng server)
 - **Wikipedia** (`wiki`): Best for general knowledge, facts, and overview information
 - **arXiv** (`arxiv`): Great for scientific and academic research, accessing preprints and papers
 - **PubMed** (`pubmed`): Excellent for biomedical literature, medical research, and health information
@@ -307,6 +324,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
   - [DuckDuckGo](https://duckduckgo.com) for web search
   - [The Guardian](https://www.theguardian.com/) for quality journalism
   - [SerpAPI](https://serpapi.com) for Google search results (requires API key)
+  - [SearXNG](https://searxng.org/) for local web-search engine 
 - Built on [LangChain](https://github.com/hwchase17/langchain) framework
 - Uses [justext](https://github.com/miso-belica/justext) for content extraction
 - [Playwright](https://playwright.dev) for web content retrieval

{local_deep_research-0.1.0 → local_deep_research-0.1.12}/README.md

@@ -29,12 +29,13 @@ A powerful AI-powered research assistant that performs deep, iterative analysis
 
 - 🌐 **Enhanced Search Integration**
   - **Auto-selection of search sources**: The "auto" search engine intelligently analyzes your query and selects the most appropriate search engine based on the query content
+  - **SearXNG** integration for local web-search engine, great for privacy, no API key required (requires a searxng server)
   - Wikipedia integration for factual knowledge
   - arXiv integration for scientific papers and academic research
   - PubMed integration for biomedical literature and medical research
   - DuckDuckGo integration for web searches (may experience rate limiting)
   - SerpAPI integration for Google search results (requires API key)
-  - **Google Programmable Search Engine** integration for custom search experiences (requires API key)
+  - Google Programmable Search Engine integration for custom search experiences (requires API key)
   - The Guardian integration for news articles and journalism (requires API key)
   - **Local RAG search for private documents** - search your own documents with vector embeddings
   - Full webpage content retrieval
@@ -65,10 +66,10 @@ This example showcases the system's ability to perform multiple research iterati
 
 1. Clone the repository:
 ```bash
-git clone https://github.com/yourusername/local-deep-research.git
+git clone https://github.com/LearningCircuit/local-deep-research.git
 cd local-deep-research
 ```
-
+(experimental pip install with new features (but not so well tested yet): **pip install local-deep-research** )
 2. Install dependencies:
 ```bash
 pip install -r requirements.txt
@@ -85,6 +86,20 @@ ollama pull mistral  # Default model - many work really well choose best for you
 ```bash
 # Copy the template
 cp .env.template .env
+```
+
+## Experimental install
+```bash
+#experimental pip install with new features (but not so well tested yet):
+pip install local-deep-research
+playwright install
+ollama pull mistral 
+```
+## Community & Support
+
+We've just launched our [Discord server](https://discord.gg/2E6gYU2Z) for this project!
+
+Our Discord server can help to exchange ideas about research approaches, discuss advanced usage patterns, and share other ideas.
 
 # Edit .env with your API keys (if using cloud LLMs)
 ANTHROPIC_API_KEY=your-api-key-here  # For Claude
@@ -214,6 +229,7 @@ You can use local search in several ways:
 The system supports multiple search engines that can be selected by changing the `search_tool` variable in `config.py`:
 
 - **Auto** (`auto`): Intelligent search engine selector that analyzes your query and chooses the most appropriate source (Wikipedia, arXiv, local collections, etc.)
+- **SearXNG** (`searxng`): Local web-search engine, great for privacy, no API key required (requires a searxng server)
 - **Wikipedia** (`wiki`): Best for general knowledge, facts, and overview information
 - **arXiv** (`arxiv`): Great for scientific and academic research, accessing preprints and papers
 - **PubMed** (`pubmed`): Excellent for biomedical literature, medical research, and health information
@@ -245,6 +261,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
   - [DuckDuckGo](https://duckduckgo.com) for web search
   - [The Guardian](https://www.theguardian.com/) for quality journalism
   - [SerpAPI](https://serpapi.com) for Google search results (requires API key)
+  - [SearXNG](https://searxng.org/) for local web-search engine 
 - Built on [LangChain](https://github.com/hwchase17/langchain) framework
 - Uses [justext](https://github.com/miso-belica/justext) for content extraction
 - [Playwright](https://playwright.dev) for web content retrieval

{local_deep_research-0.1.0 → local_deep_research-0.1.12}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "local-deep-research"
-version = "0.1.0"
+version = "0.1.12"
 description = "AI-powered research assistant with deep, iterative analysis using LLMs and web searches"
 readme = "README.md"
 requires-python = ">=3.8"
@@ -37,7 +37,7 @@ dependencies = [
     "sqlalchemy>=1.4.23",
     "wikipedia",
     "arxiv>=1.4.3",
-    "PyPDF2>=2.0.0",
+    "pypdf",
     "sentence-transformers",
     "faiss-cpu",
     "pydantic>=2.0.0",

{local_deep_research-0.1.0 → local_deep_research-0.1.12}/requirements.txt

@@ -19,7 +19,8 @@ sqlalchemy>=1.4.23
 google-search-results
 wikipedia
 arxiv>=1.4.3
-PyPDF2>=2.0.0  # Optional - for extracting text from PDFs if needed
+# PyPDF2>=2.0.0  # Optional - for extracting text from PDFs if needed
+pypdf
 
 sentence-transformers # local search
-faiss-cpu # faiss-gpu
+faiss-cpu # faiss-gpu

{local_deep_research-0.1.0 → local_deep_research-0.1.12}/src/local_deep_research/defaults/main.toml

@@ -14,6 +14,11 @@ knowledge_accumulation_context_limit = 2000000
 # Enable fact checking (experimental, works better with large LLMs)
 enable_fact_checking = false
 
+[web]
+port = 5000
+host = "0.0.0.0"
+debug = true
+
 [search]
 # Search tool to use (auto, wikipedia, arxiv, duckduckgo, serp, google_pse, etc.)
 # "auto" intelligently selects based on query content (recommended)

{local_deep_research-0.1.0 → local_deep_research-0.1.12}/src/local_deep_research/search_system.py

@@ -25,6 +25,11 @@ class AdvancedSearchSystem:
         self.citation_handler = CitationHandler(self.model)
         self.progress_callback = None
         self.all_links_of_system = list()
+        
+        # Check if search is available, log warning if not
+        if self.search is None:
+            print("WARNING: Search system initialized with no search engine! Research will not be effective.")
+            self._update_progress("WARNING: No search engine available", None, {"error": "No search engine configured properly"})
 
         
 
@@ -114,6 +119,23 @@ class AdvancedSearchSystem:
             "phase": "init",
             "iterations_planned": total_iterations
         })
+        
+        # Check if search engine is available
+        if self.search is None:
+            error_msg = "Error: No search engine available. Please check your configuration."
+            self._update_progress(error_msg, 100, {
+                "phase": "error", 
+                "error": "No search engine available",
+                "status": "failed"
+            })
+            return {
+                "findings": [],
+                "iterations": 0,
+                "questions": {},
+                "formatted_findings": "Error: Unable to conduct research without a search engine.",
+                "current_knowledge": "",
+                "error": error_msg
+            }
 
         while iteration < self.max_iterations:
             iteration_progress_base = (iteration / total_iterations) * 100
@@ -133,7 +155,21 @@ class AdvancedSearchSystem:
                                      int(question_progress_base),
                                      {"phase": "search", "iteration": iteration + 1, "question_index": q_idx + 1})
                 
-                search_results = self.search.run(question)
+                try:
+                    if self.search is None:
+                        self._update_progress(f"Search engine unavailable, skipping search for: {question}", 
+                                            int(question_progress_base + 2),
+                                            {"phase": "search_error", "error": "No search engine available"})
+                        search_results = []
+                    else:
+                        search_results = self.search.run(question)
+                except Exception as e:
+                    error_msg = f"Error during search: {str(e)}"
+                    print(f"SEARCH ERROR: {error_msg}")
+                    self._update_progress(error_msg, 
+                                        int(question_progress_base + 2),
+                                        {"phase": "search_error", "error": str(e)})
+                    search_results = []
                 
                 if search_results is None:
                     self._update_progress(f"No search results found for question: {question}", 
@@ -155,57 +191,81 @@ class AdvancedSearchSystem:
                                      int(question_progress_base + 5),
                                      {"phase": "analysis"})
                 print("NR OF SOURCES: ", len(self.all_links_of_system))
-                result = self.citation_handler.analyze_followup(
-                    question, search_results, current_knowledge, nr_of_links=len(self.all_links_of_system)
-                )
-                links = extract_links_from_search_results(search_results)
-                self.all_links_of_system.extend(links)
-                section_links.extend(links)
-                formatted_links = ""  
-                if links:
-                    formatted_links=format_links(links=links)
-                
-                logger.debug(f"Generated questions: {formatted_links}")                           
-                if result is not None:
-                    results_with_links = str(result["content"])
-                    findings.append(
-                        {
-                            "phase": f"Follow-up {iteration}.{questions.index(question) + 1}",
-                            "content": results_with_links,
-                            "question": question,
-                            "search_results": search_results,
-                            "documents": result["documents"],
-                        }
-                    )
 
-                    if settings.general.knowledge_accumulation != KnowledgeAccumulationApproach.NO_KNOWLEDGE:
-                        current_knowledge = current_knowledge + "\n\n\n New: \n" + results_with_links
-                    
-                    print(current_knowledge)
-                    if settings.general.knowledge_accumulation == KnowledgeAccumulationApproach.QUESTION:
-                        self._update_progress(f"Compress Knowledge for: {question}", 
-                                     int(question_progress_base + 0),
-                                     {"phase": "analysis"})
-                        current_knowledge = self._compress_knowledge(current_knowledge , query, section_links)
+                try:
+                    result = self.citation_handler.analyze_followup(
+                        question, search_results, current_knowledge, nr_of_links=len(self.all_links_of_system)
+                    )
+                    links = extract_links_from_search_results(search_results)
+                    self.all_links_of_system.extend(links)
+                    section_links.extend(links)
+                    formatted_links = ""  
+                    if links:
+                        formatted_links=format_links(links=links)
                     
-                    self._update_progress(f"Analysis complete for question: {question}", 
-                                         int(question_progress_base + 10),
-                                         {"phase": "analysis_complete"})
+                    logger.debug(f"Generated questions: {formatted_links}")                           
+                    if result is not None:
+                        results_with_links = str(result["content"])
+                        findings.append(
+                            {
+                                "phase": f"Follow-up {iteration}.{questions.index(question) + 1}",
+                                "content": results_with_links,
+                                "question": question,
+                                "search_results": search_results,
+                                "documents": result["documents"],
+                            }
+                        )
 
+                        if settings.general.knowledge_accumulation != KnowledgeAccumulationApproach.NO_KNOWLEDGE:
+                            current_knowledge = current_knowledge + "\n\n\n New: \n" + results_with_links
+                        
+                        print(current_knowledge)
+                        if settings.general.knowledge_accumulation == KnowledgeAccumulationApproach.QUESTION:
+                            self._update_progress(f"Compress Knowledge for: {question}", 
+                                        int(question_progress_base + 0),
+                                        {"phase": "analysis"})
+                            current_knowledge = self._compress_knowledge(current_knowledge , query, section_links)
+                        
+                        self._update_progress(f"Analysis complete for question: {question}", 
+                                            int(question_progress_base + 10),
+                                            {"phase": "analysis_complete"})
+                except Exception as e:
+                    error_msg = f"Error analyzing results: {str(e)}"
+                    print(f"ANALYSIS ERROR: {error_msg}")
+                    self._update_progress(error_msg, 
+                                        int(question_progress_base + 10),
+                                        {"phase": "analysis_error", "error": str(e)})
             iteration += 1
             
             self._update_progress(f"Compressing knowledge after iteration {iteration}", 
                                  int((iteration / total_iterations) * 100 - 5),
                                  {"phase": "knowledge_compression"})
+
             if settings.general.knowledge_accumulation == KnowledgeAccumulationApproach.ITERATION:
-                current_knowledge = self._compress_knowledge(current_knowledge , query, section_links)
+                try:
+                    current_knowledge = self._compress_knowledge(current_knowledge , query, section_links)
+                except Exception as e:
+                    error_msg = f"Error compressing knowledge: {str(e)}"
+                    print(f"COMPRESSION ERROR: {error_msg}")
+                    self._update_progress(error_msg, 
+                                        int((iteration / total_iterations) * 100 - 3),
+                                        {"phase": "compression_error", "error": str(e)})
+
 
             
             self._update_progress(f"Iteration {iteration} complete", 
                                  int((iteration / total_iterations) * 100),
                                  {"phase": "iteration_complete", "iteration": iteration})
             
-            formatted_findings = self._save_findings(findings, current_knowledge, query)
+            try:
+                formatted_findings = self._save_findings(findings, current_knowledge, query)
+            except Exception as e:
+                error_msg = f"Error saving findings: {str(e)}"
+                print(f"SAVE ERROR: {error_msg}")
+                self._update_progress(error_msg, 
+                                    int((iteration / total_iterations) * 100),
+                                    {"phase": "save_error", "error": str(e)})
+                formatted_findings = "Error: Could not format findings due to an error."
 
         self._update_progress("Research complete", 95, {"phase": "complete"})
         
@@ -213,7 +273,7 @@ class AdvancedSearchSystem:
             "findings": findings,
             "iterations": iteration,
             "questions": self.questions_by_iteration,
-            "formatted_findings": formatted_findings,
+            "formatted_findings": formatted_findings if 'formatted_findings' in locals() else "Error: Findings not available.",
             "current_knowledge": current_knowledge
         }