local-deep-research 0.1.16__py3-none-any.whl → 0.1.18__py3-none-any.whl

local_deep_research-0.1.18.dist-info/METADATA

@@ -0,0 +1,402 @@
+Metadata-Version: 2.4
+Name: local-deep-research
+Version: 0.1.18
+Summary: AI-powered research assistant with deep, iterative analysis using LLMs and web searches
+Author-email: LearningCircuit <185559241+LearningCircuit@users.noreply.github.com>, HashedViking <6432677+HashedViking@users.noreply.github.com>
+License: MIT License
+        
+        Copyright (c) 2025 LearningCircuit
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/LearningCircuit/local-deep-research
+Project-URL: Bug Tracker, https://github.com/LearningCircuit/local-deep-research/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: langchain>=0.3.18
+Requires-Dist: langchain-community>=0.3.17
+Requires-Dist: langchain-core>=0.3.34
+Requires-Dist: langchain-ollama>=0.2.3
+Requires-Dist: langchain-openai>=0.3.5
+Requires-Dist: langchain_anthropic>=0.3.7
+Requires-Dist: duckduckgo_search>=7.3.2
+Requires-Dist: python-dateutil>=2.9.0
+Requires-Dist: typing_extensions>=4.12.2
+Requires-Dist: justext
+Requires-Dist: playwright
+Requires-Dist: beautifulsoup4
+Requires-Dist: flask>=2.0.1
+Requires-Dist: flask-cors>=3.0.10
+Requires-Dist: flask-socketio>=5.1.1
+Requires-Dist: sqlalchemy>=1.4.23
+Requires-Dist: wikipedia
+Requires-Dist: arxiv>=1.4.3
+Requires-Dist: pypdf
+Requires-Dist: sentence-transformers
+Requires-Dist: faiss-cpu
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: toml>=0.10.2
+Requires-Dist: platformdirs>=3.0.0
+Requires-Dist: dynaconf
+Requires-Dist: requests>=2.28.0
+Requires-Dist: tiktoken>=0.4.0
+Requires-Dist: xmltodict>=0.13.0
+Requires-Dist: lxml>=4.9.2
+Requires-Dist: pdfplumber>=0.9.0
+Requires-Dist: unstructured>=0.10.0
+Dynamic: license-file
+
+# Local Deep Research
+
+A powerful AI-powered research assistant that performs deep, iterative analysis using multiple LLMs and web searches. The system can be run locally for privacy or configured to use cloud-based LLMs for enhanced capabilities.
+
+## Quick Start
+
+```bash
+# Install the package
+pip install local-deep-research
+
+# Install required browser automation tools
+playwright install
+
+# For local models, install Ollama
+# Download from https://ollama.ai and then pull a model
+ollama pull gemma3:12b
+```
+
+Then run:
+
+```bash
+# Start the web interface (recommended)
+ldr-web # (OR python -m local_deep_research.web.app)
+
+# OR run the command line version
+ldr # (OR python -m local_deep_research.main)
+```
+
+Access the web interface at `http://127.0.0.1:5000` in your browser.
+
+## Features
+
+- 🔍 **Advanced Research Capabilities**
+  - Automated deep research with intelligent follow-up questions
+  - Proper inline citation and source verification
+  - Multi-iteration analysis for comprehensive coverage
+  - Full webpage content analysis (not just snippets)
+
+- 🤖 **Flexible LLM Support**
+  - Local AI processing with Ollama models
+  - Cloud LLM support (Claude, GPT)
+  - Supports all Langchain models
+  - Configurable model selection based on needs
+
+- 📊 **Rich Output Options**
+  - Detailed research findings with proper citations
+  - Well-structured comprehensive research reports
+  - Quick summaries for rapid insights
+  - Source tracking and verification
+
+- 🔒 **Privacy-Focused**
+  - Runs entirely on your machine when using local models
+  - Configurable search settings
+  - Transparent data handling
+
+- 🌐 **Enhanced Search Integration**
+  - **Auto-selection of search sources**: The "auto" search engine intelligently analyzes your query and selects the most appropriate search engine
+  - Multiple search engines including Wikipedia, arXiv, PubMed, Semantic Scholar, and more
+  - **Local RAG search for private documents** - search your own documents with vector embeddings
+  - Full webpage content retrieval and intelligent filtering
+
+- 🎓 **Academic & Scientific Integration**
+  - Direct integration with PubMed, arXiv, Wikipedia, Semantic Scholar
+  - Properly formatted citations from academic sources
+  - Report structure suitable for literature reviews
+  - Cross-disciplinary synthesis of information
+
+## Configuration System
+
+The package automatically creates and manages configuration files in your user directory:
+
+- **Windows**: `Documents\LearningCircuit\local-deep-research\config\`
+- **Linux/Mac**: `~/.config/local_deep_research/config/`
+
+### Default Configuration Files
+
+When you first run the tool, it creates these configuration files:
+
+| File | Purpose |
+|------|---------|
+| `settings.toml` | General settings for research, web interface, and search |
+| `llm_config.py` | Configure which LLM to use (local or cloud-based) |
+| `search_engines.toml` | Define and configure search engines |
+| `local_collections.toml` | Configure local document collections for RAG |
+| `.secrets.toml` | Store API keys for cloud services |
+
+## Setting Up AI Models
+
+The system supports multiple LLM providers:
+
+### Local Models (via Ollama)
+
+1. [Install Ollama](https://ollama.ai) 
+2. Pull a model: `ollama pull gemma3:12b` (recommended model)
+3. Ollama runs on port 11434 by default
+
+### Cloud Models
+
+Edit your `.secrets.toml` file to add API keys:
+
+```toml
+ANTHROPIC_API_KEY = "your-api-key-here"      # For Claude models
+OPENAI_API_KEY = "your-openai-key-here"      # For GPT models
+OPENAI_ENDPOINT_API_KEY = "your-key-here"    # For OpenRouter or similar services
+```
+
+Then edit `llm_config.py` to change the default provider:
+
+```python
+# Set your preferred model provider here
+DEFAULT_PROVIDER = ModelProvider.OLLAMA  # Change to your preferred provider
+
+# Set your default model name here
+DEFAULT_MODEL = "mistral"  # Change to your preferred model
+```
+
+### Supported LLM Providers
+
+The system supports multiple LLM providers:
+
+| Provider | Type | Configuration | Notes |
+|----------|------|--------------|-------|
+| `OLLAMA` | Local | No API key needed | Default - install from ollama.ai |
+| `OPENAI` | Cloud | Requires `OPENAI_API_KEY` | GPT models (3.5, 4, 4o) |
+| `ANTHROPIC` | Cloud | Requires `ANTHROPIC_API_KEY` | Claude models (3 Opus, Sonnet, Haiku) |
+| `OPENAI_ENDPOINT` | Cloud | Requires `OPENAI_ENDPOINT_API_KEY` | For any OpenAI-compatible API |
+| `VLLM` | Local | No API key needed | For hosting models via vLLM |
+
+You can configure the OpenAI-compatible endpoint URL in `llm_config.py`:
+
+```python
+# For OpenRouter, Together.ai, Azure OpenAI, or any compatible endpoint
+OPENAI_ENDPOINT_URL = "https://openrouter.ai/api/v1"
+```
+
+The `OPENAI_ENDPOINT` provider can access any service with an OpenAI-compatible API, including:
+- OpenRouter (access to hundreds of models)
+- Azure OpenAI
+- Together.ai
+- Groq
+- Anyscale
+- Self-hosted LLM servers with OpenAI compatibility
+- Any other service that implements the OpenAI API specification
+
+## Setting Up Search Engines
+
+The system includes multiple search engines. Some require API keys:
+
+```toml
+# Add to .secrets.toml
+SERP_API_KEY = "your-serpapi-key-here"        # For Google results via SerpAPI
+GOOGLE_PSE_API_KEY = "your-google-key-here"   # For Google Programmable Search
+GOOGLE_PSE_ENGINE_ID = "your-pse-id-here"     # For Google Programmable Search
+BRAVE_API_KEY = "your-brave-search-key-here"  # For Brave Search
+GUARDIAN_API_KEY = "your-guardian-key-here"   # For The Guardian
+```
+
+No API key required for: Wikipedia, arXiv, PubMed, Semantic Scholar, and local collections.
+
+## Local Document Search (RAG)
+
+The system can search through your local documents using vector embeddings.
+
+### Setting Up Document Collections
+
+1. Define collections in `local_collections.toml`. Default collections include:
+
+```toml
+[project_docs]
+name = "Project Documents"
+description = "Project documentation and specifications"
+paths = ["@format ${DOCS_DIR}/project_documents"]
+enabled = true
+embedding_model = "all-MiniLM-L6-v2"
+embedding_device = "cpu"
+embedding_model_type = "sentence_transformers"
+max_results = 20
+max_filtered_results = 5
+chunk_size = 1000
+chunk_overlap = 200
+cache_dir = "__CACHE_DIR__/local_search/project_docs"
+
+# More collections defined in the file...
+```
+
+2. Create your document directories:
+   - The `${DOCS_DIR}` variable points to a default location in your Documents folder
+   - Documents are automatically indexed when the search is first used
+
+### Using Local Search
+
+You can use local document search in several ways:
+
+1. **Auto-selection**: Set `tool = "auto"` in `settings.toml` [search] section
+2. **Explicit collection**: Set `tool = "project_docs"` to search only that collection
+3. **All collections**: Set `tool = "local_all"` to search across all collections
+4. **Query syntax**: Type `collection:project_docs your query` to target a specific collection
+
+## Available Search Engines
+
+| Engine | Purpose | API Key Required? | Rate Limit |
+|--------|---------|-------------------|------------|
+| `auto` | Intelligently selects the best engine | No | Based on selected engine |
+| `wikipedia` | General knowledge and facts | No | No strict limit |
+| `arxiv` | Scientific papers and research | No | No strict limit |
+| `pubmed` | Medical and biomedical research | No | No strict limit |
+| `semantic_scholar` | Academic literature across all fields | No | 100/5min |
+| `github` | Code repositories and documentation | No | 60/hour (unauthenticated) |
+| `brave` | Web search (privacy-focused) | Yes | Based on plan |
+| `serpapi` | Google search results | Yes | Based on plan |
+| `google_pse` | Custom Google search | Yes | 100/day free tier |
+| `wayback` | Historical web content | No | No strict limit |
+| `searxng` | Local web search engine | No (requires local server) | No limit |
+| Any collection name | Search your local documents | No | No limit |
+
+> **Support Free Knowledge:** If you frequently use the search engines in this tool, please consider making a donation to these organizations. They provide valuable services and rely on user support to maintain their operations:
+> - [Donate to Wikipedia](https://donate.wikimedia.org)
+> - [Support arXiv](https://arxiv.org/about/give)
+> - [Donate to DuckDuckGo](https://duckduckgo.com/donations)
+> - [Support PubMed/NCBI](https://www.nlm.nih.gov/pubs/donations/donations.html)
+
+## Advanced Configuration
+
+### Research Parameters
+
+Edit `settings.toml` to customize research parameters:
+
+```toml
+[search]
+# Search tool to use (auto, wikipedia, arxiv, etc.)
+tool = "auto"
+
+# Number of research cycles
+iterations = 2
+
+# Questions generated per cycle
+questions_per_iteration = 2
+
+# Results per search query
+max_results = 50
+
+# Results after relevance filtering
+max_filtered_results = 5
+
+# More settings available...
+```
+
+## Web Interface
+
+The web interface offers several features:
+
+- **Dashboard**: Start and manage research queries
+- **Real-time Updates**: Track research progress
+- **Research History**: Access past queries
+- **PDF Export**: Download reports
+- **Research Management**: Terminate processes or delete records
+
+![Web Interface](./web1.png)
+![Web Interface](./web2.png)
+
+## Command Line Interface
+
+The CLI version allows you to:
+
+1. Choose between a quick summary or detailed report
+2. Enter your research query
+3. View results directly in the terminal
+4. Save reports automatically to the configured output directory
+
+## Development Setup
+
+If you want to develop or modify the package, you can install it in development mode:
+
+```bash
+# Clone the repository
+git clone https://github.com/LearningCircuit/local-deep-research.git
+cd local-deep-research
+
+# Install in development mode
+pip install -e .
+```
+
+This creates an "editable" installation that uses your local code, so any changes you make are immediately available without reinstalling.
+
+You can run the application directly using Python module syntax:
+
+```bash
+# Run the web interface
+python -m local_deep_research.web.app
+
+# Run the CLI version
+python -m local_deep_research.main
+```
+
+This approach is useful for development and debugging, as it provides more detailed error messages and allows you to make code changes on the fly.
+
+## Example Research
+
+The repository includes complete research examples like our [fusion energy research analysis](https://github.com/LearningCircuit/local-deep-research/blob/main/examples/fusion-energy-research-developments.md) showcasing the system's capabilities.
+
+## Community & Support
+
+Join our [Discord server](https://discord.gg/2E6gYU2Z) to exchange ideas, discuss usage patterns, and share research approaches.
+
+## License
+
+This project is licensed under the MIT License.
+
+## Acknowledgments
+
+- Built with [Ollama](https://ollama.ai) for local AI processing
+- Search powered by multiple sources:
+  - [Wikipedia](https://www.wikipedia.org/) for factual knowledge (default search engine)
+  - [arXiv](https://arxiv.org/) for scientific papers
+  - [PubMed](https://pubmed.ncbi.nlm.nih.gov/) for biomedical literature
+  - [DuckDuckGo](https://duckduckgo.com) for web search
+  - [The Guardian](https://www.theguardian.com/) for journalism
+  - [SerpAPI](https://serpapi.com) for Google search results (requires API key)
+  - [SearXNG](https://searxng.org/) for local web-search engine
+  - [Brave Search](https://search.brave.com/) for privacy-focused web search
+  - [Semantic Scholar](https://www.semanticscholar.org/) for academic literature
+- Built on [LangChain](https://github.com/hwchase17/langchain) framework
+- Uses [justext](https://github.com/miso-belica/justext), [Playwright](https://playwright.dev), [FAISS](https://github.com/facebookresearch/faiss), and more
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request

{local_deep_research-0.1.16.dist-info → local_deep_research-0.1.18.dist-info}/RECORD

@@ -1,23 +1,22 @@
 local_deep_research/__init__.py,sha256=pfHzjzYc6Szo8VCNLtFZRXyAlEz7CViY7r2fH9O7yms,584
-local_deep_research/citation_handler.py,sha256=1Znl5SPkVwOZ3JvFowvaW1uriaw37s1nPYhXbpa2UZ0,4423
-local_deep_research/config.py,sha256=lucqOE4KeNm1ynYdcHYWJLE5fJ0QN-1QKZpRcBPsHe8,8674
-local_deep_research/local_collections.py,sha256=SB-fdptT7qS0klJUVx_Rs9OgDwafMUgI46984WlZGKI,6076
+local_deep_research/citation_handler.py,sha256=v_fwTy-2XvUuoH3OQRzmBrvaiN7mBk8jbNfySslmt5g,4357
+local_deep_research/config.py,sha256=PAruLZutlrjkGOKrv49hk8U4q9JPWWgAKLiY8Ukpsks,8572
 local_deep_research/main.py,sha256=uQXtGQ6LtZNd5Qw63D5ke4Q_LjYimouWVSUknVsk3JQ,3645
-local_deep_research/report_generator.py,sha256=UOiSw_vPHgtUpI8L9_UaOlpBVBloPB-ilhAo-1d2B9M,8200
-local_deep_research/search_system.py,sha256=KNSn_8ciEGfSRR0k0ggIzZF6jqNXoYsCnUmdKgtKO0E,15481
+local_deep_research/report_generator.py,sha256=EvaArnWirMgg42fMzmZeJczoEYujEbJ2ryHHYuuoXx8,8058
+local_deep_research/search_system.py,sha256=yY3BEzX68vdtUcYF9h6lC3yVao0YA_NSBj6W3-RwlKk,15459
 local_deep_research/defaults/__init__.py,sha256=2Vvlkl-gmP_qPYWegE4JBgummypogl3VXrQ1XzptFDU,1381
-local_deep_research/defaults/llm_config.py,sha256=88IGWPPvikSKmAqfqsGovBx2Jac5eh2sBY_LIW624Ik,7910
+local_deep_research/defaults/llm_config.py,sha256=T03pntyNtOk1fvu-RZ-iEoh7L2D2hcICr8usIPpULuo,7870
 local_deep_research/defaults/local_collections.toml,sha256=_edVWVHrhunMfazjejhJlGPRkHKKIP51qQtNkMgNEiA,1406
-local_deep_research/defaults/main.toml,sha256=l_J9JAPhKEp63IsLBO0hQDVimxogEpnrEVnNjiOeUxg,1403
+local_deep_research/defaults/main.toml,sha256=DLhFq88vdE2_psLaWhPV9BWPixqTHvR2Rllaj_rmjJ4,1403
 local_deep_research/defaults/search_engines.toml,sha256=TYkOqVaZq9JPawz4fIPyGdkAtYa4t8F9H50VY-wv2ak,8101
 local_deep_research/utilties/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 local_deep_research/utilties/enums.py,sha256=TVAZiu9szNbdacfb7whgaQJJlSk7oYByADaAierD4CE,229
 local_deep_research/utilties/llm_utils.py,sha256=IGv-_gJWqLTpO3_op1NHIwxKaFEzmXhhVYSLTTSMnIA,4522
-local_deep_research/utilties/search_utilities.py,sha256=C8ycFd7blcq5vtnd6GxP8dkepZT6EEqHFtT3WYxF0Ck,4151
+local_deep_research/utilties/search_utilities.py,sha256=-wj_-HCotqDUXYcPaKGN1C_t6WH4xubNhem4RE60AuM,4098
 local_deep_research/utilties/setup_utils.py,sha256=t6GNp7lK1nLPdPNCkYUk82IATGM62vqy8UBD-KqShOs,215
 local_deep_research/web/__init__.py,sha256=3oHMatNu8r24FBtpojriIVbHYOVSHj4Q-quycMKOuDk,62
 local_deep_research/web/app.py,sha256=5_VLNdhJOqdgacucglUdS_lVURNgYNbXhK9vME6JmzA,72431
-local_deep_research/web/static/css/styles.css,sha256=mW217FfZNW1pzMtlbuXE2fRBJekeIdIoy4m-yXFirj4,23782
+local_deep_research/web/static/css/styles.css,sha256=_26yBV1fKM51Dfv67CxKSbK8aeoYK5Tl7b2TPs5Whuo,24641
 local_deep_research/web/static/js/app.js,sha256=GPncdWpw2YNTs56JY-0tjTTr9JnX-fIZSZX0agwKZMU,172813
 local_deep_research/web/templates/api_keys_config.html,sha256=jA8Y-nfUGJ1dTvbw2jK_8xPy2x6UG_5gHpbrTJAex2g,3527
 local_deep_research/web/templates/collections_config.html,sha256=Dci7KumXBON8rAXRX8TVjgqS-bbht7d6aQiedDUnxQ0,3560
@@ -28,30 +27,29 @@ local_deep_research/web/templates/search_engines_config.html,sha256=z_krznfdhF3e
 local_deep_research/web/templates/settings.html,sha256=S9A-tdpzMhP2Zw7kp2jxKlwaWtoRil_IHX_a376vXsA,29575
 local_deep_research/web/templates/settings_dashboard.html,sha256=De-v1KNdVvkXme5i3YZ6sIfU9aAKDc_N-AW9n4PZoso,9109
 local_deep_research/web_search_engines/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-local_deep_research/web_search_engines/full_search.py,sha256=3SSTvD12g4pNlZCSGh8jwsyYWpQglgqjADnq8dG1zyI,9756
-local_deep_research/web_search_engines/search_engine_base.py,sha256=Knmf45pMYd7hYc9x8jG8gOtMnribsXDcOXGrA50LK3E,8100
-local_deep_research/web_search_engines/search_engine_factory.py,sha256=B_QaqoAwnVXCmHNdqGbo94LekWY6wpBw_PWNkI120qE,10728
+local_deep_research/web_search_engines/search_engine_base.py,sha256=QmhfjuHK2deomh8tARghKuYnF-5t3wwBB661odS2VtU,8065
+local_deep_research/web_search_engines/search_engine_factory.py,sha256=Sld6bYTwcyTxgVLx04t00sD7vfJhSHFOl6iiGJ08ZUE,11118
 local_deep_research/web_search_engines/search_engines_config.py,sha256=bNCuR09NOk5cjnKIgDQfhPipqmvDKeE7WP_6p8LLZf0,1979
 local_deep_research/web_search_engines/engines/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-local_deep_research/web_search_engines/engines/full_search.py,sha256=mcxS8o7-WmOQc3_H4232adhBHevZfSHWmaOFoia68UU,4711
+local_deep_research/web_search_engines/engines/full_search.py,sha256=BuOz8dX-XocazCG7gGBKFnIY99FZtNFI0-Wq3fhsfp4,4689
 local_deep_research/web_search_engines/engines/meta_search_engine.py,sha256=Zky4sowCortEaIj1pBU0sKuggXr5izkubgrD7cM8IOQ,11485
-local_deep_research/web_search_engines/engines/search_engine_arxiv.py,sha256=WlKQSB1kgdEaLtDj4JwpIiP3eZPustJt4ryJ9wiulT0,15316
-local_deep_research/web_search_engines/engines/search_engine_brave.py,sha256=zAofutKEnXb_CYAi7hrptNVrQ15PQoYK_Eg3spcD1Ig,9583
-local_deep_research/web_search_engines/engines/search_engine_ddg.py,sha256=-_rWr2uwSoxHLkAV-WrHj_vuoSActr_jzfveI7dE10c,4845
+local_deep_research/web_search_engines/engines/search_engine_arxiv.py,sha256=cf8OzhSzE1zqaiOZ6EFQGy_6hTCJMaTysYd8rs1KJNU,15408
+local_deep_research/web_search_engines/engines/search_engine_brave.py,sha256=J242byUGG5ROQ_bh-mU292_t7Q7m20_9O0r1w5z6d9A,9688
+local_deep_research/web_search_engines/engines/search_engine_ddg.py,sha256=qK2i65dbPtr_ppoKPU_YA0mDqM_sDAvN6ZztvdFjsCk,4910
 local_deep_research/web_search_engines/engines/search_engine_github.py,sha256=qqipsw2ycjlRbR6mmMmxzGU3LEcFDJJJ7Ez7xUgWjRM,26768
 local_deep_research/web_search_engines/engines/search_engine_google_pse.py,sha256=YkXvBmgcqTImCxuyy6580SGRAvImGc6SzInXZgo1kNE,11294
 local_deep_research/web_search_engines/engines/search_engine_guardian.py,sha256=MW4WIwtNAwcpdigNXronyezAxr50EIZTV1NMedrAv2o,23912
-local_deep_research/web_search_engines/engines/search_engine_local.py,sha256=uAsNtaFV3AH4ltNpudHIl546jsOKKyGDjUwU5J7gKts,36766
+local_deep_research/web_search_engines/engines/search_engine_local.py,sha256=rfmPiA9DVmjbaB3KQtlq7s6BRMgHRgzP7AhktZNDw2M,36772
 local_deep_research/web_search_engines/engines/search_engine_local_all.py,sha256=CRNcxBzNd9kanyIJYaUDB7qfXYxVCvd4L2mX8jL73v0,5955
 local_deep_research/web_search_engines/engines/search_engine_pubmed.py,sha256=MayfzM2R0XoI7cpXlG1XJ1ktfTN_6H-Xs9RmD89UAao,39236
 local_deep_research/web_search_engines/engines/search_engine_searxng.py,sha256=GMy6qDMSaVBtjWRm48XBu6TjLAy1HfcO2EFTwr8S9rk,18048
 local_deep_research/web_search_engines/engines/search_engine_semantic_scholar.py,sha256=6VMymjFJ7pyV2nv5dRfFofXgg0kG82rkwbICVnNDNH4,23352
-local_deep_research/web_search_engines/engines/search_engine_serpapi.py,sha256=iy-QmT99Tf2cJlfCrPbEhtMB7a_zCKppvlUKi7VBrlE,9118
+local_deep_research/web_search_engines/engines/search_engine_serpapi.py,sha256=XikEYnM-pAaR70VeAJ28lbqpRzCj4bCA9xY29taTV8g,9215
 local_deep_research/web_search_engines/engines/search_engine_wayback.py,sha256=astAvSLajDZ6rwgthJ3iBcHSWuDSYPO7uilIxaJhXmU,18132
 local_deep_research/web_search_engines/engines/search_engine_wikipedia.py,sha256=KSGJECbEcxZpVK-PhYsTCtzedSK0l1AjQmvGtx8KBks,9799
-local_deep_research-0.1.16.dist-info/licenses/LICENSE,sha256=Qg2CaTdu6SWnSqk1_JtgBPp_Da-LdqJDhT1Vt1MUc5s,1072
-local_deep_research-0.1.16.dist-info/METADATA,sha256=KHrqDTRQmo_FAt1KFZLmYlbO0eQKfjqvpHFu_kRUd_w,15151
-local_deep_research-0.1.16.dist-info/WHEEL,sha256=1tXe9gY0PYatrMPMDd6jXqjfpz_B-Wqm32CPfRC58XU,91
-local_deep_research-0.1.16.dist-info/entry_points.txt,sha256=u-Y6Z3MWtR3dmsTDFYhXyfkPv7mALUA7YAnY4Fi1XDs,97
-local_deep_research-0.1.16.dist-info/top_level.txt,sha256=h6-uVE_wSuLOcoWwT9szhX23mBWufu77MqmM25UfbCY,20
-local_deep_research-0.1.16.dist-info/RECORD,,
+local_deep_research-0.1.18.dist-info/licenses/LICENSE,sha256=Qg2CaTdu6SWnSqk1_JtgBPp_Da-LdqJDhT1Vt1MUc5s,1072
+local_deep_research-0.1.18.dist-info/METADATA,sha256=6F15QvEwfPPqHVEGRUCPEYVXKoR8PvbXOwzM9Vemem8,15013
+local_deep_research-0.1.18.dist-info/WHEEL,sha256=1tXe9gY0PYatrMPMDd6jXqjfpz_B-Wqm32CPfRC58XU,91
+local_deep_research-0.1.18.dist-info/entry_points.txt,sha256=u-Y6Z3MWtR3dmsTDFYhXyfkPv7mALUA7YAnY4Fi1XDs,97
+local_deep_research-0.1.18.dist-info/top_level.txt,sha256=h6-uVE_wSuLOcoWwT9szhX23mBWufu77MqmM25UfbCY,20
+local_deep_research-0.1.18.dist-info/RECORD,,

local_deep_research/local_collections.py

@@ -1,141 +0,0 @@
-# local_collections.py
-"""
-Configuration file for local document collections.
-Each collection functions as an independent search engine.
-"""
-
-import os
-from typing import Dict, Any
-
-# Registry of local document collections
-# Each collection appears as a separate search engine in the main configuration
-LOCAL_COLLECTIONS = {
-    # Project Documents Collection
-    "project_docs": {
-        "name": "Project Documents",
-        "description": "Project documentation and specifications",
-        "paths": [os.path.abspath("./local_search_files/project_documents")],
-        "enabled": True,
-        "embedding_model": "all-MiniLM-L6-v2",
-        "embedding_device": "cpu",
-        "embedding_model_type": "sentence_transformers",
-        "max_results": 20,
-        "max_filtered_results": 5,
-        "chunk_size": 1000,
-        "chunk_overlap": 200,
-        "cache_dir": ".cache/local_search/project_docs"
-    },
-    
-    # Research Papers Collection
-    "research_papers": {
-        "name": "Research Papers",
-        "description": "Academic research papers and articles",
-        "paths": [os.path.abspath("local_search_files/research_papers")],
-        "enabled": True,
-        "embedding_model": "all-MiniLM-L6-v2",
-        "embedding_device": "cpu",
-        "embedding_model_type": "sentence_transformers",
-        "max_results": 20,
-        "max_filtered_results": 5,
-        "chunk_size": 800,  # Smaller chunks for academic content
-        "chunk_overlap": 150,
-        "cache_dir": ".cache/local_search/research_papers"
-    },
-    
-    # Personal Notes Collection
-    "personal_notes": {
-        "name": "Personal Notes",
-        "description": "Personal notes and documents",
-        "paths": [os.path.abspath("./local_search_files/personal_notes")],
-        "enabled": True,
-        "embedding_model": "all-MiniLM-L6-v2",
-        "embedding_device": "cpu",
-        "embedding_model_type": "sentence_transformers",
-        "max_results": 30,
-        "max_filtered_results": 10,
-        "chunk_size": 500,  # Smaller chunks for notes
-        "chunk_overlap": 100,
-        "cache_dir": ".cache/local_search/personal_notes"
-    }
-}
-
-# Configuration for local search integration
-LOCAL_SEARCH_CONFIG = {
-    # General embedding options
-    "DEFAULT_EMBEDDING_MODEL": "all-MiniLM-L6-v2",
-    "DEFAULT_EMBEDDING_DEVICE": "cpu",  # "cpu" or "cuda" for GPU acceleration
-    "DEFAULT_EMBEDDING_MODEL_TYPE": "sentence_transformers",  # or "ollama"
-    
-    # Ollama settings (only used if model type is "ollama")
-    # Note: You must run 'ollama pull nomic-embed-text' first if using Ollama for embeddings
-    "OLLAMA_BASE_URL": "http://localhost:11434",
-    "OLLAMA_EMBEDDING_MODEL": "nomic-embed-text",
-    
-    # Default indexing options
-    "FORCE_REINDEX": True,  # Force reindexing on startup
-    "CACHE_DIR": ".cache/local_search",  # Base directory for cache
-}
-
-def register_local_collections(search_engines_dict: Dict[str, Any]) -> None:
-    """
-    Register all enabled local collections as search engines.
-    
-    Args:
-        search_engines_dict: The main search engines dictionary to update
-    """
-    for collection_id, collection in LOCAL_COLLECTIONS.items():
-        print(collection_id, collection)
-        if collection.get("enabled", True):
-            # Skip if already defined (don't override)
-            if collection_id in search_engines_dict:
-                continue
-                
-            # Validate paths exist
-            paths = collection.get("paths", [])
-            valid_paths = []
-            for path in paths:
-                if os.path.exists(path) and os.path.isdir(path):
-                    valid_paths.append(path)
-                else:
-                    print(f"Warning: Collection '{collection_id}' contains non-existent folder: {path}")
-            
-            # Log warning if no valid paths
-            if not valid_paths and paths:
-                print(f"Warning: Collection '{collection_id}' has no valid folders. It will be registered but won't return results.")
-                
-            # Create a search engine entry for this collection
-            search_engines_dict[collection_id] = {
-                "module_path": "local_deep_research.web_search_engines.engines.search_engine_local",
-                "class_name": "LocalSearchEngine",
-                "requires_api_key": False,
-                "reliability": 0.9,  # High reliability for local documents
-                "strengths": ["personal documents", "offline access", 
-                             collection.get("description", "local documents")],
-                "weaknesses": ["requires indexing", "limited to specific folders"],
-                "default_params": {
-                    "folder_paths": collection.get("paths", []),
-                    "embedding_model": collection.get(
-                        "embedding_model", 
-                        LOCAL_SEARCH_CONFIG["DEFAULT_EMBEDDING_MODEL"]
-                    ),
-                    "embedding_device": collection.get(
-                        "embedding_device", 
-                        LOCAL_SEARCH_CONFIG["DEFAULT_EMBEDDING_DEVICE"]
-                    ),
-                    "embedding_model_type": collection.get(
-                        "embedding_model_type", 
-                        LOCAL_SEARCH_CONFIG["DEFAULT_EMBEDDING_MODEL_TYPE"]
-                    ),
-                    "chunk_size": collection.get("chunk_size", 1000),
-                    "chunk_overlap": collection.get("chunk_overlap", 200),
-                    "cache_dir": collection.get(
-                        "cache_dir", 
-                        f"{LOCAL_SEARCH_CONFIG['CACHE_DIR']}/{collection_id}"
-                    ),
-                    "max_results": collection.get("max_results", 20),
-                    "max_filtered_results": collection.get("max_filtered_results", 5),
-                    "collection_name": collection.get("name", collection_id),
-                    "collection_description": collection.get("description", "")
-                },
-                "requires_llm": True
-            }