dotagent-web-research 1.2.0__py3-none-any.whl

dotagent_web_research-1.2.0.dist-info/METADATA

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: dotagent-web-research
+Version: 1.2.0
+Summary: MCP server for web search and reading (search, scrape, map, extract, crawl) that Dot, an always-on agent for macOS, uses
+Author-email: Senol Dogan <senoldogan@users.noreply.github.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/senoldogann/web-research-stack
+Project-URL: Repository, https://github.com/senoldogann/web-research-stack.git
+Project-URL: Issues, https://github.com/senoldogann/web-research-stack/issues
+Project-URL: Documentation, https://github.com/senoldogann/web-research-stack/tree/main/docs
+Keywords: mcp,model-context-protocol,web-search,web-scraping,crawler,dot,agent
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.26.0
+Requires-Dist: curl_cffi>=0.7.3
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: lxml>=4.9.0
+Requires-Dist: defusedxml>=0.7.1
+Requires-Dist: fastapi>=0.109.0
+Requires-Dist: uvicorn[standard]>=0.27.0
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: python-dotenv>=1.1.0
+Requires-Dist: click>=8.1.0
+Requires-Dist: tqdm>=4.66.0
+Requires-Dist: pypdf>=5.1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: httpx>=0.26.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: redis[hiredis]>=5.0.0; extra == "dev"
+Provides-Extra: playwright
+Requires-Dist: playwright>=1.40.0; extra == "playwright"
+Requires-Dist: playwright-stealth>=1.0.6; extra == "playwright"
+Provides-Extra: camoufox
+Requires-Dist: camoufox[geoip]>=0.4.0; extra == "camoufox"
+Provides-Extra: redis
+Requires-Dist: redis[hiredis]>=5.0.0; extra == "redis"
+Dynamic: license-file
+
+# Web Research Stack / Web Araştırma Yığını
+
+<div align="center">
+  https://github.com/senoldogann/web-research-stack/raw/main/GITHUB-video.mp4
+  <br/>
+  <em>2-Minute Demonstration / 2 Dakikalık Tanıtım Videosu</em>
+</div>
+
+[English](#english) | [Türkçe](#türkçe)
+
+---
+
+## English
+
+### Overview
+
+**Web Scraper** is a powerful, production-ready web scraping and research tool with full-stack capabilities. It combines a Python FastAPI backend with a modern React Next.js frontend, featuring LLM-powered research capabilities with citation support.
+
+### Install
+
+The MCP server for web search and reading that Dot, the always-on agent for macOS, uses ships as a Python package, `dotagent-web-research` (the import package is `web_scraper`):
+
+```bash
+pip install dotagent-web-research
+# or, as a command of its own
+uv tool install dotagent-web-research
+```
+
+It installs two commands: `web-research-mcp`, the MCP server over stdio (tools `web_search`, `web_scrape`, `web_map`, `web_extract`, `web_crawl_start`, `web_crawl_status`, `web_crawl_results`, `web_crawl_cancel`), and `web-scraper`, the CLI and REST server described below. The headless-browser fallbacks are extras: `pip install "dotagent-web-research[camoufox]"` or `"[playwright]"`.
+
+**How Dot uses it.** Dot's package depends on this one, so a fresh Dot install brings it. Dot starts `web-research-mcp` as one of its MCP servers (named `web-research`) and its tools reach the agent as `mcp__web-research__web_search`, `mcp__web-research__web_scrape` and so on. In Dot's `config.toml` it is one table; [docs/web-tools.md](docs/web-tools.md) has it and the settings (`WEB_TOOLS_*`) the server reads.
+
+```toml
+[[mcp]]
+name = "web-research"
+command = "web-research-mcp"
+```
+
+### Features
+
+- **Dual Interface**: CLI commands and REST API
+- **Local Web Research Tools**: HTTP-first multi-angle search with field-weighted BM25, weighted angle consensus, freshness filters, caching and provider cooldown, plus scrape, map, background crawl, and structured extract, exposed over REST and a Dot-ready stdio MCP server (headless-only browser fallback)
+- **LLM Research Tool**: AI-powered web research with citations
+- **Async Architecture**: High-performance concurrent scraping
+- **Production Ready**: Docker, monitoring (Prometheus/Grafana), rate limiting, circuit breaker
+- **Security First**: SSRF protection, input validation, security headers
+- **Multi-language UI**: English and Turkish language support
+- **Streaming Support**: Real-time research progress via Server-Sent Events (SSE)
+- **Code Block Preservation**: Scraper preserves code blocks from source pages with syntax highlighting
+- **Internationalized Status Messages**: Research progress messages adapt to query language (TR/EN)
+- **Resilient JSON Parsing**: 3-stage recovery for LLM synthesis output (strict → repair → retry)
+
+### Tech Stack
+
+| Component | Technology |
+|-----------|-------------|
+| Backend | Python 3.9+, FastAPI, asyncio |
+| Frontend | React 19, Next.js 15, TypeScript, Tailwind CSS |
+| Database | SQLite (history & caching) |
+| HTTP Client | httpx, curl_cffi |
+| HTML Parsing | BeautifulSoup4, lxml |
+| AI/ML | Ollama, OpenAI API compatible |
+
+### Quick Start
+
+```bash
+# Clone the repository
+git clone https://github.com/senoldogann/web-research-stack.git
+cd web-research-stack
+
+# Install dependencies
+pip install -e .
+
+# Start the API server
+web-scraper serve
+
+# Or use Docker
+docker compose up --build
+```
+
+### Google search stays off your screen
+
+Google no longer serves results to plain HTTP clients, and headless Chrome is caught within one request. A headed Chrome gets the real page, so the searcher starts Chrome itself, **minimises its window over CDP before the first search**, and keeps one browser and one tab for the process: ~2 s to start (a brief flash of a small window), then 0.4–1 s per search. If Chrome is missing or refused, **Camoufox** — a patched headless Firefox — takes over.
+
+```bash
+pip install -e ".[playwright]"   # the Chrome path
+pip install -e ".[camoufox]" && python -m camoufox fetch   # the headless fallback
+```
+
+- `GOOGLE_ALLOW_VISIBLE_BROWSER=true` — leave the Chrome window on screen (debugging).
+- `GOOGLE_USE_PLAYWRIGHT=false` — skip Google entirely and rely on DuckDuckGo.
+
+### API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/` | GET | API information |
+| `/health` | GET | Health check |
+| `/api/v1/metrics` | GET | Prometheus metrics |
+| `/api/v1/tools` | GET | Tool manifest for LLM |
+| `/api/v1/scrape` | POST | Scrape a single URL |
+| `/api/v1/scrape/batch` | POST | Batch scrape multiple URLs |
+| `/api/v1/tools/web-research` | POST | Run AI-powered research |
+| `/api/v1/tools/web-research/stream` | POST | Stream research progress |
+| `/api/v1/web/search` | POST | Discover URLs with a search adapter |
+| `/api/v1/web/scrape` | POST | Scrape one page to Markdown, metadata, and links |
+| `/api/v1/web/map` | POST | List URLs from sitemaps and page links |
+| `/api/v1/web/extract` | POST | Extract structured fields with provenance |
+| `/api/v1/web/crawl` | POST | Start a background crawl and return its job id |
+| `/api/v1/web/crawl/{job_id}` | GET | Crawl progress and counters |
+| `/api/v1/web/crawl/{job_id}/results` | GET | Paginated crawl results |
+| `/api/v1/web/crawl/{job_id}/cancel` | POST | Cancel a queued or in-flight crawl |
+| `/api/v1/web/stats` | GET | Web tool latency, cache, and browser state |
+| `/api/v1/research` | POST | Legacy research endpoint |
+| `/api/v1/research/stream` | POST | Legacy streaming research |
+
+### Example: Research Request
+
+```bash
+curl -X POST "http://127.0.0.1:8000/api/v1/tools/web-research" \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-api-key" \
+  -d '{
+    "query": "Latest developments in AI",
+    "max_sources": 5,
+    "deep_mode": false,
+    "provider": "ollama"
+  }'
+```
+
+### Documentation
+
+- [Installation Guide](./docs/installation-tr.md#installation-guide)
+- [User Guide](./docs/user-guide.md#english)
+- [API Documentation](./docs/api.md#english)
+- [LLM Integration](./docs/llm-tool-integration.md)
+- [Local Web Tools](./docs/web-tools.md)
+- [Production Deployment](./docs/production-deployment.md)
+
+### License
+
+MIT License
+
+---
+
+## Türkçe
+
+### Genel Bakış
+
+**Web Scraper**, güçlü ve üretim ortamına hazır bir web kazıma ve araştırma aracıdır. Python FastAPI backend ile modern React Next.js frontend'i birleştirir, alıntılı LLM destekli araştırma özelliklerine sahiptir.
+
+### Kurulum
+
+Dot'un (macOS için her zaman açık ajan) kullandığı web arama ve okuma MCP sunucusu bir Python paketi olarak gelir: `dotagent-web-research` (içe aktarılan paket `web_scraper`):
+
+```bash
+pip install dotagent-web-research
+# veya kendi başına bir komut olarak
+uv tool install dotagent-web-research
+```
+
+İki komut kurar: stdio üzerinden MCP sunucusu `web-research-mcp` ve aşağıda anlatılan CLI ile REST sunucusu `web-scraper`. Headless tarayıcı yedekleri ek pakettir: `pip install "dotagent-web-research[camoufox]"` veya `"[playwright]"`.
+
+**Dot bunu nasıl kullanır.** Dot'un paketi buna bağlıdır; yeni bir Dot kurulumu bunu da getirir. Dot `web-research-mcp`'yi MCP sunucularından biri olarak (`web-research` adıyla) başlatır; araçlar ajana `mcp__web-research__web_search`, `mcp__web-research__web_scrape` gibi görünür. Ayrıntı ve ayarlar için [docs/web-tools.md](docs/web-tools.md).
+
+### Özellikler
+
+- **Çift Arayüz**: CLI komutları ve REST API
+- **Yerel Web Araştırma Araçları**: Alan ağırlıklı BM25, ağırlıklı açı uzlaşması, güncellik filtreleri, önbellek ve sağlayıcı soğuması içeren çok açılı HTTP öncelikli arama; kazıma, haritalama, arka plan taraması ve yapılandırılmış çıkarım; REST ve Dot'a hazır stdio MCP sunucusu (yalnızca headless tarayıcı yedeği)
+- **LLM Araştırma Aracı**: Alıntılı AI destekli web araştırması
+- **Asenkron Mimari**: Yüksek performanslı eşzamanlı kazıma
+- **Üretim Hazır**: Docker, izleme (Prometheus/Grafana), hız sınırlama, circuit breaker
+- **Güvenlik Öncelikli**: SSRF koruması, giriş doğrulama, güvenlik başlıkları
+- **Çok Dilli UI**: İngilizce ve Türkçe dil desteği
+- **Yayın Akışı**: Sunucu Tarafından Gönderilen Olaylar (SSE) ile gerçek zamanlı araştırma ilerlemesi
+- **Kod Bloğu Koruma**: Kaynak sayfalardan kod blokları söz dizimi vurgulamasıyla korunur
+- **Uluslararasılaştırılmış Durum Mesajları**: Araştırma ilerleme mesajları sorgu diline göre uyarlanır (TR/EN)
+- **Dayanıklı JSON Ayrıştırma**: LLM sentez çıktısı için 3 aşamalı kurtarma (katı → onarım → yeniden deneme)
+
+### Teknoloji Yığını
+
+| Bileşen | Teknoloji |
+|---------|-----------|
+| Backend | Python 3.9+, FastAPI, asyncio |
+| Frontend | React 19, Next.js 15, TypeScript, Tailwind CSS |
+| Veritabanı | SQLite (geçmiş ve önbellek) |
+| HTTP İstemcisi | httpx, curl_cffi |
+| HTML Ayrıştırma | BeautifulSoup4, lxml |
+| AI/ML | Ollama, OpenAI API uyumlu |
+
+### Hızlı Başlangıç
+
+```bash
+# Depoyu klonla
+git clone https://github.com/senoldogann/web-research-stack.git
+cd web-research-stack
+
+# Bağımlılıkları yükle
+pip install -e .
+
+# API sunucusunu başlat
+web-scraper serve
+
+# Veya Docker kullan
+docker compose up --build
+```
+
+### Google araması ekranda görünmez
+
+Google artık düz HTTP istemcilerine sonuç döndürmüyor ve headless Chrome tek istekte yakalanıyor. Başlı (headed) Chrome gerçek sayfayı alıyor; bu yüzden arayıcı Chrome'u kendisi başlatıp **penceresini ilk aramadan önce CDP ile küçültüyor** ve süreç boyunca tek tarayıcı + tek sekme kullanıyor: başlatma ~2 sn (küçük bir pencere kısa süre görünür), sonrası arama başına 0,4–1 sn. Chrome yoksa ya da reddederse **Camoufox** — yamalı, headless Firefox — devralıyor.
+
+```bash
+pip install -e ".[playwright]"   # Chrome yolu
+pip install -e ".[camoufox]" && python -m camoufox fetch   # görünmez yedek
+```
+
+- `GOOGLE_ALLOW_VISIBLE_BROWSER=true` — Chrome penceresi ekranda kalsın (hata ayıklama).
+- `GOOGLE_USE_PLAYWRIGHT=false` — Google'ı tamamen atla, DuckDuckGo kullan.
+
+### API Uç Noktaları
+
+| Uç Nokta | Metod | Açıklama |
+|----------|-------|----------|
+| `/` | GET | API bilgileri |
+| `/health` | GET | Sağlık kontrolü |
+| `/api/v1/metrics` | GET | Prometheus metrikleri |
+| `/api/v1/tools` | GET | LLM için araç manifestosu |
+| `/api/v1/scrape` | POST | Tek URL kazıma |
+| `/api/v1/scrape/batch` | POST | Toplu URL kazıma |
+| `/api/v1/tools/web-research` | POST | AI destekli araştırma |
+| `/api/v1/tools/web-research/stream` | POST | Araştırma ilerlemesi yayını |
+| `/api/v1/web/search` | POST | Arama adaptörüyle URL keşfi |
+| `/api/v1/web/scrape` | POST | Tek sayfayı Markdown, üst veri ve bağlantılara dönüştür |
+| `/api/v1/web/map` | POST | Site haritaları ve sayfa bağlantılarından URL listele |
+| `/api/v1/web/extract` | POST | Kaynak bilgisiyle yapılandırılmış alan çıkar |
+| `/api/v1/web/crawl` | POST | Arka plan taraması başlat, iş kimliğini hemen döndür |
+| `/api/v1/web/crawl/{job_id}` | GET | Tarama ilerlemesi ve sayaçları |
+| `/api/v1/web/crawl/{job_id}/results` | GET | Sayfalanmış tarama sonuçları |
+| `/api/v1/web/crawl/{job_id}/cancel` | POST | Sıradaki veya sürmekte olan taramayı iptal et |
+| `/api/v1/web/stats` | GET | Web aracı gecikmesi, önbellek ve tarayıcı durumu |
+| `/api/v1/research` | POST | Eski araştırma uç noktası |
+| `/api/v1/research/stream` | POST | Eski yayın akışı araştırması |
+
+### Örnek: Araştırma İsteği
+
+```bash
+curl -X POST "http://127.0.0.1:8000/api/v1/tools/web-research" \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: sizin-api-anahtariniz" \
+  -d '{
+    "query": "Yapay zeka son gelişmeler",
+    "max_sources": 5,
+    "deep_mode": false,
+    "provider": "ollama"
+  }'
+```
+
+### Dokümantasyon
+
+- [Kurulum Rehberi](./docs/installation-tr.md#kurulum-rehberi)
+- [Kullanım Kılavuzu](./docs/user-guide.md#türkçe)
+- [API Dokümantasyonu](./docs/api.md#türkçe)
+- [LLM Entegrasyonu](./docs/llm-tool-integration.md)
+- [Yerel Web Araçları](./docs/web-tools.md)
+- [Üretim Dağıtımı](./docs/production-deployment.md)
+
+### Lisans
+
+MIT Lisansı
+
+---
+
+## Contributing / Katkıda Bulunma
+
+Contributions are welcome! Please read our [Contributing Guide](./CONTRIBUTING.md).
+
+Katkılarınızı bekliyoruz! Lütfen [Katkı Rehberimizi](./CONTRIBUTING.md) okuyun.
+
+---
+
+&copy; 2026 Senol Dogan. MIT License.

dotagent_web_research-1.2.0.dist-info/RECORD

@@ -0,0 +1,60 @@
+dotagent_web_research-1.2.0.dist-info/licenses/LICENSE,sha256=97vp88nO0uKuooYG2QSz0rNfjuhHcMMij_dtgZn2guk,1068
+web_scraper/__init__.py,sha256=GJbvh_9N3M0uqnw8QgHhJoyGWYhD6JjLQLf43ZJF-qY,123
+web_scraper/ai_agent.py,sha256=_Kks-ps0vsw8R1tH9oFz2QybI8IBn8makEpgXAB5Hyk,6923
+web_scraper/api.py,sha256=U0oemES_Llqx9gh9DOvYF_bkXT5OeQFFYfTDwpvBCW4,59546
+web_scraper/api_models.py,sha256=ZEtQcPYrs5GrHt_pFPlzcDDnDy1Y3NVCxlWcDPZr1tM,25917
+web_scraper/api_runtime.py,sha256=17q8pyNjZJqHMb_5Q2uolCXM2zAbF5y7UcuLQkB3wIk,16352
+web_scraper/async_scrapers.py,sha256=C2_Sc1aILlqj1goeow1vUKxnoOHZkodK-LX8ooOtnLQ,14061
+web_scraper/cli.py,sha256=CEbcafI1aDNq6lH6RLZZOaK0tLIwFh6WO_wp9-uBarc,11555
+web_scraper/config.py,sha256=wWmcFM2nglJ_7mG_C27x4t-z-krZoCfQ5UYMT951s90,19097
+web_scraper/content_safety.py,sha256=zs6BlWBbLRW7j-AU3VZ5W4N4ku8Xa-XpgFogLIoYR4o,6591
+web_scraper/duckduckgo_search.py,sha256=sA4sDuFPE54M4Zbw4zW0qxTFfrJXYslzn4a-0G2Ua1Q,16599
+web_scraper/google_search.py,sha256=oA7n5pUYJXzFPAerHElZGyGfhcowbo5Uz2it-wUvLag,33781
+web_scraper/local_env.py,sha256=JugcZkJ5MjNhnAq7gukl1JnWFbAKPgpTtQCHtCktnMQ,951
+web_scraper/mcp_server.py,sha256=7zZLcR3F8ICM6VTgLNgJk6RRYrnC1DOXvG6Sn63aXwE,24947
+web_scraper/network_safety.py,sha256=xB-4cwjZn6TEt4JZj6YLppLQ_q0zcHePW-bBDPE3neU,3557
+web_scraper/playwright_scrapers.py,sha256=eyMGPcUkxTKrhaVcIqDDtjsJnli8Da0T-Awaay1QnOc,12309
+web_scraper/redis_cache.py,sha256=VoAlwxAchJ5remLIkS9Q_BQ4uN7jWrNGfkLMAcrnlQM,7610
+web_scraper/research_agent.py,sha256=1r9cuvRxv5FQzj3GVrKu7lfu5Z9ju33UEk6mW44SMQo,474
+web_scraper/scrapers.py,sha256=tbp16ZTw_3770yMyuYmc3TPm5k_eKUmIhRNGVm_5Zgc,15968
+web_scraper/stealth.py,sha256=mfdb2x5OIteHVPdon11b7yU8qiX6REAabWfQJ5oe4jc,4746
+web_scraper/yahoo_search.py,sha256=YpBHIOFUYCWJAqHE6hNKsIsiBiB_vDap-2e4wp7LzSQ,8164
+web_scraper/research/__init__.py,sha256=UpZ9S3JM7OSAK5HRp5cfjaYtBAry70hZR1ipCduUSms,359
+web_scraper/research/agent.py,sha256=YFl5M5p1uZMYzSFameU3s38yhhEzjxxv7exBsyrVkAg,96949
+web_scraper/research/citation_verifier.py,sha256=xLA3WOT4Kcmmw5JrifFFNhkRzd9TzhMLMfhLMv482e4,6349
+web_scraper/research/constants.py,sha256=8R2UTpW5bp7KEEg9ULnJfBneQpA6NtufwqgMBbUVbaY,13903
+web_scraper/research/llm_client.py,sha256=GFy1C9a2j8fTrKkZfqbIoIfPr5OGaSJbcoVK_Dcex1A,5151
+web_scraper/research/models.py,sha256=BcDlAwv0g6KogaO6lRgV5M14qaLvX9w7p-NkTQXhjPM,1185
+web_scraper/research/profile_collectors.py,sha256=kEiIvl7znmdnUy6wJIbSxhP2EGc8oWXcVd9_B0OC7JA,17406
+web_scraper/research/prompts.py,sha256=0kRRd83_XQAYH9APo8cuaibS2k4aCvbq2MJfSXNGb34,34855
+web_scraper/research/ranking.py,sha256=D9rnP4HooMIOMhcg1bPV49Nn3Otex-bL4B-DnGbVEXs,13220
+web_scraper/research/retry_utils.py,sha256=wwE5W8mT5HF4_Es-uBXXe_RC1mbq3lehLfpCOlpDhDU,2527
+web_scraper/research/text_utils.py,sha256=VTzUPXnyRSs7W_u762rN0bJlMUEBQFg6f686heyBaWY,8999
+web_scraper/research/url_utils.py,sha256=tJfvCr6SByhmwtt_70yFIPCso3HCvxcotKj2XIveBWo,4223
+web_scraper/web_tools/__init__.py,sha256=oKm85LqRlpR-Qf2CqtyV5Yic_ebZUQVUH1fCV690cf8,231
+web_scraper/web_tools/archive_fetcher.py,sha256=8_5VGKijga8ugTuelCHwFEJPRZhaJDOMuyT8RWj0mF8,7518
+web_scraper/web_tools/browser.py,sha256=mpyA8LDWhaZp0FbxMO-72xODdfOZqG11K9dsbM0k49k,12587
+web_scraper/web_tools/budget.py,sha256=dlWTkU5-sIRirNedMkAv8bFURQIBeAMMWISkwNRcgMI,8603
+web_scraper/web_tools/content.py,sha256=mzCxfo2RiJEgJmTTIO_hlwF2te445qVAfuRilGLUWc4,27268
+web_scraper/web_tools/crawl.py,sha256=xlrgN-ea829Hu4i-CwwXz3WYKHgWgV681yFugCIR1v0,25742
+web_scraper/web_tools/crawl_store.py,sha256=ZIabI-JJDk022NcB0wGP3H7CEK7GU8dOGoUHaOSgkrM,6700
+web_scraper/web_tools/discovery.py,sha256=pnKBrz3pBIeLNCioqta3g7dZ4ph04YLngWnGD4iH0tM,41838
+web_scraper/web_tools/errors.py,sha256=iRpiNT3oC5b3EQPKKK6p0wg0oeP141gILZxxzirZ4Uw,1627
+web_scraper/web_tools/fetcher.py,sha256=eq3rtW-YjyXBBc0ODp5BLyeB20InDE4d6xkYdsgj3KA,15857
+web_scraper/web_tools/fields.py,sha256=-aKmYI2QxQSm8LCF5oCN-nV_V6KEjET9pEbY5hhLUG0,18603
+web_scraper/web_tools/metrics.py,sha256=BHXrGU1T99RcIV0UtlPBy5lCiaNK4VlzlNeRrX4bs08,2523
+web_scraper/web_tools/pagecache.py,sha256=99kaYf1UCIIDuodU_ejvulfRx4LLjIp3SpQpKtfQMgk,3693
+web_scraper/web_tools/pdf.py,sha256=RMHmEvGpGb4AXYwf1BHxr0xTT3ZHi0jpBa4vKOFOmw4,5811
+web_scraper/web_tools/providers.py,sha256=SYNyz0t1yeicphiTYOdh8QAkPTVAYLtJR9c24cWxi6g,2490
+web_scraper/web_tools/relevance.py,sha256=u3g3YIjpMHL1xqA95ufAoTgiUo55iaFTcPFK2f9kRBE,3398
+web_scraper/web_tools/robots.py,sha256=yJOxRbuRRS-TY1gi8obFdF5Rtinwxn7aRrcqn0aubiU,2500
+web_scraper/web_tools/search.py,sha256=qhVrhfU4N2Cr4NcXcP6SunSqIdmjV-5p1wxKTWTTDb8,10554
+web_scraper/web_tools/service.py,sha256=7Bg52eiGVNMEbC1Azkb2cYZLq-hHCfzM6lb3D7kiOEc,81863
+web_scraper/web_tools/settings.py,sha256=guMDA0Kqst1UWycbH9ysy3vhTpugBf_qB1OtPhFi1oc,17335
+web_scraper/web_tools/sources.py,sha256=tyDCWOrr2ES2UZwzBGoQZjUJQyfW3LqfxG8jTZDI2Ss,9160
+web_scraper/web_tools/urls.py,sha256=zgOVmmgZsSHrfTZJ4X7QcyCSNiixA6r-CIUV1P06DE8,7337
+dotagent_web_research-1.2.0.dist-info/METADATA,sha256=SgndqzxZg7I-s9yzyICcfh3CK3O3plIoeoUllZTlR44,14717
+dotagent_web_research-1.2.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+dotagent_web_research-1.2.0.dist-info/entry_points.txt,sha256=xdDR2nX0ek-BRCuNK3hDbn8v4rQm95Hf9ZlptbdBuLc,100
+dotagent_web_research-1.2.0.dist-info/top_level.txt,sha256=0ynyaW_79oDijS6kQjJfk67X11tO-Qm6XfuhhuziCkk,12
+dotagent_web_research-1.2.0.dist-info/RECORD,,

dotagent_web_research-1.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

dotagent_web_research-1.2.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+web-research-mcp = web_scraper.mcp_server:main
+web-scraper = web_scraper.cli:main

dotagent_web_research-1.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Senol Dogan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dotagent_web_research-1.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+web_scraper

web_scraper/__init__.py

@@ -0,0 +1,4 @@
+"""Web Scraper - A powerful web scraping tool with CLI and API support."""
+
+__version__ = "1.2.0"
+__author__ = "Developer"

web_scraper/ai_agent.py

@@ -0,0 +1,203 @@
+"""AI Agent for natural language web scraping commands using Ollama."""
+
+import json
+import re
+from typing import Optional
+
+import httpx
+
+from web_scraper.config import config
+
+
+class OllamaAgent:
+    """AI agent that processes natural language commands for web scraping."""
+
+    def __init__(self, model: Optional[str] = None, host: Optional[str] = None):
+        """
+        Initialize the Ollama agent.
+
+        Args:
+            model: Ollama model name (default: gpt-oss:120b-cloud)
+            host: Ollama API host (default: http://localhost:11434)
+        """
+        self.model = model or config.default_research_model
+        self.host = host or config.ollama_host
+        self.api_url = f"{self.host}/api/generate"
+
+    async def process_command(self, command: str) -> dict:
+        """
+        Process a natural language command and return scraping parameters.
+
+        Args:
+            command: Natural language command (e.g., "scrape anthropic site and get me the last 10 news")
+
+        Returns:
+            Dictionary with URL, scraping mode, and filters
+        """
+        prompt = self._create_prompt(command)
+
+        try:
+            async with httpx.AsyncClient() as client:
+                response = await client.post(
+                    self.api_url,
+                    json={
+                        "model": self.model,
+                        "prompt": prompt,
+                        "stream": False,
+                    },
+                    timeout=config.research_source_selection_timeout_seconds,
+                )
+            response.raise_for_status()
+
+            result = response.json()
+            ai_response = result.get("response", "")
+
+            # Parse the AI response
+            return self._parse_response(ai_response, command)
+
+        except Exception:
+            # Fallback to manual parsing if AI fails
+            return self._fallback_parse(command)
+
+    def _create_prompt(self, command: str) -> str:
+        """Create a prompt for the AI model."""
+        return f"""You are a web scraping assistant. Extract the following information from the user's command and return ONLY a JSON object:
+
+User command: "{command}"
+
+Extract:
+1. url: The website URL to scrape (add https:// if missing)
+2. mode: "playwright" if JavaScript-heavy site (like React/Vue/Angular) or protected by Cloudflare, otherwise "basic"
+3. max_items: Number of items to extract (news, articles, etc.) if specified
+4. filters: Any specific content to look for (news, blog posts, products, etc.)
+
+Return ONLY this JSON format:
+{{
+    "url": "https://example.com",
+    "mode": "basic",
+    "max_items": 10,
+    "filters": "news"
+}}
+
+If URL cannot be determined, use empty string. If max_items not specified, use 0 (all)."""
+
+    def _parse_response(self, response: str, original_command: str) -> dict:
+        """Parse the AI response into a structured dictionary."""
+        try:
+            # Try to extract JSON from the response
+            json_match = re.search(r"\{[^}]*\}", response, re.DOTALL)
+            if json_match:
+                data = json.loads(json_match.group())
+                return {
+                    "url": data.get("url", ""),
+                    "mode": data.get("mode", "basic"),
+                    "max_items": data.get("max_items", 0),
+                    "filters": data.get("filters", ""),
+                    "original_command": original_command,
+                }
+        except json.JSONDecodeError:
+            pass
+
+        return self._fallback_parse(original_command)
+
+    def _fallback_parse(self, command: str) -> dict:
+        """Fallback parser when AI is not available."""
+        # Extract URL from command
+        url_pattern = r"(https?://[^\s]+|www\.[^\s]+|[a-zA-Z0-9-]+\.(com|org|net|io|dev|co)[^\s]*)"
+        url_match = re.search(url_pattern, command)
+
+        url = ""
+        if url_match:
+            url = url_match.group(1)
+            if not url.startswith("http"):
+                url = f"https://{url}"
+
+        # Extract number
+        number_match = re.search(r"(\d+)\s*(news|articles|posts|items)?", command, re.IGNORECASE)
+        max_items = int(number_match.group(1)) if number_match else 0
+
+        # Determine if playwright needed
+        mode = (
+            "playwright"
+            if any(
+                word in command.lower() for word in ["javascript", "react", "dynamic", "cloudflare"]
+            )
+            else "basic"
+        )
+
+        # Extract filters
+        filters = ""
+        if "news" in command.lower():
+            filters = "news"
+        elif "blog" in command.lower():
+            filters = "blog"
+        elif "article" in command.lower():
+            filters = "articles"
+
+        return {
+            "url": url,
+            "mode": mode,
+            "max_items": max_items,
+            "filters": filters,
+            "original_command": command,
+        }
+
+    async def summarize_content(self, content: str, filters: str = "", max_items: int = 0) -> str:
+        """
+        Summarize scraped content using AI.
+
+        Args:
+            content: The scraped content
+            filters: Type of content to focus on
+            max_items: Maximum number of items to extract
+
+        Returns:
+            Formatted summary
+        """
+        # Truncate content if too long
+        if len(content) > config.research_deep_content_limit_chars:
+            content = content[: config.research_deep_content_limit_chars] + "..."
+
+        prompt = f"""Analyze this website content and extract the key information{f" about {filters}" if filters else ""}.
+
+{f"Extract up to {max_items} items if applicable." if max_items > 0 else ""}
+
+Content:
+{content}
+
+Format your response as a clean, structured output with:
+1. Main topics/sections found
+2. Key information points
+3. Any lists or enumerated items
+
+Keep it concise and well-organized."""
+
+        try:
+            async with httpx.AsyncClient() as client:
+                response = await client.post(
+                    self.api_url,
+                    json={
+                        "model": self.model,
+                        "prompt": prompt,
+                        "stream": False,
+                    },
+                    timeout=config.research_synthesis_timeout_seconds,
+                )
+            response.raise_for_status()
+
+            result = response.json()
+            return result.get("response", "Could not generate summary.")
+
+        except Exception as e:
+            return f"Error generating summary: {str(e)}"
+
+    def is_available(self) -> bool:
+        """Check if Ollama is running. Stays sync — called before the event loop."""
+        try:
+            response = httpx.get(
+                f"{self.host}/api/tags",
+                timeout=min(5.0, config.research_planning_timeout_seconds),
+            )
+            return response.status_code == 200
+        except Exception:
+            return False