sitewise-crawler 0.1.0__tar.gz

sitewise_crawler-0.1.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.4
+Name: sitewise-crawler
+Version: 0.1.0
+Summary: A flexible and advanced web crawler for modern SPAs and traditional websites.
+Author-email: TarXemo <info@tarxemo.com>
+Project-URL: Homepage, https://github.com/tarxemo/sitewise-crawler
+Project-URL: Bug Tracker, https://github.com/tarxemo/sitewise-crawler/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: playwright
+Requires-Dist: trafilatura
+Requires-Dist: lxml
+Requires-Dist: pydantic
+Requires-Dist: aiohttp
+Requires-Dist: tenacity
+Requires-Dist: pypdf
+Requires-Dist: python-docx
+Requires-Dist: groq
+
+# Sitewise Crawler 🕷️
+
+An advanced, flexible, and production-ready web crawler for modern websites. Automatically detects SPAs (Single Page Applications) and switches between fast `requests` fetching and full JavaScript rendering with `Playwright`.
+
+## ✨ Features
+
+- 🚀 **Hybrid Rendering**: Automatically detects React, Vue, Angular, and Next.js to switch rendering engines on the fly.
+- 🧠 **Smart Extraction**: Built-in main content extraction that removes headers, footers, and sidebars.
+- 🔗 **SPA Link Discovery**: Discovers links even in complex client-side routers.
+- 🛠️ **Fully Configurable**: Control depth, concurrency, rate limits, and custom wait selectors.
+- 📝 **Pydantic Models**: Type-safe configuration and results.
+
+## 📦 Installation
+
+```bash
+pip install sitewise-crawler
+playwright install chromium
+```
+
+## 🚀 Quick Start
+
+```python
+import asyncio
+from sitewise_crawler import SPACrawler, CrawlerConfig
+
+async def main():
+    # 1. Configure the crawler
+    config = CrawlerConfig(
+        start_url="https://example.com",
+        max_depth=2,
+        max_pages=10,
+        use_playwright=True,
+        headless=True
+    )
+
+    # 2. Initialize and run
+    crawler = SPACrawler(config)
+    
+    # Optional: Add a callback for each page crawled
+    crawler.on_page_crawled = lambda page: print(f"Crawled: {page.url} | Title: {page.title}")
+
+    result = await crawler.crawl()
+
+    # 3. Process results
+    if result.success:
+        print(f"\n✅ Crawl complete! Found {result.total_pages} pages.")
+        for page in result.pages_all:
+            print(f"- {page.url} ({len(page.content)} chars)")
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## ⚙️ Configuration Options
+
+The `CrawlerConfig` class supports the following parameters:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `start_url` | `str` | *Required* | The entry point for the crawler. |
+| `max_depth` | `int` | `3` | Maximum crawl depth from the start URL. |
+| `max_pages` | `int` | `100` | Stop crawling after this many pages. |
+| `use_playwright` | `bool` | `True` | Enable JavaScript rendering for SPAs. |
+| `headless` | `bool` | `True` | Run browser in headless mode. |
+| `rate_limit_delay` | `float` | `1.0` | Seconds to wait between requests. |
+| `wait_for_selector`| `str` | `None` | CSS selector to wait for before extracting SPA content. |
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+# sitewise_crawler

sitewise_crawler-0.1.0/README.md

@@ -0,0 +1,75 @@
+# Sitewise Crawler 🕷️
+
+An advanced, flexible, and production-ready web crawler for modern websites. Automatically detects SPAs (Single Page Applications) and switches between fast `requests` fetching and full JavaScript rendering with `Playwright`.
+
+## ✨ Features
+
+- 🚀 **Hybrid Rendering**: Automatically detects React, Vue, Angular, and Next.js to switch rendering engines on the fly.
+- 🧠 **Smart Extraction**: Built-in main content extraction that removes headers, footers, and sidebars.
+- 🔗 **SPA Link Discovery**: Discovers links even in complex client-side routers.
+- 🛠️ **Fully Configurable**: Control depth, concurrency, rate limits, and custom wait selectors.
+- 📝 **Pydantic Models**: Type-safe configuration and results.
+
+## 📦 Installation
+
+```bash
+pip install sitewise-crawler
+playwright install chromium
+```
+
+## 🚀 Quick Start
+
+```python
+import asyncio
+from sitewise_crawler import SPACrawler, CrawlerConfig
+
+async def main():
+    # 1. Configure the crawler
+    config = CrawlerConfig(
+        start_url="https://example.com",
+        max_depth=2,
+        max_pages=10,
+        use_playwright=True,
+        headless=True
+    )
+
+    # 2. Initialize and run
+    crawler = SPACrawler(config)
+    
+    # Optional: Add a callback for each page crawled
+    crawler.on_page_crawled = lambda page: print(f"Crawled: {page.url} | Title: {page.title}")
+
+    result = await crawler.crawl()
+
+    # 3. Process results
+    if result.success:
+        print(f"\n✅ Crawl complete! Found {result.total_pages} pages.")
+        for page in result.pages_all:
+            print(f"- {page.url} ({len(page.content)} chars)")
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## ⚙️ Configuration Options
+
+The `CrawlerConfig` class supports the following parameters:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `start_url` | `str` | *Required* | The entry point for the crawler. |
+| `max_depth` | `int` | `3` | Maximum crawl depth from the start URL. |
+| `max_pages` | `int` | `100` | Stop crawling after this many pages. |
+| `use_playwright` | `bool` | `True` | Enable JavaScript rendering for SPAs. |
+| `headless` | `bool` | `True` | Run browser in headless mode. |
+| `rate_limit_delay` | `float` | `1.0` | Seconds to wait between requests. |
+| `wait_for_selector`| `str` | `None` | CSS selector to wait for before extracting SPA content. |
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+# sitewise_crawler

sitewise_crawler-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sitewise-crawler"
+version = "0.1.0"
+authors = [
+  { name="TarXemo", email="info@tarxemo.com" },
+]
+description = "A flexible and advanced web crawler for modern SPAs and traditional websites."
+readme = "README.md"
+requires-python = ">=3.10"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+]
+dependencies = [
+    "requests",
+    "beautifulsoup4",
+    "playwright",
+    "trafilatura",
+    "lxml",
+    "pydantic",
+    "aiohttp",
+    "tenacity",
+    "pypdf",
+    "python-docx",
+    "groq",
+]
+
+[project.urls]
+"Homepage" = "https://github.com/tarxemo/sitewise-crawler"
+"Bug Tracker" = "https://github.com/tarxemo/sitewise-crawler/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]

sitewise_crawler-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sitewise_crawler-0.1.0/src/sitewise_crawler/__init__.py

@@ -0,0 +1,21 @@
+from .crawler import SPACrawler
+from .models import CrawlerConfig, PageData, CrawlResult, UserInsight, CategoryScore
+from .fetchers import RequestsFetcher, PlaywrightFetcher
+from .extractors import LinkExtractor, ContentExtractor, SPADetector
+from .analyzer import InsightEngine
+
+__version__ = "0.1.0"
+__all__ = [
+    "SPACrawler",
+    "CrawlerConfig",
+    "PageData",
+    "CrawlResult",
+    "UserInsight",
+    "CategoryScore",
+    "RequestsFetcher",
+    "PlaywrightFetcher",
+    "LinkExtractor",
+    "ContentExtractor",
+    "SPADetector",
+    "InsightEngine",
+]

sitewise_crawler-0.1.0/src/sitewise_crawler/analyzer.py

@@ -0,0 +1,162 @@
+import logging
+import json
+import asyncio
+from typing import List, Optional, Dict, Any
+from datetime import datetime
+from groq import Groq
+from .crawler import SPACrawler
+from .models import CrawlerConfig, UserInsight, CategoryScore, PageData
+
+logger = logging.getLogger(__name__)
+
+class InsightEngine:
+    """
+    Advanced engine for analyzing user behavior based on visited content.
+    Uses AI (Groq) to provide deep insights.
+    """
+    def __init__(self, api_key: str, model: str = "llama-3.1-70b-versatile"):
+        self.client = Groq(api_key=api_key)
+        self.model = model
+
+    async def analyze_user_behavior(
+        self, 
+        user_id: str, 
+        urls: List[str], 
+        crawler_config: Optional[CrawlerConfig] = None
+    ) -> UserInsight:
+        """
+        Scrapes a list of URLs and performs intensive AI analysis on the content.
+        """
+        logger.info(f"Starting behavioral analysis for user {user_id} with {len(urls)} URLs")
+        
+        # 1. Scrape all URLs
+        if not crawler_config:
+            crawler_config = CrawlerConfig(start_url=urls[0], max_pages=len(urls))
+            
+        crawler = SPACrawler(crawler_config)
+        
+        tasks = [crawler.scrape_page(url) for url in urls]
+        pages: List[PageData] = await asyncio.gather(*tasks)
+        pages = [p for p in pages if p is not None]
+        
+        if not pages:
+            raise ValueError("No content could be extracted from the provided URLs.")
+
+        # 2. Consolidate content for analysis
+        # We take chunks of content from each page to stay within LLM context limits
+        consolidated_content = ""
+        for page in pages:
+            preview = page.content[:1500]  # Take first 1500 chars from each page
+            consolidated_content += f"\n--- URL: {page.url} ---\nTitle: {page.title}\nContent: {preview}\n"
+
+        # 3. Perform AI Analysis
+        analysis_result = await self._call_ai_analyzer(consolidated_content)
+        
+        # 4. Construct UserInsight model
+        from .models import Entity
+        return UserInsight(
+            user_id=user_id,
+            total_urls_analyzed=len(pages),
+            # Core
+            primary_interests=analysis_result.get("primary_interests", []),
+            top_categories=[
+                CategoryScore(category=c["name"], score=c["score"]) 
+                for c in analysis_result.get("top_categories", [])
+            ],
+            content_languages=analysis_result.get("content_languages", []),
+            content_complexity_score=analysis_result.get("content_complexity_score", 0.0),
+            technical_proficiency_level=analysis_result.get("technical_proficiency_level", "Unknown"),
+            
+            # Sentiment
+            overall_sentiment=analysis_result.get("overall_sentiment", "Neutral"),
+            average_sentiment_score=analysis_result.get("average_sentiment_score", 0.0),
+            stress_or_anxiety_indicators=analysis_result.get("stress_or_anxiety_indicators", 0.0),
+            
+            # Behavior
+            estimated_intent=analysis_result.get("estimated_intent", "Unknown"),
+            productivity_rating=analysis_result.get("productivity_rating", 0.0),
+            time_wasting_probability=analysis_result.get("time_wasting_probability", 0.0),
+            focus_score=analysis_result.get("focus_score", 0.0),
+            
+            # Academic/Career
+            educational_alignment=analysis_result.get("educational_alignment", 0.0),
+            academic_relevance_score=analysis_result.get("academic_relevance_score", 0.0),
+            career_development_focus=analysis_result.get("career_development_focus", 0.0),
+            
+            # Commercial
+            shopping_intent_score=analysis_result.get("shopping_intent_score", 0.0),
+            likely_in_market_for=analysis_result.get("likely_in_market_for", []),
+            
+            # Entities
+            top_entities=[
+                Entity(**e) for e in analysis_result.get("top_entities", [])
+            ],
+            
+            # Risk
+            risk_assessment_summary=analysis_result.get("risk_assessment_summary"),
+            nsfw_or_inappropriate_probability=analysis_result.get("nsfw_or_inappropriate_probability", 0.0),
+            
+            # Summary
+            behavioral_summary=analysis_result.get("behavioral_summary", "No summary available."),
+            actionable_recommendation=analysis_result.get("actionable_recommendation"),
+            raw_ai_response=analysis_result
+        )
+
+    async def _call_ai_analyzer(self, content: str) -> Dict[str, Any]:
+        """Calls Groq to analyze the consolidated content."""
+        prompt = f"""
+        You are a highly advanced Behavioral Profiling AI working for an institutional analytics platform.
+        Analyze the following browsing history content and provide a massive, highly detailed behavioral profile of the user.
+        Return ONLY a JSON object exactly matching this structure (fill in the values based on your analysis):
+        {{
+            "primary_interests": ["topic1", "topic2", "topic3"],
+            "top_categories": [{{"name": "CategoryName", "score": 0.9}}],
+            "content_languages": ["English"],
+            "content_complexity_score": 0.8,
+            "technical_proficiency_level": "Intermediate",
+            "overall_sentiment": "Positive",
+            "average_sentiment_score": 0.5,
+            "stress_or_anxiety_indicators": 0.1,
+            "estimated_intent": "Research",
+            "productivity_rating": 0.85,
+            "time_wasting_probability": 0.15,
+            "focus_score": 0.9,
+            "educational_alignment": 0.9,
+            "academic_relevance_score": 0.8,
+            "career_development_focus": 0.7,
+            "shopping_intent_score": 0.0,
+            "likely_in_market_for": [],
+            "top_entities": [{{"name": "OpenAI", "type": "Organization", "frequency": 5}}],
+            "risk_assessment_summary": "Low risk. Professional content.",
+            "nsfw_or_inappropriate_probability": 0.0,
+            "behavioral_summary": "A comprehensive paragraph summarizing habits.",
+            "actionable_recommendation": "Suggest resource X based on interest Y."
+        }}
+
+        Content to analyze:
+        {content}
+        """
+
+        try:
+            chat_completion = self.client.chat.completions.create(
+                messages=[
+                    {
+                        "role": "system",
+                        "content": "You are a specialized User Behavior Analyst. You extract deep, multi-dimensional insights from web content history. Output strictly valid JSON."
+                    },
+                    {
+                        "role": "user",
+                        "content": prompt,
+                    }
+                ],
+                model=self.model,
+                response_format={"type": "json_object"}
+            )
+            
+            return json.loads(chat_completion.choices[0].message.content)
+        except Exception as e:
+            logger.error(f"AI Analysis failed: {e}")
+            return {
+                "behavioral_summary": f"Failed to perform AI analysis: {str(e)}",
+                "overall_sentiment": "Unknown"
+            }

sitewise_crawler-0.1.0/src/sitewise_crawler/crawler.py

@@ -0,0 +1,132 @@
+import asyncio
+import logging
+import time
+from collections import deque
+from typing import Set, List, Optional, Callable
+from .models import CrawlerConfig, PageData, CrawlResult
+from .fetchers import RequestsFetcher, PlaywrightFetcher
+from .extractors import LinkExtractor, ContentExtractor, SPADetector
+
+logger = logging.getLogger(__name__)
+
+class SPACrawler:
+    """
+    Advanced Crawler Engine that automatically handles SPAs and traditional websites.
+    """
+    def __init__(self, config: CrawlerConfig):
+        self.config = config
+        self.visited: Set[str] = set()
+        self.queue = deque([(config.start_url, 0)])
+        self.results: List[PageData] = []
+        self.failed_urls: List[str] = []
+        
+        self.requests_fetcher = RequestsFetcher()
+        self.playwright_fetcher = PlaywrightFetcher()
+        
+        # User-defined hook for page processing
+        self.on_page_crawled: Optional[Callable[[PageData], None]] = None
+
+    def _should_crawl(self, url: str, depth: int) -> bool:
+        if url in self.visited:
+            return False
+        if depth > self.config.max_depth:
+            return False
+        if len(self.results) >= self.config.max_pages:
+            return False
+        if self.config.allowed_domains and not LinkExtractor.is_same_domain(url, self.config.start_url):
+            return False
+        return True
+
+    async def crawl(self) -> CrawlResult:
+        """Starts the full BFS crawling process based on config."""
+        start_time = time.time()
+        logger.info(f"Starting crawl for {self.config.start_url}")
+
+        while self.queue and len(self.results) < self.config.max_pages:
+            url, depth = self.queue.popleft()
+            
+            if not self._should_crawl(url, depth):
+                continue
+            
+            self.visited.add(url)
+            
+            # Step 1: Extract the page
+            page_data = await self.scrape_page(url, depth=depth)
+            
+            if not page_data:
+                self.failed_urls.append(url)
+                continue
+            
+            self.results.append(page_data)
+            
+            # Callback
+            if self.on_page_crawled:
+                self.on_page_crawled(page_data)
+            
+            # Step 2: Add new links to queue (only if we haven't hit max depth)
+            if depth < self.config.max_depth:
+                for link in page_data.links:
+                    if LinkExtractor.is_same_domain(link, self.config.start_url):
+                        self.queue.append((link, depth + 1))
+            
+            # Rate limiting
+            await asyncio.sleep(self.config.rate_limit_delay)
+
+        # Cleanup
+        await self.playwright_fetcher.close()
+        
+        duration = time.time() - start_time
+        return CrawlResult(
+            success=len(self.results) > 0,
+            pages_all=self.results,
+            failed_urls=self.failed_urls,
+            duration_seconds=duration,
+            total_pages=len(self.results)
+        )
+
+    async def scrape_page(self, url: str, depth: int = 0) -> Optional[PageData]:
+        """
+        Directly extracts data from a single URL. 
+        Supports HTML, SPAs, PDFs, and Word Documents.
+        """
+        logger.info(f"Scraping page: {url}")
+        
+        # Step 1: Fetch content
+        # We start with Requests for efficiency and document handling
+        content, status, title, content_type = await self.requests_fetcher.fetch(url, self.config)
+        
+        if not content:
+            return None
+            
+        is_spa = False
+        is_binary = False
+        
+        # Step 2: Handle based on content type
+        if 'html' in content_type:
+            # Check for SPA
+            if SPADetector.is_spa(content) and self.config.use_playwright:
+                logger.info(f"SPA detected for {url}, switching to Playwright")
+                is_spa = True
+                content, status, title, content_type = await self.playwright_fetcher.fetch(url, self.config)
+            
+            # Extract HTML content
+            text_content = ContentExtractor.clean_text(content)
+            links = LinkExtractor.extract_links(content, url)
+        else:
+            # Handle binary documents
+            logger.info(f"Binary document detected ({content_type}) for {url}")
+            is_binary = True
+            text_content = ContentExtractor.extract_from_binary(content, content_type)
+            links = [] # Binary files usually don't have crawlable links for our BFS
+            
+        return PageData(
+            url=url,
+            title=title or url.split('/')[-1],
+            content=text_content,
+            html=content if (isinstance(content, str) and self.config.max_pages < 10) else None,
+            depth=depth,
+            status_code=status,
+            is_spa=is_spa,
+            links=links,
+            metadata={'content_type': content_type, 'is_binary': is_binary}
+        )

sitewise_crawler-0.1.0/src/sitewise_crawler/extractors.py

@@ -0,0 +1,113 @@
+import re
+from urllib.parse import urljoin, urlparse, urlunparse
+from typing import List, Set
+from bs4 import BeautifulSoup
+import trafilatura
+
+class LinkExtractor:
+    @staticmethod
+    def normalize_url(url: str, base_url: str) -> str:
+        """Normalize URL and remove fragments."""
+        parsed = urlparse(url)
+        if not parsed.netloc:
+            url = urljoin(base_url, url)
+            parsed = urlparse(url)
+        
+        # Remove fragments and normalize
+        parsed = parsed._replace(fragment='')
+        normalized = urlunparse(parsed)
+        return normalized.rstrip('/')
+
+    @staticmethod
+    def is_same_domain(url: str, base_url: str) -> bool:
+        return urlparse(url).netloc == urlparse(base_url).netloc
+
+    @staticmethod
+    def extract_links(html: str, base_url: str) -> List[str]:
+        soup = BeautifulSoup(html, 'lxml')
+        links = set()
+        for a in soup.find_all('a', href=True):
+            links.add(a['href'])
+        
+        normalized = []
+        for link in links:
+            try:
+                norm = LinkExtractor.normalize_url(link, base_url)
+                normalized.append(norm)
+            except:
+                continue
+        return list(set(normalized))
+
+class ContentExtractor:
+    @staticmethod
+    def clean_text(html: str) -> str:
+        """Extract main content text, removing boilerplates."""
+        # Use trafilatura for high-quality extraction
+        content = trafilatura.extract(html, include_comments=False, include_tables=True, no_fallback=False)
+        if not content:
+            # Fallback to BeautifulSoup if trafilatura fails
+            soup = BeautifulSoup(html, 'lxml')
+            for script in soup(["script", "style", "nav", "footer", "header"]):
+                script.decompose()
+            content = soup.get_text(separator=' ')
+        
+        # Final cleanup
+        content = re.sub(r'\s+', ' ', content).strip()
+        return content
+
+    @staticmethod
+    def extract_from_binary(content: bytes, content_type: str) -> str:
+        """Extract text from non-HTML binary files (PDF, Docx)."""
+        import io
+        
+        # Handle PDF
+        if 'pdf' in content_type:
+            try:
+                from pypdf import PdfReader
+                reader = PdfReader(io.BytesIO(content))
+                text = ""
+                for page in reader.pages:
+                    text += page.extract_text() + "\n"
+                return text.strip()
+            except Exception as e:
+                return f"Error extracting PDF: {e}"
+        
+        # Handle Word Documents
+        elif 'word' in content_type or 'officedocument.wordprocessingml' in content_type:
+            try:
+                from docx import Document
+                doc = Document(io.BytesIO(content))
+                text = "\n".join([para.text for para in doc.paragraphs])
+                return text.strip()
+            except Exception as e:
+                return f"Error extracting Word doc: {e}"
+                
+        return "Unsupported binary format"
+
+class SPADetector:
+    FRAMEWORK_PATTERNS = {
+        'react': [r'react-root', r'_reactRootContainer', r'data-reactid', r'data-reactroot'],
+        'vue': [r'v-bind:', r'v-on:', r'__vue__', r'data-v-'],
+        'angular': [r'ng-version', r'ng-app', r'ng-controller', r'ng-repeat'],
+        'nextjs': [r'__NEXT_DATA__', r'_next/static'],
+        'nuxt': [r'__NUXT__'],
+    }
+
+    @staticmethod
+    def is_spa(html: str) -> bool:
+        """Detect if the page is likely a Single Page Application."""
+        for framework, patterns in SPADetector.FRAMEWORK_PATTERNS.items():
+            for pattern in patterns:
+                if re.search(pattern, html, re.IGNORECASE):
+                    return True
+        
+        # Check for empty body with lots of scripts
+        body_match = re.search(r'<body[^>]*>(.*?)</body>', html, re.DOTALL | re.IGNORECASE)
+        if body_match:
+            body_content = body_match.group(1)
+            # If body is mostly empty but has many script tags
+            clean_body = re.sub(r'<script[^>]*>.*?</script>', '', body_content, flags=re.DOTALL | re.IGNORECASE)
+            if len(clean_body.strip()) < 200 and '<script' in body_content:
+                return True
+                
+        return False

sitewise_crawler-0.1.0/src/sitewise_crawler/fetchers.py

@@ -0,0 +1,98 @@
+import logging
+from abc import ABC, abstractmethod
+from typing import Tuple, Optional
+import requests
+from bs4 import BeautifulSoup
+from .models import CrawlerConfig
+
+logger = logging.getLogger(__name__)
+
+class BaseFetcher(ABC):
+    @abstractmethod
+    async def fetch(self, url: str, config: CrawlerConfig) -> Tuple[Optional[Any], int, Optional[str], Optional[str]]:
+        """
+        Fetch content from a URL.
+        Returns: (content, status_code, title, content_type)
+        """
+        pass
+
+class RequestsFetcher(BaseFetcher):
+    async def fetch(self, url: str, config: CrawlerConfig) -> Tuple[Optional[Any], int, Optional[str], Optional[str]]:
+        try:
+            headers = {"User-Agent": config.user_agent}
+            response = requests.get(url, headers=headers, timeout=config.timeout_ms/1000, stream=True)
+            content_type = response.headers.get('Content-Type', '').split(';')[0].lower()
+            
+            if response.status_code == 200:
+                # Handle text-based content
+                if 'html' in content_type or 'text' in content_type:
+                    soup = BeautifulSoup(response.text, 'lxml')
+                    title = soup.title.string if soup.title else ""
+                    return response.text, response.status_code, title, content_type
+                
+                # Handle binary content (PDF, Docx, etc)
+                return response.content, response.status_code, url.split('/')[-1], content_type
+                
+            return None, response.status_code, None, content_type
+        except Exception as e:
+            logger.error(f"RequestsFetcher error for {url}: {e}")
+            return None, 0, None, None
+
+class PlaywrightFetcher(BaseFetcher):
+    def __init__(self):
+        self.playwright = None
+        self.browser = None
+
+    async def _ensure_browser(self, config: CrawlerConfig):
+        if not self.browser:
+            from playwright.async_api import async_playwright
+            self.playwright = await async_playwright().start()
+            self.browser = await self.playwright.chromium.launch(headless=config.headless)
+
+    async def fetch(self, url: str, config: CrawlerConfig) -> Tuple[Optional[Any], int, Optional[str], Optional[str]]:
+        try:
+            await self._ensure_browser(config)
+            context = await self.browser.new_context(user_agent=config.user_agent)
+            page = await context.new_page()
+            
+            # Playwright is mainly for HTML/SPA, but it can handle navigation to documents
+            response = await page.goto(url, wait_until="networkidle", timeout=config.timeout_ms)
+            
+            if not response:
+                return None, 0, None, None
+            
+            content_type = response.headers.get('content-type', '').split(';')[0].lower()
+            
+            # If it's a binary file, Playwright might trigger a download or show a PDF viewer
+            # For simplicity, we fallback to requests for binary files in the crawler engine,
+            # but here we return what we can.
+            if 'html' not in content_type:
+                # Use raw response body for non-html
+                content = await response.body()
+                await context.close()
+                return content, response.status, url.split('/')[-1], content_type
+
+            import asyncio
+            await asyncio.sleep(config.js_wait_time / 1000)
+            
+            if config.wait_for_selector:
+                try:
+                    await page.wait_for_selector(config.wait_for_selector, timeout=5000)
+                except:
+                    pass
+
+            html = await page.content()
+            status = response.status
+            title = await page.title()
+            
+            await context.close()
+            return html, status, title, content_type
+        except Exception as e:
+            logger.error(f"PlaywrightFetcher error for {url}: {e}")
+            return None, 0, None, None
+
+    async def close(self):
+        if self.browser:
+            await self.browser.close()
+        if self.playwright:
+            await self.playwright.stop()

sitewise_crawler-0.1.0/src/sitewise_crawler/models.py

@@ -0,0 +1,106 @@
+from typing import List, Optional, Dict, Any
+from pydantic import BaseModel, Field, HttpUrl
+from datetime import datetime
+
+class CrawlerConfig(BaseModel):
+    """Configuration for the Crawler engine."""
+    start_url: str
+    max_depth: int = Field(default=3, ge=0)
+    max_pages: int = Field(default=100, ge=1)
+    concurrency: int = Field(default=3, ge=1)
+    timeout_ms: int = Field(default=30000, ge=1000)
+    rate_limit_delay: float = Field(default=1.0, ge=0.0)
+    
+    # SPA Settings
+    use_playwright: bool = True
+    headless: bool = True
+    wait_for_selector: Optional[str] = None
+    js_wait_time: int = 2000
+    
+    # Filtering
+    allowed_domains: List[str] = []
+    ignore_patterns: List[str] = [
+        r"\.(css|js|png|jpg|jpeg|gif|svg|ico|woff|woff2|ttf|eot)$",
+        r"^javascript:",
+        r"^mailto:",
+        r"^tel:",
+    ]
+    
+    # Custom Headers
+    user_agent: str = "SitewiseCrawler/0.1.0 (+https://github.com/tarxemo/sitewise-crawler)"
+
+class PageData(BaseModel):
+    """Data extracted from a single page."""
+    url: str
+    title: Optional[str] = None
+    content: str
+    html: Optional[str] = None
+    depth: int
+    status_code: int
+    is_spa: bool = False
+    metadata: Dict[str, Any] = {}
+    links: List[str] = []
+    timestamp: datetime = Field(default_factory=datetime.now)
+
+class CrawlResult(BaseModel):
+    """Summary result of a crawl session."""
+    success: bool
+    pages_all: List[PageData]
+    failed_urls: List[str]
+    duration_seconds: float
+    total_pages: int
+
+class CategoryScore(BaseModel):
+    category: str
+    score: float  # 0.0 to 1.0
+
+class Entity(BaseModel):
+    name: str
+    type: str  # Person, Organization, Location, Product, Concept
+    frequency: int
+
+class UserInsight(BaseModel):
+    """Advanced behavioral analysis based on content consumed. Highly detailed for institutional use."""
+    user_id: str
+    analyzed_at: datetime = Field(default_factory=datetime.now)
+    total_urls_analyzed: int
+    
+    # --- Core Content Analysis ---
+    primary_interests: List[str] = []
+    top_categories: List[CategoryScore] = []
+    content_languages: List[str] = []
+    content_complexity_score: float = 0.0  # 0 to 1 (Simple vs Academic/Professional)
+    technical_proficiency_level: str = "Unknown"  # Beginner, Intermediate, Advanced, Expert
+    
+    # --- Sentiment & Psychological Indicators ---
+    overall_sentiment: str  # Positive, Neutral, Negative, Mixed
+    average_sentiment_score: float  # -1.0 to 1.0
+    stress_or_anxiety_indicators: float = 0.0  # 0 to 1 (Useful for student/employee wellbeing)
+    
+    # --- Behavioral & Productivity Insights ---
+    estimated_intent: str  # Research, Information, Entertainment, Transactional, Social
+    productivity_rating: float = 0.0  # 0 to 1
+    time_wasting_probability: float = 0.0  # 0 to 1 (High if consuming excessive social media/entertainment)
+    focus_score: float = 0.0 # 0 to 1 (Is browsing highly concentrated or scattered?)
+    
+    # --- Academic & Career Alignment ---
+    educational_alignment: float = 0.0  # 0 to 1
+    academic_relevance_score: float = 0.0 # 0 to 1
+    career_development_focus: float = 0.0 # 0 to 1
+    
+    # --- Commercial Intent ---
+    shopping_intent_score: float = 0.0 # 0 to 1
+    likely_in_market_for: List[str] = []
+    
+    # --- Extracted Entities ---
+    top_entities: List[Entity] = []
+    
+    # --- Institutional Risk Assessment ---
+    risk_assessment_summary: Optional[str] = None
+    nsfw_or_inappropriate_probability: float = 0.0 # 0 to 1
+    
+    # --- AI Synthesized Summaries ---
+    behavioral_summary: str
+    actionable_recommendation: Optional[str] = None # E.g., "User might need study resources for Python"
+    
+    raw_ai_response: Optional[Dict[str, Any]] = None

sitewise_crawler-0.1.0/src/sitewise_crawler.egg-info/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.4
+Name: sitewise-crawler
+Version: 0.1.0
+Summary: A flexible and advanced web crawler for modern SPAs and traditional websites.
+Author-email: TarXemo <info@tarxemo.com>
+Project-URL: Homepage, https://github.com/tarxemo/sitewise-crawler
+Project-URL: Bug Tracker, https://github.com/tarxemo/sitewise-crawler/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: playwright
+Requires-Dist: trafilatura
+Requires-Dist: lxml
+Requires-Dist: pydantic
+Requires-Dist: aiohttp
+Requires-Dist: tenacity
+Requires-Dist: pypdf
+Requires-Dist: python-docx
+Requires-Dist: groq
+
+# Sitewise Crawler 🕷️
+
+An advanced, flexible, and production-ready web crawler for modern websites. Automatically detects SPAs (Single Page Applications) and switches between fast `requests` fetching and full JavaScript rendering with `Playwright`.
+
+## ✨ Features
+
+- 🚀 **Hybrid Rendering**: Automatically detects React, Vue, Angular, and Next.js to switch rendering engines on the fly.
+- 🧠 **Smart Extraction**: Built-in main content extraction that removes headers, footers, and sidebars.
+- 🔗 **SPA Link Discovery**: Discovers links even in complex client-side routers.
+- 🛠️ **Fully Configurable**: Control depth, concurrency, rate limits, and custom wait selectors.
+- 📝 **Pydantic Models**: Type-safe configuration and results.
+
+## 📦 Installation
+
+```bash
+pip install sitewise-crawler
+playwright install chromium
+```
+
+## 🚀 Quick Start
+
+```python
+import asyncio
+from sitewise_crawler import SPACrawler, CrawlerConfig
+
+async def main():
+    # 1. Configure the crawler
+    config = CrawlerConfig(
+        start_url="https://example.com",
+        max_depth=2,
+        max_pages=10,
+        use_playwright=True,
+        headless=True
+    )
+
+    # 2. Initialize and run
+    crawler = SPACrawler(config)
+    
+    # Optional: Add a callback for each page crawled
+    crawler.on_page_crawled = lambda page: print(f"Crawled: {page.url} | Title: {page.title}")
+
+    result = await crawler.crawl()
+
+    # 3. Process results
+    if result.success:
+        print(f"\n✅ Crawl complete! Found {result.total_pages} pages.")
+        for page in result.pages_all:
+            print(f"- {page.url} ({len(page.content)} chars)")
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## ⚙️ Configuration Options
+
+The `CrawlerConfig` class supports the following parameters:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `start_url` | `str` | *Required* | The entry point for the crawler. |
+| `max_depth` | `int` | `3` | Maximum crawl depth from the start URL. |
+| `max_pages` | `int` | `100` | Stop crawling after this many pages. |
+| `use_playwright` | `bool` | `True` | Enable JavaScript rendering for SPAs. |
+| `headless` | `bool` | `True` | Run browser in headless mode. |
+| `rate_limit_delay` | `float` | `1.0` | Seconds to wait between requests. |
+| `wait_for_selector`| `str` | `None` | CSS selector to wait for before extracting SPA content. |
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+# sitewise_crawler

sitewise_crawler-0.1.0/src/sitewise_crawler.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+README.md
+pyproject.toml
+src/sitewise_crawler/__init__.py
+src/sitewise_crawler/analyzer.py
+src/sitewise_crawler/crawler.py
+src/sitewise_crawler/extractors.py
+src/sitewise_crawler/fetchers.py
+src/sitewise_crawler/models.py
+src/sitewise_crawler.egg-info/PKG-INFO
+src/sitewise_crawler.egg-info/SOURCES.txt
+src/sitewise_crawler.egg-info/dependency_links.txt
+src/sitewise_crawler.egg-info/requires.txt
+src/sitewise_crawler.egg-info/top_level.txt
+tests/test_analyzer.py
+tests/test_local.py

sitewise_crawler-0.1.0/src/sitewise_crawler.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sitewise_crawler-0.1.0/src/sitewise_crawler.egg-info/requires.txt

@@ -0,0 +1,11 @@
+requests
+beautifulsoup4
+playwright
+trafilatura
+lxml
+pydantic
+aiohttp
+tenacity
+pypdf
+python-docx
+groq

sitewise_crawler-0.1.0/src/sitewise_crawler.egg-info/top_level.txt

@@ -0,0 +1 @@
+sitewise_crawler

sitewise_crawler-0.1.0/tests/test_analyzer.py

@@ -0,0 +1,48 @@
+import asyncio
+import sys
+import os
+import json
+
+# Add src to path for local testing
+sys.path.append(os.path.join(os.getcwd(), 'src'))
+
+from sitewise_crawler import InsightEngine, CrawlerConfig
+
+async def test_analyzer():
+    # Replace with your actual Groq API Key
+    api_key = os.getenv("GROQ_API_KEY", "your-groq-api-key")
+    
+    if api_key == "your-groq-api-key":
+        print("⚠️ Please set your GROQ_API_KEY environment variable to test the AI analyzer.")
+        return
+
+    print("🚀 Starting InsightEngine test...")
+    engine = InsightEngine(api_key=api_key)
+    
+    # We use some fast, public URLs to simulate a user's browsing history
+    urls_to_analyze = [
+        "https://en.wikipedia.org/wiki/Machine_learning",
+        "https://www.python.org/"
+    ]
+    
+    try:
+        # We can pass a custom CrawlerConfig to speed up the test (e.g., disable playwright)
+        config = CrawlerConfig(start_url=urls_to_analyze[0], max_pages=len(urls_to_analyze), use_playwright=False)
+        
+        insight = await engine.analyze_user_behavior(
+            user_id="test_user_001",
+            urls=urls_to_analyze,
+            crawler_config=config
+        )
+        
+        print("\n✅ Analysis Complete! Here is the data ready for your database:\n")
+        
+        # Convert to dictionary and print as formatted JSON
+        insight_dict = insight.model_dump(mode='json')
+        print(json.dumps(insight_dict, indent=2))
+        
+    except Exception as e:
+        print(f"❌ Error during analysis: {e}")
+
+if __name__ == "__main__":
+    asyncio.run(test_analyzer())

sitewise_crawler-0.1.0/tests/test_local.py

@@ -0,0 +1,38 @@
+import asyncio
+import sys
+import os
+
+# Add src to path for local testing
+sys.path.append(os.path.join(os.getcwd(), 'src'))
+
+from sitewise_crawler import SPACrawler, CrawlerConfig
+
+async def test_crawl():
+    print("🚀 Starting test crawl...")
+    config = CrawlerConfig(
+        start_url="https://www.google.com", # Fast, traditional site
+        max_depth=1,
+        max_pages=2,
+        use_playwright=False # Keep it fast for testing
+    )
+    
+    crawler = SPACrawler(config)
+    
+    def on_page(page):
+        print(f"DEBUG: Processed {page.url}")
+        
+    crawler.on_page_crawled = on_page
+    
+    result = await crawler.crawl()
+    
+    print(f"\nSummary:")
+    print(f"Success: {result.success}")
+    print(f"Pages: {result.total_pages}")
+    print(f"Duration: {result.duration_seconds:.2f}s")
+    
+    if result.success:
+        for p in result.pages_all:
+            print(f"- {p.url} ({p.title})")
+
+if __name__ == "__main__":
+    asyncio.run(test_crawl())