softhauzpy 0.0.1__tar.gz

softhauzpy-0.0.1/PKG-INFO

@@ -0,0 +1,69 @@
+Metadata-Version: 2.1
+Name: softhauzpy
+Version: 0.0.1
+Description-Content-Type: text/markdown
+
+# SofthauzPy
+**SofthauzPy** is a comprehensive Python toolkit built for developers creating intelligent, data-driven web applications. It provides a powerful suite of web utilities including web scraping tools, crawling systems, content extraction pipelines, and search engine components that help developers build fully customizable in-house website search solutions.
+
+Designed for scalability and flexibility, Softhauz enables teams to collect, process, index, and search website content efficiently — all within a clean Python-first development ecosystem.
+
+Built for developers who need scalable web data tools and intelligent search capabilities, Softhauz simplifies the process of scraping, processing, indexing, and searching website content.
+From lightweight crawlers to fully customizable in-house search engine functionality, Softhauz helps developers build smarter web applications without relying heavily on external search services.
+
+
+## Key Features
+
+**Web Scraping & Crawling**
+
+-   High-performance web scraping utilities
+-   HTML parsing and structured data extraction
+-   Recursive website crawling
+-   Sitemap discovery and URL indexing
+-   Support for asynchronous scraping workflows
+-   Rate limiting and request handling utilities
+
+**Search Engine Toolkit**
+
+-   In-house website search engine creation
+-   Full-text indexing and querying
+-   Custom relevance ranking algorithms
+-   Search filtering and query optimization
+-   Incremental indexing support
+-   Lightweight search infrastructure for internal platforms
+
+**Content Processing**
+
+-   Text normalization and cleaning
+-   Metadata extraction
+-   Duplicate content detection
+-   Keyword extraction and tagging
+-   Content chunking for AI and search applications
+
+**AI & Semantic Search Ready**
+
+-   Embedding generation helpers
+-   Vector database compatibility
+-   Semantic similarity search utilities
+-   Retrieval-Augmented Generation (RAG) support
+-   AI-powered content indexing workflows
+
+**Developer Experience**
+
+-   Modular and extensible architecture
+-   Framework-friendly design for Flask, Django, and FastAPI
+-   Easy API integration
+-   Clean, Pythonic interfaces
+-   Production-ready utilities for scalable deployments
+
+> This program may incorporate artificial intelligence (AI) tools solely
+> to support and enhance development efficiency, code quality, and
+> overall performance. All software design, implementation, testing,
+> validation, and quality assurance processes are conducted and reviewed
+> by a qualified human software professional to ensure accuracy,
+> reliability, security, and compliance with applicable standards.
+
+Author:
+**Urate, Karen**<br>
+*Softhauz Software Architect*<br>
+[softhauz.ca](https://softhauz.ca)

softhauzpy-0.0.1/README.md

@@ -0,0 +1,64 @@
+# SofthauzPy
+**SofthauzPy** is a comprehensive Python toolkit built for developers creating intelligent, data-driven web applications. It provides a powerful suite of web utilities including web scraping tools, crawling systems, content extraction pipelines, and search engine components that help developers build fully customizable in-house website search solutions.
+
+Designed for scalability and flexibility, Softhauz enables teams to collect, process, index, and search website content efficiently — all within a clean Python-first development ecosystem.
+
+Built for developers who need scalable web data tools and intelligent search capabilities, Softhauz simplifies the process of scraping, processing, indexing, and searching website content.
+From lightweight crawlers to fully customizable in-house search engine functionality, Softhauz helps developers build smarter web applications without relying heavily on external search services.
+
+
+## Key Features
+
+**Web Scraping & Crawling**
+
+-   High-performance web scraping utilities
+-   HTML parsing and structured data extraction
+-   Recursive website crawling
+-   Sitemap discovery and URL indexing
+-   Support for asynchronous scraping workflows
+-   Rate limiting and request handling utilities
+
+**Search Engine Toolkit**
+
+-   In-house website search engine creation
+-   Full-text indexing and querying
+-   Custom relevance ranking algorithms
+-   Search filtering and query optimization
+-   Incremental indexing support
+-   Lightweight search infrastructure for internal platforms
+
+**Content Processing**
+
+-   Text normalization and cleaning
+-   Metadata extraction
+-   Duplicate content detection
+-   Keyword extraction and tagging
+-   Content chunking for AI and search applications
+
+**AI & Semantic Search Ready**
+
+-   Embedding generation helpers
+-   Vector database compatibility
+-   Semantic similarity search utilities
+-   Retrieval-Augmented Generation (RAG) support
+-   AI-powered content indexing workflows
+
+**Developer Experience**
+
+-   Modular and extensible architecture
+-   Framework-friendly design for Flask, Django, and FastAPI
+-   Easy API integration
+-   Clean, Pythonic interfaces
+-   Production-ready utilities for scalable deployments
+
+> This program may incorporate artificial intelligence (AI) tools solely
+> to support and enhance development efficiency, code quality, and
+> overall performance. All software design, implementation, testing,
+> validation, and quality assurance processes are conducted and reviewed
+> by a qualified human software professional to ensure accuracy,
+> reliability, security, and compliance with applicable standards.
+
+Author:
+**Urate, Karen**<br>
+*Softhauz Software Architect*<br>
+[softhauz.ca](https://softhauz.ca)

softhauzpy-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

softhauzpy-0.0.1/setup.py

@@ -0,0 +1,17 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r") as f:
+    description = f.read()
+
+setup(
+    name='softhauzpy',
+    version='0.0.1',
+    packages=find_packages(),
+    install_requires=[
+        'requests>=2.32.3',
+        'beautifulsoup4>=4.12.3',
+        'nltk>=3.9.4'
+    ],
+    long_description=description,
+    long_description_content_type="text/markdown",
+)

softhauzpy-0.0.1/softhauzpy/__init__.py

@@ -0,0 +1,18 @@
+# fingerprints and mappings
+from .main import incremental_update, highlight_query_terms, build_sitemap_urls
+from .main import fingerprint_page, generate_snippet
+
+# extractions
+from .main import extract_structured_data, extract_headings
+from .main import extract_metadata, extract_links, extract_pure_text
+
+# indexing
+from .main import load_index, save_index, search_index, compute_tfidf, build_inverted_index
+
+# crawls and scrapes
+from .main import tokenize, chunk_text, crawl_site, parse_html, fetch_page, get_search_results_list
+
+
+
+
+

softhauzpy-0.0.1/softhauzpy/main.py

@@ -0,0 +1,1018 @@
+"""
+
+Softhauz is a modern Python package built for software engineers, software developers, and web application architects who need scalable web data tools and intelligent search capabilities. It provides a powerful suite of web utilities including web scraping tools, content extraction pipelines, and search engine components that help developers build fully customizable in-house website search solutions. Softhauz simplifies the process of scraping, processing, indexing, and searching website content.
+
+From lightweight crawlers to fully customizable in-house search engine functionality, Softhauz allows for a seamless building of smarter web applications without relying heavily on external search services. Softhauz helps transform web content into structured, searchable, and intelligent systems with minimal overhead.
+
+Softhauz combines modern web scraping, intelligent indexing, and customizable search capabilities into a unified Python toolkit. Instead of stitching together multiple libraries and services, developers can use Softhauz as a centralized foundation for building scalable web data and search infrastructures tailored to their applications.
+
+Author: Urate, Karen
+Creation Date: 2026-05-09
+External Package List:
+
+- requests >= (v. 2.34.2)
+- beautifulsoup4 >= (v. 4.14.3)
+- nltk >= (v. 3.9.4)
+
+"""
+
+import re
+import json
+import math
+import time
+import hashlib
+import heapq
+from collections import defaultdict, Counter
+from urllib.parse import urljoin, urlparse, urldefrag
+from pathlib import Path
+
+import requests
+from bs4 import BeautifulSoup
+
+_SKIP_TAGS = {
+    "script", "style", "noscript", "head", "meta",
+    "link", "comment", "template", "svg", "iframe",
+}
+
+# ---------------------------------------------------------------------------
+# Optional: nltk for stemming / stopwords.  Gracefully degrade, if missing.
+# ---------------------------------------------------------------------------
+try:
+    import nltk
+    from nltk.stem import PorterStemmer
+    from nltk.corpus import stopwords as nltk_stopwords
+
+    nltk.download("punkt", quiet=True)
+    nltk.download("stopwords", quiet=True)
+    _stemmer = PorterStemmer()
+    _STOPWORDS = set(nltk_stopwords.words("english"))
+    _NLTK_AVAILABLE = True
+except Exception:
+    _stemmer = None
+    _STOPWORDS = {
+        "a", "an", "the", "is", "it", "in", "on", "at", "to", "for",
+        "of", "and", "or", "but", "not", "with", "this", "that", "are",
+        "was", "be", "by", "from", "as", "we", "i", "you", "he", "she",
+    }
+    _NLTK_AVAILABLE = False
+
+"""
+
+    Fetch a webpage and return only the pure text content found within its HTML tags.
+
+    Parameters
+    ----------
+    url           : The URL to fetch.
+    title         : Optional document title (included in the returned text header when provided).
+    author        : Optional document author (included in the returned text header when provided).
+    description   : Optional description (included in the returned text header when provided).
+    creation_date : Optional creation date string (included in the returned text header when provided).
+    modified_date : Optional last-modified date string (included in the returned text header when provided).
+
+
+    Returns
+    -------
+    dict with keys:
+        "url"           : str
+        "title"         : str | None
+        "author"        : str | None
+        "description"   : str | None
+        "creation_date" : str | None
+        "modified_date" : str | None
+        "content"       : str  — pure text extracted from the page
+        "meta_data"     : str  — meta data provided in the parameters
+
+    Raises
+    ------
+    requests.HTTPError
+        If the server returns a non-2xx status code.
+
+"""
+
+
+def extract_pure_text(
+        page_url: str,
+        *,
+        title: str | None = None,
+        author: str | None = None,
+        description: str | None = None,
+        creation_date: str | None = None,
+        modified_date: str | None = None) -> dict:
+    response = fetch_page(page_url, timeout=15)
+    response.raise_for_status()
+
+    soup = BeautifulSoup(response.text, "html.parser")
+
+    for tag in soup.find_all(_SKIP_TAGS):
+        tag.decompose()
+
+    raw_text = soup.get_text(separator=" ", strip=True)
+    lines = " ".join(raw_text.split()).strip()
+
+    header_parts = []
+    if title:
+        header_parts.append(f"Title:         {title}")
+    if author:
+        header_parts.append(f"Author:        {author}")
+    if description:
+        header_parts.append(f"Description:   {description}")
+    if creation_date:
+        header_parts.append(f"Created:       {creation_date}")
+    if modified_date:
+        header_parts.append(f"Last Modified: {modified_date}")
+    if page_url:
+        header_parts.append(f"URL:           {page_url}")
+
+    header = " ".join(header_parts)
+    result = {
+        "url": page_url,
+        "title": title,
+        "author": author,
+        "description": description,
+        "creation_date": creation_date,
+        "modified_date": modified_date,
+        "content": lines,
+        "meta_data": header
+    }
+
+    return result
+
+
+"""
+   Searches a list of pages for entries that match the provided keywords.
+
+   This method iterates over a list of pages and returns a filtered list of tuples 
+   containing pages whose content matches the given keywords. Each tuple in the 
+   returned list contains detailed information about a page.
+
+   Parameters:
+       page_list (list of tuples): A list where each tuple represents a page with the following elements:
+           - url (str): The URL of the page.
+           - title (str): The title of the page.
+           - author (str): The author of the page.
+           - description (str): A brief description of the page.
+           - creation_date (str): The date the page was created.
+           - modified_date (str): The date the page was last modified.
+       keywords (str): A string containing keywords to search for within the page entries.
+
+   Returns:
+       list of tuples: A list of tuples matching the search criteria. Each tuple contains:
+           - url (str)
+           - title (str)
+           - author (str)
+           - description (str)
+           - creation_date (str)
+           - modified_date (str)
+
+   Example:
+       >>> pages = [
+       ...     ("https://example.com", "Example Page", "Alice", "A sample page", "2023-01-01", "2023-01-05"),
+       ...     ("https://another.com", "Another Page", "Bob", "Another sample page", "2023-02-01", "2023-02-05")
+       ... ]
+       >>> search_pages(pages, "sample")
+       [
+           ("https://example.com", "Example Page", "Alice", "A sample page", "2023-01-01", "2023-01-05"),
+           ("https://another.com", "Another Page", "Bob", "Another sample page", "2023-02-01", "2023-02-05")
+       ]
+"""
+
+
+def get_search_results_list(page_list=[], keywords='') -> list:
+    results = []
+
+    for page in page_list:
+
+        url = page[0]
+
+        if len(url) == 0 or len(url) < 1:
+            continue
+
+        title = page[1] or ''
+        author = page[2] or ''
+        description = page[3] or ''
+        creation_date = page[4] or ''
+        modified_date = page[5] or ''
+
+        if keywords in extract_pure_text(url, title, author, description, creation_date, modified_date)["content"]:
+            results.append((url, title, author, description, creation_date, modified_date))
+
+    return results
+
+
+
+"""
+    Fetch a single URL with retry logic and polite delay.
+
+    Args:
+        url:      Target URL.
+        timeout:  Per-request timeout in seconds.
+        retries:  Maximum number of attempts before giving up.
+        delay:    Seconds to wait between retries (doubles on each failure).
+        headers:  Optional extra HTTP headers (merged with a default UA).
+        session:  An existing requests.Session (useful for cookie sharing).
+
+    Returns:
+        A requests.Response on success, or None after all retries fail.
+
+    Example:
+        resp = fetch_page("https://example.com/docs/api")
+        if resp:
+            print(resp.text[:200])
+"""
+def fetch_page(
+        url: str,
+        *,
+        timeout: int = 10,
+        retries: int = 3,
+        delay: float = 1.0,
+        headers: dict | None = None,
+        session: requests.Session | None = None,
+) -> requests.Response | None:
+    _headers = {
+        "User-Agent": (
+            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+        )
+    }
+    if headers:
+        _headers.update(headers)
+
+    requester = session or requests
+    wait = delay
+    for attempt in range(1, retries + 1):
+        try:
+            resp = requester.get(url, headers=_headers, timeout=timeout)
+            resp.raise_for_status()
+            return resp
+        except requests.RequestException as exc:
+            if attempt == retries:
+                print(f"[fetch_page] FAILED AFTER {retries} ATTEMPTS: {exc}")
+                return None
+            time.sleep(wait)
+            wait *= 2
+
+
+"""
+    Parse raw HTML into a BeautifulSoup tree.
+
+    Args:
+        html:   Raw HTML string or bytes.
+        parser: BS4 parser backend ('lxml', 'html.parser', 'html5lib').
+
+    Returns:
+        A BeautifulSoup object ready for querying.
+
+    Example:
+        soup = parse_html(resp.text)
+        title = soup.find("h1").get_text()
+"""
+
+
+def parse_html(
+        html: str | bytes,
+        *,
+        parser: str = "lxml",
+) -> BeautifulSoup:
+    return BeautifulSoup(html, parser)
+
+
+"""
+    Pull structured metadata from a page's <head> section.
+
+    Extracts title, description, keywords, Open Graph tags, canonical URL,
+    language, and author where available.
+
+    Args:
+        soup: Parsed BeautifulSoup object.
+        url:  Original URL (used as fallback for canonical).
+
+    Returns:
+        Dict with keys: title, description, keywords, og_title,
+        og_description, og_image, canonical, lang, author.
+
+    Example:
+        meta = extract_metadata(soup, url="https://example.com/page")
+        print(meta["title"])
+"""
+
+
+def extract_metadata(soup: BeautifulSoup, url: str = "") -> dict:
+
+    def _meta(name: str, attr: str = "name") -> str:
+        tag = soup.find("meta", {attr: name})
+        return tag["content"].strip() if tag and tag.get("content") else ""
+
+    canonical_tag = soup.find("link", rel="canonical")
+    canonical = (
+        canonical_tag["href"] if canonical_tag and canonical_tag.get("href")
+        else url
+    )
+
+    return {
+        "title": (soup.title.string.strip() if soup.title else ""),
+        "description": _meta("description"),
+        "keywords": _meta("keywords"),
+        "og_title": _meta("og:title", "property"),
+        "og_description": _meta("og:description", "property"),
+        "og_image": _meta("og:image", "property"),
+        "canonical": canonical,
+        "lang": soup.html.get("lang", "") if soup.html else "",
+        "author": _meta("author"),
+    }
+
+
+"""
+    Collect all hyperlinks from a page, normalised to absolute URLs.
+
+    Args:
+        soup:               Parsed page.
+        base_url:           Absolute URL of the page being parsed.
+        same_domain_only:   When True, filters out external domains.
+        exclude_extensions: File extensions to skip (e.g. ['.pdf', '.jpg']).
+
+    Returns:
+        Deduplicated list of absolute URL strings.
+
+    Example:
+        links = extract_links(soup, "https://docs.example.com/intro")
+        # => ['https://docs.example.com/api', 'https://docs.example.com/faq']
+"""
+
+
+def extract_links(
+        soup: BeautifulSoup,
+        base_url: str,
+        *,
+        same_domain_only: bool = True,
+        exclude_extensions: list[str] | None = None,
+) -> list[str]:
+    skip_ext = set(exclude_extensions or [
+        ".pdf", ".jpg", ".jpeg", ".png", ".gif", ".svg",
+        ".zip", ".tar", ".gz", ".mp4", ".mp3",
+    ])
+    base_parsed = urlparse(base_url)
+    seen: set[str] = set()
+    result: list[str] = []
+
+    for tag in soup.find_all("a", href=True):
+        raw = tag["href"].strip()
+        if raw.startswith(("mailto:", "tel:", "javascript:", "#")):
+            continue
+        abs_url, _ = urldefrag(urljoin(base_url, raw))
+        parsed = urlparse(abs_url)
+        if same_domain_only and parsed.netloc != base_parsed.netloc:
+            continue
+        if any(parsed.path.lower().endswith(e) for e in skip_ext):
+            continue
+        if abs_url not in seen:
+            seen.add(abs_url)
+            result.append(abs_url)
+
+    return result
+
+
+"""
+    BFS crawler that visits pages starting from start_url.
+
+    Each visited page is fetched, parsed, and its links are queued.
+    Returns a list of page records for further processing.
+
+    Args:
+        start_url:        Root URL to begin crawling.
+        max_pages:        Hard cap on pages visited.
+        same_domain_only: Stay within the same hostname.
+        delay:            Polite pause (seconds) between requests.
+        session:          Reusable requests.Session.
+
+    Returns:
+        List of dicts, each with keys: url, html, soup, status_code.
+
+    Example:
+        pages = crawl_site("https://docs.example.com", max_pages=50)
+        print(f"Crawled {len(pages)} pages")
+"""
+
+
+def crawl_site(
+        start_url: str,
+        *,
+        max_pages: int = 200,
+        same_domain_only: bool = True,
+        delay: float = 0.5,
+        session: requests.Session | None = None,
+) -> list[dict]:
+    visited: set[str] = set()
+    queue: list[str] = [start_url]
+    results: list[dict] = []
+
+    sess = session or requests.Session()
+
+    while queue and len(visited) < max_pages:
+        url = queue.pop(0)
+        if url in visited:
+            continue
+        visited.add(url)
+
+        resp = fetch_page(url, session=sess)
+        if resp is None:
+            continue
+
+        soup = parse_html(resp.text)
+        results.append({
+            "url": url,
+            "html": resp.text,
+            "soup": soup,
+            "status_code": resp.status_code,
+        })
+
+        new_links = extract_links(
+            soup, url, same_domain_only=same_domain_only
+        )
+        for link in new_links:
+            if link not in visited:
+                queue.append(link)
+
+        time.sleep(delay)
+
+    return results
+
+
+"""
+    Split a long document into overlapping word-level chunks.
+
+    Overlapping windows ensure that relevant phrases spanning a chunk
+    boundary are still findable.
+
+    Args:
+        text:       Full document text.
+        chunk_size: Maximum words per chunk.
+        overlap:    Words shared between consecutive chunks.
+
+    Returns:
+        List of text chunks.
+
+    Example:
+        chunks = chunk_text(long_article, chunk_size=200, overlap=30)
+"""
+
+
+def chunk_text(
+        text: str,
+        *,
+        chunk_size: int = 300,
+        overlap: int = 50,
+) -> list[str]:
+    words = text.split()
+    step = max(1, chunk_size - overlap)
+    return [
+        " ".join(words[i: i + chunk_size])
+        for i in range(0, len(words), step)
+        if words[i: i + chunk_size]
+    ]
+
+
+"""
+    Normalise text into a list of meaningful tokens.
+
+    Lowercases, strips punctuation, removes stopwords, and optionally
+    applies Porter stemming (if nltk is installed).
+
+    Args:
+        text:             Input string.
+        remove_stopwords: Filter common English stopwords.
+        stem:             Apply stemming for root-form matching.
+        min_token_len:    Discard tokens shorter than this length.
+
+    Returns:
+        List of processed token strings.
+
+    Example:
+        tokens = tokenize("Building scalable search engines")
+        # => ['build', 'scalabl', 'search', 'engin']  (with stemming)
+"""
+
+
+def tokenize(
+        text: str,
+        *,
+        remove_stopwords: bool = True,
+        stem: bool = True,
+        min_token_len: int = 2,
+) -> list[str]:
+    text = text.lower()
+    text = re.sub(r"[^a-z0-9\s]", " ", text)
+    tokens = [t for t in text.split() if len(t) >= min_token_len]
+
+    if remove_stopwords:
+        tokens = [t for t in tokens if t not in _STOPWORDS]
+
+    if stem and _stemmer:
+        tokens = [_stemmer.stem(t) for t in tokens]
+
+    return tokens
+
+
+"""
+    Build an inverted index mapping tokens to list of (doc_id, frequency).
+
+    The index is the core data structure that enables fast keyword lookups
+    without scanning every document on every query.
+
+    Args:
+        documents:  List of dicts, each containing at least text_field and
+                    id_field.
+        text_field: Key whose value is the text to index.
+        id_field:   Key used as the document identifier.
+
+    Returns:
+        Dict: { token: [(doc_id, freq), ...] }
+
+    Example:
+        index = build_inverted_index(docs)
+        print(index.get("search"))   # [("https://…/search", 12), …]
+
+    ------------------------------------------------------------------------------------
+
+    # SAMPLE LIST OF DICTIONARIES 
+    documents = [
+        {"url": "https://example.com/python-basics",
+         "text": "Python is a versatile programming language used for web development and data science"},
+        {"url": "https://example.com/data-science",
+         "text": "Data science uses Python and statistics to extract insights from data"},
+        {"url": "https://example.com/web-dev",
+         "text": "Web development with Python frameworks like Django makes building apps fast and fun"},
+        {"url": "https://example.com/machine-learning",
+         "text": "Machine learning is a subset of data science that trains models on data"},
+    ]
+
+    # BUILD THE INDEX
+    index = build_inverted_index(documents, text_field="text", id_field="url")
+
+    # EXAMPLE OUTPUT
+    index["python"] →
+    [("https://example.com/python-basics", 1), ("https://example.com/data-science", 1), ("https://example.com/web-dev", 1)]
+"""
+
+
+def build_inverted_index(
+        documents: list[dict],
+        *,
+        text_field: str = "text",
+        id_field: str = "url",
+) -> dict:
+    index: dict[str, list[tuple[str, int]]] = defaultdict(list)
+
+    for doc in documents:
+        doc_id = doc[id_field]
+        text = doc.get(text_field, "")
+        token_freq = Counter(tokenize(text))
+        for token, freq in token_freq.items():
+            index[token].append((doc_id, freq))
+
+    return dict(index)
+
+
+"""
+    Compute TF-IDF scores for every (document, token) pair.
+
+    TF-IDF (Term Frequency x Inverse Document Frequency) balances how
+    often a term appears in one document against how rare it is across
+    the whole corpus — the backbone of classical relevance ranking.
+
+    Args:
+        documents:  Corpus as a list of dicts.
+        text_field: Field containing raw text.
+        id_field:   Field used as document identifier.
+
+    Returns:
+        Nested dict: { doc_id: { token: tfidf_score } }
+
+    Example:
+        scores = compute_tfidf(docs)
+        top = sorted(scores["https://…"].items(), key=lambda x: -x[1])[:5]
+"""
+
+
+def compute_tfidf(
+        documents: list[dict],
+        *,
+        text_field: str = "text",
+        id_field: str = "url",
+) -> dict[str, dict[str, float]]:
+    N = len(documents)
+    tf_store: dict[str, dict[str, float]] = {}
+    doc_freq: Counter = Counter()
+
+    for doc in documents:
+        doc_id = doc[id_field]
+        tokens = tokenize(doc.get(text_field, ""))
+        total = len(tokens) or 1
+        freq = Counter(tokens)
+        tf_store[doc_id] = {t: c / total for t, c in freq.items()}
+        doc_freq.update(freq.keys())
+
+    tfidf: dict[str, dict[str, float]] = {}
+    for doc in documents:
+        doc_id = doc[id_field]
+        tfidf[doc_id] = {
+            t: tf * math.log((N + 1) / (doc_freq[t] + 1))
+            for t, tf in tf_store[doc_id].items()
+        }
+
+    return tfidf
+
+
+"""
+    Score and rank documents for a query using the inverted index + TF-IDF.
+
+    Accumulates TF-IDF scores for all query tokens found in the index,
+    then returns the top-k results by total relevance score.
+
+    Args:
+        query:  Raw user query string.
+        index:  Inverted index from build_inverted_index().
+        tfidf:  TF-IDF matrix from compute_tfidf().
+        top_k:  Maximum results to return.
+
+    Returns:
+        List of (doc_id, score) tuples, highest score first.
+
+    Example:
+        results = search_index("authentication tokens", index, tfidf)
+        for url, score in results:
+            print(f"{score:.3f}  {url}")
+"""
+
+
+def search_index(
+        query: str,
+        index: dict,
+        tfidf: dict[str, dict[str, float]],
+        *,
+        top_k: int = 10,
+) -> list[tuple[str, float]]:
+    query_tokens = tokenize(query)
+    scores: dict[str, float] = defaultdict(float)
+
+    for token in query_tokens:
+        if token in index:
+            for doc_id, _freq in index[token]:
+                scores[doc_id] += tfidf.get(doc_id, {}).get(token, 0.0)
+
+    return heapq.nlargest(top_k, scores.items(), key=lambda x: x[1])
+
+
+"""
+    Return an ordered list of all headings (h1-h6) with their level and text.
+
+    Headings provide document structure signals — boosting a result whose
+    h1 matches a query is a simple way to improve ranking quality.
+
+    Args:
+        soup: Parsed BeautifulSoup object.
+
+    Returns:
+        List of dicts: [{ "level": 1, "text": "Getting Started" }, …]
+
+    Example:
+        headings = extract_headings(soup)
+        print(headings[0])  # {"level": 1, "text": "Introduction"}
+"""
+def extract_headings(soup: BeautifulSoup) -> list[dict]:
+
+    return [
+        {"level": int(tag.name[1]), "text": tag.get_text(strip=True)}
+        for tag in soup.find_all(re.compile(r"^h[1-6]$"))
+        if tag.get_text(strip=True)
+    ]
+
+
+"""
+    Extract a query-centred excerpt to display in search results.
+
+    Finds the first occurrence of any query keyword in the text and
+    returns the surrounding word window, mimicking Google's snippet style.
+
+    Args:
+        text:       Full document text.
+        query:      User's search query.
+        window:     Words to show on each side of the match.
+        max_length: Hard character cap on the returned snippet.
+
+    Returns:
+        A short excerpt string, potentially with leading/trailing ellipsis.
+
+    Example:
+        snippet = generate_snippet(article_text, "authentication tokens")
+        # => "…Users obtain authentication tokens via the /api/auth endpoint…"
+"""
+
+
+def generate_snippet(
+        text: str,
+        query: str,
+        *,
+        window: int = 40,
+        max_length: int = 300,
+) -> str:
+    words = text.split()
+    query_tokens = set(tokenize(query, stem=False, remove_stopwords=False))
+    match_idx = next(
+        (i for i, w in enumerate(words) if re.sub(r"[^a-z]", "", w.lower()) in query_tokens),
+        None,
+    )
+
+    if match_idx is None:
+        snippet = " ".join(words[:window * 2])
+        return (snippet[:max_length] + "…") if len(snippet) > max_length else snippet
+
+    start = max(0, match_idx - window)
+    end = min(len(words), match_idx + window + 1)
+    snippet = " ".join(words[start:end])
+    if start > 0:
+        snippet = "…" + snippet
+    if end < len(words):
+        snippet += "…"
+
+    return snippet[:max_length] or snippet
+
+
+"""
+    Generate a stable SHA-256 fingerprint of a page's text content.
+
+    Store the fingerprint alongside each crawled document to detect
+    whether a page has changed since the last crawl, avoiding re-indexing
+    unchanged content.
+
+    Args:
+        text: Extracted page text (from extract_text).
+
+    Returns:
+        64-character hex string (SHA-256 digest).
+
+    Example:
+        fp = fingerprint_page(page_text)
+        if fp != stored_fingerprints.get(url):
+            re_index(url)
+"""
+
+
+def fingerprint_page(text: str) -> str:
+    normalised = " ".join(text.lower().split())
+    return hashlib.sha256(normalised.encode("utf-8")).hexdigest()
+
+
+"""
+    Serialise the full search index to a JSON file for persistence.
+
+    The saved file can be reloaded on the next run, avoiding a full
+    re-crawl every time the search service starts.
+
+    Args:
+        index:    Inverted index from build_inverted_index().
+        tfidf:    TF-IDF scores from compute_tfidf().
+        metadata: Per-page metadata records (list of dicts).
+        path:     Output file path.
+
+    Example:
+        save_index(index, tfidf, page_metadata, "data/index.json")
+"""
+
+
+def save_index(
+        index: dict,
+        tfidf: dict,
+        metadata: list[dict],
+        path: str = "search_index.json",
+) -> None:
+    payload = {
+        "inverted_index": {k: v for k, v in index.items()},
+        "tfidf": tfidf,
+        "metadata": metadata,
+    }
+    Path(path).parent.mkdir(parents=True, exist_ok=True)
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(payload, f, indent=2, ensure_ascii=False)
+    print(f"[save_index] Saved index to {path}")
+
+
+"""
+    Deserialise a previously saved search index from JSON.
+
+    Args:
+        path: File path written by save_index().
+
+    Returns:
+        Tuple (inverted_index, tfidf, metadata_list).
+
+    Raises:
+        FileNotFoundError: If the file does not exist.
+
+    Example:
+        index, tfidf, metadata = load_index("data/index.json")
+"""
+
+
+def load_index(path: str = "search_index.json") -> tuple[dict, dict, list]:
+    with open(path, "r", encoding="utf-8") as f:
+        payload = json.load(f)
+
+    index = payload["inverted_index"]
+    index = {k: [tuple(pair) for pair in v] for k, v in index.items()}
+    return index, payload["tfidf"], payload["metadata"]
+
+
+"""
+    Extract JSON-LD structured data blocks embedded in a page.
+
+    Many modern websites include schema.org markup (Article, FAQPage,
+    BreadcrumbList, etc.) which provides clean, machine-readable content
+    ideal for enriching search results.
+
+    Args:
+        soup: Parsed BeautifulSoup object.
+
+    Returns:
+        List of parsed JSON-LD objects found on the page.
+
+    Example:
+        schemas = extract_structured_data(soup)
+        for s in schemas:
+            print(s.get("@type"), s.get("name"))
+"""
+
+
+def extract_structured_data(soup: BeautifulSoup) -> list[dict]:
+    results = []
+    for tag in soup.find_all("script", type="application/ld+json"):
+        try:
+            data = json.loads(tag.string or "{}")
+            if isinstance(data, list):
+                results.extend(data)
+            else:
+                results.append(data)
+        except json.JSONDecodeError:
+            continue
+    return results
+
+
+"""
+    Parse a site's sitemap.xml to seed the crawler URL queue.
+
+    Tries /sitemap.xml and /sitemap_index.xml; follows nested sitemaps
+    (sitemapindex elements) one level deep.
+
+    Args:
+        base_url: Root URL of the site, e.g. "https://docs.example.com".
+        session:  Optional reusable requests.Session.
+
+    Returns:
+        Sorted, deduplicated list of page URLs listed in the sitemap(s).
+
+    Example:
+        urls = build_sitemap_urls("https://docs.example.com")
+        print(f"Found {len(urls)} URLs in sitemap")
+"""
+
+
+def build_sitemap_urls(
+        base_url: str,
+        *,
+        session: requests.Session | None = None,
+) -> list[str]:
+    parsed = urlparse(base_url)
+    origin = f"{parsed.scheme}://{parsed.netloc}"
+    candidates = [f"{origin}/sitemap.xml", f"{origin}/sitemap_index.xml"]
+    sess = session or requests
+
+    def _parse_sitemap(url: str) -> list[str]:
+        resp = fetch_page(url, session=sess)
+        if not resp:
+            return []
+        soup = BeautifulSoup(resp.content, "lxml-xml")
+        child_maps = [loc.text.strip() for loc in soup.find_all("sitemap") if loc.find("loc")]
+        urls: list[str] = []
+        for sm in child_maps:
+            urls.extend(_parse_sitemap(sm))
+        urls.extend(loc.text.strip() for loc in soup.find_all("url") if loc.find("loc"))
+        return urls
+
+    all_urls: set[str] = set()
+    for candidate in candidates:
+        found = _parse_sitemap(candidate)
+        all_urls.update(found)
+        if all_urls:
+            break
+
+    return sorted(all_urls)
+
+
+"""
+    Wrap query keywords in HTML highlight tags within a snippet string.
+
+    Use in search result UIs to visually emphasise where the match occurs.
+    The matching is case-insensitive and handles whole words only.
+
+    Args:
+        snippet:   Text excerpt (from generate_snippet).
+        query:     Original user query.
+        open_tag:  Opening HTML tag (default <mark>).
+        close_tag: Closing HTML tag (default </mark>).
+
+    Returns:
+        Snippet string with matching keywords wrapped in tags.
+
+    Example:
+        highlighted = highlight_query_terms(
+            "retrieve authentication tokens from the vault",
+            "authentication tokens"
+        )
+        # => "retrieve <mark>authentication</mark> <mark>tokens</mark>…"
+"""
+
+
+def highlight_query_terms(
+        snippet: str,
+        query: str,
+        *,
+        open_tag: str = "<mark>",
+        close_tag: str = "</mark>",
+) -> str:
+    keywords = [
+        re.escape(w) for w in query.split()
+        if w.lower() not in _STOPWORDS and len(w) > 1
+    ]
+    if not keywords:
+        return snippet
+
+    pattern = re.compile(r"(" + "|".join(keywords) + r")", re.IGNORECASE)
+    return pattern.sub(rf"{open_tag}\1{close_tag}", snippet)
+
+
+"""
+    Re-index a single URL only if its content has changed.
+
+    Compares the page's current fingerprint against a stored one; if
+    unchanged the function exits early, making scheduled re-crawls cheap.
+
+    Args:
+        url:          Page to check and potentially re-index.
+        index:        Mutable inverted index (modified in place).
+        tfidf:        Mutable TF-IDF store (modified in place).
+        metadata:     Mutable metadata list (modified in place).
+        fingerprints: Dict mapping url to last known fingerprint (mutable).
+        session:      Optional requests.Session.
+
+    Returns:
+        True if the page was re-indexed, False if it was unchanged.
+
+    Example:
+        changed = incremental_update(url, index, tfidf, metadata, fps)
+        print("Updated" if changed else "No change")
+"""
+
+
+def incremental_update(
+        url: str,
+        index: dict,
+        tfidf: dict,
+        metadata: list[dict],
+        fingerprints: dict[str, str],
+        *,
+        session: requests.Session | None = None,
+) -> bool:
+    resp = fetch_page(url, session=session)
+    if resp is None:
+        return False
+
+    soup = parse_html(resp.text)
+    text = extract_text(soup)
+    fp = fingerprint_page(text)
+
+    if fingerprints.get(url) == fp:
+        return False  # No change — skip re-indexing
+
+    fingerprints[url] = fp
+
+    # Remove stale entries from index
+    for token in list(index.keys()):
+        index[token] = [(doc_id, freq) for doc_id, freq in index[token] if doc_id != url]
+        if not index[token]:
+            del index[token]
+
+    # Remove stale tfidf and metadata entries
+    tfidf.pop(url, None)
+    metadata[:] = [m for m in metadata if m.get("url") != url]
+
+    # Build fresh entries for this page
+    token_freq = Counter(tokenize(text))
+    total = len(list(token_freq.elements())) or 1
+    for token, freq in token_freq.items():
+        index.setdefault(token, []).append((url, freq))
+    tfidf[url] = {t: (c / total) for t, c in token_freq.items()}
+
+    meta = extract_metadata(soup, url)
+    meta["url"] = url
+    meta["fingerprint"] = fp
+    metadata.append(meta)
+
+    return True

softhauzpy-0.0.1/softhauzpy.egg-info/PKG-INFO

@@ -0,0 +1,69 @@
+Metadata-Version: 2.1
+Name: softhauzpy
+Version: 0.0.1
+Description-Content-Type: text/markdown
+
+# SofthauzPy
+**SofthauzPy** is a comprehensive Python toolkit built for developers creating intelligent, data-driven web applications. It provides a powerful suite of web utilities including web scraping tools, crawling systems, content extraction pipelines, and search engine components that help developers build fully customizable in-house website search solutions.
+
+Designed for scalability and flexibility, Softhauz enables teams to collect, process, index, and search website content efficiently — all within a clean Python-first development ecosystem.
+
+Built for developers who need scalable web data tools and intelligent search capabilities, Softhauz simplifies the process of scraping, processing, indexing, and searching website content.
+From lightweight crawlers to fully customizable in-house search engine functionality, Softhauz helps developers build smarter web applications without relying heavily on external search services.
+
+
+## Key Features
+
+**Web Scraping & Crawling**
+
+-   High-performance web scraping utilities
+-   HTML parsing and structured data extraction
+-   Recursive website crawling
+-   Sitemap discovery and URL indexing
+-   Support for asynchronous scraping workflows
+-   Rate limiting and request handling utilities
+
+**Search Engine Toolkit**
+
+-   In-house website search engine creation
+-   Full-text indexing and querying
+-   Custom relevance ranking algorithms
+-   Search filtering and query optimization
+-   Incremental indexing support
+-   Lightweight search infrastructure for internal platforms
+
+**Content Processing**
+
+-   Text normalization and cleaning
+-   Metadata extraction
+-   Duplicate content detection
+-   Keyword extraction and tagging
+-   Content chunking for AI and search applications
+
+**AI & Semantic Search Ready**
+
+-   Embedding generation helpers
+-   Vector database compatibility
+-   Semantic similarity search utilities
+-   Retrieval-Augmented Generation (RAG) support
+-   AI-powered content indexing workflows
+
+**Developer Experience**
+
+-   Modular and extensible architecture
+-   Framework-friendly design for Flask, Django, and FastAPI
+-   Easy API integration
+-   Clean, Pythonic interfaces
+-   Production-ready utilities for scalable deployments
+
+> This program may incorporate artificial intelligence (AI) tools solely
+> to support and enhance development efficiency, code quality, and
+> overall performance. All software design, implementation, testing,
+> validation, and quality assurance processes are conducted and reviewed
+> by a qualified human software professional to ensure accuracy,
+> reliability, security, and compliance with applicable standards.
+
+Author:
+**Urate, Karen**<br>
+*Softhauz Software Architect*<br>
+[softhauz.ca](https://softhauz.ca)

softhauzpy-0.0.1/softhauzpy.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+README.md
+setup.py
+softhauzpy/__init__.py
+softhauzpy/main.py
+softhauzpy.egg-info/PKG-INFO
+softhauzpy.egg-info/SOURCES.txt
+softhauzpy.egg-info/dependency_links.txt
+softhauzpy.egg-info/requires.txt
+softhauzpy.egg-info/top_level.txt

softhauzpy-0.0.1/softhauzpy.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

softhauzpy-0.0.1/softhauzpy.egg-info/requires.txt

@@ -0,0 +1,3 @@
+requests>=2.32.3
+beautifulsoup4>=4.12.3
+nltk>=3.9.4

softhauzpy-0.0.1/softhauzpy.egg-info/top_level.txt

@@ -0,0 +1 @@
+softhauzpy