segment_classifier 0.1.0__tar.gz

segment_classifier-0.1.0/PKG-INFO

@@ -0,0 +1,95 @@
+Metadata-Version: 2.3
+Name: segment_classifier
+Version: 0.1.0
+Summary: Async segment classifier library
+Author: Gagandeep Singh
+Author-email: gagan@innerkore.com
+Requires-Python: >=3.12,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: aiofiles (>=23.0,<24.0)
+Requires-Dist: beautifulsoup4 (>=4.12,<5.0)
+Requires-Dist: litellm (>=1.40,<2.0)
+Requires-Dist: lxml (>=5.0,<6.0)
+Requires-Dist: numpy (>=1.26,<2.0)
+Requires-Dist: pydantic (>=2.7,<3.0)
+Requires-Dist: pydantic-settings (>=2.2,<3.0)
+Requires-Dist: scikit-learn (>=1.5,<2.0)
+Description-Content-Type: text/markdown
+
+# Segment Classifier
+
+An asynchronous Python library that classifies HTML segments extracted by a page-segmenter into structured component types.
+
+## Overview
+
+The `segment_classifier` implements a 4-stage classification pipeline with progressive fallback to optimize for cost and speed:
+
+1. **Rule-based heuristics** — Zero LLM cost. Uses DOM structure, text density, siblings, and attributes.
+2. **L1 exact fingerprint cache** — Zero LLM cost. Exact matching on structural DOM fingerprint hashes.
+3. **L2 fuzzy cluster cache** — Zero LLM cost. TF-IDF and cosine similarity on fingerprint tokens.
+4. **LLM batch classification** — Batched fallback via LiteLLM with feature-based model routing based on segment complexity.
+
+## Installation
+
+You can install the package using poetry:
+```bash
+poetry install
+```
+
+Or via pip (once published):
+```bash
+pip install segment-classifier
+```
+
+## Setup
+
+The library uses `pydantic-settings` to manage configuration via a `.env` file or environment variables.
+
+Required environment variables:
+```env
+CLASSIFIER_LITELLM_API_KEY="your-api-key"
+```
+
+## Usage
+
+```python
+import asyncio
+from segment_classifier import ClassifierPipeline
+from segment_classifier.config import ClassifierSettings
+from segment_classifier.models import InputSegment, SegmentPosition
+
+async def main():
+    settings = ClassifierSettings()
+    pipeline = ClassifierPipeline(settings)
+    await pipeline.initialize()
+
+    segments = [
+        InputSegment(
+            segment_id="seg_001",
+            page_url="https://example.com/products",
+            page_slug="products",
+            raw_html="<div class='product-card'>...</div>",
+            text_content="Product Item",
+            position_hint=SegmentPosition.MIDDLE,
+            sibling_count=3,
+        )
+    ]
+
+    result = await pipeline.run(segments)
+    await pipeline.shutdown()
+
+    for seg in result.classified:
+        print(seg.component_type)
+
+asyncio.run(main())
+```
+
+## Caching
+
+Caches are stored by default in `.cache/l1_fingerprints.json` and `.cache/l2_clusters.json` / `.cache/l2_embeddings.npy`.
+
+## Stages Breakdown
+Every returned `ClassifiedSegment` will be marked with a `classification_stage` indicating which of the 4 stages resolved the query.
+

segment_classifier-0.1.0/README.md

@@ -0,0 +1,74 @@
+# Segment Classifier
+
+An asynchronous Python library that classifies HTML segments extracted by a page-segmenter into structured component types.
+
+## Overview
+
+The `segment_classifier` implements a 4-stage classification pipeline with progressive fallback to optimize for cost and speed:
+
+1. **Rule-based heuristics** — Zero LLM cost. Uses DOM structure, text density, siblings, and attributes.
+2. **L1 exact fingerprint cache** — Zero LLM cost. Exact matching on structural DOM fingerprint hashes.
+3. **L2 fuzzy cluster cache** — Zero LLM cost. TF-IDF and cosine similarity on fingerprint tokens.
+4. **LLM batch classification** — Batched fallback via LiteLLM with feature-based model routing based on segment complexity.
+
+## Installation
+
+You can install the package using poetry:
+```bash
+poetry install
+```
+
+Or via pip (once published):
+```bash
+pip install segment-classifier
+```
+
+## Setup
+
+The library uses `pydantic-settings` to manage configuration via a `.env` file or environment variables.
+
+Required environment variables:
+```env
+CLASSIFIER_LITELLM_API_KEY="your-api-key"
+```
+
+## Usage
+
+```python
+import asyncio
+from segment_classifier import ClassifierPipeline
+from segment_classifier.config import ClassifierSettings
+from segment_classifier.models import InputSegment, SegmentPosition
+
+async def main():
+    settings = ClassifierSettings()
+    pipeline = ClassifierPipeline(settings)
+    await pipeline.initialize()
+
+    segments = [
+        InputSegment(
+            segment_id="seg_001",
+            page_url="https://example.com/products",
+            page_slug="products",
+            raw_html="<div class='product-card'>...</div>",
+            text_content="Product Item",
+            position_hint=SegmentPosition.MIDDLE,
+            sibling_count=3,
+        )
+    ]
+
+    result = await pipeline.run(segments)
+    await pipeline.shutdown()
+
+    for seg in result.classified:
+        print(seg.component_type)
+
+asyncio.run(main())
+```
+
+## Caching
+
+Caches are stored by default in `.cache/l1_fingerprints.json` and `.cache/l2_clusters.json` / `.cache/l2_embeddings.npy`.
+
+## Stages Breakdown
+Every returned `ClassifiedSegment` will be marked with a `classification_stage` indicating which of the 4 stages resolved the query.

segment_classifier-0.1.0/pyproject.toml

@@ -0,0 +1,24 @@
+[tool.poetry]
+name = "segment_classifier"
+version = "0.1.0"
+description = "Async segment classifier library"
+authors = ["Gagandeep Singh <gagan@innerkore.com>"]
+readme = "README.md"
+packages = [
+    { include = "segment_classifier" }
+]
+
+[tool.poetry.dependencies]
+python = "^3.12"
+beautifulsoup4 = "^4.12"
+lxml = "^5.0"
+pydantic = "^2.7"
+pydantic-settings = "^2.2"
+litellm = "^1.40"
+scikit-learn = "^1.5"
+numpy = "^1.26"
+aiofiles = "^23.0"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

segment_classifier-0.1.0/segment_classifier/__init__.py

@@ -0,0 +1,4 @@
+from .pipeline import ClassifierPipeline
+from .config import ClassifierSettings
+
+__all__ = ["ClassifierPipeline", "ClassifierSettings"]

segment_classifier-0.1.0/segment_classifier/cache/__init__.py

@@ -0,0 +1,4 @@
+from .l1_cache import L1FingerprintCache
+from .l2_cache import L2FuzzyCache
+
+__all__ = ["L1FingerprintCache", "L2FuzzyCache"]

segment_classifier-0.1.0/segment_classifier/cache/l1_cache.py

@@ -0,0 +1,71 @@
+import asyncio
+import json
+from pathlib import Path
+import aiofiles
+from pydantic import ValidationError
+from segment_classifier.models import FingerprintRecord
+
+
+class L1FingerprintCache:
+    def __init__(self, cache_path: str, auto_persist_every: int = 50):
+        self._path = Path(cache_path)
+        self._store: dict[str, FingerprintRecord] = {}
+        self._lock = asyncio.Lock()
+        self._write_count = 0
+        self._auto_persist_every = auto_persist_every
+
+    async def load(self) -> None:
+        if not self._path.exists():
+            return
+
+        async with aiofiles.open(self._path, "r", encoding="utf-8") as f:
+            content = await f.read()
+            if not content.strip():
+                return
+
+            try:
+                data = json.loads(content)
+                for key, val in data.items():
+                    self._store[key] = FingerprintRecord.model_validate(val)
+            except (json.JSONDecodeError, ValidationError) as e:
+                # Log error in real app, we'll just ignore and start fresh here
+                pass
+
+    async def get(self, fingerprint_hash: str) -> FingerprintRecord | None:
+        async with self._lock:
+            return self._store.get(fingerprint_hash)
+
+    async def set(self, fingerprint_hash: str, record: FingerprintRecord) -> None:
+        async with self._lock:
+            self._store[fingerprint_hash] = record
+            self._write_count += 1
+            if self._write_count >= self._auto_persist_every:
+                self._write_count = 0
+                await self._persist_unsafe()
+
+    async def increment_hit(self, fingerprint_hash: str) -> None:
+        async with self._lock:
+            record = self._store.get(fingerprint_hash)
+            if record:
+                record.hit_count += 1
+                self._store[fingerprint_hash] = record
+                self._write_count += 1
+                if self._write_count >= self._auto_persist_every:
+                    self._write_count = 0
+                    await self._persist_unsafe()
+
+    async def _persist_unsafe(self) -> None:
+        """Called internally when lock is already acquired."""
+        self._path.parent.mkdir(parents=True, exist_ok=True)
+        data = {k: v.model_dump(mode="json") for k, v in self._store.items()}
+        async with aiofiles.open(self._path, "w", encoding="utf-8") as f:
+            await f.write(json.dumps(data, indent=2))
+
+    async def persist(self) -> None:
+        async with self._lock:
+            self._write_count = 0
+            await self._persist_unsafe()
+
+    @property
+    def size(self) -> int:
+        return len(self._store)

segment_classifier-0.1.0/segment_classifier/cache/l2_cache.py

@@ -0,0 +1,157 @@
+import asyncio
+import json
+import uuid
+from pathlib import Path
+import aiofiles
+import numpy as np
+from pydantic import ValidationError
+from segment_classifier.models import ClusterRecord, ComponentType
+
+
+class L2FuzzyCache:
+    def __init__(
+        self,
+        cache_path: str,
+        embeddings_path: str,
+        similarity_threshold: float = 0.85,
+        max_cluster_size: int = 50,
+        persist_on_update: bool = True
+    ):
+        self._path = Path(cache_path)
+        self._embeddings_path = Path(embeddings_path)
+        self._similarity_threshold = similarity_threshold
+        self._max_cluster_size = max_cluster_size
+        self._persist_on_update = persist_on_update
+
+        self._store: list[ClusterRecord] = []
+        # Matrix storing centroids, parallel to self._store
+        self._centroids: np.ndarray | None = None
+
+        self._lock = asyncio.Lock()
+
+    async def load(self) -> None:
+        if not self._path.exists() or not self._embeddings_path.exists():
+            return
+
+        async with aiofiles.open(self._path, "r", encoding="utf-8") as f:
+            content = await f.read()
+            if content.strip():
+                try:
+                    data = json.loads(content)
+                    self._store = [ClusterRecord.model_validate(val) for val in data]
+                except (json.JSONDecodeError, ValidationError):
+                    self._store = []
+
+        if self._store:
+            try:
+                self._centroids = np.load(str(self._embeddings_path))
+                if len(self._store) != self._centroids.shape[0]:
+                    # Mismatch between JSON and npy, reset
+                    self._store = []
+                    self._centroids = None
+            except Exception:
+                self._store = []
+                self._centroids = None
+
+    async def find_nearest(self, vector: list[float], threshold: float | None = None) -> ClusterRecord | None:
+        async with self._lock:
+            if not self._store or self._centroids is None:
+                return None
+
+            query = np.array(vector)
+            query_norm = np.linalg.norm(query)
+            if query_norm == 0:
+                return None
+
+            # Cosine similarity
+            dot_products = np.dot(self._centroids, query)
+            centroid_norms = np.linalg.norm(self._centroids, axis=1)
+            # Avoid division by zero
+            centroid_norms[centroid_norms == 0] = 1
+
+            similarities = dot_products / (centroid_norms * query_norm)
+
+            best_idx = np.argmax(similarities)
+            best_sim = similarities[best_idx]
+
+            check_threshold = threshold if threshold is not None else self._similarity_threshold
+
+            if best_sim >= check_threshold:
+                return self._store[best_idx]
+            return None
+
+    async def add_to_cluster(self, cluster_id: str, fingerprint_hash: str, vector: list[float]) -> None:
+        async with self._lock:
+            idx = next((i for i, c in enumerate(self._store) if c.cluster_id == cluster_id), None)
+            if idx is not None and self._centroids is not None:
+                cluster = self._store[idx]
+
+                # Check size
+                if len(cluster.member_fingerprints) < self._max_cluster_size:
+                    if fingerprint_hash not in cluster.member_fingerprints:
+                        cluster.member_fingerprints.append(fingerprint_hash)
+
+                        # Update centroid (running mean)
+                        n = len(cluster.member_fingerprints)
+                        old_centroid = self._centroids[idx]
+                        new_vec = np.array(vector)
+                        # (old_centroid * (n-1) + new_vec) / n
+                        new_centroid = (old_centroid * (n - 1) + new_vec) / n
+
+                        self._centroids[idx] = new_centroid
+                        cluster.centroid_vector = new_centroid.tolist()
+
+                        if self._persist_on_update:
+                            await self._persist_unsafe()
+
+    async def create_cluster(
+        self,
+        fingerprint_hash: str,
+        vector: list[float],
+        component_type: ComponentType,
+        confidence: float,
+    ) -> ClusterRecord:
+        async with self._lock:
+            cluster_id = str(uuid.uuid4())
+            record = ClusterRecord(
+                cluster_id=cluster_id,
+                centroid_vector=vector,
+                component_type=component_type,
+                confidence=confidence,
+                member_fingerprints=[fingerprint_hash]
+            )
+
+            self._store.append(record)
+
+            new_vec = np.array([vector])
+            if self._centroids is None:
+                self._centroids = new_vec
+            else:
+                self._centroids = np.vstack([self._centroids, new_vec])
+
+            if self._persist_on_update:
+                await self._persist_unsafe()
+
+            return record
+
+    async def _persist_unsafe(self) -> None:
+        self._path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Write JSON
+        data = [c.model_dump(mode="json") for c in self._store]
+        async with aiofiles.open(self._path, "w", encoding="utf-8") as f:
+            await f.write(json.dumps(data, indent=2))
+
+        # Write npy
+        if self._centroids is not None:
+            # We can't do aiofiles easily for numpy, doing it sync for now
+            # as it's a small matrix and we use it as memory store
+            np.save(str(self._embeddings_path), self._centroids)
+
+    async def persist(self) -> None:
+        async with self._lock:
+            await self._persist_unsafe()
+
+    @property
+    def size(self) -> int:
+        return len(self._store)

segment_classifier-0.1.0/segment_classifier/config.py

@@ -0,0 +1,53 @@
+from pydantic import BaseModel
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class ModelFeatureConfig(BaseModel):
+    """
+    Feature-based LLM model routing.
+
+    Selection priority (highest to lowest):
+    1. high_complexity  → use for ambiguous, deeply nested, multi-role segments
+    2. standard         → default for most unknown segments
+    3. fast             → simple segments with weak signals but not rule-matchable
+
+    Complexity is determined by:
+    - dom_depth > threshold
+    - child_tag_counts diversity (many unique tags = complex)
+    - text_density_ratio (very high or very low = complex)
+    - sibling_count == 0 (one-off sections = complex)
+    """
+    high_complexity_model: str = "anthropic/claude-opus-4"
+    standard_model: str = "anthropic/claude-sonnet-4-5"
+    fast_model: str = "anthropic/claude-haiku-4-5"
+
+    high_complexity_dom_depth_threshold: int = 6
+    high_complexity_unique_tag_threshold: int = 8
+    fast_model_max_dom_depth: int = 3
+
+
+class CacheConfig(BaseModel):
+    l1_cache_path: str = ".cache/l1_fingerprints.json"
+    l2_cache_path: str = ".cache/l2_clusters.json"
+    l2_embeddings_path: str = ".cache/l2_embeddings.npy"
+    l2_similarity_threshold: float = 0.85
+    l2_max_cluster_size: int = 50
+    persist_on_update: bool = True
+
+
+class ClassifierSettings(BaseSettings):
+    model_config = SettingsConfigDict(env_file=".env", env_prefix="CLASSIFIER_")
+
+    # LiteLLM
+    litellm_api_key: str = ""
+    litellm_batch_size: int = 20         # max segments per LLM batch call
+    litellm_max_concurrent_batches: int = 5
+    litellm_timeout_seconds: int = 60
+
+    # Pipeline
+    rule_based_confidence_threshold: float = 0.90
+    l1_min_confidence: float = 0.85
+    l2_min_confidence: float = 0.75
+
+    model_routing: ModelFeatureConfig = ModelFeatureConfig()
+    cache: CacheConfig = CacheConfig()

segment_classifier-0.1.0/segment_classifier/models.py

@@ -0,0 +1,142 @@
+from enum import Enum
+from typing import Any
+from pydantic import BaseModel, Field
+
+
+class ClassificationStage(str, Enum):
+    RULE_BASED = "rule_based"
+    L1_EXACT_CACHE = "l1_exact_cache"
+    L2_FUZZY_CACHE = "l2_fuzzy_cache"
+    LLM = "llm"
+
+
+class SegmentPosition(str, Enum):
+    TOP = "top"           # top 5% of page
+    BOTTOM = "bottom"     # bottom 10% of page
+    MIDDLE = "middle"
+    UNKNOWN = "unknown"
+
+
+class ComponentType(str, Enum):
+    # Layout
+    LAYOUT_HEADER = "layout.header"
+    LAYOUT_FOOTER = "layout.footer"
+    LAYOUT_NAV = "layout.nav"
+    LAYOUT_SIDEBAR = "layout.sidebar"
+    LAYOUT_BREADCRUMB = "layout.breadcrumb"
+
+    # Collections
+    COLLECTION_PRODUCT_CARD = "collection.product_card"
+    COLLECTION_PRODUCT_LIST = "collection.product_list"
+    COLLECTION_BLOG_CARD = "collection.blog_card"
+    COLLECTION_BLOG_LIST = "collection.blog_list"
+    COLLECTION_NEWS_ITEM = "collection.news_item"
+    COLLECTION_NEWS_LIST = "collection.news_list"
+
+    # Sections
+    SECTION_HERO = "section.hero"
+    SECTION_FEATURE_GRID = "section.feature_grid"
+    SECTION_TESTIMONIAL = "section.testimonial"
+    SECTION_CTA = "section.cta"
+    SECTION_FAQ = "section.faq"
+    SECTION_PRICING = "section.pricing"
+
+    # UI Elements
+    UI_FORM = "ui.form"
+    UI_MODAL = "ui.modal"
+    UI_TABLE = "ui.table"
+    UI_CAROUSEL = "ui.carousel"
+    UI_PAGINATION = "ui.pagination"
+    UI_SEARCH = "ui.search"
+
+    # Content
+    CONTENT_ARTICLE = "content.article"
+    CONTENT_RICH_TEXT = "content.rich_text"
+    CONTENT_MEDIA = "content.media"
+
+    UNKNOWN = "unknown"
+
+
+class InputSegment(BaseModel):
+    """Raw segment from the page-segmenter tool."""
+    segment_id: str
+    page_url: str
+    page_slug: str
+    raw_html: str
+    text_content: str
+    position_hint: SegmentPosition = SegmentPosition.UNKNOWN
+    dom_position: str = ""                    # CSS selector path e.g. "main > section:nth-child(2)"
+    sibling_count: int = 0                    # how many same-fingerprint siblings on same page
+    url_path_segments: list[str] = Field(default_factory=list)  # e.g. ["products", "shoes"]
+
+
+class ClassifiedSegment(BaseModel):
+    """Segment with classification result and metadata."""
+    segment_id: str
+    page_url: str
+    page_slug: str
+    raw_html: str
+    text_content: str
+    position_hint: SegmentPosition
+
+    # Classification output
+    component_type: ComponentType
+    classification_stage: ClassificationStage
+    confidence: float = Field(ge=0.0, le=1.0)
+
+    # Fingerprint computed during pipeline
+    fingerprint_hash: str = ""
+    cluster_id: str | None = None
+
+    # LLM metadata (populated only for stage=LLM)
+    llm_model_used: str | None = None
+    llm_raw_response: str | None = None
+
+
+class FingerprintRecord(BaseModel):
+    """Stored in L1 cache: fingerprint → classification."""
+    fingerprint_hash: str
+    component_type: ComponentType
+    confidence: float
+    hit_count: int = 1
+    example_segment_id: str = ""
+
+
+class ClusterRecord(BaseModel):
+    """Stored in L2 cache: cluster of similar fingerprints."""
+    cluster_id: str
+    centroid_vector: list[float]
+    component_type: ComponentType
+    confidence: float
+    member_fingerprints: list[str] = Field(default_factory=list)
+
+
+class LLMClassificationRequest(BaseModel):
+    """Batch item sent to LLM."""
+    segment_id: str
+    fingerprint_hash: str
+    normalized_html: str          # skeleton only, no content
+    position_hint: SegmentPosition
+    sibling_count: int
+    url_hints: list[str]
+    dom_depth: int
+    child_tag_counts: dict[str, int]
+    text_density_ratio: float
+
+
+class LLMClassificationResult(BaseModel):
+    """Parsed result from LLM for one segment."""
+    segment_id: str
+    component_type: ComponentType
+    confidence: float
+    reasoning: str
+
+
+class PipelineResult(BaseModel):
+    """Final output of the full classification pipeline run."""
+    total_segments: int
+    classified: list[ClassifiedSegment]
+    stage_breakdown: dict[ClassificationStage, int]
+    llm_calls_made: int
+    llm_model_usage: dict[str, int]   # model_name → call count
+    cache_hit_rate: float