vlm-sieve 0.1.0__tar.gz

vlm_sieve-0.1.0/.agent/rules.md

@@ -0,0 +1,59 @@
+# vlm-sieve: Workspace Architectural Rules & Engineering Guidelines
+
+## 1. Project Identity & Purpose
+- **Distribution Name:** `vlm-sieve`
+- **Import Package Name:** `vlmsieve`
+- **Core Purpose:** High-throughput, event-driven video frame gatekeeper, active Region-of-Interest (ROI) cropper, and temporal storyboard packer. It selectively filters redundant video frames and intelligently packs salient visual information to reduce Vision-Language Model (VLM) image token consumption by >95% while operating purely on CPU.
+
+---
+
+## 2. Strict Architectural Constraints
+
+### 2.1 Core Dependency Guardrails
+- **Heavy Deep-Learning Prohibition:** Heavy deep-learning frameworks (`torch`, `tensorflow`, `transformers`, `jax`, `onnxruntime`, etc.) are **STRICTLY FORBIDDEN** in core dependencies.
+- **Allowed Core Dependencies Only:**
+  - `numpy>=1.24.0`
+  - `opencv-python-headless>=4.8.0`
+  - `Pillow>=10.0.0`
+  - `pydantic>=2.5.0`
+- Any optional neural/model integrations must live in separated optional extras (e.g., `vlm-sieve[torch]`) and must never be imported in core modules without graceful fallbacks.
+
+### 2.2 Performance & Latency Targets
+- **CPU Frame Gating Latency:** `< 3ms` per frame at 1080p input resolution.
+- **Downscaled Grayscale Processing:** Gating decisions must operate on downscaled grayscale representations (e.g., 360p or lower, pyramid representations) to guarantee sub-3ms evaluation on commodity single-core CPUs.
+- **Memory Footprint:** Zero-copy operations and buffer reuse wherever possible when piping NumPy arrays between OpenCV and Pillow. Avoid redundant array allocations.
+
+### 2.3 Implementation & Code Integrity
+- **Zero Placeholder Stubs:** Every class, function, and method must be completely implemented, mathematically sound, and functional. No empty `pass` bodies, no `...` ellipses in concrete logic, and no `TODO` or unhandled edge cases.
+- **Strict Typing:** All modules must enforce Python 3.9+ type annotations on all function signatures, return values, and class attributes. Utilize `typing` / `pydantic` types and adhere to strict static typing checks.
+
+---
+
+## 3. Core Functional Pillars
+
+### 3.1 Temporal Frame Gating (`vlmsieve.gating`)
+- Detect meaningful visual delta between consecutive and sliding-window frames.
+- Employ lightweight algorithms:
+  - Structural Similarity Metric (SSIM) approximations on downscaled tiles.
+  - Pixel-difference statistics with adaptive luminosity normalization.
+  - Motion energy accumulation (frame differencing / background subtraction).
+- Fast drop of static or sub-threshold frames before any expensive image processing.
+
+### 3.2 Active Region-of-Interest Cropper (`vlmsieve.cropper`)
+- Identify dynamic bounding boxes containing the active motion or high-frequency visual change.
+- Morphological dilation and contour clustering to avoid fragmented crops.
+- Dynamic aspect-ratio padding to preserve contextual cues without transmitting vast static backgrounds to VLMs.
+
+### 3.3 Temporal Storyboard Packer (`vlmsieve.packer`)
+- Composite multiple sequential keyframes or ROI patches into high-density temporal grid storyboards.
+- Timestamp and sequence metadata overlay/badges for multi-frame contextual alignment within a single VLM input image.
+- Drastic reduction of API calls and token count by consolidating $N$ frames into single composite tensors/images.
+
+---
+
+## 4. Coding & Quality Standards
+
+- **Python Version Support:** `>=3.9`
+- **Linting & Formatting:** `ruff` rules must pass cleanly.
+- **Data Validation:** All user-facing configuration, thresholds, and outputs must be validated using `pydantic.BaseModel`.
+- **Testing:** Unit tests under `tests/` verifying mathematical correctness, threshold edge cases, memory cleanup, and throughput benchmarking.

vlm_sieve-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    name: Test (Python ${{ matrix.python-version }} on ${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run Ruff linter and formatter check
+        run: |
+          ruff check src/ tests/ scripts/
+          ruff format --check src/ tests/ scripts/
+
+      - name: Run Pytest test suite
+        run: |
+          pytest -v

vlm_sieve-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,41 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  build-and-publish:
+    name: Build distribution and publish to PyPI
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/vlm-sieve
+    permissions:
+      id-token: write  # Mandatory for PyPI Trusted Publishing
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build and packaging tools
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+
+      - name: Build sdist and wheel
+        run: python -m build
+
+      - name: Check package distribution artifacts
+        run: twine check dist/*
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

vlm_sieve-0.1.0/.gitignore

@@ -0,0 +1,45 @@
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Build, distribution, and packaging artifacts
+dist/
+build/
+*.egg-info/
+*.whl
+*.tar.gz
+
+# Python cache & test artifacts
+__pycache__/
+*.py[cod]
+*$py.class
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Local IDE / Agent workspace state (keep .agent/rules.md tracked or untracked as preferred, ignore temp logs)
+.vscode/
+.idea/
+
+# Personal webcam & CLI output directories (STRICT PRIVACY LOCK)
+live_webcam_test/
+webcam_verified_v2/
+*webcam*/
+sieve_out/
+*test_out*/
+preview.jpg
+summary.json
+
+# Block all stray images and videos everywhere by default
+*.jpg
+*.jpeg
+*.png
+*.mp4
+*.avi
+*.mov
+
+# Explicitly whitelist ONLY the synthetic robotics showcase image
+!assets/
+!assets/demo_storyboard.jpg

vlm_sieve-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 vlm-sieve contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vlm_sieve-0.1.0/PKG-INFO

@@ -0,0 +1,264 @@
+Metadata-Version: 2.5
+Name: vlm-sieve
+Version: 0.1.0
+Summary: Event-driven video frame gatekeeper, active ROI cropper, and temporal storyboard packer
+Project-URL: Homepage, https://github.com/Virajcreates/vlm-sieve
+Project-URL: Repository, https://github.com/Virajcreates/vlm-sieve
+Project-URL: Issues, https://github.com/Virajcreates/vlm-sieve/issues
+Author: vlm-sieve contributors
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: opencv-python-headless>=4.8.0
+Requires-Dist: pillow>=10.0.0
+Requires-Dist: pydantic>=2.5.0
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.6.0; extra == 'dev'
+Requires-Dist: twine>=5.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# vlm-sieve
+
+[![PyPI version](https://img.shields.io/pypi/v/vlm-sieve.svg)](https://pypi.org/project/vlm-sieve/)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Zero-GPU CPU Engine](https://img.shields.io/badge/CPU%20Engine-%3C2ms%2Fframe-purple.svg)](#performance--benchmarks)
+
+> **Event-Driven Video Frame Gatekeeper, Active ROI Cropper & Temporal Storyboard Packer for Vision-Language Models.**
+
+---
+
+## Overview
+
+Sending raw 30fps or 60fps video directly to frontier Vision-Language Models (such as **Google Gemini**, **OpenAI GPT-4o**, or **Anthropic Claude 3.5**) quickly becomes cost-prohibitive, introduces severe latency, and floods context windows with static background tokens.
+
+`vlm-sieve` (`vlmsieve`) is an ultra-fast, CPU-only intelligent pre-filter that monitors live webcams, RTSP security feeds, or recorded video files. It discards redundant or idle frames, tracks active motion regions, and packs salient temporal moments into structured, timestamped, and spatially grounded **storyboard image grids**—reducing VLM token consumption by **>95%** while boosting temporal and spatial comprehension.
+
+---
+
+## Visual Showcase
+
+![vlm-sieve Structured Storyboard Demo](https://raw.githubusercontent.com/Virajcreates/vlm-sieve/main/assets/demo_storyboard.jpg)
+
+In the example above, `vlm-sieve` monitored a 150-frame ($5.0\text{s}$) simulated surveillance feed. Instead of transmitting 150 separate images ($168,000$ naive tokens), `vlm-sieve` emitted a **single $2 \times 3$ storyboard** ($1,120$ tokens)—achieving **99.33% token savings (150x compression)** while providing:
+- **Chronological Indexing:** Each tile contains a high-contrast badge (`#1 | +0.00s` to `#6 | +2.53s`) specifying exact relative offsets.
+- **$4 \times 4$ Spatial Coordinate Grounding:** Tiles feature cell coordinate labels (`A1`–`D4`) allowing VLMs to reason about exact spatial trajectories (e.g., *"vehicle entered at B1, traversed C2-C3, and parked at D4"*).
+- **Active ROI Zooming:** Automatically crops out uninformative backgrounds without losing context.
+
+---
+
+## The 4-Stage Architecture Funnel
+
+```
+Raw Video Stream (30/60 FPS)
+ │
+ ├──► [Stage 1: Downscaled CPU Motion Gatekeeper]
+ │     • Sub-2ms frame evaluation on CPU via 320p grayscale differencing
+ │     • Rejects camera sensor noise, exposure flickers, and static intervals
+ │     • Automated webcam sensor warmup & 1-frame settling tail capture
+ │
+ ├──► [Stage 2: Temporal-Peak Keyframe Sampler]
+ │     • Pins event onset (first motion) and termination (settled action)
+ │     • Stratified bucket sampling combining motion energy (60%) + visual diversity (40%)
+ │
+ ├──► [Stage 3: Active ROI Auto-Cropper]
+ │     • Computes union bounding box over entire motion trajectory
+ │     • Applies contextual padding with edge-aware minimum ROI size clamping
+ │
+ └──► [Stage 4: Structured Storyboard Packer]
+       • Assembles RxC composite image grid with uniform gutters
+       • Burns chronological relative timestamps and optional A1-D4 spatial overlays
+       • Outputs ready-to-prompt PIL.Image, raw bytes, or base64 data URIs
+```
+
+---
+
+## Installation
+
+```bash
+pip install vlm-sieve
+```
+
+### Core Dependency Guardrails
+`vlm-sieve` is engineered for ultra-lean edge and cloud deployment. Core dependencies are strictly limited to `numpy`, `opencv-python-headless`, `Pillow`, and `pydantic`. **Zero heavy deep-learning frameworks (PyTorch, TensorFlow, Transformers) are installed or required.**
+
+---
+
+## Quickstart Code Examples
+
+### 1. Basic 5-Line Python SDK
+
+```python
+from vlmsieve import VideoSieve
+
+# Works with local video files, RTSP/HTTP streams, or webcam index (0)
+sieve = VideoSieve("warehouse_cam.mp4", grid_layout=(2, 3), spatial_grid_overlay=True)
+
+for event in sieve.stream_events():
+    print(f"Captured Event #{event.event_id} ({event.raw_frame_count} frames)")
+    # Save storyboard directly to disk or send to VLM
+    event.save(f"event_{event.event_id}.jpg")
+
+report = sieve.get_savings_report()
+print(f"Tokens saved: {report.tokens_saved_pct:.1f}% ({report.compression_ratio:.1f}x compression)")
+```
+
+---
+
+### 2. Google Gemini (`google-genai`) Integration
+
+Pass storyboard PIL images directly to Google Gemini 2.0 / 1.5 Flash:
+
+```python
+from google import genai
+from vlmsieve import VideoSieve
+
+client = genai.Client()
+sieve = VideoSieve("security_feed.mp4", spatial_grid_overlay=True)
+
+for event in sieve.stream_events():
+    prompt = (
+        "Analyze this chronological 2x3 video storyboard. Each tile has a relative timestamp badge "
+        "and a 4x4 spatial grid (rows A-D, columns 1-4). "
+        "Describe what occurred, noting exact grid coordinates and timestamps."
+    )
+    
+    response = client.models.generate_content(
+        model="gemini-2.0-flash",
+        contents=[event.storyboard_pil, prompt]
+    )
+    print(f"Gemini Analysis for Event #{event.event_id}:\n{response.text}\n")
+```
+
+---
+
+### 3. OpenAI (`gpt-4o` / `gpt-4o-mini`) Integration
+
+Encode the storyboard to a base64 Data URI for OpenAI's Chat Completions API:
+
+```python
+from openai import OpenAI
+from vlmsieve import VideoSieve
+
+client = OpenAI()
+sieve = VideoSieve("traffic_cam.mp4", spatial_grid_overlay=True)
+
+for event in sieve.stream_events():
+    response = client.chat.completions.create(
+        model="gpt-4o",
+        messages=[
+            {
+                "role": "user",
+                "content": [
+                    {
+                        "type": "text",
+                        "text": "Identify any safety incidents in this 6-panel storyboard.",
+                    },
+                    {
+                        "type": "image_url",
+                        "image_url": {"url": event.to_data_uri(format="JPEG", quality=88)},
+                    },
+                ],
+            }
+        ],
+    )
+    print(response.choices[0].message.content)
+```
+
+---
+
+### 4. Non-Blocking Async Streaming (FastAPI / Agent Loops)
+
+```python
+import asyncio
+from vlmsieve import VideoSieve
+
+async def live_monitoring_agent():
+    sieve = VideoSieve("rtsp://admin:pass@192.168.1.50/stream")
+    
+    # astream_events runs CPU frame gating on worker threads without blocking asyncio
+    async for event in sieve.astream_events():
+        print(f"New event #{event.event_id} captured at {event.start_time_sec:.1f}s")
+        # Trigger background webhook, alerts, or asynchronous LLM calls
+        await dispatch_vlm_task(event.to_bytes())
+
+asyncio.run(live_monitoring_agent())
+```
+
+---
+
+## Command-Line Interface (CLI)
+
+`vlm-sieve` ships with a command-line tool `vlmsieve`:
+
+```bash
+# Process video into 2x3 storyboards with spatial coordinate grid
+vlmsieve warehouse.mp4 --output-dir ./events --grid 2x3 --spatial-grid
+
+# Inspect live webcam feed (index 0) with higher sensitivity threshold
+vlmsieve 0 --threshold 0.01 --max-frames 300 -o ./webcam_events
+
+# Process RTSP feed with full frames (disable active ROI cropping)
+vlmsieve "rtsp://camera.local/live" --no-crop -o ./rtsp_events
+```
+
+### Sample CLI Output
+```
+============================================================
+                vlm-sieve Processing Summary
+============================================================
+ Raw Frames Inspected  : 1,800
+ Storyboards Emitted   : 4
+ Frames Dropped        : 1,776
+ Naive Token Estimate  : 2,016,000
+ Sieve Token Estimate  : 4,480
+ Compression Ratio     : 450.0x
+ Token Savings         : 99.78%
+============================================================
+ Output Directory      : C:\Users\user\Documents\events
+ Total Events Saved    : 4
+============================================================
+```
+
+---
+
+## `SieveConfig` Reference Guide
+
+Every hyperparameter can be fine-tuned via `SieveConfig`:
+
+| Parameter | Type | Default | Description |
+|:---|:---:|:---:|:---|
+| `downscale_width` | `int` | `320` | Downscaled frame width for motion evaluation. Guarantees sub-2ms CPU gating latency. |
+| `motion_threshold` | `float` | `0.015` | Minimum fraction of moving pixels to trigger an active motion state. |
+| `pixel_diff_threshold` | `int` | `25` | Pixel intensity delta threshold in $[0, 255]$ for frame differencing. |
+| `min_contour_area` | `int` | `16` | Minimum contour area in downscaled pixels required to eliminate sensor noise. |
+| `cooldown_frames` | `int` | `12` | Static frames required after motion ceases before closing an event. |
+| `min_event_frames` | `int` | `3` | Minimum motion frames required to emit an event (rejects single-frame flicker). |
+| `max_event_frames` | `int` | `150` | Maximum frames in a single event buffer before flushing to prevent unbounded latency. |
+| `grid_layout` | `tuple[int, int]` | `(2, 3)` | Storyboard grid layout specified as `(rows, cols)` (e.g., `(2, 3)` or `(3, 3)`). |
+| `auto_crop_roi` | `bool` | `True` | Automatically detect and crop the active motion region across the action trajectory. |
+| `roi_padding_ratio` | `float` | `0.15` | Symmetrical padding ratio added around the motion bounding box. |
+| `min_roi_size` | `int` | `224` | Minimum cropped ROI width and height, preserving framing for small objects. |
+| `burn_timestamps` | `bool` | `True` | Burn relative elapsed timestamp badges (`#1 \| +0.00s`) onto each tile. |
+| `spatial_grid_overlay` | `bool` | `False` | Render a $4 \times 4$ spatial grid (`A1`–`D4`) across each tile for coordinate grounding. |
+| `tile_max_width` | `int` | `640` | Maximum pixel width per tile before proportional Lanczos downscaling. |
+| `tokens_per_image` | `int` | `1120` | Standard VLM image tile token cost used to compute savings reports. |
+| `camera_warmup_frames` | `int` | `8` | Number of initial frames discarded when opening live webcams for exposure stabilization. |
+
+---
+
+## Performance & Benchmarks
+
+- **CPU Frame Gating Latency:** `< 2.5ms` per 1080p frame and `< 1.2ms` per 720p frame on single-core standard x86_64 CPUs.
+- **Zero GPU Memory:** All downscaling, differencing, and morphological operations execute within CPU L2/L3 cache without requiring CUDA or GPU buffers.
+- **Token Compression:** Typically compresses continuous video by **95% to 99%**, translating directly to a 20x to 100x reduction in API costs.
+
+---
+
+## License
+
+This project is licensed under the terms of the [MIT License](LICENSE).