napsack 0.1.0__tar.gz

napsack-0.1.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: napsack
+Version: 0.1.0
+Summary: NAPsack records and aggregates your computer use — screenshots plus input events (click, keypress, scroll, cursor move). It groups activity into event bursts and uses a VLM pipeline to generate human-readable captions describing what happened.
+Requires-Python: <=3.13,>=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: mss<11.0.0,>=10.0.0
+Requires-Dist: numpy==2.2
+Requires-Dist: opencv-python<5.0.0.0,>=4.11.0.86
+Requires-Dist: pandas<3.0.0,>=2.3.0
+Requires-Dist: matplotlib<4.0.0,>=3.10.3
+Requires-Dist: scikit-learn<2.0.0,>=1.7.0
+Requires-Dist: ruptures<2.0.0,>=1.1.9
+Requires-Dist: ipdb<0.14.0,>=0.13.13
+Requires-Dist: plotly<7.0.0,>=6.2.0
+Requires-Dist: nbformat<6.0.0,>=5.10.4
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: google-generativeai>=0.8.5
+Requires-Dist: datasets<4.0.0,>=3.0.0
+Requires-Dist: pillow>=11.3.0
+Requires-Dist: imageio>=2.37.0
+Requires-Dist: screeninfo>=0.8.1
+Requires-Dist: pynput>=1.8.1
+Requires-Dist: scikit-image>=0.25.2
+Requires-Dist: google-genai>=1.45.0
+Requires-Dist: openai>=2.8.1
+Requires-Dist: google-cloud-storage>=3.6.0
+Requires-Dist: google-cloud-bigquery>=3.38.0
+
+# NAPsack
+
+**NAPsack** records and structures your computer use by generating natural language caption from screenshots and input events (click, keypress, scroll, cursor move).
+
+<img alt="napsack_overview" src="https://github.com/user-attachments/assets/dd9ca2c5-288c-4977-8dc9-10ca343e56db" />
+
+---
+# Quickstart
+
+> Requires Python 3.11+ and `ffmpeg` for video generation. Use `uv` to run the commands below.
+
+## API Keys
+
+NAPsack uses a VLM to generate captions. Create a `.env` file in the project root (or export variables in your shell):
+
+```shell
+cp .env.example .env
+```
+
+Then fill in the key for your chosen client:
+
+| Client | Variable | Where to get it |
+|--------|----------|-----------------|
+| `gemini` (default) | `GEMINI_API_KEY` | [Google AI Studio](https://aistudio.google.com/apikey) |
+| `vllm` | _(none — pass `--vllm-url`)_ | Self-hosted vLLM server |
+| `bigquery` | _(uses Application Default Credentials)_ | `gcloud auth application-default login` |
+
+For Gemini, your `.env` should contain:
+
+```
+GEMINI_API_KEY=your_key_here
+```
+
+**Record** a session (press CTRL+C to stop)
+```shell
+uv run -m record --monitor
+```
+**Label** the recorded session
+```shell
+uv run -m label --session logs/session_name --client gemini
+```
+
+> NAPsack supports `gemini` and `vllm` for data labeling and integrates with `big query`
+
+# Output
+
+```shell
+logs/session_name
+├── screenshots         # Recorded screenshots
+├── aggregations.jsonl  # Recorded event bursts
+├── captions.jsonl	    # All VLM-generated captions
+├── annotated.mp4       # Final video showing generated captions and input events
+└── data.jsonl          # Final data containing raw input events and LLM generated captions
+```
+
+# Method
+
+## Record
+
+NAPsack groups temporally adjacent input events of the same type into **event bursts**. An event is assigned to the current burst if the time since the preceding event of that type does not exceed the corresponding **gap** threshold and the elapsed time since the burst start remains within the **max** duration.
+* If the **gap** threshold is exceeded, a new burst is started.
+* If the **max** duration is exceeded, the first half of the current burst is finalized and saved, while the second half becomes the active burst.
+A burst is force-restarted when the active monitor changes.
+
+## Label
+
+The `label` module:
+
+* Loads sessions or raw screenshots and chunks.
+* Uses prompts (in `label/prompts`) to instruct the VLM to generate captions that describe the user's actions and context.
+* Produces `captions.jsonl` and `data.jsonl` (captions aligned to screenshots and events).
+* Optionally renders an annotated video (`annotated.mp4`) showing captions and event visualizations overlayed on frames.
+
+The label step performs a second layer of aggregation: it uses the bursts detected at recording time and further refines and annotates them with VLM outputs to create final human-readable summaries.
+
+

napsack-0.1.0/README.md

@@ -0,0 +1,76 @@
+# NAPsack
+
+**NAPsack** records and structures your computer use by generating natural language caption from screenshots and input events (click, keypress, scroll, cursor move).
+
+<img alt="napsack_overview" src="https://github.com/user-attachments/assets/dd9ca2c5-288c-4977-8dc9-10ca343e56db" />
+
+---
+# Quickstart
+
+> Requires Python 3.11+ and `ffmpeg` for video generation. Use `uv` to run the commands below.
+
+## API Keys
+
+NAPsack uses a VLM to generate captions. Create a `.env` file in the project root (or export variables in your shell):
+
+```shell
+cp .env.example .env
+```
+
+Then fill in the key for your chosen client:
+
+| Client | Variable | Where to get it |
+|--------|----------|-----------------|
+| `gemini` (default) | `GEMINI_API_KEY` | [Google AI Studio](https://aistudio.google.com/apikey) |
+| `vllm` | _(none — pass `--vllm-url`)_ | Self-hosted vLLM server |
+| `bigquery` | _(uses Application Default Credentials)_ | `gcloud auth application-default login` |
+
+For Gemini, your `.env` should contain:
+
+```
+GEMINI_API_KEY=your_key_here
+```
+
+**Record** a session (press CTRL+C to stop)
+```shell
+uv run -m record --monitor
+```
+**Label** the recorded session
+```shell
+uv run -m label --session logs/session_name --client gemini
+```
+
+> NAPsack supports `gemini` and `vllm` for data labeling and integrates with `big query`
+
+# Output
+
+```shell
+logs/session_name
+├── screenshots         # Recorded screenshots
+├── aggregations.jsonl  # Recorded event bursts
+├── captions.jsonl	    # All VLM-generated captions
+├── annotated.mp4       # Final video showing generated captions and input events
+└── data.jsonl          # Final data containing raw input events and LLM generated captions
+```
+
+# Method
+
+## Record
+
+NAPsack groups temporally adjacent input events of the same type into **event bursts**. An event is assigned to the current burst if the time since the preceding event of that type does not exceed the corresponding **gap** threshold and the elapsed time since the burst start remains within the **max** duration.
+* If the **gap** threshold is exceeded, a new burst is started.
+* If the **max** duration is exceeded, the first half of the current burst is finalized and saved, while the second half becomes the active burst.
+A burst is force-restarted when the active monitor changes.
+
+## Label
+
+The `label` module:
+
+* Loads sessions or raw screenshots and chunks.
+* Uses prompts (in `label/prompts`) to instruct the VLM to generate captions that describe the user's actions and context.
+* Produces `captions.jsonl` and `data.jsonl` (captions aligned to screenshots and events).
+* Optionally renders an annotated video (`annotated.mp4`) showing captions and event visualizations overlayed on frames.
+
+The label step performs a second layer of aggregation: it uses the bursts detected at recording time and further refines and annotates them with VLM outputs to create final human-readable summaries.
+
+

napsack-0.1.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "napsack"
+version = "0.1.0"
+readme = "README.md"
+description = "NAPsack records and aggregates your computer use — screenshots plus input events (click, keypress, scroll, cursor move). It groups activity into event bursts and uses a VLM pipeline to generate human-readable captions describing what happened."
+requires-python = ">=3.11,<=3.13"
+dependencies = [
+    "mss (>=10.0.0,<11.0.0)",
+    "numpy==2.2",
+    "opencv-python (>=4.11.0.86,<5.0.0.0)",
+    "pandas (>=2.3.0,<3.0.0)",
+    "matplotlib (>=3.10.3,<4.0.0)",
+    "scikit-learn (>=1.7.0,<2.0.0)",
+    "ruptures (>=1.1.9,<2.0.0)",
+    "ipdb>=0.13.13,<0.14.0",
+    "plotly (>=6.2.0,<7.0.0)",
+    "nbformat (>=5.10.4,<6.0.0)",
+    "python-dotenv>=1.0.0",
+    "google-generativeai>=0.8.5",
+    "datasets>=3.0.0,<4.0.0",
+    "pillow>=11.3.0",
+    "imageio>=2.37.0",
+    "screeninfo>=0.8.1",
+    "pynput>=1.8.1",
+    "scikit-image>=0.25.2",
+    "google-genai>=1.45.0",
+    "openai>=2.8.1",
+    "google-cloud-storage>=3.6.0",
+    "google-cloud-bigquery>=3.38.0",
+]
+
+[tool.uv]
+package = true
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+include-package-data = true
+
+[tool.setuptools.package-data]
+label = ["prompts/*.txt"]
+
+[project.scripts]
+napsack-record = "record.__main__:main"
+napsack-label = "label.__main__:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["label*", "record*", "napsack*"]

napsack-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

napsack-0.1.0/src/label/__main__.py

@@ -0,0 +1,210 @@
+from pathlib import Path
+import argparse
+from dotenv import load_dotenv
+
+from label.discovery import discover_sessions, discover_screenshots_sessions, create_single_config
+from label.clients import create_client
+from label.processor import Processor
+from label.visualizer import Visualizer
+
+load_dotenv()
+
+
+def parse_args():
+    p = argparse.ArgumentParser(description="Process session recordings with VLM")
+
+    session_group = p.add_mutually_exclusive_group(required=True)
+    session_group.add_argument("--session", type=Path)
+    session_group.add_argument("--sessions-root", type=Path)
+
+    p.add_argument("--chunk-duration", type=int, default=60, help="Chunk duration in seconds")
+    p.add_argument("--fps", type=int, default=1, help="Frames per second for video processing")
+
+    p.add_argument("--screenshots-only", action="store_true", help="Process screenshots folder only without aggregations or annotations")
+    p.add_argument("--image-extensions", nargs="+", default=[".jpg", ".jpeg", ".png"], help="Image file extensions to consider")
+    p.add_argument("--max-time-gap", type=float, default=300.0, help="Maximum time gap (seconds) between images before forcing a video split (default: 120 = 2 minutes)")
+    p.add_argument("--prompt-file", default=None, help="Path to prompt file (default: prompts/default.txt or prompts/screenshots_only.txt if screenshots only)")
+    p.add_argument("--hash-cache", type=str, default=None, help="Path to hash_cache.json for deduplicating consecutive similar images")
+    p.add_argument("--dedupe-threshold", type=int, default=1, help="Hamming distance threshold for deduplication (drop if <= threshold, default: 1)")
+    p.add_argument("--annotate", action="store_true", help="Annotate videos with cursor positions and clicks (only for standard processing)")
+    p.add_argument("--skip-existing", action="store_true", help="Skip sessions that have already been processed")
+    p.add_argument("--visualize", action="store_true", help="Create annotated video visualizations after processing")
+    p.add_argument("--encode-only", action="store_true", help="Only encode videos (create chunks), skip labeling. Useful for pre-processing before running the full pipeline.")
+
+    p.add_argument("--client", choices=["gemini", "vllm", "bigquery"], default="gemini")
+    p.add_argument("--model", default="")
+    p.add_argument("--encode-workers", type=int, default=8, help="Number of parallel workers for video encoding")
+    p.add_argument("--label-workers", type=int, default=4, help="Number of parallel workers for VLM labeling")
+
+    vllm_group = p.add_argument_group("vLLM Options")
+    vllm_group.add_argument("--vllm-url")
+
+    bq_group = p.add_argument_group("BigQuery Options")
+    bq_group.add_argument("--bq-project", help="GCP project ID for AI Platform endpoint")
+    bq_group.add_argument("--bq-bucket-name", help="GCS bucket name for uploading videos")
+    bq_group.add_argument("--bq-gcs-prefix", default="video_chunks", help="Prefix/folder path in GCS bucket")
+    bq_group.add_argument("--bq-object-table-location", default="us", help="Object table location (e.g., 'us' or 'us.screenomics-gemini')")
+
+    args = p.parse_args()
+
+    if not args.model:
+        if args.client == 'gemini':
+            args.model = 'gemini-3-flash-preview'
+        elif args.client == 'vllm':
+            args.model = 'Qwen/Qwen3-VL-8B-Thinking-FP8'
+        elif args.client == 'bigquery':
+            args.model = 'dataset.model'  # Placeholder - user must provide full model reference
+    if not args.prompt_file:
+        args.prompt_file = "prompts/screenshots_only.txt" if args.screenshots_only else "prompts/default.txt"
+
+    return args
+
+
+def setup_configs(args):
+    if args.session:
+        configs = [create_single_config(
+            args.session,
+            args.chunk_duration,
+            args.screenshots_only,
+            tuple(args.image_extensions),
+        )]
+    else:
+        if args.screenshots_only:
+            configs = discover_screenshots_sessions(
+                args.sessions_root,
+                args.chunk_duration,
+                tuple(args.image_extensions),
+            )
+        else:
+            configs = discover_sessions(
+                args.sessions_root,
+                args.chunk_duration,
+                args.skip_existing,
+            )
+
+        if not configs:
+            print(f"No sessions found in {args.sessions_root}")
+            return []
+
+    return configs
+
+
+def process_with_gemini(args, configs):
+    client = create_client(
+        'gemini',
+        model_name=args.model,
+    )
+
+    processor = Processor(
+        client=client,
+        encode_workers=args.encode_workers,
+        label_workers=args.label_workers,
+        screenshots_only=args.screenshots_only,
+        prompt_file=args.prompt_file,
+        max_time_gap=args.max_time_gap,
+        hash_cache_path=args.hash_cache,
+        dedupe_threshold=args.dedupe_threshold,
+    )
+
+    return processor.process_sessions(
+        configs,
+        fps=args.fps,
+        annotate=args.annotate and not args.screenshots_only,
+        encode_only=args.encode_only,
+    )
+
+
+def process_with_vllm(args, configs):
+    client = create_client(
+        'vllm',
+        base_url=args.vllm_url if args.vllm_url.endswith('/v1') else f"{args.vllm_url}/v1",
+        model_name=args.model
+    )
+
+    processor = Processor(
+        client=client,
+        encode_workers=args.encode_workers,
+        label_workers=args.label_workers,
+        screenshots_only=args.screenshots_only,
+        prompt_file=args.prompt_file,
+        max_time_gap=args.max_time_gap,
+        hash_cache_path=args.hash_cache,
+        dedupe_threshold=args.dedupe_threshold,
+    )
+
+    return processor.process_sessions(
+        configs,
+        fps=args.fps,
+        annotate=args.annotate and not args.screenshots_only,
+        encode_only=args.encode_only,
+    )
+
+
+def process_with_bigquery(args, configs):
+    client = create_client(
+        'bigquery',
+        model_name=args.model,
+        bucket_name=args.bq_bucket_name,
+        gcs_prefix=args.bq_gcs_prefix,
+        object_table_location=args.bq_object_table_location,
+        project_id=args.bq_project,
+    )
+
+    processor = Processor(
+        client=client,
+        encode_workers=args.encode_workers,
+        label_workers=args.label_workers,
+        screenshots_only=args.screenshots_only,
+        prompt_file=args.prompt_file,
+        max_time_gap=args.max_time_gap,
+        hash_cache_path=args.hash_cache,
+        dedupe_threshold=args.dedupe_threshold,
+    )
+
+    return processor.process_sessions(
+        configs,
+        fps=args.fps,
+        annotate=args.annotate and not args.screenshots_only,
+        encode_only=args.encode_only,
+    )
+
+
+def main():
+    args = parse_args()
+
+    configs = setup_configs(args)
+    if not configs:
+        return
+
+    print(f"Processing {len(configs)} sessions")
+
+    if args.client == 'gemini':
+        results = process_with_gemini(args, configs)
+    elif args.client == 'vllm':
+        results = process_with_vllm(args, configs)
+    elif args.client == 'bigquery':
+        results = process_with_bigquery(args, configs)
+    else:
+        raise ValueError(f"Unknown client: {args.client}")
+
+    print(f"✓ Processed {len(results)} sessions")
+
+    if args.visualize:
+        print("\nCreating visualizations...")
+        visualizer = Visualizer(args.annotate)
+
+        for config in configs:
+            if not config.matched_captions_jsonl.exists():
+                print(f"Skipping Visualizing {config.session_id}: no data.jsonl")
+                continue
+
+            try:
+                output = config.session_folder / "annotated.mp4"
+                visualizer.visualize(config.session_folder, output, args.fps)
+                print(f"✓ {config.session_id}: {output}")
+            except Exception as e:
+                print(f"✗ {config.session_id}: {e}")
+
+
+if __name__ == '__main__':
+    main()

napsack-0.1.0/src/label/caption_matching.py

@@ -0,0 +1,150 @@
+from pathlib import Path
+from typing import List, Dict, Any, Optional
+import json
+
+
+def match_captions_with_events(
+    captions_path: Path,
+    aggregations_path: Path,
+    output_path: Path,
+    fps: int = 1
+) -> List[Dict[str, Any]]:
+    """
+    Match captions with aggregated events based on timestamps.
+
+    Args:
+        captions_path: Path to captions.jsonl
+        aggregations_path: Path to aggregations.jsonl
+        output_path: Path to save matched_captions.jsonl
+        fps: Frames per second used in video creation
+
+    Returns:
+        List of matched caption-event objects
+    """
+    # Load captions
+    captions = []
+    with open(captions_path, 'r', encoding='utf-8') as f:
+        for line in f:
+            if line.strip():
+                captions.append(json.loads(line))
+
+    # Load aggregations
+    aggregations = []
+    with open(aggregations_path, 'r', encoding='utf-8') as f:
+        for line in f:
+            if line.strip():
+                agg = json.loads(line)
+                aggregations.append(agg)
+
+    # Sort aggregations by timestamp
+    aggregations.sort(key=lambda x: x.get('timestamp', 0))
+
+    if not aggregations:
+        print("[Matcher] Warning: No aggregations found")
+        return []
+
+    # Get first aggregation timestamp (video start time)
+    first_timestamp = aggregations[0].get('timestamp', 0)
+
+    print(f"[Matcher] Video start time: {first_timestamp}")
+    print(f"[Matcher] Total aggregations: {len(aggregations)}")
+    print(f"[Matcher] FPS: {fps}")
+
+    # Match captions with events
+    matched_data = []
+
+    for caption in captions:
+        # Convert MM:SS to seconds
+        start_seconds = caption['start_seconds']
+        end_seconds = caption['end_seconds']
+
+        # Convert video time to aggregation indices
+        # Each aggregation represents 1 frame, so index = seconds * fps
+        start_index = int(start_seconds * fps)
+        end_index = int(end_seconds * fps)
+
+        # Clamp to valid range
+        start_index = max(0, min(start_index, len(aggregations) - 1))
+        end_index = max(start_index, min(end_index, len(aggregations) - 1))
+
+        print(f"[Matcher] Caption '{caption['caption'][:50]}...' -> indices [{start_index}, {end_index}]")
+
+        # Get aggregations in this range
+        matched_aggs = aggregations[start_index:end_index + 1]
+
+        if not matched_aggs:
+            # No events matched, but still save the caption
+            matched_entry = {
+                "start_time": first_timestamp + start_seconds,
+                "end_time": first_timestamp + end_seconds,
+                "start_index": start_index,
+                "end_index": end_index,
+                "img": None,
+                "caption": caption['caption'],
+                "raw_events": [],
+                "num_aggregations": 0,
+                "start_formatted": caption['start'],
+                "end_formatted": caption['end'],
+            }
+        else:
+            # Get first and last aggregation for time and image
+            first_agg = matched_aggs[0]
+            last_agg = matched_aggs[-1]
+
+            # Concatenate all events from matched aggregations
+            all_events = []
+            for agg in matched_aggs:
+                events = agg.get('events', [])
+                all_events.extend(events)
+
+            matched_entry = {
+                "start_time": first_agg.get('timestamp'),
+                "end_time": last_agg.get('timestamp'),
+                "start_index": start_index,
+                "end_index": end_index,
+                "img": first_agg.get('screenshot_path'),
+                "caption": caption['caption'],
+                "raw_events": all_events,
+                "num_aggregations": len(matched_aggs),
+                "start_formatted": caption['start'],
+                "end_formatted": caption['end'],
+            }
+
+        matched_data.append(matched_entry)
+
+    # Save matched data
+    with open(output_path, 'w', encoding='utf-8') as f:
+        for entry in matched_data:
+            f.write(json.dumps(entry, ensure_ascii=False) + '\n')
+
+    print(f"[Matcher] Saved {len(matched_data)} matched entries to {output_path}")
+
+    return matched_data
+
+
+def create_matched_captions_for_session(session_dir: Path, fps: int = 1) -> Optional[Path]:
+    """
+    Create matched_captions.jsonl for a session directory.
+
+    Args:
+        session_dir: Path to session directory
+        fps: Frames per second used in video creation
+
+    Returns:
+        Path to created matched_captions.jsonl or None if failed
+    """
+    captions_path = session_dir / "captions.jsonl"
+    aggregations_path = session_dir / "aggregations.jsonl"
+    output_path = session_dir / "matched_captions.jsonl"
+
+    if not captions_path.exists():
+        print(f"[Matcher] Warning: {captions_path} not found")
+        return None
+
+    if not aggregations_path.exists():
+        print(f"[Matcher] Warning: {aggregations_path} not found")
+        return None
+
+    match_captions_with_events(captions_path, aggregations_path, output_path, fps)
+
+    return output_path

napsack-0.1.0/src/label/clients/__init__.py

@@ -0,0 +1,28 @@
+from label.clients.client import VLMClient, CAPTION_SCHEMA
+from label.clients.gemini import GeminiClient, GeminiResponse
+from label.clients.vllm import VLLMClient, VLLMResponse
+from label.clients.bigquery import BigQueryClient, BigQueryResponse
+
+
+def create_client(client_type: str, **kwargs) -> VLMClient:
+    if client_type == 'gemini':
+        return GeminiClient(**kwargs)
+    elif client_type == 'vllm':
+        return VLLMClient(**kwargs)
+    elif client_type == 'bigquery':
+        return BigQueryClient(**kwargs)
+    else:
+        raise ValueError(f"Unknown client type: {client_type}")
+
+
+__all__ = [
+    "VLMClient",
+    "GeminiClient",
+    "GeminiResponse",
+    "VLLMClient",
+    "VLLMResponse",
+    "BigQueryClient",
+    "BigQueryResponse",
+    "CAPTION_SCHEMA",
+    "create_client",
+]