tokeye 0.9.0__tar.gz

tokeye-0.9.0/PKG-INFO

@@ -0,0 +1,138 @@
+Metadata-Version: 2.3
+Name: tokeye
+Version: 0.9.0
+Summary: Add your description here
+Requires-Dist: gradio
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: torch
+Requires-Dist: torchcodec
+Requires-Dist: torchvision
+Requires-Dist: torchinfo
+Requires-Dist: omegaconf
+Requires-Dist: tables
+Requires-Dist: pydantic
+Requires-Dist: mypy ; extra == 'dev'
+Requires-Dist: pytest ; extra == 'dev'
+Requires-Dist: pytest-cov ; extra == 'dev'
+Requires-Dist: pytest-xdist ; extra == 'dev'
+Requires-Dist: ruff ; extra == 'dev'
+Requires-Dist: ipykernel ; extra == 'dev'
+Requires-Dist: ipywidgets ; extra == 'dev'
+Requires-Dist: matplotlib ; extra == 'dev'
+Requires-Dist: wavio ; extra == 'dev'
+Requires-Dist: h5py ; extra == 'dev'
+Requires-Dist: fusionaihub ; extra == 'dev'
+Requires-Dist: scikit-learn ; extra == 'dev'
+Requires-Dist: lightning ; extra == 'train'
+Requires-Python: >=3.13
+Provides-Extra: dev
+Provides-Extra: train
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="assets/logo.png" alt="TokEye Logo" width="400">
+</p>
+
+# TokEye
+
+[![Python package](https://github.com/PlasmaControl/tokeye/actions/workflows/python-package.yml/badge.svg)](https://github.com/PlasmaControl/tokeye/actions/workflows/python-package.yml)
+
+TokEye is a open-source Python-based application for automatic classification and localization of fluctuating signals.
+It is designed to be used in the context of plasma physics, but can be used for any type of fluctuating signal.
+
+Check out [this poster from APS DPP 2025](assets/aps_dpp_2025.pdf) for more information.
+
+## Example Demonstration
+![Example Demonstration](assets/example.gif)
+
+Expected processing time:
+- A100: < 0.5 seconds on any size spectrogram after warmup.
+- CPU: not yet tested.
+
+## Verified Datatypes
+- DIII-D Fast Magnetics (cite)
+- DIII-D CO2 Interferometer (cite)
+- DIII-D Electron Cyclotron Emission (cite)
+- DIII-D Beam Emission Spectroscopy (cite)
+
+## Evaluation
+Recall Scores:
+- TJII2021: 0.8254
+- DCLDE2011 (Delphinus capensis): 0.7708
+- DCLDE2011 (Delphinus delphis): 0.7953
+
+With more data, comes better models. Please contribute to the project!
+
+## Installation
+
+[uv](https://docs.astral.sh/uv/) (recommended)
+```bash
+git clone git@github.com:PlasmaControl/TokEye.git
+cd TokEye
+uv sync
+```
+
+pip (from source)
+```bash
+git clone git@github.com:PlasmaControl/TokEye.git
+cd TokEye
+python3 -m venv .venv
+source venv/bin/activate
+pip install uv
+uv sync
+```
+
+pip (from PyPI)
+```bash
+pip install tokeye
+```
+Coming soon.
+
+Containerized installation (Docker)
+Coming soon.
+
+
+## Usage
+```bash
+python -m TokEye.app
+```
+
+This will start a web app on `http://localhost:8888`.
+
+If you are on a remote server, you can use SSH port forwarding to access the web app on your local machine:
+```bash
+ssh -L 8888:localhost:7860 user@remote_server
+```
+Then open your web browser and navigate to `http://localhost:8888`.
+
+## Models
+Pre-trained models are available at [this link](https://drive.google.com/drive/folders/1rXllPXB3eWhMvSIlp0CDSFx68lJOQG1u?usp=drive_link).
+Copy them into the `models/` directory after downloading them.
+- big_mode_v1.pt: Original training regime (window = 1024, hop = 128)
+- big_mode_v2.pt: Trained on multiscale (multiwindow, multihop) spectrograms
+
+Input should be a tensor that has shape (B, 1, H, W) where B, H, and W can vary
+Output will be a tensor of shape (B, 2, H, W)
+
+Best performance when spectrograms are oriented so that when they are plotted with matplotlib, the lowest frequency bin is oriented with the bottom when `origin='lower'`. Spectrograms should be standardized (mean = 0, std = 1). If baseline activity is very strong, clipping the input may help, but is generally not needed.
+
+The first channel of the output will return preferential measurements of coherent activity (useful for most tasks)
+THe second channel of the output will return preferential measurements of transient activity
+
+## Data
+Right now, keep all data as 1d numpy float arrays. No need to normalize or preprocess them.
+Copy them into the `data/` directory.
+
+## Citation
+If you use this code in your research, please cite:
+```bibtex
+@article{NaN,
+  title={Paper not yet published},
+  author={Nathaniel Chen},
+  year={2025}
+}
+```
+
+## Contact
+Please check back for updates or reach out to Nathaniel Chen at nathaniel@princeton.edu.

tokeye-0.9.0/README.md

@@ -0,0 +1,106 @@
+<p align="center">
+  <img src="assets/logo.png" alt="TokEye Logo" width="400">
+</p>
+
+# TokEye
+
+[![Python package](https://github.com/PlasmaControl/tokeye/actions/workflows/python-package.yml/badge.svg)](https://github.com/PlasmaControl/tokeye/actions/workflows/python-package.yml)
+
+TokEye is a open-source Python-based application for automatic classification and localization of fluctuating signals.
+It is designed to be used in the context of plasma physics, but can be used for any type of fluctuating signal.
+
+Check out [this poster from APS DPP 2025](assets/aps_dpp_2025.pdf) for more information.
+
+## Example Demonstration
+![Example Demonstration](assets/example.gif)
+
+Expected processing time:
+- A100: < 0.5 seconds on any size spectrogram after warmup.
+- CPU: not yet tested.
+
+## Verified Datatypes
+- DIII-D Fast Magnetics (cite)
+- DIII-D CO2 Interferometer (cite)
+- DIII-D Electron Cyclotron Emission (cite)
+- DIII-D Beam Emission Spectroscopy (cite)
+
+## Evaluation
+Recall Scores:
+- TJII2021: 0.8254
+- DCLDE2011 (Delphinus capensis): 0.7708
+- DCLDE2011 (Delphinus delphis): 0.7953
+
+With more data, comes better models. Please contribute to the project!
+
+## Installation
+
+[uv](https://docs.astral.sh/uv/) (recommended)
+```bash
+git clone git@github.com:PlasmaControl/TokEye.git
+cd TokEye
+uv sync
+```
+
+pip (from source)
+```bash
+git clone git@github.com:PlasmaControl/TokEye.git
+cd TokEye
+python3 -m venv .venv
+source venv/bin/activate
+pip install uv
+uv sync
+```
+
+pip (from PyPI)
+```bash
+pip install tokeye
+```
+Coming soon.
+
+Containerized installation (Docker)
+Coming soon.
+
+
+## Usage
+```bash
+python -m TokEye.app
+```
+
+This will start a web app on `http://localhost:8888`.
+
+If you are on a remote server, you can use SSH port forwarding to access the web app on your local machine:
+```bash
+ssh -L 8888:localhost:7860 user@remote_server
+```
+Then open your web browser and navigate to `http://localhost:8888`.
+
+## Models
+Pre-trained models are available at [this link](https://drive.google.com/drive/folders/1rXllPXB3eWhMvSIlp0CDSFx68lJOQG1u?usp=drive_link).
+Copy them into the `models/` directory after downloading them.
+- big_mode_v1.pt: Original training regime (window = 1024, hop = 128)
+- big_mode_v2.pt: Trained on multiscale (multiwindow, multihop) spectrograms
+
+Input should be a tensor that has shape (B, 1, H, W) where B, H, and W can vary
+Output will be a tensor of shape (B, 2, H, W)
+
+Best performance when spectrograms are oriented so that when they are plotted with matplotlib, the lowest frequency bin is oriented with the bottom when `origin='lower'`. Spectrograms should be standardized (mean = 0, std = 1). If baseline activity is very strong, clipping the input may help, but is generally not needed.
+
+The first channel of the output will return preferential measurements of coherent activity (useful for most tasks)
+THe second channel of the output will return preferential measurements of transient activity
+
+## Data
+Right now, keep all data as 1d numpy float arrays. No need to normalize or preprocess them.
+Copy them into the `data/` directory.
+
+## Citation
+If you use this code in your research, please cite:
+```bibtex
+@article{NaN,
+  title={Paper not yet published},
+  author={Nathaniel Chen},
+  year={2025}
+}
+```
+
+## Contact
+Please check back for updates or reach out to Nathaniel Chen at nathaniel@princeton.edu.

tokeye-0.9.0/pyproject.toml

@@ -0,0 +1,91 @@
+[build-system]
+requires = ["uv_build"]
+build-backend = "uv_build"
+
+[project]
+name = "tokeye"
+version = "0.9.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "gradio",
+
+    "numpy",
+    "scipy",
+
+    "torch",
+    "torchcodec",
+    "torchvision",
+    "torchinfo",
+
+    "omegaconf",
+    "tables",
+    "pydantic",
+]
+
+[project.optional-dependencies]
+dev = [
+    "mypy",
+    "pytest",
+    "pytest-cov",
+    "pytest-xdist",
+    "ruff",
+
+    "ipykernel",
+    "ipywidgets",
+    "matplotlib",
+    "wavio",
+    "h5py",
+    "fusionaihub",
+    "scikit-learn",
+]
+train = [
+    "lightning",
+]
+
+[tool.ruff]
+line-length = 88
+
+[tool.ruff.lint]
+extend-select = [
+    "F",        # Pyflakes rules
+    "W",        # PyCodeStyle warnings
+    "E",        # PyCodeStyle errors
+    "I",        # Sort imports properly
+    "UP",       # Warn if certain things can changed due to newer Python versions
+    "C4",       # Catch incorrect use of comprehensions, dict, list, etc
+    "FA",       # Enforce from __future__ import annotations
+    "ISC",      # Good use of string concatenation
+    "ICN",      # Use common import conventions
+    "RET",      # Good return practices
+    "SIM",      # Common simplification rules
+    "TID",      # Some good import practices
+    "TC",       # Enforce importing certain types in a TYPE_CHECKING block
+    "PTH",      # Use pathlib instead of os.path
+    "TD",       # Be diligent with TODO comments
+    "NPY",      # Some numpy-specific things
+]
+ignore = [
+    "E501",
+    "B008",
+    ]
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["F401"]
+"tests/*" = ["ARG"]
+"*.ipynb" = ["E402"]
+
+[tool.uv]
+package = true
+
+[tool.uv.build-backend]
+module-name = "tokeye"
+module-root = "src"
+
+[pytest]
+minversion = "9.0"
+addopts = ["-ra", "-q"]
+testpaths = [
+    "tests",
+]

tokeye-0.9.0/src/tokeye/analysis/__init__.py

@@ -0,0 +1,4 @@
+from .batch_analysis import default_settings, run_batch_analysis
+
+__all__ = ["run_batch_analysis", "default_settings"]
+

tokeye-0.9.0/src/tokeye/analysis/batch_analysis.py

@@ -0,0 +1,240 @@
+import logging
+import time
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.nn as nn
+
+# Import model loading function from the analyze module
+from TokEye.app.analyze.load import model_load
+from tqdm.auto import tqdm
+
+# Configure logging
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+# Default settings
+default_settings = {
+    "model_path": "data/models/big_mode_v1.pth",
+    "input_path": "data/batch_inputs",
+    "output_path": "data/batch_outputs",
+    "analysis_mode": "amplitude",
+    "threshold": 0.5,
+    "polling_interval": 5,  # seconds between directory scans
+}
+
+
+def load_spectrogram(filepath: Path) -> np.ndarray | None:
+    """Load a 2D spectrogram from a numpy file.
+
+    Args:
+        filepath: Path to the .npy file containing a 2D spectrogram
+
+    Returns:
+        2D numpy array or None if loading fails
+    """
+    try:
+        spec = np.load(filepath)
+        if spec.ndim != 2:
+            logger.error(f"Expected 2D array, got {spec.ndim}D: {filepath.name}")
+            return None
+        if spec.size == 0:
+            logger.error(f"Empty array: {filepath.name}")
+            return None
+        return spec
+    except Exception as e:
+        logger.error(f"Failed to load {filepath.name}: {e}")
+        return None
+
+
+def run_inference(
+    spec: np.ndarray,
+    model: nn.Module | torch.export.ExportedProgram,
+) -> np.ndarray | None:
+    """Run inference on a 2D spectrogram.
+
+    Args:
+        spec: 2D numpy array (H, W)
+        model: Loaded PyTorch model
+
+    Returns:
+        3D numpy array (2, H, W) with two output channels or None if inference fails
+    """
+    try:
+        device = next(model.parameters()).device
+
+        # Normalize input
+        spec_norm = (spec - spec.mean()) / (spec.std() + 1e-6)
+
+        # Convert to tensor: (1, 1, H, W)
+        inp_tensor = torch.from_numpy(spec_norm)
+        inp_tensor = inp_tensor.unsqueeze(0).unsqueeze(0).float()
+        inp_tensor = inp_tensor.to(device)
+
+        # Run inference
+        with torch.no_grad():
+            out_tensor = model(inp_tensor)
+
+        # Remove batch dimension: (1, 2, H, W) -> (2, H, W)
+        out_tensor = out_tensor[0]
+
+        # Apply sigmoid activation
+        out_tensor = torch.sigmoid(out_tensor)
+
+        # Convert to numpy
+        return out_tensor.cpu().numpy()
+
+    except Exception as e:
+        logger.error(f"Inference failed: {e}")
+        return None
+
+
+def process_file(
+    input_path: Path,
+    output_path: Path,
+    model: nn.Module | torch.export.ExportedProgram,
+) -> bool:
+    """Process a single file: load, infer, save.
+
+    Args:
+        input_path: Path to input .npy file
+        output_path: Path to output .npy file
+        model: Loaded PyTorch model
+
+    Returns:
+        True if successful, False otherwise
+    """
+    # Load spectrogram
+    spec = load_spectrogram(input_path)
+    if spec is None:
+        return False
+
+    # Run inference
+    result = run_inference(spec, model)
+    if result is None:
+        return False
+
+    # Save result
+    try:
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        np.save(output_path, result)
+        return True
+    except Exception as e:
+        logger.error(f"Failed to save {output_path.name}: {e}")
+        return False
+
+
+def scan_directory(input_dir: Path) -> set[str]:
+    """Scan directory for .npy files.
+
+    Args:
+        input_dir: Directory to scan
+
+    Returns:
+        Set of filenames (not full paths)
+    """
+    if not input_dir.exists():
+        return set()
+
+    npy_files = input_dir.glob("*.npy")
+    return {f.name for f in npy_files}
+
+
+def run_batch_analysis(settings: dict):
+    """Main batch analysis loop.
+
+    Continuously monitors input directory for new spectrogram files,
+    runs inference, and saves results. Runs until interrupted with Ctrl+C.
+
+    Args:
+        settings: Dictionary with configuration:
+            - model_path: Path to model file
+            - input_path: Input directory to monitor
+            - output_path: Output directory for results
+            - polling_interval: Seconds between directory scans
+    """
+    # Extract settings
+    model_path = Path(settings["model_path"])
+    input_dir = Path(settings["input_path"])
+    output_dir = Path(settings["output_path"])
+    polling_interval = settings.get("polling_interval", 5)
+
+    # Ensure directories exist
+    input_dir.mkdir(parents=True, exist_ok=True)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load model
+    logger.info("=" * 60)
+    logger.info("Starting Batch Analysis System")
+    logger.info("=" * 60)
+    logger.info(f"Model: {model_path}")
+    logger.info(f"Input directory: {input_dir}")
+    logger.info(f"Output directory: {output_dir}")
+    logger.info(f"Polling interval: {polling_interval}s")
+    logger.info("=" * 60)
+
+    try:
+        model = model_load(model_path)
+    except Exception as e:
+        logger.error(f"Failed to load model: {e}")
+        return
+
+    # Initialize tracking
+    processed_files = set()
+
+    logger.info("Monitoring for new files... (Press Ctrl+C to stop)")
+    logger.info("")
+
+    # Main loop
+    try:
+        while True:
+            # Scan for all files in directory
+            current_files = scan_directory(input_dir)
+
+            # Update tracking: remove deleted files
+            deleted_files = processed_files - current_files
+            if deleted_files:
+                for fname in deleted_files:
+                    processed_files.discard(fname)
+                    logger.info(f"File deleted, will reprocess if added again: {fname}")
+
+            # Find new files to process
+            new_files = current_files - processed_files
+
+            if new_files:
+                logger.info(f"Found {len(new_files)} new file(s)")
+
+                # Process files with progress bar
+                new_files_list = sorted(new_files)
+                for fname in tqdm(new_files_list, desc="Processing", unit="file"):
+                    input_path = input_dir / fname
+                    output_path = output_dir / fname
+
+                    # Process the file
+                    success = process_file(input_path, output_path, model)
+
+                    if success:
+                        processed_files.add(fname)
+                        logger.info(f"✓ Processed: {fname}")
+                    else:
+                        logger.warning(f"✗ Failed: {fname}")
+
+                logger.info("")
+
+            # Sleep before next scan
+            time.sleep(polling_interval)
+
+    except KeyboardInterrupt:
+        logger.info("")
+        logger.info("=" * 60)
+        logger.info("Shutting down gracefully...")
+        logger.info(f"Total files processed: {len(processed_files)}")
+        logger.info("=" * 60)
+
+
+if __name__ == "__main__":
+    run_batch_analysis(default_settings)

tokeye-0.9.0/src/tokeye/app/__main__.py

@@ -0,0 +1,84 @@
+"""
+TokEye Main Inference
+"""
+
+import logging
+import sys
+from pathlib import Path
+
+import gradio as gr
+
+# Import tabs
+from .analyze.analyze import analyze_tab
+from .tabs.annotate import annotate_tab
+from .tabs.utilities import utilities_tab
+from .utils.theme import make_theme
+
+# Constants
+APP_TITLE = "TokEye"
+DEFAULT_PORT = 7860
+MAX_PORT_ATTEMPTS = 10
+
+# Set up logging
+logging.getLogger("uvicorn").setLevel(logging.WARNING)
+logging.getLogger("httpx").setLevel(logging.WARNING)
+
+# Current working directory
+cwd = Path.cwd()
+
+
+def create_app() -> gr.Blocks:
+    with gr.Blocks(
+        title=APP_TITLE,
+        theme=make_theme(),
+        css="footer{display:none !important}",
+    ) as app:
+        gr.Image(
+            str(Path.cwd() / "assets" / "logo.png"),
+            height=300,
+            interactive=False,
+            container=False,
+            show_download_button=False,
+            show_fullscreen_button=False,
+        )
+        with gr.Tab("Analyze"):
+            analyze_tab()
+        with gr.Tab("Annotate"):
+            annotate_tab()
+        with gr.Tab("Utilities"):
+            utilities_tab()
+    return app
+
+
+def get_port():
+    if "--port" in sys.argv:
+        port_index = sys.argv.index("--port") + 1
+        if port_index < len(sys.argv):
+            return int(sys.argv[port_index])
+    return DEFAULT_PORT
+
+
+def launch(app, port):
+    app.launch(
+        # favicon_path="assets/ICON.ico",  # Set up favicon later
+        share="--share" in sys.argv,
+        inbrowser="--open" in sys.argv,
+        server_port=port,
+    )
+
+
+if __name__ == "__main__":
+    logging.info(f"Initializing TokEye in: {cwd}")
+    # Set up
+    app = create_app()
+    # Launch application
+    port = get_port()
+    for _ in range(MAX_PORT_ATTEMPTS):
+        try:
+            launch(app, port)
+        except OSError:
+            print(f"Failed on port {port}")
+            port -= 1
+        except Exception as error:
+            print(f"{error}")
+            break