jabs-behavior-classifier 0.46.0__tar.gz → 0.46.2__tar.gz

{jabs_behavior_classifier-0.46.0 → jabs_behavior_classifier-0.46.2}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: jabs-behavior-classifier
-Version: 0.46.0
+Version: 0.46.2
 Summary: 
 Author: Glen Beane, Brian Geuther, Keith Sheppard, Alexander Berger-Liedtka
 License-Expression: MIT
 License-File: LICENSE
-Requires-Dist: jabs-behavior==0.46.0
-Requires-Dist: jabs-core==0.46.0
-Requires-Dist: jabs-io==0.46.0
+Requires-Dist: jabs-behavior==0.46.2
+Requires-Dist: jabs-core==0.46.2
+Requires-Dist: jabs-io==0.46.2
 Requires-Dist: h5py>=3.10.0,<4.0.0
 Requires-Dist: markdown2>=2.5.1,<3.0.0
 Requires-Dist: numpy>=2.0.0,<3.0.0
@@ -20,7 +20,7 @@ Requires-Dist: shapely>=2.0.1,<3.0.0
 Requires-Dist: tabulate>=0.9.0,<1.0.0
 Requires-Dist: toml>=0.10.2,<0.11.0
 Requires-Dist: xgboost>=2.0.0,<3.0.0
-Requires-Dist: pyarrow>=20.0.0,<23.0.0
+Requires-Dist: pyarrow>=23.0.1,<24.0.0
 Requires-Dist: rich>=14.0.0,<15.0.0
 Requires-Dist: rich-argparse>=1.7.1,<2.0.0
 Requires-Dist: intervaltree>=3.1.0,<4.0.0
@@ -51,14 +51,14 @@ Description-Content-Type: text/markdown
 
 [![Python 3.10-3.14](https://img.shields.io/badge/python-3.10--3.14-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://img.shields.io/pypi/v/jabs-behavior-classifier.svg)](https://pypi.org/project/jabs-behavior-classifier/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/LICENSE)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/LICENSE)
 [![DOI](https://img.shields.io/badge/DOI-10.7554%2FeLife.107259.2-green.svg)](https://doi.org/10.7554/eLife.107259.2)
 
-[Documentation](https://jabs-tutorial.readthedocs.io/) · [User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/user-guide/overview.md) · [Sample Data](https://doi.org/10.5281/zenodo.16697331) · [Contact Us](mailto:jabs@jax.org)
+[Documentation](https://jabs-tutorial.readthedocs.io/) · [User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/user-guide/overview.md) · [Sample Data](https://doi.org/10.5281/zenodo.16697331) · [Contact Us](mailto:jabs@jax.org)
 
 ---
 
-![JABS Screenshot](https://raw.githubusercontent.com/KumarLabJax/JABS-behavior-classifier/v0.46.0/docs/assets/images/jabs_screenshot.png)
+![JABS Screenshot](https://raw.githubusercontent.com/KumarLabJax/JABS-behavior-classifier/v0.46.2/docs/assets/images/jabs_screenshot.png)
 
 </div>
 
@@ -188,7 +188,7 @@ JABS requires pose files generated from the Kumar Lab's mouse pose estimation ne
 
 We provide Singularity/Apptainer definition files and SLURM batch scripts for running
 JABS on Linux compute clusters. See
-[vm/README.md](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/vm/README.md)
+[vm/README.md](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/vm/README.md)
 for build instructions and usage details.
 
 ## Project Portability
@@ -224,14 +224,14 @@ JABS uses four version numbers to track compatibility:
 ## Documentation
 
 - **[ReadTheDocs Tutorial](https://jabs-tutorial.readthedocs.io/)** — Complete user guide and tutorials
-- **[User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/user-guide/overview.md)** — Markdown documentation
+- **[User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/user-guide/overview.md)** — Markdown documentation
 - **[Sample Data](https://doi.org/10.5281/zenodo.16697331)** — Test datasets for demonstration
 
 ## Contributing
 
 Interested in contributing? Check out our:
-- [Contributing Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/CONTRIBUTING.md)
-- [Development Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/development/development.md)
+- [Contributing Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/CONTRIBUTING.md)
+- [Development Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/development/development.md)
 
 ## Citation
 

{jabs_behavior_classifier-0.46.0 → jabs_behavior_classifier-0.46.2}/README.md

@@ -8,14 +8,14 @@
 
 [![Python 3.10-3.14](https://img.shields.io/badge/python-3.10--3.14-blue.svg)](https://www.python.org/downloads/)
 [![PyPI version](https://img.shields.io/pypi/v/jabs-behavior-classifier.svg)](https://pypi.org/project/jabs-behavior-classifier/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/LICENSE)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/LICENSE)
 [![DOI](https://img.shields.io/badge/DOI-10.7554%2FeLife.107259.2-green.svg)](https://doi.org/10.7554/eLife.107259.2)
 
-[Documentation](https://jabs-tutorial.readthedocs.io/) · [User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/user-guide/overview.md) · [Sample Data](https://doi.org/10.5281/zenodo.16697331) · [Contact Us](mailto:jabs@jax.org)
+[Documentation](https://jabs-tutorial.readthedocs.io/) · [User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/user-guide/overview.md) · [Sample Data](https://doi.org/10.5281/zenodo.16697331) · [Contact Us](mailto:jabs@jax.org)
 
 ---
 
-![JABS Screenshot](https://raw.githubusercontent.com/KumarLabJax/JABS-behavior-classifier/v0.46.0/docs/assets/images/jabs_screenshot.png)
+![JABS Screenshot](https://raw.githubusercontent.com/KumarLabJax/JABS-behavior-classifier/v0.46.2/docs/assets/images/jabs_screenshot.png)
 
 </div>
 
@@ -145,7 +145,7 @@ JABS requires pose files generated from the Kumar Lab's mouse pose estimation ne
 
 We provide Singularity/Apptainer definition files and SLURM batch scripts for running
 JABS on Linux compute clusters. See
-[vm/README.md](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/vm/README.md)
+[vm/README.md](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/vm/README.md)
 for build instructions and usage details.
 
 ## Project Portability
@@ -181,14 +181,14 @@ JABS uses four version numbers to track compatibility:
 ## Documentation
 
 - **[ReadTheDocs Tutorial](https://jabs-tutorial.readthedocs.io/)** — Complete user guide and tutorials
-- **[User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/user-guide/overview.md)** — Markdown documentation
+- **[User Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/user-guide/overview.md)** — Markdown documentation
 - **[Sample Data](https://doi.org/10.5281/zenodo.16697331)** — Test datasets for demonstration
 
 ## Contributing
 
 Interested in contributing? Check out our:
-- [Contributing Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/CONTRIBUTING.md)
-- [Development Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.0/docs/development/development.md)
+- [Contributing Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/CONTRIBUTING.md)
+- [Development Guide](https://github.com/KumarLabJax/JABS-behavior-classifier/blob/v0.46.2/docs/development/development.md)
 
 ## Citation
 

{jabs_behavior_classifier-0.46.0 → jabs_behavior_classifier-0.46.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "jabs-behavior-classifier"
-version = "0.46.0"
+version = "0.46.2"
 description = ""
 readme = "README.md"
 requires-python = ">=3.10,<3.15"
@@ -15,9 +15,9 @@ authors = [
 ]
 
 dependencies = [
-  "jabs-behavior==0.46.0",
-  "jabs-core==0.46.0",
-  "jabs-io==0.46.0",
+  "jabs-behavior==0.46.2",
+  "jabs-core==0.46.2",
+  "jabs-io==0.46.2",
   "h5py>=3.10.0,<4.0.0",
   "markdown2>=2.5.1,<3.0.0",
   "numpy>=2.0.0,<3.0.0",
@@ -30,7 +30,7 @@ dependencies = [
   "tabulate>=0.9.0,<1.0.0",
   "toml>=0.10.2,<0.11.0",
   "xgboost>=2.0.0,<3.0.0",
-  "pyarrow>=20.0.0,<23.0.0",
+  "pyarrow>=23.0.1,<24.0.0",
   "rich>=14.0.0,<15.0.0",
   "rich-argparse>=1.7.1,<2.0.0",
   "intervaltree>=3.1.0,<4.0.0",
@@ -47,6 +47,13 @@ nwb = ["jabs-io[nwb]"]
 yaml = ["pyyaml>=6.0.0"]
 mlflow = ["mlflow>=3.8.1"]
 
+[tool.uv]
+# Security floors for transitive dependencies (Dependabot). These packages are
+# pulled indirectly (e.g. cryptography via the optional `mlflow` extra ->
+# google-auth); constraining them here bumps the locked version past known
+# vulnerabilities without adding them as direct dependencies.
+constraint-dependencies = ["cryptography>=48.0.1"]
+
 [tool.uv.sources]
 jabs-behavior = { workspace = true }
 jabs-io = { workspace = true }

{jabs_behavior_classifier-0.46.0 → jabs_behavior_classifier-0.46.2}/src/jabs/classifier/mlflow_logging.py

@@ -25,6 +25,8 @@ import logging
 import math
 import os
 import subprocess
+import tempfile
+import zipfile
 from pathlib import Path
 from typing import TYPE_CHECKING
 
@@ -37,6 +39,10 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
+#: File name used for the zipped annotations artifact attached to each run. Kept
+#: constant across runs so the artifact is easy to find and compare in the UI.
+ANNOTATIONS_ARCHIVE_NAME = "annotations.zip"
+
 
 def mlflow_available() -> bool:
     """Return True if an import spec for the optional ``mlflow`` package is found.
@@ -130,6 +136,83 @@ def load_env_file(env_file: Path | None, *, override: bool = True) -> dict[str,
     return values
 
 
+def archive_annotations(annotations_dir: Path, output_path: Path) -> Path | None:
+    """Zip a JABS project's annotations directory for upload as an MLflow artifact.
+
+    Captures the label set the cross-validation run was computed from, so a run's
+    metrics can be traced back to (and reproduced from) the exact annotations.
+
+    Archive members are stored under a single top-level directory named after
+    ``annotations_dir`` (normally ``annotations/``), so unpacking the zip recreates
+    the directory rather than scattering JSON files into the current directory.
+    Files are added in sorted order to keep archives reproducible.
+
+    Symlinks are **not** archived, and neither are files reached through one. The
+    archive is uploaded to a tracking server, so it must contain only what really
+    lives in the project: zipping a symlink stores the *target's* content, which
+    would publish files from outside the project. Skipped symlinks are logged.
+
+    Args:
+        annotations_dir: The project's ``jabs/annotations`` directory.
+        output_path: Destination path for the ``.zip`` file.
+
+    Returns:
+        ``output_path`` if an archive was written, or None if ``annotations_dir``
+        does not exist or holds no archivable files -- there is then nothing worth
+        uploading.
+
+    Raises:
+        OSError: If reading the annotations or writing the archive fails.
+    """
+    if not annotations_dir.is_dir():
+        logger.warning(
+            "Annotations directory not found, no annotations artifact: %s", annotations_dir
+        )
+        return None
+
+    real_root = annotations_dir.resolve()
+    files: list[Path] = []
+    skipped_links = 0
+    for path in sorted(annotations_dir.rglob("*")):
+        # Skip symlinks themselves, and any entry whose real location falls
+        # outside the annotations directory -- the latter guards against a
+        # directory symlink being traversed (which pathlib does not currently do,
+        # but is not contractually guaranteed across Python versions).
+        if path.is_symlink() or not path.resolve().is_relative_to(real_root):
+            skipped_links += 1
+            continue
+        if path.is_file():
+            files.append(path)
+
+    if skipped_links:
+        logger.warning(
+            "Skipped %d symlinked entr%s under %s: symlinks are not archived",
+            skipped_links,
+            "y" if skipped_links == 1 else "ies",
+            annotations_dir,
+        )
+
+    if not files:
+        logger.warning(
+            "No archivable annotation files in %s, no annotations artifact", annotations_dir
+        )
+        return None
+
+    root = Path(annotations_dir.name)
+    with zipfile.ZipFile(output_path, "w", compression=zipfile.ZIP_DEFLATED) as archive:
+        for path in files:
+            archive.write(path, arcname=str(root / path.relative_to(annotations_dir)))
+
+    logger.info(
+        "Archived %d annotation file(s) from %s into %s (%d bytes)",
+        len(files),
+        annotations_dir,
+        output_path.name,
+        output_path.stat().st_size,
+    )
+    return output_path
+
+
 def _git_sha() -> str | None:
     """Short git SHA of the jabs checkout, or None if unavailable."""
     try:
@@ -266,23 +349,29 @@ def log_cross_validation_to_mlflow(
     *,
     report_data: TrainingReportData,
     report_file: Path | None = None,
+    annotations_dir: Path | None = None,
     env_file: Path | None = None,
     experiment_name: str | None = None,
     run_name: str | None = None,
     tags: dict[str, str] | None = None,
     log_report_artifact: bool = True,
+    log_annotations_artifact: bool = True,
 ) -> tuple[str, str]:
     """Create one MLflow run for a cross-validation run and return its ids.
 
     Logs aggregate CV metrics, curated params, auto-derived plus caller tags,
-    and (optionally) the training report as an artifact. Connection config comes
-    from the environment; ``env_file``, if given, is loaded into it first.
+    and (optionally) the training report and a zip of the project's annotations
+    as artifacts. Connection config comes from the environment; ``env_file``, if
+    given, is loaded into it first.
 
     Args:
         report_data: Completed training report data.
         report_file: Path to the saved training report to upload as an artifact.
             Ignored if None or missing on disk, or if ``log_report_artifact`` is
             False.
+        annotations_dir: The project's ``jabs/annotations`` directory, zipped and
+            uploaded as ``annotations.zip``. Ignored if None, if the directory is
+            missing or empty, or if ``log_annotations_artifact`` is False.
         env_file: Optional ``.env`` file with ``MLFLOW_*`` connection settings.
             If None, connection config comes from the ambient environment.
         experiment_name: Explicit MLflow experiment name. If None, the experiment is
@@ -294,12 +383,15 @@ def log_cross_validation_to_mlflow(
         tags: Caller-supplied run tags; merged over the auto-derived tags (so a
             user tag with the same key wins).
         log_report_artifact: Whether to upload the training report artifact.
+        log_annotations_artifact: Whether to upload the zipped annotations artifact.
 
     Returns:
         A ``(run_id, tracking_uri)`` tuple for the created MLflow run.
 
     Raises:
-        MlflowLoggingError: If the ``mlflow`` package is not installed.
+        MlflowLoggingError: If the ``mlflow`` package is not installed. A failure
+            to archive the annotations is *not* fatal: it is logged as a warning
+            and the run keeps its metrics, params, and report artifact.
     """
     try:
         import mlflow
@@ -340,6 +432,24 @@ def log_cross_validation_to_mlflow(
         if log_report_artifact and report_file is not None and Path(report_file).is_file():
             mlflow.log_artifact(str(report_file))
 
+        if log_annotations_artifact and annotations_dir is not None:
+            # The annotations are supplementary provenance, so a failure to
+            # archive them must not cost the run its metrics, params, or report.
+            try:
+                with tempfile.TemporaryDirectory() as staging_dir:
+                    archive = archive_annotations(
+                        Path(annotations_dir), Path(staging_dir) / ANNOTATIONS_ARCHIVE_NAME
+                    )
+                    if archive is not None:
+                        mlflow.log_artifact(str(archive))
+            except OSError:
+                logger.warning(
+                    "Failed to archive annotations from %s; run %s has no annotations artifact",
+                    annotations_dir,
+                    run.info.run_id,
+                    exc_info=True,
+                )
+
         run_id = run.info.run_id
 
     tracking_uri = mlflow.get_tracking_uri()

{jabs_behavior_classifier-0.46.0 → jabs_behavior_classifier-0.46.2}/src/jabs/project/parallel_workers.py

@@ -21,7 +21,7 @@ from jabs.core.constants import MULTICLASS_NONE_BEHAVIOR
 from jabs.core.enums import CacheFormat
 from jabs.pose_estimation import open_pose_file
 from jabs.video_reader import VideoReader
-from jabs.video_reader.utilities import get_fps
+from jabs.video_reader.utilities import get_fps_and_nframes
 
 from .track_labels import TrackLabels
 from .video_labels import VideoLabels
@@ -32,6 +32,21 @@ if TYPE_CHECKING:
     from jabs.pose_estimation import PoseEstimation
 
 
+def preload_worker_modules() -> None:
+    """Process-pool initializer that warms each worker's import cache.
+
+    Intended to be passed as the ``initializer`` for the application
+    ``ProcessPoolManager``. A worker must import this module to resolve this
+    reference, and importing it (with its transitive dependencies) pulls in the
+    heavy stack workers need: numpy, pandas and h5py directly, plus
+    ``jabs.feature_extraction`` (which brings in scipy) and the pose/video
+    readers (which bring in OpenCV). Doing that once at pool warm-up means the
+    first project-load scan and the first training run don't pay the import
+    cost. The function body is intentionally empty; the import triggered by
+    resolving it is the whole point.
+    """
+
+
 class _BaseFeatureLoadJobSpec(TypedDict):
     """Shared fields for binary and multi-class feature-load jobs."""
 
@@ -212,13 +227,39 @@ def _apply_macos_fork_lapack_workaround() -> None:
         fe.feature_base_class._use_numpy_detrend = True
 
 
+def _warn_on_frame_count_mismatch(
+    video: str, video_frame_count: int, pose_frame_count: int
+) -> None:
+    """Log a warning when a video and its pose file disagree on frame count.
+
+    The video/pose frame-count check is deferred from project load. Feature
+    extraction and predictions are indexed to pose frames, so a mismatch does
+    not corrupt them; this surfaces the discrepancy for headless/batch runs
+    where no video is opened interactively.
+
+    Args:
+        video: Video filename, used in the log message.
+        video_frame_count: Frame count read from the video file.
+        pose_frame_count: Frame count from the pose file.
+    """
+    if video_frame_count != pose_frame_count:
+        logger.warning(
+            "%s: video frame count (%d) does not match pose file frame count (%d); "
+            "features and predictions are pose-indexed and unaffected",
+            video,
+            video_frame_count,
+            pose_frame_count,
+        )
+
+
 def _open_pose_and_labels(
     job: _BaseFeatureLoadJobSpec,
 ) -> "tuple[PoseEstimation, VideoLabels | None, float]":
     """Set up the macOS workaround and open pose + label resources for a job."""
     _apply_macos_fork_lapack_workaround()
     pose_est = open_pose_file(job["pose_path"], job["cache_dir"])
-    fps = get_fps(str(job["video_path"]))
+    fps, video_frame_count = get_fps_and_nframes(str(job["video_path"]))
+    _warn_on_frame_count_mismatch(job["video"], video_frame_count, pose_est.num_frames)
     labels_obj = _load_video_labels(job["annotations_path"], pose_est)
     return pose_est, labels_obj, fps
 

jabs_behavior_classifier-0.46.2/src/jabs/project/pose_attribute_cache.py

@@ -0,0 +1,88 @@
+"""Persistent cache of per-video pose attributes to skip the load-time pose scan.
+
+The project-load scan opens every pose HDF5 file to read a handful of small,
+intrinsic attributes (frame count, identity count, static objects, lixit
+keypoint count, cm-per-pixel flag). Those values change only when the pose file
+itself changes, so they are cached in ``jabs/cache/pose_attribute_cache.json``
+keyed by video filename and gated by a cheap ``stat`` token of the pose file. A
+subsequent load then rescans only the videos whose pose file is new or changed.
+
+The cache is an optimization only: a missing, unreadable, or schema-mismatched
+cache simply triggers a full rescan, and a failed write is logged and ignored.
+"""
+
+import json
+import logging
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+# Bump when the cached entry schema changes so stale caches are ignored wholesale.
+SCHEMA_VERSION = 1
+
+
+def pose_token(pose_path: Path) -> str:
+    """Return a cheap change-detection token for a pose file.
+
+    Uses a single ``stat`` (size and modification time) rather than opening or
+    hashing the file, so computing the token stays far cheaper than the scan it
+    guards.
+
+    Args:
+        pose_path: Path to the pose HDF5 file.
+
+    Returns:
+        A ``"<size>:<mtime_ns>"`` token string.
+    """
+    st = pose_path.stat()
+    return f"{st.st_size}:{st.st_mtime_ns}"
+
+
+def load(cache_path: Path | None) -> dict[str, dict]:
+    """Load the per-video attribute map from the cache file.
+
+    Args:
+        cache_path: Path to the cache JSON file, or ``None`` when caching is
+            disabled (``use_cache=False``).
+
+    Returns:
+        Mapping of video filename to its cached entry. Returns an empty mapping
+        when caching is disabled, or the file is missing, unreadable, or written
+        by a different schema version.
+    """
+    if cache_path is None or not cache_path.exists():
+        return {}
+    try:
+        with cache_path.open("r") as f:
+            data = json.load(f)
+    except (OSError, ValueError):
+        logger.warning("Could not read pose attribute cache %s; rescanning", cache_path)
+        return {}
+    if not isinstance(data, dict) or data.get("schema_version") != SCHEMA_VERSION:
+        return {}
+    videos = data.get("videos")
+    return videos if isinstance(videos, dict) else {}
+
+
+def save(cache_path: Path | None, videos: dict[str, dict]) -> None:
+    """Atomically write the per-video attribute map to the cache file.
+
+    A write failure is logged and swallowed: the cache is an optimization, so
+    failing to persist it must never break project loading.
+
+    Args:
+        cache_path: Path to the cache JSON file, or ``None`` when caching is
+            disabled (in which case this is a no-op).
+        videos: Mapping of video filename to its cached entry.
+    """
+    if cache_path is None:
+        return
+    payload = {"schema_version": SCHEMA_VERSION, "videos": videos}
+    tmp = cache_path.with_suffix(".json.tmp")
+    try:
+        cache_path.parent.mkdir(parents=True, exist_ok=True)
+        with tmp.open("w") as f:
+            json.dump(payload, f, indent=2, sort_keys=True)
+        tmp.replace(cache_path)
+    except OSError:
+        logger.warning("Could not write pose attribute cache %s", cache_path, exc_info=True)