protspace 4.16.2__tar.gz → 4.16.4__tar.gz

{protspace-4.16.2 → protspace-4.16.4}/CHANGELOG.md

@@ -1,6 +1,57 @@
 # CHANGELOG
 
 
+## v4.16.4 (2026-10-15)
+
+### Bug Fixes
+
+- **reader**: Refuse only a bundle file as save_data's default target
+  ([`d91ff68`](https://github.com/tsenoner/protspace/commit/d91ff68fa4b82dc02e006e7633a140728e60ce7b))
+
+- **reader**: Refuse save_data without a directory to write into
+  ([`b8c8b8c`](https://github.com/tsenoner/protspace/commit/b8c8b8c6a242baef45b577ef7ef5283a79d8b448))
+
+- **reader**: Save annotations where ArrowReader reloads them
+  ([`6b5baa8`](https://github.com/tsenoner/protspace/commit/6b5baa84b4c450d6c0547bfc5a99f294bbb12494))
+
+### Refactoring
+
+- **reader**: Write save_data's tables in one loop, test via the shared fixture
+  ([`2e3ed84`](https://github.com/tsenoner/protspace/commit/2e3ed8478b0b7cdec3a461c530403626c1fbe729))
+
+
+## v4.16.3 (2026-10-13)
+
+### Bug Fixes
+
+- **reader**: Read a .parquetbundle directly instead of refusing it
+  ([`ed1d127`](https://github.com/tsenoner/protspace/commit/ed1d12737089df465b896b4ac3c85c405a1c73cf))
+
+### Chores
+
+- **protspace**: Ban src.* imports with ruff TID251
+  ([`cca83a1`](https://github.com/tsenoner/protspace/commit/cca83a173c127462c4d019cae3879290837861be))
+
+### Refactoring
+
+- **reader**: Fold the path check into _load_data and test it
+  ([`d662690`](https://github.com/tsenoner/protspace/commit/d662690b5869f0361bd23342b2c61264beb194d9))
+
+### Testing
+
+- **protspace**: Import protspace, not src.protspace, in the tests
+  ([`238d432`](https://github.com/tsenoner/protspace/commit/238d432afda2adc43d23c2a9c44722cd751d4a51))
+
+- **protspace**: Match the remaining transfer CLI errors through plain()
+  ([`aa15d57`](https://github.com/tsenoner/protspace/commit/aa15d57a6bf0979d35616a6ce6a0bad08f0e14b8))
+
+- **protspace**: Pin the embed usage errors and the MDS max(s) shift
+  ([`07a8403`](https://github.com/tsenoner/protspace/commit/07a8403fc4b3dde50475a0ffc473a4c80dec9639))
+
+- **protspace**: Reuse plain() and one _forbid helper in the audit tests
+  ([`7aa03f1`](https://github.com/tsenoner/protspace/commit/7aa03f1da2d0749035fabc96673ae7437040b01e))
+
+
 ## v4.16.2 (2026-10-07)
 
 ### Bug Fixes
@@ -886,6 +937,9 @@
 - **prepare**: Make -f reach every HDF5 input and reject a path that is not there
   ([`0591944`](https://github.com/tsenoner/protspace/commit/0591944008d7a79786a36ccab125f8970a004d57))
 
+- **reader**: Raise instead of returning an empty dataset for a bad path
+  ([`a6b00ec`](https://github.com/tsenoner/protspace/commit/a6b00ec52d7d2fbe586ba8a963ceca2a6b30c5da))
+
 ### Build System
 
 - **deps**: Give protobuf a floor instead of leaving it unconstrained

{protspace-4.16.2 → protspace-4.16.4}/CLAUDE.md

@@ -317,13 +317,13 @@ For a live count run `uv run pytest tests/ --collect-only -q`.
 | `test_config_validation.py` | DimensionReductionConfig parameter validation |
 | `test_style_warnings.py` | `protspace style` warnings: numeric-column detection (tsenoner/protspace-legacy#67) + `selectedPaletteId` validation (categorical vs gradient palette, per column type) + pinned palette-catalog contract |
 | `test_h5_parse_identifier.py` | HDF5 key parsing, identifier extraction |
-| `test_base_data_processor.py` | BaseProcessor: reduction, output creation, save (incl. settings in unbundled output) |
+| `test_base_data_processor.py` | BaseProcessor: reduction (the MDS input actually fed: a similarity matrix converted with `sqrt(max(s) - s)`, a distance matrix passed through), output creation, save (incl. settings in unbundled output) |
 | `test_ted_retriever.py` | TED domain retriever (mocked AlphaFold API, CATH names), final retry pass for failed lookups (10-in-a-row cut-off, 404 never retried), parallel lookups equal to one at a time under jitter, one session, CATH names loaded once, a slow first-pass lookup not holding up the others, a tripped final-pass breaker stopping lookups still retrying |
 | `test_pfam_clan.py` | Pfam CLAN transformer (mapping, dedup, edge cases) |
 | `test_formatters.py` | ProteinAnnotations → DataFrame formatting |
 | `test_bundle_settings.py` | Parquetbundle settings read/write |
 | `test_annotation_encoding.py` | Percent-encoding round-trip, reserved-char-only encoding, schema-metadata stamping through parquet, `upgrade_cell_grammar` taking the version from the caller |
-| `test_transfer_cli.py` | Transfer orchestration core and CLI registration |
+| `test_transfer_cli.py` | Transfer orchestration core (k>1 confidence, cosine metric, self-transfer) and the `transfer` CLI: bad input (no matching embedding, no query match, both id columns) as a usage error with its message, `--help` rendering, `protein_id` and legacy v1 bundles end to end |
 | `test_predictions_overlay.py` | Building the per-cell prediction overlay columns |
 | `test_display_decode.py` | Display-side decoding of encoded values, multi-hit rendering, gated-off passthrough |
 | `test_toxprot_demo.py` | Signal-peptide bound parsing, mature-FASTA stripping, bundle post-processing (column filter/reorder) |
@@ -334,6 +334,7 @@ For a live count run `uv run pytest tests/ --collect-only -q`.
 | `test_atomic_publication.py` | `data/io/atomic.py`: staged rename keeps the previous content on failure, and a published file (bundle, statistics parquet, retained FASTA) carries the process umask rather than `mkstemp`'s owner-only mode |
 | `test_classification.py` | Query/reference rules: id-prefix and case-insensitive `where` substring, query-over-reference precedence, empty-match and missing-column errors |
 | `test_bundle_version.py` | `format_version=2` stamped on the table the `prepare` factory hands to `write_bundle`; a `prepare` bundle reads back its proteins, annotation cells and coordinates; `bundle -a` passes stamped annotate output and the `prepare` annotation cache through (an unstamped cache is recognised by its cache-version attribute), reads an unstamped table as v1 plain text (a pandas `category` column too), stores a list column as hits and reports an unstorable column as a usage error; `ArrowReader.save_data` keeps the stamp it read |
+| `test_arrow_reader.py` | `ArrowReader(Path)` reads a `.parquetbundle` in memory exactly as its extracted directory (data, settings, format version), and refuses a path that holds no ProtSpace data (a non-bundle file, a missing path, a directory with no core table) instead of reading it as an empty dataset; an annotations-only directory still loads; `save_data` writes a directory that reloads with its annotations, projections and restyled colours (the `protspace style` path for an unbundled parquet directory) and recreates its source directory when saved in place, a bundle saves to a directory that reads back alike, while `save_data` without an output directory (bundle source) or on a dict-built reader is refused before anything is written |
 | `test_bundle_v3_encode.py` | v3 encoder: the physical contract behind the browser's zero-copy read (non-nullable PLAIN columns, one row group, little-endian payloads) and v2-reader-parity classification and code order (part 1 rows and projections in the v2 browser's order, numeric inference over the placed proteins with the `placedNumeric` mark, a v1 hit split at its last pipe); NaN for uncovered projections, added unannotated rows, dimension from the data, `true`/`false` booleans, list columns as one hit per element, unstamped and v1 tables refused |
 | `test_bundle_v3_decode.py` | v3 decoder: `decode_v3(encode_v3(T)) == T` on pipeline-shaped tables, plus the deliberate canonicalisations; `sourceType` restoration (64-bit integers past 2^53 exactly), finite-only projection rows, manifest dimensions, payload tiling checks |
 | `test_bundle_v3_container.py` | Six-part container boundary: every write emits v3 with part 6 pinned, every read hands back v2-shaped tables, legacy bundles read as written, the delimiter guard covers part 6; detection by `protspace_container_version`, `style` writing a legacy input as v3, `replace_annotations` keeping label columns labels and the `placedNumeric` mark, a corrupt part 1 as a bundle error |

{protspace-4.16.2 → protspace-4.16.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: protspace
-Version: 4.16.2
+Version: 4.16.4
 Summary: Prepare protein language model (pLM) embeddings for interactive exploration at protspace.app — embed, project, annotate, transfer labels (EAT), and bundle.
 Project-URL: Homepage, https://protspace.app
 Project-URL: Repository, https://github.com/tsenoner/protspace

{protspace-4.16.2 → protspace-4.16.4}/packages/protlabel/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "protlabel"
-version = "4.16.2"
+version = "4.16.4"
 description = "Embedding Annotation Transfer (EAT): nearest-neighbour label transfer in protein-language-model embedding space"
 authors = [{ name = "Tobias Senoner", email = "tobias.senoner@tum.de" }]
 readme = "README.md"

{protspace-4.16.2 → protspace-4.16.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "protspace"
-version = "4.16.2"
+version = "4.16.4"
 description = "Prepare protein language model (pLM) embeddings for interactive exploration at protspace.app — embed, project, annotate, transfer labels (EAT), and bundle."
 authors = [{ name = "Tobias Senoner", email = "tobias.senoner@tum.de" }]
 readme = "README.md"
@@ -136,6 +136,7 @@ select = [
     "ARG001", # unused-function-args
     "F401", # unused-import
     "F841", # unused-variable
+    "TID251", # banned-api: see flake8-tidy-imports.banned-api below
 ]
 ignore = [
     "E501", # line too long, handled by black
@@ -143,6 +144,11 @@ ignore = [
     "C901", # too complex
 ]
 
+[tool.ruff.lint.flake8-tidy-imports.banned-api]
+# Importing through `src.` loads a second copy of every protspace module next to the
+# installed one, so a `patch("src.protspace...")` misses the copy the code under test uses.
+"src".msg = "Import from `protspace`, not `src.protspace`."
+
 [tool.ruff.lint.isort]
 # protlabel lives in the packages/protlabel workspace member (not src/), so
 # declare both packages first-party for stable import grouping.

protspace-4.16.4/src/protspace/__init__.py

@@ -0,0 +1 @@
+__version__ = "4.16.4"

{protspace-4.16.2 → protspace-4.16.4}/src/protspace/main.py

@@ -3,7 +3,6 @@ import warnings
 from pathlib import Path
 
 from protspace.app import ProtSpace
-from protspace.data.io.bundle import extract_bundle_to_dir
 
 DEFAULT_PORT = 8050
 
@@ -11,18 +10,18 @@ warnings.filterwarnings("ignore", category=SyntaxWarning)
 
 
 def detect_data_type(data_path: str) -> str:
-    """Resolve *data_path* to a directory of Parquet files.
+    """Resolve *data_path* to something ``ArrowReader`` reads.
 
-    Accepts a ``.parquetbundle`` file (extracted to a temp dir) or a
-    directory containing ``.parquet`` files.
+    Accepts a ``.parquetbundle`` file or a directory containing ``.parquet``
+    files (or a ``data.parquetbundle``).
 
     Returns:
-        Path to the Parquet directory.
+        Path to the bundle file or the Parquet directory.
     """
     path = Path(data_path)
 
     if path.is_file() and path.suffix.lower() == ".parquetbundle":
-        return extract_bundle_to_dir(path)
+        return str(path)
     elif path.is_dir():
         parquet_files = list(path.glob("*.parquet"))
         if parquet_files:
@@ -30,7 +29,7 @@ def detect_data_type(data_path: str) -> str:
 
         bundle_file = path / "data.parquetbundle"
         if bundle_file.exists():
-            return extract_bundle_to_dir(bundle_file)
+            return str(bundle_file)
 
         raise ValueError(
             f"Directory '{data_path}' does not contain .parquet files or data.parquetbundle"

{protspace-4.16.2 → protspace-4.16.4}/src/protspace/utils/add_annotation_style.py

@@ -281,15 +281,8 @@ def generate_template(input_file: str) -> dict:
     with values in frequency-descending order. ``<NA>`` gets a default
     light-gray color.
     """
-    data_format = detect_data_format(input_file)
-
-    if data_format == "parquetbundle":
-        from protspace.data.io.bundle import extract_bundle_to_dir
-
-        temp_dir = extract_bundle_to_dir(Path(input_file))
-        reader = ArrowReader(Path(temp_dir))
-    elif data_format == "parquet":
-        reader = ArrowReader(Path(input_file))
+    detect_data_format(input_file)
+    reader = ArrowReader(Path(input_file))
 
     frequencies = compute_value_frequencies(reader)
     template: dict = {}
@@ -366,7 +359,7 @@ def add_annotation_styles_bundle(
 ) -> None:
     """Add annotation styles to a .parquetbundle file.
 
-    Extracts the bundle to a temp dir, applies styles via ArrowReader, converts
+    Reads the bundle with ArrowReader, applies styles, converts
     the visualization_state back to settings_json, then writes the output
     bundle with updated settings.
 
@@ -375,25 +368,15 @@ def add_annotation_styles_bundle(
     ``selectedPaletteId``) alongside ``colors`` and ``shapes``.  These are
     forwarded to the settings converter.
     """
-    from protspace.data.io.bundle import (
-        SETTINGS_FILENAME,
-        extract_bundle_to_dir,
-        read_settings_from_file,
-        replace_settings_in_bundle,
-    )
+    from protspace.data.io.bundle import replace_settings_in_bundle
     from protspace.data.io.settings_converter import visualization_state_to_settings
 
-    # Extract bundle so ArrowReader can load the data
-    temp_dir = extract_bundle_to_dir(Path(bundle_file))
-    reader = ArrowReader(Path(temp_dir))
+    reader = ArrowReader(Path(bundle_file))
 
-    # Read existing settings (if any) to preserve extra fields.  From the
-    # extracted copy, not the bundle again: this command reads the bundle once
-    # (and so logs a legacy-format warning once).
-    settings_path = Path(temp_dir) / SETTINGS_FILENAME
-    existing_settings = (
-        read_settings_from_file(settings_path) if settings_path.exists() else None
-    )
+    # Existing settings (if any), to preserve extra fields.  From the reader,
+    # not the bundle again: this command reads the bundle once (and so logs a
+    # legacy-format warning once).
+    existing_settings = reader.settings
 
     # Collect settings-level overrides from the styles input
     style_overrides: dict[str, dict] = {}

{protspace-4.16.2 → protspace-4.16.4}/src/protspace/utils/arrow_reader.py

@@ -28,27 +28,30 @@ def _read_format_version(schema_metadata: dict | None) -> int:
 class ArrowReader:
     """Read and manipulate ProtSpace data from Parquet files or a dict.
 
-    Accepts either a ``Path`` to a directory of Parquet files or a
-    pre-built ``dict`` (the same structure previously handled by the
-    legacy ``JsonReader``).
+    Accepts a ``Path`` to a ``.parquetbundle`` file or to a directory of
+    Parquet files, or a pre-built ``dict`` (the same structure previously
+    handled by the legacy ``JsonReader``).
     """
 
     def __init__(self, source: Path | dict):
         """
         Args:
-            source: Path to directory containing .parquet files, or a
-                pre-built data dict (protein_data, projections, …).
+            source: Path to a .parquetbundle file or to a directory
+                containing .parquet files, or a pre-built data dict
+                (protein_data, projections, …).
         """
         if isinstance(source, dict):
             # Direct dict input
             self.data = source
             self.data_path = None
+            self.settings = None
             self._protein_annotations_df = None
             self._projections_metadata_df = None
             self._projections_data_df = None
         else:
             # Parquet file loading
             self.data_path = Path(source)
+            self.settings = None
             self._protein_annotations_df = None
             self._projections_metadata_df = None
             self._projections_data_df = None
@@ -61,43 +64,61 @@ class ArrowReader:
             self._build_data_structure()
 
     def _load_data(self):
-        """Load data from Parquet files."""
-        try:
-            protein_annotations_path = self.data_path / "selected_annotations.parquet"
-            projections_metadata_path = self.data_path / "projections_metadata.parquet"
-            projections_data_path = self.data_path / "projections_data.parquet"
-
-            if protein_annotations_path.exists():
-                annotations_table = pq.read_table(str(protein_annotations_path))
-                self._protein_annotations_df = annotations_table.to_pandas()
-                self.data["format_version"] = _read_format_version(
-                    annotations_table.schema.metadata
-                )
-            else:
-                self._protein_annotations_df = pd.DataFrame(columns=["protein_id"])
-
-            if projections_metadata_path.exists():
-                self._projections_metadata_df = pq.read_table(
-                    str(projections_metadata_path)
-                ).to_pandas()
-            else:
-                self._projections_metadata_df = pd.DataFrame(
-                    columns=["projection_name", "dimensions", "info_json"]
-                )
+        """Load the core tables from a .parquetbundle file or a directory."""
+        path = self.data_path
+        if path.is_file():
+            tables = self._read_bundle(path)
+        elif path.is_dir():
+            tables = self._read_directory(path)
+        else:
+            raise FileNotFoundError(f"No such file or directory: '{path}'")
 
-            if projections_data_path.exists():
-                self._projections_data_df = pq.read_table(
-                    str(projections_data_path)
-                ).to_pandas()
-            else:
-                self._projections_data_df = pd.DataFrame(
-                    columns=["projection_name", "identifier", "x", "y", "z"]
-                )
+        annotations, metadata, projections = tables
+        if annotations is not None:
+            self._protein_annotations_df = annotations.to_pandas()
+            self.data["format_version"] = _read_format_version(
+                annotations.schema.metadata
+            )
+        else:
+            self._protein_annotations_df = pd.DataFrame(columns=["protein_id"])
+        self._projections_metadata_df = (
+            metadata.to_pandas()
+            if metadata is not None
+            else pd.DataFrame(columns=["projection_name", "dimensions", "info_json"])
+        )
+        self._projections_data_df = (
+            projections.to_pandas()
+            if projections is not None
+            else pd.DataFrame(columns=["projection_name", "identifier", "x", "y", "z"])
+        )
 
-        except Exception as e:
+    def _read_bundle(self, path: Path) -> tuple[pa.Table, pa.Table, pa.Table]:
+        """Read a bundle in memory, whatever its container version."""
+        from protspace.data.io.bundle import read_bundle_contents
+
+        contents = read_bundle_contents(path)
+        self.settings = contents.settings
+        return contents.annotations, contents.metadata, contents.projections
+
+    def _read_directory(self, path: Path) -> tuple[pa.Table | None, ...]:
+        """Read the core tables a directory holds; a missing one reads as None.
+
+        An annotations-only directory is valid, but one holding none of the core
+        tables is not ProtSpace data and must not read as an empty dataset.
+        """
+        from protspace.data.io.bundle import CORE_FILENAMES
+
+        core_paths = [path / name for name in CORE_FILENAMES]
+        if not any(p.exists() for p in core_paths):
             raise ValueError(
-                f"Error loading Arrow data from {self.data_path}: {e}"
-            ) from e
+                f"Directory '{path}' contains none of {', '.join(CORE_FILENAMES)}."
+            )
+        try:
+            return tuple(
+                pq.read_table(str(p)) if p.exists() else None for p in core_paths
+            )
+        except Exception as e:
+            raise ValueError(f"Error loading Arrow data from {path}: {e}") from e
 
     def _build_data_structure(self):
         """Build the data structure matching the ProtSpace data schema."""
@@ -158,65 +179,85 @@ class ArrowReader:
     def _load_visualization_state(self):
         """Load visualization state.
 
-        Priority: visualization_state.json > settings.parquet (from bundle).
+        A bundle carries it in its settings part. A directory may hold
+        visualization_state.json, which wins over settings.parquet.
         """
-        viz_state_path = self.data_path / "visualization_state.json"
-        if viz_state_path.exists():
-            try:
-                with open(viz_state_path) as f:
-                    viz_state = json.load(f)
-                    self.data["visualization_state"] = viz_state
-                    return
-            except (json.JSONDecodeError, FileNotFoundError):
-                pass
-
-        # Fall back to settings.parquet (extracted from a 4-part bundle)
-        settings_path = self.data_path / "settings.parquet"
-        if settings_path.exists():
-            try:
-                from protspace.data.io.bundle import read_settings_from_file
-                from protspace.data.io.settings_converter import (
-                    settings_to_visualization_state,
-                )
+        if self.data_path.is_dir():
+            viz_state_path = self.data_path / "visualization_state.json"
+            if viz_state_path.exists():
+                try:
+                    with open(viz_state_path) as f:
+                        self.data["visualization_state"] = json.load(f)
+                        return
+                except (json.JSONDecodeError, FileNotFoundError):
+                    pass
+
+        try:
+            from protspace.data.io.bundle import (
+                SETTINGS_FILENAME,
+                read_settings_from_file,
+            )
+            from protspace.data.io.settings_converter import (
+                settings_to_visualization_state,
+            )
 
-                settings = read_settings_from_file(settings_path)
+            settings_path = self.data_path / SETTINGS_FILENAME
+            if self.data_path.is_dir() and settings_path.exists():
+                self.settings = read_settings_from_file(settings_path)
+            if self.settings:
                 self.data["visualization_state"] = settings_to_visualization_state(
-                    settings
-                )
-            except Exception:
-                logger.debug(
-                    "Failed to load visualization state from settings", exc_info=True
+                    self.settings
                 )
+        except Exception:
+            logger.debug(
+                "Failed to load visualization state from settings", exc_info=True
+            )
+
+    def save_data(self, output_path: Path | None = None):
+        """Save the current data back to parquet files.
 
-    def save_data(self, output_path: Path = None):
-        """Save the current data back to parquet files."""
+        Defaults to the directory the data was read from. A reader built from
+        a .parquetbundle needs an explicit ``output_path``: the bundle is a
+        file, not a directory the tables could be written into.
+        """
+        if self.data_path is None:
+            raise ValueError(
+                "save_data writes the parquet tables a reader loaded; this "
+                "reader was built from a dict and holds none."
+            )
         if output_path is None:
+            if self.data_path.is_file():
+                raise ValueError(
+                    f"'{self.data_path}' is a bundle file, not a directory; "
+                    "pass save_data an output directory."
+                )
             output_path = self.data_path
         else:
             output_path = Path(output_path)
 
         output_path.mkdir(parents=True, exist_ok=True)
 
-        # Save protein annotations
-        protein_annotations_path = output_path / "protein_annotations.parquet"
-        protein_annotations_table = pa.Table.from_pandas(self._protein_annotations_df)
+        # The names _read_directory reads back (and a bundle's parts extract to)
+        from protspace.data.io.bundle import CORE_FILENAMES
+
+        annotations, metadata, projections = (
+            pa.Table.from_pandas(df)
+            for df in (
+                self._protein_annotations_df,
+                self._projections_metadata_df,
+                self._projections_data_df,
+            )
+        )
         if self.should_decode():
             # The cells are written back as they were read, so they keep the
             # grammar stamp that says so; unstamped they would read as v1.
             from protspace.data.annotations.encoding import stamp_format_version
 
-            protein_annotations_table = stamp_format_version(protein_annotations_table)
-        pq.write_table(protein_annotations_table, str(protein_annotations_path))
-
-        # Save projections metadata
-        projections_metadata_path = output_path / "projections_metadata.parquet"
-        projections_metadata_table = pa.Table.from_pandas(self._projections_metadata_df)
-        pq.write_table(projections_metadata_table, str(projections_metadata_path))
-
-        # Save projections data
-        projections_data_path = output_path / "projections_data.parquet"
-        projections_data_table = pa.Table.from_pandas(self._projections_data_df)
-        pq.write_table(projections_data_table, str(projections_data_path))
+            annotations = stamp_format_version(annotations)
+        for name, table in zip(
+            CORE_FILENAMES, (annotations, metadata, projections), strict=True
+        ):
+            pq.write_table(table, str(output_path / name))
 
         # Save visualization state as a separate JSON file
         viz_state_path = output_path / "visualization_state.json"