protspace 4.0.2__tar.gz → 4.2.0__tar.gz

{protspace-4.0.2 → protspace-4.2.0}/CHANGELOG.md

@@ -1,6 +1,67 @@
 # CHANGELOG
 
 
+## v4.2.0 (2026-03-28)
+
+### Features
+
+* feat: replace unipressed with direct UniProt REST API calls
+
+Replace the unipressed library (community UniProt API wrapper) with
+direct HTTP calls to rest.uniprot.org. Adds _fetch_many_accessions()
+and _search_sec_acc() helpers using the same Link-header pagination
+pattern as the taxonomy retriever.
+
+Simplifies the sec_acc search fallback from 8 lines of page-parsing
+to a single function call.
+
+Closes #32
+
+Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> ([`5605a93`](https://github.com/tsenoner/protspace/commit/5605a935e680132ee7d22adb30a45b3ff26df02b))
+
+### Refactoring
+
+* refactor: extract shared paginated_get() utility for REST API calls
+
+Consolidate the duplicated Link-header pagination loop (4 instances
+across uniprot_retriever, taxonomy_retriever, and uniprot_parser) into
+a single paginated_get() function in http_utils.py. Each caller is
+now a 1-3 line function.
+
+Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> ([`cd528ed`](https://github.com/tsenoner/protspace/commit/cd528ed68ba7cf9bf3a33183345b483c00b4f9d1))
+
+### Unknown
+
+* Merge pull request #39 from tsenoner/feat/replace-unipressed-with-direct-api
+
+Replace unipressed with direct UniProt REST API calls ([`5d365c7`](https://github.com/tsenoner/protspace/commit/5d365c70dfb66fe1754fb610c7df3acb3cfae480))
+
+
+## v4.1.0 (2026-03-28)
+
+### Features
+
+* feat: replace taxopy with UniProt Taxonomy API for taxonomy lookups
+
+Replace the taxopy-based taxonomy retriever (which required downloading
+the full NCBI taxonomy database ~50 MB on first use) with the UniProt
+Taxonomy API (/taxonomy/search). This eliminates the slow first-run
+download, weekly cache refresh, and ~120 lines of cache management code.
+
+Also fix typer[all] → typer (the [all] extra was removed) and add
+requests as an explicit core dependency.
+
+Closes #36
+
+Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> ([`754fa9b`](https://github.com/tsenoner/protspace/commit/754fa9b45cf5a913a97823b2b45012c49221ca7a))
+
+### Unknown
+
+* Merge pull request #37 from tsenoner/feat/replace-taxopy-with-uniprot-api
+
+Replace taxopy with UniProt Taxonomy API ([`034c0ec`](https://github.com/tsenoner/protspace/commit/034c0ece32238144fe68819ce2b6e5e43d8b57a5))
+
+
 ## v4.0.2 (2026-03-28)
 
 ### Fixes

{protspace-4.0.2 → protspace-4.2.0}/CLAUDE.md

@@ -205,7 +205,7 @@ uv run pytest tests/ --cov=src/protspace     # With coverage
 | `test_biocentral_embedder.py` | 23 | Biocentral API client, embedding flow |
 | `test_fasta.py` | 17 | FASTA parsing, edge cases, CSV annotation loading |
 | `test_config_validation.py` | 12 | DimensionReductionConfig parameter validation |
-| `test_taxonomy_annotation_retriever.py` | 12 | Taxonomy database lookups |
+| `test_taxonomy_annotation_retriever.py` | 15 | Taxonomy via UniProt Taxonomy API (mocked + integration) |
 | `test_h5_parse_identifier.py` | 9 | HDF5 key parsing, identifier extraction |
 | `test_base_data_processor.py` | 8 | BaseProcessor: reduction, output creation, save |
 | `test_formatters.py` | 5 | ProteinAnnotations → DataFrame formatting |
@@ -225,7 +225,7 @@ Located in `notebooks/`:
 
 ## Dependencies
 
-**Core:** h5py, scikit-learn, umap-learn, pacmap (includes annoy), numpy, pandas, pyarrow, tqdm, taxopy, pymmseqs, unipressed, biocentral-api, typer, rich
+**Core:** h5py, scikit-learn, umap-learn, pacmap (includes annoy), numpy, pandas, pyarrow, tqdm, requests, pymmseqs, biocentral-api, typer, rich
 
 **Frontend (optional):** dash, plotly, dash-bootstrap-components, dash-molstar, gunicorn
 

{protspace-4.0.2 → protspace-4.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: protspace
-Version: 4.0.2
+Version: 4.2.0
 Summary: A visualisation tool for protein embeddings from pLMs
 Author-email: Tobias Senoner <tobias.senoner@tum.de>
 License-Expression: GPL-3.0
@@ -13,13 +13,12 @@ Requires-Dist: pacmap>=0.8.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pyarrow>=20.0.0
 Requires-Dist: pymmseqs>=1.0.4
-Requires-Dist: rich>=13.0.0
+Requires-Dist: requests>=2.32.4
+Requires-Dist: rich>=14.3.3
 Requires-Dist: scikit-learn>=1.6.1
-Requires-Dist: taxopy>=0.14.0
 Requires-Dist: tqdm>=4.67.1
-Requires-Dist: typer[all]>=0.15.0
+Requires-Dist: typer>=0.24.1
 Requires-Dist: umap-learn>=0.5.10
-Requires-Dist: unipressed>=1.4.0
 Provides-Extra: frontend
 Requires-Dist: dash-bootstrap-components>=1.6.0; extra == 'frontend'
 Requires-Dist: dash-daq>=0.5.0; extra == 'frontend'

{protspace-4.0.2 → protspace-4.2.0}/docs/annotations.md

@@ -46,7 +46,7 @@ With `--keep-tmp`, only API-fetched annotations are cached; the CSV is always re
 
 ## UniProt Annotations
 
-14 annotations retrieved from the [UniProt REST API](https://rest.uniprot.org/) via `unipressed` (batch size: 100):
+14 annotations retrieved from the [UniProt REST API](https://rest.uniprot.org/) (batch size: 100):
 
 | Name                      | Description                          | Example                                                        |
 | ------------------------- | ------------------------------------ | -------------------------------------------------------------- |

{protspace-4.0.2 → protspace-4.2.0}/notebooks/ProtSpace_Preparation.ipynb

@@ -45,7 +45,7 @@
     "cellView": "form"
    },
    "outputs": [],
-   "source": "# @title 3. Generate & Download {display-mode: \"form\"}\n# @markdown Configure methods and annotations, then click Generate.\n# @markdown Processing time depends on dataset size and selected methods.\n\nfrom ipywidgets import (\n    HTML,\n    Button,\n    Checkbox,\n    Dropdown,\n    FileUpload,\n    FloatSlider,\n    GridBox,\n    HBox,\n    IntSlider,\n    Output,\n    ToggleButton,\n    VBox,\n)\n\nMETHODS = [\"PCA\", \"UMAP\", \"t-SNE\", \"MDS\", \"PaCMAP\", \"LocalMAP\"]\nANNOTATIONS = {\n    \"UniProt\": [\"annotation_score\", \"cc_subcellular_location\", \"ec\", \"fragment\", \"go_bp\", \"go_cc\", \"go_mf\", \"keyword\", \"length\", \"protein_existence\", \"protein_families\", \"reviewed\", \"xref_pdb\"],\n    \"InterPro\": [\"cath\", \"cdd\", \"panther\", \"pfam\", \"prints\", \"prosite\", \"signal_peptide\", \"smart\", \"superfamily\"],\n    \"Taxonomy\": [\"root\", \"domain\", \"kingdom\", \"phylum\", \"class\", \"order\", \"family\", \"genus\", \"species\"],\n}\nANNOTATION_DEFAULTS = {\"ec\", \"keyword\", \"length\", \"protein_families\", \"reviewed\"}\n\n# --- Methods ---\nmethod_toggles = {}\nfor m in METHODS:\n    sel = m in {\"PCA\", \"UMAP\"}\n    method_toggles[m] = ToggleButton(value=sel, description=m, button_style=\"info\" if sel else \"\", layout={\"width\": \"auto\", \"height\": \"32px\"})\n\n# --- DR Parameters ---\n_s = {\"description_width\": \"110px\"}\n_l = {\"width\": \"100%\"}\n_g = {\"border\": \"1px solid #666\", \"padding\": \"6px 10px\", \"margin\": \"2px\", \"flex\": \"1 1 220px\", \"overflow\": \"hidden\"}\n\nparam_groups = []\nw_nn = IntSlider(value=25, min=2, max=500, description=\"n_neighbors:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>UMAP / PaCMAP / LocalMAP</b>\"), w_nn], layout=_g), {\"UMAP\", \"PaCMAP\", \"LocalMAP\"}))\nw_md = FloatSlider(value=0.4, min=0.0, max=0.99, step=0.01, description=\"min_dist:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>UMAP</b>\"), w_md], layout=_g), {\"UMAP\"}))\nw_mn = FloatSlider(value=0.5, min=0.0, max=1.0, step=0.1, description=\"mn_ratio:\", style=_s, layout=_l)\nw_fp = FloatSlider(value=2.0, min=0.0, max=5.0, step=0.1, description=\"fp_ratio:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>PaCMAP / LocalMAP</b>\"), w_mn, w_fp], layout=_g), {\"PaCMAP\", \"LocalMAP\"}))\nw_pp = FloatSlider(value=30.0, min=5.0, max=500.0, step=1.0, description=\"perplexity:\", style=_s, layout=_l)\nw_lr = FloatSlider(value=200.0, min=1.0, max=1000.0, step=10.0, description=\"learning_rate:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>t-SNE</b>\"), w_pp, w_lr], layout=_g), {\"t-SNE\"}))\n\nno_params = HTML(\"<i>PCA/MDS have no adjustable parameters.</i>\")\npw = {\"n_neighbors\": w_nn, \"min_dist\": w_md, \"perplexity\": w_pp, \"learning_rate\": w_lr, \"mn_ratio\": w_mn, \"fp_ratio\": w_fp}\nparam_grid = HBox([g for g, _ in param_groups], layout={\"flex_flow\": \"row wrap\", \"gap\": \"6px\", \"width\": \"100%\"})\n\n\ndef _update_params(_c=None):\n    sel = {m for m, t in method_toggles.items() if t.value}\n    any_vis = False\n    for gw, req in param_groups:\n        vis = bool(req & sel)\n        gw.layout.display = \"\" if vis else \"none\"\n        if vis:\n            any_vis = True\n    no_params.layout.display = \"\" if not any_vis else \"none\"\n\n\nfor tb in method_toggles.values():\n    def _toggle(c):\n        c[\"owner\"].button_style = \"info\" if c[\"new\"] else \"\"\n        _update_params()\n    tb.observe(_toggle, names=\"value\")\n_update_params()\n\n# --- Annotations ---\nann_preset = Dropdown(\n    options=[\"Default (recommended)\", \"All annotations\", \"Custom...\"],\n    value=\"Default (recommended)\",\n    description=\"Annotations:\",\n    style={\"description_width\": \"initial\"},\n)\nann_cbs = {}\nann_cols = []\nfor cat, names in ANNOTATIONS.items():\n    cbs = {n: Checkbox(value=n in ANNOTATION_DEFAULTS, description=n, indent=False, layout={\"width\": \"auto\"}) for n in names}\n    ann_cbs[cat] = cbs\n    grid = GridBox(list(cbs.values()), layout={\"grid_template_columns\": \"repeat(2, auto)\", \"grid_gap\": \"0 6px\"})\n    ann_cols.append(VBox([HTML(f\"<b>{cat}</b>\"), grid], layout={\"flex\": \"1 1 200px\", \"border\": \"1px solid #666\", \"padding\": \"6px\", \"margin\": \"2px\"}))\ncustom_ann = HBox(ann_cols, layout={\"flex_flow\": \"row wrap\"})\ncustom_ann.layout.display = \"none\"\nann_preset.observe(lambda c: setattr(custom_ann.layout, \"display\", \"\" if c[\"new\"] == \"Custom...\" else \"none\"), names=\"value\")\n\n# --- Custom CSV upload ---\ncsv_upload = FileUpload(accept=\".csv,.tsv\", multiple=False, description=\"Choose CSV/TSV\")\n\n\ndef _get_ann():\n    p = ann_preset.value\n    if p == \"Default (recommended)\":\n        return [\"default\"]\n    if p == \"All annotations\":\n        return [\"all\"]\n    sel = [n for cbs in ann_cbs.values() for n, cb in cbs.items() if cb.value]\n    return sel if sel else [\"default\"]\n\n\ndef _save_csv():\n    \"\"\"Save uploaded CSV to disk and return the path, or None.\"\"\"\n    if not csv_upload.value:\n        return None\n    fname = list(csv_upload.value.keys())[0]\n    content = csv_upload.value[fname][\"content\"]\n    Path(fname).write_bytes(content)\n    return fname\n\n\n# --- Generate ---\ngen_btn = Button(description=\"Generate\", button_style=\"primary\", icon=\"play\", layout={\"width\": \"180px\", \"height\": \"40px\"})\ngen_out = Output()\n\n\ndef _on_gen(_b):\n    with gen_out:\n        clear_output()\n        if not input_args:\n            print(\"Select input data in cell above first.\")\n            return\n        method_specs = [m.lower().replace(\"-\", \"\") + \"2\" for m, t in method_toggles.items() if t.value]\n        if not method_specs:\n            print(\"Select at least one method.\")\n            return\n\n        ann = _get_ann() or []\n        csv_path = _save_csv()\n        if csv_path:\n            ann.append(csv_path)\n        if not ann:\n            ann = None\n\n        inp = input_args\n\n        out_dir = Path(\"output\")\n        out_dir.mkdir(exist_ok=True)\n        cache_dir = out_dir / \"tmp\"\n        cache_dir.mkdir(exist_ok=True)\n        output_path = out_dir / \"data.parquetbundle\"\n\n        step_html = HTML(value=\"<b>Step 1/4: Loading embeddings...</b>\")\n        display(step_html)\n\n        try:\n            t0 = _time.time()\n            embedding_sets = []\n\n            if inp[\"type\"] == \"query\":\n                embs = _get_selected_embedders()\n                if not embs:\n                    print(\"Select at least one embedder.\")\n                    return\n                step_html.value = \"<b>Step 1/6: Downloading FASTA...</b>\"\n                headers, fasta_path = query_uniprot(inp[\"query\"])\n                if not headers:\n                    print(f\"No sequences found for query: {inp['query']}\")\n                    return\n                from protspace.data.embedding.biocentral import EmbedConfig\n                for emb_name in embs:\n                    step_html.value = f\"<b>Step 2/6: Computing {emb_name} embeddings...</b>\"\n                    emb_set = embed_fasta(fasta_path, emb_name, embed_config=EmbedConfig())\n                    emb_set.fasta_path = fasta_path\n                    embedding_sets.append(emb_set)\n            elif inp[\"type\"] == \"fasta\":\n                embs = _get_selected_embedders()\n                if not embs:\n                    print(\"Select at least one embedder.\")\n                    return\n                from protspace.data.embedding.biocentral import EmbedConfig\n                for emb_name in embs:\n                    step_html.value = f\"<b>Step 1/5: Computing {emb_name} embeddings...</b>\"\n                    emb_set = embed_fasta(Path(inp[\"path\"]), emb_name, embed_config=EmbedConfig())\n                    emb_set.fasta_path = Path(inp[\"path\"])\n                    embedding_sets.append(emb_set)\n            else:\n                h5_path = Path(inp[\"path\"])\n                name_override = inp.get(\"name\")\n                emb_set = load_h5([h5_path], name_override=name_override)\n                embedding_sets.append(emb_set)\n\n            n_proteins = len(embedding_sets[0].headers)\n\n            # Build pipeline with caching enabled\n            reducer_params = ReducerParams(\n                n_neighbors=pw[\"n_neighbors\"].value,\n                min_dist=pw[\"min_dist\"].value,\n                perplexity=pw[\"perplexity\"].value,\n                learning_rate=pw[\"learning_rate\"].value,\n                mn_ratio=pw[\"mn_ratio\"].value,\n                fp_ratio=pw[\"fp_ratio\"].value,\n            )\n            config = PipelineConfig(\n                methods=method_specs,\n                output_path=output_path,\n                bundled=True,\n                keep_tmp=True,\n                intermediate_dir=cache_dir,\n                annotations=ann,\n                reducer_params=reducer_params,\n            )\n            pipeline = ReductionPipeline(config)\n\n            # Step 2: Annotations (cached after first run)\n            step_html.value = \"<b>Step 2/4: Fetching annotations...</b>\"\n            metadata = pipeline._fetch_annotations(embedding_sets[0].headers)\n\n            # Step 3: Dimensionality reduction\n            step_html.value = \"<b>Step 3/4: Reducing dimensions...</b>\"\n\n            # Build full metadata\n            all_headers = embedding_sets[0].headers\n            full_metadata = _pd.DataFrame({\"identifier\": all_headers})\n            if len(metadata.columns) > 1:\n                metadata = metadata.astype(str)\n                id_col = metadata.columns[0]\n                if id_col != \"identifier\":\n                    metadata = metadata.rename(columns={id_col: \"identifier\"})\n                full_metadata = full_metadata.merge(\n                    metadata.drop_duplicates(\"identifier\"),\n                    on=\"identifier\",\n                    how=\"left\",\n                )\n            metadata = full_metadata\n\n            all_reductions = pipeline._run_reductions(embedding_sets)\n\n            # Step 4: Bundle & download\n            step_html.value = \"<b>Step 4/4: Bundling & downloading...</b>\"\n            output = pipeline.base.create_output(metadata, all_reductions, all_headers)\n            pipeline.base.save_output(output, output_path, bundled=True)\n\n            kb = output_path.stat().st_size / 1024\n            total_time = _time.time() - t0\n            step_html.value = f\"<b>Done! ({kb:.0f} KB, {total_time:.0f}s)</b>\"\n            print(f\"Processed {n_proteins} proteins with {len(method_specs)} method(s)\")\n            print(f\"\\nUpload at: https://protspace.app/explore\")\n            files.download(str(output_path))\n\n        except Exception as e:\n            step_html.value = \"<b>Failed</b>\"\n            print(f\"\\nError: {e}\")\n            print(\"\\nTroubleshooting:\")\n            print(\"- Check that protein IDs are UniProt accessions (e.g., P12345)\")\n            print(\"- Try setting Annotations to 'None' to skip annotation fetching\")\n\n\ngen_btn.on_click(_on_gen)\n\ndisplay(VBox([\n    HTML(\"<h4>Methods</h4>\"),\n    HBox(list(method_toggles.values()), layout={\"flex_flow\": \"row wrap\", \"gap\": \"6px\"}),\n    no_params,\n    param_grid,\n    HTML(\"<h4>Annotations</h4>\"),\n    HBox([ann_preset, csv_upload], layout={\"gap\": \"10px\", \"align_items\": \"center\"}),\n    HTML(\"<p><i>Optional: upload a CSV/TSV with custom annotations (first column = protein IDs). \"\n         \"Taxonomy requires a one-time database download (~1 min).</i></p>\"),\n    custom_ann,\n    gen_btn,\n    HTML(\"<p><i>Processing time depends on dataset size and selected methods.</i></p>\"),\n    gen_out,\n]))"
+   "source": "# @title 3. Generate & Download {display-mode: \"form\"}\n# @markdown Configure methods and annotations, then click Generate.\n# @markdown Processing time depends on dataset size and selected methods.\n\nfrom ipywidgets import (\n    HTML,\n    Button,\n    Checkbox,\n    Dropdown,\n    FileUpload,\n    FloatSlider,\n    GridBox,\n    HBox,\n    IntSlider,\n    Output,\n    ToggleButton,\n    VBox,\n)\n\nMETHODS = [\"PCA\", \"UMAP\", \"t-SNE\", \"MDS\", \"PaCMAP\", \"LocalMAP\"]\nANNOTATIONS = {\n    \"UniProt\": [\"annotation_score\", \"cc_subcellular_location\", \"ec\", \"fragment\", \"go_bp\", \"go_cc\", \"go_mf\", \"keyword\", \"length\", \"protein_existence\", \"protein_families\", \"reviewed\", \"xref_pdb\"],\n    \"InterPro\": [\"cath\", \"cdd\", \"panther\", \"pfam\", \"prints\", \"prosite\", \"signal_peptide\", \"smart\", \"superfamily\"],\n    \"Taxonomy\": [\"root\", \"domain\", \"kingdom\", \"phylum\", \"class\", \"order\", \"family\", \"genus\", \"species\"],\n}\nANNOTATION_DEFAULTS = {\"ec\", \"keyword\", \"length\", \"protein_families\", \"reviewed\"}\n\n# --- Methods ---\nmethod_toggles = {}\nfor m in METHODS:\n    sel = m in {\"PCA\", \"UMAP\"}\n    method_toggles[m] = ToggleButton(value=sel, description=m, button_style=\"info\" if sel else \"\", layout={\"width\": \"auto\", \"height\": \"32px\"})\n\n# --- DR Parameters ---\n_s = {\"description_width\": \"110px\"}\n_l = {\"width\": \"100%\"}\n_g = {\"border\": \"1px solid #666\", \"padding\": \"6px 10px\", \"margin\": \"2px\", \"flex\": \"1 1 220px\", \"overflow\": \"hidden\"}\n\nparam_groups = []\nw_nn = IntSlider(value=25, min=2, max=500, description=\"n_neighbors:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>UMAP / PaCMAP / LocalMAP</b>\"), w_nn], layout=_g), {\"UMAP\", \"PaCMAP\", \"LocalMAP\"}))\nw_md = FloatSlider(value=0.4, min=0.0, max=0.99, step=0.01, description=\"min_dist:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>UMAP</b>\"), w_md], layout=_g), {\"UMAP\"}))\nw_mn = FloatSlider(value=0.5, min=0.0, max=1.0, step=0.1, description=\"mn_ratio:\", style=_s, layout=_l)\nw_fp = FloatSlider(value=2.0, min=0.0, max=5.0, step=0.1, description=\"fp_ratio:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>PaCMAP / LocalMAP</b>\"), w_mn, w_fp], layout=_g), {\"PaCMAP\", \"LocalMAP\"}))\nw_pp = FloatSlider(value=30.0, min=5.0, max=500.0, step=1.0, description=\"perplexity:\", style=_s, layout=_l)\nw_lr = FloatSlider(value=200.0, min=1.0, max=1000.0, step=10.0, description=\"learning_rate:\", style=_s, layout=_l)\nparam_groups.append((VBox([HTML(\"<b>t-SNE</b>\"), w_pp, w_lr], layout=_g), {\"t-SNE\"}))\n\nno_params = HTML(\"<i>PCA/MDS have no adjustable parameters.</i>\")\npw = {\"n_neighbors\": w_nn, \"min_dist\": w_md, \"perplexity\": w_pp, \"learning_rate\": w_lr, \"mn_ratio\": w_mn, \"fp_ratio\": w_fp}\nparam_grid = HBox([g for g, _ in param_groups], layout={\"flex_flow\": \"row wrap\", \"gap\": \"6px\", \"width\": \"100%\"})\n\n\ndef _update_params(_c=None):\n    sel = {m for m, t in method_toggles.items() if t.value}\n    any_vis = False\n    for gw, req in param_groups:\n        vis = bool(req & sel)\n        gw.layout.display = \"\" if vis else \"none\"\n        if vis:\n            any_vis = True\n    no_params.layout.display = \"\" if not any_vis else \"none\"\n\n\nfor tb in method_toggles.values():\n    def _toggle(c):\n        c[\"owner\"].button_style = \"info\" if c[\"new\"] else \"\"\n        _update_params()\n    tb.observe(_toggle, names=\"value\")\n_update_params()\n\n# --- Annotations ---\nann_preset = Dropdown(\n    options=[\"Default (recommended)\", \"All annotations\", \"Custom...\"],\n    value=\"Default (recommended)\",\n    description=\"Annotations:\",\n    style={\"description_width\": \"initial\"},\n)\nann_cbs = {}\nann_cols = []\nfor cat, names in ANNOTATIONS.items():\n    cbs = {n: Checkbox(value=n in ANNOTATION_DEFAULTS, description=n, indent=False, layout={\"width\": \"auto\"}) for n in names}\n    ann_cbs[cat] = cbs\n    grid = GridBox(list(cbs.values()), layout={\"grid_template_columns\": \"repeat(2, auto)\", \"grid_gap\": \"0 6px\"})\n    ann_cols.append(VBox([HTML(f\"<b>{cat}</b>\"), grid], layout={\"flex\": \"1 1 200px\", \"border\": \"1px solid #666\", \"padding\": \"6px\", \"margin\": \"2px\"}))\ncustom_ann = HBox(ann_cols, layout={\"flex_flow\": \"row wrap\"})\ncustom_ann.layout.display = \"none\"\nann_preset.observe(lambda c: setattr(custom_ann.layout, \"display\", \"\" if c[\"new\"] == \"Custom...\" else \"none\"), names=\"value\")\n\n# --- Custom CSV upload ---\ncsv_upload = FileUpload(accept=\".csv,.tsv\", multiple=False, description=\"Choose CSV/TSV\")\n\n\ndef _get_ann():\n    p = ann_preset.value\n    if p == \"Default (recommended)\":\n        return [\"default\"]\n    if p == \"All annotations\":\n        return [\"all\"]\n    sel = [n for cbs in ann_cbs.values() for n, cb in cbs.items() if cb.value]\n    return sel if sel else [\"default\"]\n\n\ndef _save_csv():\n    \"\"\"Save uploaded CSV to disk and return the path, or None.\"\"\"\n    if not csv_upload.value:\n        return None\n    fname = list(csv_upload.value.keys())[0]\n    content = csv_upload.value[fname][\"content\"]\n    Path(fname).write_bytes(content)\n    return fname\n\n\n# --- Generate ---\ngen_btn = Button(description=\"Generate\", button_style=\"primary\", icon=\"play\", layout={\"width\": \"180px\", \"height\": \"40px\"})\ngen_out = Output()\n\n\ndef _on_gen(_b):\n    with gen_out:\n        clear_output()\n        if not input_args:\n            print(\"Select input data in cell above first.\")\n            return\n        method_specs = [m.lower().replace(\"-\", \"\") + \"2\" for m, t in method_toggles.items() if t.value]\n        if not method_specs:\n            print(\"Select at least one method.\")\n            return\n\n        ann = _get_ann() or []\n        csv_path = _save_csv()\n        if csv_path:\n            ann.append(csv_path)\n        if not ann:\n            ann = None\n\n        inp = input_args\n\n        out_dir = Path(\"output\")\n        out_dir.mkdir(exist_ok=True)\n        cache_dir = out_dir / \"tmp\"\n        cache_dir.mkdir(exist_ok=True)\n        output_path = out_dir / \"data.parquetbundle\"\n\n        step_html = HTML(value=\"<b>Step 1/4: Loading embeddings...</b>\")\n        display(step_html)\n\n        try:\n            t0 = _time.time()\n            embedding_sets = []\n\n            if inp[\"type\"] == \"query\":\n                embs = _get_selected_embedders()\n                if not embs:\n                    print(\"Select at least one embedder.\")\n                    return\n                step_html.value = \"<b>Step 1/6: Downloading FASTA...</b>\"\n                headers, fasta_path = query_uniprot(inp[\"query\"])\n                if not headers:\n                    print(f\"No sequences found for query: {inp['query']}\")\n                    return\n                from protspace.data.embedding.biocentral import EmbedConfig\n                for emb_name in embs:\n                    step_html.value = f\"<b>Step 2/6: Computing {emb_name} embeddings...</b>\"\n                    emb_set = embed_fasta(fasta_path, emb_name, embed_config=EmbedConfig())\n                    emb_set.fasta_path = fasta_path\n                    embedding_sets.append(emb_set)\n            elif inp[\"type\"] == \"fasta\":\n                embs = _get_selected_embedders()\n                if not embs:\n                    print(\"Select at least one embedder.\")\n                    return\n                from protspace.data.embedding.biocentral import EmbedConfig\n                for emb_name in embs:\n                    step_html.value = f\"<b>Step 1/5: Computing {emb_name} embeddings...</b>\"\n                    emb_set = embed_fasta(Path(inp[\"path\"]), emb_name, embed_config=EmbedConfig())\n                    emb_set.fasta_path = Path(inp[\"path\"])\n                    embedding_sets.append(emb_set)\n            else:\n                h5_path = Path(inp[\"path\"])\n                name_override = inp.get(\"name\")\n                emb_set = load_h5([h5_path], name_override=name_override)\n                embedding_sets.append(emb_set)\n\n            n_proteins = len(embedding_sets[0].headers)\n\n            # Build pipeline with caching enabled\n            reducer_params = ReducerParams(\n                n_neighbors=pw[\"n_neighbors\"].value,\n                min_dist=pw[\"min_dist\"].value,\n                perplexity=pw[\"perplexity\"].value,\n                learning_rate=pw[\"learning_rate\"].value,\n                mn_ratio=pw[\"mn_ratio\"].value,\n                fp_ratio=pw[\"fp_ratio\"].value,\n            )\n            config = PipelineConfig(\n                methods=method_specs,\n                output_path=output_path,\n                bundled=True,\n                keep_tmp=True,\n                intermediate_dir=cache_dir,\n                annotations=ann,\n                reducer_params=reducer_params,\n            )\n            pipeline = ReductionPipeline(config)\n\n            # Step 2: Annotations (cached after first run)\n            step_html.value = \"<b>Step 2/4: Fetching annotations...</b>\"\n            metadata = pipeline._fetch_annotations(embedding_sets[0].headers)\n\n            # Step 3: Dimensionality reduction\n            step_html.value = \"<b>Step 3/4: Reducing dimensions...</b>\"\n\n            # Build full metadata\n            all_headers = embedding_sets[0].headers\n            full_metadata = _pd.DataFrame({\"identifier\": all_headers})\n            if len(metadata.columns) > 1:\n                metadata = metadata.astype(str)\n                id_col = metadata.columns[0]\n                if id_col != \"identifier\":\n                    metadata = metadata.rename(columns={id_col: \"identifier\"})\n                full_metadata = full_metadata.merge(\n                    metadata.drop_duplicates(\"identifier\"),\n                    on=\"identifier\",\n                    how=\"left\",\n                )\n            metadata = full_metadata\n\n            all_reductions = pipeline._run_reductions(embedding_sets)\n\n            # Step 4: Bundle & download\n            step_html.value = \"<b>Step 4/4: Bundling & downloading...</b>\"\n            output = pipeline.base.create_output(metadata, all_reductions, all_headers)\n            pipeline.base.save_output(output, output_path, bundled=True)\n\n            kb = output_path.stat().st_size / 1024\n            total_time = _time.time() - t0\n            step_html.value = f\"<b>Done! ({kb:.0f} KB, {total_time:.0f}s)</b>\"\n            print(f\"Processed {n_proteins} proteins with {len(method_specs)} method(s)\")\n            print(f\"\\nUpload at: https://protspace.app/explore\")\n            files.download(str(output_path))\n\n        except Exception as e:\n            step_html.value = \"<b>Failed</b>\"\n            print(f\"\\nError: {e}\")\n            print(\"\\nTroubleshooting:\")\n            print(\"- Check that protein IDs are UniProt accessions (e.g., P12345)\")\n            print(\"- Try setting Annotations to 'None' to skip annotation fetching\")\n\n\ngen_btn.on_click(_on_gen)\n\ndisplay(VBox([\n    HTML(\"<h4>Methods</h4>\"),\n    HBox(list(method_toggles.values()), layout={\"flex_flow\": \"row wrap\", \"gap\": \"6px\"}),\n    no_params,\n    param_grid,\n    HTML(\"<h4>Annotations</h4>\"),\n    HBox([ann_preset, csv_upload], layout={\"gap\": \"10px\", \"align_items\": \"center\"}),\n    HTML(\"<p><i>Optional: upload a CSV/TSV with custom annotations (first column = protein IDs).</i></p>\"),\n    custom_ann,\n    gen_btn,\n    HTML(\"<p><i>Processing time depends on dataset size and selected methods.</i></p>\"),\n    gen_out,\n]))"
   },
   {
    "cell_type": "markdown",

{protspace-4.0.2 → protspace-4.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "protspace"
-version = "4.0.2"
+version = "4.2.0"
 description = "A visualisation tool for protein embeddings from pLMs"
 authors = [{ name = "Tobias Senoner", email = "tobias.senoner@tum.de" }]
 readme = "README.md"
@@ -15,13 +15,12 @@ dependencies = [
     "numpy>=1.23.0",
     "pandas>=2.0.0",
     "tqdm>=4.67.1",
-    "taxopy>=0.14.0",
     "pymmseqs>=1.0.4",
     "pyarrow>=20.0.0",
-    "unipressed>=1.4.0",
     "biocentral-api>=1.1.2",
-    "typer[all]>=0.15.0",
-    "rich>=13.0.0",
+    "requests>=2.32.4",
+    "typer>=0.24.1",
+    "rich>=14.3.3",
 ]
 
 [project.optional-dependencies]
@@ -60,7 +59,6 @@ dev = [
     "seaborn>=0.13.2",
     "pytest>=8.3.3",
     "polars>=1.26.0",
-    "taxopy>=0.14.0",
     "jupytext>=1.17.1",
     "pytest-cov>=7.0.0",
 ]

protspace-4.2.0/src/protspace/__init__.py

@@ -0,0 +1 @@
+__version__ = "4.2.0"

protspace-4.2.0/src/protspace/data/annotations/retrievers/http_utils.py

@@ -0,0 +1,39 @@
+"""Shared HTTP utilities for UniProt-style REST API calls."""
+
+import logging
+
+import requests
+
+logger = logging.getLogger(__name__)
+
+API_TIMEOUT = 30
+
+
+def paginated_get(
+    url: str,
+    params: dict | None = None,
+    timeout: int = API_TIMEOUT,
+    result_key: str = "results",
+) -> list[dict]:
+    """Fetch all pages from a UniProt-style REST API endpoint.
+
+    Follows Link headers with rel="next" for automatic pagination.
+    Returns the concatenated contents of the ``result_key`` array
+    across all pages.
+    """
+    results = []
+
+    while url:
+        resp = requests.get(url, params=params, timeout=timeout)
+        resp.raise_for_status()
+        data = resp.json()
+        results.extend(data.get(result_key, []))
+
+        # Follow Link header for next page
+        link = resp.headers.get("Link", "")
+        url = None
+        params = None  # next-page URL already contains all params
+        if 'rel="next"' in link:
+            url = link.split(";")[0].strip(" <>")
+
+    return results

protspace-4.2.0/src/protspace/data/annotations/retrievers/taxonomy_retriever.py

@@ -0,0 +1,124 @@
+import logging
+from typing import Any
+
+from tqdm import tqdm
+
+from protspace.data.annotations.retrievers.base_retriever import BaseAnnotationRetriever
+from protspace.data.annotations.retrievers.http_utils import paginated_get
+
+logger = logging.getLogger(__name__)
+
+TAXONOMY_API_URL = "https://rest.uniprot.org/taxonomy/search"
+_BATCH_SIZE = 100  # Max taxon IDs per request (URL length safety)
+
+# Taxonomy annotations
+TAXONOMY_ANNOTATIONS = [
+    "root",
+    "domain",
+    "kingdom",
+    "phylum",
+    "class",
+    "order",
+    "family",
+    "genus",
+    "species",
+]
+
+
+class TaxonomyRetriever(BaseAnnotationRetriever):
+    """Retrieves taxonomy lineage data from the UniProt Taxonomy API."""
+
+    def __init__(self, taxon_ids: list[int], annotations: list = None):
+        # Don't call super().__init__() as we use taxon_ids instead of headers
+        self.taxon_ids = self._validate_taxon_ids(taxon_ids)
+        self.annotations = annotations
+
+    def fetch_annotations(self) -> dict[int, dict[str, Any]]:
+        result = {}
+
+        with tqdm(
+            total=len(self.taxon_ids),
+            desc="Fetching taxonomy annotations",
+            unit="taxon",
+        ) as pbar:
+            taxonomies_info = self._get_taxonomy_info(self.taxon_ids)
+
+            for taxon_id in self.taxon_ids:
+                if taxon_id in taxonomies_info:
+                    result[taxon_id] = {"annotations": taxonomies_info[taxon_id]}
+                else:
+                    result[taxon_id] = {
+                        "annotations": dict.fromkeys(self.annotations, "")
+                    }
+                pbar.update(1)
+
+        return result
+
+    def _validate_taxon_ids(self, taxon_ids: list[int]) -> list[int]:
+        for taxon_id in taxon_ids:
+            if not isinstance(taxon_id, int):
+                raise ValueError(f"Taxon ID {taxon_id} is not an integer")
+        return taxon_ids
+
+    def _get_taxonomy_info(self, taxon_ids: list[int]) -> dict[int, dict[str, str]]:
+        result = {}
+
+        # Fetch in batches to stay within URL length limits
+        for i in range(0, len(taxon_ids), _BATCH_SIZE):
+            batch = taxon_ids[i : i + _BATCH_SIZE]
+            batch_results = self._fetch_batch(batch)
+            result.update(batch_results)
+
+        return result
+
+    def _fetch_batch(self, taxon_ids: list[int]) -> dict[int, dict[str, str]]:
+        """Fetch taxonomy info for a batch of taxon IDs from UniProt Taxonomy API."""
+        result = {}
+        query = " OR ".join(f"id:{tid}" for tid in taxon_ids)
+
+        try:
+            entries = paginated_get(
+                TAXONOMY_API_URL,
+                params={"query": query, "format": "json", "size": "500"},
+            )
+            for entry in entries:
+                taxon_id = entry.get("taxonId")
+                if taxon_id is not None:
+                    result[taxon_id] = self._extract_taxonomy(entry)
+
+        except Exception as e:
+            logger.error(f"Failed to fetch taxonomy batch: {e}")
+            for tid in taxon_ids:
+                if tid not in result:
+                    result[tid] = dict.fromkeys(self.annotations, "")
+
+        return result
+
+    def _extract_taxonomy(self, entry: dict) -> dict[str, str]:
+        """Extract taxonomy ranks from a UniProt Taxonomy API result."""
+        lineage = entry.get("lineage", [])
+        rank_map = {item["rank"]: item["scientificName"] for item in lineage}
+
+        # The lineage only contains ancestors. If the taxon itself is a
+        # species (e.g., Human 9606), include it from the entry's own rank.
+        own_rank = entry.get("rank", "")
+        if own_rank and own_rank not in rank_map:
+            rank_map[own_rank] = entry.get("scientificName", "")
+
+        full_taxonomy_info = {
+            "root": rank_map.get("no rank", ""),
+            "domain": rank_map.get("domain", "") or rank_map.get("realm", ""),
+            "kingdom": rank_map.get("kingdom", ""),
+            "phylum": rank_map.get("phylum", ""),
+            "class": rank_map.get("class", ""),
+            "order": rank_map.get("order", ""),
+            "family": rank_map.get("family", ""),
+            "genus": rank_map.get("genus", ""),
+            "species": rank_map.get("species", ""),
+        }
+
+        # Filter based on requested taxonomy annotations
+        return {
+            annotation: full_taxonomy_info.get(annotation, "")
+            for annotation in self.annotations
+        }

{protspace-4.0.2 → protspace-4.2.0}/src/protspace/data/annotations/retrievers/uniprot_retriever.py

@@ -4,15 +4,14 @@ UniProt annotation retriever.
 This module fetches protein annotations from the UniProt API.
 """
 
-import json
 import logging
 from collections import namedtuple
 
 import requests
 from tqdm import tqdm
-from unipressed import UniprotkbClient
 
 from protspace.data.annotations.retrievers.base_retriever import BaseAnnotationRetriever
+from protspace.data.annotations.retrievers.http_utils import API_TIMEOUT, paginated_get
 from protspace.data.parsers.uniprot_parser import UniProtEntry
 
 logger = logging.getLogger(__name__)
@@ -42,15 +41,8 @@ UNIPROT_ANNOTATIONS = [
 ProteinAnnotations = namedtuple("ProteinAnnotations", ["identifier", "annotations"])
 
 
-_API_TIMEOUT = 30  # seconds per HTTP request
-
-
-def _fetch_one_with_timeout(accession: str, timeout: int = _API_TIMEOUT) -> dict:
-    """Fetch a single UniProt entry with a timeout.
-
-    The unipressed library's fetch_one() uses requests.get without a timeout,
-    which can hang indefinitely. This wrapper adds timeout protection.
-    """
+def _fetch_one_with_timeout(accession: str, timeout: int = API_TIMEOUT) -> dict:
+    """Fetch a single UniProt entry by accession with timeout protection."""
     url = f"https://rest.uniprot.org/uniprotkb/{accession}.json"
     resp = requests.get(url, timeout=timeout)
     resp.raise_for_status()
@@ -58,15 +50,9 @@ def _fetch_one_with_timeout(accession: str, timeout: int = _API_TIMEOUT) -> dict
 
 
 def _fetch_uniparc_sequence(
-    uniparc_id: str, timeout: int = _API_TIMEOUT
+    uniparc_id: str, timeout: int = API_TIMEOUT
 ) -> tuple[str, int]:
-    """Fetch sequence and length from UniParc.
-
-    Deleted UniProt entries still have their sequence archived in UniParc.
-
-    Returns:
-        Tuple of (sequence_string, length) or ("", 0) on failure.
-    """
+    """Fetch sequence and length from UniParc for deleted entries."""
     url = f"https://rest.uniprot.org/uniparc/{uniparc_id}.json"
     try:
         resp = requests.get(url, timeout=timeout)
@@ -81,6 +67,22 @@ def _fetch_uniparc_sequence(
         return "", 0
 
 
+def _fetch_many_accessions(accessions: list[str]) -> list[dict]:
+    """Fetch multiple UniProt entries by accession."""
+    return paginated_get(
+        "https://rest.uniprot.org/uniprotkb/accessions",
+        params={"accessions": ",".join(accessions)},
+    )
+
+
+def _search_sec_acc(accession: str) -> list[dict]:
+    """Search UniProt by secondary accession (fallback for inactive entries)."""
+    return paginated_get(
+        "https://rest.uniprot.org/uniprotkb/search",
+        params={"query": f"sec_acc:{accession}", "format": "json", "size": "500"},
+    )
+
+
 class UniProtRetriever(BaseAnnotationRetriever):
     """Retrieves annotations from UniProt API."""
 
@@ -207,15 +209,7 @@ class UniProtRetriever(BaseAnnotationRetriever):
             except Exception:
                 # fetch_one failed — fall back to sec_acc: search
                 try:
-                    records = []
-                    for page in UniprotkbClient.search(
-                        query=f"sec_acc:{accession}", format="json"
-                    ).each_page():
-                        content = page.read() if hasattr(page, "read") else page
-                        parsed = (
-                            json.loads(content) if isinstance(content, str) else content
-                        )
-                        records.extend(parsed.get("results", []))
+                    records = _search_sec_acc(accession)
                     if records:
                         entry = UniProtEntry(records[0])
                         annotations_dict = self._extract_annotations(entry)
@@ -277,8 +271,7 @@ class UniProtRetriever(BaseAnnotationRetriever):
                 batch = self.headers[i : i + batch_size]
 
                 try:
-                    # Fetch records using unipressed
-                    records = UniprotkbClient.fetch_many(batch)
+                    records = _fetch_many_accessions(batch)
 
                     # Parse each record and track returned identifiers
                     returned_ids = set()

{protspace-4.0.2 → protspace-4.2.0}/src/protspace/data/parsers/uniprot_parser.py

@@ -50,7 +50,8 @@ xref_pdb                 - PDB cross-references (list)
 from typing import Any
 
 import pandas as pd
-from unipressed import UniprotkbClient
+
+from protspace.data.annotations.retrievers.http_utils import paginated_get
 
 # ECO evidence code mapping (ECO ID → short human-readable code)
 ECO_TO_SHORT: dict[str, str] = {
@@ -586,9 +587,12 @@ def fetch_uniprot_data(
                 f"Invalid properties: {invalid}. Available: {AVAILABLE_PROPERTIES}"
             )
 
-    # Fetch records
-    records = UniprotkbClient.fetch_many(accessions)
-    entries = [UniProtEntry(record) for record in records]
+    # Fetch records via UniProt REST API
+    results = paginated_get(
+        "https://rest.uniprot.org/uniprotkb/accessions",
+        params={"accessions": ",".join(accessions)},
+    )
+    entries = [UniProtEntry(record) for record in results]
 
     # Extract properties
     data = []