variantgrid-api 1.9.0__tar.gz → 1.11.0__tar.gz

{variantgrid_api-1.9.0/src/variantgrid_api.egg-info → variantgrid_api-1.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: variantgrid_api
-Version: 1.9.0
+Version: 1.11.0
 Summary: A Python API client for VariantGrid
 Author-email: Dave Lawrence <davmlaw@gmail.com>
 License-Expression: MIT
@@ -64,7 +64,10 @@ $ vg_api annotate_vcf input.vcf.gz -o results/        # once it's done
 Annotated vcf written to results/input.vcf_annotated_v254_GRCh38.vcf.gz
 ```
 
-Add `--verbose` to log each request, response and the full upload status to stderr.
+`-o` takes a file or a directory. A path ending in `/` is a directory, created if it doesn't exist yet, and the
+download keeps the server's filename. `--export-type csv` downloads a `.csv.zip`. Add `--compression none` for a
+plain `.csv` (needs a server at or after SACGF/variantgrid#2368). Add `--verbose` to log each request, response
+and the full upload status to stderr.
 
 From Python it's `upload_file()` / `poll_upload_status()` / `download_annotated()`, or the blocking
 `annotate_vcf()` one-liner. See
@@ -90,6 +93,21 @@ api.upload_file("sample.vcf.gz", path=None,
 A bare string names a record by its local reference. Use `ExternalReference(code=..., external_type=...)` to
 name it by a LIMS identifier instead. See `examples/example_tso500.py` for a full run.
 
+Once the sample sheet is registered, link its sequencing samples to their extractions in one call, keyed on
+sample name (what `SequencingSampleLookup.sample_name` holds):
+
+```
+api.link_sequencing_sample_extractions(sample_sheet_lookup, {"sample_1": "2600000001C",
+                                                             "sample_2": "2600000002C"})
+```
+
+It returns a list with one `{"sequencing_sample", "match_status", "match_error", "extraction"}` per sample. An
+extraction the server doesn't have yet is `Pending` for that sample only (HTTP 202) and attaches itself once the
+extraction is created; a sample name that isn't on the sheet is a 400 for the whole batch. On a server from
+before SACGF/variantgrid#2176 (no `link_extractions_bulk` feature) the client makes one
+`link_sequencing_sample_extraction` call per sample instead, so pipelines can use the bulk call unconditionally.
+`link_sequencing_sample_extraction` is still there for a single late link. See `examples/example_haem_20_999.py`.
+
 ## BAMs and CRAMs
 
 A sequencing sample can have several alignment files, eg a BAM and its recalibrated BAM, or a BAM plus a CRAM.
@@ -113,17 +131,22 @@ file as `bam_file`, and a QC's first alignment file as its `bam_file`. A CRAM th
 
 Servers of different ages accept different calls. The client asks the server which features it has
 (`GET api/v1/capabilities`, fetched once on first use), and each call that needs a newer server
-checks that list first: the patient / specimen / extraction calls, specimen measures,
-`link_sequencing_sample_extraction`, the annotate flow (`poll_upload_status`, `wait_for_annotation`,
-`download_annotated`, `annotate_vcf`), and `upload_file` with `metadata` or `file_type`. A server without the
-capabilities endpoint counts as legacy and reports no features.
+checks that list first: the patient / specimen / extraction calls,
+`link_sequencing_sample_extraction` and `link_sequencing_sample_extractions`, the annotate flow
+(`poll_upload_status`, `wait_for_annotation`, `download_annotated`, `annotate_vcf`), and `upload_file` with
+`metadata`, `file_type` or `required_features`. A server without the capabilities endpoint counts as legacy and
+reports no features. From SACGF/variantgrid#2732 the endpoint is public, so `VariantGridAPI(server, None)` can
+probe a server before you have a token.
 
 By default an unsupported call raises `UnsupportedFeatureError`. To run the same code against old and new
-servers, skip those calls instead: they log a warning and return `None`. `upload_file` is the exception: on a
-server without upload metadata it drops the `metadata` and still uploads the file.
+servers, skip those calls instead: they log a warning and return `None`. Two calls downgrade rather than skip
+when the server has the older form of the feature: `upload_file` on a server without upload metadata drops the
+`metadata` and still uploads the file, and `link_sequencing_sample_extractions` on a server with only
+`link_extraction` makes one call per sample.
 
 ```
 from variantgrid_api.api_client import VariantGridAPI, UnsupportedFeaturePolicy
+from variantgrid_api.data_models import ServerFeature
 
 api = VariantGridAPI(server, api_token, unsupported_feature_policy=UnsupportedFeaturePolicy.SKIP)
 api.create_patient(patient)        # skipped on a server without patients
@@ -131,6 +154,9 @@ api.create_patient(patient)        # skipped on a server without patients
 # Uploads only when the server has an importer for this file type
 api.upload_file("sample_CombinedVariantOutput.tsv", path=None,
                 file_type="dragen_tso500_combined_variant_output")
+
+# An SV/CNV VCF with symbolic alts (<DEL>, <DUP>, <INV>, <CNV>) - only a server that imports them (not VG3)
+api.upload_file("sample.sv.vcf.gz", path=None, required_features=[ServerFeature.SYMBOLIC_ALTS])
 ```
 
 Where the fallback isn't simply "do nothing", branch on `api.supports("feature")` or
@@ -163,10 +189,10 @@ specimen measures. See `examples/example_tso500.py` for a full run.
 ## Permission to write sequencing data
 
 Posts to the server's `seqauto/api/` endpoints - the sequencing `create_*` calls from `create_experiment` through
-`create_sequencing_data` and the QC calls, and `link_sequencing_sample_extraction` - need a superuser's API token,
-or one belonging to a member of the server's SeqAuto write group (`seqauto_api_write` unless the server's
-`SEQAUTO_API_WRITE_GROUP` setting says otherwise). Anyone else gets a 403 (`requests.HTTPError`). Reads, such as
-`sequencing_run_has_vcf()`, need only a valid token.
+`create_sequencing_data` and the QC calls, and the `link_sequencing_sample_extraction(s)` calls - need a
+superuser's API token, or one belonging to a member of the server's SeqAuto write group (`seqauto_api_write`
+unless the server's `SEQAUTO_API_WRITE_GROUP` setting says otherwise). Anyone else gets a 403
+(`requests.HTTPError`). Reads, such as `sequencing_run_has_vcf()`, need only a valid token.
 
 ## Testing
 

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/README.md

@@ -43,7 +43,10 @@ $ vg_api annotate_vcf input.vcf.gz -o results/        # once it's done
 Annotated vcf written to results/input.vcf_annotated_v254_GRCh38.vcf.gz
 ```
 
-Add `--verbose` to log each request, response and the full upload status to stderr.
+`-o` takes a file or a directory. A path ending in `/` is a directory, created if it doesn't exist yet, and the
+download keeps the server's filename. `--export-type csv` downloads a `.csv.zip`. Add `--compression none` for a
+plain `.csv` (needs a server at or after SACGF/variantgrid#2368). Add `--verbose` to log each request, response
+and the full upload status to stderr.
 
 From Python it's `upload_file()` / `poll_upload_status()` / `download_annotated()`, or the blocking
 `annotate_vcf()` one-liner. See
@@ -69,6 +72,21 @@ api.upload_file("sample.vcf.gz", path=None,
 A bare string names a record by its local reference. Use `ExternalReference(code=..., external_type=...)` to
 name it by a LIMS identifier instead. See `examples/example_tso500.py` for a full run.
 
+Once the sample sheet is registered, link its sequencing samples to their extractions in one call, keyed on
+sample name (what `SequencingSampleLookup.sample_name` holds):
+
+```
+api.link_sequencing_sample_extractions(sample_sheet_lookup, {"sample_1": "2600000001C",
+                                                             "sample_2": "2600000002C"})
+```
+
+It returns a list with one `{"sequencing_sample", "match_status", "match_error", "extraction"}` per sample. An
+extraction the server doesn't have yet is `Pending` for that sample only (HTTP 202) and attaches itself once the
+extraction is created; a sample name that isn't on the sheet is a 400 for the whole batch. On a server from
+before SACGF/variantgrid#2176 (no `link_extractions_bulk` feature) the client makes one
+`link_sequencing_sample_extraction` call per sample instead, so pipelines can use the bulk call unconditionally.
+`link_sequencing_sample_extraction` is still there for a single late link. See `examples/example_haem_20_999.py`.
+
 ## BAMs and CRAMs
 
 A sequencing sample can have several alignment files, eg a BAM and its recalibrated BAM, or a BAM plus a CRAM.
@@ -92,17 +110,22 @@ file as `bam_file`, and a QC's first alignment file as its `bam_file`. A CRAM th
 
 Servers of different ages accept different calls. The client asks the server which features it has
 (`GET api/v1/capabilities`, fetched once on first use), and each call that needs a newer server
-checks that list first: the patient / specimen / extraction calls, specimen measures,
-`link_sequencing_sample_extraction`, the annotate flow (`poll_upload_status`, `wait_for_annotation`,
-`download_annotated`, `annotate_vcf`), and `upload_file` with `metadata` or `file_type`. A server without the
-capabilities endpoint counts as legacy and reports no features.
+checks that list first: the patient / specimen / extraction calls,
+`link_sequencing_sample_extraction` and `link_sequencing_sample_extractions`, the annotate flow
+(`poll_upload_status`, `wait_for_annotation`, `download_annotated`, `annotate_vcf`), and `upload_file` with
+`metadata`, `file_type` or `required_features`. A server without the capabilities endpoint counts as legacy and
+reports no features. From SACGF/variantgrid#2732 the endpoint is public, so `VariantGridAPI(server, None)` can
+probe a server before you have a token.
 
 By default an unsupported call raises `UnsupportedFeatureError`. To run the same code against old and new
-servers, skip those calls instead: they log a warning and return `None`. `upload_file` is the exception: on a
-server without upload metadata it drops the `metadata` and still uploads the file.
+servers, skip those calls instead: they log a warning and return `None`. Two calls downgrade rather than skip
+when the server has the older form of the feature: `upload_file` on a server without upload metadata drops the
+`metadata` and still uploads the file, and `link_sequencing_sample_extractions` on a server with only
+`link_extraction` makes one call per sample.
 
 ```
 from variantgrid_api.api_client import VariantGridAPI, UnsupportedFeaturePolicy
+from variantgrid_api.data_models import ServerFeature
 
 api = VariantGridAPI(server, api_token, unsupported_feature_policy=UnsupportedFeaturePolicy.SKIP)
 api.create_patient(patient)        # skipped on a server without patients
@@ -110,6 +133,9 @@ api.create_patient(patient)        # skipped on a server without patients
 # Uploads only when the server has an importer for this file type
 api.upload_file("sample_CombinedVariantOutput.tsv", path=None,
                 file_type="dragen_tso500_combined_variant_output")
+
+# An SV/CNV VCF with symbolic alts (<DEL>, <DUP>, <INV>, <CNV>) - only a server that imports them (not VG3)
+api.upload_file("sample.sv.vcf.gz", path=None, required_features=[ServerFeature.SYMBOLIC_ALTS])
 ```
 
 Where the fallback isn't simply "do nothing", branch on `api.supports("feature")` or
@@ -142,10 +168,10 @@ specimen measures. See `examples/example_tso500.py` for a full run.
 ## Permission to write sequencing data
 
 Posts to the server's `seqauto/api/` endpoints - the sequencing `create_*` calls from `create_experiment` through
-`create_sequencing_data` and the QC calls, and `link_sequencing_sample_extraction` - need a superuser's API token,
-or one belonging to a member of the server's SeqAuto write group (`seqauto_api_write` unless the server's
-`SEQAUTO_API_WRITE_GROUP` setting says otherwise). Anyone else gets a 403 (`requests.HTTPError`). Reads, such as
-`sequencing_run_has_vcf()`, need only a valid token.
+`create_sequencing_data` and the QC calls, and the `link_sequencing_sample_extraction(s)` calls - need a
+superuser's API token, or one belonging to a member of the server's SeqAuto write group (`seqauto_api_write`
+unless the server's `SEQAUTO_API_WRITE_GROUP` setting says otherwise). Anyone else gets a 403
+(`requests.HTTPError`). Reads, such as `sequencing_run_has_vcf()`, need only a valid token.
 
 ## Testing
 

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "variantgrid_api"
-version = "1.9.0"
+version = "1.11.0"
 description = "A Python API client for VariantGrid"
 authors = [
     { name = "Dave Lawrence", email = "davmlaw@gmail.com" }

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/src/variantgrid_api/api_client.py

@@ -1,13 +1,14 @@
 import datetime
 import json
 import logging
+import os
 import re
 import time
 import urllib
 import warnings
 from enum import Enum
 from pathlib import Path
-from typing import List, Optional, Callable, Union
+from typing import Dict, Iterable, List, Optional, Callable, Union
 
 import requests
 
@@ -63,7 +64,9 @@ class VariantGridAPI:
                  unsupported_feature_policy=UnsupportedFeaturePolicy.ERROR
     ):
         self.server = server
-        self.headers = {"Authorization": f"Token {api_token}"}
+        # api_token=None sends no Authorization header - enough for the public capabilities endpoint
+        # (SACGF/variantgrid#2732); everything else needs a token
+        self.headers = {"Authorization": f"Token {api_token}"} if api_token is not None else {}
         if logger is None:
             logger = logging.getLogger(__name__)
         self.logger = logger
@@ -438,8 +441,47 @@ class VariantGridAPI:
         }
         return self._post("seqauto/api/v1/sequencing_sample/link_extraction", json_data)
 
+    def link_sequencing_sample_extractions(self, sample_sheet_lookup: SampleSheetLookup,
+                                           extractions: Dict[str, ReferenceLike]) -> Optional[List[dict]]:
+        """ link_sequencing_sample_extraction() for a whole sample sheet in one call. extractions maps each
+            sequencing sample's name (SequencingSampleLookup.sample_name) to its extraction reference.
+
+            Returns one {"sequencing_sample", "match_status", "match_error", "extraction"} per sample, in the
+            order given. match_status is per sample: an extraction the server doesn't have yet is 'Pending'
+            (the response is a 202) and attaches itself once created, without affecting the others. A sample
+            name that isn't on the sample sheet is a 400 for the whole batch - nothing is linked.
+
+            Needs ServerFeature.LINK_EXTRACTIONS_BULK (SACGF/variantgrid#2176). A server with only
+            ServerFeature.LINK_EXTRACTION gets one link_sequencing_sample_extraction() call per sample and
+            the results are collected, so pipelines can call this unconditionally. unsupported_feature_policy
+            applies only when the server has neither feature. """
+        if not self.supports(ServerFeature.LINK_EXTRACTIONS_BULK) and not self._require(ServerFeature.LINK_EXTRACTION):
+            return None
+        self._validate_object("sample_sheet_lookup", sample_sheet_lookup)
+        if not extractions:
+            self.validation_handler("extractions: empty")
+        for sample_name, extraction_reference in extractions.items():
+            self._validate_string("extractions: sample name", sample_name)
+            self._validate_reference(f"extractions['{sample_name}']", extraction_reference)
+
+        if not self.supports(ServerFeature.LINK_EXTRACTIONS_BULK):
+            self.logger.info("Server doesn't support feature '%s' - linking %d sequencing sample(s) one at a time",
+                             ServerFeature.LINK_EXTRACTIONS_BULK, len(extractions))
+            return [self.link_sequencing_sample_extraction(
+                        SequencingSampleLookup(sample_sheet_lookup=sample_sheet_lookup, sample_name=sample_name),
+                        extraction_reference)
+                    for sample_name, extraction_reference in extractions.items()]
+
+        json_data = {
+            "sample_sheet": sample_sheet_lookup.to_dict(),
+            "links": [{"sample_name": sample_name, "extraction": reference_json(extraction_reference)}
+                      for sample_name, extraction_reference in extractions.items()],
+        }
+        return self._post("seqauto/api/v1/sequencing_sample/link_extractions", json_data)
+
     def upload_file(self, filename: str, path=_UNSET, metadata: Optional[dict] = None,
-                    file_type: Optional[Union[UploadFileType, str]] = None):
+                    file_type: Optional[Union[UploadFileType, str]] = None,
+                    required_features: Optional[Iterable[Union[ServerFeature, str]]] = None):
         """ Upload a file via multipart POST to upload/api/v1/file_upload.
 
             Returns {"uploaded_file_id": <id>, "sha256_hash": <hash>, ...}; identify the upload by
@@ -467,7 +509,14 @@ class VariantGridAPI:
 
             file_type: the server's name for what this file is, eg UploadFileType.DRAGEN_TSO500_COMBINED_VARIANT_OUTPUT.
                   Not sent (the server decides from the filename) - if given, the upload only happens when
-                  accepts_upload(file_type), so an older server doesn't mis-import it as something else """
+                  accepts_upload(file_type), so an older server doesn't mis-import it as something else
+
+            required_features: server features the file needs, eg [ServerFeature.SYMBOLIC_ALTS] for an SV/CNV VCF
+                  with symbolic alts (<DEL>, <DUP>, <INV>, <CNV>). Not sent - the upload only happens when the
+                  server supports them all, otherwise unsupported_feature_policy applies (SKIP doesn't upload) """
+        for feature in required_features or ():
+            if not self._require(feature):
+                return None
         if metadata and not self.supports(ServerFeature.UPLOAD_METADATA):
             self._unsupported(f"upload metadata for '{filename}' (server doesn't support feature 'upload_metadata')")
             metadata = None
@@ -592,37 +641,65 @@ class VariantGridAPI:
             sleep(poll_interval)
 
     @staticmethod
-    def _attachment_filename(response, export_type: str) -> str:
+    def _attachment_filename(response, export_type: str, compression: str = "zip") -> str:
         content_disposition = response.headers.get("Content-Disposition", "")
         if m := re.search(r'filename\*?=(?:UTF-8\'\')?"?([^";]+)"?', content_disposition):
             return urllib.parse.unquote(m.group(1))
-        ext = ".vcf.gz" if export_type == "vcf" else ".csv.zip"
+        if export_type == "vcf":
+            ext = ".vcf.gz"
+        else:
+            ext = ".csv" if compression == "none" else ".csv.zip"
         return f"download{ext}"
 
+    DOWNLOAD_COMPRESSIONS = ("zip", "none")
+
+    def _download_compression(self, export_type: str, compression: str) -> str:
+        """ Validates compression and returns what to ask the server for. 'none' needs the server feature
+            'annotated_download_compression' - without it, SKIP falls back to 'zip' (with a warning) and ERROR raises """
+        if export_type not in ("vcf", "csv"):
+            raise ValueError(f"export_type must be 'vcf' or 'csv', got {export_type!r}")
+        if compression not in self.DOWNLOAD_COMPRESSIONS:
+            raise ValueError(f"compression must be one of {self.DOWNLOAD_COMPRESSIONS}, got {compression!r}")
+        if compression == "zip":
+            return compression
+        if export_type != "csv":
+            raise ValueError(f"compression={compression!r} applies only to export_type 'csv' - the VCF is always gzipped")
+        if not self.supports(ServerFeature.ANNOTATED_DOWNLOAD_COMPRESSION):
+            self._unsupported(f"plain CSV download (server doesn't support feature "
+                              f"'{ServerFeature.ANNOTATED_DOWNLOAD_COMPRESSION}') - downloading the zip")
+            return "zip"
+        return compression
+
     def download_annotated(self, uploaded_file_id: Optional[int] = None, sha256: Optional[str] = None,
                            export_type: str = "vcf", dest_path: Optional[Union[str, Path]] = None,
                            timeout: float = 3600, poll_interval: float = 10,
-                           sleep: Callable = time.sleep) -> Optional[Path]:
+                           sleep: Callable = time.sleep, compression: str = "zip") -> Optional[Path]:
         """ Download the cohort-level annotated export of an uploaded VCF, saving it to disk.
 
-            export_type is 'vcf' (gzipped *.vcf.gz) or 'csv' (zipped *.csv.zip). The export covers all samples
-            in the uploaded VCF (single-sample VCFs included).
+            export_type is 'vcf' (gzipped *.vcf.gz) or 'csv'. The export covers all samples in the uploaded VCF
+            (single-sample VCFs included).
+
+            compression (CSV only): 'zip' (default, *.csv.zip) or 'none' for a plain *.csv. 'none' needs the server
+            feature 'annotated_download_compression' (SACGF/variantgrid#2368) - without it, unsupported_feature_policy
+            SKIP downloads the zip (with a warning) and ERROR raises UnsupportedFeatureError. The VCF is always gzipped.
 
             The endpoint returns 202 while the file is still being generated - this polls until it is ready (200),
             then streams the attachment to dest_path. If dest_path is None the server's attachment filename is used
-            in the current directory; if it is an existing directory the attachment filename is placed inside it;
-            otherwise it is treated as the full destination path. Returns the Path written.
+            in the current directory; if it is an existing directory, or a str ending in '/' (a directory, created
+            if it doesn't exist yet), the attachment filename is placed inside it; otherwise it is treated as the
+            full destination path. Returns the Path written.
 
             Raises TimeoutError if the file isn't ready within 'timeout' seconds. Needs server feature 'upload_status' """
         if not self._require(ServerFeature.UPLOAD_STATUS):
             return None
-        if export_type not in ("vcf", "csv"):
-            raise ValueError(f"export_type must be 'vcf' or 'csv', got {export_type!r}")
+        compression = self._download_compression(export_type, compression)
         segment = self._upload_key_segment(uploaded_file_id, sha256)
         url = self._get_url(f"upload/api/v1/download/{segment}/{export_type}")
+        # Only sent for a plain CSV, so the default request is exactly what older clients sent
+        params = {"compression": compression} if compression != "zip" else None
         deadline = time.monotonic() + timeout
         while True:
-            response = requests.get(url, headers=self.headers, stream=True)
+            response = requests.get(url, params=params, headers=self.headers, stream=True)
             if response.status_code == 202:
                 if self.log_response:
                     self.logger.info("Download of '%s' still generating: %s", url, self._safe_json(response))
@@ -633,22 +710,34 @@ class VariantGridAPI:
             if not response.ok:
                 # Raises with logging (JSON 'error' message from the server)
                 self._handle_json_response(response, f"download {export_type} for {segment}")
-            dest = self._resolve_download_dest(dest_path, response, export_type)
+            dest = self._resolve_download_dest(dest_path, response, export_type, compression)
             with open(dest, "wb") as f:
                 for chunk in response.iter_content(chunk_size=8192):
                     if chunk:
                         f.write(chunk)
             return dest
 
-    def _resolve_download_dest(self, dest_path: Optional[Union[str, Path]], response, export_type: str) -> Path:
-        filename = self._attachment_filename(response, export_type)
+    def _resolve_download_dest(self, dest_path: Optional[Union[str, Path]], response, export_type: str,
+                               compression: str = "zip") -> Path:
+        filename = self._attachment_filename(response, export_type, compression)
         if dest_path is None:
             return Path(filename)
         dest = Path(dest_path)
+        if self._names_directory(dest_path):
+            # Path() drops the trailing slash, so check the string - 'results/' is a directory even if it's new
+            dest.mkdir(parents=True, exist_ok=True)
         if dest.is_dir():
             return dest / filename
         return dest
 
+    @staticmethod
+    def _names_directory(dest_path: Union[str, Path]) -> bool:
+        """ A str ending in a path separator names a directory (Path objects can't carry one) """
+        if not isinstance(dest_path, str):
+            return False
+        separators = (os.sep,) + ((os.altsep,) if os.altsep else ())
+        return dest_path.endswith(separators)
+
     @staticmethod
     def _safe_json(response):
         try:
@@ -657,20 +746,24 @@ class VariantGridAPI:
             return None
 
     def annotate_vcf(self, filename: str, export_type: str = "vcf", dest_path: Optional[Union[str, Path]] = None,
-                     timeout: float = 3600, poll_interval: float = 10, sleep: Callable = time.sleep) -> Optional[Path]:
+                     timeout: float = 3600, poll_interval: float = 10, sleep: Callable = time.sleep,
+                     compression: str = "zip") -> Optional[Path]:
         """ Convenience one-liner: upload a VCF, wait for annotation to finish, download the annotated export.
 
             Chains upload_file -> wait_for_annotation -> download_annotated and returns the Path written.
-            Needs server feature 'upload_status' - checked before uploading """
+            Needs server feature 'upload_status', and for compression='none' 'annotated_download_compression'
+            (see download_annotated) - both checked before uploading """
         if not self._require(ServerFeature.UPLOAD_STATUS):
             return None
+        compression = self._download_compression(export_type, compression)
         # path is SeqAuto-only and makes ad-hoc uploads fail the import - omit it for the annotate flow
         upload = self.upload_file(filename, path=None)
         uploaded_file_id = upload["uploaded_file_id"]
         self.wait_for_annotation(uploaded_file_id=uploaded_file_id,
                                  timeout=timeout, poll_interval=poll_interval, sleep=sleep)
         return self.download_annotated(uploaded_file_id=uploaded_file_id, export_type=export_type,
-                                       dest_path=dest_path, timeout=timeout, poll_interval=poll_interval, sleep=sleep)
+                                       dest_path=dest_path, timeout=timeout, poll_interval=poll_interval, sleep=sleep,
+                                       compression=compression)
 
 
 

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/src/variantgrid_api/cli.py

@@ -15,6 +15,7 @@ import sys
 import requests
 
 from variantgrid_api.api_client import VariantGridAPI, AnnotationError, UnsupportedFeatureError
+from variantgrid_api.data_models import ServerFeature
 
 DEFAULT_SERVER = "https://variantgrid.com"
 
@@ -116,7 +117,8 @@ def _probe_status(api, logger, sha256, verbose=False):
 
 
 def _download(api, args, sha256):
-    path = api.download_annotated(sha256=sha256, export_type=args.export_type, dest_path=args.dest)
+    path = api.download_annotated(sha256=sha256, export_type=args.export_type, dest_path=args.dest,
+                                  compression=args.compression)
     print(f"Annotated {args.export_type} written to {path}")
     return EXIT_OK
 
@@ -125,8 +127,19 @@ def annotate_vcf_cmd(args):
     if not os.path.isfile(args.vcf):
         print(f"No such file: {args.vcf}", file=sys.stderr)
         return EXIT_ERROR
+    if args.compression != "zip" and args.export_type != "csv":
+        print(f"--compression {args.compression} applies only to --export-type csv (the VCF is always gzipped)",
+              file=sys.stderr)
+        return EXIT_ERROR
 
     api, logger = _build_api(args)
+    # Checked before uploading, so we don't upload a file we then can't download as asked. A server without
+    # upload_status at all is reported by the first status poll instead
+    if (args.compression != "zip" and api.supports(ServerFeature.UPLOAD_STATUS)
+            and not api.supports(ServerFeature.ANNOTATED_DOWNLOAD_COMPRESSION)):
+        print(f"This VariantGrid server (version '{api.capabilities.version}') can't send a plain CSV - "
+              f"leave out --compression {args.compression} to download the .csv.zip", file=sys.stderr)
+        return EXIT_ERROR
     name = os.path.basename(args.vcf)
     sha256 = _sha256(args.vcf)
 
@@ -181,8 +194,13 @@ def build_parser():
     p.add_argument("vcf", help="Path to the VCF (.vcf / .vcf.gz) to annotate")
     p.add_argument("--export-type", choices=("vcf", "csv"), default="vcf",
                    help="Download format (default: vcf)")
+    p.add_argument("--compression", choices=("zip", "none"), default="zip",
+                   help="CSV download compression: zip (default, .csv.zip) or none (plain .csv - needs a server "
+                        "with the annotated_download_compression feature)")
     p.add_argument("-o", "--dest", default=".",
-                   help="Destination directory or file for the download (default: current directory)")
+                   help="Destination directory or file for the download (default: current directory). "
+                        "A path ending in '/' is a directory, created if it doesn't exist, and the download "
+                        "keeps the server's filename")
     p.add_argument("--wait", action="store_true",
                    help="Poll until annotation finishes and download it, instead of returning immediately "
                         "(can take hours)")

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/src/variantgrid_api/data_models.py

@@ -520,11 +520,17 @@ class ServerFeature(_ServerName):
         variantgrid/views_rest.py). An older server just doesn't list a newer one """
     PATIENTS = "patients"
     LINK_EXTRACTION = "link_extraction"
+    LINK_EXTRACTIONS_BULK = "link_extractions_bulk"  # sequencing_sample/link_extractions (SACGF/variantgrid#2176)
     UPLOAD_STATUS = "upload_status"
     JOINT_CALLED_VCF_CROSS_RUN = "joint_called_vcf_cross_run"
     UPLOAD_METADATA = "upload_metadata"
     CRAM_ALIGNMENT_FILES = "cram_alignment_files"  # a 'bam_file' path ending .cram is stored as a CRAM
     ALIGNMENT_FILES = "alignment_files"  # SequencingFile and QC 'alignment_files' (SACGF/variantgrid#2105)
+    # Annotated CSV download takes ?compression=zip|none, default zip (SACGF/variantgrid#2368)
+    ANNOTATED_DOWNLOAD_COMPRESSION = "annotated_download_compression"
+    # Imports VCFs with symbolic alts (<DEL>, <DUP>, <INV>, <CNV>) as structural variants - VG4, not VG3
+    # (SACGF/variantgrid#2732). Gene-level files have their own UploadFileType instead
+    SYMBOLIC_ALTS = "symbolic_alts"
 
 
 class UploadFileType(_ServerName):
@@ -553,6 +559,9 @@ class UploadFileType(_ServerName):
 class ServerCapabilities:
     """ What a server accepts, from GET api/v1/capabilities (SACGF/variantgrid_sapath#443).
 
+        A server at or after SACGF/variantgrid#2732 answers anonymous callers too, without git_hash and with
+        version lacking its commit suffix
+
         A server without that endpoint (404) is LEGACY: version 'legacy', no features, no upload file types """
     LEGACY: ClassVar["ServerCapabilities"]
 

{variantgrid_api-1.9.0 → variantgrid_api-1.11.0}/src/variantgrid_api/mock_variantgrid_api.py

@@ -19,7 +19,8 @@ from pathlib import Path
 from typing import Any, List, Optional, Tuple, Union
 
 from variantgrid_api.api_client import UnsupportedFeaturePolicy, UnsupportedFeatureError
-from variantgrid_api.data_models import reference_json, ServerCapabilities, ServerFeature, UploadFileType
+from variantgrid_api.data_models import reference_json, SequencingSampleLookup, ServerCapabilities, \
+    ServerFeature, UploadFileType
 
 _UNSET = object()
 
@@ -219,7 +220,28 @@ class MockVariantGridAPI:
             "extraction": str(reference_json(extraction_reference)),
         })
 
-    def upload_file(self, filename, path=_UNSET, metadata=None, file_type=None):
+    def link_sequencing_sample_extractions(self, sample_sheet_lookup, extractions):
+        """ As the real client, a server without 'link_extractions_bulk' gets one
+            link_sequencing_sample_extraction call per sample - those are what's recorded, not this call """
+        if not self.supports(ServerFeature.LINK_EXTRACTIONS_BULK):
+            if not self._require(ServerFeature.LINK_EXTRACTION):
+                return None
+            return [self.link_sequencing_sample_extraction(
+                        SequencingSampleLookup(sample_sheet_lookup=sample_sheet_lookup, sample_name=sample_name),
+                        extraction_reference)
+                    for sample_name, extraction_reference in extractions.items()]
+        self._record("link_sequencing_sample_extractions", sample_sheet_lookup, extractions)
+        return self._ret("link_sequencing_sample_extractions", [{
+            "sequencing_sample": sample_name,
+            "match_status": "Matched",
+            "match_error": None,
+            "extraction": str(reference_json(extraction_reference)),
+        } for sample_name, extraction_reference in extractions.items()])
+
+    def upload_file(self, filename, path=_UNSET, metadata=None, file_type=None, required_features=None):
+        for feature in required_features or ():
+            if not self._require(feature):
+                return None
         if metadata and not self.supports(ServerFeature.UPLOAD_METADATA):
             self._unsupported(f"upload metadata for '{filename}' (server doesn't support feature 'upload_metadata')")
             metadata = None
@@ -231,6 +253,8 @@ class MockVariantGridAPI:
         extra = {"metadata": metadata} if metadata is not None else {}
         if file_type is not None:
             extra["file_type"] = file_type
+        if required_features is not None:
+            extra["required_features"] = required_features
         self._record("upload_file", filename, path=path, **extra)
         return self._ret("upload_file", {"uploaded_file_id": 1, "sha256_hash": "deadbeef",
                                          "path": path, "status": "ok"})
@@ -259,21 +283,42 @@ class MockVariantGridAPI:
             "error": None,
         })
 
+    def _download_compression(self, export_type, compression):
+        """ Same validation and gating as VariantGridAPI._download_compression """
+        if export_type not in ("vcf", "csv"):
+            raise ValueError(f"export_type must be 'vcf' or 'csv', got {export_type!r}")
+        if compression not in ("zip", "none"):
+            raise ValueError(f"compression must be one of ('zip', 'none'), got {compression!r}")
+        if compression == "zip":
+            return compression
+        if export_type != "csv":
+            raise ValueError(f"compression={compression!r} applies only to export_type 'csv' - the VCF is always gzipped")
+        if not self.supports(ServerFeature.ANNOTATED_DOWNLOAD_COMPRESSION):
+            self._unsupported(f"plain CSV download (server doesn't support feature "
+                              f"'{ServerFeature.ANNOTATED_DOWNLOAD_COMPRESSION}') - downloading the zip")
+            return "zip"
+        return compression
+
     def download_annotated(self, uploaded_file_id=None, sha256=None, export_type="vcf",
-                           dest_path=None, timeout=3600, poll_interval=10, sleep=None):
+                           dest_path=None, timeout=3600, poll_interval=10, sleep=None, compression="zip"):
         if not self._require(ServerFeature.UPLOAD_STATUS):
             return None
+        compression = self._download_compression(export_type, compression)
+        # compression only recorded when not the default, so existing assertions on the recorded kwargs still hold
+        extra = {"compression": compression} if compression != "zip" else {}
         self._record("download_annotated", uploaded_file_id, sha256, export_type=export_type,
-                     dest_path=dest_path, timeout=timeout, poll_interval=poll_interval, sleep=sleep)
+                     dest_path=dest_path, timeout=timeout, poll_interval=poll_interval, sleep=sleep, **extra)
         default = Path(dest_path) if dest_path is not None else Path(f"download.{export_type}")
         return self._ret("download_annotated", default)
 
     def annotate_vcf(self, filename, export_type="vcf", dest_path=None,
-                     timeout=3600, poll_interval=10, sleep=None):
+                     timeout=3600, poll_interval=10, sleep=None, compression="zip"):
         if not self._require(ServerFeature.UPLOAD_STATUS):
             return None
+        compression = self._download_compression(export_type, compression)
+        extra = {"compression": compression} if compression != "zip" else {}
         self._record("annotate_vcf", filename, export_type=export_type, dest_path=dest_path,
-                     timeout=timeout, poll_interval=poll_interval, sleep=sleep)
+                     timeout=timeout, poll_interval=poll_interval, sleep=sleep, **extra)
         default = Path(dest_path) if dest_path is not None else Path(f"download.{export_type}")
         return self._ret("annotate_vcf", default)