scout-browser 4.96.0__tar.gz → 4.98.0__tar.gz

{scout_browser-4.96.0 → scout_browser-4.98.0}/CHANGELOG.md

@@ -4,6 +4,55 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 About changelog [here](https://keepachangelog.com/en/1.0.0/)
 
+## [4.98]
+### Added
+- Documentation on how to delete variants for one or more cases
+- Document the option to collect green genes from any panel when updating the PanelApp green genes panel
+- On the institute's filters page, display also any soft filters applied to institute's variants
+### Fixed
+- Case page patch for research cases without WTS outliers
+
+## [4.97]
+### Added
+- Software version and link to the relative release on GitHub on the top left dropdown menu
+- Option to sort WTS outliers by p_value, Δψ, ψ value, zscore or l2fc
+- Display pLI score and LOEUF on rare diseases and cancer SNV pages
+- Preselect MANE SELECT transcripts in the multi-step ClinVar variant add to submission process
+- Allow updating case with WTS Fraser and Outrider research files
+- Load research WTS outliers using the `scout load variants --outliers-research` command
+- Chanjo2 gene coverage completeness indicator and report from variant page, summary card
+- Enhanced SNV and SV filtering for cancer and rare disease cases, now supporting size thresholds (≥ or < a specified base pair length)
+- Option to exclude ClinVar significance status in SNVs filters form
+- Made HRD a config parameter and display it for cancer cases.
+- Preset institute-level soft filters for variants (filtering based on "filters" values on variant documents). Settings editable by admins on the institute's settings page. Allows e.g. hiding tumor `in_normal` and `germline_risk` filter status variants.
+- Load pedigree and sex check from Somalier, provided by e.g. the Nallo pipeline
+- Expand the command line to remove more types of variants. Now supports: `cancer`, `cancer_sv`, `fusion`, `mei`, `outlier`, `snv`, `str`, and `sv`.
+- New `prioritise_clinvar` checkbox on rare diseases cases, SNVs page, used by clinical filter or for expanding the search to always return variants that match the selected ClinVar conditions
+- ClinVar CLNSIG Exclude option on cancer variantS filters
+### Changed
+- Do not show overlapping gene panels badge on variants from cases runned without gene panels
+- Set case as research case if it contains any type of research variants
+- Update igv.js to 3.2.0
+- IGV DNA alignment track defaults to group by tag:HP and color by methylation (useful for LRS), and show soft-clips
+- Update gnomAD constraint to v4.1
+- HG38 genes track in igv.js browser, to correctly display gene names
+- Refactored code for prioritizing the order of variant loading
+- Modified the web pages body style to adapt content to smaller screens
+- Refactored filters to filter variants by ClinVar significance, CLINSIG Confident and ClinVar hits at the same time
+- Improved tooltips for ClinVar filter in SNVs filter form
+- `showSoftClips` parameter in igv.js is set to false by default for WES and PANEL samples
+- Updated dependencies in uv.lock file
+### Fixed
+- Don't save any "-1", "." or "0" frequency values for SNVs - same as for SVs
+- Downloading and parsing of genes from Ensembl (including MT-TP)
+- Don't parse SV frequencies for SNVs even if the name matches. Also accept "." as missing value for SV frequencies.
+- HPO search on WTS Outliers page
+- Stop using dynamic gene panel (HPO generated list) for clinical filter when the last gene is removed from the dynamic gene panel
+- Return only variants with ClinVar annotation when `ClinVar hits` checkbox is checked on variants search form
+- Legacy variant filter option `clinsig_confident_always_returned` on saved filters is remapped as `prioritised_clivar` and `clinvar_trusted_revstat`
+- Variants queries excluding ClinVar tags without `prioritise_clinvar` checkbox checked
+- Pedigree QC Somalier loading demo ancestry file and operator priority
+
 ## [4.96]
 ### Added
 - Support case status assignment upon loading (by providing case status in the case config file)
@@ -24,7 +73,6 @@ About changelog [here](https://keepachangelog.com/en/1.0.0/)
 - Avoid recurrent error by removing variant ranking settings in unranked demo case
 - Actually re-raise exception after load aborts and has rolled back variant insertion
 
-
 ## [4.95]
 ### Added
 - CCV score / temperature on case reports
@@ -1754,6 +1802,7 @@ About changelog [here](https://keepachangelog.com/en/1.0.0/)
 - Accept only numbers in managed variants filter as position and end coordinates
 - SNP id format and links in Variant page, ClinVar submission form and general report
 - Case groups tooltip triggered only when mouse is on the panel header
+- Loadable filters displayed in alphabetical order on variants page
 ### Changed
 - A more compact case groups panel
 - Added landscape orientation CSS style to cancer coverage and QC demo report

{scout_browser-4.96.0 → scout_browser-4.98.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scout-browser
-Version: 4.96.0
+Version: 4.98.0
 Summary: Clinical DNA variant visualizer and browser
 Project-URL: Repository, https://github.com/Clinical-Genomics/scout
 Project-URL: Changelog, https://github.com/Clinical-Genomics/scout/blob/main/CHANGELOG.md

scout_browser-4.98.0/docs/admin-guide/deleting-variants.md

@@ -0,0 +1,78 @@
+# Deleting Variants
+
+The Scout command line provides instructions to remove variants from one or more cases using the `scout delete variants` command.
+
+## Usage
+
+To view available options, run:
+
+```shell
+scout delete variants --help
+```
+
+This will display:
+
+```shell
+Usage: scout delete variants [OPTIONS]
+
+  Delete variants for one or more cases.
+
+Options:
+  -u, --user TEXT                 User running this command (email) [required]
+  -c, --case-id TEXT              Case ID (e.g., expertpoodle or helpedgoat)
+  -f, --case-file PATH            Path to file containing a list of case IDs
+  -i, --institute TEXT            Restrict to cases with the specified institute ID
+  --status [prioritized|inactive|active|solved|archived|ignored]
+                                  Restrict to cases with the specified status
+  --older-than INTEGER            Remove variants from cases older than (months)
+  --analysis-type [external|mixed|ogm|panel|panel-umi|unknown|wes|wgs|wts]
+                                  Restrict to cases with the specified analysis type
+  --rank-threshold INTEGER        Remove only variants with a rank lower than this threshold
+  --variants-threshold INTEGER    Remove variants only from cases containing at least this number of variants
+  --rm-ctg [cancer|cancer_sv|fusion|mei|outlier|snv|str|sv]
+                                  Remove only the specified variant categories
+  --keep-ctg [cancer|cancer_sv|fusion|mei|outlier|snv|str|sv]
+                                  Keep only the specified variant categories
+  --dry-run                       Perform a simulation without removing any variants
+  --help                          Show this message and exit.
+```
+
+### Parameter Descriptions
+
+| Option                 | Description                                                                                                                                              |
+| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--user (-u)`          | **Required.** The email of the user executing the command. The user must exist in the Scout database.                                                    |
+| `--case-id (-c)`       | **Optional.** The case ID (e.g., `expertpoodle` or `helpedgoat`).                                                                                        |
+| `--case-file (-f)`     | **Optional.** Path to a file containing a list of case IDs.                                                                                              |
+| `--institute (-i)`     | **Optional.** Institute ID. Not required if a case ID is provided.                                                                                       |
+| `--status`             | **Optional.** Restrict removal to cases with the specified status (e.g., `solved`).                                                                      |
+| `--older-than`         | **Optional.** Remove variants from cases older than the specified number of months (e.g., `12` for cases older than a year).                             |
+| `--analysis-type`      | **Optional.** Restrict removal to cases with the specified analysis type.                                                                                |
+| `--rank-threshold`     | **Optional.** Remove only variants with a rank lower than this threshold.                                                                                |
+| `--variants-threshold` | **Optional.** Remove variants only from cases containing at least this number of variants.                                                               |
+| `--rm-ctg`             | **Optional.** Remove only the specified variant categories. Example: `--rm-ctg snv --rm-ctg sv`.                                                         |
+| `--keep-ctg`           | **Optional.** Keep only the specified variant categories. Cannot be used together with `--rm-ctg`. Use multiple times to specify more than one category. |
+| `--dry-run`            | **Optional.** Runs a simulation, showing an estimate of the number of variants to be removed without actually deleting anything.                                   |
+
+> **Note:** If you are removing variants from Scout for the first time, it is strongly recommended to run a simulation first using the `--dry-run` option.
+
+### Example Command
+
+To simulate the deletion of variants that:
+
+-   Have a rank lower than `5`
+-   Belong to cases with `wgs` analysis type
+-   Contain at least `100,000` variants
+-   Are older than `24` months
+
+Run:
+
+```shell
+scout delete variants --loglevel WARNING \
+  -u youremail \
+  --rank-threshold 5 \
+  --analysis-type wgs \
+  --variants-threshold 100000 \
+  --older-than 24 \
+  --dry-run
+```

{scout_browser-4.96.0 → scout_browser-4.98.0}/docs/admin-guide/load-config.md

@@ -63,6 +63,7 @@ Below are available configuration parameters for a Scout case. Names marked with
     - **d4_path** _String_ Path to [.d4 file][d4_file]. Required for Chanjo2 integration
     - **expected_coverage** _Int_ The level of expected coverage.
     - **father** _String/Int_ Sample ID for father or 0.
+    - **hrd** _Int_ Homologous recombination deficiency.
     - **is_sma** _Bool/None_ if SMA status determined - None if not done.
     - **is_sma_carrier**  _Bool/None_  # True / False if SMA carriership determined - None if not done.
     - **mitodel** _String_ Path to mitodel file.
@@ -98,6 +99,9 @@ Below are available configuration parameters for a Scout case. Names marked with
     - **upd_sites_bed** _String_ Path to BED file to view alignments [Reference][upd].
     - **vcf2cytosure** _String_ Path to CGH file to allow download per individual. Such SV files can be visualized using standard arrayCGH analysis tools. See [vcf2cytosure](https://github.com/NBISweden/vcf2cytosure/blob/master/README.md).
 - **smn_tsv** _String_ Path to an SMN TSV file.
+- **somalier_ancestry** _String_ Path to a [Somalier][somalier] ancestry tsv file.
+- **somalier_pairs** _String_ Path to a [Somalier][somalier] pairs tsv file.
+- **somalier_samples** _String_ Path to a [Somalier][somalier] samples tsv file.
 - **status** _String_ Any of the following strings: 'prioritized', 'inactive', 'ignored', 'active', 'solved' or 'archived'.
 - **synopsis** _String_ Synopsis of case.
 - **sv_rank_model_version** _String_ Rank model that was used when scoring the variants.
@@ -146,6 +150,7 @@ vcf_snv: scout/demo/643594.clinical.vcf.gz
 [rnafusion-inspector]: https://nf-co.re/rnafusion/output#fusioninspector
 [rnafusion-report]: https://nf-co.re/rnafusion/output#fusion-report
 [rhocall]: https://github.com/dnil/rhocall
+[somalier]: https://github.com/brentp/somalier
 [srs]: https://github.com/Clinical-Genomics/Scout-REViewer-service
 [tiddit]: https://github.com/SciLifeLab/TIDDIT
 [tomte]: https://github.com/genomic-medicine-sweden/tomte

{scout_browser-4.96.0 → scout_browser-4.98.0}/docs/admin-guide/panelapp_panels.md

@@ -68,10 +68,14 @@ At the time of writing, the following panel types are available:
 12. **Research**
 13. **Submitted List**
 14. **Superpanel**
+15. **all** - all types above
 
 #### Default Behavior
 If no panel type is selected (i.e., the user presses Enter without input), Green Genes will be selected from the following default panel types: `3`, `4`, `6`, `7`, `8`, `9`, `10`.
 
+### Include all available panels
+By typing `all` at the prompt, green genes will be collected from any panel available in PanelApp
+
 #### Important Note
 The `--force` or (`-f`) parameter is required to create a new version of the gene panel if the number of green genes retrieved from the PanelApp server is lower than the number of genes in the older version of the panel. This ensures the panel is updated despite the reduction in gene count.
 

{scout_browser-4.96.0 → scout_browser-4.98.0}/mkdocs.yml

@@ -67,6 +67,7 @@ nav:
       - Updating a case without a new config: 'admin-guide/updating-case.md'
       - Updating case individuals tracks: 'admin-guide/updating-individuals.md'
       - Loading variants: 'admin-guide/loading-variants.md'
+      - Removing variants: 'admin-guide/deleting-variants.md'
       - Gene definitions: 'admin-guide/genes.md'
       - PanelApp gene panels: 'admin-guide/panelapp_panels.md'
       - HPO terms: 'admin-guide/hpo.md'

{scout_browser-4.96.0 → scout_browser-4.98.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "scout-browser"
-version = "4.96.0"
+version = "4.98.0"
 description = "Clinical DNA variant visualizer and browser"
 authors = [{name="Chiara Rasi", email="chiara.rasi@scilifelab.se"}, {name="Daniel Nilsson", email="daniel.nilsson@ki.se"}, {name="Robin Andeer", email="robin.andeer@gmail.com"}, {name="Mans Magnuson", email="monsunas@gmail.com"}]
 license = {text = "MIT License"}

{scout_browser-4.96.0 → scout_browser-4.98.0}/scout/adapter/mongo/case.py

@@ -3,6 +3,7 @@ import datetime
 import logging
 import operator
 import re
+from collections import OrderedDict
 from copy import deepcopy
 from typing import Any, Dict, List, Optional
 
@@ -11,11 +12,16 @@ from bson import ObjectId
 from werkzeug.datastructures import ImmutableMultiDict
 
 from scout.build.case import build_case
-from scout.constants import ACMG_MAP, CCV_MAP, FILE_TYPE_MAP, ID_PROJECTION, OMICS_FILE_TYPE_MAP
+from scout.constants import (
+    ACMG_MAP,
+    CCV_MAP,
+    ID_PROJECTION,
+    ORDERED_FILE_TYPE_MAP,
+    ORDERED_OMICS_FILE_TYPE_MAP,
+)
 from scout.exceptions import ConfigError, IntegrityError
 from scout.parse.variant.ids import parse_document_id
 from scout.utils.algorithms import ui_score
-from scout.utils.sort import get_load_priority
 
 LOG = logging.getLogger(__name__)
 EXISTS = "$exists"
@@ -886,11 +892,17 @@ class CaseHandler(object):
             if key in old_case:
                 new_case[key] = old_case[key]
 
-    def _load_omics_variants(self, case_obj: dict, build: str, update: bool = False):
+    def _load_clinical_omics_variants(self, case_obj: dict, build: str, update: bool = False):
         """Load omics variants. The OMICS FILE type dict contains all we need to
         determine how to load variants (type, category etc)."""
 
-        for omics_file in OMICS_FILE_TYPE_MAP.keys():
+        CLINICAL_ORDERED_OMICS_FILE_TYPE_MAP = OrderedDict(
+            (key, value)
+            for key, value in ORDERED_OMICS_FILE_TYPE_MAP.items()
+            if value["variant_type"] != "research"
+        )
+
+        for omics_file in CLINICAL_ORDERED_OMICS_FILE_TYPE_MAP.keys():
             if not case_obj["omics_files"].get(omics_file):
                 LOG.debug("didn't find %s for case, skipping", omics_file)
                 continue
@@ -901,6 +913,38 @@ class CaseHandler(object):
 
             self.load_omics_variants(case_obj=case_obj, build=build, file_type=omics_file)
 
+    def _load_clinical_variants(self, case_obj: dict, build: str, update: bool = False):
+        """Load variants in the order specified by CLINICAL_ORDERED_FILE_TYPE_MAP."""
+        CLINICAL_ORDERED_FILE_TYPE_MAP = OrderedDict(
+            (key, value)
+            for key, value in ORDERED_FILE_TYPE_MAP.items()
+            if value["variant_type"] != "research"
+        )
+        load_type_cat = set()
+        for file_name, vcf_dict in CLINICAL_ORDERED_FILE_TYPE_MAP.items():
+            if not case_obj["vcf_files"].get(file_name):
+                LOG.debug("didn't find {}, skipping".format(file_name))
+                continue
+            load_type_cat.add((vcf_dict["variant_type"], vcf_dict["category"]))
+
+        for variant_type, category in load_type_cat:
+            if update:
+                self.delete_variants(
+                    case_id=case_obj["_id"],
+                    variant_type=variant_type,
+                    category=category,
+                )
+            self.load_variants(
+                case_obj=case_obj,
+                variant_type=variant_type,
+                category=category,
+                build=build,
+                rank_threshold=case_obj.get("rank_score_threshold", 5),
+                custom_images=self._get_variants_custom_images(
+                    variant_category=category, case=case_obj
+                ),
+            )
+
     def load_case(self, config_data: dict, update: bool = False, keep_actions: bool = True) -> dict:
         """Load a case into the database
 
@@ -950,50 +994,9 @@ class CaseHandler(object):
             old_evaluated_variants = list(
                 self.evaluated_variants(case_obj["_id"], case_obj["owner"])
             )
-
-        # load from files
-        files = [
-            {
-                "file_name": file_type,
-                "variant_type": FILE_TYPE_MAP[file_type]["variant_type"],
-                "category": FILE_TYPE_MAP[file_type]["category"],
-            }
-            for file_type in FILE_TYPE_MAP.keys()
-            if FILE_TYPE_MAP[file_type]["variant_type"] != "research"
-        ]
-
-        # (type, category) tuples are not unique - eg SNV, SNV_MT
-        load_variants = set()
         try:
-            for vcf_file in files:
-                # Check if any file of this kind is configured for case
-                if not case_obj["vcf_files"].get(vcf_file["file_name"]):
-                    LOG.debug("didn't find {}, skipping".format(vcf_file["file_name"]))
-                    continue
-                load_variants.add((vcf_file["variant_type"], vcf_file["category"]))
-
-            for variant_type, category in sorted(
-                load_variants,
-                key=lambda tup: get_load_priority(variant_type=tup[0], category=tup[1]),
-            ):
-                if update:
-                    self.delete_variants(
-                        case_id=case_obj["_id"],
-                        variant_type=variant_type,
-                        category=category,
-                    )
-                self.load_variants(
-                    case_obj=case_obj,
-                    variant_type=variant_type,
-                    category=category,
-                    build=genome_build,
-                    rank_threshold=case_obj.get("rank_score_threshold", 5),
-                    custom_images=self._get_variants_custom_images(
-                        variant_category=category, case=case_obj
-                    ),
-                )
-
-                self._load_omics_variants(case_obj, build=genome_build, update=update)
+            self._load_clinical_variants(case_obj, build=genome_build, update=update)
+            self._load_clinical_omics_variants(case_obj, build=genome_build, update=update)
 
         except (IntegrityError, ValueError, ConfigError, KeyError) as error:
             LOG.exception(error)
@@ -1087,6 +1090,7 @@ class CaseHandler(object):
             Returns:
                 updated_case(dict): The updated case information
         """
+
         LOG.info("Updating case {0}".format(case_obj["_id"]))
         old_case = self.case_collection.find_one({"_id": case_obj["_id"]})
 

{scout_browser-4.96.0 → scout_browser-4.98.0}/scout/adapter/mongo/filter.py

@@ -24,10 +24,10 @@ class FilterHandler(object):
         Returns:
             filter_obj(dict)
         """
-        filter_obj = None
-        LOG.debug("Retrieve filter {}".format(filter_id))
         filter_obj = self.filter_collection.find_one({"_id": ObjectId(filter_id)})
+
         if filter_obj is not None:
+            self.set_legacy_options(filter_obj)
             # use _id to preselect the currently loaded filter, and drop it while we are at it
             filter_obj.update([("filters", filter_obj.pop("_id", None))])
         return filter_obj
@@ -49,15 +49,6 @@ class FilterHandler(object):
         Returns:
             filter_id(str) - a unique id that can be cast to ObjectId
         """
-
-        LOG.info(
-            "Stashing filter for user '%s' and institute %s.",
-            user_obj.get("email"),
-            institute_obj.get("display_name"),
-        )
-
-        LOG.info("Filter object {}".format(filter_obj))
-
         institute_id = institute_obj.get("_id")
         filter_dict = {"institute_id": institute_id, "category": category}
 
@@ -271,3 +262,29 @@ class FilterHandler(object):
         )
 
         return filters_res
+
+    def set_legacy_options(self, filter_obj):
+        """Update remaining legacy filter options,
+        i.e. filter controls that changed names or functionality.
+        In particular, clinsig_confident_always_returned was split into two different
+        options: clinvar_trusted_revstat and prioritise_clinvar.
+        """
+        if "clinsig_confident_always_returned" not in filter_obj:
+            return
+
+        filter_value = filter_obj.pop("clinsig_confident_always_returned", ["True"])
+        filter_obj["clinvar_trusted_revstat"] = filter_value
+        filter_obj["prioritise_clinvar"] = filter_value
+
+        self.filter_collection.find_one_and_update(
+            {"_id": filter_obj["_id"]},
+            {
+                "$set": {
+                    "clinvar_trusted_revstat": filter_value,
+                    "prioritise_clinvar": filter_value,
+                },
+                "$unset": {
+                    "clinsig_confident_always_returned": "",
+                },
+            },
+        )

{scout_browser-4.96.0 → scout_browser-4.98.0}/scout/adapter/mongo/institute.py

@@ -56,6 +56,7 @@ class InstituteHandler(object):
         check_show_all_vars: Optional[str] = None,
         clinvar_key: Optional[str] = None,
         clinvar_submitters: Optional[List[str]] = None,
+        soft_filters: Optional[dict] = None,
     ) -> Union[dict, str]:
         """Update the information for an institute."""
 
@@ -127,6 +128,7 @@ class InstituteHandler(object):
             "alamut_institution": alamut_institution,
             "clinvar_key": clinvar_key,
             "show_all_cases_status": show_all_cases_status,
+            "soft_filters": soft_filters,
         }
         for key, value in ADMIN_SETTINGS.items():
             if value not in [None, "", []]:

{scout_browser-4.96.0 → scout_browser-4.98.0}/scout/adapter/mongo/omics_variant.py

@@ -1,11 +1,14 @@
 import logging
 from typing import Dict, Optional
 
-from scout.constants import OMICS_FILE_TYPE_MAP
+from pymongo import ASCENDING, DESCENDING
+
+from scout.constants import ORDERED_OMICS_FILE_TYPE_MAP
 from scout.models.omics_variant import OmicsVariantLoader
 from scout.parse.omics_variant import parse_omics_file
 
 LOG = logging.getLogger(__name__)
+SORT_ORDER = {"asc": ASCENDING, "desc": DESCENDING}
 
 
 class OmicsVariantHandler:
@@ -30,7 +33,7 @@ class OmicsVariantHandler:
 
     def delete_omics_variants(self, case_id: str, file_type: str):
         """Delete OMICS variants for a case"""
-        omics_file_type = OMICS_FILE_TYPE_MAP.get(file_type)
+        omics_file_type = ORDERED_OMICS_FILE_TYPE_MAP.get(file_type)
         category = omics_file_type["category"]
         sub_category = omics_file_type["sub_category"]
         variant_type = omics_file_type["variant_type"]
@@ -123,7 +126,7 @@ class OmicsVariantHandler:
         case_panels = case_obj.get("panels", [])
         gene_to_panels = self.gene_to_panels(case_obj)
 
-        omics_file_type: dict = OMICS_FILE_TYPE_MAP.get(file_type)
+        omics_file_type: dict = ORDERED_OMICS_FILE_TYPE_MAP.get(file_type)
 
         nr_inserted = 0
 
@@ -180,9 +183,21 @@ class OmicsVariantHandler:
         else:
             nr_of_variants = skip + nr_of_variants
 
-        query = self.build_query(case_id, query=query, category=category, build=build)
+        variants_query = self.build_query(case_id, query=query, category=category, build=build)
+
+        if query.get("sort_by") and query.get("sort_order"):
+            return (
+                self.omics_variant_collection.find(variants_query, projection)
+                .sort([(query.get("sort_by"), SORT_ORDER[query.get("sort_order")])])
+                .skip(skip)
+                .limit(nr_of_variants)
+            )
+
         return self.omics_variant_collection.find(
-            query, projection, skip=skip, limit=nr_of_variants
+            variants_query,
+            projection,
+            skip=skip,
+            limit=nr_of_variants,
         )
 
     def count_omics_variants(