scout-browser 4.82.2__tar.gz → 4.84__tar.gz

{scout-browser-4.82.2 → scout-browser-4.84}/CHANGELOG.md

@@ -4,6 +4,47 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 About changelog [here](https://keepachangelog.com/en/1.0.0/)
 
+## [4.84]
+### Changed
+- Clearer error message when a loqusdb query fails for an instance that initially connected
+- Do not load chanjo-report module if not needed and more visible message when it fails loading
+- Converted the HgncGene class into a Pydantic class
+- Swap menu open and collapse indicator chevrons - down is now displayed-open, right hidden-closed
+- Linters and actions now all use python 3.11
+### Fixed
+- Safer way to update variant genes and compounds that avoids saving temporary decorators into variants' database documents
+- Link to HGNC gene report on gene page
+- Case file load priority so that e.g. SNV get loaded before SV, or clinical before research, for consistent variant_id collisions
+
+## [4.83]
+### Added
+- Edit ACMG classifications from variant page (only for classifications with criteria)
+- Events for case CLI events (load case, update case, update individual)
+- Support for loading and displaying local custom IGV tracks
+- MANE IGV track to be used as a local track for igv.js (see scout demo config file)
+- Optional separate MT VCFs, for `nf-core/raredisease`
+### Changed
+- Avoid passing verbs from CaseHandler - functions for case sample and individual in CaseEventHandler
+- Hide mtDNA report and coverage report links on case sidebar for cases with WTS data only
+- Modified OMIM-AUTO gene panel to include genes in both genome builds
+- Moved chanjo code into a dedicated extension
+- Optimise the function that collects "match-safe" genes for an institute by avoiding duplicated genes from different panels
+- Users must actively select "show matching causatives/managed" on a case page to see matching numbers
+- Upgraded python version from 3.8 to 3.11 in Docker images
+### Fixed
+- Fix several tests that relied on number of events after setup to be 0
+- Removed unused load case function
+- Artwork logo sync sketch with png and export svg
+- Clearer exception handling on chanjo-report setup - fail early and visibly
+- mtDNA report crashing when one or more samples from a case is not in the chanjo database
+- Case page crashing on missing phenotype terms
+- ACMG benign modifiers
+- Speed up tests by caching python env correctly in Github action and adding two more test groups
+- Agile issue templates were added globally to the CG-org. Adding custom issue templates to avoid exposing customers
+- PanelApp panel not saving genes with empty `EnsembleGeneIds` list
+- Speed up checking outdated gene panels
+- Do not load research variants automatically when loading a case
+
 ## [4.82.2]
 ### Fixed
 - Warning icon in case pages for individuals where `confirmed_sex` is false

scout-browser-4.84/PKG-INFO

@@ -0,0 +1,300 @@
+Metadata-Version: 2.1
+Name: scout-browser
+Version: 4.84
+Summary: Clinical DNA variant visualizer and browser.
+Home-page: https://github.com/Clinical-Genomics/scout
+Author: Måns Magnusson
+Author-email: mans.magnusson@scilifelab.se
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Programming Language :: Python :: 3.6
+Description-Content-Type: text/markdown
+Provides-Extra: coverage
+License-File: LICENSE
+
+
+<p align="center">
+	<a href="https://Clinical-Genomics.github.io/scout/">
+		<img height="165" width="637" src="artwork/logo-display.png">
+	</a>
+	<h3 align="center">Analyze VCFs and collaborate on solving rare diseases quicker</h3>
+</p>
+
+![Build Status - GitHub][actions-build-status]
+[![PyPI Version][pypi-img]][pypi-url]
+![GitHub Release Date][github-release-date]
+[![GitHub commits latest](https://img.shields.io/github/commits-since/Clinical-Genomics/scout/latest)](https://GitHub.com/Clinical-Genomics/scout/commit/)
+[![GitHub commit rate](https://img.shields.io/github/commit-activity/w/Clinical-Genomics/scout)](https://GitHub.com/Clinical-Genomics/scout/pulse/)
+[![GitHub commits](https://img.shields.io/github/commits-since/Clinical-Genomics/scout/v4.0.0.svg)](https://GitHub.com/Clinical-Genomics/scout/commit/)
+[![GitHub issues-closed][closed-issues-img]][closed-issues-url]
+[![Average time to resolve an issue][ismaintained-resolve-img]][ismaintained-resolve-url]
+[![Percentage of issues still open][ismaintained-open-rate-img]][ismaintained-open-rate-url]
+[![Coverage Status][codecov-img]][codecov-url]
+[![CodeFactor](https://www.codefactor.io/repository/github/clinical-genomics/scout/badge)](https://www.codefactor.io/repository/github/clinical-genomics/scout)
+[![Code style: black][black-image]][black-url]
+[![Woke][woke-image]][woke-url]
+
+
+## What is Scout?
+
+- **Simple** - Analyze variants in a simple to use web interface.
+- **Aggregation** - Combine results from multiple analyses and VCFs into a centralized database.
+- **Collaboration** - Write comments and share cases between users and institutes.
+
+## Documentation
+
+This README only gives a brief overview of Scout, for a more complete reference, please check out our docs: https://clinical-genomics.github.io/scout .
+
+## Runnable demo image - does not require installing of software and database
+
+A simple demo instance of Scout requires the installation of Docker and can be launched either by using the command:
+`docker-compose up -d` or `make up`.
+
+The repository includes a Makefile with common shortcuts to simplify setting up and working with Scout. To see a full list and description of these shortcuts run: `make help`.
+
+This demo is consisting of 3 containers:
+- a MongoDB instance, on the default port 27017 in the container, mapped to host port 27013
+- scout-cli --> the Scout command line, connected to the database. Populates the database with demo data
+- scout-web --> the Scout web app, that serves the app on localhost, port 8000.
+
+Once the server has started you and open the app in the web browser at the following address: http://localhost:8000/
+
+The command to stop the demo are either `docker-compose down` or `make down`.
+
+Instructions on how to run a Scout image connected to your local database or a custom database are present [on this page](docs/admin-guide/containers/container-deploy.md).
+
+## Installation
+
+<!-- You can install the latest release of Scout using `pip`:
+
+```bash
+pip install scout-browser
+
+# ... to include optional coverage tools you would use:
+pip install scout-browser[coverage]
+```
+
+If you would like to install Scout for local development: -->
+
+```bash
+git clone https://github.com/Clinical-Genomics/scout
+cd scout
+pip install --editable .
+```
+
+Scout PDF reports are created using [Flask-WeasyPrint](https://pythonhosted.org/Flask-WeasyPrint/). This library requires external dependencies which need be installed separately (namely Cairo and Pango). See platform-specific instructions for Linux, macOS and Windows available on the WeasyPrint installation [pages](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#installation).
+
+<b>NB</b>: in order to convert HTML reports into PDF reports, we have recently switched from the WeasyPrint lib to [python-pdfkit](https://github.com/JazzCore/python-pdfkit). For this reason, when upgrading to a Scout version >4.47, you need to install an additional [wkhtmltopdf system library](https://wkhtmltopdf.org/).
+
+You also need to have an instance of MongoDB running. I've found that it's easiest to do using the official Docker image:
+
+```bash
+docker run --name mongo -p 27017:27017 mongo
+```
+
+## Usage
+
+### Demo - requires pip-installing the app in a container and a running instance of mongodb
+
+Once installed, you can setup Scout by running a few commands using the included command line interface. Given you have a MongoDB server listening on the default port (27017), this is how you would setup a fully working Scout demo:
+
+```bash
+scout setup demo
+```
+
+This will setup an instance of scout with a database called `scout-demo`. Now run
+
+```bash
+scout --demo serve
+```
+And play around with the interface. A user has been created with email clark.kent@mail.com so use that address to get access
+
+### Initialize scout
+
+To initialize a working instance with all genes, diseases etc run
+
+```bash
+scout setup database
+```
+
+for more info, run `scout --help`
+
+The previous command initializes the database with a curated collection of gene definitions with links to OMIM along with HPO phenotype terms. Now we will load some example data. Scout expects the analysis to be accomplished using various gene panels so let's load one and then our first analysis case:
+
+```bash
+scout load panel scout/demo/panel_1.txt
+scout load case scout/demo/643594.config.yaml
+```
+
+## Integration with chanjo and chanjo2 for coverage report visualization
+
+Scout may be configured to visualize coverage reports produced by [Chanjo][chanjo] or [chanjo2][chanjo2]. Instructions on
+how to enable this feature can be found in the document [chanjo_coverage_integration][chanjo-scout].
+
+## Integration with loqusdb for integrating local variant frequencies
+
+Scout may be configured to visualize local variant frequencies monitored by [Loqusdb][loqusdb].
+Instructions on how to enable this feature can be found in the document
+[loqusdb integration][loqusdb-scout].
+
+## Integration with Gens for displaying copy number profiles for variants
+
+Scout may be configured to link to a local [Gens][gens] installation.
+Instructions on how to enable this feature can be found in the document
+[Gens integration][gens-scout].
+
+## Server setup
+
+Scout needs a server config to know which databases to connect to etc. Depending on which
+information you provide you activate different parts of the interface automatically,
+including user authentication, coverage, and local observations.
+
+This is an example of the config file:
+
+```python
+# scoutconfig.py
+
+# list of email addresses to send errors to in production
+ADMINS = ['paul.anderson@magnolia.com']
+
+MONGO_HOST = 'localhost'
+MONGO_PORT = 27017
+MONGO_DBNAME = 'scout'
+MONGO_USERNAME = 'testUser'
+MONGO_PASSWORD = 'testPass'
+
+# enable user authentication using Google OAuth 2.0
+GOOGLE = dict(
+   client_id="client_id_string.apps.googleusercontent.com",
+   client_secret="client_secret_string",
+   discovery_url="https://accounts.google.com/.well-known/openid-configuration"
+)
+
+# enable Phenomizer gene predictions from phenotype terms
+PHENOMIZER_USERNAME = '???'
+PHENOMIZER_PASSWORD = '???'
+
+# enable Chanjo coverage integration
+SQLALCHEMY_DATABASE_URI = '???'
+REPORT_LANGUAGE = 'en'  # or 'sv'
+
+# other interesting settings
+SQLALCHEMY_TRACK_MODIFICATIONS = False  # this is essential in production
+TEMPLATES_AUTO_RELOAD = False  			# consider turning off in production
+SECRET_KEY = 'secret key'               # override in production!
+```
+
+Most of the config settings are optional. A minimal config would consist of SECRET_KEY and MONGO_DBNAME.
+
+Starting the server in now really easy, for the demo and local development we will use the CLI:
+
+```bash
+scout --flask-config config.py serve
+```
+
+![Scout Interface demo](artwork/scout-variant-demo.png)
+
+### Hosting a production server
+
+When running the server in production you will likely want to use a proper Python server solution
+such as Gunicorn.
+This is also how we can multiprocess the server and use encrypted HTTPS connections.
+
+```bash
+SCOUT_CONFIG=./config.py gunicorn --workers 4 --bind 0.0.0.0:8080 scout.server.auto:app
+```
+
+For added security and flexibility, we recommend a reverse proxy solution like NGINX.
+
+### Setting up a user login system
+Scout currently supports 3 mutually exclusive types of login:
+- Google authentication via OpenID Connect (OAuth 2.0)
+- LDAP authentication
+- Simple authentication using userid and password
+
+The first 2 solutions are both suitable for a production server. A description on how to set up an advanced login system is available in the [admin guide](docs/admin-guide/login-system.md)
+
+
+### Integration with Matchmaker Exchange
+
+Starting from release 4.4, Scout offers integration for patient data sharing via Matchmaker
+Exchange.
+General info about Matchmaker and patient matching could be found in [this paper][matchmaker-pub].
+For a technical guideline of our implementation of Matchmaker Exchange at Clinical Genomics and its
+integration with Scout check scouts [matchmaker docs][matchmaker-scout].
+A user-oriented guide describing how to share case and variant data to Matchmaker using Scout can
+be found [here][matchmaker-scout-sharing].
+
+
+## Development
+
+To keep the code base consistent, formatting with [Black](https://github.com/psf/black) is always applied as part of the PR submission process via GitHub Actions. While not strictly required, to avoid confusion, it is suggested that developers apply Black locally.
+Black defaults to 88 characters per line, we use 100.
+
+To format all the files in the project run:
+
+```bash
+black --line-length 100 .
+```
+
+We recommend using Black with [pre-commit](https://github.com/pre-commit/pre-commit).
+In `.pre-commit-config.yaml` you can find the pre-commit configuration.
+To enable this configuration run:
+
+```bash
+pre-commit install
+```
+
+### Test
+
+To run unit tests:
+
+```bash
+pytest
+```
+
+
+### Contributing to Scout
+
+If you want to contribute and make Scout better, you help is very appreciated! Bug reports or
+feature requests are really helpful and can be submitted via github issues.
+Feel free to open a pull request to add a new functionality or fixing a bug, we welcome any help,
+regardless of the amount of code provided or your skills as a programmer.
+More info on how to contribute to the project and a description of the Scout branching workflow can
+be found in [CONTRIBUTING](CONTRIBUTING.md).
+
+
+
+[chanjo]: https://github.com/Clinical-Genomics/chanjo
+[chanjo2]: https://github.com/Clinical-Genomics/chanjo2
+[chanjo-scout]: docs/admin-guide/chanjo_coverage_integration.md
+[loqusdb]: https://github.com/moonso/loqusdb
+[loqusdb-scout]: docs/admin-guide/loqusdb_integration.md
+[gens]: https://github.com/Clinical-Genomics-Lund/gens
+[gens-scout]: docs/admin-guide/gens_integration.md
+[github-release-date]: https://img.shields.io/github/release-date/Clinical-Genomics/scout
+[matchmaker-pub]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6016856/
+[matchmaker-scout]: docs/admin-guide/matchmaker_exchange_integration.md
+[matchmaker-scout-sharing]: docs/user-guide/cases.md#matchmaker-exchange-integration
+[travis-img]: https://img.shields.io/travis/Clinical-Genomics/scout/develop.svg?style=flat-square
+[travis-url]: https://travis-ci.org/Clinical-Genomics/scout
+[pypi-img]: https://img.shields.io/pypi/v/scout-browser.svg?style=flat-square
+[pypi-url]: https://pypi.python.org/pypi/scout-browser/
+[ismaintained-resolve-img]: http://isitmaintained.com/badge/resolution/Clinical-Genomics/scout.svg
+[ismaintained-resolve-url]: http://isitmaintained.com/project/Clinical-Genomics/scout
+[ismaintained-open-rate-img]: http://isitmaintained.com/badge/open/Clinical-Genomics/scout.svg
+[ismaintained-open-rate-url]: http://isitmaintained.com/project/Clinical-Genomics/scout
+[closed-issues-img]: https://img.shields.io/github/issues-closed/Clinical-Genomics/scout.svg
+[closed-issues-url]: https://GitHub.com/Clinical-Genomics/scout/issues?q=is%3Aissue+is%3Aclosed
+[codecov-img]: https://codecov.io/gh/Clinical-Genomics/scout/branch/main/graph/badge.svg?token=BOhMTZ4T3c
+[codecov-url]: https://codecov.io/gh/Clinical-Genomics/scout
+[actions-build-status]: https://github.com/Clinical-Genomics/scout/actions/workflows/build_and_publish.yml/badge.svg
+[black-image]: https://img.shields.io/badge/code%20style-black-000000.svg
+[black-url]: https://github.com/psf/black
+[woke-image]: https://github.com/Clinical-Genomics/scout/actions/workflows/woke.yml/badge.svg
+[woke-url]: https://github.com/Clinical-Genomics/scout/actions/workflows/woke.yml

scout-browser-4.84/scout/__version__.py

@@ -0,0 +1 @@
+__version__ = "4.84"

{scout-browser-4.82.2 → scout-browser-4.84}/scout/adapter/client.py

@@ -4,6 +4,7 @@ client.py
 Establish a connection to the database
 
 """
+
 import logging
 
 from pymongo import MongoClient

{scout-browser-4.82.2 → scout-browser-4.84}/scout/adapter/mongo/base.py

@@ -79,7 +79,6 @@ class MongoAdapter(
     PhenoModelHandler,
     RankModelHandler,
 ):
-
     """Adapter for communication with a Mongo database."""
 
     def __init__(self, database=None):

{scout-browser-4.82.2 → scout-browser-4.84}/scout/adapter/mongo/case.py

@@ -10,10 +10,11 @@ import pymongo
 from bson import ObjectId
 
 from scout.build.case import build_case
-from scout.constants import ACMG_MAP, ID_PROJECTION
+from scout.constants import ACMG_MAP, FILE_TYPE_MAP, ID_PROJECTION
 from scout.exceptions import ConfigError, IntegrityError
 from scout.parse.variant.ids import parse_document_id
 from scout.utils.algorithms import ui_score
+from scout.utils.sort import get_load_priority
 
 LOG = logging.getLogger(__name__)
 
@@ -883,44 +884,37 @@ class CaseHandler(object):
                 self.evaluated_variants(case_obj["_id"], case_obj["owner"])
             )
 
+        # load from files
         files = [
-            {"file_name": "vcf_snv", "variant_type": "clinical", "category": "snv"},
-            {"file_name": "vcf_sv", "variant_type": "clinical", "category": "sv"},
             {
-                "file_name": "vcf_cancer",
-                "variant_type": "clinical",
-                "category": "cancer",
-            },
-            {
-                "file_name": "vcf_cancer_sv",
-                "variant_type": "clinical",
-                "category": "cancer_sv",
-            },
-            {"file_name": "vcf_str", "variant_type": "clinical", "category": "str"},
-            {"file_name": "vcf_mei", "variant_type": "clinical", "category": "mei"},
-            {
-                "file_name": "vcf_fusion",
-                "variant_type": "clinical",
-                "category": "fusion",
-            },
+                "file_name": file_type,
+                "variant_type": FILE_TYPE_MAP[file_type]["variant_type"],
+                "category": FILE_TYPE_MAP[file_type]["category"],
+            }
+            for file_type in FILE_TYPE_MAP.keys()
+            if FILE_TYPE_MAP[file_type]["variant_type"] != "research"
         ]
 
+        # (type, category) tuples are not unique - eg SNV, SNV_MT
+        load_variants = set()
         try:
             for vcf_file in files:
-                # Check if file exists
+                # Check if any file of this kind is configured for case
                 if not case_obj["vcf_files"].get(vcf_file["file_name"]):
                     LOG.debug("didn't find {}, skipping".format(vcf_file["file_name"]))
                     continue
+                load_variants.add((vcf_file["variant_type"], vcf_file["category"]))
 
-                variant_type = vcf_file["variant_type"]
-                category = vcf_file["category"]
+            for variant_type, category in sorted(
+                load_variants,
+                key=lambda tup: get_load_priority(variant_type=tup[0], category=tup[1]),
+            ):
                 if update:
                     self.delete_variants(
                         case_id=case_obj["_id"],
                         variant_type=variant_type,
                         category=category,
                     )
-
                 # add variants
                 self.load_variants(
                     case_obj=case_obj,
@@ -948,7 +942,7 @@ class CaseHandler(object):
                 force_update_case=True,
             )
 
-            self.update_case(case_obj)
+            self.update_case_cli(case_obj, institute_obj)
             # update Sanger status for the new inserted variants
             self.update_case_sanger_variants(institute_obj, case_obj, old_sanger_variants)
 
@@ -957,7 +951,7 @@ class CaseHandler(object):
 
         else:
             LOG.info("Loading case %s into database", case_obj["display_name"])
-            self._add_case(case_obj)
+            self.add_case(case_obj, institute_obj)
 
         return case_obj
 
@@ -972,18 +966,6 @@ class CaseHandler(object):
                 "custom_images"
             ].get(variant_category)
 
-    def _add_case(self, case_obj):
-        """Add a case to the database
-        If the case already exists exception is raised
-
-         Args:
-             case_obj(Case)
-        """
-        if self.case(case_obj["_id"], projection=ID_PROJECTION):
-            raise IntegrityError("Case %s already exists in database" % case_obj["_id"])
-
-        return self.case_collection.insert_one(case_obj)
-
     def update_case(self, case_obj, keep_date=False):
         """Update a case in the database.
         While updating the case, it compares the date of the latest analysis (case_obj["analysis_date"]) against

{scout-browser-4.82.2 → scout-browser-4.84}/scout/adapter/mongo/case_events.py

@@ -1,10 +1,12 @@
 import logging
 from collections import Counter
+from os import getlogin
 from typing import Dict, List, Optional
 
 import pymongo
 
-from scout.constants import CASE_STATUSES, CASE_TAGS
+from scout.constants import CASE_STATUSES, CASE_TAGS, ID_PROJECTION
+from scout.exceptions import IntegrityError
 
 LOG = logging.getLogger(__name__)
 
@@ -12,6 +14,100 @@ LOG = logging.getLogger(__name__)
 class CaseEventHandler(object):
     """Class to handle case events for the mongo adapter"""
 
+    def get_cli_user(self) -> dict:
+        """
+        Return a faux CLI user with a login username from OS CLI if it is available.
+        """
+        try:
+            cli_user_name = getlogin()
+        except OSError:
+            # no controlling terminal
+            cli_user_name = "CLI user"
+
+        return {"_id": "CLI", "name": cli_user_name}
+
+    def add_case(self, case_obj: dict, institute_obj: dict):
+        """Add a case to the database
+        If the case already exists exception is raised.
+        Add case will only be called from CLI, or tests, so the user will be the faux CLI user with
+        a login username from OS CLI if available.
+        """
+        if self.case(case_obj["_id"], projection=ID_PROJECTION):
+            raise IntegrityError("Case %s already exists in database" % case_obj["_id"])
+        link = f"/{case_obj['owner']}/{case_obj['display_name']}"
+
+        self.create_event(
+            institute=institute_obj,
+            case=case_obj,
+            user=self.get_cli_user(),
+            link=link,
+            category="case",
+            verb="add_case",
+            subject=case_obj["display_name"],
+        )
+
+        return self.case_collection.insert_one(case_obj)
+
+    def update_case_individual(
+        self, case_obj: dict, user_obj: dict, institute_obj: dict, link: str, keep_date: bool = True
+    ):
+        """Update case with new individual data (age and/or Tissue type) for a case
+        and create an associated event"""
+        self._update_case_component(
+            case_obj, user_obj, institute_obj, link, verb="update_individual", keep_date=keep_date
+        )
+
+    def update_case_sample(
+        self, case_obj: dict, user_obj: dict, institute_obj: dict, link: str, keep_date=True
+    ):
+        """Handle update of sample data data (tissue, tumor_type, tumor_purity) for a cancer case
+        and create an associated event"""
+        self._update_case_component(
+            case_obj, user_obj, institute_obj, link, verb="update_sample", keep_date=keep_date
+        )
+
+    def _update_case_component(
+        self,
+        case_obj: dict,
+        user_obj: Optional[dict],
+        institute_obj: dict,
+        link: str,
+        verb: str,
+        keep_date: bool = True,
+    ):
+        """Update case with new sample data, and create an associated event"""
+        self.update_case(case_obj, keep_date)
+
+        if not user_obj:
+            user_obj = self.get_cli_user()
+
+        self.create_event(
+            institute=institute_obj,
+            case=case_obj,
+            user=user_obj,
+            link=link,
+            category="case",
+            verb=verb,
+            subject=case_obj["display_name"],
+        )
+
+    def update_case_cli(self, case_obj: dict, institute_obj: dict):
+        """Update case with new case obj, and create an associated CLI user event."""
+
+        link = f"/{case_obj['owner']}/{case_obj['display_name']}"
+
+        self.create_event(
+            institute=institute_obj,
+            case=case_obj,
+            user=self.get_cli_user(),
+            link=link,
+            category="case",
+            verb="update_case",
+            subject=case_obj["display_name"],
+        )
+
+        self.update_case(case_obj)
+
     def assign(self, institute, case, user, link):
         """Assign a user to a case.
 
@@ -550,7 +646,7 @@ class CaseEventHandler(object):
         updated_diagnoses = []
         case_diagnoses = case.get("diagnosis_phenotypes") or []
 
-        if remove is True:  # Remove term from case diagnoses list
+        if remove:  # Remove term from case diagnoses list
             for case_dia in case_diagnoses:
                 if case_dia.get("disease_id") == disease_id:
                     continue