pmhclib 0.9.0__tar.gz

pmhclib-0.9.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2023 David Wales, Jonathan Stucken, South Western Sydney Primary
+Health Network
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pmhclib-0.9.0/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: pmhclib
+Version: 0.9.0
+Summary: Python wrapper for unofficial PMHC MDS portal API
+License-File: LICENSE
+Author: David Wales
+Author-email: david.wales@swsphn.com.au
+Requires-Python: >=3.10
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: beautifulsoup4 (>=4.12.3,<5.0.0)
+Requires-Dist: pyotp (>=2.9.0,<3.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Requires-Dist: rich (>=13.9.4,<14.0.0)
+Description-Content-Type: text/markdown
+
+# pmhclib
+
+pmhclib is a Python wrapper for the unofficial PMHC MDS portal API. You
+can use it to automate uploads and downloads to the PMHC portal.
+
+## Install
+
+`pmhclib` is a Python package. You should be able to install
+it directly with `pip` or `poetry`:
+
+``` sh
+pip install pmhclib@git+https://github.com/swsphn/pmhclib.git
+```
+
+OR
+
+``` sh
+poetry add pmhclib@git+https://github.com/swsphn/pmhclib.git
+```
+
+This will install `pmhclib` as an importable Python library.
+
+
+## Usage
+
+`pmhclib.PMHC` is intended to be used with a context manager. This
+ensures that the requests session which performs the login process and the
+API requests is correctly closed when the script exits. The standard
+use pattern is as follows:
+
+``` python
+from pmhclib import PMHC
+with PMHC('PHN105') as pmhc:
+    pmhc.login()
+    ...
+    # other pmhc methods.
+```
+
+`pmhc.login()` will read credentials from the following environment
+variables if they are set:
+
+```
+PMHC_USERNAME
+PMHC_PASSWORD
+PMHC_TOTP_SECRET
+```
+
+Otherwise, you will be prompted for credentials interactively.
+
+In PowerShell, you can set the environment variables interactively as
+follows:
+
+``` ps1
+$env:PMHC_USERNAME='your_username_here'
+$env:PMHC_PASSWORD=python -c 'import getpass; print(getpass.getpass())'
+$env:PMHC_TOTP_SECRET=python -c 'import getpass; print(getpass.getpass("TOTP Secret: "))'
+```
+
+In a Unix shell (Mac, Linux), you can do:
+
+``` bash
+export PMHC_USERNAME='your_username_here'
+read -rs PMHC_PASSWORD && export PMHC_PASSWORD
+read -rs PMHC_TOTP_SECRET && export PMHC_TOTP_SECRET
+```
+
+NOTE: `PMHC_TOTP_SECRET` is the unchanging base32-encoded TOTP secret,
+not the time-based six-digit code. You can likely find this secret in
+the 'advanced' section of your TOTP app. It will be a long string of
+upper-case letters and digits. See below for a list of TOTP apps which
+support viewing the TOTP secret. It is also possible to get the secret
+by scanning the setup QR code with a generic QR code reader, or by
+clicking the _Trouble scanning_ button on the website to manually
+configure the TOTP app:
+
+![manually view totp secret](./images/view-totp-secret-at-setup.png)
+
+The six-digit code will be automatically calculated based on the current
+time as required if `PMHC_TOTP_SECRET` is specified. Otherwise, the user
+will be prompted to enter the current six-digit code.
+
+Not all TOTP apps support viewing the secret. The following are known
+to support this:
+
+- [Aegis Authenticator](https://getaegis.app/) (Android only)
+- [Bitwarden
+  Authenticator](https://bitwarden.com/products/authenticator/)
+- [Ente Auth](https://github.com/ente-io/ente/tree/main/auth#readme)
+- [2FA Authenticator (2FAS)](https://2fas.com/)
+
+For more details, see the [list of recommended authenticator
+apps][mfa-apps] on our Data Wiki.
+
+## Documentation
+
+See the [online documentation][docs].
+
+### Built-in docs
+
+Review the built-in documentation from Python as follows:
+
+``` python
+>>> from pmhclib import PMHC
+>>> help(PMHC)
+```
+
+### Build docs
+
+You can also generate html documentation locally using [Sphinx][] if you
+have a local copy of the repository.
+
+Linux:
+
+```
+cd docs
+make html
+```
+
+PowerShell:
+
+```
+cd docs
+./make.bat html
+```
+
+The generated documentation can be viewed at `docs/_build/html/index.html`.
+
+[Sphinx]: https://www.sphinx-doc.org/
+[docs]: https://swsphn.github.io/pmhclib/
+[mfa-apps]: https://datawiki.swsphn.com.au/software/gui-tools/multi-factor-authentication-apps/
+

pmhclib-0.9.0/README.md

@@ -0,0 +1,130 @@
+# pmhclib
+
+pmhclib is a Python wrapper for the unofficial PMHC MDS portal API. You
+can use it to automate uploads and downloads to the PMHC portal.
+
+## Install
+
+`pmhclib` is a Python package. You should be able to install
+it directly with `pip` or `poetry`:
+
+``` sh
+pip install pmhclib@git+https://github.com/swsphn/pmhclib.git
+```
+
+OR
+
+``` sh
+poetry add pmhclib@git+https://github.com/swsphn/pmhclib.git
+```
+
+This will install `pmhclib` as an importable Python library.
+
+
+## Usage
+
+`pmhclib.PMHC` is intended to be used with a context manager. This
+ensures that the requests session which performs the login process and the
+API requests is correctly closed when the script exits. The standard
+use pattern is as follows:
+
+``` python
+from pmhclib import PMHC
+with PMHC('PHN105') as pmhc:
+    pmhc.login()
+    ...
+    # other pmhc methods.
+```
+
+`pmhc.login()` will read credentials from the following environment
+variables if they are set:
+
+```
+PMHC_USERNAME
+PMHC_PASSWORD
+PMHC_TOTP_SECRET
+```
+
+Otherwise, you will be prompted for credentials interactively.
+
+In PowerShell, you can set the environment variables interactively as
+follows:
+
+``` ps1
+$env:PMHC_USERNAME='your_username_here'
+$env:PMHC_PASSWORD=python -c 'import getpass; print(getpass.getpass())'
+$env:PMHC_TOTP_SECRET=python -c 'import getpass; print(getpass.getpass("TOTP Secret: "))'
+```
+
+In a Unix shell (Mac, Linux), you can do:
+
+``` bash
+export PMHC_USERNAME='your_username_here'
+read -rs PMHC_PASSWORD && export PMHC_PASSWORD
+read -rs PMHC_TOTP_SECRET && export PMHC_TOTP_SECRET
+```
+
+NOTE: `PMHC_TOTP_SECRET` is the unchanging base32-encoded TOTP secret,
+not the time-based six-digit code. You can likely find this secret in
+the 'advanced' section of your TOTP app. It will be a long string of
+upper-case letters and digits. See below for a list of TOTP apps which
+support viewing the TOTP secret. It is also possible to get the secret
+by scanning the setup QR code with a generic QR code reader, or by
+clicking the _Trouble scanning_ button on the website to manually
+configure the TOTP app:
+
+![manually view totp secret](./images/view-totp-secret-at-setup.png)
+
+The six-digit code will be automatically calculated based on the current
+time as required if `PMHC_TOTP_SECRET` is specified. Otherwise, the user
+will be prompted to enter the current six-digit code.
+
+Not all TOTP apps support viewing the secret. The following are known
+to support this:
+
+- [Aegis Authenticator](https://getaegis.app/) (Android only)
+- [Bitwarden
+  Authenticator](https://bitwarden.com/products/authenticator/)
+- [Ente Auth](https://github.com/ente-io/ente/tree/main/auth#readme)
+- [2FA Authenticator (2FAS)](https://2fas.com/)
+
+For more details, see the [list of recommended authenticator
+apps][mfa-apps] on our Data Wiki.
+
+## Documentation
+
+See the [online documentation][docs].
+
+### Built-in docs
+
+Review the built-in documentation from Python as follows:
+
+``` python
+>>> from pmhclib import PMHC
+>>> help(PMHC)
+```
+
+### Build docs
+
+You can also generate html documentation locally using [Sphinx][] if you
+have a local copy of the repository.
+
+Linux:
+
+```
+cd docs
+make html
+```
+
+PowerShell:
+
+```
+cd docs
+./make.bat html
+```
+
+The generated documentation can be viewed at `docs/_build/html/index.html`.
+
+[Sphinx]: https://www.sphinx-doc.org/
+[docs]: https://swsphn.github.io/pmhclib/
+[mfa-apps]: https://datawiki.swsphn.com.au/software/gui-tools/multi-factor-authentication-apps/

pmhclib-0.9.0/pyproject.toml

@@ -0,0 +1,27 @@
+[tool.poetry]
+name = "pmhclib"
+version = "0.9.0"
+description = "Python wrapper for unofficial PMHC MDS portal API"
+authors = [
+    "David Wales <david.wales@swsphn.com.au>",
+    "Jonathan Stucken <jonathan.stucken@swsphn.com.au>"
+]
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = ">=3.10"
+rich = "^13.9.4"
+pyotp = "^2.9.0"
+requests = "^2.32.3"
+beautifulsoup4 = "^4.12.3"
+
+[tool.poetry.group.docs.dependencies]
+sphinx = "^7.0.0"
+myst-parser = "^2.0.0"
+sphinx-autoapi = "^3.0.0"
+sphinx-rtd-theme = "^2.0.0"
+typing-extensions = "^4.12.2"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

pmhclib-0.9.0/src/pmhclib/__init__.py

@@ -0,0 +1,10 @@
+# read version from installed package
+from importlib.metadata import version
+__version__ = version("pmhclib")
+
+# Import classes to be available at the top level.
+# Allows importing with:
+#     from pmhclib import PMHC
+# rather than
+#     from pmhclib.pmhc import PMHC
+from .pmhc import PMHC, PMHCSpecification

pmhclib-0.9.0/src/pmhclib/pmhc.py

@@ -0,0 +1,675 @@
+"""
+This class provides a wrapper around the unofficial PMHC internal API.
+It is useful for automating uploads and downloads from the PMHC portal.
+
+No login details are saved anywhere
+To speed up usage when doing repeated calls, create the following local env variables:
+PMHC_USERNAME
+PMHC_PASSWORD
+PMHC_TOTP_SECRET
+
+Note: See PMHC.login() documentation for details about `PMHC_TOTP_SECRET`.
+"""
+
+import functools
+import logging
+import mimetypes
+import os
+import time
+from dataclasses import dataclass
+from datetime import date, timedelta
+from enum import Enum, unique
+from getpass import getpass
+from pathlib import Path
+from typing import Optional
+from urllib.parse import urlsplit, parse_qs
+
+import pyotp
+import requests
+from bs4 import BeautifulSoup
+from rich.progress import Progress, TimeElapsedColumn
+
+
+class FileNotFoundException(Exception):
+    """Custom error handler for when no file is found"""
+
+
+class IncorrectFileType(Exception):
+    """Custom error handler for when an incorrect file is provided"""
+
+
+class InvalidPmhcUser(Exception):
+    """Custom error handler for when a PMHC login is unsuccessful"""
+
+
+class CouldNotFindPmhcUpload(Exception):
+    """Custom error handler for when an upload cannot be found on PMHC"""
+
+
+class PmhcServerError(Exception):
+    """Custom exception for when a PMHC Server error is encountered"""
+
+
+class MaxRetriesExceeded(Exception):
+    """Custom exception for when the maximum retries is exceeded"""
+
+
+class SecureString(str):
+    """Show `'***'` instead of string value in tracebacks"""
+
+    def __repr__(self):
+        return "***"
+
+
+@dataclass
+class PMHCSpecificationRepresentation:
+    """Dataclass which provides structure for PMHCSpecification Enum."""
+
+    term: str
+    filter_term: str
+
+
+@unique
+class PMHCSpecification(PMHCSpecificationRepresentation, Enum):
+    """Enum of valid PMHC specifications"""
+
+    ALL = "meta", "Include data from all specifications"
+    HEADSPACE = "headspace", "headspace 4.1"
+    PMHC = "pmhc", "PMHC 5.0"
+    SURVEY = "survey", "SURVEY 1.0"
+
+
+class PMHC:
+    """This class wraps the unofficial PMHC API.
+
+    Use it to automate tasks such as uploading to the PMHC website,
+    downloading error reports, downloading PMHC extracts, etc.
+
+    Usage:
+
+    This class is intended to be used with a context manager. This ensures
+    that the requests session is correctly closed. For
+    example:
+
+    >>> with PMHC('PHN105') as pmhc:
+    ...     pmhc.login()
+    ...     pmhc.download_error_json('94edf5e3-36b1-46d3-9178-bf3b142da6a1')
+
+    Args:
+        organisation_path: Your organisation's PMHC organisation_path
+    """
+
+    default_timeout = 60  # seconds
+
+    def __enter__(self):
+        """Initialise requests session. (Called automatically by context
+        manager.)
+        """
+        s = requests.Session()
+        # Ensure requests eventually timeout
+        # https://github.com/psf/requests/issues/2011#issuecomment-490050252
+        s.request = functools.partial(s.request, timeout=self.default_timeout)
+        self.s = s
+        return self  # Return the instance of this class
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        """Close requests session. (Called automatically when context
+        manager exits.)
+        """
+        # exc_type, exc_value, and traceback are required parameters in __exit__()
+        self.s.close()
+
+    def __init__(self, organisation_path: str):
+        # user_info is set by login()
+        self.user_info = None
+        self.organisation_path = organisation_path
+
+    def login(
+        self,
+        username: str | None = None,
+        password: str | None = None,
+        totp_secret: str | None = None,
+    ):
+        """Logs in to PMHC website. This allows us to reuse the login the session
+        across other class methods.
+
+        Set the following environment variables to skip interactive login prompt:
+
+        - `PMHC_USERNAME`
+        - `PMHC_PASSWORD`
+        - `PMHC_TOTP_SECRET`
+
+        NOTE: `PMHC_TOTP_SECRET` is _not_ the 6 digit time-dependent TOTP code,
+        but rather the long base32 encoded random secret. You might find this in
+        the 'advanced' section when editing the record in your TOTP app. It will
+        likely be a long string containing uppercase letters and numbers. This
+        will be automatically combined with the current time to derive the
+        correct 6 digit code.
+
+        Args:
+            username: PMHC username
+            password: PMHC password
+            totp_secret: static base32 encoded random totp secret Used to
+                generate the 6-digit time-based totp code. See note above.
+        """
+
+        pmhc_auth_url = "https://pmhc-mds.net/api/auth/login"
+        pmhc_login_url = "https://pmhc-mds.net/api/current-user"
+
+        # Prompt user for credentials if not set in env.
+        username = username or os.getenv("PMHC_USERNAME")
+        password = SecureString(password or os.getenv("PMHC_PASSWORD") or "")
+        totp_secret = SecureString(totp_secret or os.getenv("PMHC_TOTP_SECRET") or "")
+
+        while not username:
+            username = input("Enter PMHC username: ")
+
+        while not password:
+            password = SecureString(
+                getpass("Enter PMHC password (keyboard input will be hidden): ")
+            )
+
+        def extract_state(url: str) -> str:
+            """Extract state parameter from url query"""
+            return parse_qs(urlsplit(url).query)["state"][0]
+
+        initial_request = self.s.get(pmhc_auth_url)
+
+        # After redirects, the URL needed for the next POST is initial_request.url
+        identifier_url = initial_request.url
+        identifier_request = self.s.post(
+            identifier_url,
+            data={
+                "state": extract_state(identifier_url),
+                "username": username,
+            },
+        )
+
+        password_url = identifier_request.url
+        password_request = self.s.post(
+            password_url,
+            data={
+                "state": extract_state(password_url),
+                "username": username,
+                "password": password,
+            },
+        )
+
+        mfa_detect_url = password_request.url
+        mfa_detect_request = self.s.post(
+            mfa_detect_url,
+            data={
+                "state": extract_state(mfa_detect_url),
+                "action": "default",
+                "js-available": "false",
+                "webauthn-available": "true",
+                " is-brave": "false",
+                "webauthn-platform-available": "false",
+            },
+        )
+
+        mfa_url = mfa_detect_request.url
+
+        # Detect invalid username or password error
+        # If the username and password are correct, we should be
+        # redirected to /u/mfa-otp-challenge.
+        # If we are still on /u/login/password, then the username and
+        # password are probably invalid.
+        if mfa_url.startswith("https://login.logicly.com.au/u/login/password"):
+            logging.debug("Got password URL instead of expected MFA URL:")
+            logging.debug(f"{mfa_url}")
+            error_soup = BeautifulSoup(password_request.text, "html.parser")
+            error_message = error_soup.select_one(
+                'span[id="error-element-password"]'
+            ).get_text()
+            raise InvalidPmhcUser(
+                f"Did not reach expected OTP page. Error message:\n{error_message}"
+            )
+
+        # Note: We get the code _after_ loading the page and entering
+        # the username and password, to ensure that it is still valid
+        # when we submit it.
+        logging.info("Entering TOTP MFA code")
+        if totp_secret:
+            totp = pyotp.TOTP(totp_secret)
+            totp_code = SecureString(totp.now())
+        else:
+            totp_code = None
+            while not totp_code:
+                totp_code = SecureString(
+                    getpass(
+                        "Enter six-digit MFA code (keyboard input will be hidden): "
+                    )
+                )
+
+        mfa_request = self.s.post(
+            mfa_url,
+            data={
+                "state": extract_state(mfa_url),
+                "code": totp_code,
+            },
+        )
+
+        # confirm login was successful
+        user_query = self.s.get("https://pmhc-mds.net/api/current-user")
+        self.user_info = user_query.json()
+
+        # error key will be present if login was unsuccessful
+        if "error" in self.user_info:
+            raise InvalidPmhcUser(
+                "PMHC login was unsuccessful. Are you sure you entered "
+                "correct credentials?"
+            )
+
+    def upload_file(
+        self,
+        input_file: Path,
+        test: bool = True,
+    ) -> Path:
+        """Uploads a user specified file to PMHC website.
+
+        Args:
+            input_file: Path to the file e.g.
+                `'PMHC_MDS_20230101_20230131.xlsx'`
+            test: Upload in 'test' or 'live' mode? Defaults to `True`
+                ('test'). Use `False` ('live') with care!
+
+        Raises:
+            IncorrectFileType: If user uploads a bad filetype
+            FileNotFoundException: If we cannot find user file
+
+        Returns:
+            Filename of the new file we generated for matching purposes
+        """
+
+        # check file looks ok
+        if input_file.suffix not in (".xlsx", ".zip"):
+            raise IncorrectFileType(
+                "Only .xlsx or .zip (containing multiple csv's) are acceptable PMHC "
+                "input files"
+            )
+
+        if not input_file.exists():
+            raise FileNotFoundException(
+                "Input file does not exist - please check the file path and try again"
+            )
+
+        # check no uploads are currently being processed
+        # PMHC only allows one upload at a time per user account.
+        # This usually only occurs if the user is also using their browser to upload
+        # manually at the same time as running this script
+        self.wait_for_upload()
+
+        mode = "test" if test else "live"
+        print(
+            f"Uploading '{input_file}' to PMHC as a '{mode}' file\n"
+            "It usually takes approx 3-10 minutes for PMHC to process xlsx files "
+            "depending on the number of months included in the data, less for zipped "
+            "csv files"
+        )
+
+        # First PUT the file and receive a uuid
+        with open(input_file, "rb") as file:
+            upload_response = self.s.put(
+                "https://uploader.strategicdata.com.au/upload",
+                files={
+                    "file": (
+                        input_file.name,  # file name
+                        file,  # file object
+                        mimetypes.guess_type(input_file)[0],  # content type
+                    )
+                },
+            )
+
+        upload_status = upload_response.json()
+        logging.debug("Upload status:")
+        logging.debug(upload_status)
+
+        uuid = upload_status["id"]
+
+        # Second POST the upload details
+        # This is required to register the upload with the PMHC portal
+        post_response = self.s.post(
+            f"https://pmhc-mds.net/api/organisations/{self.organisation_path}/uploads",
+            json={
+                "uuid": uuid,
+                "filename": input_file.name,
+                "test": test,
+                "encoded_organisation_path": self.organisation_path,
+            },
+        )
+        logging.info("Upload details POST response:")
+        logging.info(post_response)
+        logging.info(post_response.text)
+
+        return uuid
+
+    def wait_for_upload(self):
+        """Waits for a PMHC upload to complete processing in 'test' mode"""
+
+        # check to see if the PMHC upload queue is free
+        delay = 10
+        with Progress(*Progress.get_default_columns(), TimeElapsedColumn()) as progress:
+            processing_task = progress.add_task(
+                "Checking PMHC upload queue...", total=None
+            )
+            while self.is_upload_processing():
+                progress.update(
+                    processing_task, description="Waiting for PMHC processing..."
+                )
+                time.sleep(delay)
+
+    def download_error_json(self, uuid: str, download_folder: Path = Path(".")) -> Path:
+        """Downloads a JSON error file from PMHC
+        This is useful for matching against uploaded files and processing
+
+        Args:
+            uuid: PMHC upload uuid from View Uploads page. For
+                example: `'94edf5e3-36b1-46d3-9178-bf3b142da6a1'`.
+                The uuid is found in the URL to the upload summary page.
+            download_folder: Location to save the downloaded error
+                JSON.
+
+        Returns:
+            Path to JSON file saved to local disk
+        """
+
+        url = f"https://pmhc-mds.net/api/organisations/{self.organisation_path}/uploads/{uuid}"
+        upload_errors_json = self.s.get(url)
+
+        download_folder.mkdir(parents=True, exist_ok=True)
+        filename = download_folder / f"{uuid}.json"
+        with open(filename, "wb") as file:
+            file.write(upload_errors_json.content)
+
+        logging.info(f"Saved JSON file to disk: '{filename}'")
+
+        return filename
+
+    def is_upload_processing(self) -> bool:
+        """Checks if the user has an upload currently processing in either live or
+        test mode. Useful for checking before we do certain actions e.g. try upload
+        another file, because this script can only handle one 'processing' file at a time
+
+        Returns:
+            `True` if an upload is currently processing, otherwise `False`.
+        """
+        # Get a list of all this user's uploads ('processing', 'complete'
+        # and 'error' status)
+        # The filter parameter only accepts 'name', not 'username' or 'email'
+        # This is not ideal, as if there is another user with the same name,
+        # uploading at the same time, then you will be blocked from uploading
+        # until their upload completes. But this seems to be the best we can
+        # do within the limits of the unofficial PMHC Portal API.
+        pmhc_name = self.user_info["name"]
+        json_list = self.s.get(
+            f"https://pmhc-mds.net/api/uploads?name={pmhc_name}&sort=-date",
+            headers={"Range": "0-19"},
+        ).json()
+        # see if any are in a 'processing' state
+        for json in json_list:
+            if "status" in json and json["status"] == "processing":
+                return True
+
+        # all ok, none are processing, we are free to now upload a new file
+        return False
+
+    def wait_for_extract(self, uuid: str, max_retries: int = 20) -> bool:
+        """Wait for an extract with given uuid to have status
+        'Completed'.
+
+        Both PMHC server errors and incomplete processing extracts
+        return the same HTTP status (400) and JSON response when
+        trying to fetch the extract by UUID:
+
+        .. code-block:: text
+
+            https://pmhc-mds.net/api/extract/{download_uuid}/fetch
+
+            {
+              "errors": {
+                "export_fetch": "Can not fetch extract for uuid
+                    [123...]. Extract is not complete. Extract has
+                    expired."
+              }
+            }
+
+        For this reason, it's not sufficient to simply try the
+        download URL until we get a success code. If there is
+        a PMHC server error, we will end up retrying forever.
+
+        Instead, we need to fetch the list of extracts and filter
+        for one with the required uuid. We can then check the
+        extract status explicitly, which should be one of the
+        following values:
+
+        - Completed
+        - Processing
+        - Queued
+        - Error
+
+        If Completed, we can download the extract.
+        If Processing or Queued, keep looping and waiting.
+        If Error, the extract has failed. Exit.
+        """
+        retries = 0
+        while retries < max_retries:
+            logging.info(f"wait_for_extract: attempt: {retries}")
+            time.sleep(30)
+            try:
+                extracts_request = self.s.get(
+                    "https://pmhc-mds.net/api/extract?sort=-date"
+                )
+
+                extracts = extracts_request.json()
+                extract = next(filter(lambda item: item.get("uuid") == uuid, extracts))
+                status = extract["status"]
+
+                if status == "Completed":
+                    return True
+                if status == "Error":
+                    logging.error(f"PMHC extract with uuid {uuid} has failed.")
+                    logging.error("See PMHC Server error:")
+                    logging.error(extract["stash"]["error"])
+                    raise PmhcServerError("The PMHC extract has failed on the server.")
+
+            except (requests.ReadTimeout, requests.ConnectionError) as err:
+                retries += 1
+                logging.warning(err)
+                logging.warning(
+                    f"Request timed out ({retries} of {max_retries}). Retrying."
+                )
+
+        else:
+            raise MaxRetriesExceeded(
+                f"Tried fetching PMHC extract list {retries} times"
+            )
+
+    def download_extract_request(
+        self,
+        start_date: date = date.today() - timedelta(days=30),
+        end_date: date = date.today(),
+        organisation_path: Optional[str] = None,
+        specification: PMHCSpecification = PMHCSpecification.PMHC,
+        without_associated_dates: bool = False,
+        matched_episodes: bool = False,
+        max_retries: int = 20,
+        **kwargs,
+    ) -> Path:
+        """Extract PMHC MDS Data within the date range. If no date range
+        is given, `start_date` defaults to 30 days before the current
+        date and `end_date` defaults to the current date.
+
+        Returns a requests.Response object for the generated data
+        extract. This enables you to use any of the supported
+        requests.Response methods, rather than simply downloading to a
+        local file. (If you want to just download to a local file, use
+        the download_pmhc_mds() method.)
+
+        Args:
+            start_date: start date for extract
+            end_date: end date for extract (default: today)
+            organisation_path: Organisation path for downloaded extract.
+                Defaults to your organisation as specified when
+                initialising `pmhclib.PMHC`. However, can be a different
+                organisation, for example if you are a PHN, but only
+                want to download data for a single provider
+                organisation.
+            specification: Specification for extract. (default:
+                `PMHCSpecification.PMHC`, which returns data from the
+                current PMHC specification.)
+            without_associated_dates: Enable extract option
+                "Include data without associated dates"
+            matched_episodes: Enable extract option
+                "Include all data associated with matched episodes"
+            max_retries: Number of times to retry after timeout when
+                waiting for extract to be generated by PMHC website.
+            kwargs: Additional arguments passed through to
+                requests.get()
+
+        Returns:
+            requests.Response
+
+        Examples:
+
+            Stream download to disk, avoiding excessive RAM usage.
+
+            >>> with PMHC("PHN105") as pmhc:
+            ...     pmhc.login()
+            ...     r = pmhc.download_extract_request(stream=True)
+            ...     with open('extract.zip', 'wb') as file:
+            ...         for content in r.iter_content(chunk_size=65536):
+            ...             file.write(content)
+        """
+
+        if organisation_path is None:
+            organisation_path = self.organisation_path
+
+        # Queue download from PMHC
+        logging.info("Queuing extract...")
+        params = {
+            "organisation_path": f"{organisation_path}",
+            "encoded_organisation_path": f"{organisation_path}",
+            "file_type": "csv",
+            "start_date": f"{start_date:%Y-%m-%d}",
+            "end_date": f"{end_date:%Y-%m-%d}",
+            # These need to be interpreted as a JS boolean
+            # (true or 1, rather than True).
+            "childless": int(without_associated_dates),
+            "all_episode_children": int(matched_episodes),
+            "spec_type": specification.term,
+        }
+
+        download_request = self.s.get(
+            "https://pmhc-mds.net/api/extract/csv",
+            params=params,
+        )
+        download_response = download_request.json()
+        try:
+            download_uuid = download_response["uuid"]
+        except KeyError as err:
+            progress.stop()
+            logging.error("Could not find uuid in the following JSON:")
+            logging.error(download_response)
+            logging.error(
+                "Ensure your PMHC user has the 'Reporting' role and you have\n"
+                "set the correct organisation_path."
+            )
+            raise err
+
+        # Wait for extract to be ready
+        logging.info("Waiting for extract...")
+        self.wait_for_extract(download_uuid, max_retries)
+
+        # We know the URL which will give us the final download URL,
+        # as we have the uuid. We have confirmed above that the
+        # extract is completed.
+        retries = 0
+        while retries < max_retries:
+            try:
+                download_url_request = self.s.get(
+                    f"https://pmhc-mds.net/api/extract/{download_uuid}/fetch"
+                )
+                # Successful status codes are between 200 and 299
+                if 200 <= download_url_request.status_code <= 299:
+                    break
+            except (requests.ReadTimeout, requests.ConnectionError) as err:
+                retries += 1
+                logging.warning(err)
+                logging.warning(
+                    f"Request timed out ({retries} of {max_retries}). Retrying."
+                )
+
+            # Wait before retrying
+            time.sleep(30)
+
+        else:
+            raise MaxRetriesExceeded(f"Tried fetching PMHC extract {retries} times.")
+
+        download_url_json = download_url_request.json()
+        download_url = download_url_json["location"]
+
+        logging.info("Downloading extract...")
+        return self.s.get(download_url, **kwargs)
+
+    def download_pmhc_mds(
+        self,
+        output_directory: Path = Path("."),
+        start_date: date = date.today() - timedelta(days=30),
+        end_date: date = date.today(),
+        organisation_path: Optional[str] = None,
+        specification: PMHCSpecification = PMHCSpecification.PMHC,
+        without_associated_dates: bool = False,
+        matched_episodes: bool = False,
+        max_retries: int = 20,
+    ) -> Path:
+        """Extract PMHC MDS Data within the date range. If no date range
+        is given, `start_date` defaults to 30 days before the current
+        date and `end_date` defaults to the current date.
+
+        If you just want the raw requests.Response object, use the
+        download_extract_request() method.
+
+        Args:
+            output_directory: directory to save download
+            start_date: start date for extract
+            end_date: end date for extract (default: today)
+            organisation_path: Organisation path for downloaded extract.
+                Defaults to your organisation as specified when
+                initialising `pmhclib.PMHC`. However, can be a different
+                organisation, for example if you are a PHN, but only
+                want to download data for a single provider
+                organisation.
+            specification: Specification for extract. (default:
+                `PMHCSpecification.PMHC`, which returns data from the
+                current PMHC specification.)
+            without_associated_dates: Enable extract option
+                "Include data without associated dates"
+            matched_episodes: Enable extract option
+                "Include all data associated with matched episodes"
+            max_retries: Number of times to retry after timeout when
+                waiting for extract to be generated by PMHC website.
+
+        Returns:
+            Path to downloaded extract.
+        """
+
+        output_file = output_directory / f"pmhc_extract_{start_date}_{end_date}.zip"
+        logging.info(f"Saving output to {output_file}")
+
+        r = self.download_extract_request(
+            start_date=start_date,
+            end_date=end_date,
+            organisation_path=organisation_path,
+            specification=specification,
+            without_associated_dates=without_associated_dates,
+            matched_episodes=matched_episodes,
+            max_retries=max_retries,
+            stream=True,
+        )
+
+        with open(output_file, "wb") as fp:
+            for content in r.iter_content(chunk_size=65536):
+                fp.write(content)
+
+        return output_file