dic-helper-toolkit 1.0.2__py3-none-any.whl

dic_helper_toolkit-1.0.2.dist-info/METADATA

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: dic-helper-toolkit
+Version: 1.0.2
+Summary: Helper libraries for data integration centers' daily tasks
+Project-URL: Repository, https://github.com/haemka/dic-helper-toolkit
+Project-URL: Issues, https://github.com/haemka/dic-helper-toolkit/issues
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: requests~=2.34.2
+Requires-Dist: requests-oauthlib~=2.0.0
+Requires-Dist: oauthlib~=3.3.1
+Requires-Dist: fhir.resources~=8.2.0
+Requires-Dist: pytest~=9.0.3
+Requires-Dist: certifi~=2026.5.20
+Requires-Dist: charset-normalizer~=3.4.7
+Requires-Dist: pydantic~=2.13.4
+Requires-Dist: typing-extensions~=4.15.0
+Requires-Dist: typing-inspection~=0.4.2
+Requires-Dist: responses~=0.26.1
+Provides-Extra: xml
+Requires-Dist: lxml~=6.1; extra == "xml"
+Provides-Extra: tests
+Requires-Dist: pytest; extra == "tests"
+Requires-Dist: responses; extra == "tests"
+Requires-Dist: lxml~=6.1; extra == "tests"
+
+# Python DIC Helper Toolkit
+
+This Toolkit contains helper libraries for the data integration centers daily tasks.
+
+## Installation
+
+```shell
+pip install dic-helper-toolkit
+```
+
+The package is published on [PyPI](https://pypi.org/project/dic-helper-toolkit/). To use the FHIR XML reader, install
+the `xml` extra: `pip install dic-helper-toolkit[xml]`.
+
+## Contents
+
+### HTTP client
+
+A wrapper around the `requests` library which allows to create an object for an endpoint holding all needed
+configuration including authentication and certificate verification. It supports Basic Auth and OAuth 2.0.
+
+#### Usage:
+
+```python
+from dic_helpers.http_client import HTTPClient
+
+## No authentication or CA certificate at all
+client = HTTPClient(base_url="https://example.com")
+
+## Basic Auth with CA certificate
+#client = HTTPClient(base_url="https://example.com", username="user", password="P4SSW0RD", ca_file="MyCA.crt")
+
+## OAuth 2.0 with username and password
+#client = HTTPClient(base_url="https://example.com", username="user", password="P4SSW0RD",
+#                    auth_url="https://keycloak.example.com", realm="MyRealm", client_id="MyClient",
+#                    client_secret="00000000-00000000-00000000-00000000", ca_file="MyCA.crt")
+
+with client.get("index.html") as result:
+    print(result)
+```
+
+
+### FHIR client
+
+A simple FHIR client library which uses the HTTP client from above and handles resources directly as `fhir.resources`
+(https://github.com/nazrulworld/fhir.resources) object.
+
+#### Usage:
+
+```python
+from dic_helpers.fhir_client import FHIRClient
+
+client = FHIRClient(base_url="https://example.com", username="user", password="P4SSW0RD",
+                    auth_url="https://keycloak.example.com", realm="MyRealm", client_id="MyClient",
+                    client_secret="00000000-00000000-00000000-00000000", ca_file="MyCA.crt")
+
+# Fetch a patient 
+patients = client.search("Patient", "identifier=0123456789&_lastUpdated=ge2025-01-01"):
+for patient in patients:
+    print(patient)
+```
+*Keep in mind, that this returns an object.* 
+
+### Directory reader
+
+Lists the files of a directory, optionally including all subdirectories, whose names match shell-style patterns.
+
+#### Usage:
+
+```python
+from dic_helpers.directory_reader import DirectoryReader
+
+reader = DirectoryReader("/data/export", patterns=["*.json", "*.ndjson"], recursive=True)
+for path in reader.list_files():
+    print(path)
+```
+
+- **Only regular files** are listed, sorted by path, as `pathlib.Path` objects that start with the directory as given.
+- **`patterns`** (default `"*"`) are matched case-sensitively against the file names (`*`, `?`, `[seq]`). A single
+  pattern can be a string; a file is listed if it matches any pattern. Special characters in the directory's own name
+  don't matter.
+- **`recursive`** (default `False`) also lists the files of all subdirectories.
+- **`include_hidden`** (default `False`): hidden files (names starting with `.`) are skipped and hidden directories
+  are not read, unless this is `True`.
+- **`follow_symlinks`** (default `False`): symbolic links to directories are only read if this is `True`. Every
+  directory is read once at most, so symlink loops end. Symbolic links to files are always listed, broken links never.
+- **Errors:** a missing directory raises `FileNotFoundError`, a file instead of a directory `NotADirectoryError`, and
+  an unreadable directory `PermissionError`. An empty list therefore always means that no file matched.
+
+
+### FHIR file reader
+
+Reads FHIR R4B resources from files as `fhir.resources` objects, together with the file and line they were read from.
+Combined with the directory reader:
+
+```python
+from dic_helpers.directory_reader import DirectoryReader
+from dic_helpers.fhir_file_reader import FHIRFileReader
+
+paths = DirectoryReader("/data/export", patterns=["*.json", "*.ndjson", "*.xml", "*.gz"], recursive=True).list_files()
+for item in FHIRFileReader(paths).resources():
+    print(item.path, item.line, item.resource.get_resource_type(), item.resource.id)
+```
+
+**Formats**, by file extension (case-insensitive):
+
+| Extension | Content | `item.line` |
+|---|---|---|
+| `.json` | one resource | `None` |
+| `.ndjson` | one resource per line (FHIR bulk data), empty lines are skipped | line number, from 1 |
+| `.xml` | one resource in FHIR XML, the resource type comes from the root element | `None` |
+| `.json.gz`, `.ndjson.gz`, `.xml.gz` | the same, gzip compressed | as above |
+
+Other extensions are an error, the format is not guessed. Text is UTF-8, a byte order mark is allowed. Reading XML
+needs `lxml`: install `dic-helper-toolkit[xml]`. External entities are not resolved, an XML file that references one is
+invalid.
+
+**Parameters:**
+- **`paths`**: the files, read in this order; a single path can be given on its own. Resources come in the order they
+  appear in each file. NDJSON files are read line by line, also compressed ones; JSON and XML files as a whole.
+- **`unpack_bundles`** (default `True`): a Bundle gives the resources of its entries, of any Bundle type. Only the
+  outermost Bundle is unpacked, entries without a resource are left out. With `False` the Bundle is returned.
+- **`on_error`** (default `"raise"`): an unreadable or invalid file, NDJSON line or resource raises `FHIRFileError`,
+  whose message starts with the file and line, e.g. `export/Patient.ndjson:17: invalid JSON: …` or
+  `patient.json: invalid Patient: birthDate: Date value string does not match spec regex.`. With `"skip"` the message
+  is logged as a warning, and reading continues with the next line (NDJSON) or the next file. A missing `lxml` is
+  always an error.

dic_helper_toolkit-1.0.2.dist-info/RECORD

@@ -0,0 +1,9 @@
+dic_helpers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dic_helpers/directory_reader.py,sha256=JQQMaBfoORzOwXAWCNJvcJiHJERe8F_p69yKXP_5uKg,3997
+dic_helpers/fhir_client.py,sha256=yYuo2hPQGa2F11eiBaOMuKfjrXb0DYTOHCOcqom6LOM,14050
+dic_helpers/fhir_file_reader.py,sha256=8Y43J-2cc3X79iuQpupRV08JsgxhprkwnihfvpEj11A,13315
+dic_helpers/http_client.py,sha256=tej3q1BsDTRQ3t8Iw8ISuElVuV1tUYEtX1YAA-KrIxU,11759
+dic_helper_toolkit-1.0.2.dist-info/METADATA,sha256=gyZKqm22Wq-WWgBAn2cMfAiO1pysGjVgE3xcYTStx0U,6569
+dic_helper_toolkit-1.0.2.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+dic_helper_toolkit-1.0.2.dist-info/top_level.txt,sha256=BNkYErUWeTgH4-0z0EnR2h1rzj5-JCq_tw14_q53f_w,12
+dic_helper_toolkit-1.0.2.dist-info/RECORD,,

dic_helper_toolkit-1.0.2.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

dic_helper_toolkit-1.0.2.dist-info/top_level.txt

@@ -0,0 +1 @@
+dic_helpers

dic_helpers/directory_reader.py

@@ -0,0 +1,80 @@
+"""Listing the files of a directory, optionally including its subdirectories."""
+import fnmatch
+import os
+from pathlib import Path
+from typing import Iterable, List, Set, Tuple, Union
+
+
+class DirectoryReader:
+    """
+    Lists the files of a directory whose names match one of the given patterns.
+    """
+
+    def __init__(self, directory: Union[str, os.PathLike], patterns: Union[str, Iterable[str]] = ("*",),
+                 recursive: bool = False, include_hidden: bool = False, follow_symlinks: bool = False) -> None:
+        """
+        Initialize the DirectoryReader. The directory is only read by list_files().
+
+        :param directory: The directory to read.
+        :type directory: Union[str, os.PathLike]
+        :param patterns: Shell-style patterns (fnmatch: *, ?, [seq]) matched case-sensitively against the file names,
+            e.g. ("*.json", "*.ndjson"). A single pattern can be given as a string. A file is listed if it matches any
+            pattern.
+        :type patterns: Union[str, Iterable[str]]
+        :param recursive: Also list the files of all subdirectories.
+        :type recursive: bool
+        :param include_hidden: Also list hidden files (names starting with '.') and read hidden subdirectories.
+        :type include_hidden: bool
+        :param follow_symlinks: Also read subdirectories that are symbolic links. Every directory is read once at most,
+            so symlink loops end. Symbolic links to files are always listed.
+        :type follow_symlinks: bool
+        """
+        self.directory = Path(directory)
+        self.patterns: Tuple[str, ...] = (patterns,) if isinstance(patterns, str) else tuple(patterns)
+        self.recursive = recursive
+        self.include_hidden = include_hidden
+        self.follow_symlinks = follow_symlinks
+
+    def list_files(self) -> List[Path]:
+        """
+        List the matching files.
+
+        :return: The paths of the matching regular files, sorted. Each path starts with the directory as given.
+        :rtype: List[Path]
+        :raises FileNotFoundError: If the directory doesn't exist.
+        :raises NotADirectoryError: If the path is not a directory.
+        :raises PermissionError: If the directory or one of the subdirectories to read can't be read.
+        """
+        if not self.directory.exists():
+            raise FileNotFoundError(f"Directory not found: {self.directory}")
+        if not self.directory.is_dir():
+            raise NotADirectoryError(f"Not a directory: {self.directory}")
+        files: List[Path] = []
+        self._read(self.directory, files, visited=set())
+        return sorted(files)
+
+    def _read(self, directory: Path, files: List[Path], visited: Set[Tuple[int, int]]) -> None:
+        """
+        Add the matching files of a directory to files, and read its subdirectories if recursive.
+
+        :param directory: The directory to read.
+        :type directory: Path
+        :param files: The list the matching files are added to.
+        :type files: List[Path]
+        :param visited: Device and inode of the directories read so far; directories in it are not read again.
+        :type visited: Set[Tuple[int, int]]
+        """
+        status = directory.stat()
+        visited.add((status.st_dev, status.st_ino))
+        with os.scandir(directory) as entries:
+            for entry in entries:
+                if not self.include_hidden and entry.name.startswith("."):
+                    continue
+                # is_file() follows symbolic links: links to files are listed, broken links are not
+                if entry.is_file():
+                    if any(fnmatch.fnmatchcase(entry.name, pattern) for pattern in self.patterns):
+                        files.append(Path(entry.path))
+                elif self.recursive and entry.is_dir(follow_symlinks=self.follow_symlinks):
+                    status = entry.stat()
+                    if (status.st_dev, status.st_ino) not in visited:
+                        self._read(Path(entry.path), files, visited)

dic_helpers/fhir_client.py

@@ -0,0 +1,314 @@
+import logging
+import json
+from typing import Dict, Any, Generator, Optional
+from urllib.parse import urlparse, parse_qs, urlencode, urljoin
+from fhir.resources.R4B import get_fhir_model_class
+from fhir.resources.R4B.bundle import Bundle
+from dic_helpers.http_client import HTTPClient
+
+
+logger = logging.getLogger(__name__)
+
+
+class FHIRClient(HTTPClient):
+    """
+    A FHIR API client that abstracts an advanced HTTP client.
+
+    This client uses the custom implementation for a HTTP client from the same package (dic-helper-toolkit). It offers
+    easy to use function for FHIR requests and uses fhir.resources as object representations of resources.
+    """
+    def __init__(self, page_size=50, next_param: Optional[str] = None, *args, **kwargs) -> None:
+        """Initialize the FHIRClient
+
+        :param page_size: Number of entries to request per search page.
+        :param next_param: Query parameter for offset paging, for servers that don't provide standard FHIR paging
+            links (e.g. '_offset'). Defaults to None, which uses standard FHIR paging only.
+        :param args: Positional arguments to pass to the parent HTTPClient.
+        :param kwargs: Keyword arguments to pass to the parent HTTPClient.
+        """
+        super().__init__(*args, **kwargs)
+        self.page_size = page_size
+        # An empty string counts as not configured
+        self.next_param: Optional[str] = next_param or None
+
+    def _is_standard_paging(self, bundle: Dict[str, Any]) -> bool:
+        """Check if the bundle follows standard FHIR paging
+
+        :param bundle: A dictionary representing a FHIR Bundle.
+        :type bundle: Dict[str, Any]
+        :return: True if the bundle uses standard paging, False otherwise.
+        :rtype: bool
+        """
+        if 'link' in bundle:
+            return any(link['relation'] in ['next', 'previous'] for link in bundle['link'])
+        return False
+
+    def _get_next_url(self, bundle: Dict[str, Any]) -> Optional[str]:
+        """Get the 'next' URL from the bundle
+
+        :param bundle: A dictionary representing a FHIR Bundle.
+        :type bundle: Dict[str, Any]
+        :return: The next URL from the bundle, None if it has no 'next' link.
+        :rtype: Optional[str]
+        """
+        if 'link' in bundle:
+            next_link = next((link['url'] for link in bundle['link'] if link['relation'] == 'next'), None)
+            return next_link
+        return None
+
+    def _construct_next_params(self, original_params: Dict[str, Any], offset: int) -> Dict[str, Any]:
+        """Construct the search parameters of the next page for non-standard (offset) paging
+
+        :param original_params: The search parameters of the current page; they are changed and returned.
+        :type original_params: Dict[str, Any]
+        :param offset: The offset of the next page, set as the next_param parameter.
+        :type offset: int
+        :return: The search parameters of the next page, _count unchanged.
+        :rtype: Dict[str, Any]
+        """
+        # _count stays as requested, the page size of the whole search
+        query_params = original_params
+        query_params[self.next_param] = str(offset)
+        return query_params
+
+    @staticmethod
+    def _parse_params(params: str) -> Dict[str, Any]:
+        """Parse search parameters given as a query string into a dictionary
+
+        :param params: The parameters, e.g. 'identifier=0123456789&_lastUpdated=ge2025-01-01', not URL encoded.
+        :type params: str
+        :return: The parameters by name; a repeated parameter keeps its last value.
+        :rtype: Dict[str, Any]
+        """
+        return dict(param.split('=', 1) for param in params.split('&'))
+
+    def search(self, resource_type: str, params: Optional[Any] = None) -> Generator[Bundle, None, None]:
+        """Perform a FHIR search and yield every result page as a Bundle
+
+        Pages are followed by the Bundle's 'next' link (standard FHIR paging), or, if next_param is configured and a
+        page is full, by requesting the next offset. A server that returns the same page again for the offset stops
+        the search with a warning. Without next_param, a warning is logged if the Bundle's total shows that results
+        are missing.
+
+        :param resource_type: The resource type to search for.
+        :type resource_type: str
+        :param params: Search parameters as a dictionary (a list value repeats the parameter) or as a query string.
+            '_count' defaults to the client's page_size.
+        :type params: Dict[str, Any] or str, optional
+        :return: A generator that yields the result pages.
+        :rtype: Generator[Bundle, None, None]
+        """
+        if params is None:
+            params = {}
+        elif isinstance(params, str):
+            params = self._parse_params(params)
+        params['_count'] = params.get('_count', self.page_size)
+        # The page size actually requested: a _count given by the caller overrides the client's page_size
+        count = params['_count'][0] if isinstance(params['_count'], list) else params['_count']
+        page_size = int(count)
+
+        url = urljoin(self.base_url, resource_type)
+        path = resource_type
+        logger.debug(f"Search request with path {path} and params {params}")
+
+        previous_entries = None
+        received = 0
+        while path:
+            response = self.get(path, params=params)
+            bundle_dict = response.json()
+            entries = bundle_dict.get('entry') or []
+            # A server ignoring the offset parameter returns the same page again, which would otherwise loop forever
+            if entries and entries == previous_entries:
+                logger.warning(f"Server returned the same page again for paging parameter '{self.next_param}'. It "
+                               f"probably does not support this parameter. Stopping the search.")
+                break
+            previous_entries = entries
+            received += len(entries)
+            resource_class = get_fhir_model_class('Bundle')
+            bundle = resource_class.model_validate_json(json.dumps(bundle_dict))
+
+            yield bundle
+
+            if self._is_standard_paging(bundle_dict):
+                # Standard FHIR paging
+                next_url = self._get_next_url(bundle_dict)
+                if next_url:
+                    parsed_next_url = urlparse(next_url)
+                    path = parsed_next_url.path
+                    params = parse_qs(parsed_next_url.query)  # Correctly parse query parameters
+                else:
+                    path = None
+            elif self.next_param and len(entries) == page_size:
+                # Non-standard offset paging, only if configured
+                current_offset = int(params.get(self.next_param, 0))
+                next_offset = current_offset + page_size
+
+                params = self._construct_next_params(params, next_offset)
+            else:
+                total = bundle_dict.get('total')
+                if self.next_param is None and isinstance(total, int) and total > received:
+                    logger.warning(f"Search for {resource_type} returned {received} of {total} results without a next "
+                                   f"link. The server probably doesn't support standard FHIR paging, configure "
+                                   f"next_param to use offset paging.")
+                path = None
+            if path:
+                logger.debug(f"Next URL has path '{path}' and params '{params}'")
+
+    def search_all(self, resource_type: str, params: Optional[Dict[str, Any]] = None) -> Bundle:
+        """Perform a FHIR search and return a single Bundle containing all entries.
+
+        :param resource_type: The resource type to search for.
+        :type resource_type: str
+        :param params: Search parameters, as for search().
+        :type params: Dict[str, Any], optional
+        :return: A searchset Bundle with the entries of all result pages.
+        :rtype: Bundle
+        """
+        # Initialize an empty Bundle to store all entries
+        all_entries_bundle = Bundle(
+            type='searchset',
+            total=0,
+            entry=[]
+        )
+
+        # Call the original search function
+        for page_bundle in self.search(resource_type, params):
+            # Add entries from each page to the all_entries_bundle
+            if page_bundle.entry:
+                all_entries_bundle.entry.extend(page_bundle.entry)
+
+            # Copy metadata from the last page
+            all_entries_bundle.type = page_bundle.type
+            all_entries_bundle.total = page_bundle.total
+
+            # Update other relevant fields
+            if hasattr(page_bundle, 'meta'):
+                all_entries_bundle.meta = page_bundle.meta
+            if hasattr(page_bundle, 'link'):
+                all_entries_bundle.link = page_bundle.link
+
+        # If total wasn't set in the original bundles, use the length of entries
+        if all_entries_bundle.total is None or all_entries_bundle.total == 0:
+            all_entries_bundle.total = len(all_entries_bundle.entry)
+
+        return all_entries_bundle
+
+    def get_resource(self, resource_type: str, id: str, **kwargs: Any) -> Any:
+        """Retrieve a single resource
+
+        :param resource_type: The resource type to search for.
+        :type resource_type: str
+        :param id: The id of the resource to retrieve.
+        :type id: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The resource retrieved.
+        :rtype: Any
+        """
+        response = self.get(f"{resource_type}/{id}", **kwargs)
+        resource_dict = response.json()
+        resource_class = get_fhir_model_class(resource_type)
+        return resource_class.model_validate_json(json.dumps(resource_dict))
+
+    def post_resource(self, resource: Any, **kwargs: Any) -> Any:
+        """Create a new resource
+
+        :param resource: The resource to create.
+        :type resource: Any
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The created resource.
+        :rtype: Any
+        """
+        resource_type = resource.get_resource_type()
+        path = None if resource_type == "Bundle" else resource_type
+        response = self.post(path=path, json=json.loads(resource.model_dump_json()), **kwargs)
+        created_resource_dict = response.json()
+        resource_class = get_fhir_model_class(resource_type)
+        return resource_class.model_validate_json(json.dumps(created_resource_dict))
+
+    # Alias for post_resource()
+    create_resource = post_resource
+
+    def put_resource(self, resource: Any, **kwargs: Any) -> Any:
+        """Update an existing resource
+
+        :param resource: The resource to update.
+        :type resource: Any
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The updated resource.
+        :rtype: Any
+        """
+        resource_type = resource.get_resource_type()
+        resource_id = resource.id
+        response = self.put(f"{resource_type}/{resource_id}",
+                            json=json.loads(resource.model_dump_json()), **kwargs)
+        updated_resource_dict = response.json()
+        resource_class = get_fhir_model_class(resource_type)
+        return resource_class.model_validate_json(json.dumps(updated_resource_dict))
+
+    # Alias for put_resource()
+    update_resource = put_resource
+
+    def delete_resource(self, resource_type: str, id: str, **kwargs: Any) -> None:
+        """Delete a resource
+
+        :param resource_type: The resource type to search for.
+        :type resource_type: str
+        :param kwargs: Additional arguments to pass to the request.
+        :param id: The id of the resource to delete.
+        :type id: str
+        """
+        self.delete(f"{resource_type}/{id}", **kwargs)
+
+    def post_bundle(self, bundle: Bundle, **kwargs: Any) -> Bundle:
+        """Execute a transaction or batch Bundle
+
+        :param bundle: The Bundle to execute.
+        :type bundle: Bundle
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The executed Bundle.
+        :rtype: Bundle
+        """
+        response = self.post("", json=json.loads(bundle.model_dump_json()), **kwargs)
+        result_bundle_dict = response.json()
+        resource_class = get_fhir_model_class('Bundle')
+        return resource_class.model_validate_json(json.dumps(result_bundle_dict))
+
+    # Alias for post_bundle()
+    execute_bundle = post_bundle
+
+    def get_capability_statement(self) -> Any:
+        """Retrieve the server's capability statement
+
+        :return: The server's capability statement.
+        :rtype: Any
+        """
+        response = self.get("metadata")
+        capability_statement_dict = response.json()
+        resource_class = get_fhir_model_class('CapabilityStatement')
+        return resource_class.model_validate_json(json.dumps(capability_statement_dict))
+
+    def execute_operation(self, operation_name: str, resource_type: Optional[str] = None,
+                          id: Optional[str] = None, parameters: Optional[Dict[str, Any]] = None) -> Any:
+        """Execute a FHIR operation
+
+        :param operation_name: The operation to execute.
+        :type operation_name: str
+        :param resource_type: The resource type to search for.
+        :type resource_type: str
+        :param id: The id of the resource to execute.
+        :type id: str
+        :param parameters: Optional parameters to pass to the parent HTTPClient.
+        :type parameters: Dict[str, Any]
+        :return: The executed operation.
+        :rtype: Any
+        """
+        path = f"${operation_name}"
+        if resource_type and id:
+            path = f"{resource_type}/{id}/" + path
+        elif resource_type:
+            path = f"{resource_type}/" + path
+
+        response = self.post(path, json=parameters)
+        result_dict = response.json()
+        resource_class = get_fhir_model_class('Parameters')
+        return resource_class.model_validate_json(json.dumps(result_dict))

dic_helpers/fhir_file_reader.py

@@ -0,0 +1,307 @@
+"""Reading FHIR resources from files: JSON, NDJSON and FHIR XML, each optionally gzip compressed."""
+import gzip
+import json
+import logging
+import os
+import zlib
+from dataclasses import dataclass
+from pathlib import Path
+from typing import IO, Iterable, Iterator, Literal, Optional, Tuple, Union
+
+from fhir.resources.R4B import get_fhir_model_class
+from fhir.resources.R4B.resource import Resource
+from pydantic import ValidationError
+
+logger = logging.getLogger(__name__)
+
+# File formats by extension, before an optional .gz
+FORMATS = {".ndjson": "ndjson", ".json": "json", ".xml": "xml"}
+FHIR_NAMESPACE = "http://hl7.org/fhir"
+
+
+class FHIRFileError(Exception):
+    """An unreadable or invalid file, NDJSON line or resource. The message starts with the path (and line)."""
+
+
+@dataclass(frozen=True)
+class FHIRFileItem:
+    """A resource and where it was read: the file, and for NDJSON the line (from 1), otherwise None."""
+    path: Path
+    line: Optional[int]
+    resource: Resource
+
+
+class FHIRFileReader:
+    """
+    Reads FHIR R4B resources from files. The format comes from the file extension (case-insensitive): .json (a
+    resource), .ndjson (a resource per line, empty lines are skipped) or .xml (a resource in FHIR XML, needs the extra
+    dic-helper-toolkit[xml]), each optionally followed by .gz for gzip compression. Text is UTF-8, optionally with BOM.
+    """
+
+    def __init__(self, paths: Union[str, os.PathLike, Iterable[Union[str, os.PathLike]]], unpack_bundles: bool = True,
+                 on_error: Literal["raise", "skip"] = "raise") -> None:
+        """
+        Initialize the FHIRFileReader. The files are only read by resources().
+
+        :param paths: The files to read, in this order. A single path can be given on its own.
+        :type paths: Union[str, os.PathLike, Iterable[Union[str, os.PathLike]]]
+        :param unpack_bundles: Return the resources of a Bundle's entries instead of the Bundle (only the outermost
+            Bundle; entries without a resource are left out).
+        :type unpack_bundles: bool
+        :param on_error: 'raise': an unreadable or invalid file, line or resource raises FHIRFileError. 'skip': it is
+            logged as a warning instead, and reading continues with the next line (NDJSON) or the next file.
+        :type on_error: Literal["raise", "skip"]
+        :raises ValueError: If on_error is neither 'raise' nor 'skip'.
+        """
+        if on_error not in ("raise", "skip"):
+            raise ValueError(f"on_error must be 'raise' or 'skip', not {on_error!r}")
+        if isinstance(paths, (str, os.PathLike)):
+            paths = [paths]
+        self.paths = [Path(path) for path in paths]
+        self.unpack_bundles = unpack_bundles
+        self.on_error = on_error
+
+    def resources(self) -> Iterator[FHIRFileItem]:
+        """
+        Read the files.
+
+        :return: The resources with their origin, in the order of the files and within each file.
+        :rtype: Iterator[FHIRFileItem]
+        :raises FHIRFileError: With on_error 'raise', for an unreadable or invalid file, line or resource.
+        :raises ImportError: For an XML file without lxml installed, also with on_error 'skip'.
+        """
+        for path in self.paths:
+            yield from self._read_file(path)
+
+    def _read_file(self, path: Path) -> Iterator[FHIRFileItem]:
+        """
+        Read the resources of one file in its format. Errors of the whole file go to _fail().
+
+        :param path: The file to read.
+        :type path: Path
+        :return: The resources of the file with their origin.
+        :rtype: Iterator[FHIRFileItem]
+        """
+        try:
+            file_format, compressed = self._format(path)
+            with self._open(path, compressed) as file:
+                if file_format == "ndjson":
+                    yield from self._read_ndjson(path, file)
+                else:
+                    content = file.read()
+                    if file_format == "json":
+                        resource = self._from_json(self._decode(content, str(path)), str(path))
+                    else:
+                        resource = self._from_xml(content, str(path))
+                    yield from self._items(path, None, resource)
+        except FHIRFileError as error:
+            self._fail(error)
+        except (OSError, EOFError, zlib.error) as error:
+            # Can't be opened, or isn't valid gzip
+            self._fail(FHIRFileError(f"{path}: cannot read file: {error}"))
+
+    def _read_ndjson(self, path: Path, file: IO[bytes]) -> Iterator[FHIRFileItem]:
+        """
+        Read an NDJSON file line by line. Errors of a line go to _fail(), so with on_error 'skip' only that line is
+        left out.
+
+        :param path: The file, for the origin and for messages.
+        :type path: Path
+        :param file: The opened file, binary and already decompressed.
+        :type file: IO[bytes]
+        :return: The resources of the lines with their origin.
+        :rtype: Iterator[FHIRFileItem]
+        """
+        for number, raw in enumerate(file, start=1):
+            location = f"{path}:{number}"
+            try:
+                text = self._decode(raw, location).strip()
+                if not text:
+                    continue
+                resource = self._from_json(text, location)
+            except FHIRFileError as error:
+                self._fail(error)
+                continue
+            yield from self._items(path, number, resource)
+
+    def _items(self, path: Path, line: Optional[int], resource: Resource) -> Iterator[FHIRFileItem]:
+        """
+        The results for a resource read from a file: the resources of its entries for a Bundle with unpack_bundles,
+        otherwise the resource itself.
+
+        :param path: The file the resource was read from.
+        :type path: Path
+        :param line: The NDJSON line the resource was read from, None for other formats.
+        :type line: Optional[int]
+        :param resource: The resource.
+        :type resource: Resource
+        :return: The resources with their origin.
+        :rtype: Iterator[FHIRFileItem]
+        """
+        if self.unpack_bundles and resource.get_resource_type() == "Bundle":
+            for entry in resource.entry or []:
+                if entry.resource is not None:
+                    yield FHIRFileItem(path, line, entry.resource)
+        else:
+            yield FHIRFileItem(path, line, resource)
+
+    def _fail(self, error: FHIRFileError) -> None:
+        """
+        Handle an error according to on_error: raise it, or log it as a warning.
+
+        :param error: The error, its message names the file (and line).
+        :type error: FHIRFileError
+        :raises FHIRFileError: With on_error 'raise'.
+        """
+        if self.on_error == "raise":
+            raise error
+        logger.warning(str(error))
+
+    @staticmethod
+    def _format(path: Path) -> Tuple[str, bool]:
+        """
+        The format of a file and whether it is gzip compressed, from its extension (case-insensitive).
+
+        :param path: The file.
+        :type path: Path
+        :return: The format ('json', 'ndjson' or 'xml') and whether the file ends with .gz.
+        :rtype: Tuple[str, bool]
+        :raises FHIRFileError: For any other extension.
+        """
+        name = path.name.lower()
+        compressed = name.endswith(".gz")
+        if compressed:
+            name = name.removesuffix(".gz")
+        for extension, file_format in FORMATS.items():
+            if name.endswith(extension):
+                return file_format, compressed
+        raise FHIRFileError(f"{path}: unknown file type, expected .json, .ndjson or .xml, optionally with .gz")
+
+    @staticmethod
+    def _open(path: Path, compressed: bool) -> IO[bytes]:
+        """
+        Open a file for reading bytes, decompressing it if it is gzip compressed.
+
+        :param path: The file.
+        :type path: Path
+        :param compressed: Whether the file is gzip compressed.
+        :type compressed: bool
+        :return: The opened file.
+        :rtype: IO[bytes]
+        :raises OSError: If the file can't be opened.
+        """
+        return gzip.open(path, "rb") if compressed else open(path, "rb")
+
+    @staticmethod
+    def _decode(content: bytes, location: str) -> str:
+        """
+        Decode UTF-8 text, with or without a byte order mark.
+
+        :param content: The bytes of a file or an NDJSON line.
+        :type content: bytes
+        :param location: The file (and line) for the error message.
+        :type location: str
+        :return: The text, without byte order mark.
+        :rtype: str
+        :raises FHIRFileError: If the content is not UTF-8.
+        """
+        try:
+            # utf-8-sig removes a BOM, and reads text without BOM like utf-8
+            return content.decode("utf-8-sig")
+        except UnicodeDecodeError as error:
+            raise FHIRFileError(f"{location}: not UTF-8: {error}") from None
+
+    @staticmethod
+    def _model_class(resource_type: object, location: str) -> type:
+        """
+        The fhir.resources R4B model class of a resource type.
+
+        :param resource_type: The resource type, e.g. 'Patient'.
+        :type resource_type: object
+        :param location: The file (and line) for the error message.
+        :type location: str
+        :return: The model class.
+        :rtype: type
+        :raises FHIRFileError: If it is not a FHIR resource type, also for data types like 'HumanName'.
+        """
+        try:
+            model_class = get_fhir_model_class(str(resource_type))
+        except ValueError:
+            model_class = None
+        if model_class is None or not model_class.has_resource_base():
+            raise FHIRFileError(f"{location}: unknown resource type {resource_type!r}")
+        return model_class
+
+    def _from_json(self, text: str, location: str) -> Resource:
+        """
+        Parse and validate a resource in JSON.
+
+        :param text: The JSON text of a file or an NDJSON line.
+        :type text: str
+        :param location: The file (and line) for error messages.
+        :type location: str
+        :return: The resource.
+        :rtype: Resource
+        :raises FHIRFileError: For invalid JSON, a missing or unknown resourceType, or an invalid resource.
+        """
+        try:
+            data = json.loads(text)
+        except json.JSONDecodeError as error:
+            raise FHIRFileError(f"{location}: invalid JSON: {error}") from None
+        if not isinstance(data, dict) or "resourceType" not in data:
+            raise FHIRFileError(f"{location}: not a FHIR resource, resourceType is missing")
+        model_class = self._model_class(data["resourceType"], location)
+        try:
+            return model_class.model_validate(data)
+        except ValidationError as error:
+            raise FHIRFileError(f"{location}: invalid {data['resourceType']}: {_problems(error)}") from None
+
+    def _from_xml(self, content: bytes, location: str) -> Resource:
+        """
+        Parse and validate a resource in FHIR XML. The resource type is the name of the root element, which must be in
+        the FHIR namespace. External entities are not resolved.
+
+        :param content: The bytes of the file, XML parses the encoding itself.
+        :type content: bytes
+        :param location: The file for error messages.
+        :type location: str
+        :return: The resource.
+        :rtype: Resource
+        :raises FHIRFileError: For invalid XML, a root element outside the FHIR namespace, an unknown resource type or
+            an invalid resource.
+        :raises ImportError: If lxml is not installed.
+        """
+        try:
+            from lxml import etree
+        except ImportError:
+            raise ImportError("Reading FHIR XML needs lxml, install dic-helper-toolkit[xml]") from None
+        # No external entities and no network access while parsing
+        parser = etree.XMLParser(resolve_entities=False, no_network=True)
+        try:
+            root = etree.fromstring(content, parser)
+        except etree.XMLSyntaxError as error:
+            raise FHIRFileError(f"{location}: invalid XML: {error}") from None
+        name = etree.QName(root)
+        if name.namespace != FHIR_NAMESPACE:
+            raise FHIRFileError(f"{location}: not a FHIR resource, the root element is not in the namespace "
+                                f"{FHIR_NAMESPACE}")
+        model_class = self._model_class(name.localname, location)
+        try:
+            return model_class.model_validate_xml(content, xmlparser=parser)
+        except ValidationError as error:
+            raise FHIRFileError(f"{location}: invalid {name.localname}: {_problems(error)}") from None
+        except ValueError as error:
+            raise FHIRFileError(f"{location}: invalid {name.localname}: {error}") from None
+
+
+def _problems(error: ValidationError) -> str:
+    """
+    The problems of a validation error on one line, e.g. 'birthDate: Date value string does not match spec regex.'.
+
+    :param error: The validation error of a resource.
+    :type error: ValidationError
+    :return: Every problem as '<element path>: <message>', separated by '; '.
+    :rtype: str
+    """
+    return "; ".join(f"{'.'.join(str(part) for part in problem['loc'])}: {problem['msg'].removeprefix('Value error, ')}"
+                     for problem in error.errors())

dic_helpers/http_client.py

@@ -0,0 +1,300 @@
+import requests
+import ssl
+import logging
+from typing import Optional, Dict, Any
+from requests.auth import HTTPBasicAuth
+from requests_oauthlib import OAuth2Session
+from oauthlib.oauth2 import LegacyApplicationClient, BackendApplicationClient, TokenExpiredError
+from urllib.parse import urljoin
+
+
+logger = logging.getLogger(__name__)
+
+
+class NoRFC5280HTTPAdapter(requests.adapters.HTTPAdapter):
+    """
+    HTTP adapter that verifies certificates without Python's strict RFC 5280 checks (``VERIFY_X509_STRICT``).
+
+    Certificate chain and hostname verification stay enabled; only the additional strictness checks enabled by
+    default since Python 3.13 are relaxed. Applies to direct connections as well as connections through proxies.
+    """
+
+    def __init__(self, *args, ca_file: Optional[str] = None, **kwargs):
+        """Initialize the adapter with an SSL context without the strict RFC 5280 checks.
+
+        :param args: Positional arguments to pass to requests' HTTPAdapter.
+        :param ca_file: Path to a CA certificate file; if given, only this file is trusted, otherwise the system's CAs.
+        :type ca_file: str, optional
+        :param kwargs: Keyword arguments to pass to requests' HTTPAdapter.
+        """
+        self._ssl_context = ssl.create_default_context(cafile=ca_file)
+        self._ssl_context.verify_flags &= ~ssl.VERIFY_X509_STRICT
+        super().__init__(*args, **kwargs)
+
+    def init_poolmanager(self, *args, **kwargs):
+        """Create the pool manager for direct connections with this adapter's SSL context.
+
+        :param args: Positional arguments to pass to requests' HTTPAdapter.init_poolmanager.
+        :param kwargs: Keyword arguments to pass to requests' HTTPAdapter.init_poolmanager.
+        """
+        kwargs["ssl_context"] = self._ssl_context
+        super().init_poolmanager(*args, **kwargs)
+
+    def proxy_manager_for(self, proxy, **proxy_kwargs):
+        """Return the pool manager for connections through a proxy, with this adapter's SSL context.
+
+        :param proxy: The proxy URL.
+        :type proxy: str
+        :param proxy_kwargs: Keyword arguments to pass to requests' HTTPAdapter.proxy_manager_for.
+        :return: The proxy manager.
+        :rtype: urllib3.ProxyManager
+        """
+        proxy_kwargs["ssl_context"] = self._ssl_context
+        return super().proxy_manager_for(proxy, **proxy_kwargs)
+
+
+class HTTPClient:
+    """
+    A client for making HTTP requests with support for OAuth2 authentication.
+
+    This client can handle basic authentication, OAuth2 Resource Owner Password Credentials flow, and OAuth2 Client
+    Credentials flow.
+    """
+
+    def __init__(self, base_url: str, username: Optional[str] = None, password: Optional[str] = None,
+                 auth_url: Optional[str] = None, realm: Optional[str] = None,
+                 client_id: Optional[str] = None, client_secret: Optional[str] = None,
+                 ca_file: Optional[str] = None, rfc5280: bool = True) -> None:
+        """Initialize the HTTPClient.
+
+        :param base_url: The base URL for API requests.
+        :type base_url: str
+        :param username: Username for authentication.
+        :type username: str, optional
+        :param password: Password for authentication.
+        :type password: str, optional
+        :param auth_url: URL for OAuth2 authentication.
+        :type auth_url: str, optional
+        :param realm: Realm for OAuth2 authentication.
+        :type realm: str, optional
+        :param client_id: Client ID for OAuth2 authentication.
+        :type client_id: str, optional
+        :param client_secret: Client secret for OAuth2 authentication.
+        :type client_secret: str, optional
+        :param ca_file: Path to CA certificate file for SSL verification.
+        :type ca_file: str, optional
+        :param rfc5280: Enforce Python's strict RFC 5280 certificate checks (``VERIFY_X509_STRICT``). Defaults to
+            True. False relaxes only these checks so that certificates from non-conforming CAs are accepted;
+            chain and hostname verification remain active either way.
+        :type rfc5280: bool, optional
+        """
+        self.base_url: str = self._normalize_url(base_url)
+        self.username: Optional[str] = username
+        self.password: Optional[str] = password
+        self.auth_url: Optional[str] = self._normalize_url(auth_url) if auth_url else None
+        self.realm: Optional[str] = realm
+        self.client_id: Optional[str] = client_id
+        self.client_secret: Optional[str] = client_secret
+        self.ca_file: Optional[str] = ca_file
+        self.rfc5280: bool = rfc5280
+        self.session: Optional[requests.Session] = None
+        self.token: Optional[Dict[str, Any]] = None
+        self.token_url: Optional[str] = None
+        self._create_session()
+
+    @staticmethod
+    def _normalize_url(url: str) -> str:
+        """
+        Ensure the URL ends with a forward slash.
+
+        :param url: The URL to normalize.
+        :type url: str
+        :return: The normalized URL.
+        :rtype: str
+        """
+        if url.endswith('/'):
+            return url
+        else:
+            return f"{url}/"
+
+    def _get_openid_configuration(self) -> Dict[str, Any]:
+        """
+        Retrieve the OpenID configuration from the authentication server.
+
+        :return: The OpenID configuration.
+        :rtype: dict
+        :raises requests.HTTPError: If the request fails.
+        """
+        config_url = urljoin(urljoin(self.auth_url, f"realms/{self.realm}/"),
+                             ".well-known/openid-configuration")
+        response = self.session.get(config_url)
+        response.raise_for_status()
+        return response.json()
+
+    def _fetch_token(self) -> Dict[str, Any]:
+        """
+        Fetch an OAuth2 token from the authentication server.
+
+        :return: The OAuth2 token.
+        :rtype: dict
+        :raises oauthlib.oauth2.OAuth2Error: If token fetching fails.
+        """
+        if self.username and self.password:
+            # Resource Owner Password Credentials flow
+            return self.session.fetch_token(
+                token_url=self.token_url,
+                username=self.username,
+                password=self.password,
+                client_id=self.client_id,
+                client_secret=self.client_secret
+            )
+        else:
+            # Client Credentials flow
+            # TODO: untested, needs to be tested, after KeyCloak update
+            return self.session.fetch_token(
+                token_url=self.token_url,
+                client_id=self.client_id,
+                client_secret=self.client_secret
+            )
+
+    def _create_session(self) -> None:
+        """
+        Create and configure the HTTP session based on the authentication method.
+        """
+        oauth = self.auth_url and self.realm and self.client_id and self.client_secret
+        if oauth:
+            if self.username and self.password:
+                self.session = OAuth2Session(client=LegacyApplicationClient(client_id=self.client_id))
+            else:
+                self.session = OAuth2Session(client=BackendApplicationClient(client_id=self.client_id))
+        else:
+            self.session = requests.Session()
+            if self.username and self.password:
+                self.session.auth = HTTPBasicAuth(self.username, self.password)
+
+        if self.ca_file:
+            self.session.verify = self.ca_file
+
+        if not self.rfc5280:
+            self.session.mount('https://', NoRFC5280HTTPAdapter(ca_file=self.ca_file))
+
+        if oauth:
+            self.token_url = self._get_openid_configuration()['token_endpoint']
+            self.token = self._fetch_token()
+
+    def _token_updater(self, token: Dict[str, Any]) -> None:
+        """
+        Update the stored token when it's refreshed.
+
+        :param token: The new token.
+        :type token: dict
+        """
+        logger.debug(f"Token updater called with token {token}")
+        self.token = token
+
+    def _make_request(self, method: str, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Make an HTTP request, handling token expiration if necessary.
+
+        :param method: The HTTP method (GET, POST, etc.).
+        :type method: str
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        :raises requests.HTTPError: If the request fails.
+        """
+        url = urljoin(self.base_url, path)
+        try:
+            response = self.session.request(method, url, **kwargs)
+        except TokenExpiredError:
+            logger.debug(f"Token expired for {url}, Re-fetching...")
+            self.token = self._fetch_token()
+            response = self.session.request(method, url, **kwargs)
+
+        if response.status_code == 401:
+            logger.debug(f"Token expired for {url}, Re-fetching...")
+            self.token = self._fetch_token()
+            response = self.session.request(method, url, **kwargs)
+
+        response.raise_for_status()
+        return response
+
+    def get(self, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a GET request.
+
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        """
+        return self._make_request('GET', path, **kwargs)
+
+    def post(self, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a POST request.
+
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        """
+        return self._make_request('POST', path, **kwargs)
+
+    def put(self, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a PUT request.
+
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        """
+        return self._make_request('PUT', path, **kwargs)
+
+    def delete(self, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a DELETE request.
+
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        """
+        return self._make_request('DELETE', path, **kwargs)
+
+    def patch(self, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a PATCH request.
+
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        """
+        return self._make_request('PATCH', path, **kwargs)
+
+    def request(self, method: str, path: str, **kwargs: Any) -> requests.Response:
+        """
+        Send a custom HTTP request.
+
+        This method allows sending any HTTP request method and is used internally
+        by other convenience methods like get(), post(), etc.
+
+        :param method: The HTTP method (GET, POST, PUT, DELETE, etc.).
+        :type method: str
+        :param path: The path to request.
+        :type path: str
+        :param kwargs: Additional arguments to pass to the request.
+        :return: The HTTP response.
+        :rtype: requests.Response
+        :raises requests.HTTPError: If the request fails.
+        """
+        return self._make_request(method, path, **kwargs)