nhp-model 5.0.0__tar.gz

nhp_model-5.0.0/.coveragerc

@@ -0,0 +1,3 @@
+[run]
+omit =
+  tests/*

nhp_model-5.0.0/.github/copilot-instructions.md

@@ -0,0 +1,292 @@
+# NHP Model - Copilot Coding Agent Instructions
+
+## Repository Overview
+
+This is the **New Hospital Programme (NHP) Demand Model**, a Python package for healthcare activity prediction. The model provides modeling capabilities for inpatients, outpatients, and A&E (Accident & Emergency) services. It is built as a Python library using modern packaging tools and is deployed as both a Python package and a Docker container to Azure.
+
+**Key Facts:**
+- **Project Type:** Python package/library with Docker containerization
+- **Python Version:** Requires Python 3.11 or higher (specified in pyproject.toml)
+- **Package Manager:** `uv` (modern Python package manager from Astral)
+- **Build System:** setuptools with setuptools-scm for versioning
+- **Primary Language:** Python
+- **Project Size:** Medium-sized Python project
+- **Main Modules:** nhp.model (core model code), nhp.docker (Docker runtime)
+
+## Environment Setup and Build Instructions
+
+### Initial Setup
+
+**ALWAYS start by installing uv and project dependencies:**
+
+```bash
+# Install uv using the recommended approach from Astral
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Install project dependencies (production only)
+uv sync
+
+# Install with dev dependencies for development/testing (RECOMMENDED for development)
+uv sync --extra dev
+
+# Install with docs dependencies for documentation
+uv sync --extra docs
+
+# Install multiple extras at once
+uv sync --extra dev --extra docs
+```
+
+**Important:** The `uv sync` command only installs production dependencies. For development work (linting, testing), use `uv sync --extra dev` to install the dev dependencies.
+
+**Python Version:** The project requires Python 3.11+. The CI uses Python 3.11 specifically via `uv python install` in workflows.
+
+### Build Commands
+
+**To build the package:**
+
+```bash
+# Standard build - creates wheel and source distribution
+uv build
+
+# Build for development (sets version to 0.dev0)
+SETUPTOOLS_SCM_PRETEND_VERSION=0.dev0 uv build
+```
+
+The build creates:
+- `dist/nhp_model-<version>-py3-none-any.whl`
+- `dist/nhp_model-<version>.tar.gz`
+
+**Note:** The Dockerfile includes a TODO comment about forcing version numbers during Docker builds. Currently it uses `ENV SETUPTOOLS_SCM_PRETEND_VERSION=v0.0.0` as a workaround.
+
+### Testing
+
+**Unit Tests (ALWAYS run these before committing):**
+
+```bash
+# Run all unit tests
+uv run pytest tests/unit --verbose
+
+# Run unit tests with coverage report
+uv run pytest --cov=. tests/unit --ignore=tests --cov-branch --cov-report xml:coverage.xml
+```
+
+**Integration Tests:**
+
+```bash
+# Integration tests require test data in a specific format
+# These are located in tests/integration/ but may require data setup
+uv run pytest tests/integration --verbose
+```
+
+**All unit tests must pass. Test failures are NOT acceptable.**
+
+### Linting and Formatting
+
+**ALWAYS run linting before committing. All linting checks MUST pass:**
+
+```bash
+# Run ruff linting check
+uvx ruff check .
+
+# Run ruff format check (no auto-formatting)
+uvx ruff format --check .
+
+# Auto-format code (if needed)
+uvx ruff format .
+
+# Run type checking with ty
+uvx ty check .
+```
+
+**Linting Configuration:**
+- Ruff config is in `pyproject.toml` under `[tool.ruff]`
+- Line length: 100 characters
+- Target Python version: 3.11
+- Excludes: `notebooks/` directory
+- Key rules: pydocstyle (D), pycodestyle (E/W), isort (I), pylint (PL), pandas-vet (PD), numpy (NPY), ruff-specific (RUF)
+- Docstring convention: Google style
+
+**The notebooks directory is excluded from linting and should not be linted.**
+
+### Documentation
+
+```bash
+# Build documentation (requires docs dependencies)
+uv run mkdocs build --clean
+
+# Serve documentation locally
+uv run mkdocs serve
+```
+
+Documentation is deployed automatically to Connect via CI on main branch pushes.
+
+### Running the Model
+
+**Local execution:**
+
+```bash
+# Run with sample parameters (requires data in specified path)
+uv run python -m nhp.model queue/params-sample.json -d data/synth --type all
+
+# Run single model type
+uv run python -m nhp.model queue/params-sample.json -d data --type ip   # inpatients
+uv run python -m nhp.model queue/params-sample.json -d data --type op   # outpatients
+uv run python -m nhp.model queue/params-sample.json -d data --type aae  # A&E
+
+# Run specific model iteration for debugging
+uv run python -m nhp.model queue/params-sample.json -d data --model-run 1 --type ip
+```
+
+**Command-line arguments:**
+- `params_file`: Path to JSON parameters file (default: `queue/params-sample.json`)
+- `-d, --data-path`: Path to data directory (default: `data`)
+- `-r, --model-run`: Which model iteration to run (default: 1)
+- `-t, --type`: Model type - `all`, `ip`, `op`, or `aae` (default: `all`)
+- `--save-full-model-results`: Save complete model results
+
+**Data Requirements:**
+The model expects data in parquet format organized by fiscal year and dataset:
+- Format: `{data_path}/{file}/fyear={year}/dataset={dataset}/`
+- Required files: `ip`, `op`, `aae`, `demographic_factors`, `birth_factors`, `hsa_activity_tables`, `hsa_gams` (pickle)
+- Sample data location: `data/synth/` (synthetic dataset for testing - see GitHub issue #347)
+
+## Project Structure
+
+### Directory Layout
+
+**Core Directories:**
+- `.github/workflows/` - CI/CD pipelines (linting, codecov, build, deploy)
+- `src/nhp/model/` - Core model: `__main__.py`, `model.py`, `inpatients.py`, `outpatients.py`, `aae.py`, `run.py`, `results.py`, `data/`
+- `src/nhp/docker/` - Docker runtime with Azure Storage integration
+- `tests/unit/` - Unit tests
+- `tests/integration/` - Integration tests (require data)
+- `docs/` - MkDocs documentation
+- `notebooks/` - Databricks notebooks (excluded from linting)
+- `queue/` - Parameter files (params-sample.json)
+
+**Key Configuration Files:**
+- `pyproject.toml` - Project metadata, dependencies, ruff/pytest/setuptools config
+- `uv.lock` - Locked dependency versions (DO NOT modify manually)
+- `params-schema.json` - JSON schema for model parameters (deployed to GitHub Pages)
+
+### Architecture Overview
+
+**Model Hierarchy:**
+- `Model` (base class in model.py) - Common model functionality
+  - `InpatientsModel` - Inpatient demand modeling
+  - `OutpatientsModel` - Outpatient demand modeling  
+  - `AaEModel` - A&E demand modeling
+
+**Execution Flow:**
+1. `__main__.py` parses CLI arguments and loads parameters
+2. `run.py` orchestrates model execution (single or parallel runs)
+3. `ModelIteration` runs a single model iteration
+4. Results are aggregated and saved by `results.py`
+
+**Data Loading:**
+- Abstract `Data` interface allows multiple data sources
+- `Local` loads from local parquet files
+- `DatabricksNational` loads from Databricks (used in notebooks)
+
+## CI/CD Validation Pipeline
+
+### Pull Request Checks
+
+**Every pull request triggers these workflows (ALL MUST PASS):**
+
+1. **Linting** (`.github/workflows/linting.yaml`):
+   - `ruff check` - Code quality checks
+   - `ruff format --check` - Code formatting verification
+   - `ty check .` - Type checking
+
+2. **Code Coverage** (`.github/workflows/codecov.yaml`):
+   - Runs unit tests with coverage
+   - Uploads to Codecov
+   - Requires passing tests
+
+**IMPORTANT:** All linting and test checks must pass before merge. DO NOT skip or disable these checks.
+
+### Main Branch / Release Workflows
+
+On push to main or tags:
+
+1. **build_app.yaml**: Builds Python wheel, uploads to Azure Storage and GitHub releases
+2. **build_schema.yaml**: Deploys params-schema.json to GitHub Pages
+3. **build_container.yaml**: Builds and pushes Docker image to GitHub Container Registry
+4. **deploy_docs.yaml**: Builds and deploys MkDocs documentation to RStudio Connect
+
+### Docker Deployment
+
+The model is containerized using:
+- Base image: `ghcr.io/astral-sh/uv:python3.11-alpine`
+- Build args: `app_version`, `data_version`, `storage_account`
+- Entry point: `python -m nhp.docker`
+- Tags: `dev` (PRs), `v*.*.*` (releases), `latest` (latest release)
+
+## Common Issues and Workarounds
+
+**Known Issues:**
+1. **Dockerfile Version**: Uses `ENV SETUPTOOLS_SCM_PRETEND_VERSION=v0.0.0` because setuptools-scm needs git metadata (TODO: build wheel and copy instead)
+2. **Data Structure**: Model expects parquet files at `{data_path}/{file}/fyear={year}/dataset={dataset}/`. Missing files cause runtime errors.
+3. **Notebooks**: `notebooks/` directory excluded from linting - don't lint these Databricks notebooks.
+
+**Environment Variables (Docker):**
+- `APP_VERSION`, `DATA_VERSION` (default: "dev")
+- `STORAGE_ACCOUNT` (required for Azure), `BATCH_SIZE` (default: 16)
+- `.env` file supported via python-dotenv for local development
+
+## Testing Strategy
+
+- **Unit Tests**: `tests/unit/` - Mock-based, parameterized. **ALWAYS run before committing.**
+- **Integration Tests**: `tests/integration/` - Require properly formatted test data, test end-to-end runs
+- **Test Organization**: pytest-mock for mocking, fixtures in `tests/conftest.py`
+- **Coverage**: High coverage maintained via Codecov integration
+
+## Best Practices for Coding Agents
+
+1. **ALWAYS install dependencies first**: Run `uv sync --extra dev` before any development work.
+
+2. **ALWAYS run linting before committing**: Run `uvx ruff check .` and `uvx ruff format --check .` - these MUST pass.
+
+3. **ALWAYS run unit tests**: Run `uv run pytest tests/unit` before committing - all tests MUST pass.
+
+4. **Follow Google docstring convention**: All public functions/classes must have Google-style docstrings (enforced by ruff).
+
+5. **Respect line length**: Maximum 100 characters per line (ruff will enforce this).
+
+6. **Don't modify notebooks**: The `notebooks/` directory is excluded from linting for a reason. These are Databricks notebooks with special formatting.
+
+7. **Use uv for all Python commands**: Prefix commands with `uv run` to ensure correct virtual environment usage.
+
+8. **Don't modify uv.lock manually**: Use `uv sync` to update dependencies.
+
+9. **Test locally before pushing**: The CI checks are strict and will fail if linting/tests don't pass.
+
+10. **Understand the data structure**: The model requires specific data formats. If testing model execution, ensure proper test data is available or use existing test fixtures.
+
+## Quick Reference
+
+```bash
+# Setup (production + dev dependencies)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+uv sync --extra dev
+
+# Lint (MUST pass)
+uvx ruff check .
+uvx ruff format --check .
+
+# Test (MUST pass)  
+uv run pytest tests/unit --verbose
+
+# Build
+uv build
+
+# Run model (requires data)
+uv run python -m nhp.model queue/params-sample.json -d data --type all
+
+# Build docs (requires docs extras)
+uv sync --extra docs
+uv run mkdocs build --clean
+```
+
+**When in doubt, check the CI workflows in `.github/workflows/` - they define the exact validation steps used in the pipeline.**

nhp_model-5.0.0/.github/dependabot.yml

@@ -0,0 +1,16 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  - package-ecosystem: "uv"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+
+    groups:
+      all-dependencies:
+        patterns:
+          - "*"

nhp_model-5.0.0/.github/workflows/build_app.yaml

@@ -0,0 +1,79 @@
+on:
+  push:
+    branches:
+      - main
+    tags:
+      - "v*"
+
+jobs:
+  build-app:
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout GitHub Action"
+        uses: actions/checkout@v6
+
+      - name: Install the latest version of uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: Build (release)
+        if: github.ref != 'refs/heads/main'
+        run: uv build
+
+      - name: Build (dev)
+        if: github.ref == 'refs/heads/main'
+        env:
+          SETUPTOOLS_SCM_PRETEND_VERSION: 0.dev0
+        run: uv build
+
+      - name: Generate artifact
+        uses: actions/upload-artifact@v6
+        with:
+          name: dist-whl
+          path: dist/*.whl
+
+  upload-build-to-storage-account:
+    runs-on: ubuntu-latest
+    needs: ["build-app"]
+
+    steps:
+      - name: Download build artifact
+        uses: actions/download-artifact@v8
+        with:
+          name: dist-whl
+          path: .
+
+      - name: Install Azure CLI
+        uses: Azure/setup-azd@v2
+
+      - name: Upload to blob storage
+        run: |
+          az storage blob upload \
+            --account-name ${{ secrets.NHP_STORAGE_ACCOUNT }} \
+            --container-name app \
+            --file $(ls *.whl) \
+            --sas-token "${{ secrets.APP_CONTAINER_SAS }}" \
+            --overwrite
+
+  add-build-to-release:
+    runs-on: ubuntu-latest
+    needs: ["build-app"]
+    permissions:
+      contents: write
+
+    steps:
+      - name: Download build artifact
+        uses: actions/download-artifact@v8
+        with:
+          name: dist-whl
+          path: .
+      - name: Upload artifact to the GitHub Release
+        uses: softprops/action-gh-release@v2
+        if: github.ref_type == 'tag'
+        with:
+          files: "*.whl"
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

nhp_model-5.0.0/.github/workflows/build_container.yaml

@@ -0,0 +1,48 @@
+name: Build and push container image
+
+on:
+  workflow_call:
+    inputs:
+      docker-tag:
+        required: true
+        default: dev
+        type: string
+      app-version:
+        required: true
+        default: dev
+        type: string
+      data-version:
+        required: true
+        default: dev
+        type: string
+
+jobs:
+  build-container:
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout GitHub Action"
+        uses: actions/checkout@v6
+
+      - name: "Login to GitHub Container Registry"
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: "Build image"
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          tags: ${{ inputs.docker-tag }}
+          push: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+          platforms: linux/amd64
+          provenance: false
+          sbom: false
+          build-args: |
+            app_version=${{ inputs.app-version }}
+            data_version=${{ inputs.data-version }}

nhp_model-5.0.0/.github/workflows/build_schema.yaml

@@ -0,0 +1,57 @@
+name: Deploy schema.json to GitHub Pages
+
+on:
+  workflow_call:
+    inputs:
+      schema-tag:
+        required: true
+        default: dev
+        type: string
+
+permissions:
+  pages: write
+  id-token: write
+  contents: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Clone existing schemas branch content
+        run: |
+          git fetch --depth=1 origin schemas
+          git worktree add schemas schemas
+
+      - name: Copy schema to app version path
+        run: |
+          mkdir -p schemas/${{ inputs.schema-tag }}
+          sed '/$id/ s/dev/${{ inputs.schema-tag }}/' src/nhp/model/params/params-schema.json > schemas/${{ inputs.schema-tag }}/params-schema.json
+
+      - name: Commit the schema
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          pushd schemas
+          git add ${{ inputs.schema-tag }}/params-schema.json
+          git commit -m "adding schema for ${{ inputs.schema-tag }}" || echo "No changes to commit"
+          git push origin schemas
+          popd
+
+      - name: Upload to GitHub Pages
+        uses: actions/upload-pages-artifact@v4
+        with:
+          path: schemas
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

nhp_model-5.0.0/.github/workflows/codecov.yaml

@@ -0,0 +1,35 @@
+name: CodeCov
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  run:
+    runs-on: ubuntu-latest
+
+    defaults:
+      run:
+        shell: bash -l {0}
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install the latest version of uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"
+          activate-environment: true
+
+      - name: Install dependencies
+        run: uv sync --dev
+
+      - name: Generate Report
+        run: uv run pytest --cov=. tests/unit --ignore=tests --cov-branch --cov-report xml:coverage.xml
+
+      - name: Upload Coverage to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}

nhp_model-5.0.0/.github/workflows/deploy_dev.yaml

@@ -0,0 +1,22 @@
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+name: Deploy Dev
+
+jobs:
+  deploy-ghcr-dev:
+    uses: ./.github/workflows/build_container.yaml
+    with:
+      docker-tag: ghcr.io/the-strategy-unit/nhp_model:dev
+      app-version: dev
+      data-version: dev
+    secrets: inherit
+
+  deploy-dev-schema:
+    uses: ./.github/workflows/build_schema.yaml
+    with:
+      schema-tag: dev
+    secrets: inherit

nhp_model-5.0.0/.github/workflows/deploy_docs.yaml

@@ -0,0 +1,57 @@
+name: Deploy Documentation
+
+on:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: "pages"
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Set up Python
+        run: uv python install
+
+      - name: Install dependencies
+        run: uv sync --group docs
+
+      - name: Build documentation
+        run: uv run mkdocs build --clean
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v6
+        with:
+          name: site
+          path: ./site
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: Download artifact
+        uses: actions/download-artifact@v8
+        with:
+          name: site
+          path: ./site
+
+      - name: Install uv (for rsconnect)
+        uses: astral-sh/setup-uv@v7
+
+      - name: Configure Connect
+        run: uvx rsconnect add -s ${{ secrets.RSCONNECT_URL }} -n connect -k ${{ secrets.RSCONNECT_API_KEY }}
+
+      - name: Deploy to Connect
+        run: uvx rsconnect deploy html site -a ${{ vars.CONNECT_DOCS_APP_ID }}

nhp_model-5.0.0/.github/workflows/deploy_pr.yaml

@@ -0,0 +1,47 @@
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+
+name: Deploy PR
+
+jobs:
+  deploy-ghcr-pr:
+    # only allow this workflow to run on PRs from *this* repository, not from forks
+    # this is to prevent the possibility of a malicious actor creating a PR from a fork that triggers this workflow and
+    # pushes a container image to our registry
+    if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository }}
+    uses: ./.github/workflows/build_container.yaml
+    with:
+      docker-tag: ghcr.io/the-strategy-unit/nhp_model:pr-${{ github.event.number }}
+      app-version: dev
+      data-version: dev
+    secrets: inherit
+
+  add-comment-to-pr:
+    runs-on: ubuntu-latest
+    needs: ["deploy-ghcr-pr"]
+    steps:
+      - name: Find Comment
+        uses: peter-evans/find-comment@v4
+        id: fc
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          comment-author: "github-actions[bot]"
+          body-includes: "## ✅ A new build is available"
+
+      - name: Comment with container image link
+        if: github.event_name == 'pull_request'
+        uses: peter-evans/create-or-update-comment@v5
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          comment-id: ${{ steps.fc.outputs.comment-id }}
+          issue-number: ${{ github.event.pull_request.number }}
+          body: |
+            ## ✅ A new build is available.
+
+            You can use the following to use pull the image into your local environment:
+
+            ``` bash
+            docker pull ghcr.io/the-strategy-unit/nhp_model:pr-${{ github.event.number }}
+            ```
+          edit-mode: replace

nhp_model-5.0.0/.github/workflows/deploy_release.yaml

@@ -0,0 +1,35 @@
+on:
+  push:
+    tags:
+      - "v*.*.*"
+
+name: Deploy Production
+
+jobs:
+  set-tag:
+    runs-on: ubuntu-latest
+    outputs:
+      tag: ${{ steps.create-tag.outputs.TAG }}
+
+    steps:
+      - name: Create tag
+        id: create-tag
+        run: |
+          TAG=`echo ${{ github.ref_name }} | awk 'BEGIN { FS="."; } { print ""$1"."$2; }'`
+          echo "TAG=$TAG" >> $GITHUB_OUTPUT
+
+  deploy-ghcr-production:
+    needs: [set-tag]
+    uses: ./.github/workflows/build_container.yaml
+    with:
+      docker-tag: ghcr.io/the-strategy-unit/nhp_model:${{ needs.set-tag.outputs.tag }},ghcr.io/the-strategy-unit/nhp_model:latest
+      app-version: ${{ github.ref_name }}
+      data-version: ${{vars.data_version}}
+    secrets: inherit
+
+  deploy-schema:
+    needs: [set-tag]
+
+    uses: ./.github/workflows/build_schema.yaml
+    with:
+      schema-tag: ${{ needs.set-tag.outputs.tag }}