matchbox-db 0.8.0__tar.gz → 0.8.2.dev56__tar.gz

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/.github/workflows/ci.yml

@@ -55,6 +55,21 @@ jobs:
       - name: Copy environment variables
         run: cp environments/development.env .env
 
+      - name: Generate test keys
+        run: |
+          uv run test/scripts/authorisation.py keygen > /tmp/keys.json
+          
+          echo "MB__DEV__PRIVATE_KEY=\"$(jq -r '.private_key' /tmp/keys.json)\"" >> .env
+          
+          echo "MB__SERVER__PUBLIC_KEY=\"$(jq -r '.public_key' /tmp/keys.json)\"" >> .env
+                
+          echo "MB__CLIENT__JWT=$(jq -r .private_key /tmp/keys.json | \
+            uv run test/scripts/authorisation.py jwt \
+              --sub e9ba93b3-e4d5-4dee-868c-d011116142bd \
+              --api-root https://localhost:8000/ \
+              --expiry 999999 | \
+            tr -d '\n')" >> .env
+
       - name: Run databases and API
         run: docker compose --env-file .env --env-file environments/containers.env up -d --wait
 
@@ -132,4 +147,4 @@ jobs:
     
     steps:
       - name: CI success
-        run: echo "All CI checks passed!"
+        run: echo "All CI checks passed!"

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/.pre-commit-config.yaml

@@ -31,7 +31,7 @@ repos:
 
 # Mandatory internal hooks
 - repo: https://github.com/uktrade/github-standards
-  rev: v1.2.1  # update periodically with pre-commit autoupdate
+  rev: v1.3.0  # update periodically with pre-commit autoupdate
   hooks:
     - id: run-security-scan
       verbose: false

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: matchbox-db
-Version: 0.8.0
+Version: 0.8.2.dev56
 Summary: A framework for orchestrating and comparing data linking and deduplication methodologies.
 Author: Department for Business and Trade
 Project-URL: Documentation, https://uktrade.github.io/matchbox/

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/docs/api/common/factories/index.md

@@ -221,17 +221,7 @@ def test_my_adapter_function(my_backend_adapter):
 
 The scenario system is cached, so subsequent runs of the same scenario are significantly faster.
 
-### Available scenarios
-
-The following built-in scenarios are available. They are built on top of each other, so `link` includes all the steps from `dedupe`, which includes `index`, and so on.
-
-*   **`bare`**: Creates a set of linked sources and writes them to the data warehouse, but does not interact with the matchbox backend.
-*   **`index`**: Takes the `bare` scenario and indexes all the sources in the matchbox backend.
-*   **`dedupe`**: Takes the `index` scenario and adds perfectly deduplicated models for each source.
-*   **`probabilistic_dedupe`**: Like `dedupe`, but the models produce probabilistic scores rather than perfect matches.
-*   **`link`**: Takes the `dedupe` scenario and adds linking models between the deduplicated sources, culminating in a `final_join` resolution.
-*   **`alt_dedupe`**: A specialised scenario with two alternative deduplication models for the same source.
-*   **`convergent`**: A specialised scenario where two different sources index to almost identical data.
+See [the scenarios API][matchbox.common.factories.scenarios] for a full list of available scenarios.
 
 ### Creating new scenarios
 

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/docs/contributing.md

@@ -11,6 +11,24 @@ This document describes how you can get started with developing Matchbox.
 
 Set up environment variables by creating a `.env` file under project directory. See [`/environments/development.env`](https://github.com/uktrade/matchbox/blob/main/environments/development.env) for sensible defaults, which should work as is with the Docker Compose set-up and the tests.
 
+Generate dummy public/private keys and a JWT for local testing. The below will add the required variables to the bottom of your `.env` file.
+
+```shell
+uv run test/scripts/authorisation.py keygen > /tmp/keys.json
+
+echo "MB__DEV__PRIVATE_KEY=\"$(jq -r '.private_key' /tmp/keys.json)\"" >> .env
+
+echo "MB__SERVER__PUBLIC_KEY=\"$(jq -r '.public_key' /tmp/keys.json)\"" >> .env
+
+echo "MB__CLIENT__JWT=$(jq -r .private_key /tmp/keys.json | \
+    uv run test/scripts/authorisation.py jwt \
+        --sub e9ba93b3-e4d5-4dee-868c-d011116142bd \
+        --email test@example.org \
+        --api-root https://localhost:8000/ \
+        --expiry 999999 | \
+    tr -d '\n')" >> .env
+```
+
 This project is managed by [uv](https://docs.astral.sh/uv/), linted and formated with [ruff](https://docs.astral.sh/ruff/), and tested with [pytest](https://docs.pytest.org/en/stable/). [Docker](https://www.docker.com) is used for local development. Documentation is build using [mkdocs](https://www.mkdocs.org).
 
 To install all dependencies for this project, run:
@@ -184,4 +202,4 @@ We suggest using tools like [`wayneashleyberry/gh-act`](https://github.com/wayne
 gh act update --pin
 ```
 
-You will still need to independently verify that the new pins are safe.
+You will still need to independently verify that the new pins are safe.

matchbox_db-0.8.2.dev56/docs/server/cli.md

@@ -0,0 +1,59 @@
+# Command line interface
+
+Matchbox comes with a command line interface (CLI) that allows users to perform certain management tasks.
+
+To run the CLI:
+
+```shell
+mbx [OPTIONS] COMMAND [ARGS]
+```
+
+To get help:
+
+```shell
+mbx --help
+```
+
+To get help on specific commands:
+
+```shell
+mbx COMMAND --help
+```
+
+## How do I?
+
+### Bootstrap Matchbox
+
+The first user is always the system admin. As soon as you run the server for the first time, log in to automatically be added to the admins group.
+
+```
+mbx login
+```
+
+### Manage permissions
+
+Permissions are defined between groups and resources. To manage groups:
+
+```shell
+mbx groups
+```
+
+For example, to manage permissions on a collection resource, see its related commands:
+
+```shell
+mbx collections -c COLLECTION permissions
+mbx collections -c COLLECTION grant -g GROUP -p PERMISSION
+mbx collections -c COLLECTION revoke -g GROUP -p PERMISSION
+```
+
+See [permissions](permissions.md) for more information.
+
+### Delete orphans
+
+When resolutions are modified or deleted, it is possible that the database ends up having clusters which are not related to any table containing sources, models or evaluations. These clusters are considered orphaned, and they should be deleted regularly to reduce bloat.
+
+```shell
+mbx admin prune
+```
+
+This command will print the number of orphaned clusters deleted.

matchbox_db-0.8.2.dev56/docs/server/permissions.md

@@ -0,0 +1,45 @@
+!!! warning
+    Permissions are an experimental feature in Matchbox. Disable them with `MB__SERVER__AUTHORISATION=False`.
+
+    They currently assume an external identity provider uses a public, private and JWT generated elsewhere, and places them into the Matchbox client and server environments.
+
+Matchbox uses a Role-Based Access Control (RBAC) system to manage security. Permissions are granted to **Groups**, and **Users** are assigned membership in those groups.
+
+## Bootstrapping
+
+When a Matchbox server is first deployed, it has no admins. To bootstrap setup:
+
+```shell
+mbx login
+```
+
+* The **first user** to log in is automatically assigned to the **admins** group
+* This user becomes the initial system administrator
+* All subsequent users are treated as standard users
+
+Bootstrap as soon as the server is deployed. If you boostrap and aren't made admin, either:
+
+* Delete the database and try again
+* Contact whoever became the admin user is to add you to the admin group
+
+## Defaults
+
+* **admins**: Members of this group have full control over the system
+* **public**: Every user who logs in is automatically added to the `public` group. This ensures baseline access control for all authenticated users
+
+## Permission hierarchy
+
+Permissions in Matchbox are hierarchical. Possessing a higher-level permission automatically grants all lower-level permissions for that resource.
+
+| Level | Capability | Includes |
+| :--- | :--- | :--- |
+| **Admin** | Manage permissions, delete resources, full control. | Read, Write |
+| **Write** | Create and modify data (e.g., upload sources, run models). | Read |
+| **Read** | View data, configurations, and results. | *None* |
+
+### Resources
+
+Permissions are applied to specific resources:
+
+* **System**: Permissions apply to the system as a whole, such as group creation or dumping the database
+* **Collection**: Permissions apply only to a specific named collection, such as read access to the "finance" collection

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/docs/server/risks.md

@@ -2,8 +2,8 @@ This page contains a list of known data and security caveats determined by the c
 
 ## Sensitive data sources
 
-!!! danger
-    Matchbox allows **any user** to view all identifier values in an indexed dataset. There is no authorisation model for reading data. Anyone can query any source.
+!!! warning
+    Matchbox allows users with **read** permission on a collection to view all identifier values in that collection's indexed datasets.
 
 ### What's usually safe
 
@@ -26,12 +26,10 @@ Problems arise when:
 !!! tip
     Do not use Matchbox to index sources where the key fields are sensitive data.
 
-Future versions of Matchbox will address this limitation by introducing an authorisation mechanism.
-
 ## Table metadata
 
-!!! danger
-    Matchbox allows any user see the **list of fields** and extract-transform logic of an indexed source.
+!!! warning
+    Matchbox allows users with **read** permission to see the **list of fields** and extract-transform logic of an indexed source.
 
 !!! tip
     Avoid indexing sources where the **field names** or **structure** are considered sensitive.
@@ -44,7 +42,7 @@ Future versions of Matchbox will address this limitation by introducing an autho
 **Example**:
 
 * A source from `RelationalDBLocation` stores SQL queries used during indexing.
-* If an automated pipeline reuses these queries, a malicious user could inject harmful SQL (e.g., `DROP DATABASE`).
+* If an automated pipeline reuses these queries, a malicious user with **write** permission could inject harmful SQL (e.g., `DROP DATABASE`).
 
 Matchbox performs **basic validation**, but **cannot guarantee query safety**.
 

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/environments/development.env

@@ -1,4 +1,5 @@
 ## COPY THIS FILE to .env in the project root ##
+## See conttributing.md for instructions on setting keys and JWTs ##
 
 # Port configuration for Docker Compose
 # If you change these values, you might have to update a corresponding
@@ -12,9 +13,8 @@ MB__DEV__WAREHOUSE_PORT=7654
 MB__DEV__POSTGRES_BACKEND_PORT=5432
 MB__DEV__REDIS_PORT=6379
 MB__DEV__FLOWER_PORT=5555
-# Set with MB__SERVER__PUBLIC_KEY using uv run test/scripts/authorisation.py keygen
-# trufflehog:ignore
-MB__DEV__PRIVATE_KEY=-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VwBCIEIPMJVIQACD9QNGi0219FHyDQUk8OUP1vkGwkdsis58hZ\n-----END PRIVATE KEY-----\n
+# See contributing.md for instructions on setting
+# MB__DEV__PRIVATE_KEY=
 
 # Client configuration
 MB__CLIENT__API_ROOT=http://localhost:8000
@@ -22,20 +22,21 @@ MB__CLIENT__TIMEOUT=10
 MB__CLIENT__RETRY_DELAY=2
 MB__CLIENT__USER=scott.mcgregor
 MB__CLIENT__BATCH_SIZE=250_000
+# See contributing.md for instructions on setting
+# MB__CLIENT__JWT=
 
 # Server configuration for when the API is run outside of Docker
 # e.g. when running in a debugger, or when checking local DB migrations
 MB__SERVER__LOG_LEVEL=INFO
 MB__SERVER__BATCH_SIZE=250_000
 MB__SERVER__BACKEND_TYPE=postgres
-MB__SERVER__AUTHORISATION=False
+MB__SERVER__AUTHORISATION=True
 
 MB__SERVER__TASK_RUNNER=celery
 MB__SERVER__REDIS_URI=redis://localhost:6379/0
 MB__SERVER__UPLOADS_EXPIRY_MINUTES=10_080
-# Set with MB__DEV__PRIVATE_KEY using uv run test/scripts/authorisation.py keygen
-# trufflehog:ignore
-MB__SERVER__PUBLIC_KEY=-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEAg6Jqls79JB04WSlqMkP5ta9gD62t5DyKNj0v6fWCemg=\n-----END PUBLIC KEY-----\n
+# See contributing.md for instructions on setting
+# MB__SERVER__PUBLIC_KEY=
 
 MB__SERVER__DATASTORE__HOST=localhost
 MB__SERVER__DATASTORE__PORT=9000

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/justfile

@@ -1,5 +1,3 @@
-set dotenv-load := true
-
 # Unit testing
 mod test 'test/justfile'
 # PostgreSQL migration

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/mkdocs.yml

@@ -19,6 +19,7 @@ nav:
 - Server:
   - Matchbox concepts: server/concepts.md
   - Installation: server/install.md
+  - Access control and permissions: server/permissions.md
   - CLI: server/cli.md
   - Data and security risks: server/risks.md
   - API: 

{matchbox_db-0.8.0 → matchbox_db-0.8.2.dev56}/pyproject.toml

@@ -74,7 +74,8 @@ typing = [
 ]
 
 [project.scripts]
-matchbox = "matchbox.client.cli.main:app"
+matchbox = "matchbox.client.cli.main:run"
+mbx = "matchbox.client.cli.main:run"
 
 [project.urls]
 "Documentation" = "https://uktrade.github.io/matchbox/"
@@ -86,7 +87,7 @@ build-backend = "setuptools.build_meta"
 
 # Presence enables setuptools-scm
 [tool.setuptools_scm]
-
+fallback_version = "0.0.0"
 
 [tool.uv]
 default-groups = ["dev", "typing"]
@@ -96,7 +97,8 @@ constraint-dependencies = [
     "urllib3>=2.6.3",
     "werkzeug>=3.1.5",
     "setuptools-scm>=9.2.0",
-    "filelock>=3.20.1"
+    "filelock>=3.20.3",
+    "virtualenv>=20.36.1",
 ]
 
 [tool.ruff]

matchbox_db-0.8.2.dev56/src/matchbox/client/_handler/__init__.py

@@ -0,0 +1,88 @@
+"""Functions abstracting the interaction with the server API."""
+
+from matchbox.client._handler.admin import (
+    count_backend_items,
+    delete_orphans,
+)
+from matchbox.client._handler.auth import auth_status, login
+from matchbox.client._handler.collections import (
+    create_collection,
+    create_resolution,
+    create_run,
+    delete_collection,
+    delete_resolution,
+    delete_run,
+    get_collection,
+    get_collection_permissions,
+    get_resolution,
+    get_resolution_stage,
+    get_results,
+    get_run,
+    grant_collection_permission,
+    list_collections,
+    revoke_collection_permission,
+    set_data,
+    set_run_default,
+    set_run_mutable,
+    update_resolution,
+)
+from matchbox.client._handler.eval import (
+    download_eval_data,
+    sample_for_eval,
+    send_eval_judgement,
+)
+from matchbox.client._handler.groups import (
+    add_user_to_group,
+    create_group,
+    delete_group,
+    get_group,
+    list_groups,
+    remove_user_from_group,
+)
+from matchbox.client._handler.main import healthcheck
+from matchbox.client._handler.query import match, query
+
+__all__ = [
+    # auth
+    "auth_status",
+    "login",
+    # admin
+    "count_backend_items",
+    "delete_orphans",
+    # groups
+    "add_user_to_group",
+    "create_group",
+    "delete_group",
+    "get_group",
+    "list_groups",
+    "remove_user_from_group",
+    # main
+    "healthcheck",
+    # eval
+    "download_eval_data",
+    "sample_for_eval",
+    "send_eval_judgement",
+    # query
+    "query",
+    "match",
+    # collections
+    "create_collection",
+    "create_resolution",
+    "create_run",
+    "delete_collection",
+    "delete_resolution",
+    "delete_run",
+    "get_collection",
+    "get_collection_permissions",
+    "get_resolution",
+    "get_resolution_stage",
+    "get_results",
+    "get_run",
+    "grant_collection_permission",
+    "list_collections",
+    "revoke_collection_permission",
+    "set_data",
+    "set_run_default",
+    "set_run_mutable",
+    "update_resolution",
+]

matchbox_db-0.8.2.dev56/src/matchbox/client/_handler/admin.py

@@ -0,0 +1,39 @@
+"""Admin functions for the client handler."""
+
+from matchbox.client._handler.main import CLIENT, http_retry, url_params
+from matchbox.common.dtos import (
+    BackendCountableType,
+    ResourceOperationStatus,
+)
+from matchbox.common.logging import logger
+
+
+@http_retry
+def count_backend_items(
+    entity: BackendCountableType | None = None,
+) -> dict[str, int]:
+    """Count the number of various entities in the backend."""
+    if entity is not None and entity not in BackendCountableType:
+        raise ValueError(
+            f"Invalid entity type: {entity}. "
+            f"Must be one of {list(BackendCountableType)} "
+        )
+
+    log_prefix = "Backend count"
+    logger.debug("Counting", prefix=log_prefix)
+
+    res = CLIENT.get("/database/count", params=url_params({"entity": entity}))
+
+    counts = res.json()
+    logger.debug(f"Counts: {counts}", prefix=log_prefix)
+
+    return counts
+
+
+@http_retry
+def delete_orphans() -> ResourceOperationStatus:
+    """Delete orphaned clusters."""
+    logger.debug("Deleting orphans")
+
+    res = CLIENT.delete("/database/orphans")
+    return ResourceOperationStatus.model_validate(res.json())

matchbox_db-0.8.2.dev56/src/matchbox/client/_handler/auth.py

@@ -0,0 +1,24 @@
+"""Authentication functions for the client handler."""
+
+from matchbox.client._handler.main import CLIENT, http_retry
+from matchbox.common.dtos import (
+    AuthStatusResponse,
+    LoginResponse,
+)
+from matchbox.common.logging import logger
+
+
+@http_retry
+def login() -> LoginResponse:
+    """Log into Matchbox."""
+    logger.debug("Login attempt using JWT")
+    response = CLIENT.post("/auth/login")
+    return LoginResponse.model_validate(response.json())
+
+
+@http_retry
+def auth_status() -> AuthStatusResponse:
+    """Check authentication status."""
+    logger.debug("Checking authentication status")
+    response = CLIENT.get("/auth/status")
+    return AuthStatusResponse.model_validate(response.json())