microdata-tools 1.14.0__tar.gz → 2.0.0__tar.gz

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: microdata-tools
-Version: 1.14.0
+Version: 2.0.0
 Summary: Tools for the microdata.no platform
 Author: microdata-developers
 License: MIT
@@ -35,14 +35,13 @@ The CSV file is optional in some cases.
 The `package_dataset()` function will encrypt and package your dataset as a tar archive. The process is as follows:
 
 1. Generate the symmetric key for a dataset.
-2. Encrypt the dataset data (CSV) using the symmetric key and store the encrypted file as `<DATASET_NAME>.csv.encr`
-3. Encrypt the symmetric key using the asymmetric RSA public key `microdata_public_key.pem` 
-   and store the encrypted file as `<DATASET_NAME>.symkey.encr`
-4. Gather the encrypted CSV, encrypted symmetric key and metadata (JSON) file in one tar file.
+2. Encrypt the dataset data (CSV) using an AES-256-GCM symmetric key and store the encrypted file as `<DATASET_NAME>.csv.encr`
+3. Encrypt the symmetric key using HPKE with the combined ML-KEM-768/X25519 public key from `microdata_public_key.pem` and store the resulting HPKE ciphertext as <DATASET_NAME>.kem.encr
+4. Gather the encrypted CSV, ciphertext file and metadata (JSON) file in one tar file.
 
 ### Unpackage dataset
-The `unpackage_dataset()` function will untar and decrypt your dataset using the `microdata_private_key.pem`
-RSA private key.
+The `unpackage_dataset()` function will untar and your dataset and use the combined ML-KEM-768/X25519 private key from `microdata_private_key.pem` to recover the symmetric key, which is then used to decrypt the dataset.
+
 
 The packaged file has to have the `<DATASET_NAME>.tar` extension. Its contents should be as follows:
 
@@ -50,30 +49,33 @@ The packaged file has to have the `<DATASET_NAME>.tar` extension. Its contents s
 
 ```<DATASET_NAME>.csv.encr``` : Optional encrypted dataset file.
 
-```<DATASET_NAME>.symkey.encr``` : Optional encrypted file containing the symmetrical key used to decrypt the dataset file. Required if the `.csv.encr` file is present.
+```<DATASET_NAME>.kem.encr``` : Optional HPKE ciphertext file containing the encrypted symmetric key required to decrypt the dataset file. Required if the `.csv.encr` file is present.
 
-Decryption uses the RSA private key located at ```RSA_KEY_DIR```.
+Decryption uses the combined ML-KEM-768/X25519 private key located at ```PRIVATE_KEY_DIR``` to recover the symmetric decryption key.
 
 The packaged file is then stored in `output_dir/archive/unpackaged` after a successful run or `output_dir/archive/failed` after an unsuccessful run.
 
+
 ## Example
-Python script that uses a RSA public key named `microdata_public_key.pem` and packages a dataset:
+
+Store your metadata and data files according to the structure described above, and put
+the provided public key in a directory of your choice.
+Then:
 
 ```py
 from pathlib import Path
 from microdata_tools import package_dataset
 
-RSA_KEYS_DIRECTORY = Path("tests/resources/rsa_keys")
-DATASET_DIRECTORY = Path("tests/resources/input_package/DATASET_1")
-OUTPUT_DIRECTORY = Path("tests/resources/output")
-
 package_dataset(
-   rsa_keys_dir=RSA_KEYS_DIRECTORY,
-   dataset_dir=DATASET_DIRECTORY,
-   output_dir=OUTPUT_DIRECTORY,
+    public_key_dir=Path("path/to/key_directory"),
+    dataset_dir=Path("path/to/MY_DATASET_NAME"),
+    output_dir=Path("path/to/output"),
 )
 ```
 
+This produces `path/to/output/MY_DATASET_NAME.tar`, which can be uploaded to microdata.
+
+
 ### Validation
 
 Once you have your metadata and data files ready to go, they should be named and stored like this:
@@ -104,7 +106,7 @@ else:
         print(error)
 ```
 
- For a more in-depth explanation of usage visit [the usage documentation](https://statisticsnorway.github.io/microdata-tools/usage.md).
+ For a more in-depth explanation of usage visit [the usage documentation](https://statisticsnorway.github.io/microdata-tools/USAGE).
 
  ### Data format description
 A dataset as defined in microdata consists of one data file, and one metadata file.

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/README.md

@@ -21,14 +21,13 @@ The CSV file is optional in some cases.
 The `package_dataset()` function will encrypt and package your dataset as a tar archive. The process is as follows:
 
 1. Generate the symmetric key for a dataset.
-2. Encrypt the dataset data (CSV) using the symmetric key and store the encrypted file as `<DATASET_NAME>.csv.encr`
-3. Encrypt the symmetric key using the asymmetric RSA public key `microdata_public_key.pem` 
-   and store the encrypted file as `<DATASET_NAME>.symkey.encr`
-4. Gather the encrypted CSV, encrypted symmetric key and metadata (JSON) file in one tar file.
+2. Encrypt the dataset data (CSV) using an AES-256-GCM symmetric key and store the encrypted file as `<DATASET_NAME>.csv.encr`
+3. Encrypt the symmetric key using HPKE with the combined ML-KEM-768/X25519 public key from `microdata_public_key.pem` and store the resulting HPKE ciphertext as <DATASET_NAME>.kem.encr
+4. Gather the encrypted CSV, ciphertext file and metadata (JSON) file in one tar file.
 
 ### Unpackage dataset
-The `unpackage_dataset()` function will untar and decrypt your dataset using the `microdata_private_key.pem`
-RSA private key.
+The `unpackage_dataset()` function will untar and your dataset and use the combined ML-KEM-768/X25519 private key from `microdata_private_key.pem` to recover the symmetric key, which is then used to decrypt the dataset.
+
 
 The packaged file has to have the `<DATASET_NAME>.tar` extension. Its contents should be as follows:
 
@@ -36,30 +35,33 @@ The packaged file has to have the `<DATASET_NAME>.tar` extension. Its contents s
 
 ```<DATASET_NAME>.csv.encr``` : Optional encrypted dataset file.
 
-```<DATASET_NAME>.symkey.encr``` : Optional encrypted file containing the symmetrical key used to decrypt the dataset file. Required if the `.csv.encr` file is present.
+```<DATASET_NAME>.kem.encr``` : Optional HPKE ciphertext file containing the encrypted symmetric key required to decrypt the dataset file. Required if the `.csv.encr` file is present.
 
-Decryption uses the RSA private key located at ```RSA_KEY_DIR```.
+Decryption uses the combined ML-KEM-768/X25519 private key located at ```PRIVATE_KEY_DIR``` to recover the symmetric decryption key.
 
 The packaged file is then stored in `output_dir/archive/unpackaged` after a successful run or `output_dir/archive/failed` after an unsuccessful run.
 
+
 ## Example
-Python script that uses a RSA public key named `microdata_public_key.pem` and packages a dataset:
+
+Store your metadata and data files according to the structure described above, and put
+the provided public key in a directory of your choice.
+Then:
 
 ```py
 from pathlib import Path
 from microdata_tools import package_dataset
 
-RSA_KEYS_DIRECTORY = Path("tests/resources/rsa_keys")
-DATASET_DIRECTORY = Path("tests/resources/input_package/DATASET_1")
-OUTPUT_DIRECTORY = Path("tests/resources/output")
-
 package_dataset(
-   rsa_keys_dir=RSA_KEYS_DIRECTORY,
-   dataset_dir=DATASET_DIRECTORY,
-   output_dir=OUTPUT_DIRECTORY,
+    public_key_dir=Path("path/to/key_directory"),
+    dataset_dir=Path("path/to/MY_DATASET_NAME"),
+    output_dir=Path("path/to/output"),
 )
 ```
 
+This produces `path/to/output/MY_DATASET_NAME.tar`, which can be uploaded to microdata.
+
+
 ### Validation
 
 Once you have your metadata and data files ready to go, they should be named and stored like this:
@@ -90,7 +92,7 @@ else:
         print(error)
 ```
 
- For a more in-depth explanation of usage visit [the usage documentation](https://statisticsnorway.github.io/microdata-tools/usage.md).
+ For a more in-depth explanation of usage visit [the usage documentation](https://statisticsnorway.github.io/microdata-tools/USAGE).
 
  ### Data format description
 A dataset as defined in microdata consists of one data file, and one metadata file.

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/microdata_tools/__init__.py

@@ -1,4 +1,8 @@
-from microdata_tools.packaging import package_dataset, unpackage_dataset
+from microdata_tools.packaging import (
+    package_dataset,
+    unpackage_dataset,
+)
+from microdata_tools.packaging.keys import PrivateKey, PublicKey
 from microdata_tools.validation import (
     get_unit_id_type_for_unit_type,
     validate_dataset,
@@ -11,4 +15,6 @@ __all__ = [
     "validate_dataset",
     "validate_metadata",
     "get_unit_id_type_for_unit_type",
+    "PrivateKey",
+    "PublicKey",
 ]

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/microdata_tools/packaging/__init__.py

@@ -23,14 +23,15 @@ logger = logging.getLogger()
 
 
 def package_dataset(
-    rsa_keys_dir: Path, dataset_dir: Path, output_dir: Path
+    public_key_dir: Path, dataset_dir: Path, output_dir: Path
 ) -> None:
     """
     Packages a dataset. It will encrypt and tar the dataset using
-    the provided RSA public key. Only the CSV file will be encrypted.
+    the provided combined ML-KEM-768/X25519 public key.
+    Only the CSV file will be encrypted.
     Creates a checksum file for the CSV file.
 
-    :param rsa_keys_dir:
+    :param public_key_dir:
         directory containing public key file microdata_public_key.pem
     :param dataset_dir:
         directory containing the dataset files (CSV and JSON)
@@ -59,7 +60,7 @@ def package_dataset(
         if len(csv_files) == 1:
             write_checksum_to_file(csv_files[0])
             encrypt_dataset(
-                rsa_keys_dir=rsa_keys_dir,
+                public_key_dir=public_key_dir,
                 dataset_dir=dataset_dir,
                 output_dir=output_dir,
             )
@@ -67,10 +68,10 @@ def package_dataset(
             if not dataset_output_dir.exists():
                 os.makedirs(dataset_output_dir)
 
-        if Path(dataset_dir / f"{dataset_name}.md5").exists():
+        if Path(dataset_dir / f"{dataset_name}.sha256").exists():
             shutil.move(
-                dataset_dir / f"{dataset_name}.md5",
-                dataset_output_dir / f"{dataset_name}.md5",
+                dataset_dir / f"{dataset_name}.sha256",
+                dataset_output_dir / f"{dataset_name}.sha256",
             )
 
         shutil.copyfile(
@@ -87,17 +88,18 @@ def package_dataset(
 
 def unpackage_dataset(
     packaged_file_path: Path,
-    rsa_keys_dir: Path,
+    private_key_dir: Path,
     output_dir: Path,
 ) -> None:
     """
     Unpackages a dataset. It will untar and decrypt the dataset using
-    the provided RSA private key. Only the CSV file will be decrypted.
+    the provided combined ML-KEM-768/X25519 private key.
+    Only the CSV file will be decrypted.
     Validates the checksum of the CSV file.
 
     :param packaged_file_path:
         a Path to the .tar file containing the dataset files
-    :param rsa_keys_dir:
+    :param private_key_dir:
         directory containing the private key file microdata_private_key.pem
     :param output_dir:
         output directory
@@ -105,12 +107,12 @@ def unpackage_dataset(
         None
     """
     check_exists(packaged_file_path)
-    check_exists(rsa_keys_dir)
+    check_exists(private_key_dir)
 
     if not output_dir.exists():
         os.makedirs(output_dir)
 
-    private_key_path = rsa_keys_dir / "microdata_private_key.pem"
+    private_key_path = private_key_dir / "microdata_private_key.pem"
     check_exists(private_key_path)
 
     dataset_name = packaged_file_path.stem
@@ -119,7 +121,7 @@ def unpackage_dataset(
 
     try:
         untar_encrypted_dataset(packaged_file_path, dataset_name, dataset_dir)
-        decrypt(rsa_keys_dir, dataset_dir, output_dir)
+        decrypt(private_key_dir, dataset_dir, output_dir)
         _validate_csv_consistency(dataset_name, dataset_dir, output_dir)
 
         if Path(dataset_dir).exists():
@@ -139,5 +141,5 @@ def _validate_csv_consistency(
             output_dir / dataset_name / f"{dataset_name}.csv"
         )
         compare_checksum_with_file(
-            dataset_dir / f"{dataset_name}.md5", calculated_checksum
+            dataset_dir / f"{dataset_name}.sha256", calculated_checksum
         )

microdata_tools-2.0.0/microdata_tools/packaging/_crypto_config.py

@@ -0,0 +1,10 @@
+from cryptography.hazmat.primitives.hpke import (
+    AEAD,
+    KDF,
+    KEM,
+    Suite,
+)
+
+NONCE_SIZE_BYTES = 12
+HPKE_SUITE = Suite(KEM.MLKEM768_X25519, KDF.HKDF_SHA256, AEAD.AES_256_GCM)
+HPKE_INFO = b"microdata-tools symmetric-key encryption"

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/microdata_tools/packaging/_decrypt.py

@@ -5,24 +5,34 @@ import tarfile
 from pathlib import Path
 from typing import List, Tuple
 
-from cryptography.fernet import Fernet, InvalidToken
-from cryptography.hazmat.backends import default_backend
-from cryptography.hazmat.primitives import hashes, serialization
-from cryptography.hazmat.primitives.asymmetric import padding, rsa
+from cryptography.exceptions import InvalidTag
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+from cryptography.hazmat.primitives.hpke import (
+    MLKEM768X25519PrivateKey,
+)
 
-from microdata_tools.packaging._utils import check_exists
+from microdata_tools.packaging._crypto_config import (
+    HPKE_INFO,
+    HPKE_SUITE,
+    NONCE_SIZE_BYTES,
+)
+from microdata_tools.packaging._utils import (
+    check_exists,
+)
 from microdata_tools.packaging.exceptions import (
     InvalidKeyError,
     InvalidTarFileContents,
 )
+from microdata_tools.packaging.keys import PrivateKey
 
 logger = logging.getLogger()
 
 
-def decrypt(rsa_keys_dir: Path, dataset_dir: Path, output_dir: Path) -> None:
+def decrypt(private_key_dir: Path, dataset_dir: Path, output_dir: Path) -> None:
     """
     Decrypts a dataset as follows:
-        1. Decrypts the symmetric key using the RSA private key.
+        1. Recovers the symmetric key using the hybrid private key
+           (HPKE with ML-KEM-768 + X25519).
         2. Decrypts each chunk using the symmetric key.
         3. Merges the decrypted chunks into a single file.
     """
@@ -39,36 +49,35 @@ def decrypt(rsa_keys_dir: Path, dataset_dir: Path, output_dir: Path) -> None:
     decrypted_dir = output_dataset_dir / "decrypted"
     os.makedirs(decrypted_dir, exist_ok=True)
 
-    if not output_dataset_dir.exists():
-        os.makedirs(output_dataset_dir)
-
     if chunk_dir.exists() and first_encrypted_chunk.exists():
         logger.info(f"Encrypted file found in {dataset_dir}")
 
-        with open(
-            Path(rsa_keys_dir / "microdata_private_key.pem"), "rb"
-        ) as key_file:
-            private_key = serialization.load_pem_private_key(
-                key_file.read(), password=None, backend=default_backend()
+        # Read private key from file
+        private_key = PrivateKey.load_from_file(
+            Path(private_key_dir / "microdata_private_key.pem")
+        ).to_hpke_key()
+
+        if not isinstance(private_key, MLKEM768X25519PrivateKey):
+            raise TypeError(
+                "Private key is not a hybrid ML-KEM-768-X25519 private key."
             )
+        # Read the hpke ciphertext from file and recover the symmetric key
+        hpke_ciphertext_path = Path(dataset_dir / f"{dataset_name}.kem.encr")
+        check_exists(hpke_ciphertext_path)
 
-        encrypted_symkey = Path(dataset_dir / f"{dataset_name}.symkey.encr")
-        check_exists(encrypted_symkey)
-
-        if not isinstance(private_key, rsa.RSAPrivateKey):
-            raise TypeError("Privatkey is not RSA. Cannot use .decrypt().")
-
-        with open(encrypted_symkey, "rb") as f:
-            symkey = f.read()
-        decrypted_symkey = private_key.decrypt(
-            symkey,
-            padding.OAEP(
-                mgf=padding.MGF1(algorithm=hashes.SHA256()),
-                algorithm=hashes.SHA256(),
-                label=None,
-            ),
-        )
-        fernet = Fernet(decrypted_symkey)
+        with open(hpke_ciphertext_path, "rb") as f:
+            hpke_ciphertext = f.read()
+
+        try:
+            decrypted_symkey = HPKE_SUITE.decrypt(
+                hpke_ciphertext, private_key, info=HPKE_INFO
+            )
+        except Exception as e:
+            raise InvalidKeyError(
+                "Unable to recover symmetric key. Is the private key correct?"
+            ) from e
+
+        aesgcm = AESGCM(decrypted_symkey)
 
         # Decrypt all the encrypted csv files in the directory
         for encrypted_file in chunk_dir.iterdir():
@@ -78,14 +87,18 @@ def decrypt(rsa_keys_dir: Path, dataset_dir: Path, output_dir: Path) -> None:
                 # Decrypt csv file
                 with open(encrypted_file, "rb") as file:
                     data = file.read()
+                    nonce, ciphertext = (
+                        data[:NONCE_SIZE_BYTES],
+                        data[NONCE_SIZE_BYTES:],
+                    )
 
                 try:
-                    decrypted_data = fernet.decrypt(data)
+                    decrypted_data = aesgcm.decrypt(nonce, ciphertext, None)
 
                     with open(decrypted_dir / f"{csv_file}", "wb") as file:
                         file.write(decrypted_data)
 
-                except InvalidToken as exc:
+                except InvalidTag as exc:
                     raise InvalidKeyError(
                         f"Not able to decrypt {encrypted_file}, "
                         f"is symkey correct?"
@@ -93,8 +106,8 @@ def decrypt(rsa_keys_dir: Path, dataset_dir: Path, output_dir: Path) -> None:
 
                 logger.debug(f"Decrypted {encrypted_file}")
 
-        del symkey
-        del fernet
+        del decrypted_symkey
+        del aesgcm
 
         # Merges the decrypted csv files into a single file
         _combine_csv_files(decrypted_dir, dataset_dir / f"{dataset_name}.csv")
@@ -140,14 +153,15 @@ def untar_encrypted_dataset(
 
 
 def _validate_tar_contents(files: List[str], dataset_name: str) -> None:
-    if f"{dataset_name}.json" not in files:
+    filenames = {os.path.basename(f) for f in files}
+    if f"{dataset_name}.json" not in filenames:
         raise InvalidTarFileContents(f"{dataset_name}.json not in .tar file")
 
     if len(files) > 1:
-        if f"{dataset_name}.symkey.encr" not in files:
+        if f"{dataset_name}.kem.encr" not in filenames:
             raise InvalidTarFileContents(
                 f"Tar file for {dataset_name} does not contain the required "
-                f"{dataset_name}.symkey.encr file"
+                f"{dataset_name}.kem.encr file"
             )
 
         chunk_files = [str for str in files if str.endswith(".csv.encr")]
@@ -157,10 +171,10 @@ def _validate_tar_contents(files: List[str], dataset_name: str) -> None:
                 f"Tar file for {dataset_name} does not contain any chunks files"
             )
 
-        if f"{dataset_name}.md5" not in files:
+        if f"{dataset_name}.sha256" not in filenames:
             raise InvalidTarFileContents(
                 f"Tar file for {dataset_name} does not contain the required "
-                f"{dataset_name}.md5 file"
+                f"{dataset_name}.sha256 file"
             )
 
 

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/microdata_tools/packaging/_encrypt.py

@@ -4,13 +4,21 @@ import shutil
 import tarfile
 from pathlib import Path
 
-from cryptography.fernet import Fernet
-from cryptography.hazmat.backends import default_backend
-from cryptography.hazmat.primitives import hashes, serialization
-from cryptography.hazmat.primitives.asymmetric import padding, rsa
-
-from microdata_tools.packaging._utils import check_exists
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+from cryptography.hazmat.primitives.hpke import (
+    MLKEM768X25519PublicKey,
+)
+
+from microdata_tools.packaging._crypto_config import (
+    HPKE_INFO,
+    HPKE_SUITE,
+    NONCE_SIZE_BYTES,
+)
+from microdata_tools.packaging._utils import (
+    check_exists,
+)
 from microdata_tools.packaging.exceptions import ValidationException
+from microdata_tools.packaging.keys import PublicKey
 
 logger = logging.getLogger()
 
@@ -18,7 +26,7 @@ CHUNK_SIZE_BYTES = 250_000_000  # 250 MB per chunk
 
 
 def encrypt_dataset(
-    rsa_keys_dir: Path,
+    public_key_dir: Path,
     dataset_dir: Path,
     output_dir: Path,
 ) -> None:
@@ -27,28 +35,29 @@ def encrypt_dataset(
         1. Generates the symmetric key for this dataset.
         2. Splits the dataset into chunks.
         3. Encrypts each chunk using the symmetric key.
-        4. Encrypts the symmetric key using the RSA public key.
+        4. Uses HPKE with a hybrid ML-KEM-768/X25519 public key to encrypt the
+           symmetric key and stores the resulting encapsulated ciphertext.
     """
 
-    check_exists(rsa_keys_dir)
+    check_exists(public_key_dir)
     check_exists(dataset_dir)
 
     if not output_dir.exists():
         os.makedirs(output_dir)
 
-    public_key_location = rsa_keys_dir / "microdata_public_key.pem"
+    public_key_location = public_key_dir / "microdata_public_key.pem"
     check_exists(public_key_location)
 
     # Read public key from file
-    with open(public_key_location, "rb") as key_file:
-        public_key = serialization.load_pem_public_key(
-            key_file.read(), backend=default_backend()
-        )
+    public_key = PublicKey.load_from_file(public_key_location).to_hpke_key()
 
+    if not isinstance(public_key, MLKEM768X25519PublicKey):
+        raise TypeError(
+            "Public key is not a hybrid ML-KEM-768-X25519 public key."
+        )
     csv_files = [
         file for file in dataset_dir.iterdir() if file.suffix == ".csv"
     ]
-
     csv_file = csv_files[0]
     dataset_name = csv_file.stem
 
@@ -62,11 +71,10 @@ def encrypt_dataset(
     os.makedirs(dataset_output_dir)
     os.makedirs(dataset_output_dir / "chunks", exist_ok=True)
 
-    encrypted_symkey_file = dataset_output_dir / f"{dataset_name}.symkey.encr"
+    hpke_ciphertext_file = dataset_output_dir / f"{dataset_name}.kem.encr"
 
-    # Generate and store symmetric key for this file
-    symkey = Fernet.generate_key()
-    fernet = Fernet(symkey)
+    symkey = AESGCM.generate_key(bit_length=256)
+    aesgcm = AESGCM(symkey)
 
     # Encrypt csv file
     chunk_count = 0
@@ -79,7 +87,9 @@ def encrypt_dataset(
                 break
 
             chunk_count += 1
-            encrypted = fernet.encrypt(data)
+            nonce = os.urandom(NONCE_SIZE_BYTES)
+            # The nonce is stored along with the ciphertext to decrypt later
+            encrypted = nonce + aesgcm.encrypt(nonce, data, None)
 
             chunk_file = (
                 dataset_output_dir / "chunks" / f"{chunk_count}.csv.encr"
@@ -89,27 +99,23 @@ def encrypt_dataset(
 
     logger.debug(f"Csv file {csv_file} encrypted into {chunk_count} chunks")
 
-    if not isinstance(public_key, rsa.RSAPublicKey):
-        raise TypeError("Public key is not RSA. Cannot use .encrypt().")
+    # Encrypt symmetric key using HPKE and store the ciphertext
+    try:
+        hpke_ciphertext = HPKE_SUITE.encrypt(symkey, public_key, info=HPKE_INFO)
+    except Exception as e:
+        raise ValidationException(
+            "Failed to encrypt the dataset key using HPKE. "
+            "Please check that the public key is valid and try again."
+        ) from e
 
-    encrypted_sym_key = public_key.encrypt(
-        symkey,
-        padding.OAEP(
-            mgf=padding.MGF1(algorithm=hashes.SHA256()),
-            algorithm=hashes.SHA256(),
-            label=None,
-        ),
-    )
+    with open(hpke_ciphertext_file, "wb") as file:
+        file.write(hpke_ciphertext)
 
     del symkey
-    del fernet
-
-    # Store encrypted symkey to file
-    with open(encrypted_symkey_file, "wb") as file:
-        file.write(encrypted_sym_key)
+    del aesgcm
 
     logger.debug(
-        f"Key file for {csv_file} encrypted into {encrypted_symkey_file}"
+        f"HPKE-encrypted key ciphertext stored in {hpke_ciphertext_file}"
     )
 
 
@@ -143,16 +149,16 @@ def tar_encrypted_dataset(input_dir: Path, dataset_name: str) -> None:
         if len(chunk_files) == 0:
             raise ValidationException(f"No files found in {chunk_dir}")
 
-        md5_file = dataset_dir / f"{dataset_name}.md5"
-        if not md5_file.exists():
+        sha256_file = dataset_dir / f"{dataset_name}.sha256"
+        if not sha256_file.exists():
             raise ValidationException(
-                f"The required file {md5_file} is missing"
+                f"The required file {sha256_file} is missing"
             )
 
         files_to_tar.extend(
             [
-                dataset_dir / f"{dataset_name}.symkey.encr",
-                dataset_dir / f"{dataset_name}.md5",
+                dataset_dir / f"{dataset_name}.kem.encr",
+                dataset_dir / f"{dataset_name}.sha256",
             ]
         )
 

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/microdata_tools/packaging/_utils.py

@@ -14,29 +14,29 @@ def check_exists(path: Path) -> None:
 
 def calculate_checksum(csv_file: Path) -> str:
     """
-    Reads a file in chunks and returns the MD5 hash of the file
+    Reads a file in chunks and returns the SHA-256 hash of the file
     """
     CHUNK_SIZE = 32768
-    hash_md5 = hashlib.md5()
+    hash_sha256 = hashlib.sha256()
     with open(csv_file, "rb") as f:
         for chunk in iter(lambda: f.read(CHUNK_SIZE), b""):
-            hash_md5.update(chunk)
+            hash_sha256.update(chunk)
 
-    return hash_md5.hexdigest()
+    return hash_sha256.hexdigest()
 
 
 def write_checksum_to_file(csv_file: Path) -> None:
-    hash_file = str(csv_file).replace(".csv", ".md5")
+    hash_file = str(csv_file).replace(".csv", ".sha256")
     with open(hash_file, "w") as file:
         file.write(calculate_checksum(csv_file))
 
 
 def compare_checksum_with_file(
-    md5_file: Path, calculated_checksum: str
+    sha256_file: Path, calculated_checksum: str
 ) -> None:
-    with open(md5_file, "r") as file:
+    with open(sha256_file, "r") as file:
         checksum_from_file = file.readlines()[0]
         if calculated_checksum != checksum_from_file.strip():
             raise CsvConsistencyException(
-                "MD5 checksums do not match. The csv file may be corrupted!"
+                "SHA-256 checksums do not match. The csv file may be corrupted!"
             )

microdata_tools-2.0.0/microdata_tools/packaging/keys.py

@@ -0,0 +1,192 @@
+import re
+from pathlib import Path
+
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import mlkem, x25519
+from cryptography.hazmat.primitives.hpke import (
+    MLKEM768X25519PrivateKey,
+    MLKEM768X25519PublicKey,
+)
+
+PEM_PRIVATE_BLOCK = re.compile(
+    rb"-----BEGIN PRIVATE KEY-----.*?-----END PRIVATE KEY-----",
+    re.DOTALL,
+)
+
+PEM_PUBLIC_BLOCK = re.compile(
+    rb"-----BEGIN PUBLIC KEY-----.*?-----END PUBLIC KEY-----",
+    re.DOTALL,
+)
+
+
+class PrivateKey:
+    """Hybrid ML-KEM-768 + X25519 private key for use with HPKE."""
+
+    FILENAME = "microdata_private_key.pem"
+
+    def __init__(
+        self,
+        mlkem_key: mlkem.MLKEM768PrivateKey,
+        x25519_key: x25519.X25519PrivateKey,
+    ) -> None:
+        self.mlkem_key = mlkem_key
+        self.x25519_key = x25519_key
+
+    @classmethod
+    def generate(cls) -> "PrivateKey":
+        """Generates a new hybrid private key."""
+        return cls(
+            mlkem_key=mlkem.MLKEM768PrivateKey.generate(),
+            x25519_key=x25519.X25519PrivateKey.generate(),
+        )
+
+    @classmethod
+    def load_from_file(cls, path: Path) -> "PrivateKey":
+        """Loads a hybrid private key from a PEM file containing one
+        ML-KEM-768 and one X25519 PRIVATE KEY block, in any order."""
+        blocks = PEM_PRIVATE_BLOCK.findall(path.read_bytes())
+        if len(blocks) != 2:
+            raise ValueError(
+                f"Expected 2 PEM blocks in hybrid private key file, "
+                f"found {len(blocks)}"
+            )
+
+        parsed_keys = [
+            serialization.load_pem_private_key(block, password=None)
+            for block in blocks
+        ]
+
+        mlkem_key = next(
+            (
+                key
+                for key in parsed_keys
+                if isinstance(key, mlkem.MLKEM768PrivateKey)
+            ),
+            None,
+        )
+        x25519_key = next(
+            (
+                key
+                for key in parsed_keys
+                if isinstance(key, x25519.X25519PrivateKey)
+            ),
+            None,
+        )
+
+        if mlkem_key is None or x25519_key is None:
+            raise ValueError(
+                "Hybrid private key file must contain one ML-KEM-768 key "
+                "and one X25519 key."
+            )
+
+        return cls(mlkem_key=mlkem_key, x25519_key=x25519_key)
+
+    def public_key(self) -> "PublicKey":
+        """Returns the matching hybrid PublicKey."""
+        return PublicKey(
+            mlkem_key=self.mlkem_key.public_key(),
+            x25519_key=self.x25519_key.public_key(),
+        )
+
+    def serialize(self) -> bytes:
+        """Serializes both halves as concatenated PKCS8 PEM blocks."""
+        return self.mlkem_key.private_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PrivateFormat.PKCS8,
+            encryption_algorithm=serialization.NoEncryption(),
+        ) + self.x25519_key.private_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PrivateFormat.PKCS8,
+            encryption_algorithm=serialization.NoEncryption(),
+        )
+
+    def write_to_file(self, target_dir: Path) -> None:
+        """Writes the serialized key to
+        ``<target_dir>/<PrivateKey.FILENAME>``."""
+        (target_dir / self.FILENAME).write_bytes(self.serialize())
+
+    def to_hpke_key(self) -> MLKEM768X25519PrivateKey:
+        """Returns the combined HPKE-usable key object."""
+        return MLKEM768X25519PrivateKey(
+            mlkem_key=self.mlkem_key,
+            x25519_key=self.x25519_key,
+        )
+
+
+class PublicKey:
+    """Hybrid ML-KEM-768 + X25519 public key for use with HPKE."""
+
+    FILENAME = "microdata_public_key.pem"
+
+    def __init__(
+        self,
+        mlkem_key: mlkem.MLKEM768PublicKey,
+        x25519_key: x25519.X25519PublicKey,
+    ) -> None:
+        self.mlkem_key = mlkem_key
+        self.x25519_key = x25519_key
+
+    @classmethod
+    def load_from_file(cls, path: Path) -> "PublicKey":
+        """Loads a hybrid public key from a PEM file containing one
+        ML-KEM-768 and one X25519 PUBLIC KEY block, in any order."""
+        return cls.from_pem(path.read_bytes())
+
+    @classmethod
+    def from_pem(cls, pem_data: bytes) -> "PublicKey":
+        """Loads a hybrid public key from PEM data containing one
+        ML-KEM-768 and one X25519 PUBLIC KEY block, in any order."""
+        blocks = PEM_PUBLIC_BLOCK.findall(pem_data)
+        if len(blocks) != 2:
+            raise ValueError(
+                "Expected 2 PEM blocks in hybrid public key file, "
+                f"found {len(blocks)}"
+            )
+
+        parsed_keys = [serialization.load_pem_public_key(b) for b in blocks]
+        mlkem_key = next(
+            (
+                key
+                for key in parsed_keys
+                if isinstance(key, mlkem.MLKEM768PublicKey)
+            ),
+            None,
+        )
+        x25519_key = next(
+            (
+                key
+                for key in parsed_keys
+                if isinstance(key, x25519.X25519PublicKey)
+            ),
+            None,
+        )
+
+        if mlkem_key is None or x25519_key is None:
+            raise ValueError(
+                "Hybrid public key file must contain one ML-KEM-768 key "
+                "and one X25519 key."
+            )
+
+        return cls(mlkem_key=mlkem_key, x25519_key=x25519_key)
+
+    def serialize(self) -> bytes:
+        """Serializes both keys as concatenated PEM blocks."""
+        return self.mlkem_key.public_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PublicFormat.SubjectPublicKeyInfo,
+        ) + self.x25519_key.public_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PublicFormat.SubjectPublicKeyInfo,
+        )
+
+    def write_to_file(self, target_dir: Path) -> None:
+        """Writes the serialized keys to
+        ``<target_dir>/<PublicKey.FILENAME>``."""
+        (target_dir / self.FILENAME).write_bytes(self.serialize())
+
+    def to_hpke_key(self) -> MLKEM768X25519PublicKey:
+        """Returns the combined HPKE-usable key object."""
+        return MLKEM768X25519PublicKey(
+            mlkem_key=self.mlkem_key,
+            x25519_key=self.x25519_key,
+        )

{microdata_tools-1.14.0 → microdata_tools-2.0.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "microdata-tools"
-version = "1.14.0"
+version = "2.0.0"
 description = "Tools for the microdata.no platform"
 authors = [{ name = "microdata-developers" }]
 license = { text = "MIT" }