microdata-tools 2.2.3__py3-none-any.whl → 2.2.5__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -149,9 +149,23 @@ def untar_encrypted_dataset(
149
149
  ) -> None:
150
150
  with tarfile.open(input_file) as tar:
151
151
  _validate_tar_contents(tar.getnames(), dataset_name)
152
+ _check_no_path_traversal(tar.getmembers())
152
153
  tar.extractall(path=untar_dir)
153
154
 
154
155
 
156
+ def _check_no_path_traversal(members: List[tarfile.TarInfo]) -> None:
157
+ for member in members:
158
+ member_path = Path(member.name)
159
+ if member_path.is_absolute() or ".." in member_path.parts:
160
+ raise InvalidTarFileContents(
161
+ f"Tar member '{member.name}' contains a path traversal sequence"
162
+ )
163
+ if not (member.isfile() or member.isdir()):
164
+ raise InvalidTarFileContents(
165
+ f"Tar member '{member.name}' is not a regular file or directory"
166
+ )
167
+
168
+
155
169
  def _validate_tar_contents(files: List[str], dataset_name: str) -> None:
156
170
  filenames = {os.path.basename(f) for f in files}
157
171
  if f"{dataset_name}.json" not in filenames:
@@ -0,0 +1,105 @@
1
+ Metadata-Version: 2.5
2
+ Name: microdata-tools
3
+ Version: 2.2.5
4
+ Summary: Tools for the microdata.no platform
5
+ Author: microdata-developers
6
+ License: MIT
7
+ License-File: LICENSE.md
8
+ Requires-Python: >=3.10
9
+ Requires-Dist: adbc-driver-manager==1.12.0
10
+ Requires-Dist: adbc-driver-sqlite==1.12.0
11
+ Requires-Dist: cryptography>=50.0.1
12
+ Requires-Dist: numpy>=2.2.0
13
+ Requires-Dist: pyarrow==25.0.1
14
+ Requires-Dist: pydantic>=2.13.5
15
+ Description-Content-Type: text/markdown
16
+
17
+ # microdata-tools
18
+ Tools for the [microdata.no](https://www.microdata.no/) platform
19
+
20
+ `microdata-tools` is a Python library with two audiences:
21
+ - **Data owners** preparing datasets for microdata.no: validate your data and metadata, then encrypt and package the files into a `.tar` archive ready for upload.
22
+ - **microdata.no services** (such as job-executor) that receive packaged datasets: unpackage and decrypt them, and validate them on the receiving end.
23
+
24
+ A dataset normally contains:
25
+ - A CSV file with one identifier, one value and the dates that describe when the value applies.
26
+ - A JSON file describing the dataset, its variable and its values.
27
+
28
+ ## Installation
29
+ `microdata-tools` can be installed from PyPI using pip:
30
+ ```
31
+ pip install microdata-tools
32
+ ```
33
+
34
+ ## Documentation
35
+
36
+ If you're a data owner preparing a dataset, the full guide lives on the documentation site:
37
+
38
+ - **[Getting started](https://statisticsnorway.github.io/microdata-tools/)** — install, prepare, validate and package your dataset, step by step.
39
+ - **[The metadata model](https://statisticsnorway.github.io/microdata-tools/metadata-model/)** — a field-by-field reference for the metadata JSON file.
40
+ - **[The data file format](https://statisticsnorway.github.io/microdata-tools/data-file-format/)** — a reference for the CSV data file, including validation rules.
41
+ - **[The packaging format](https://statisticsnorway.github.io/microdata-tools/packaging-format/)** — how your data is encrypted and packaged.
42
+
43
+ The sections below are aimed at developers integrating `microdata-tools` into another service (such as job-executor), and at contributors to this repository.
44
+
45
+ ## Unpackaging a dataset
46
+
47
+ The `unpackage_dataset()` function will reverse the packaging step: it untars the archive and use the combined ML-KEM-768/X25519 private key from `microdata_private_key.pem` to recover the symmetric key, which is then used to decrypt the data file.
48
+
49
+ The packaged file must have the `<DATASET_NAME>.tar` extension. Its contents should be:
50
+
51
+ - `<DATASET_NAME>.json`: Required metadata file.
52
+ - `<DATASET_NAME>.csv.encr`: Optional encrypted data file.
53
+ - `<DATASET_NAME>.kem.encr`: Optional HPKE ciphertext file containing the encrypted symmetric key. Required if `.csv.encr` is present.
54
+
55
+ Decryption uses the combined ML-KEM-768/X25519 private key located at `PRIVATE_KEY_DIR` to recover the symmetric decryption key. The dataset is then stored in `output_dir/archive/unpackaged` after a successful run, or `output_dir/archive/failed` after an unsuccessful one.
56
+ For details on how packaging encrypts a dataset in the first place, see [the packaging format](https://statisticsnorway.github.io/microdata-tools/packaging-format/).
57
+
58
+
59
+ ## Contribute
60
+
61
+ ### Set up
62
+ To work on this repository you need to install [uv](https://docs.astral.sh/uv/):
63
+ ```
64
+ # macOS / linux / BashOnWindows
65
+ curl -LsSf https://astral.sh/uv/install.sh | sh
66
+
67
+ # Windows powershell
68
+ powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
69
+ ```
70
+ Then install the virtual environment from the root directory:
71
+ ```
72
+ uv sync
73
+ ```
74
+
75
+ ### Running unit tests
76
+ Open terminal and go to root directory of the project and run:
77
+ ````
78
+ uv run pytest
79
+ ````
80
+
81
+ ### Pre-commit
82
+ There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file.
83
+ Install pre-commit
84
+ ```sh
85
+ pip install pre-commit
86
+ ```
87
+ If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:
88
+ ```sh
89
+ pre-commit install
90
+ ```
91
+ Now it should run when you do:
92
+ ```sh
93
+ git commit
94
+ ```
95
+
96
+ By default it only runs against changed files. To force the hooks to run against all files:
97
+ ```sh
98
+ pre-commit run --all-files
99
+ ```
100
+ if you dont have it installed on your system you can use:
101
+ (but then it won't run when you use the git-cli)
102
+ ```sh
103
+ uv run pre-commit
104
+ ```
105
+ Read more about [pre-commit](https://pre-commit.com/#intro)
@@ -1,7 +1,7 @@
1
1
  microdata_tools/__init__.py,sha256=lW8W1kL7ePkQlhXzhGhgPoqlY6pOdyuONeSU_6ghcY0,460
2
2
  microdata_tools/packaging/__init__.py,sha256=p4UgB4Q_pLi0zXlmHuSFgrwU5JueRnGSwT-ffRNIv5M,4486
3
3
  microdata_tools/packaging/_crypto_config.py,sha256=Mjf5BmOmbDy1QbMpMQl0N16jpSBTMLQ7oQ5nCRdD3yY,245
4
- microdata_tools/packaging/_decrypt.py,sha256=eA-d6IdaJ02X_8n-hIKmhDuGxikCTRO1m7R6C93u0UA,6821
4
+ microdata_tools/packaging/_decrypt.py,sha256=0dCTXf1g48q_s0YEz1IJ84j1kt-4k_UvRe2-D-LTGAM,7404
5
5
  microdata_tools/packaging/_encrypt.py,sha256=mMamdx8rbFJiyzeKTSAWll8RPoiNiTZ8qkATwPPSrmk,5416
6
6
  microdata_tools/packaging/_utils.py,sha256=LO6qsme90lrLAgQu1jeM5S33dLC5zp_4-J56Drn2T1g,1226
7
7
  microdata_tools/packaging/keys.py,sha256=77xlcSOKMVWDYf5YmlZpDufqlKfnkoSkjv_jFMFy3Q8,6274
@@ -42,7 +42,7 @@ microdata_tools/validation/steps/data_reader.py,sha256=vJ_RzSHFsNDnMLIDFcG2fg4Z7
42
42
  microdata_tools/validation/steps/dataset_validator.py,sha256=4z1H4O3NreqVegxjoloJwJIMaIHpra26iIiEXO_EaY8,13186
43
43
  microdata_tools/validation/steps/metadata_enricher.py,sha256=TiyEjO6k5vEDNvUA9kAcTmmcu8L6JSu6sLBqK_q64pU,1055
44
44
  microdata_tools/validation/steps/metadata_reader.py,sha256=erB46QQ2L3fmHvK3efzor_XfpaK8r0OTvrSaSlXOaU4,7671
45
- microdata_tools-2.2.3.dist-info/METADATA,sha256=O-j2wIkny_-sMckX_OTretoBE_GNGyq37TM8-W1h1BI,6191
46
- microdata_tools-2.2.3.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
47
- microdata_tools-2.2.3.dist-info/licenses/LICENSE.md,sha256=hxspefYgWP3U6OZFhCifqWMI5ksnKzgFxNKgQnG7Ozc,1074
48
- microdata_tools-2.2.3.dist-info/RECORD,,
45
+ microdata_tools-2.2.5.dist-info/METADATA,sha256=XCZsE8PX3BB2kRJBe-PWWGByZKnFakD7ObIOmbAtd08,4282
46
+ microdata_tools-2.2.5.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
47
+ microdata_tools-2.2.5.dist-info/licenses/LICENSE.md,sha256=hxspefYgWP3U6OZFhCifqWMI5ksnKzgFxNKgQnG7Ozc,1074
48
+ microdata_tools-2.2.5.dist-info/RECORD,,
@@ -1,4 +1,4 @@
1
1
  Wheel-Version: 1.0
2
- Generator: hatchling 1.32.4
2
+ Generator: hatchling 1.33.0
3
3
  Root-Is-Purelib: true
4
4
  Tag: py3-none-any
@@ -1,182 +0,0 @@
1
- Metadata-Version: 2.5
2
- Name: microdata-tools
3
- Version: 2.2.3
4
- Summary: Tools for the microdata.no platform
5
- Author: microdata-developers
6
- License: MIT
7
- License-File: LICENSE.md
8
- Requires-Python: >=3.10
9
- Requires-Dist: adbc-driver-manager==1.12.0
10
- Requires-Dist: adbc-driver-sqlite==1.12.0
11
- Requires-Dist: cryptography>=50.0.1
12
- Requires-Dist: numpy>=2.2.0
13
- Requires-Dist: pyarrow==25.0.1
14
- Requires-Dist: pydantic>=2.13.5
15
- Description-Content-Type: text/markdown
16
-
17
- # microdata-tools
18
- Tools for the [microdata.no](https://www.microdata.no/) platform
19
-
20
- ## Installation
21
- `microdata-tools` can be installed from PyPI using pip:
22
- ```
23
- pip install microdata-tools
24
- ```
25
-
26
- ## Usage
27
- Once you have your metadata and data files ready to go, they should be named and stored like this:
28
- ```
29
- my-input-directory/
30
- MY_DATASET_NAME/
31
- MY_DATASET_NAME.csv
32
- MY_DATASET_NAME.json
33
- ```
34
- The CSV file is optional in some cases.
35
-
36
- ### Package dataset
37
- The `package_dataset()` function will encrypt and package your dataset as a tar archive. The process is as follows:
38
-
39
- 1. Generate the symmetric key for a dataset.
40
- 2. Encrypt the dataset data (CSV) using an AES-256-GCM symmetric key and store the encrypted file as `<DATASET_NAME>.csv.encr`
41
- 3. Encrypt the symmetric key using HPKE with the combined ML-KEM-768/X25519 public key from `microdata_public_key.pem` and store the resulting HPKE ciphertext as <DATASET_NAME>.kem.encr
42
- 4. Gather the encrypted CSV, ciphertext file and metadata (JSON) file in one tar file.
43
-
44
- ### Unpackage dataset
45
- The `unpackage_dataset()` function will untar and your dataset and use the combined ML-KEM-768/X25519 private key from `microdata_private_key.pem` to recover the symmetric key, which is then used to decrypt the dataset.
46
-
47
-
48
- The packaged file has to have the `<DATASET_NAME>.tar` extension. Its contents should be as follows:
49
-
50
- ```<DATASET_NAME>.json``` : Required medata file.
51
-
52
- ```<DATASET_NAME>.csv.encr``` : Optional encrypted dataset file.
53
-
54
- ```<DATASET_NAME>.kem.encr``` : Optional HPKE ciphertext file containing the encrypted symmetric key required to decrypt the dataset file. Required if the `.csv.encr` file is present.
55
-
56
- Decryption uses the combined ML-KEM-768/X25519 private key located at ```PRIVATE_KEY_DIR``` to recover the symmetric decryption key.
57
-
58
- The packaged file is then stored in `output_dir/archive/unpackaged` after a successful run or `output_dir/archive/failed` after an unsuccessful run.
59
-
60
-
61
- ## Example
62
-
63
- Store your metadata and data files according to the structure described above, and put
64
- the provided public key in a directory of your choice.
65
- Then:
66
-
67
- ```py
68
- from pathlib import Path
69
- from microdata_tools import package_dataset
70
-
71
- package_dataset(
72
- public_key_dir=Path("path/to/key_directory"),
73
- dataset_dir=Path("path/to/MY_DATASET_NAME"),
74
- output_dir=Path("path/to/output"),
75
- )
76
- ```
77
-
78
- This produces `path/to/output/MY_DATASET_NAME.tar`, which can be uploaded to microdata.
79
-
80
-
81
- ### Validation
82
-
83
- Once you have your metadata and data files ready to go, they should be named and stored like this:
84
- ```
85
- my-input-directory/
86
- MY_DATASET_NAME/
87
- MY_DATASET_NAME.csv
88
- MY_DATASET_NAME.json
89
- ```
90
- Note that the filename only allows upper case letters A-Z, number 0-9 and underscores.
91
-
92
-
93
- Import microdata-tools in your script and validate your files:
94
- ```py
95
- from microdata_tools import validate_dataset
96
-
97
- validation_errors = validate_dataset(
98
- "MY_DATASET_NAME", input_directory="path/to/my-input-directory"
99
- )
100
-
101
- if not validation_errors:
102
- print("My dataset is valid")
103
- else:
104
- print("Dataset is invalid :(")
105
- # You can print your errors like this:
106
- for error in validation_errors:
107
- print(error)
108
- ```
109
-
110
- For a more in-depth explanation of usage visit [the usage documentation](https://statisticsnorway.github.io/microdata-tools/USAGE).
111
-
112
- ### Data format description
113
- A dataset as defined in microdata consists of one data file, and one metadata file.
114
-
115
- The data file is a csv file seperated by semicolons. A valid example would be:
116
- ```csv
117
- 000000000000001;123;2020-01-01;2020-12-31;
118
- 000000000000002;123;2020-01-01;2020-12-31;
119
- 000000000000003;123;2020-01-01;2020-12-31;
120
- 000000000000004;123;2020-01-01;2020-12-31;
121
- ```
122
-
123
- For datasets with temporalityType FIXED the value does not change over time, so there is no time period to describe — both the start and stop columns should be left empty. On microdata.no such variables are shown with an infinite validity period ("∞"), see [BEFOLKNING_KJOENN](https://microdata.no/discovery/variable/no.ssb.fdb/56/BEFOLKNING_KJOENN) for an example.
124
- ```csv
125
- 000000000000001;123;;;
126
- 000000000000002;123;;;
127
- 000000000000003;123;;;
128
- 000000000000004;123;;;
129
- ```
130
-
131
- Read more about the data format and columns in [the documentation](https://statisticsnorway.github.io/microdata-tools/).
132
-
133
- The metadata files should be in json format. The requirements for the metadata is best described through the [Pydantic model](/microdata_tools/validation/model/metadata.py), [the examples](/docs/examples), and [the metadata model](https://statisticsnorway.github.io/microdata-tools/metadata-model/).
134
-
135
-
136
- ## Contribute
137
-
138
- ### Set up
139
- To work on this repository you need to install [uv](https://docs.astral.sh/uv/):
140
- ```
141
- # macOS / linux / BashOnWindows
142
- curl -LsSf https://astral.sh/uv/install.sh | sh
143
-
144
- # Windows powershell
145
- powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
146
- ```
147
- Then install the virtual environment from the root directory:
148
- ```
149
- uv sync
150
- ```
151
-
152
- ### Running unit tests
153
- Open terminal and go to root directory of the project and run:
154
- ````
155
- uv run pytest
156
- ````
157
-
158
- ### Pre-commit
159
- There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file.
160
- Install pre-commit
161
- ```sh
162
- pip install pre-commit
163
- ```
164
- If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:
165
- ```sh
166
- pre-commit install
167
- ```
168
- Now it should run when you do:
169
- ```sh
170
- git commit
171
- ```
172
-
173
- By default it only runs against changed files. To force the hooks to run against all files:
174
- ```sh
175
- pre-commit run --all-files
176
- ```
177
- if you dont have it installed on your system you can use:
178
- (but then it won't run when you use the git-cli)
179
- ```sh
180
- uv run pre-commit
181
- ```
182
- Read more about [pre-commit](https://pre-commit.com/#intro)