refscan 0.1.4__tar.gz → 0.1.6__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {refscan-0.1.4 → refscan-0.1.6}/PKG-INFO +56 -14
- {refscan-0.1.4 → refscan-0.1.6}/README.md +55 -13
- {refscan-0.1.4 → refscan-0.1.6}/pyproject.toml +8 -1
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/Finder.py +1 -3
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/Reference.py +1 -0
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/ReferenceList.py +21 -15
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/Violation.py +1 -0
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/helpers.py +3 -12
- {refscan-0.1.4 → refscan-0.1.6}/refscan/refscan.py +80 -47
- {refscan-0.1.4 → refscan-0.1.6}/refscan/__init__.py +0 -0
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/ViolationList.py +0 -0
- {refscan-0.1.4 → refscan-0.1.6}/refscan/lib/constants.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: refscan
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.6
|
|
4
4
|
Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
|
|
5
5
|
Home-page: https://github.com/microbiomedata/refscan
|
|
6
6
|
Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
|
|
@@ -45,18 +45,18 @@ graph LR
|
|
|
45
45
|
`refscan` does its job in two stages:
|
|
46
46
|
1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
|
|
47
47
|
that _can_ exist in a database that conforms to the schema.
|
|
48
|
-
> **Example:** The schema might say that,
|
|
49
|
-
>
|
|
48
|
+
> **Example:** The schema might say that, for each document in the `biosample_set` collection that has a field named
|
|
49
|
+
> `associated_studies`, that field must contain a list of `id`s of documents in the `study_set` collection.
|
|
50
50
|
2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
|
|
51
|
-
> **Example:** For each document in the
|
|
52
|
-
> field
|
|
51
|
+
> **Example:** For each document in the `biosample_set` collection that _has_ a field named `associated_studies`,
|
|
52
|
+
> for each value in that field, confirm there _is_ a document having that `id` in the `study_set` collection.
|
|
53
53
|
|
|
54
54
|
## Limitations
|
|
55
55
|
|
|
56
|
-
`refscan` was designed under the assumption that **
|
|
57
|
-
whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class
|
|
58
|
-
`refscan`
|
|
59
|
-
|
|
56
|
+
`refscan` was designed under the assumption that **every document** in **every collection described by the schema** has
|
|
57
|
+
a **field named `type`**, whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class the document represents an instance
|
|
58
|
+
of. `refscan` uses that `class_uri` value (in that `type` field) to determine the name of that schema class,
|
|
59
|
+
which it, in turn, uses to determine _which fields_ of that document can contain references.
|
|
60
60
|
|
|
61
61
|
## Usage
|
|
62
62
|
|
|
@@ -167,6 +167,15 @@ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-sc
|
|
|
167
167
|
---
|
|
168
168
|
</details>
|
|
169
169
|
|
|
170
|
+
#### Output
|
|
171
|
+
|
|
172
|
+
While `refscan` is running, it will display console output indicating what it's currently doing.
|
|
173
|
+
|
|
174
|
+

|
|
175
|
+
|
|
176
|
+
Once the scan is complete, the reference report (TSV file) and violation report (TSV file) will be available
|
|
177
|
+
in the current directory (or in custom directories, if any were specified via CLI options).
|
|
178
|
+
|
|
170
179
|
### Updating
|
|
171
180
|
|
|
172
181
|
You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
|
|
@@ -185,10 +194,10 @@ pipx uninstall refscan
|
|
|
185
194
|
|
|
186
195
|
## Development
|
|
187
196
|
|
|
188
|
-
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b)
|
|
197
|
+
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) build distributable packages that can be published to PyPI.
|
|
189
198
|
|
|
190
|
-
- `pyproject.toml`: Configuration file for Poetry and other tools (was
|
|
191
|
-
- `poetry.lock`: List of dependencies, direct and indirect
|
|
199
|
+
- `pyproject.toml`: Configuration file for Poetry and other tools (was initialized via `$ poetry init`)
|
|
200
|
+
- `poetry.lock`: List of dependencies, both direct and [indirect/transitive](https://en.wikipedia.org/wiki/Transitive_dependency)
|
|
192
201
|
|
|
193
202
|
### Clone repository
|
|
194
203
|
|
|
@@ -229,6 +238,41 @@ While editing the tool's source code, you can run the tool as you normally would
|
|
|
229
238
|
refscan --help
|
|
230
239
|
```
|
|
231
240
|
|
|
241
|
+
### Run tests
|
|
242
|
+
|
|
243
|
+
We use [pytest](https://docs.pytest.org/en/8.2.x/) as the testing framework for `refscan`.
|
|
244
|
+
|
|
245
|
+
You can run the tests by running the following command from the root directory of the repository:
|
|
246
|
+
|
|
247
|
+
```shell
|
|
248
|
+
poetry run pytest
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
Tests are defined in the `tests` directory.
|
|
252
|
+
|
|
253
|
+
### Format code
|
|
254
|
+
|
|
255
|
+
We use [`black`](https://black.readthedocs.io/en/stable/) as the code formatter for `refscan`.
|
|
256
|
+
|
|
257
|
+
We do not use it with its default options. Instead, we include an option that allows lines to be 120 characters
|
|
258
|
+
instead of the default 88 characters.
|
|
259
|
+
|
|
260
|
+
You can format all the Python code in the repository by running this command
|
|
261
|
+
from the root directory of the repository:
|
|
262
|
+
|
|
263
|
+
```shell
|
|
264
|
+
poetry run black --line-length 120 .
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
#### Check format
|
|
268
|
+
|
|
269
|
+
You can _check_ the format of the Python code by including the `--check` option, like this:
|
|
270
|
+
|
|
271
|
+
```shell
|
|
272
|
+
poetry run black --line-length 120 --check .
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
|
|
232
276
|
## Building and publishing
|
|
233
277
|
|
|
234
278
|
### Build for production
|
|
@@ -238,8 +282,6 @@ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.
|
|
|
238
282
|
will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
|
|
239
283
|
That package will have a version identifier that matches the name of the Git tag associated with the Release.
|
|
240
284
|
|
|
241
|
-
The GitHub Actions workflow will automatically
|
|
242
|
-
|
|
243
285
|
### Test the build process locally
|
|
244
286
|
|
|
245
287
|
In case you want to test the build process locally, you can do so by running:
|
|
@@ -26,18 +26,18 @@ graph LR
|
|
|
26
26
|
`refscan` does its job in two stages:
|
|
27
27
|
1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
|
|
28
28
|
that _can_ exist in a database that conforms to the schema.
|
|
29
|
-
> **Example:** The schema might say that,
|
|
30
|
-
>
|
|
29
|
+
> **Example:** The schema might say that, for each document in the `biosample_set` collection that has a field named
|
|
30
|
+
> `associated_studies`, that field must contain a list of `id`s of documents in the `study_set` collection.
|
|
31
31
|
2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
|
|
32
|
-
> **Example:** For each document in the
|
|
33
|
-
> field
|
|
32
|
+
> **Example:** For each document in the `biosample_set` collection that _has_ a field named `associated_studies`,
|
|
33
|
+
> for each value in that field, confirm there _is_ a document having that `id` in the `study_set` collection.
|
|
34
34
|
|
|
35
35
|
## Limitations
|
|
36
36
|
|
|
37
|
-
`refscan` was designed under the assumption that **
|
|
38
|
-
whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class
|
|
39
|
-
`refscan`
|
|
40
|
-
|
|
37
|
+
`refscan` was designed under the assumption that **every document** in **every collection described by the schema** has
|
|
38
|
+
a **field named `type`**, whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class the document represents an instance
|
|
39
|
+
of. `refscan` uses that `class_uri` value (in that `type` field) to determine the name of that schema class,
|
|
40
|
+
which it, in turn, uses to determine _which fields_ of that document can contain references.
|
|
41
41
|
|
|
42
42
|
## Usage
|
|
43
43
|
|
|
@@ -148,6 +148,15 @@ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-sc
|
|
|
148
148
|
---
|
|
149
149
|
</details>
|
|
150
150
|
|
|
151
|
+
#### Output
|
|
152
|
+
|
|
153
|
+
While `refscan` is running, it will display console output indicating what it's currently doing.
|
|
154
|
+
|
|
155
|
+

|
|
156
|
+
|
|
157
|
+
Once the scan is complete, the reference report (TSV file) and violation report (TSV file) will be available
|
|
158
|
+
in the current directory (or in custom directories, if any were specified via CLI options).
|
|
159
|
+
|
|
151
160
|
### Updating
|
|
152
161
|
|
|
153
162
|
You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
|
|
@@ -166,10 +175,10 @@ pipx uninstall refscan
|
|
|
166
175
|
|
|
167
176
|
## Development
|
|
168
177
|
|
|
169
|
-
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b)
|
|
178
|
+
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) build distributable packages that can be published to PyPI.
|
|
170
179
|
|
|
171
|
-
- `pyproject.toml`: Configuration file for Poetry and other tools (was
|
|
172
|
-
- `poetry.lock`: List of dependencies, direct and indirect
|
|
180
|
+
- `pyproject.toml`: Configuration file for Poetry and other tools (was initialized via `$ poetry init`)
|
|
181
|
+
- `poetry.lock`: List of dependencies, both direct and [indirect/transitive](https://en.wikipedia.org/wiki/Transitive_dependency)
|
|
173
182
|
|
|
174
183
|
### Clone repository
|
|
175
184
|
|
|
@@ -210,6 +219,41 @@ While editing the tool's source code, you can run the tool as you normally would
|
|
|
210
219
|
refscan --help
|
|
211
220
|
```
|
|
212
221
|
|
|
222
|
+
### Run tests
|
|
223
|
+
|
|
224
|
+
We use [pytest](https://docs.pytest.org/en/8.2.x/) as the testing framework for `refscan`.
|
|
225
|
+
|
|
226
|
+
You can run the tests by running the following command from the root directory of the repository:
|
|
227
|
+
|
|
228
|
+
```shell
|
|
229
|
+
poetry run pytest
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Tests are defined in the `tests` directory.
|
|
233
|
+
|
|
234
|
+
### Format code
|
|
235
|
+
|
|
236
|
+
We use [`black`](https://black.readthedocs.io/en/stable/) as the code formatter for `refscan`.
|
|
237
|
+
|
|
238
|
+
We do not use it with its default options. Instead, we include an option that allows lines to be 120 characters
|
|
239
|
+
instead of the default 88 characters.
|
|
240
|
+
|
|
241
|
+
You can format all the Python code in the repository by running this command
|
|
242
|
+
from the root directory of the repository:
|
|
243
|
+
|
|
244
|
+
```shell
|
|
245
|
+
poetry run black --line-length 120 .
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
#### Check format
|
|
249
|
+
|
|
250
|
+
You can _check_ the format of the Python code by including the `--check` option, like this:
|
|
251
|
+
|
|
252
|
+
```shell
|
|
253
|
+
poetry run black --line-length 120 --check .
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
|
|
213
257
|
## Building and publishing
|
|
214
258
|
|
|
215
259
|
### Build for production
|
|
@@ -219,8 +263,6 @@ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.
|
|
|
219
263
|
will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
|
|
220
264
|
That package will have a version identifier that matches the name of the Git tag associated with the Release.
|
|
221
265
|
|
|
222
|
-
The GitHub Actions workflow will automatically
|
|
223
|
-
|
|
224
266
|
### Test the build process locally
|
|
225
267
|
|
|
226
268
|
In case you want to test the build process locally, you can do so by running:
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
[tool.poetry]
|
|
4
4
|
name = "refscan"
|
|
5
|
-
version = "v0.1.
|
|
5
|
+
version = "v0.1.6"
|
|
6
6
|
description = "Command-line program that scans the NMDC MongoDB database for referential integrity violations"
|
|
7
7
|
authors = []
|
|
8
8
|
readme = "README.md"
|
|
@@ -27,11 +27,18 @@ rich = "^13.7.1"
|
|
|
27
27
|
typer = "^0.12.3"
|
|
28
28
|
|
|
29
29
|
[tool.poetry.group.dev.dependencies]
|
|
30
|
+
# We use `black` for code formatting.
|
|
31
|
+
# Usage: $ poetry run black --line-length 120 .
|
|
32
|
+
# Docs: https://black.readthedocs.io/en/stable/
|
|
30
33
|
black = "^24.4.2"
|
|
31
34
|
# We use `pyinstrument` for performance profiling.
|
|
32
35
|
# Usage: $ pyinstrument --from-path refscan --help
|
|
33
36
|
# Docs: https://pyinstrument.readthedocs.io/en/latest/guide.html#profile-a-python-cli-command
|
|
34
37
|
pyinstrument = "^4.6.2"
|
|
38
|
+
# We use `pytest` to run tests.
|
|
39
|
+
# Usage: $ poetry run pytest
|
|
40
|
+
# Docs: https://docs.pytest.org/en/stable/contents.html
|
|
41
|
+
pytest = "^8.2.2"
|
|
35
42
|
|
|
36
43
|
[tool.poetry.scripts]
|
|
37
44
|
# Reference: https://python-poetry.org/docs/pyproject#scripts
|
|
@@ -20,9 +20,7 @@ class Finder:
|
|
|
20
20
|
self.name_of_most_recently_found_in_collection = None
|
|
21
21
|
|
|
22
22
|
def check_whether_document_having_id_exists_among_collections(
|
|
23
|
-
|
|
24
|
-
document_id: str,
|
|
25
|
-
collection_names: List[str]
|
|
23
|
+
self, document_id: str, collection_names: List[str]
|
|
26
24
|
) -> bool:
|
|
27
25
|
r"""
|
|
28
26
|
Checks whether any documents in any of the specified collections
|
|
@@ -10,6 +10,7 @@ class Reference:
|
|
|
10
10
|
Note: `order` means the instances have methods that help with sorting. For example, an `__eq__` method that
|
|
11
11
|
can be used to compare instances of the class as thought they were tuples of those instances' fields.
|
|
12
12
|
"""
|
|
13
|
+
|
|
13
14
|
source_collection_name: str = field() # e.g. "study_set"
|
|
14
15
|
source_class_name: str = field() # e.g. "Study"
|
|
15
16
|
source_field_name: str = field() # e.g. "part_of"
|
|
@@ -40,9 +40,9 @@ class ReferenceList(UserList):
|
|
|
40
40
|
return distinct_source_field_names
|
|
41
41
|
|
|
42
42
|
def get_target_collection_names(
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
43
|
+
self,
|
|
44
|
+
source_class_name: str,
|
|
45
|
+
source_field_name: str,
|
|
46
46
|
) -> list[str]:
|
|
47
47
|
"""
|
|
48
48
|
Returns a list of the names of the collections in which a [target] document referenced by the specified field
|
|
@@ -60,10 +60,12 @@ class ReferenceList(UserList):
|
|
|
60
60
|
distinct_target_collection_names = list(set(target_collection_names))
|
|
61
61
|
return distinct_target_collection_names
|
|
62
62
|
|
|
63
|
-
def get_groups(self, field_names: list[str])
|
|
63
|
+
def get_groups(self, field_names: list[str]):
|
|
64
64
|
r"""
|
|
65
65
|
Returns an iterable of groups, where each group has a distinct combination of values in the specified fields.
|
|
66
66
|
|
|
67
|
+
TODO: Add a type hint that indicates the return type. It's an instance of a class named `groupby` (lowercase).
|
|
68
|
+
|
|
67
69
|
Note: This method can be used to "consolidate" references that have the same source collection name,
|
|
68
70
|
source field name, and target collection name (i.e. ones that only differ by target class name).
|
|
69
71
|
"""
|
|
@@ -121,10 +123,12 @@ class ReferenceList(UserList):
|
|
|
121
123
|
# Make the data rows for the table. Data rows that would have had the same
|
|
122
124
|
# (combination of) the following fields, get consolidated into a single row;
|
|
123
125
|
# and all the target class names are displayed as a list on that row.
|
|
124
|
-
fields_to_group_rows_by = [
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
126
|
+
fields_to_group_rows_by = [
|
|
127
|
+
"source_collection_name",
|
|
128
|
+
"source_class_name",
|
|
129
|
+
"source_field_name",
|
|
130
|
+
"target_collection_name",
|
|
131
|
+
]
|
|
128
132
|
groups = self.get_groups(fields_to_group_rows_by)
|
|
129
133
|
data_rows: list[tuple[str, str, str, str, str]] = []
|
|
130
134
|
for key, group in groups:
|
|
@@ -133,13 +137,15 @@ class ReferenceList(UserList):
|
|
|
133
137
|
data_rows.append(row)
|
|
134
138
|
|
|
135
139
|
# Initialize the table, then add the data rows to it.
|
|
136
|
-
table = Table(
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
140
|
+
table = Table(
|
|
141
|
+
Column(header="Source collection", footer=f"{len(data_rows)} rows"),
|
|
142
|
+
Column(header="Source class"),
|
|
143
|
+
Column(header="Source field"),
|
|
144
|
+
Column(header="Target collection"),
|
|
145
|
+
Column(header="Target class(es)"),
|
|
146
|
+
title="References",
|
|
147
|
+
show_footer=True,
|
|
148
|
+
)
|
|
143
149
|
for row in data_rows:
|
|
144
150
|
table.add_row(*row)
|
|
145
151
|
|
|
@@ -4,14 +4,7 @@ from functools import cache
|
|
|
4
4
|
from pymongo import MongoClient, timeout
|
|
5
5
|
from linkml_runtime import SchemaView
|
|
6
6
|
from rich.console import Console
|
|
7
|
-
from rich.progress import
|
|
8
|
-
Progress,
|
|
9
|
-
TextColumn,
|
|
10
|
-
MofNCompleteColumn,
|
|
11
|
-
BarColumn,
|
|
12
|
-
TimeElapsedColumn,
|
|
13
|
-
TimeRemainingColumn
|
|
14
|
-
)
|
|
7
|
+
from rich.progress import Progress, TextColumn, MofNCompleteColumn, BarColumn, TimeElapsedColumn, TimeRemainingColumn
|
|
15
8
|
|
|
16
9
|
from refscan.lib.constants import DATABASE_CLASS_NAME, console
|
|
17
10
|
|
|
@@ -22,7 +15,7 @@ def connect_to_database(mongo_uri: str, database_name: str, verbose: bool = True
|
|
|
22
15
|
"""
|
|
23
16
|
mongo_client: MongoClient = MongoClient(host=mongo_uri, directConnection=True)
|
|
24
17
|
|
|
25
|
-
with
|
|
18
|
+
with timeout(5): # if any message exchange takes > 5 seconds, this will raise an exception
|
|
26
19
|
(host, port_number) = mongo_client.address
|
|
27
20
|
|
|
28
21
|
if verbose:
|
|
@@ -35,9 +28,7 @@ def connect_to_database(mongo_uri: str, database_name: str, verbose: bool = True
|
|
|
35
28
|
return mongo_client
|
|
36
29
|
|
|
37
30
|
|
|
38
|
-
def get_collection_names_from_schema(
|
|
39
|
-
schema_view: SchemaView
|
|
40
|
-
) -> list[str]:
|
|
31
|
+
def get_collection_names_from_schema(schema_view: SchemaView) -> list[str]:
|
|
41
32
|
"""
|
|
42
33
|
Returns the names of the slots of the `Database` class that describe database collections.
|
|
43
34
|
|
|
@@ -4,7 +4,6 @@ from typing_extensions import Annotated
|
|
|
4
4
|
|
|
5
5
|
import typer
|
|
6
6
|
import linkml_runtime
|
|
7
|
-
from rich import padding
|
|
8
7
|
|
|
9
8
|
from refscan.lib.Finder import Finder
|
|
10
9
|
from refscan.lib.constants import DATABASE_CLASS_NAME, console
|
|
@@ -47,56 +46,81 @@ def display_app_version_and_exit(is_active: bool = False) -> None:
|
|
|
47
46
|
|
|
48
47
|
@app.command("scan")
|
|
49
48
|
def scan(
|
|
50
|
-
|
|
51
|
-
|
|
49
|
+
# Reference: https://typer.tiangolo.com/tutorial/parameter-types/path/
|
|
50
|
+
schema_file_path: Annotated[
|
|
51
|
+
Path,
|
|
52
|
+
typer.Option(
|
|
52
53
|
"--schema",
|
|
53
54
|
dir_okay=False,
|
|
54
55
|
writable=False,
|
|
55
56
|
readable=True,
|
|
56
57
|
resolve_path=True,
|
|
57
58
|
help="Filesystem path at which the YAML file representing the schema is located.",
|
|
58
|
-
)
|
|
59
|
-
|
|
59
|
+
),
|
|
60
|
+
],
|
|
61
|
+
database_name: Annotated[
|
|
62
|
+
str,
|
|
63
|
+
typer.Option(
|
|
60
64
|
help="Name of the database.",
|
|
61
|
-
)
|
|
62
|
-
|
|
65
|
+
),
|
|
66
|
+
] = "nmdc",
|
|
67
|
+
mongo_uri: Annotated[
|
|
68
|
+
str,
|
|
69
|
+
typer.Option(
|
|
63
70
|
envvar="MONGO_URI",
|
|
64
71
|
help="Connection string for accessing the MongoDB server. If you have Docker installed, "
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
)
|
|
68
|
-
|
|
72
|
+
"you can spin up a temporary MongoDB server at the default URI by running: "
|
|
73
|
+
"`$ docker run --rm --detach -p 27017:27017 mongo`",
|
|
74
|
+
),
|
|
75
|
+
] = "mongodb://localhost:27017",
|
|
76
|
+
verbose: Annotated[
|
|
77
|
+
bool,
|
|
78
|
+
typer.Option(
|
|
69
79
|
"--verbose",
|
|
70
80
|
help="Show verbose output.",
|
|
71
|
-
)
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
81
|
+
),
|
|
82
|
+
] = False,
|
|
83
|
+
# Reference: https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/
|
|
84
|
+
skip_source_collection: Annotated[
|
|
85
|
+
Optional[List[str]],
|
|
86
|
+
typer.Option(
|
|
87
|
+
"--skip-source-collection",
|
|
88
|
+
"--skip",
|
|
75
89
|
help="Name of collection you do not want to search for referring documents. "
|
|
76
|
-
|
|
77
|
-
)
|
|
78
|
-
|
|
90
|
+
"Option can be used multiple times.",
|
|
91
|
+
),
|
|
92
|
+
] = None,
|
|
93
|
+
reference_report_file_path: Annotated[
|
|
94
|
+
Optional[Path],
|
|
95
|
+
typer.Option(
|
|
79
96
|
"--reference-report",
|
|
80
97
|
dir_okay=False,
|
|
81
98
|
writable=True,
|
|
82
99
|
readable=False,
|
|
83
100
|
resolve_path=True,
|
|
84
101
|
help="Filesystem path at which you want the program to generate its reference report.",
|
|
85
|
-
)
|
|
86
|
-
|
|
102
|
+
),
|
|
103
|
+
] = "references.tsv",
|
|
104
|
+
violation_report_file_path: Annotated[
|
|
105
|
+
Optional[Path],
|
|
106
|
+
typer.Option(
|
|
87
107
|
"--violation-report",
|
|
88
108
|
dir_okay=False,
|
|
89
109
|
writable=True,
|
|
90
110
|
readable=False,
|
|
91
111
|
resolve_path=True,
|
|
92
112
|
help="Filesystem path at which you want the program to generate its violation report.",
|
|
93
|
-
)
|
|
94
|
-
|
|
113
|
+
),
|
|
114
|
+
] = "violations.tsv",
|
|
115
|
+
version: Annotated[
|
|
116
|
+
Optional[bool],
|
|
117
|
+
typer.Option(
|
|
95
118
|
"--version",
|
|
96
119
|
callback=display_app_version_and_exit,
|
|
97
120
|
is_eager=True, # tells Typer to process this option first
|
|
98
121
|
help="Show version number and exit.",
|
|
99
|
-
)
|
|
122
|
+
),
|
|
123
|
+
] = None,
|
|
100
124
|
):
|
|
101
125
|
"""
|
|
102
126
|
Scans the NMDC MongoDB database for referential integrity violations.
|
|
@@ -170,11 +194,13 @@ def scan(
|
|
|
170
194
|
for name_of_eligible_target_class in names_of_eligible_target_classes:
|
|
171
195
|
for target_collection_name, class_names_in_collection in collection_name_to_class_names.items():
|
|
172
196
|
if name_of_eligible_target_class in class_names_in_collection:
|
|
173
|
-
reference = Reference(
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
197
|
+
reference = Reference(
|
|
198
|
+
source_collection_name=collection_name,
|
|
199
|
+
source_class_name=class_name,
|
|
200
|
+
source_field_name=slot_name,
|
|
201
|
+
target_collection_name=target_collection_name,
|
|
202
|
+
target_class_name=name_of_eligible_target_class,
|
|
203
|
+
)
|
|
178
204
|
references.append(reference)
|
|
179
205
|
|
|
180
206
|
console.print(f"References described by schema: {len(references)}")
|
|
@@ -232,8 +258,8 @@ def scan(
|
|
|
232
258
|
# those that have _any_ of the fields (of classes whose instances are allowed to reside in this collection)
|
|
233
259
|
# that the schema allows to contain a reference to an instance.
|
|
234
260
|
source_field_names = references.get_source_field_names_of_source_collection(source_collection_name)
|
|
235
|
-
or_terms = [{field_name: {
|
|
236
|
-
query_filter = {
|
|
261
|
+
or_terms = [{field_name: {"$exists": True}} for field_name in source_field_names]
|
|
262
|
+
query_filter = {"$or": or_terms}
|
|
237
263
|
if verbose:
|
|
238
264
|
console.print(f"{query_filter=}")
|
|
239
265
|
|
|
@@ -251,10 +277,12 @@ def scan(
|
|
|
251
277
|
|
|
252
278
|
# Set up the progress bar for the task of scanning those documents.
|
|
253
279
|
num_relevant_documents = collection.count_documents(query_filter)
|
|
254
|
-
task_id = progress.add_task(
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
280
|
+
task_id = progress.add_task(
|
|
281
|
+
f"{source_collection_name}",
|
|
282
|
+
total=num_relevant_documents,
|
|
283
|
+
num_violations=0,
|
|
284
|
+
remaining_time_label="remaining",
|
|
285
|
+
)
|
|
258
286
|
|
|
259
287
|
# Advance the progress bar by 0 (this makes it so that, even if there are 0 relevant documents, the progress
|
|
260
288
|
# bar does not continue incrementing its "elapsed time" even after a subsequent task has begun).
|
|
@@ -296,25 +324,30 @@ def scan(
|
|
|
296
324
|
|
|
297
325
|
for target_id in target_ids:
|
|
298
326
|
target_exists = finder.check_whether_document_having_id_exists_among_collections(
|
|
299
|
-
collection_names=target_collection_names,
|
|
300
|
-
document_id=target_id
|
|
327
|
+
collection_names=target_collection_names, document_id=target_id
|
|
301
328
|
)
|
|
302
329
|
if not target_exists:
|
|
303
|
-
violation = Violation(
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
330
|
+
violation = Violation(
|
|
331
|
+
source_collection_name=source_collection_name,
|
|
332
|
+
source_field_name=field_name,
|
|
333
|
+
source_document_object_id=source_document_object_id,
|
|
334
|
+
source_document_id=source_document_id,
|
|
335
|
+
target_id=target_id,
|
|
336
|
+
)
|
|
308
337
|
source_collections_and_their_violations[source_collection_name].append(violation)
|
|
309
338
|
if verbose:
|
|
310
|
-
console.print(
|
|
311
|
-
|
|
312
|
-
|
|
339
|
+
console.print(
|
|
340
|
+
f"Failed to find document having `id` '{target_id}' "
|
|
341
|
+
f"among collections: {target_collection_names}. "
|
|
342
|
+
f"{violation=}"
|
|
343
|
+
)
|
|
313
344
|
|
|
314
345
|
# Advance the progress bar to account for the current document's contribution to the violations count.
|
|
315
|
-
progress.update(
|
|
316
|
-
|
|
317
|
-
|
|
346
|
+
progress.update(
|
|
347
|
+
task_id,
|
|
348
|
+
advance=1,
|
|
349
|
+
num_violations=len(source_collections_and_their_violations[source_collection_name]),
|
|
350
|
+
)
|
|
318
351
|
|
|
319
352
|
# Update the progress bar to indicate the current task is complete.
|
|
320
353
|
progress.update(task_id, remaining_time_label="done")
|
|
File without changes
|
|
File without changes
|
|
File without changes
|