refscan 0.1.4__tar.gz → 0.1.6__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: refscan
3
- Version: 0.1.4
3
+ Version: 0.1.6
4
4
  Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
5
5
  Home-page: https://github.com/microbiomedata/refscan
6
6
  Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
@@ -45,18 +45,18 @@ graph LR
45
45
  `refscan` does its job in two stages:
46
46
  1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
47
47
  that _can_ exist in a database that conforms to the schema.
48
- > **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
49
- > a field named `owner` that contains the `id` of a document in a collection named `persons`.
48
+ > **Example:** The schema might say that, for each document in the `biosample_set` collection that has a field named
49
+ > `associated_studies`, that field must contain a list of `id`s of documents in the `study_set` collection.
50
50
  2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
51
- > **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
52
- > field matches the `id` of some document in the collection named `persons`.
51
+ > **Example:** For each document in the `biosample_set` collection that _has_ a field named `associated_studies`,
52
+ > for each value in that field, confirm there _is_ a document having that `id` in the `study_set` collection.
53
53
 
54
54
  ## Limitations
55
55
 
56
- `refscan` was designed under the assumption that **each document has a field named `type`,
57
- whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
58
- `refscan` relies on that `type` field when determining the name of the class of which a document represents an
59
- instance; which it does in order to determine "which fields" of that document can contain a reference.
56
+ `refscan` was designed under the assumption that **every document** in **every collection described by the schema** has
57
+ a **field named `type`**, whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class the document represents an instance
58
+ of. `refscan` uses that `class_uri` value (in that `type` field) to determine the name of that schema class,
59
+ which it, in turn, uses to determine _which fields_ of that document can contain references.
60
60
 
61
61
  ## Usage
62
62
 
@@ -167,6 +167,15 @@ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-sc
167
167
  ---
168
168
  </details>
169
169
 
170
+ #### Output
171
+
172
+ While `refscan` is running, it will display console output indicating what it's currently doing.
173
+
174
+ ![Screenshot of refscan console output](./docs/refscan-screenshot.png)
175
+
176
+ Once the scan is complete, the reference report (TSV file) and violation report (TSV file) will be available
177
+ in the current directory (or in custom directories, if any were specified via CLI options).
178
+
170
179
  ### Updating
171
180
 
172
181
  You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
@@ -185,10 +194,10 @@ pipx uninstall refscan
185
194
 
186
195
  ## Development
187
196
 
188
- We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
197
+ We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) build distributable packages that can be published to PyPI.
189
198
 
190
- - `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
191
- - `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
199
+ - `pyproject.toml`: Configuration file for Poetry and other tools (was initialized via `$ poetry init`)
200
+ - `poetry.lock`: List of dependencies, both direct and [indirect/transitive](https://en.wikipedia.org/wiki/Transitive_dependency)
192
201
 
193
202
  ### Clone repository
194
203
 
@@ -229,6 +238,41 @@ While editing the tool's source code, you can run the tool as you normally would
229
238
  refscan --help
230
239
  ```
231
240
 
241
+ ### Run tests
242
+
243
+ We use [pytest](https://docs.pytest.org/en/8.2.x/) as the testing framework for `refscan`.
244
+
245
+ You can run the tests by running the following command from the root directory of the repository:
246
+
247
+ ```shell
248
+ poetry run pytest
249
+ ```
250
+
251
+ Tests are defined in the `tests` directory.
252
+
253
+ ### Format code
254
+
255
+ We use [`black`](https://black.readthedocs.io/en/stable/) as the code formatter for `refscan`.
256
+
257
+ We do not use it with its default options. Instead, we include an option that allows lines to be 120 characters
258
+ instead of the default 88 characters.
259
+
260
+ You can format all the Python code in the repository by running this command
261
+ from the root directory of the repository:
262
+
263
+ ```shell
264
+ poetry run black --line-length 120 .
265
+ ```
266
+
267
+ #### Check format
268
+
269
+ You can _check_ the format of the Python code by including the `--check` option, like this:
270
+
271
+ ```shell
272
+ poetry run black --line-length 120 --check .
273
+ ```
274
+
275
+
232
276
  ## Building and publishing
233
277
 
234
278
  ### Build for production
@@ -238,8 +282,6 @@ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.
238
282
  will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
239
283
  That package will have a version identifier that matches the name of the Git tag associated with the Release.
240
284
 
241
- The GitHub Actions workflow will automatically
242
-
243
285
  ### Test the build process locally
244
286
 
245
287
  In case you want to test the build process locally, you can do so by running:
@@ -26,18 +26,18 @@ graph LR
26
26
  `refscan` does its job in two stages:
27
27
  1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
28
28
  that _can_ exist in a database that conforms to the schema.
29
- > **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
30
- > a field named `owner` that contains the `id` of a document in a collection named `persons`.
29
+ > **Example:** The schema might say that, for each document in the `biosample_set` collection that has a field named
30
+ > `associated_studies`, that field must contain a list of `id`s of documents in the `study_set` collection.
31
31
  2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
32
- > **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
33
- > field matches the `id` of some document in the collection named `persons`.
32
+ > **Example:** For each document in the `biosample_set` collection that _has_ a field named `associated_studies`,
33
+ > for each value in that field, confirm there _is_ a document having that `id` in the `study_set` collection.
34
34
 
35
35
  ## Limitations
36
36
 
37
- `refscan` was designed under the assumption that **each document has a field named `type`,
38
- whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
39
- `refscan` relies on that `type` field when determining the name of the class of which a document represents an
40
- instance; which it does in order to determine "which fields" of that document can contain a reference.
37
+ `refscan` was designed under the assumption that **every document** in **every collection described by the schema** has
38
+ a **field named `type`**, whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class the document represents an instance
39
+ of. `refscan` uses that `class_uri` value (in that `type` field) to determine the name of that schema class,
40
+ which it, in turn, uses to determine _which fields_ of that document can contain references.
41
41
 
42
42
  ## Usage
43
43
 
@@ -148,6 +148,15 @@ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-sc
148
148
  ---
149
149
  </details>
150
150
 
151
+ #### Output
152
+
153
+ While `refscan` is running, it will display console output indicating what it's currently doing.
154
+
155
+ ![Screenshot of refscan console output](./docs/refscan-screenshot.png)
156
+
157
+ Once the scan is complete, the reference report (TSV file) and violation report (TSV file) will be available
158
+ in the current directory (or in custom directories, if any were specified via CLI options).
159
+
151
160
  ### Updating
152
161
 
153
162
  You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
@@ -166,10 +175,10 @@ pipx uninstall refscan
166
175
 
167
176
  ## Development
168
177
 
169
- We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
178
+ We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) build distributable packages that can be published to PyPI.
170
179
 
171
- - `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
172
- - `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
180
+ - `pyproject.toml`: Configuration file for Poetry and other tools (was initialized via `$ poetry init`)
181
+ - `poetry.lock`: List of dependencies, both direct and [indirect/transitive](https://en.wikipedia.org/wiki/Transitive_dependency)
173
182
 
174
183
  ### Clone repository
175
184
 
@@ -210,6 +219,41 @@ While editing the tool's source code, you can run the tool as you normally would
210
219
  refscan --help
211
220
  ```
212
221
 
222
+ ### Run tests
223
+
224
+ We use [pytest](https://docs.pytest.org/en/8.2.x/) as the testing framework for `refscan`.
225
+
226
+ You can run the tests by running the following command from the root directory of the repository:
227
+
228
+ ```shell
229
+ poetry run pytest
230
+ ```
231
+
232
+ Tests are defined in the `tests` directory.
233
+
234
+ ### Format code
235
+
236
+ We use [`black`](https://black.readthedocs.io/en/stable/) as the code formatter for `refscan`.
237
+
238
+ We do not use it with its default options. Instead, we include an option that allows lines to be 120 characters
239
+ instead of the default 88 characters.
240
+
241
+ You can format all the Python code in the repository by running this command
242
+ from the root directory of the repository:
243
+
244
+ ```shell
245
+ poetry run black --line-length 120 .
246
+ ```
247
+
248
+ #### Check format
249
+
250
+ You can _check_ the format of the Python code by including the `--check` option, like this:
251
+
252
+ ```shell
253
+ poetry run black --line-length 120 --check .
254
+ ```
255
+
256
+
213
257
  ## Building and publishing
214
258
 
215
259
  ### Build for production
@@ -219,8 +263,6 @@ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.
219
263
  will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
220
264
  That package will have a version identifier that matches the name of the Git tag associated with the Release.
221
265
 
222
- The GitHub Actions workflow will automatically
223
-
224
266
  ### Test the build process locally
225
267
 
226
268
  In case you want to test the build process locally, you can do so by running:
@@ -2,7 +2,7 @@
2
2
 
3
3
  [tool.poetry]
4
4
  name = "refscan"
5
- version = "v0.1.4"
5
+ version = "v0.1.6"
6
6
  description = "Command-line program that scans the NMDC MongoDB database for referential integrity violations"
7
7
  authors = []
8
8
  readme = "README.md"
@@ -27,11 +27,18 @@ rich = "^13.7.1"
27
27
  typer = "^0.12.3"
28
28
 
29
29
  [tool.poetry.group.dev.dependencies]
30
+ # We use `black` for code formatting.
31
+ # Usage: $ poetry run black --line-length 120 .
32
+ # Docs: https://black.readthedocs.io/en/stable/
30
33
  black = "^24.4.2"
31
34
  # We use `pyinstrument` for performance profiling.
32
35
  # Usage: $ pyinstrument --from-path refscan --help
33
36
  # Docs: https://pyinstrument.readthedocs.io/en/latest/guide.html#profile-a-python-cli-command
34
37
  pyinstrument = "^4.6.2"
38
+ # We use `pytest` to run tests.
39
+ # Usage: $ poetry run pytest
40
+ # Docs: https://docs.pytest.org/en/stable/contents.html
41
+ pytest = "^8.2.2"
35
42
 
36
43
  [tool.poetry.scripts]
37
44
  # Reference: https://python-poetry.org/docs/pyproject#scripts
@@ -20,9 +20,7 @@ class Finder:
20
20
  self.name_of_most_recently_found_in_collection = None
21
21
 
22
22
  def check_whether_document_having_id_exists_among_collections(
23
- self,
24
- document_id: str,
25
- collection_names: List[str]
23
+ self, document_id: str, collection_names: List[str]
26
24
  ) -> bool:
27
25
  r"""
28
26
  Checks whether any documents in any of the specified collections
@@ -10,6 +10,7 @@ class Reference:
10
10
  Note: `order` means the instances have methods that help with sorting. For example, an `__eq__` method that
11
11
  can be used to compare instances of the class as thought they were tuples of those instances' fields.
12
12
  """
13
+
13
14
  source_collection_name: str = field() # e.g. "study_set"
14
15
  source_class_name: str = field() # e.g. "Study"
15
16
  source_field_name: str = field() # e.g. "part_of"
@@ -40,9 +40,9 @@ class ReferenceList(UserList):
40
40
  return distinct_source_field_names
41
41
 
42
42
  def get_target_collection_names(
43
- self,
44
- source_class_name: str,
45
- source_field_name: str,
43
+ self,
44
+ source_class_name: str,
45
+ source_field_name: str,
46
46
  ) -> list[str]:
47
47
  """
48
48
  Returns a list of the names of the collections in which a [target] document referenced by the specified field
@@ -60,10 +60,12 @@ class ReferenceList(UserList):
60
60
  distinct_target_collection_names = list(set(target_collection_names))
61
61
  return distinct_target_collection_names
62
62
 
63
- def get_groups(self, field_names: list[str]) -> list[tuple[str, str, str, str, list[str]]]:
63
+ def get_groups(self, field_names: list[str]):
64
64
  r"""
65
65
  Returns an iterable of groups, where each group has a distinct combination of values in the specified fields.
66
66
 
67
+ TODO: Add a type hint that indicates the return type. It's an instance of a class named `groupby` (lowercase).
68
+
67
69
  Note: This method can be used to "consolidate" references that have the same source collection name,
68
70
  source field name, and target collection name (i.e. ones that only differ by target class name).
69
71
  """
@@ -121,10 +123,12 @@ class ReferenceList(UserList):
121
123
  # Make the data rows for the table. Data rows that would have had the same
122
124
  # (combination of) the following fields, get consolidated into a single row;
123
125
  # and all the target class names are displayed as a list on that row.
124
- fields_to_group_rows_by = ["source_collection_name",
125
- "source_class_name",
126
- "source_field_name",
127
- "target_collection_name"]
126
+ fields_to_group_rows_by = [
127
+ "source_collection_name",
128
+ "source_class_name",
129
+ "source_field_name",
130
+ "target_collection_name",
131
+ ]
128
132
  groups = self.get_groups(fields_to_group_rows_by)
129
133
  data_rows: list[tuple[str, str, str, str, str]] = []
130
134
  for key, group in groups:
@@ -133,13 +137,15 @@ class ReferenceList(UserList):
133
137
  data_rows.append(row)
134
138
 
135
139
  # Initialize the table, then add the data rows to it.
136
- table = Table(Column(header="Source collection", footer=f"{len(data_rows)} rows"),
137
- Column(header="Source class"),
138
- Column(header="Source field"),
139
- Column(header="Target collection"),
140
- Column(header="Target class(es)"),
141
- title="References",
142
- show_footer=True)
140
+ table = Table(
141
+ Column(header="Source collection", footer=f"{len(data_rows)} rows"),
142
+ Column(header="Source class"),
143
+ Column(header="Source field"),
144
+ Column(header="Target collection"),
145
+ Column(header="Target class(es)"),
146
+ title="References",
147
+ show_footer=True,
148
+ )
143
149
  for row in data_rows:
144
150
  table.add_row(*row)
145
151
 
@@ -6,6 +6,7 @@ class Violation:
6
6
  """
7
7
  A specific reference that lacks integrity.
8
8
  """
9
+
9
10
  source_collection_name: str = field()
10
11
  source_field_name: str = field()
11
12
  source_document_object_id: str = field()
@@ -4,14 +4,7 @@ from functools import cache
4
4
  from pymongo import MongoClient, timeout
5
5
  from linkml_runtime import SchemaView
6
6
  from rich.console import Console
7
- from rich.progress import (
8
- Progress,
9
- TextColumn,
10
- MofNCompleteColumn,
11
- BarColumn,
12
- TimeElapsedColumn,
13
- TimeRemainingColumn
14
- )
7
+ from rich.progress import Progress, TextColumn, MofNCompleteColumn, BarColumn, TimeElapsedColumn, TimeRemainingColumn
15
8
 
16
9
  from refscan.lib.constants import DATABASE_CLASS_NAME, console
17
10
 
@@ -22,7 +15,7 @@ def connect_to_database(mongo_uri: str, database_name: str, verbose: bool = True
22
15
  """
23
16
  mongo_client: MongoClient = MongoClient(host=mongo_uri, directConnection=True)
24
17
 
25
- with (timeout(5)): # if any message exchange takes > 5 seconds, this will raise an exception
18
+ with timeout(5): # if any message exchange takes > 5 seconds, this will raise an exception
26
19
  (host, port_number) = mongo_client.address
27
20
 
28
21
  if verbose:
@@ -35,9 +28,7 @@ def connect_to_database(mongo_uri: str, database_name: str, verbose: bool = True
35
28
  return mongo_client
36
29
 
37
30
 
38
- def get_collection_names_from_schema(
39
- schema_view: SchemaView
40
- ) -> list[str]:
31
+ def get_collection_names_from_schema(schema_view: SchemaView) -> list[str]:
41
32
  """
42
33
  Returns the names of the slots of the `Database` class that describe database collections.
43
34
 
@@ -4,7 +4,6 @@ from typing_extensions import Annotated
4
4
 
5
5
  import typer
6
6
  import linkml_runtime
7
- from rich import padding
8
7
 
9
8
  from refscan.lib.Finder import Finder
10
9
  from refscan.lib.constants import DATABASE_CLASS_NAME, console
@@ -47,56 +46,81 @@ def display_app_version_and_exit(is_active: bool = False) -> None:
47
46
 
48
47
  @app.command("scan")
49
48
  def scan(
50
- # Reference: https://typer.tiangolo.com/tutorial/parameter-types/path/
51
- schema_file_path: Annotated[Path, typer.Option(
49
+ # Reference: https://typer.tiangolo.com/tutorial/parameter-types/path/
50
+ schema_file_path: Annotated[
51
+ Path,
52
+ typer.Option(
52
53
  "--schema",
53
54
  dir_okay=False,
54
55
  writable=False,
55
56
  readable=True,
56
57
  resolve_path=True,
57
58
  help="Filesystem path at which the YAML file representing the schema is located.",
58
- )],
59
- database_name: Annotated[str, typer.Option(
59
+ ),
60
+ ],
61
+ database_name: Annotated[
62
+ str,
63
+ typer.Option(
60
64
  help="Name of the database.",
61
- )] = "nmdc",
62
- mongo_uri: Annotated[str, typer.Option(
65
+ ),
66
+ ] = "nmdc",
67
+ mongo_uri: Annotated[
68
+ str,
69
+ typer.Option(
63
70
  envvar="MONGO_URI",
64
71
  help="Connection string for accessing the MongoDB server. If you have Docker installed, "
65
- "you can spin up a temporary MongoDB server at the default URI by running: "
66
- "`$ docker run --rm --detach -p 27017:27017 mongo`",
67
- )] = "mongodb://localhost:27017",
68
- verbose: Annotated[bool, typer.Option(
72
+ "you can spin up a temporary MongoDB server at the default URI by running: "
73
+ "`$ docker run --rm --detach -p 27017:27017 mongo`",
74
+ ),
75
+ ] = "mongodb://localhost:27017",
76
+ verbose: Annotated[
77
+ bool,
78
+ typer.Option(
69
79
  "--verbose",
70
80
  help="Show verbose output.",
71
- )] = False,
72
- # Reference: https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/
73
- skip_source_collection: Annotated[Optional[List[str]], typer.Option(
74
- "--skip-source-collection", "--skip",
81
+ ),
82
+ ] = False,
83
+ # Reference: https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/
84
+ skip_source_collection: Annotated[
85
+ Optional[List[str]],
86
+ typer.Option(
87
+ "--skip-source-collection",
88
+ "--skip",
75
89
  help="Name of collection you do not want to search for referring documents. "
76
- "Option can be used multiple times.",
77
- )] = None,
78
- reference_report_file_path: Annotated[Optional[Path], typer.Option(
90
+ "Option can be used multiple times.",
91
+ ),
92
+ ] = None,
93
+ reference_report_file_path: Annotated[
94
+ Optional[Path],
95
+ typer.Option(
79
96
  "--reference-report",
80
97
  dir_okay=False,
81
98
  writable=True,
82
99
  readable=False,
83
100
  resolve_path=True,
84
101
  help="Filesystem path at which you want the program to generate its reference report.",
85
- )] = "references.tsv",
86
- violation_report_file_path: Annotated[Optional[Path], typer.Option(
102
+ ),
103
+ ] = "references.tsv",
104
+ violation_report_file_path: Annotated[
105
+ Optional[Path],
106
+ typer.Option(
87
107
  "--violation-report",
88
108
  dir_okay=False,
89
109
  writable=True,
90
110
  readable=False,
91
111
  resolve_path=True,
92
112
  help="Filesystem path at which you want the program to generate its violation report.",
93
- )] = "violations.tsv",
94
- version: Annotated[Optional[bool], typer.Option(
113
+ ),
114
+ ] = "violations.tsv",
115
+ version: Annotated[
116
+ Optional[bool],
117
+ typer.Option(
95
118
  "--version",
96
119
  callback=display_app_version_and_exit,
97
120
  is_eager=True, # tells Typer to process this option first
98
121
  help="Show version number and exit.",
99
- )] = None,
122
+ ),
123
+ ] = None,
100
124
  ):
101
125
  """
102
126
  Scans the NMDC MongoDB database for referential integrity violations.
@@ -170,11 +194,13 @@ def scan(
170
194
  for name_of_eligible_target_class in names_of_eligible_target_classes:
171
195
  for target_collection_name, class_names_in_collection in collection_name_to_class_names.items():
172
196
  if name_of_eligible_target_class in class_names_in_collection:
173
- reference = Reference(source_collection_name=collection_name,
174
- source_class_name=class_name,
175
- source_field_name=slot_name,
176
- target_collection_name=target_collection_name,
177
- target_class_name=name_of_eligible_target_class)
197
+ reference = Reference(
198
+ source_collection_name=collection_name,
199
+ source_class_name=class_name,
200
+ source_field_name=slot_name,
201
+ target_collection_name=target_collection_name,
202
+ target_class_name=name_of_eligible_target_class,
203
+ )
178
204
  references.append(reference)
179
205
 
180
206
  console.print(f"References described by schema: {len(references)}")
@@ -232,8 +258,8 @@ def scan(
232
258
  # those that have _any_ of the fields (of classes whose instances are allowed to reside in this collection)
233
259
  # that the schema allows to contain a reference to an instance.
234
260
  source_field_names = references.get_source_field_names_of_source_collection(source_collection_name)
235
- or_terms = [{field_name: {'$exists': True}} for field_name in source_field_names]
236
- query_filter = {'$or': or_terms}
261
+ or_terms = [{field_name: {"$exists": True}} for field_name in source_field_names]
262
+ query_filter = {"$or": or_terms}
237
263
  if verbose:
238
264
  console.print(f"{query_filter=}")
239
265
 
@@ -251,10 +277,12 @@ def scan(
251
277
 
252
278
  # Set up the progress bar for the task of scanning those documents.
253
279
  num_relevant_documents = collection.count_documents(query_filter)
254
- task_id = progress.add_task(f"{source_collection_name}",
255
- total=num_relevant_documents,
256
- num_violations=0,
257
- remaining_time_label="remaining")
280
+ task_id = progress.add_task(
281
+ f"{source_collection_name}",
282
+ total=num_relevant_documents,
283
+ num_violations=0,
284
+ remaining_time_label="remaining",
285
+ )
258
286
 
259
287
  # Advance the progress bar by 0 (this makes it so that, even if there are 0 relevant documents, the progress
260
288
  # bar does not continue incrementing its "elapsed time" even after a subsequent task has begun).
@@ -296,25 +324,30 @@ def scan(
296
324
 
297
325
  for target_id in target_ids:
298
326
  target_exists = finder.check_whether_document_having_id_exists_among_collections(
299
- collection_names=target_collection_names,
300
- document_id=target_id
327
+ collection_names=target_collection_names, document_id=target_id
301
328
  )
302
329
  if not target_exists:
303
- violation = Violation(source_collection_name=source_collection_name,
304
- source_field_name=field_name,
305
- source_document_object_id=source_document_object_id,
306
- source_document_id=source_document_id,
307
- target_id=target_id)
330
+ violation = Violation(
331
+ source_collection_name=source_collection_name,
332
+ source_field_name=field_name,
333
+ source_document_object_id=source_document_object_id,
334
+ source_document_id=source_document_id,
335
+ target_id=target_id,
336
+ )
308
337
  source_collections_and_their_violations[source_collection_name].append(violation)
309
338
  if verbose:
310
- console.print(f"Failed to find document having `id` '{target_id}' "
311
- f"among collections: {target_collection_names}. "
312
- f"{violation=}")
339
+ console.print(
340
+ f"Failed to find document having `id` '{target_id}' "
341
+ f"among collections: {target_collection_names}. "
342
+ f"{violation=}"
343
+ )
313
344
 
314
345
  # Advance the progress bar to account for the current document's contribution to the violations count.
315
- progress.update(task_id,
316
- advance=1,
317
- num_violations=len(source_collections_and_their_violations[source_collection_name]))
346
+ progress.update(
347
+ task_id,
348
+ advance=1,
349
+ num_violations=len(source_collections_and_their_violations[source_collection_name]),
350
+ )
318
351
 
319
352
  # Update the progress bar to indicate the current task is complete.
320
353
  progress.update(task_id, remaining_time_label="done")
File without changes