refscan 0.1.0__tar.gz → 0.1.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
refscan-0.1.2/PKG-INFO ADDED
@@ -0,0 +1,256 @@
1
+ Metadata-Version: 2.1
2
+ Name: refscan
3
+ Version: 0.1.2
4
+ Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
5
+ Home-page: https://github.com/microbiomedata/refscan
6
+ Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
7
+ Requires-Python: >=3.10,<4.0
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: Programming Language :: Python :: 3.10
10
+ Classifier: Programming Language :: Python :: 3.11
11
+ Classifier: Programming Language :: Python :: 3.12
12
+ Requires-Dist: linkml-runtime (>=1.7.5,<2.0.0)
13
+ Requires-Dist: pymongo (>=4.7.1,<5.0.0)
14
+ Requires-Dist: rich (>=13.7.1,<14.0.0)
15
+ Requires-Dist: typer (>=0.12.3,<0.13.0)
16
+ Project-URL: Documentation, https://github.com/microbiomedata/refscan
17
+ Project-URL: Repository, https://github.com/microbiomedata/refscan
18
+ Description-Content-Type: text/markdown
19
+
20
+ # refscan
21
+
22
+ `refscan` is a command-line tool people can use to scan the [NMDC](https://microbiomedata.org/) MongoDB database
23
+ for referential integrity violations.
24
+
25
+ ```mermaid
26
+ %% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
27
+ %% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
28
+ %% Reference: https://github.com/pypi/warehouse/issues/13083
29
+ graph LR
30
+ schema[LinkML<br>schema]
31
+ database[(MongoDB<br>database)]
32
+ script[["refscan.py"]]
33
+ violations["List of<br>violations"]
34
+ references["List of<br>references"]:::dashed_border
35
+ schema --> script
36
+ database --> script
37
+ script -.-> references
38
+ script --> violations
39
+
40
+ classDef dashed_border stroke-dasharray: 5 5
41
+ ```
42
+
43
+ ## How it works
44
+
45
+ `refscan` does its job in two stages:
46
+ 1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
47
+ that _can_ exist in a database that conforms to the schema.
48
+ > **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
49
+ > a field named `owner` that contains the `id` of a document in a collection named `persons`.
50
+ 2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
51
+ > **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
52
+ > field matches the `id` of some document in the collection named `persons`.
53
+
54
+ ## Limitations
55
+
56
+ `refscan` was designed under the assumption that **each document has a field named `type`,
57
+ whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
58
+ `refscan` relies on that `type` field when determining the name of the class of which a document represents an
59
+ instance; which it does in order to determine "which fields" of that document can contain a reference.
60
+
61
+ ## Usage
62
+
63
+ ### Installation
64
+
65
+ Assuming you have `pipx` installed, you can install the tool by running the following command:
66
+
67
+ ```shell
68
+ pipx install refscan
69
+ ```
70
+
71
+ > [`pipx`](https://pipx.pypa.io/stable/) is a tool people can use to
72
+ > [download and install](https://pipx.pypa.io/stable/#where-does-pipx-install-apps-from)
73
+ > Python scripts that are hosted on PyPI.
74
+ > You can [install `pipx`](https://pipx.pypa.io/stable/installation/) by running `$ python -m pip install pipx`.
75
+
76
+ ### Running
77
+
78
+ You can display the tool's `--help` snippet by running:
79
+
80
+ ```shell
81
+ refscan --help
82
+ ```
83
+
84
+ At the time of this writing, the tool's `--help` snippet is:
85
+
86
+ ```console
87
+ Usage: refscan [OPTIONS]
88
+
89
+ Scans the NMDC MongoDB database for referential integrity violations.
90
+
91
+ ╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
92
+ │ * --schema FILE Filesystem path at which the YAML file │
93
+ │ representing the schema is located. │
94
+ │ [default: None] │
95
+ │ [required] │
96
+ │ --database-name TEXT Name of the database. │
97
+ │ [default: nmdc] │
98
+ │ --mongo-uri TEXT Connection string for accessing the │
99
+ │ MongoDB server. If you have Docker │
100
+ │ installed, you can spin up a temporary │
101
+ │ MongoDB server at the default URI by │
102
+ │ running: $ docker run --rm --detach -p │
103
+ │ 27017:27017 mongo │
104
+ │ [env var: MONGO_URI] │
105
+ │ [default: mongodb://localhost:27017] │
106
+ │ --verbose Show verbose output. │
107
+ │ --skip-source-collection,--skip TEXT Name of collection you do not want to │
108
+ │ search for referring documents. Option │
109
+ │ can be used multiple times. │
110
+ │ [default: None] │
111
+ │ --reference-report FILE Filesystem path at which you want the │
112
+ │ program to generate its reference │
113
+ │ report. │
114
+ │ [default: references.tsv] │
115
+ │ --violation-report FILE Filesystem path at which you want the │
116
+ │ program to generate its violation │
117
+ │ report. │
118
+ │ [default: violations.tsv] │
119
+ │ --version Show version number and exit. │
120
+ │ --help Show this message and exit. │
121
+ ╰────────────────────────────────────────────────────────────────────────────────────────╯
122
+ ```
123
+
124
+ > Note: The above snippet was captured from a terminal window whose width was 90 characters.
125
+
126
+ #### The MongoDB connection string (`--mongo-uri`)
127
+
128
+ As documented in the `--help` snippet above, you can provide the MongoDB connection string to the tool via either
129
+ (a) the `--mongo-uri` option; or (b) an environment variable named `MONGO_URI`. The latter can come in handy
130
+ when the MongoDB connection string contains information you don't want to appear in your shell history,
131
+ such as a password.
132
+
133
+ Here's how you could create that environment variable:
134
+
135
+ ```shell
136
+ export MONGO_URI='mongodb://username:password@localhost:27017'
137
+ ```
138
+
139
+ #### The schema (`--schema`)
140
+
141
+ As documented in the `--help` snippet above, you can provide the path to a YAML-formatted LinkML schema file to the tool
142
+ via the `--schema` option.
143
+
144
+ <details>
145
+
146
+ <summary>
147
+ Show/hide tips for getting a schema file
148
+ </summary>
149
+
150
+ ---
151
+
152
+ If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing
153
+ the `{...}` placeholders and customizing the path):
154
+
155
+ ```shell
156
+ # Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
157
+ curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
158
+ ```
159
+
160
+ For example:
161
+
162
+ ```shell
163
+ # Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
164
+ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
165
+ ```
166
+
167
+ ---
168
+ </details>
169
+
170
+ ### Updating
171
+
172
+ You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
173
+
174
+ ```shell
175
+ pipx upgrade refscan
176
+ ```
177
+
178
+ ### Uninstallation
179
+
180
+ You can uninstall the tool from your computer by running:
181
+
182
+ ```shell
183
+ pipx uninstall refscan
184
+ ```
185
+
186
+ ## Development
187
+
188
+ We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
189
+
190
+ - `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
191
+ - `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
192
+
193
+ ### Clone repository
194
+
195
+ ```shell
196
+ git clone https://github.com/microbiomedata/refscan.git
197
+ cd refscan
198
+ ```
199
+
200
+ ### Create virtual environment
201
+
202
+ Create a Poetry virtual environment and attach to its shell:
203
+
204
+ ```shell
205
+ poetry shell
206
+ ```
207
+
208
+ > You can see information about the Poetry virtual environment by running: `$ poetry env info`
209
+
210
+ > You can detach from the Poetry virtual environment's shell by running: `$ exit`
211
+
212
+ From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."
213
+
214
+ ### Install dependencies
215
+
216
+ At the Poetry shell, install the project's dependencies:
217
+
218
+ ```shell
219
+ poetry install
220
+ ```
221
+
222
+ ### Make changes
223
+
224
+ Edit the tool's source code and documentation however you want.
225
+
226
+ While editing the tool's source code, you can run the tool as you normally would in order to test things out.
227
+
228
+ ```shell
229
+ refscan --help
230
+ ```
231
+
232
+ ## Building and publishing
233
+
234
+ ### Build for production
235
+
236
+ Whenever someone publishes a [GitHub Release](https://github.com/microbiomedata/refscan/releases) in this repository,
237
+ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.yml)
238
+ will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
239
+ That package will have a version identifier that matches the name of the Git tag associated with the Release.
240
+
241
+ The GitHub Actions workflow will automatically
242
+
243
+ ### Test the build process locally
244
+
245
+ In case you want to test the build process locally, you can do so by running:
246
+
247
+ ```shell
248
+ poetry build
249
+ ```
250
+
251
+ > That will create both a
252
+ > [source distribution](https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#creating-a-source-distribution)
253
+ > file (whose name ends with `.tar.gz`) and a
254
+ > [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format)
255
+ > file (whose name ends with `.whl`) in the `dist` directory.
256
+
@@ -0,0 +1,236 @@
1
+ # refscan
2
+
3
+ `refscan` is a command-line tool people can use to scan the [NMDC](https://microbiomedata.org/) MongoDB database
4
+ for referential integrity violations.
5
+
6
+ ```mermaid
7
+ %% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
8
+ %% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
9
+ %% Reference: https://github.com/pypi/warehouse/issues/13083
10
+ graph LR
11
+ schema[LinkML<br>schema]
12
+ database[(MongoDB<br>database)]
13
+ script[["refscan.py"]]
14
+ violations["List of<br>violations"]
15
+ references["List of<br>references"]:::dashed_border
16
+ schema --> script
17
+ database --> script
18
+ script -.-> references
19
+ script --> violations
20
+
21
+ classDef dashed_border stroke-dasharray: 5 5
22
+ ```
23
+
24
+ ## How it works
25
+
26
+ `refscan` does its job in two stages:
27
+ 1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
28
+ that _can_ exist in a database that conforms to the schema.
29
+ > **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
30
+ > a field named `owner` that contains the `id` of a document in a collection named `persons`.
31
+ 2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
32
+ > **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
33
+ > field matches the `id` of some document in the collection named `persons`.
34
+
35
+ ## Limitations
36
+
37
+ `refscan` was designed under the assumption that **each document has a field named `type`,
38
+ whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
39
+ `refscan` relies on that `type` field when determining the name of the class of which a document represents an
40
+ instance; which it does in order to determine "which fields" of that document can contain a reference.
41
+
42
+ ## Usage
43
+
44
+ ### Installation
45
+
46
+ Assuming you have `pipx` installed, you can install the tool by running the following command:
47
+
48
+ ```shell
49
+ pipx install refscan
50
+ ```
51
+
52
+ > [`pipx`](https://pipx.pypa.io/stable/) is a tool people can use to
53
+ > [download and install](https://pipx.pypa.io/stable/#where-does-pipx-install-apps-from)
54
+ > Python scripts that are hosted on PyPI.
55
+ > You can [install `pipx`](https://pipx.pypa.io/stable/installation/) by running `$ python -m pip install pipx`.
56
+
57
+ ### Running
58
+
59
+ You can display the tool's `--help` snippet by running:
60
+
61
+ ```shell
62
+ refscan --help
63
+ ```
64
+
65
+ At the time of this writing, the tool's `--help` snippet is:
66
+
67
+ ```console
68
+ Usage: refscan [OPTIONS]
69
+
70
+ Scans the NMDC MongoDB database for referential integrity violations.
71
+
72
+ ╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
73
+ │ * --schema FILE Filesystem path at which the YAML file │
74
+ │ representing the schema is located. │
75
+ │ [default: None] │
76
+ │ [required] │
77
+ │ --database-name TEXT Name of the database. │
78
+ │ [default: nmdc] │
79
+ │ --mongo-uri TEXT Connection string for accessing the │
80
+ │ MongoDB server. If you have Docker │
81
+ │ installed, you can spin up a temporary │
82
+ │ MongoDB server at the default URI by │
83
+ │ running: $ docker run --rm --detach -p │
84
+ │ 27017:27017 mongo │
85
+ │ [env var: MONGO_URI] │
86
+ │ [default: mongodb://localhost:27017] │
87
+ │ --verbose Show verbose output. │
88
+ │ --skip-source-collection,--skip TEXT Name of collection you do not want to │
89
+ │ search for referring documents. Option │
90
+ │ can be used multiple times. │
91
+ │ [default: None] │
92
+ │ --reference-report FILE Filesystem path at which you want the │
93
+ │ program to generate its reference │
94
+ │ report. │
95
+ │ [default: references.tsv] │
96
+ │ --violation-report FILE Filesystem path at which you want the │
97
+ │ program to generate its violation │
98
+ │ report. │
99
+ │ [default: violations.tsv] │
100
+ │ --version Show version number and exit. │
101
+ │ --help Show this message and exit. │
102
+ ╰────────────────────────────────────────────────────────────────────────────────────────╯
103
+ ```
104
+
105
+ > Note: The above snippet was captured from a terminal window whose width was 90 characters.
106
+
107
+ #### The MongoDB connection string (`--mongo-uri`)
108
+
109
+ As documented in the `--help` snippet above, you can provide the MongoDB connection string to the tool via either
110
+ (a) the `--mongo-uri` option; or (b) an environment variable named `MONGO_URI`. The latter can come in handy
111
+ when the MongoDB connection string contains information you don't want to appear in your shell history,
112
+ such as a password.
113
+
114
+ Here's how you could create that environment variable:
115
+
116
+ ```shell
117
+ export MONGO_URI='mongodb://username:password@localhost:27017'
118
+ ```
119
+
120
+ #### The schema (`--schema`)
121
+
122
+ As documented in the `--help` snippet above, you can provide the path to a YAML-formatted LinkML schema file to the tool
123
+ via the `--schema` option.
124
+
125
+ <details>
126
+
127
+ <summary>
128
+ Show/hide tips for getting a schema file
129
+ </summary>
130
+
131
+ ---
132
+
133
+ If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing
134
+ the `{...}` placeholders and customizing the path):
135
+
136
+ ```shell
137
+ # Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
138
+ curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
139
+ ```
140
+
141
+ For example:
142
+
143
+ ```shell
144
+ # Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
145
+ curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
146
+ ```
147
+
148
+ ---
149
+ </details>
150
+
151
+ ### Updating
152
+
153
+ You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
154
+
155
+ ```shell
156
+ pipx upgrade refscan
157
+ ```
158
+
159
+ ### Uninstallation
160
+
161
+ You can uninstall the tool from your computer by running:
162
+
163
+ ```shell
164
+ pipx uninstall refscan
165
+ ```
166
+
167
+ ## Development
168
+
169
+ We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
170
+
171
+ - `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
172
+ - `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
173
+
174
+ ### Clone repository
175
+
176
+ ```shell
177
+ git clone https://github.com/microbiomedata/refscan.git
178
+ cd refscan
179
+ ```
180
+
181
+ ### Create virtual environment
182
+
183
+ Create a Poetry virtual environment and attach to its shell:
184
+
185
+ ```shell
186
+ poetry shell
187
+ ```
188
+
189
+ > You can see information about the Poetry virtual environment by running: `$ poetry env info`
190
+
191
+ > You can detach from the Poetry virtual environment's shell by running: `$ exit`
192
+
193
+ From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."
194
+
195
+ ### Install dependencies
196
+
197
+ At the Poetry shell, install the project's dependencies:
198
+
199
+ ```shell
200
+ poetry install
201
+ ```
202
+
203
+ ### Make changes
204
+
205
+ Edit the tool's source code and documentation however you want.
206
+
207
+ While editing the tool's source code, you can run the tool as you normally would in order to test things out.
208
+
209
+ ```shell
210
+ refscan --help
211
+ ```
212
+
213
+ ## Building and publishing
214
+
215
+ ### Build for production
216
+
217
+ Whenever someone publishes a [GitHub Release](https://github.com/microbiomedata/refscan/releases) in this repository,
218
+ a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.yml)
219
+ will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
220
+ That package will have a version identifier that matches the name of the Git tag associated with the Release.
221
+
222
+ The GitHub Actions workflow will automatically
223
+
224
+ ### Test the build process locally
225
+
226
+ In case you want to test the build process locally, you can do so by running:
227
+
228
+ ```shell
229
+ poetry build
230
+ ```
231
+
232
+ > That will create both a
233
+ > [source distribution](https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#creating-a-source-distribution)
234
+ > file (whose name ends with `.tar.gz`) and a
235
+ > [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format)
236
+ > file (whose name ends with `.whl`) in the `dist` directory.
@@ -2,7 +2,7 @@
2
2
 
3
3
  [tool.poetry]
4
4
  name = "refscan"
5
- version = "v0.1.0"
5
+ version = "v0.1.2"
6
6
  description = "Command-line program that scans the NMDC MongoDB database for referential integrity violations"
7
7
  authors = []
8
8
  readme = "README.md"
@@ -0,0 +1,18 @@
1
+ from importlib.metadata import metadata, PackageMetadata, PackageNotFoundError
2
+
3
+
4
+ def get_package_metadata(key: str) -> str:
5
+ r"""
6
+ Returns metadata about the installed package.
7
+
8
+ References:
9
+ - https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata
10
+ - https://github.com/python-poetry/poetry/issues/273#issuecomment-570999678
11
+ """
12
+ metadata_value = ""
13
+ try:
14
+ package_metadata: PackageMetadata = metadata("refscan")
15
+ metadata_value = package_metadata.get(key)
16
+ except PackageNotFoundError:
17
+ pass
18
+ return metadata_value
@@ -18,6 +18,7 @@ from refscan.lib.Reference import Reference
18
18
  from refscan.lib.ReferenceList import ReferenceList
19
19
  from refscan.lib.Violation import Violation
20
20
  from refscan.lib.ViolationList import ViolationList
21
+ from refscan import get_package_metadata
21
22
 
22
23
  app = typer.Typer(
23
24
  help="Scans the NMDC MongoDB database for referential integrity violations.",
@@ -25,6 +26,22 @@ app = typer.Typer(
25
26
  rich_markup_mode="markdown", # enables use of Markdown in docstrings and CLI help
26
27
  )
27
28
 
29
+ app_version = get_package_metadata("Version")
30
+
31
+
32
+ def display_app_version_and_exit(is_active: bool = False) -> None:
33
+ r"""
34
+ Displays the app's version number, then exists.
35
+
36
+ Note: The `is_active` flag will be `True` if the program was
37
+ invoked with the associated CLI option.
38
+
39
+ Reference: https://typer.tiangolo.com/tutorial/options/version/
40
+ """
41
+ if is_active:
42
+ console.print(f"[white]{app_version}[/white]")
43
+ raise typer.Exit()
44
+
28
45
 
29
46
  @app.command("scan")
30
47
  def scan(
@@ -47,6 +64,7 @@ def scan(
47
64
  "`$ docker run --rm --detach -p 27017:27017 mongo`",
48
65
  )] = "mongodb://localhost:27017",
49
66
  verbose: Annotated[bool, typer.Option(
67
+ "--verbose",
50
68
  help="Show verbose output.",
51
69
  )] = False,
52
70
  # Reference: https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/
@@ -71,6 +89,12 @@ def scan(
71
89
  resolve_path=True,
72
90
  help="Filesystem path at which you want the program to generate its violation report.",
73
91
  )] = "violations.tsv",
92
+ version: Annotated[Optional[bool], typer.Option(
93
+ "--version",
94
+ callback=display_app_version_and_exit,
95
+ is_eager=True, # tells Typer to process this option first
96
+ help="Show version number and exit.",
97
+ )] = None,
74
98
  ):
75
99
  """
76
100
  Scans the NMDC MongoDB database for referential integrity violations.
@@ -80,9 +104,11 @@ def scan(
80
104
  console.print(f"Schema YAML file: {schema_file_path}")
81
105
  schema_view = linkml_runtime.SchemaView(schema_file_path)
82
106
 
83
- # Show high-level information about the schema and LinkML runtime.
84
- console.print(f"Schema version: {schema_view.schema.version}")
85
- console.print(f"LinkML runtime version: {linkml_runtime.__version__}")
107
+ # Show high-level information about the application and schema.
108
+ console.print(f"refscan version: {app_version}")
109
+ console.print(f"Schema version: {schema_view.schema.version}")
110
+
111
+ console.rule("[bold]Identifying references")
86
112
 
87
113
  # Make a more self-documenting alias for the CLI option that can be specified multiple times.
88
114
  names_of_source_collections_to_skip: list[str] = [] if skip_source_collection is None else skip_source_collection
@@ -158,6 +184,8 @@ def scan(
158
184
  if verbose:
159
185
  console.print(references.as_table())
160
186
 
187
+ console.rule("[bold]Scanning for violations")
188
+
161
189
  # Get a dictionary that maps source class names to the names of their fields that can contain references.
162
190
  reference_field_names_by_source_class_name = references.get_reference_field_names_by_source_class_name()
163
191
 
@@ -291,6 +319,8 @@ def scan(
291
319
  # Close the connection to the MongoDB server.
292
320
  mongo_client.close()
293
321
 
322
+ console.rule("[bold]Results summary")
323
+
294
324
  # Create a violation report in TSV format — for all collections combined.
295
325
  all_violations = ViolationList()
296
326
  for collection_name, violations in sorted(source_collections_and_their_violations.items(), key=get_lowercase_key):
refscan-0.1.0/PKG-INFO DELETED
@@ -1,68 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: refscan
3
- Version: 0.1.0
4
- Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
5
- Home-page: https://github.com/microbiomedata/refscan
6
- Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
7
- Requires-Python: >=3.10,<4.0
8
- Classifier: Programming Language :: Python :: 3
9
- Classifier: Programming Language :: Python :: 3.10
10
- Classifier: Programming Language :: Python :: 3.11
11
- Classifier: Programming Language :: Python :: 3.12
12
- Requires-Dist: linkml-runtime (>=1.7.5,<2.0.0)
13
- Requires-Dist: pymongo (>=4.7.1,<5.0.0)
14
- Requires-Dist: rich (>=13.7.1,<14.0.0)
15
- Requires-Dist: typer (>=0.12.3,<0.13.0)
16
- Project-URL: Documentation, https://github.com/microbiomedata/refscan
17
- Project-URL: Repository, https://github.com/microbiomedata/refscan
18
- Description-Content-Type: text/markdown
19
-
20
- # refscan
21
-
22
- `refscan` is a command-line tool people can use to scan the NMDC MongoDB database
23
- for referential integrity violations.
24
-
25
- ```mermaid
26
- %% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
27
- %% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
28
- %% Reference: https://github.com/pypi/warehouse/issues/13083
29
- graph LR
30
- schema[LinkML<br>schema]
31
- database[(MongoDB<br>database)]
32
- script[["refscan.py"]]
33
- violations["List of<br>violations"]
34
- references["List of<br>references"]:::dashed_border
35
- schema --> script
36
- database --> script
37
- script -.-> references
38
- script --> violations
39
-
40
- classDef dashed_border stroke-dasharray: 5 5
41
- ```
42
-
43
- ## Assumptions
44
-
45
- `refscan` was designed under some assumptions about the schema and database, including:
46
-
47
- 1. Each source document (i.e. document containing references) has a field named `type`, whose value (a string) is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance. For example, the `type` field of each document in the `study_set` collection has the value `"nmdc:Study"`.
48
-
49
- ## Development status
50
-
51
- `refscan` is in early development and its author does not recommend anyone use it for anything without reviewing its code first.
52
-
53
- ## Tips
54
-
55
- `refscan` requires the user to specify the path to a schema in YAML format. If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing the `{...}` placeholders and customizing the path):
56
-
57
- ```shell
58
- # Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
59
- curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
60
- ```
61
-
62
- For example:
63
-
64
- ```shell
65
- # Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
66
- curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
67
- ```
68
-
refscan-0.1.0/README.md DELETED
@@ -1,48 +0,0 @@
1
- # refscan
2
-
3
- `refscan` is a command-line tool people can use to scan the NMDC MongoDB database
4
- for referential integrity violations.
5
-
6
- ```mermaid
7
- %% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
8
- %% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
9
- %% Reference: https://github.com/pypi/warehouse/issues/13083
10
- graph LR
11
- schema[LinkML<br>schema]
12
- database[(MongoDB<br>database)]
13
- script[["refscan.py"]]
14
- violations["List of<br>violations"]
15
- references["List of<br>references"]:::dashed_border
16
- schema --> script
17
- database --> script
18
- script -.-> references
19
- script --> violations
20
-
21
- classDef dashed_border stroke-dasharray: 5 5
22
- ```
23
-
24
- ## Assumptions
25
-
26
- `refscan` was designed under some assumptions about the schema and database, including:
27
-
28
- 1. Each source document (i.e. document containing references) has a field named `type`, whose value (a string) is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance. For example, the `type` field of each document in the `study_set` collection has the value `"nmdc:Study"`.
29
-
30
- ## Development status
31
-
32
- `refscan` is in early development and its author does not recommend anyone use it for anything without reviewing its code first.
33
-
34
- ## Tips
35
-
36
- `refscan` requires the user to specify the path to a schema in YAML format. If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing the `{...}` placeholders and customizing the path):
37
-
38
- ```shell
39
- # Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
40
- curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
41
- ```
42
-
43
- For example:
44
-
45
- ```shell
46
- # Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
47
- curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
48
- ```
File without changes
File without changes
File without changes