refscan 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- refscan-0.1.2/PKG-INFO +256 -0
- refscan-0.1.2/README.md +236 -0
- {refscan-0.1.0 → refscan-0.1.2}/pyproject.toml +1 -1
- refscan-0.1.2/refscan/__init__.py +18 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/refscan.py +33 -3
- refscan-0.1.0/PKG-INFO +0 -68
- refscan-0.1.0/README.md +0 -48
- refscan-0.1.0/refscan/__init__.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/Finder.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/Reference.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/ReferenceList.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/Violation.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/ViolationList.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/constants.py +0 -0
- {refscan-0.1.0 → refscan-0.1.2}/refscan/lib/helpers.py +0 -0
refscan-0.1.2/PKG-INFO
ADDED
|
@@ -0,0 +1,256 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: refscan
|
|
3
|
+
Version: 0.1.2
|
|
4
|
+
Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
|
|
5
|
+
Home-page: https://github.com/microbiomedata/refscan
|
|
6
|
+
Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
|
|
7
|
+
Requires-Python: >=3.10,<4.0
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
+
Requires-Dist: linkml-runtime (>=1.7.5,<2.0.0)
|
|
13
|
+
Requires-Dist: pymongo (>=4.7.1,<5.0.0)
|
|
14
|
+
Requires-Dist: rich (>=13.7.1,<14.0.0)
|
|
15
|
+
Requires-Dist: typer (>=0.12.3,<0.13.0)
|
|
16
|
+
Project-URL: Documentation, https://github.com/microbiomedata/refscan
|
|
17
|
+
Project-URL: Repository, https://github.com/microbiomedata/refscan
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
|
|
20
|
+
# refscan
|
|
21
|
+
|
|
22
|
+
`refscan` is a command-line tool people can use to scan the [NMDC](https://microbiomedata.org/) MongoDB database
|
|
23
|
+
for referential integrity violations.
|
|
24
|
+
|
|
25
|
+
```mermaid
|
|
26
|
+
%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
|
|
27
|
+
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
|
|
28
|
+
%% Reference: https://github.com/pypi/warehouse/issues/13083
|
|
29
|
+
graph LR
|
|
30
|
+
schema[LinkML<br>schema]
|
|
31
|
+
database[(MongoDB<br>database)]
|
|
32
|
+
script[["refscan.py"]]
|
|
33
|
+
violations["List of<br>violations"]
|
|
34
|
+
references["List of<br>references"]:::dashed_border
|
|
35
|
+
schema --> script
|
|
36
|
+
database --> script
|
|
37
|
+
script -.-> references
|
|
38
|
+
script --> violations
|
|
39
|
+
|
|
40
|
+
classDef dashed_border stroke-dasharray: 5 5
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## How it works
|
|
44
|
+
|
|
45
|
+
`refscan` does its job in two stages:
|
|
46
|
+
1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
|
|
47
|
+
that _can_ exist in a database that conforms to the schema.
|
|
48
|
+
> **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
|
|
49
|
+
> a field named `owner` that contains the `id` of a document in a collection named `persons`.
|
|
50
|
+
2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
|
|
51
|
+
> **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
|
|
52
|
+
> field matches the `id` of some document in the collection named `persons`.
|
|
53
|
+
|
|
54
|
+
## Limitations
|
|
55
|
+
|
|
56
|
+
`refscan` was designed under the assumption that **each document has a field named `type`,
|
|
57
|
+
whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
|
|
58
|
+
`refscan` relies on that `type` field when determining the name of the class of which a document represents an
|
|
59
|
+
instance; which it does in order to determine "which fields" of that document can contain a reference.
|
|
60
|
+
|
|
61
|
+
## Usage
|
|
62
|
+
|
|
63
|
+
### Installation
|
|
64
|
+
|
|
65
|
+
Assuming you have `pipx` installed, you can install the tool by running the following command:
|
|
66
|
+
|
|
67
|
+
```shell
|
|
68
|
+
pipx install refscan
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
> [`pipx`](https://pipx.pypa.io/stable/) is a tool people can use to
|
|
72
|
+
> [download and install](https://pipx.pypa.io/stable/#where-does-pipx-install-apps-from)
|
|
73
|
+
> Python scripts that are hosted on PyPI.
|
|
74
|
+
> You can [install `pipx`](https://pipx.pypa.io/stable/installation/) by running `$ python -m pip install pipx`.
|
|
75
|
+
|
|
76
|
+
### Running
|
|
77
|
+
|
|
78
|
+
You can display the tool's `--help` snippet by running:
|
|
79
|
+
|
|
80
|
+
```shell
|
|
81
|
+
refscan --help
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
At the time of this writing, the tool's `--help` snippet is:
|
|
85
|
+
|
|
86
|
+
```console
|
|
87
|
+
Usage: refscan [OPTIONS]
|
|
88
|
+
|
|
89
|
+
Scans the NMDC MongoDB database for referential integrity violations.
|
|
90
|
+
|
|
91
|
+
╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
|
|
92
|
+
│ * --schema FILE Filesystem path at which the YAML file │
|
|
93
|
+
│ representing the schema is located. │
|
|
94
|
+
│ [default: None] │
|
|
95
|
+
│ [required] │
|
|
96
|
+
│ --database-name TEXT Name of the database. │
|
|
97
|
+
│ [default: nmdc] │
|
|
98
|
+
│ --mongo-uri TEXT Connection string for accessing the │
|
|
99
|
+
│ MongoDB server. If you have Docker │
|
|
100
|
+
│ installed, you can spin up a temporary │
|
|
101
|
+
│ MongoDB server at the default URI by │
|
|
102
|
+
│ running: $ docker run --rm --detach -p │
|
|
103
|
+
│ 27017:27017 mongo │
|
|
104
|
+
│ [env var: MONGO_URI] │
|
|
105
|
+
│ [default: mongodb://localhost:27017] │
|
|
106
|
+
│ --verbose Show verbose output. │
|
|
107
|
+
│ --skip-source-collection,--skip TEXT Name of collection you do not want to │
|
|
108
|
+
│ search for referring documents. Option │
|
|
109
|
+
│ can be used multiple times. │
|
|
110
|
+
│ [default: None] │
|
|
111
|
+
│ --reference-report FILE Filesystem path at which you want the │
|
|
112
|
+
│ program to generate its reference │
|
|
113
|
+
│ report. │
|
|
114
|
+
│ [default: references.tsv] │
|
|
115
|
+
│ --violation-report FILE Filesystem path at which you want the │
|
|
116
|
+
│ program to generate its violation │
|
|
117
|
+
│ report. │
|
|
118
|
+
│ [default: violations.tsv] │
|
|
119
|
+
│ --version Show version number and exit. │
|
|
120
|
+
│ --help Show this message and exit. │
|
|
121
|
+
╰────────────────────────────────────────────────────────────────────────────────────────╯
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
> Note: The above snippet was captured from a terminal window whose width was 90 characters.
|
|
125
|
+
|
|
126
|
+
#### The MongoDB connection string (`--mongo-uri`)
|
|
127
|
+
|
|
128
|
+
As documented in the `--help` snippet above, you can provide the MongoDB connection string to the tool via either
|
|
129
|
+
(a) the `--mongo-uri` option; or (b) an environment variable named `MONGO_URI`. The latter can come in handy
|
|
130
|
+
when the MongoDB connection string contains information you don't want to appear in your shell history,
|
|
131
|
+
such as a password.
|
|
132
|
+
|
|
133
|
+
Here's how you could create that environment variable:
|
|
134
|
+
|
|
135
|
+
```shell
|
|
136
|
+
export MONGO_URI='mongodb://username:password@localhost:27017'
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
#### The schema (`--schema`)
|
|
140
|
+
|
|
141
|
+
As documented in the `--help` snippet above, you can provide the path to a YAML-formatted LinkML schema file to the tool
|
|
142
|
+
via the `--schema` option.
|
|
143
|
+
|
|
144
|
+
<details>
|
|
145
|
+
|
|
146
|
+
<summary>
|
|
147
|
+
Show/hide tips for getting a schema file
|
|
148
|
+
</summary>
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing
|
|
153
|
+
the `{...}` placeholders and customizing the path):
|
|
154
|
+
|
|
155
|
+
```shell
|
|
156
|
+
# Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
|
|
157
|
+
curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
For example:
|
|
161
|
+
|
|
162
|
+
```shell
|
|
163
|
+
# Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
164
|
+
curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
</details>
|
|
169
|
+
|
|
170
|
+
### Updating
|
|
171
|
+
|
|
172
|
+
You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
|
|
173
|
+
|
|
174
|
+
```shell
|
|
175
|
+
pipx upgrade refscan
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Uninstallation
|
|
179
|
+
|
|
180
|
+
You can uninstall the tool from your computer by running:
|
|
181
|
+
|
|
182
|
+
```shell
|
|
183
|
+
pipx uninstall refscan
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## Development
|
|
187
|
+
|
|
188
|
+
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
|
|
189
|
+
|
|
190
|
+
- `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
|
|
191
|
+
- `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
|
|
192
|
+
|
|
193
|
+
### Clone repository
|
|
194
|
+
|
|
195
|
+
```shell
|
|
196
|
+
git clone https://github.com/microbiomedata/refscan.git
|
|
197
|
+
cd refscan
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
### Create virtual environment
|
|
201
|
+
|
|
202
|
+
Create a Poetry virtual environment and attach to its shell:
|
|
203
|
+
|
|
204
|
+
```shell
|
|
205
|
+
poetry shell
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
> You can see information about the Poetry virtual environment by running: `$ poetry env info`
|
|
209
|
+
|
|
210
|
+
> You can detach from the Poetry virtual environment's shell by running: `$ exit`
|
|
211
|
+
|
|
212
|
+
From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."
|
|
213
|
+
|
|
214
|
+
### Install dependencies
|
|
215
|
+
|
|
216
|
+
At the Poetry shell, install the project's dependencies:
|
|
217
|
+
|
|
218
|
+
```shell
|
|
219
|
+
poetry install
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Make changes
|
|
223
|
+
|
|
224
|
+
Edit the tool's source code and documentation however you want.
|
|
225
|
+
|
|
226
|
+
While editing the tool's source code, you can run the tool as you normally would in order to test things out.
|
|
227
|
+
|
|
228
|
+
```shell
|
|
229
|
+
refscan --help
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
## Building and publishing
|
|
233
|
+
|
|
234
|
+
### Build for production
|
|
235
|
+
|
|
236
|
+
Whenever someone publishes a [GitHub Release](https://github.com/microbiomedata/refscan/releases) in this repository,
|
|
237
|
+
a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.yml)
|
|
238
|
+
will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
|
|
239
|
+
That package will have a version identifier that matches the name of the Git tag associated with the Release.
|
|
240
|
+
|
|
241
|
+
The GitHub Actions workflow will automatically
|
|
242
|
+
|
|
243
|
+
### Test the build process locally
|
|
244
|
+
|
|
245
|
+
In case you want to test the build process locally, you can do so by running:
|
|
246
|
+
|
|
247
|
+
```shell
|
|
248
|
+
poetry build
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
> That will create both a
|
|
252
|
+
> [source distribution](https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#creating-a-source-distribution)
|
|
253
|
+
> file (whose name ends with `.tar.gz`) and a
|
|
254
|
+
> [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format)
|
|
255
|
+
> file (whose name ends with `.whl`) in the `dist` directory.
|
|
256
|
+
|
refscan-0.1.2/README.md
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
# refscan
|
|
2
|
+
|
|
3
|
+
`refscan` is a command-line tool people can use to scan the [NMDC](https://microbiomedata.org/) MongoDB database
|
|
4
|
+
for referential integrity violations.
|
|
5
|
+
|
|
6
|
+
```mermaid
|
|
7
|
+
%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
|
|
8
|
+
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
|
|
9
|
+
%% Reference: https://github.com/pypi/warehouse/issues/13083
|
|
10
|
+
graph LR
|
|
11
|
+
schema[LinkML<br>schema]
|
|
12
|
+
database[(MongoDB<br>database)]
|
|
13
|
+
script[["refscan.py"]]
|
|
14
|
+
violations["List of<br>violations"]
|
|
15
|
+
references["List of<br>references"]:::dashed_border
|
|
16
|
+
schema --> script
|
|
17
|
+
database --> script
|
|
18
|
+
script -.-> references
|
|
19
|
+
script --> violations
|
|
20
|
+
|
|
21
|
+
classDef dashed_border stroke-dasharray: 5 5
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## How it works
|
|
25
|
+
|
|
26
|
+
`refscan` does its job in two stages:
|
|
27
|
+
1. It uses the LinkML schema to determine "what to scan;" i.e. all of the document-to-document references
|
|
28
|
+
that _can_ exist in a database that conforms to the schema.
|
|
29
|
+
> **Example:** The schema might say that, if a document exists in a collection named `cars`, that document must have
|
|
30
|
+
> a field named `owner` that contains the `id` of a document in a collection named `persons`.
|
|
31
|
+
2. It scans the MongoDB database to check the integrity of all of the references that _do_ exist.
|
|
32
|
+
> **Example:** For each document in the collection named `cars`, `refscan` checks whether the value in its `owner`
|
|
33
|
+
> field matches the `id` of some document in the collection named `persons`.
|
|
34
|
+
|
|
35
|
+
## Limitations
|
|
36
|
+
|
|
37
|
+
`refscan` was designed under the assumption that **each document has a field named `type`,
|
|
38
|
+
whose value is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance**.
|
|
39
|
+
`refscan` relies on that `type` field when determining the name of the class of which a document represents an
|
|
40
|
+
instance; which it does in order to determine "which fields" of that document can contain a reference.
|
|
41
|
+
|
|
42
|
+
## Usage
|
|
43
|
+
|
|
44
|
+
### Installation
|
|
45
|
+
|
|
46
|
+
Assuming you have `pipx` installed, you can install the tool by running the following command:
|
|
47
|
+
|
|
48
|
+
```shell
|
|
49
|
+
pipx install refscan
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
> [`pipx`](https://pipx.pypa.io/stable/) is a tool people can use to
|
|
53
|
+
> [download and install](https://pipx.pypa.io/stable/#where-does-pipx-install-apps-from)
|
|
54
|
+
> Python scripts that are hosted on PyPI.
|
|
55
|
+
> You can [install `pipx`](https://pipx.pypa.io/stable/installation/) by running `$ python -m pip install pipx`.
|
|
56
|
+
|
|
57
|
+
### Running
|
|
58
|
+
|
|
59
|
+
You can display the tool's `--help` snippet by running:
|
|
60
|
+
|
|
61
|
+
```shell
|
|
62
|
+
refscan --help
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
At the time of this writing, the tool's `--help` snippet is:
|
|
66
|
+
|
|
67
|
+
```console
|
|
68
|
+
Usage: refscan [OPTIONS]
|
|
69
|
+
|
|
70
|
+
Scans the NMDC MongoDB database for referential integrity violations.
|
|
71
|
+
|
|
72
|
+
╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
|
|
73
|
+
│ * --schema FILE Filesystem path at which the YAML file │
|
|
74
|
+
│ representing the schema is located. │
|
|
75
|
+
│ [default: None] │
|
|
76
|
+
│ [required] │
|
|
77
|
+
│ --database-name TEXT Name of the database. │
|
|
78
|
+
│ [default: nmdc] │
|
|
79
|
+
│ --mongo-uri TEXT Connection string for accessing the │
|
|
80
|
+
│ MongoDB server. If you have Docker │
|
|
81
|
+
│ installed, you can spin up a temporary │
|
|
82
|
+
│ MongoDB server at the default URI by │
|
|
83
|
+
│ running: $ docker run --rm --detach -p │
|
|
84
|
+
│ 27017:27017 mongo │
|
|
85
|
+
│ [env var: MONGO_URI] │
|
|
86
|
+
│ [default: mongodb://localhost:27017] │
|
|
87
|
+
│ --verbose Show verbose output. │
|
|
88
|
+
│ --skip-source-collection,--skip TEXT Name of collection you do not want to │
|
|
89
|
+
│ search for referring documents. Option │
|
|
90
|
+
│ can be used multiple times. │
|
|
91
|
+
│ [default: None] │
|
|
92
|
+
│ --reference-report FILE Filesystem path at which you want the │
|
|
93
|
+
│ program to generate its reference │
|
|
94
|
+
│ report. │
|
|
95
|
+
│ [default: references.tsv] │
|
|
96
|
+
│ --violation-report FILE Filesystem path at which you want the │
|
|
97
|
+
│ program to generate its violation │
|
|
98
|
+
│ report. │
|
|
99
|
+
│ [default: violations.tsv] │
|
|
100
|
+
│ --version Show version number and exit. │
|
|
101
|
+
│ --help Show this message and exit. │
|
|
102
|
+
╰────────────────────────────────────────────────────────────────────────────────────────╯
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
> Note: The above snippet was captured from a terminal window whose width was 90 characters.
|
|
106
|
+
|
|
107
|
+
#### The MongoDB connection string (`--mongo-uri`)
|
|
108
|
+
|
|
109
|
+
As documented in the `--help` snippet above, you can provide the MongoDB connection string to the tool via either
|
|
110
|
+
(a) the `--mongo-uri` option; or (b) an environment variable named `MONGO_URI`. The latter can come in handy
|
|
111
|
+
when the MongoDB connection string contains information you don't want to appear in your shell history,
|
|
112
|
+
such as a password.
|
|
113
|
+
|
|
114
|
+
Here's how you could create that environment variable:
|
|
115
|
+
|
|
116
|
+
```shell
|
|
117
|
+
export MONGO_URI='mongodb://username:password@localhost:27017'
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
#### The schema (`--schema`)
|
|
121
|
+
|
|
122
|
+
As documented in the `--help` snippet above, you can provide the path to a YAML-formatted LinkML schema file to the tool
|
|
123
|
+
via the `--schema` option.
|
|
124
|
+
|
|
125
|
+
<details>
|
|
126
|
+
|
|
127
|
+
<summary>
|
|
128
|
+
Show/hide tips for getting a schema file
|
|
129
|
+
</summary>
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing
|
|
134
|
+
the `{...}` placeholders and customizing the path):
|
|
135
|
+
|
|
136
|
+
```shell
|
|
137
|
+
# Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
|
|
138
|
+
curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
For example:
|
|
142
|
+
|
|
143
|
+
```shell
|
|
144
|
+
# Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
145
|
+
curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
</details>
|
|
150
|
+
|
|
151
|
+
### Updating
|
|
152
|
+
|
|
153
|
+
You can update the tool to [the latest version available on PyPI](https://pypi.org/project/refscan/) by running:
|
|
154
|
+
|
|
155
|
+
```shell
|
|
156
|
+
pipx upgrade refscan
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
### Uninstallation
|
|
160
|
+
|
|
161
|
+
You can uninstall the tool from your computer by running:
|
|
162
|
+
|
|
163
|
+
```shell
|
|
164
|
+
pipx uninstall refscan
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
## Development
|
|
168
|
+
|
|
169
|
+
We use [Poetry](https://python-poetry.org/) to both (a) manage dependencies and (b) publish packages to PyPI.
|
|
170
|
+
|
|
171
|
+
- `pyproject.toml`: Configuration file for Poetry and other tools (was generated via `$ poetry init`)
|
|
172
|
+
- `poetry.lock`: List of dependencies, direct and indirect (was generated via `$ poetry update`)
|
|
173
|
+
|
|
174
|
+
### Clone repository
|
|
175
|
+
|
|
176
|
+
```shell
|
|
177
|
+
git clone https://github.com/microbiomedata/refscan.git
|
|
178
|
+
cd refscan
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
### Create virtual environment
|
|
182
|
+
|
|
183
|
+
Create a Poetry virtual environment and attach to its shell:
|
|
184
|
+
|
|
185
|
+
```shell
|
|
186
|
+
poetry shell
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
> You can see information about the Poetry virtual environment by running: `$ poetry env info`
|
|
190
|
+
|
|
191
|
+
> You can detach from the Poetry virtual environment's shell by running: `$ exit`
|
|
192
|
+
|
|
193
|
+
From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."
|
|
194
|
+
|
|
195
|
+
### Install dependencies
|
|
196
|
+
|
|
197
|
+
At the Poetry shell, install the project's dependencies:
|
|
198
|
+
|
|
199
|
+
```shell
|
|
200
|
+
poetry install
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
### Make changes
|
|
204
|
+
|
|
205
|
+
Edit the tool's source code and documentation however you want.
|
|
206
|
+
|
|
207
|
+
While editing the tool's source code, you can run the tool as you normally would in order to test things out.
|
|
208
|
+
|
|
209
|
+
```shell
|
|
210
|
+
refscan --help
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
## Building and publishing
|
|
214
|
+
|
|
215
|
+
### Build for production
|
|
216
|
+
|
|
217
|
+
Whenever someone publishes a [GitHub Release](https://github.com/microbiomedata/refscan/releases) in this repository,
|
|
218
|
+
a [GitHub Actions workflow](.github/workflows/build-and-publish-package-to-pypi.yml)
|
|
219
|
+
will automatically build a package and publish it to [PyPI](https://pypi.org/project/refscan/).
|
|
220
|
+
That package will have a version identifier that matches the name of the Git tag associated with the Release.
|
|
221
|
+
|
|
222
|
+
The GitHub Actions workflow will automatically
|
|
223
|
+
|
|
224
|
+
### Test the build process locally
|
|
225
|
+
|
|
226
|
+
In case you want to test the build process locally, you can do so by running:
|
|
227
|
+
|
|
228
|
+
```shell
|
|
229
|
+
poetry build
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
> That will create both a
|
|
233
|
+
> [source distribution](https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#creating-a-source-distribution)
|
|
234
|
+
> file (whose name ends with `.tar.gz`) and a
|
|
235
|
+
> [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format)
|
|
236
|
+
> file (whose name ends with `.whl`) in the `dist` directory.
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
from importlib.metadata import metadata, PackageMetadata, PackageNotFoundError
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
def get_package_metadata(key: str) -> str:
|
|
5
|
+
r"""
|
|
6
|
+
Returns metadata about the installed package.
|
|
7
|
+
|
|
8
|
+
References:
|
|
9
|
+
- https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata
|
|
10
|
+
- https://github.com/python-poetry/poetry/issues/273#issuecomment-570999678
|
|
11
|
+
"""
|
|
12
|
+
metadata_value = ""
|
|
13
|
+
try:
|
|
14
|
+
package_metadata: PackageMetadata = metadata("refscan")
|
|
15
|
+
metadata_value = package_metadata.get(key)
|
|
16
|
+
except PackageNotFoundError:
|
|
17
|
+
pass
|
|
18
|
+
return metadata_value
|
|
@@ -18,6 +18,7 @@ from refscan.lib.Reference import Reference
|
|
|
18
18
|
from refscan.lib.ReferenceList import ReferenceList
|
|
19
19
|
from refscan.lib.Violation import Violation
|
|
20
20
|
from refscan.lib.ViolationList import ViolationList
|
|
21
|
+
from refscan import get_package_metadata
|
|
21
22
|
|
|
22
23
|
app = typer.Typer(
|
|
23
24
|
help="Scans the NMDC MongoDB database for referential integrity violations.",
|
|
@@ -25,6 +26,22 @@ app = typer.Typer(
|
|
|
25
26
|
rich_markup_mode="markdown", # enables use of Markdown in docstrings and CLI help
|
|
26
27
|
)
|
|
27
28
|
|
|
29
|
+
app_version = get_package_metadata("Version")
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
def display_app_version_and_exit(is_active: bool = False) -> None:
|
|
33
|
+
r"""
|
|
34
|
+
Displays the app's version number, then exists.
|
|
35
|
+
|
|
36
|
+
Note: The `is_active` flag will be `True` if the program was
|
|
37
|
+
invoked with the associated CLI option.
|
|
38
|
+
|
|
39
|
+
Reference: https://typer.tiangolo.com/tutorial/options/version/
|
|
40
|
+
"""
|
|
41
|
+
if is_active:
|
|
42
|
+
console.print(f"[white]{app_version}[/white]")
|
|
43
|
+
raise typer.Exit()
|
|
44
|
+
|
|
28
45
|
|
|
29
46
|
@app.command("scan")
|
|
30
47
|
def scan(
|
|
@@ -47,6 +64,7 @@ def scan(
|
|
|
47
64
|
"`$ docker run --rm --detach -p 27017:27017 mongo`",
|
|
48
65
|
)] = "mongodb://localhost:27017",
|
|
49
66
|
verbose: Annotated[bool, typer.Option(
|
|
67
|
+
"--verbose",
|
|
50
68
|
help="Show verbose output.",
|
|
51
69
|
)] = False,
|
|
52
70
|
# Reference: https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/
|
|
@@ -71,6 +89,12 @@ def scan(
|
|
|
71
89
|
resolve_path=True,
|
|
72
90
|
help="Filesystem path at which you want the program to generate its violation report.",
|
|
73
91
|
)] = "violations.tsv",
|
|
92
|
+
version: Annotated[Optional[bool], typer.Option(
|
|
93
|
+
"--version",
|
|
94
|
+
callback=display_app_version_and_exit,
|
|
95
|
+
is_eager=True, # tells Typer to process this option first
|
|
96
|
+
help="Show version number and exit.",
|
|
97
|
+
)] = None,
|
|
74
98
|
):
|
|
75
99
|
"""
|
|
76
100
|
Scans the NMDC MongoDB database for referential integrity violations.
|
|
@@ -80,9 +104,11 @@ def scan(
|
|
|
80
104
|
console.print(f"Schema YAML file: {schema_file_path}")
|
|
81
105
|
schema_view = linkml_runtime.SchemaView(schema_file_path)
|
|
82
106
|
|
|
83
|
-
# Show high-level information about the
|
|
84
|
-
console.print(f"
|
|
85
|
-
console.print(f"
|
|
107
|
+
# Show high-level information about the application and schema.
|
|
108
|
+
console.print(f"refscan version: {app_version}")
|
|
109
|
+
console.print(f"Schema version: {schema_view.schema.version}")
|
|
110
|
+
|
|
111
|
+
console.rule("[bold]Identifying references")
|
|
86
112
|
|
|
87
113
|
# Make a more self-documenting alias for the CLI option that can be specified multiple times.
|
|
88
114
|
names_of_source_collections_to_skip: list[str] = [] if skip_source_collection is None else skip_source_collection
|
|
@@ -158,6 +184,8 @@ def scan(
|
|
|
158
184
|
if verbose:
|
|
159
185
|
console.print(references.as_table())
|
|
160
186
|
|
|
187
|
+
console.rule("[bold]Scanning for violations")
|
|
188
|
+
|
|
161
189
|
# Get a dictionary that maps source class names to the names of their fields that can contain references.
|
|
162
190
|
reference_field_names_by_source_class_name = references.get_reference_field_names_by_source_class_name()
|
|
163
191
|
|
|
@@ -291,6 +319,8 @@ def scan(
|
|
|
291
319
|
# Close the connection to the MongoDB server.
|
|
292
320
|
mongo_client.close()
|
|
293
321
|
|
|
322
|
+
console.rule("[bold]Results summary")
|
|
323
|
+
|
|
294
324
|
# Create a violation report in TSV format — for all collections combined.
|
|
295
325
|
all_violations = ViolationList()
|
|
296
326
|
for collection_name, violations in sorted(source_collections_and_their_violations.items(), key=get_lowercase_key):
|
refscan-0.1.0/PKG-INFO
DELETED
|
@@ -1,68 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: refscan
|
|
3
|
-
Version: 0.1.0
|
|
4
|
-
Summary: Command-line program that scans the NMDC MongoDB database for referential integrity violations
|
|
5
|
-
Home-page: https://github.com/microbiomedata/refscan
|
|
6
|
-
Keywords: mongodb,mongo,relationships,reference,database,data,referential integrity,scan
|
|
7
|
-
Requires-Python: >=3.10,<4.0
|
|
8
|
-
Classifier: Programming Language :: Python :: 3
|
|
9
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
10
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
11
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
-
Requires-Dist: linkml-runtime (>=1.7.5,<2.0.0)
|
|
13
|
-
Requires-Dist: pymongo (>=4.7.1,<5.0.0)
|
|
14
|
-
Requires-Dist: rich (>=13.7.1,<14.0.0)
|
|
15
|
-
Requires-Dist: typer (>=0.12.3,<0.13.0)
|
|
16
|
-
Project-URL: Documentation, https://github.com/microbiomedata/refscan
|
|
17
|
-
Project-URL: Repository, https://github.com/microbiomedata/refscan
|
|
18
|
-
Description-Content-Type: text/markdown
|
|
19
|
-
|
|
20
|
-
# refscan
|
|
21
|
-
|
|
22
|
-
`refscan` is a command-line tool people can use to scan the NMDC MongoDB database
|
|
23
|
-
for referential integrity violations.
|
|
24
|
-
|
|
25
|
-
```mermaid
|
|
26
|
-
%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
|
|
27
|
-
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
|
|
28
|
-
%% Reference: https://github.com/pypi/warehouse/issues/13083
|
|
29
|
-
graph LR
|
|
30
|
-
schema[LinkML<br>schema]
|
|
31
|
-
database[(MongoDB<br>database)]
|
|
32
|
-
script[["refscan.py"]]
|
|
33
|
-
violations["List of<br>violations"]
|
|
34
|
-
references["List of<br>references"]:::dashed_border
|
|
35
|
-
schema --> script
|
|
36
|
-
database --> script
|
|
37
|
-
script -.-> references
|
|
38
|
-
script --> violations
|
|
39
|
-
|
|
40
|
-
classDef dashed_border stroke-dasharray: 5 5
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
## Assumptions
|
|
44
|
-
|
|
45
|
-
`refscan` was designed under some assumptions about the schema and database, including:
|
|
46
|
-
|
|
47
|
-
1. Each source document (i.e. document containing references) has a field named `type`, whose value (a string) is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance. For example, the `type` field of each document in the `study_set` collection has the value `"nmdc:Study"`.
|
|
48
|
-
|
|
49
|
-
## Development status
|
|
50
|
-
|
|
51
|
-
`refscan` is in early development and its author does not recommend anyone use it for anything without reviewing its code first.
|
|
52
|
-
|
|
53
|
-
## Tips
|
|
54
|
-
|
|
55
|
-
`refscan` requires the user to specify the path to a schema in YAML format. If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing the `{...}` placeholders and customizing the path):
|
|
56
|
-
|
|
57
|
-
```shell
|
|
58
|
-
# Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
|
|
59
|
-
curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
For example:
|
|
63
|
-
|
|
64
|
-
```shell
|
|
65
|
-
# Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
66
|
-
curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
67
|
-
```
|
|
68
|
-
|
refscan-0.1.0/README.md
DELETED
|
@@ -1,48 +0,0 @@
|
|
|
1
|
-
# refscan
|
|
2
|
-
|
|
3
|
-
`refscan` is a command-line tool people can use to scan the NMDC MongoDB database
|
|
4
|
-
for referential integrity violations.
|
|
5
|
-
|
|
6
|
-
```mermaid
|
|
7
|
-
%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
|
|
8
|
-
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
|
|
9
|
-
%% Reference: https://github.com/pypi/warehouse/issues/13083
|
|
10
|
-
graph LR
|
|
11
|
-
schema[LinkML<br>schema]
|
|
12
|
-
database[(MongoDB<br>database)]
|
|
13
|
-
script[["refscan.py"]]
|
|
14
|
-
violations["List of<br>violations"]
|
|
15
|
-
references["List of<br>references"]:::dashed_border
|
|
16
|
-
schema --> script
|
|
17
|
-
database --> script
|
|
18
|
-
script -.-> references
|
|
19
|
-
script --> violations
|
|
20
|
-
|
|
21
|
-
classDef dashed_border stroke-dasharray: 5 5
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
## Assumptions
|
|
25
|
-
|
|
26
|
-
`refscan` was designed under some assumptions about the schema and database, including:
|
|
27
|
-
|
|
28
|
-
1. Each source document (i.e. document containing references) has a field named `type`, whose value (a string) is the [class_uri](https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri) of the schema class of which the document represents an instance. For example, the `type` field of each document in the `study_set` collection has the value `"nmdc:Study"`.
|
|
29
|
-
|
|
30
|
-
## Development status
|
|
31
|
-
|
|
32
|
-
`refscan` is in early development and its author does not recommend anyone use it for anything without reviewing its code first.
|
|
33
|
-
|
|
34
|
-
## Tips
|
|
35
|
-
|
|
36
|
-
`refscan` requires the user to specify the path to a schema in YAML format. If you have `curl` installed, you can download a YAML file from GitHub by running the following command (after replacing the `{...}` placeholders and customizing the path):
|
|
37
|
-
|
|
38
|
-
```shell
|
|
39
|
-
# Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
|
|
40
|
-
curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
For example:
|
|
44
|
-
|
|
45
|
-
```shell
|
|
46
|
-
# Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
47
|
-
curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml
|
|
48
|
-
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|