maven-data 1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- maven_data-1.0/PKG-INFO +127 -0
- maven_data-1.0/README.md +105 -0
- maven_data-1.0/maven/.streamlit/config.toml +8 -0
- maven_data-1.0/maven/__init__.py +0 -0
- maven_data-1.0/maven/_version.py +1 -0
- maven_data-1.0/maven/app.py +3407 -0
- maven_data-1.0/maven/cli.py +72 -0
- maven_data-1.0/maven/metadata_template/datasheet_sections.yaml +360 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_breakdown.yaml +2422 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_breakdown_old.yaml +2206 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_classes.yaml +5576 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_v1.2.md +1337 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_v1.2_MD.md +193 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_v1.2_YAML.yaml +1135 -0
- maven_data-1.0/maven/metadata_template/genesis_dc_v1.2_reference_guide.md +1750 -0
- maven_data-1.0/maven/remote_move.py +49 -0
- maven_data-1.0/maven/remote_register_endpoint.py +39 -0
- maven_data-1.0/maven/tier1_agent.py +91 -0
- maven_data-1.0/maven/tier1_genesis_dc_agent.py +201 -0
- maven_data-1.0/maven/ursa_autofill.py +1684 -0
- maven_data-1.0/maven/ursa_utils.py +58 -0
- maven_data-1.0/maven_data.egg-info/PKG-INFO +127 -0
- maven_data-1.0/maven_data.egg-info/SOURCES.txt +28 -0
- maven_data-1.0/maven_data.egg-info/dependency_links.txt +1 -0
- maven_data-1.0/maven_data.egg-info/entry_points.txt +2 -0
- maven_data-1.0/maven_data.egg-info/requires.txt +12 -0
- maven_data-1.0/maven_data.egg-info/top_level.txt +2 -0
- maven_data-1.0/pyproject.toml +34 -0
- maven_data-1.0/requirements.txt +12 -0
- maven_data-1.0/setup.cfg +4 -0
maven_data-1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: maven-data
|
|
3
|
+
Version: 1.0
|
|
4
|
+
Summary: Metadata Acquisition and Validation ENgine
|
|
5
|
+
Author-email: Vedant Iyer <iyer@lanl.gov>, Christopher W Johnson <cwj@lanl.gov>
|
|
6
|
+
License-Expression: BSD-3-Clause
|
|
7
|
+
Classifier: Programming Language :: Python
|
|
8
|
+
Requires-Python: >=3.12
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
Requires-Dist: streamlit>=1.52.2
|
|
11
|
+
Requires-Dist: pypdf>=5.9.0
|
|
12
|
+
Requires-Dist: python-docx>=1.2.0
|
|
13
|
+
Requires-Dist: lxml>=6.1.1
|
|
14
|
+
Requires-Dist: ursa-ai==0.15.8
|
|
15
|
+
Requires-Dist: langchain>=1.2.10
|
|
16
|
+
Requires-Dist: pandas<=3.0.0,>=2.3.1
|
|
17
|
+
Requires-Dist: dsi-workflow>=1.2.4
|
|
18
|
+
Requires-Dist: pyyaml>=6.0.3
|
|
19
|
+
Requires-Dist: reportlab>=4.4.6
|
|
20
|
+
Requires-Dist: linkml-runtime>=1.11.1
|
|
21
|
+
Requires-Dist: python-pptx>=1.0.2
|
|
22
|
+
|
|
23
|
+
# MAVEN User Guide
|
|
24
|
+
MAVEN is a local host interface designed to automate the collection of metadata for DIANA datasets using URSA.
|
|
25
|
+
It streamlines the process of gathering necessary dataset information before moving data and metadata to a specified HPC campaign directory via DSI.
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
# Quick Start
|
|
29
|
+
|
|
30
|
+
If already installed, simply run `maven` in your terminal.
|
|
31
|
+
Then, open the provided localhost URL in your preferred browser.
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
# Installation and Setup
|
|
35
|
+
|
|
36
|
+
## Prerequisites
|
|
37
|
+
- Python 3.12+ [Download](https://www.python.org/downloads/)
|
|
38
|
+
|
|
39
|
+
## Virtual Environment
|
|
40
|
+
It is recommended that users create a virtual environment to run MAVEN with Python 3.12.
|
|
41
|
+
|
|
42
|
+
**Option A: Miniconda3 [Download](https://www.anaconda.com/download/success)**
|
|
43
|
+
```
|
|
44
|
+
. ~/miniconda3/bin/activate
|
|
45
|
+
conda create -n maven_env python=3.12
|
|
46
|
+
conda activate maven_env # start here if virtual environment exists
|
|
47
|
+
```
|
|
48
|
+
**Option B: Python venv [Download python 3.12+](https://www.python.org/downloads/)**
|
|
49
|
+
```
|
|
50
|
+
python -m venv maven_env
|
|
51
|
+
source maven_env/bin/activate # start here if virtual environment exists
|
|
52
|
+
pip install --upgrade pip
|
|
53
|
+
```
|
|
54
|
+
**Option C: uv [Download](https://pydevtools.com/handbook/how-to/how-to-install-uv/)**
|
|
55
|
+
```
|
|
56
|
+
uv venv
|
|
57
|
+
source .venv/bin/activate
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Installation and Launching
|
|
61
|
+
**Option A: GitHub (Alpha version)**
|
|
62
|
+
```
|
|
63
|
+
git clone https://github.com/lanl/maven.git
|
|
64
|
+
cd maven
|
|
65
|
+
pip install .
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
**Option B: PyPI (Supported release)**
|
|
69
|
+
```
|
|
70
|
+
pip install maven-data
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
After successful install, simply run `maven` in your terminal.
|
|
74
|
+
Then, open the provided localhost URL in your preferred browser.
|
|
75
|
+
|
|
76
|
+
## Setup
|
|
77
|
+
When first launching the app, you will be required to complete 3 fields:
|
|
78
|
+
1) **Maven Directory**: Where your metadata databases will be locally saved.
|
|
79
|
+
2) **AI API Key**: Secret API key to connect to an AI model with [URSA](https://github.com/lanl/ursa)
|
|
80
|
+
3) **AI Base URL**: Endpoint to connect to that AI model
|
|
81
|
+
|
|
82
|
+
For further instructions on setting up an AI API Key for LANL users, please contact a MAVEN team member.
|
|
83
|
+
|
|
84
|
+
## HPC & DSI Setup
|
|
85
|
+
To move data to an HPC campaign space using [DSI](https://github.com/lanl/dsi), contact a MAVEN team member for setup details.
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
# App Notes
|
|
90
|
+
The app collects 3 levels of metadata:
|
|
91
|
+
|
|
92
|
+
## 1. The Genesis Datasheet (Sections 1-8)
|
|
93
|
+
These sections follow the Genesis Datasheet format. URSA leverages your **Project Description** and **Clarification** responses to autofill these fields.
|
|
94
|
+
|
|
95
|
+
## 2. Findability Metadata
|
|
96
|
+
High-level metadata used for indexing and searching.
|
|
97
|
+
|
|
98
|
+
## 3. AI-Ready Metadata
|
|
99
|
+
Domain-specific technical metadata that extracts values and types from the data.
|
|
100
|
+
|
|
101
|
+
Once all levels are complete, the app enables users to move data and metadata to the chosen HPC campaign directory.
|
|
102
|
+
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
# Contact Us
|
|
106
|
+
For general inquiries or help, reach us at maven-help (at) lanl.gov
|
|
107
|
+
|
|
108
|
+
Bugs/Feature Requests: [Maven GitHub Requests](https://github.com/lanl/maven/issues)
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
# Copyright and License
|
|
113
|
+
|
|
114
|
+
This program is Open-Source under the BSD-3 License.
|
|
115
|
+
|
|
116
|
+
© 2026. Triad National Security, LLC. All rights reserved. **O5152**
|
|
117
|
+
|
|
118
|
+
This program was produced under U.S. Government contract 89233218CNA000001 for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S. Department of Energy/National Nuclear Security Administration. All rights in the program are reserved by Triad National Security, LLC, and the U.S. Department of Energy/National Nuclear Security Administration. The Government is granted for itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare. derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
|
122
|
+
|
|
123
|
+
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
|
124
|
+
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
|
125
|
+
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
|
126
|
+
|
|
127
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
maven_data-1.0/README.md
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# MAVEN User Guide
|
|
2
|
+
MAVEN is a local host interface designed to automate the collection of metadata for DIANA datasets using URSA.
|
|
3
|
+
It streamlines the process of gathering necessary dataset information before moving data and metadata to a specified HPC campaign directory via DSI.
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
# Quick Start
|
|
7
|
+
|
|
8
|
+
If already installed, simply run `maven` in your terminal.
|
|
9
|
+
Then, open the provided localhost URL in your preferred browser.
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
# Installation and Setup
|
|
13
|
+
|
|
14
|
+
## Prerequisites
|
|
15
|
+
- Python 3.12+ [Download](https://www.python.org/downloads/)
|
|
16
|
+
|
|
17
|
+
## Virtual Environment
|
|
18
|
+
It is recommended that users create a virtual environment to run MAVEN with Python 3.12.
|
|
19
|
+
|
|
20
|
+
**Option A: Miniconda3 [Download](https://www.anaconda.com/download/success)**
|
|
21
|
+
```
|
|
22
|
+
. ~/miniconda3/bin/activate
|
|
23
|
+
conda create -n maven_env python=3.12
|
|
24
|
+
conda activate maven_env # start here if virtual environment exists
|
|
25
|
+
```
|
|
26
|
+
**Option B: Python venv [Download python 3.12+](https://www.python.org/downloads/)**
|
|
27
|
+
```
|
|
28
|
+
python -m venv maven_env
|
|
29
|
+
source maven_env/bin/activate # start here if virtual environment exists
|
|
30
|
+
pip install --upgrade pip
|
|
31
|
+
```
|
|
32
|
+
**Option C: uv [Download](https://pydevtools.com/handbook/how-to/how-to-install-uv/)**
|
|
33
|
+
```
|
|
34
|
+
uv venv
|
|
35
|
+
source .venv/bin/activate
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Installation and Launching
|
|
39
|
+
**Option A: GitHub (Alpha version)**
|
|
40
|
+
```
|
|
41
|
+
git clone https://github.com/lanl/maven.git
|
|
42
|
+
cd maven
|
|
43
|
+
pip install .
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
**Option B: PyPI (Supported release)**
|
|
47
|
+
```
|
|
48
|
+
pip install maven-data
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
After successful install, simply run `maven` in your terminal.
|
|
52
|
+
Then, open the provided localhost URL in your preferred browser.
|
|
53
|
+
|
|
54
|
+
## Setup
|
|
55
|
+
When first launching the app, you will be required to complete 3 fields:
|
|
56
|
+
1) **Maven Directory**: Where your metadata databases will be locally saved.
|
|
57
|
+
2) **AI API Key**: Secret API key to connect to an AI model with [URSA](https://github.com/lanl/ursa)
|
|
58
|
+
3) **AI Base URL**: Endpoint to connect to that AI model
|
|
59
|
+
|
|
60
|
+
For further instructions on setting up an AI API Key for LANL users, please contact a MAVEN team member.
|
|
61
|
+
|
|
62
|
+
## HPC & DSI Setup
|
|
63
|
+
To move data to an HPC campaign space using [DSI](https://github.com/lanl/dsi), contact a MAVEN team member for setup details.
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
# App Notes
|
|
68
|
+
The app collects 3 levels of metadata:
|
|
69
|
+
|
|
70
|
+
## 1. The Genesis Datasheet (Sections 1-8)
|
|
71
|
+
These sections follow the Genesis Datasheet format. URSA leverages your **Project Description** and **Clarification** responses to autofill these fields.
|
|
72
|
+
|
|
73
|
+
## 2. Findability Metadata
|
|
74
|
+
High-level metadata used for indexing and searching.
|
|
75
|
+
|
|
76
|
+
## 3. AI-Ready Metadata
|
|
77
|
+
Domain-specific technical metadata that extracts values and types from the data.
|
|
78
|
+
|
|
79
|
+
Once all levels are complete, the app enables users to move data and metadata to the chosen HPC campaign directory.
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
# Contact Us
|
|
84
|
+
For general inquiries or help, reach us at maven-help (at) lanl.gov
|
|
85
|
+
|
|
86
|
+
Bugs/Feature Requests: [Maven GitHub Requests](https://github.com/lanl/maven/issues)
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
|
|
90
|
+
# Copyright and License
|
|
91
|
+
|
|
92
|
+
This program is Open-Source under the BSD-3 License.
|
|
93
|
+
|
|
94
|
+
© 2026. Triad National Security, LLC. All rights reserved. **O5152**
|
|
95
|
+
|
|
96
|
+
This program was produced under U.S. Government contract 89233218CNA000001 for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S. Department of Energy/National Nuclear Security Administration. All rights in the program are reserved by Triad National Security, LLC, and the U.S. Department of Energy/National Nuclear Security Administration. The Government is granted for itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare. derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
|
100
|
+
|
|
101
|
+
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
|
102
|
+
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
|
103
|
+
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
|
104
|
+
|
|
105
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
File without changes
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "1.0"
|