cloudcatalog 1.0.2__tar.gz → 1.2.1__tar.gz

cloudcatalog-1.2.1/.github/workflows/pylint.yml

@@ -0,0 +1,23 @@
+name: Pylint
+
+on: [push]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10"]
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pylint
+    - name: Analysing the code with pylint
+      run: |
+        pylint $(git ls-files '*.py')

{cloudcatalog-1.0.2 → cloudcatalog-1.2.1}/.gitignore

@@ -6,6 +6,7 @@ __pycache__/
 # cloud catalog common caches
 *_cache/
 src/*_cache/
+*.bak
 
 # emacs save files
 *~

{cloudcatalog-1.0.2 → cloudcatalog-1.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: cloudcatalog
-Version: 1.0.2
+Version: 1.2.1
 Summary: API for accessing the generalized CloudCatalog (cloudcatalog) specification for sharing data in and across clouds
 Author-email: Johns Hopkins University Applied Physics Laboratory LLC <sandy.antunes@jhuapl.edu>
 License: MIT License
@@ -11,15 +11,25 @@ Keywords: cloud,index,catalog,AWS
 Classifier: Development Status :: 4 - Beta
 Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: MIT License
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.MD
 Requires-Dist: boto3
 Requires-Dist: pandas
+Dynamic: license-file
 
 # CloudCatalog (cloudcatalog) API
 
+Indexing millions of files for easy, searchable yet serverless and decentralized access is hard.  CloudCatalog is a lightweight CSV- and JSON-based indexing schema enabling HAPI-like "data ID + time range" queries on massive cloud datasets, and includes an implementation of the API and support tools in Python. Key goals include that (1) data owners control their own indices, (2) indices are static files to avoid incurring server costs, (3) searching is efficient and (4) indices are easily constructable and maintainable by the scientists/data-owners (the 'lazy' part). In addition to the FAIR principles of findability, accessibility, interoperability, and reusability, it is serverless and decentralized so that contributors can publish and update their open science data without the worries of external gatekeeping or server maintenance.  
+
 CloudCatalog is a generalized indexing specification for large cloud datasets.
+
+* For sharing datasets across cloud frameworks
+* Decentralized: data owners control their own data and access via JSON
+* RESTful & serverless (indices are flat CSV files alongside their datasets)
+* Removes need for doing slow/expensive disk ‘ls’ on large holdings
+* Searchable
+
 The push to open science means many more published datasets, and finding and accessing is important to solve. CloudCatalog is an indexing method for sharing big datasets in cloud systems. It is scientist-friendly and it is easy to generate a set of indices. It uses static index files in time-ordered CSV format that are easy to fetch, easy to access via an API, and very low cost in both money and bandwidth needed to support. Metadata is kept in a simple JSON schema. We also provide a Python client toolset for scientists to access datasets that use CloudCatalog. 
 
 The CloudCatalog specification and tools are open source, created by the HelioCloud project, and already used for 2 Petabytes of publicly available NASA and scientist-contributed data. We hope the community continues to adopt this CloudCatalog standard (in github, linked off heliocloud.org).
@@ -37,6 +47,22 @@ The CloudCatalog specification and tools are open source, created by the HelioCl
 
 The API is designed for retrieving file catalog (index) files from a specific ID entry in a catalog within a bucket. It also includes search functionality for searching through all data index catalogs found in the bucket list.
 
+## Command-line tools
+
+We also include command-line tools for creating and viewing the networked catalogs.
+
+#### Viewing tools
+
+* cloudcatalog-tree: lists or returns list of all toplevel datasets and number of dataIDs available, fast
+* cloudcatalog-spider: as 'tree' plus lists valid years and number of files, slow
+
+#### Generator/updater tools (beta, use at risk for now)
+
+* cloudcatalog-update-json: updates catalog.json using metadata from catalog_stub.json
+* cloudcatalog-update-csv: updates catalog.json using metadata from cat.csv
+* cloudcatalog-gui: GUI for selecting files for cloudcatalog-update-json
+* cloudcatalog-manifest2indices: tries to convert an AWS Manifest.csv to individual [dataID]_[YYYY].csv indices
+
 ## Use Case
 Suppose there is a mission on S3 that follows the HelioCloud 'CloudCatalog' specification, and you want to obtain specific files from this mission.
 
@@ -64,7 +90,7 @@ fr = cloudcatalog.CloudCatalog(bucket_name)
 print(fr.get_catalog())
 
 # To find the specific ID we can also get the ID + Title by
-print(fr.get_entries())
+print(fr.get_entries_id_title())
 
 # Now with the ID we can request the catalog index files as a Pandas dataframe
 fr_id = 'a_dataset_id_from_the_catalog'
@@ -93,7 +119,7 @@ mySDOlist = fr.request_cloud_catalog(dataid, start, stop)
 
 ### Add-on example for an MMS fetch of the filelist for all of a specific MMS item (64,383 files)
 ``` python
-dataid = "mms1_feeps_brst_electron"
+dataid = "MMS1_FEEPS_BRST_L2_ELECTRON"
 start, stop = fr.get_entry(dataid)['start'], fr.get_entry(dataid)['stop']
 myMMSlist = fr.request_cloud_catalog(dataid, start, stop)
 ```

{cloudcatalog-1.0.2 → cloudcatalog-1.2.1}/README.md

@@ -1,6 +1,15 @@
 # CloudCatalog (cloudcatalog) API
 
+Indexing millions of files for easy, searchable yet serverless and decentralized access is hard.  CloudCatalog is a lightweight CSV- and JSON-based indexing schema enabling HAPI-like "data ID + time range" queries on massive cloud datasets, and includes an implementation of the API and support tools in Python. Key goals include that (1) data owners control their own indices, (2) indices are static files to avoid incurring server costs, (3) searching is efficient and (4) indices are easily constructable and maintainable by the scientists/data-owners (the 'lazy' part). In addition to the FAIR principles of findability, accessibility, interoperability, and reusability, it is serverless and decentralized so that contributors can publish and update their open science data without the worries of external gatekeeping or server maintenance.  
+
 CloudCatalog is a generalized indexing specification for large cloud datasets.
+
+* For sharing datasets across cloud frameworks
+* Decentralized: data owners control their own data and access via JSON
+* RESTful & serverless (indices are flat CSV files alongside their datasets)
+* Removes need for doing slow/expensive disk ‘ls’ on large holdings
+* Searchable
+
 The push to open science means many more published datasets, and finding and accessing is important to solve. CloudCatalog is an indexing method for sharing big datasets in cloud systems. It is scientist-friendly and it is easy to generate a set of indices. It uses static index files in time-ordered CSV format that are easy to fetch, easy to access via an API, and very low cost in both money and bandwidth needed to support. Metadata is kept in a simple JSON schema. We also provide a Python client toolset for scientists to access datasets that use CloudCatalog. 
 
 The CloudCatalog specification and tools are open source, created by the HelioCloud project, and already used for 2 Petabytes of publicly available NASA and scientist-contributed data. We hope the community continues to adopt this CloudCatalog standard (in github, linked off heliocloud.org).
@@ -18,6 +27,22 @@ The CloudCatalog specification and tools are open source, created by the HelioCl
 
 The API is designed for retrieving file catalog (index) files from a specific ID entry in a catalog within a bucket. It also includes search functionality for searching through all data index catalogs found in the bucket list.
 
+## Command-line tools
+
+We also include command-line tools for creating and viewing the networked catalogs.
+
+#### Viewing tools
+
+* cloudcatalog-tree: lists or returns list of all toplevel datasets and number of dataIDs available, fast
+* cloudcatalog-spider: as 'tree' plus lists valid years and number of files, slow
+
+#### Generator/updater tools (beta, use at risk for now)
+
+* cloudcatalog-update-json: updates catalog.json using metadata from catalog_stub.json
+* cloudcatalog-update-csv: updates catalog.json using metadata from cat.csv
+* cloudcatalog-gui: GUI for selecting files for cloudcatalog-update-json
+* cloudcatalog-manifest2indices: tries to convert an AWS Manifest.csv to individual [dataID]_[YYYY].csv indices
+
 ## Use Case
 Suppose there is a mission on S3 that follows the HelioCloud 'CloudCatalog' specification, and you want to obtain specific files from this mission.
 
@@ -45,7 +70,7 @@ fr = cloudcatalog.CloudCatalog(bucket_name)
 print(fr.get_catalog())
 
 # To find the specific ID we can also get the ID + Title by
-print(fr.get_entries())
+print(fr.get_entries_id_title())
 
 # Now with the ID we can request the catalog index files as a Pandas dataframe
 fr_id = 'a_dataset_id_from_the_catalog'
@@ -74,7 +99,7 @@ mySDOlist = fr.request_cloud_catalog(dataid, start, stop)
 
 ### Add-on example for an MMS fetch of the filelist for all of a specific MMS item (64,383 files)
 ``` python
-dataid = "mms1_feeps_brst_electron"
+dataid = "MMS1_FEEPS_BRST_L2_ELECTRON"
 start, stop = fr.get_entry(dataid)['start'], fr.get_entry(dataid)['stop']
 myMMSlist = fr.request_cloud_catalog(dataid, start, stop)
 ```