cloudcatalog 0.4__tar.gz → 0.5__tar.gz

cloudcatalog-0.5/.github/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Error Log**
+If applicable, add the log dump produced by the bug.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari] (if applicable)
+ - Platform version [e.g. 1.0.0]
+ - Python version
+ - AWS CLI version
+
+
+**Additional context**
+Add any other context about the problem here.

cloudcatalog-0.5/.github/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

{cloudcatalog-0.4/src/cloudcatalog.egg-info → cloudcatalog-0.5}/PKG-INFO

@@ -1,20 +1,19 @@
 Metadata-Version: 2.1
 Name: cloudcatalog
-Version: 0.4
+Version: 0.5
 Summary: API for accessing the generalized Cloud Catalog (cloudcatalog) specification for sharing data in and across clouds
 Author-email: Johns Hopkins University Applied Physics Laboratory LLC <sandy.antunes@jhuapl.edu>
 License: MIT License
 Project-URL: Homepage, https://heliocloud.org
-Project-URL: Documentation, https://heliocloud.org
-Project-URL: Repository, https://gitlab.smce.nasa.gov
+Project-URL: Documentation, https://github.com/heliocloud-data/cloudcatalog
+Project-URL: Repository, https://github.com/heliocloud-data/cloudcatalog
 Keywords: cloud,index,catalog,AWS
 Classifier: Development Status :: 4 - Beta
 Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: LICENSE.txt
+License-File: LICENSE.MD
 Requires-Dist: boto3
 Requires-Dist: pandas
 
@@ -66,17 +65,17 @@ print(fr.get_entries())
 # also save the downloaded file index
 fr_id = 'a_dataset_id_from_the_catalog'
 start_date = '2007-02-01T00:00:00Z'  # A ISO 8601 standard time and a valid time witin the mission/file-index
-end_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
+stop_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
 myfiles = fr.request_cloud_catalog(fr_id, start_date=start_date, end_date=end_date, overwrite=False)
 ```
 
 ### Streaming Data from the File Catalog
-You now have a pandas DataFrame with startdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
+You now have a pandas DataFrame with startdate, stopdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
 
 This tool also offers a simple function for streaming the data once the file catalog is obtained:
 
 ```python
-cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, filesize: print(len(bo.read()), filesize))
+cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, stopdate, filesize: print(len(bo.read()), filesize))
 ```
 
 ### Searching the Entire Catalog

{cloudcatalog-0.4 → cloudcatalog-0.5}/README.md

@@ -46,17 +46,17 @@ print(fr.get_entries())
 # also save the downloaded file index
 fr_id = 'a_dataset_id_from_the_catalog'
 start_date = '2007-02-01T00:00:00Z'  # A ISO 8601 standard time and a valid time witin the mission/file-index
-end_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
+stop_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
 myfiles = fr.request_cloud_catalog(fr_id, start_date=start_date, end_date=end_date, overwrite=False)
 ```
 
 ### Streaming Data from the File Catalog
-You now have a pandas DataFrame with startdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
+You now have a pandas DataFrame with startdate, stopdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
 
 This tool also offers a simple function for streaming the data once the file catalog is obtained:
 
 ```python
-cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, filesize: print(len(bo.read()), filesize))
+cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, stopdate, filesize: print(len(bo.read()), filesize))
 ```
 
 ### Searching the Entire Catalog

{cloudcatalog-0.4 → cloudcatalog-0.5}/docs/Notes.md

@@ -9,7 +9,7 @@ Test the wheel
 
 Do a test package upload then install with:
 
-* twine upload -repository testpypi dist/*
+* twine upload -r testpypi dist/*
 * pip install -i https://test.pypi.org/simple/cloudcatalog==0.4
 
 Commit with

cloudcatalog-0.5/docs/cloudcatalog-spec-05.md

@@ -0,0 +1,398 @@
+<!-- TOC -->
+[1 Introduction](#1-intro)<br/>
+[2 Global data registry](#2-dataregistry)<br/>
+[3 Catalog](#3-catalog)<br/>
+[4 File Indices](#4-fileindex)<br/>
+[5 Info Metadata](#5-info)<br/>
+<!-- \TOC -->
+
+Version 0.5.0 \| HelioCloud \|
+
+# 1 The generalized Cloud Catalog specification for HelioCloud
+
+The shared Cloud Catalog specification can be used for sharing datasets across cloud frameworks.
+
+For HelioCloud, this specification creates a global data registry ('HelioDataRegistry'), maintained at the HDRL GitLab repository, then defines the file catalogs ('cloudCatalog') for each dataset, that resides in the S3 (or equivalent) bucket alongside the dataset.
+
+That 'HelioDataRegistry.json' is a JSON file consisting of **name** and **endpoint** and lists buckets as endpoints, not datasets.  Tools can visit each **endpoint** to get the **catalog.json** listing datasets available in that bucket.  The 'cloudCatalog' consists of the required index to the actual files, and an optional dataset summary file. The cloudCatalog itself is a set of ***<id>_YYYY.csv*** (or csp-zip or parquet) index files, one per year.  The optional summary file is named **<id>.json** file and provides additional potentially searchable metadata.
+
+## 1.1 Flow
+
+Findability is through the 'HelioDataRegistry' JSON index of S3 buckets, which leads to the **catalog.json** for each S3 bucket that indicates available datasets, which points to the individual dataset file catalog index files **<id>_YYYY.csv** and its optional associated **<id>.info** metadata auxillary file.
+
+(Diagram here)  github.com/whatever/HelioDataRegistry.json -> s3://aplcloud.com/mybucket/catalog.json -> individual dataset file catalogs (.csv)
+
+![Data Registry Schema](dataRegistry_diagram.png)
+
+Datasets are findable by going to 'HelioDataRegistry.json' to get endpoints, then visiting each S3 endpoint to get the catalog listing of datasets available.
+
+Accessing the actual data involves accessing each dataset's catalog index files. The cloudCatalog consists of CSV files, one per year. Therefore, a user has the choice to:
+* (a) download the CSV file
+* (b) use our provided Python API to fetch a subset of the CSV file, by date range
+* (c) use AWS Athena to run queries on the CSV file
+
+## 1.2 Uploading Data
+
+Data providers must provide both the Data. the Dataset Description (in .json as per the catalog.json spec) and a Manifest (in the .csv format we specify).
+
+We will generate an S3 Inventory of the contents and check against their Manifest, and send them the 'diff' file.  It is their responsibility to provide us with a corrected Manifest.  This iterates until their Manifest matches the actual Data provided.
+
+## 1.3 Refreshing Data
+
+Periodic updates to datasets go into an upload bucket.  The process is generally the same as for Uploading Data: provide Data and Manifest.  Once the Manifest matches the Data, we move the data from the upload bucket to the proper location and append the new Manifest to the existing Manifests.
+
+## 1.4 Scope
+
+We will produce Python tools that fit this API; a web form or similar tool for users to register datasets; automation to move datasets from an upload bucket to the data buckets; scripts to generate the AWS Inventory and compare against a user-provided Manifest; a sample use of Athena against the file catalogs for queries.
+
+This meets requirements 1, 4, 5, 6, 7 and 11 from the user stories/requirements document,, and opens up 9 (current scope is data be findable, whereas item 9 adds searchability).
+
+# 2. Global data registry 'HelioDataRegistry.json'
+
+The global data registry is an unsorted json list of names and bucket endpoints.  This is for buckets, not datasets (a bucket can hold multiple datasets).  Queries to each endpoint can fetch the list off datasets and more information from the **endpoint**/catalog.json file.
+
+The definitive listing of available dataset is kept in the **name for GSFC HelioCloud registry**.  Providers who wish to make data visibile to the public do a one-time registration of their S3 bucket (or equivalent) to the main registry
+The global data registry can be accessed directly or mirrored by anyone.
+
+Once an S3 bucket (or equivalent) is registered in the HelioDataRegistry.json file, it does not have to be updated when new datasets are added, only when new buckets are made public.
+
+For this global registry, only S3 buckets, not subbuckets, are allowed. For example, 's3://helio-public/' is allowed, but 's3://helio-public/MMS/' is not.
+
+## 2.1 Global Data Registry specification
+
+The registry is a JSON file containing the version of this specification, the date it was last modified, and the registry list of datasets.  Each dataset has two items: **endpoint** and **name**.  **endpoint**s must be unique; names do not enforce uniqueness.
+
+* **endpoint** - An accessble S3 (or equivalent) bucket link
+* **name** - A descriptive name for the dataset.
+* **provider** - default 'aws', indicates which cloud provider. Included for future federation.
+* **region** - The AWS or equivalent region the bucket resides in
+
+The **name** should be brief and one-line; length is not enforced by the specification but may be truncated when registration to keep things readable.
+
+## 2.2 Sample global data registry
+
+Here is a sample 'HelioDataRegistry.json' for three buckets at two sites.
+
+```javascript
+{
+    "version": "0.3",
+    "modificationDate": "2022-01-01T00:00.00Z",
+    "registry": [
+        {
+            "endpoint": "s3://gov-nasa-hdrl-data1/",
+            "name": "GSFC HelioCloud Set 1",
+            "provider": "aws",
+            "region": "us-east-1"
+        },
+        {
+            "endpoint": "s3://gov-nasa-hdrl-data2/",
+            "name": "GSFC HelioCloud Set 2",
+            "provider": "aws",
+            "region": "us-east-1"
+        },
+        {
+            "endpoint": "s3://edu-apl-helio-public/",
+            "name": "APL HelioCLoud",
+            "provider": "aws",
+            "region": "us-west-1"
+        }
+    ]
+}
+```
+
+# 3. Catalog of File Registries (per bucket catalog.json)
+
+The catalog.json file has an entry for each dataset stored within the given S3 bucket. **endpoint** and **name** are identical to the item in the global data registry.
+
+Globally the catalog.json describes the endpoint with the following items. Note that **endpoint** and **name** are included for clarity sake (being duplicates of the main registry) as JSON does not allow for comments, and should be the same as was provided to the global registry.
+
+* **endpoint** - same as was provided to GlobalDataRegistry.json, an accessble S3 (or equivalent) bucket link
+* **name** - same as was provided to GlobalDataRegistry.json, a descriptive name for the dataset.
+* **provider** - same as was provided to GlobalDataRegistry.json, default 'aws', indicates which cloud provider. Included for future federation.
+* **region** - same as was provided to GlobalDataRegistry.json, which AWS region
+* **egress** - one of 'no-egress', 'user-pays', 'egress-allowed', or 'none'
+* **status** - A return code, typically "1200/OK". Site owners can temporarily set this to other values
+* **contact** - Who to contact for issues with this bucket, e.g. "Dr. Contact, dr_contact@example.com"
+
+* **description** - Optional description of this collection
+* **citation** - Optional how to cite, preferably a DOI for the server
+* **comment** - A catch-all comment field for data provider and developer use. It should not contain information required to parse the data items.
+
+For each dataset, the catalog entry requires:
+
+* **id** a unique ID for the dataset that follows the ID naming requirements
+* **index** a fully qualified pointer to the object directory containing both the dataset and the required cloudCatalog. It MUST start with s3:// or "https://" (or equivalent) end in a terminating '/'.
+* **start**: string, Restricted ISO 8601 date/time of first record of data in the entire dataset OR the word 'static' for items such as model shapes that lack a time field.
+* **stop**: string, Restricted ISO 8601 date/time of end of the last 
+record of data in the entire dataset OR the word 'static' for items such as model shapes that lack a time field.
+* **modification**: string, Restricted ISO 8601 date/time of last time this dataset was updated
+* **title** a short descriptive title sufficient to identify the dataset and its utility to users
+* **indextype** Defines what format the actual cloudCatalog is, one of 'csv', 'csv-zip' or 'parquet'
+* **filetype** the file format of the actual data. Must be from the prescribed list of files. 
+
+* **description** optional description for dataset".
+* **resource** optional identifier e.g. SPASE ID, dataset description URL, DOI, json link of model parameters, or similar ancillary information".
+* **creation** optional ISO 8601 date/time of the dataset creation".
+* **expiration** optional ISO 8601 date/time after which the dataset will be expired, migrated, or not maintained".
+* **verified** optional ISO 8601 date/time for when the dataset was last tested by verifier programs".
+* **citation** optional how to cite this dataset, DOI or similar".
+* **contact** optional contact name".
+* **about** optional website URL for info, team, etc.
+* **multiyear** optional True/False field (default: False) for use when dataitems span multiple years (see 3.2 below)
+
+For file formats, there is a prescibed list. As new file formats are introduced, we will update this specification to give a single unique identifier for it.  The reason for the prescribed list is to avoid ambiguity or the need for users to parse (for example, avoiding figuring out '.fts', 'fits', '.FTS', etc)  Repositories with multiple files types can specify them as a comma-separated list with no spaces, e.g. 'fits,csv' for a dataset that contains both images and event lists. Currently defined types are 'fits,csv,cdf,netcdf3,netcdf4,hdf5,datamap,txt,binary,other'.
+
+The catalog.json file has to be updated when new data is added to a dataset, by updating the **stop** item. Also, the catalog.json file is updated when a new dataset is put into that S3 bucket.
+
+Note, currently this specification is defined around "s3://" architecture with some support for "https://" endpoints; future versions may support other protocols.
+
+## 3.1 ID naming requirements
+
+The **id** field can only contain alphanumeric characters, dashes, or underscores. No spaces or other characters are allowed.
+
+The **id** field will match the cloudCatalog files but does not have to match the sub-bucket names.  The cloudCatalog includes the **id**_YYYY.csv file indices and the optional **id**.json metadata file.
+
+## 3.2 Data items spanning multiple years
+
+Some data or model outputs span multiple years. An output product that covers 3 years, for example (2011-2013) would be index in "id"_2011.csv (based on the start date of the data).  A time-based search that is looking for 2012 coverage would therefore not be able to find it, as no "id"_2012.csv file exists or is needed.  To accommodate these edge cases, the **multiyear** field should be set to True, which will allow subsequent search layers to be aware that long-duration files exist in this dataset.
+
+This field should not be set for the typical case where a datafile extends slightly into the next year, but only when a datafile exists to provide data for a calendar year and that datafile would not be findable based purely on its given start date.
+
+
+## 3.2 Status codes
+
+The default status code for a catalog.json item is "1200/OK" indicating the data is available.  Status codes are informative and do not enforce any limits. They exist to communicate to users and client programs if a dataset is temporarily down or has other constraints.
+Other status codes defined so far include:
+* "code": "1200", "message": "OK": system is up and running
+* "code": "1400", "message": "temporarily unavailable": providers can set this if they temporarily are doing maintenance or need to stop access due to costs
+
+## 3.3 Example
+
+Here is an example catalog, for which only the first item has decided to fill out the optional 'ownership' block.  If this was the GSFC catalog.json, it would reside at s3://gov-nasa-helio-public/catalog.json.
+
+```javascript
+{
+    "version": "0.3",
+    "endpoint": "s3://gov-nasa-helio-public/",
+    "name": "GSFC HelioCloud",
+    "provider": "aws",
+    "region": "us-east-1",
+    "egress": "no-egress",
+    "contact": "Dr. Contact, dr_contact@example.com",
+    "description": "Optional description of this collection",
+    "citation": "Optional how to cite, preferably a DOI for the server",
+    "catalog":[
+        {
+            "id": "euvml",
+            "index": "s3://gov-nasa-helio-public/euvml/",
+            "title": "EUV-ML dataset",
+            "start": "1995-01-01T00:00.00Z",
+            "stop": "2022-01-01T00:00.00Z",
+            "modification": "2022-01-01T00:00.00Z",
+            "indextype": "csv",
+            "filetype": "fits",
+            "description": "Optional description for dataset",
+            "resource": "optional SPASE ID, DOI, URL, or json modelset",
+            "creation": "optional ISO 8601 date/time of the dataset creation",
+            "citation": "optional how to cite this dataset, DOI or similar",
+            "contact": "optional contact name",
+            "about": "optional website URL for info, team, etc"
+        },
+        {
+            "id": "mms_hmi",
+            "index": "s3://gov-nasa-helio-public/mms/hmi/",
+            "title": "MMS HMI data"
+            "start": "2015-01-01T00:00.00Z",
+            "stop": "2022-01-01T00:00.00Z",
+            "modification": "2022-01-01T00:00.00Z",
+            "indextype": "csv-zip",
+            "filetype": "cdf"
+        },
+        {
+            "id": "mms_feeps",
+            "index": "s3://gov-nasa-helio-public/mms/feeps/",
+            "title": "MMS FEEPS data"
+            "start": "2015-01-01T00:00.00Z",
+            "stop": "2022-01-01T00:00.00Z",
+            "modification": "2022-01-01T00:00.00Z",
+            "indextype": "csv-zip",
+            "filetype": "cdf"
+        },
+        {
+            "id": "fluxrope",
+            "index": "s3://heliotest/models/",
+            "title": "Instatiation of 3D fluxropes"
+            "start": "static",
+            "stop": "static",
+            "modification": "2022-01-01T00:00.00Z",
+            "indextype": "csv-zip",
+            "filetype": "cdf"
+        }
+    ],
+    "status": {
+        "code": 1200,
+        "message": "OK request successful"
+    }
+}
+```
+
+## 3.4 Indexes should reside in same bucket as data
+
+Index CSV files must be in the same bucket as the data, but do not have to be in the same sub-bucket or directory as their data.  The catalog points to the index files, and the index files use absolute paths to point to the data items.
+
+This also enables design a collection of datasets, wherein the data in the actual file catalog <ID>_<YYYY>.csv files can span sub-buckets. The use of absolute file paths is mandated.
+
+```
+Example data itself is in:
+  s3://example/mms1/feeps/
+  s3://example/mms2/feeps/
+  s3://example/mms3/feeps/
+  s3://example/mms4/feeps/
+```
+
+```
+Case 1: Matching
+    file catalog locations:
+    	 s3://example/mms1/feeps/mms1_feeps.CSV
+    	 s3://example/mms2/feeps/mms2_feeps.CSV
+    	 s3://example/mms3/feeps/mms3_feeps.CSV
+    	 s3://example/mms4/feeps/mms4_feeps.CSV
+```
+
+```
+Case 2: Not Matching
+    file catalog locations:
+         s3://example/mms_all/feeps/mms_feeps.CSV (contents point to 4 subbuckets)
+```
+
+The first case matches the idea of a 'dataset' and MUST be provided for any provided dataset.  The second matches the idea of a 'collection' and is supported but not required (i.e. additional extra cloudCatalog endpoints do not have to match the underlying data structure).
+
+## 3.5 Concerns
+
+Concerns were voiced about clashes if multiple people attempt to edit the catalog.json for a bucket at the same time.  Our default HelioCloud will maintain the catalog.json contents in a serverless DynamoDB (which will handle transaction collisions and provide rollback), which then outputs the 'catalog.json' file that users and data requests use.
+
+# 4 File Catalog
+
+The file catalog consists of one index for each year of the dataset in either csv, zipped csv, or parquet format.  The file must be in time sequence and the first four items must be the **start**, **stop**, **datakey**, and **filesize** fields in that order.
+
+The index cloudCatalog is a set of CSV or Parquet files named "index"/"id"_YYYY.csv, "index"/"id"_YYYY.csv.zip or "index"/"id"_YYYY.parquet.  For the case of static non-time sequence outputs, the index cloudCatalog are named "index"/"id"_static.csv (or .csv.zip or .parquet).
+
+## 4.1 Required Items
+
+* **start**: string, Restricted ISO 8601 date/time of start for that data OR the word 'static' for items such as model shapes that lack a time field
+* **stop**: string, Restricted ISO 8601 date/time of stop for that data OR the word 'static' for items such as model shapes that lack a time field
+* **datakey**: string, full filename or S3 object identifier sufficient to actually obtain the file
+* **filesize**: integer, file size in bytes
+
+## 4.2 Optional Items
+
+* **checksum**: checksum for that file. If given, **checksum_algorithm** must also be listed
+* **checksum_algorithm**: checksum algorithm used if checksums are generated. Examples include SHA, others.
+
+## 4.3 Optional Parameters
+
+The default expected search capability is _filetime_ within requested time range.
+
+Anything past **filesize** is fully optional; the minimal API expects only a start time, datakey aka filehandle, and file size IN THAT ORDER in the actual file index.  It is up to individual client programs to do anything past that.
+
+Any metadata included in this per-file index must be defined in the <id>.info json file to allow parseability.
+
+The csv or zipped csv files may or may not include a one-line header that is prefaced by "#".  It is the responsibility of client programs to determine if there is a skippable header or not.
+
+Any optional parameters from the above list or the info JSON must be in the same order specified in the JSON.
+
+For 'static' items, since the 'start' field will be the same constant value 'static' for all items, optional parameters to distinguish the items are suggested.
+
+## 4.4 Accessing
+
+Users have 3 options with the cloudCatalog CSV file:
+* download it directly and parse yourself,
+* use our Python API (provided) to extract a subset of filehandles from CSV,
+* use AWS Athena for queries
+
+## 4.5 Justification for Yearly CSV/Parquet Files
+
+Unlike a database, yearly index files are both fetchable and parseable. They have a lower cost profile than the equivalent database, reasonably fast access, and allow for client and search programs independent of a specific database implementation.
+
+Using yearly files rather than a single file or a database is to maintain an inexpensive, stateless, easily updated file index.  In AWS, the "S3 Inventory" command can generate a list of filenames and datakeys, or filenames and datakeys since the last time inventory was run. Being able to add to the catalog index files incrementally is easier served if they are chunked into yearly files.
+
+In addition, many use cases for long time baseline datasets will not need to access the entire multi-decadal span of the data, so parsing into years reduces the downloads needed to obtain the indices.
+
+The use of CSV or Parquet also enables AWS Athena searches within the index with little overhead, so long as optional metadata is provided.
+
+## 4.6 Example File Catalog
+
+Here is a short minimal CSV example index file.
+
+```
+# start, stop, datakey, filesize
+'2010-05-08T12:05:30.000Z','2010-05-08T12:06:14.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120530_n4euA.fts','246000'
+'2010-05-08T12:06:15.000Z','2010-05-08T12:10:29.00Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120615_n4euA.fts','246000'
+'2010-05-08T12:10:30.000Z','2010-05-08T12:14:29.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_121030_n4euA.fts','246000'
+```
+
+Here is an example with additional metadata and a CSV header as well.
+
+```
+# start, stop, datakey, filesize, wavelength, carr_lon, carr_lat
+'2010-05-08T12:05:30.000Z','2010-05-08T12:06:14.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120530_n4euA.fts','246000','195','20.4','30.0'
+'2010-05-08T12:06:15.000Z','2010-05-08T12:10:29.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120615_n4euA.fts','246000','195','21.8','30.0'
+'2010-05-08T12:10:30.000Z','2010-05-08T12:14:29.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_121030_n4euA.fts','246000','195','22.4','30.0'
+```
+Here is an example with additional metadata and a CSV header as the EUV-ML project would like.  Items in CAPS are directly from FITS keywords.
+```
+# start, stop, datakey, filesize, spacecraft, instrument, WAVELNTH, CRLT_OBS, CRLN_OBS, CRPIX1, CRPIX2, RSUN, quality, generation_flag
+'2010-05-08T12:05:30.000Z','2010-05-08T12:06:14.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120530_n4euA.fts','246000','A','euvi','195,45.0, 23.1, 512, 510, 26.5, 1, 1
+'2010-05-08T12:06:15.000Z','2010-05-08T12:10:29.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_120615_n4euA.fts','246000','A','euvi','195',45.0, 23.1, 512, 510, 26.5, 1, 1
+'2010-05-08T12:10:30.000Z','2010-05-08T12:14:29.000Z','s3://edu-apl-helio-public/euvml/stereo/a/195/20100508_121030_n4euA.fts','246000','A','euvi','195',45.0, 23.1, 512, 510, 26.5, 1, 1
+```
+
+# 5 Time Specification and ISO 8601
+
+Time values are always strings, and the SCR Time format (taken from the HAPI Time format) is a subset of the ISO 8601 standard. The restriction on the ISO 8601 standard is that time must be represented as
+
+```
+yyyy-mm-ddThh:mm:ss.sssZ
+```
+
+and the trailing Z is required. Strings with less precision are allowed as per ISO 8601. Any date or time elements missing from the string are assumed to take on their smallest possible value. For example, the string 2017-01-15T23:00:00.000Z could be given in truncated form as 2017-01-15T23:00Z.  A dataset must use only one format and length within that given dataset. The times values must not have any local time zone offset, and they must indicate this by including the trailing Z.
+
+
+# 6 Info Metadata
+
+Each dataset may also include an optional info json file that gives the time range, date last modified, ownership information, and optional additional metadata for that dataset.  The files are by default searchable and selectable by time window.  Additional search capability is not within scope of the file catalog per se, but data providers can indicate metadata for adding a search layer.
+
+If an <id>.info file exists and lists additional parameters, the resulting catalog index files must contain those parameters in the same order as expressed in this json file.
+
+## 6.1 Optional Items
+
+* **parameters**: optional list of searchable parameters available in the actual catalog index files
+file.
+(* = available from S3 Inventory)
+
+## 6.2 Example
+
+Here is an example for a sample optional Info json file.  This is used to indicate additional searchable metadata that exists within the catalog index files, and enables higher searchability in datasets.
+
+```javascript
+{
+    "version": "0.3",
+    "parameters": [
+        {"name": "spacecraft", "type": "string"},
+        {"name": "wavelength", "type": "int", "units": "Angstroms"},
+        {"name": "crlt", "type": "double", "units": "degrees", "desc": "Carrington latitude"},
+        {"name": "crln", "type": "double", "units": "degrees", "desc": "Carrington longitude"},
+        {"name": "rsun", "type": "double", "units": "pixels", "desc": "Size of sun in pizels"},
+        {"name": "crpix1", "type": "integer", "units": "pixels", "desc": "x coord of sun center"},
+        {"name": "crpix2", "type": "integer", "units": "pixels", "desc": "x coord of sun center"},
+        {"name": "quality", "type": "integer", "desc": "data quality and level of interpolation"}
+    ]
+}
+```
+
+## 7.0 Changes from 0.4 to 0.5
+
+Changed
+```
+Added 'stop' as a mandatory field
+```

{cloudcatalog-0.4 → cloudcatalog-0.5}/docs/cloudcatalog_demo.py

@@ -30,8 +30,6 @@ cat = cr.get_catalog()
 print("get_catalog:",cat)
 reg = cr.get_registry()
 print("get registry:",reg)
-link = cr.get_entries()
-print("get_entries:",link)
 url = cr.get_endpoint("HelioCloud, including SDO")
 print("get_endpoint:", url)
 

{cloudcatalog-0.4 → cloudcatalog-0.5}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = 'setuptools.build_meta'
     
 [project]
 name = "cloudcatalog"
-version = "0.4"
+version = "0.5"
 authors = [ {name = "Johns Hopkins University Applied Physics Laboratory LLC", email = "sandy.antunes@jhuapl.edu"}]
 license = {text = "MIT License"}
 description = "API for accessing the generalized Cloud Catalog (cloudcatalog) specification for sharing data in and across clouds"
@@ -25,5 +25,5 @@ dependencies = [
 
 [project.urls]
 Homepage = "https://heliocloud.org"
-Documentation = "https://heliocloud.org"
-Repository = "https://gitlab.smce.nasa.gov"
+Documentation = "https://github.com/heliocloud-data/cloudcatalog"
+Repository = "https://github.com/heliocloud-data/cloudcatalog"

{cloudcatalog-0.4 → cloudcatalog-0.5/src/cloudcatalog.egg-info}/PKG-INFO

@@ -1,20 +1,19 @@
 Metadata-Version: 2.1
 Name: cloudcatalog
-Version: 0.4
+Version: 0.5
 Summary: API for accessing the generalized Cloud Catalog (cloudcatalog) specification for sharing data in and across clouds
 Author-email: Johns Hopkins University Applied Physics Laboratory LLC <sandy.antunes@jhuapl.edu>
 License: MIT License
 Project-URL: Homepage, https://heliocloud.org
-Project-URL: Documentation, https://heliocloud.org
-Project-URL: Repository, https://gitlab.smce.nasa.gov
+Project-URL: Documentation, https://github.com/heliocloud-data/cloudcatalog
+Project-URL: Repository, https://github.com/heliocloud-data/cloudcatalog
 Keywords: cloud,index,catalog,AWS
 Classifier: Development Status :: 4 - Beta
 Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: MIT License
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: LICENSE.txt
+License-File: LICENSE.MD
 Requires-Dist: boto3
 Requires-Dist: pandas
 
@@ -66,17 +65,17 @@ print(fr.get_entries())
 # also save the downloaded file index
 fr_id = 'a_dataset_id_from_the_catalog'
 start_date = '2007-02-01T00:00:00Z'  # A ISO 8601 standard time and a valid time witin the mission/file-index
-end_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
+stop_date = None  # A ISO 8601 standard time or None if want all the file indices after start_date
 myfiles = fr.request_cloud_catalog(fr_id, start_date=start_date, end_date=end_date, overwrite=False)
 ```
 
 ### Streaming Data from the File Catalog
-You now have a pandas DataFrame with startdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
+You now have a pandas DataFrame with startdate, stopdate, key, and filesize for all the files of the mission within your specified start and end dates. From here, you can use the key to stream some of the data through EC2, a Lambda, or other processing methods.
 
 This tool also offers a simple function for streaming the data once the file catalog is obtained:
 
 ```python
-cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, filesize: print(len(bo.read()), filesize))
+cloudcatalog.CloudCatalog.stream(cloud_catalog, lambda bfile, startdate, stopdate, filesize: print(len(bo.read()), filesize))
 ```
 
 ### Searching the Entire Catalog

{cloudcatalog-0.4 → cloudcatalog-0.5}/src/cloudcatalog.egg-info/SOURCES.txt

@@ -1,19 +1,22 @@
 .gitignore
-LICENSE
-LICENSE.txt
+LICENSE.MD
 README.md
 TODO.md
 pyproject.toml
+.github/bug_report.md
+.github/feature_request.md
 docs/Makefile
 docs/Notes.md
-docs/cloudcatalog-spec-04.md
+docs/cloudcatalog-spec-05.md
 docs/cloudcatalog_demo.py
 docs/conf.py
 docs/index.rst
 docs/make.bat
 docs/scr_logo.png
+docs/earlier/cloudcatalog-spec-04.md
 src/__init__.py
 src/cloudcatalog.py
+src/quickfail.py
 src/cloudcatalog.egg-info/PKG-INFO
 src/cloudcatalog.egg-info/SOURCES.txt
 src/cloudcatalog.egg-info/dependency_links.txt

cloudcatalog-0.5/src/cloudcatalog.egg-info/top_level.txt

@@ -0,0 +1,5 @@
+__init__
+cloudcatalog
+data
+helio-public_cache
+quickfail

{cloudcatalog-0.4 → cloudcatalog-0.5}/src/cloudcatalog.py

@@ -10,7 +10,110 @@ import logging
 import dateutil
 import re
 from typing import List, Dict, Tuple, Union, Optional, Callable
+from botocore import UNSIGNED
+from botocore.client import Config
 
+# Added handler that first tries S3 anonymous, then tries https/egreess
+def s3url_to_https(s3url):
+    """ Formula is  s3://BUCKET/KEY -> https://BUCKET.s3.amazonaws.com/KEY
+    """
+    mybucket, mykey = s3url_to_bucketkey(s3url)
+    url = "https://" + mybucket + ".s3.amazonaws.com/" + mykey
+    return url
+    
+def s3url_to_bucketkey(s3url):
+    """
+    Extracts the S3 bucket name and file key from an S3 URL.
+
+    S3 paths are weird, bucket + everything else, e.g.
+    s3://b1/b2/b3/t.txt would be bucket b1, file b2/b3/t.txt
+
+    :param s3url: The S3 URL to extract the bucket name and file key from.
+
+    :returns: A tuple containing the S3 bucket name and file key.
+    """
+    # S3 paths are weird, bucket + everything else, e.g.
+    # s3://b1/b2/b3/t.txt would be bucket b1, file b2/b3/t.txt
+    name2 = re.sub(r"s3://", "", s3url)
+    s = name2.split("/", 1)
+    mybucket = s[0]
+    myfilekey = s[1] if len(s) > 1 else ""  # Want None if no key?
+    return mybucket, myfilekey
+    
+def fetch_S3(s3url,unsigned=True,region=None,rawbytes=False, **client_kwargs):
+    # default is JSON, but can return raw bytes
+    #print("Trying S3, unsigned=",unsigned,"region=",region)
+    mybucket, mykey = s3url_to_bucketkey(s3url)
+    #print("Looking for: ",mybucket,mykey)
+    if unsigned:
+        if region != None:
+            s3_client = boto3.client("s3", config=Config(signature_version=UNSIGNED),region=region, **client_kwargs)
+        else:
+            s3_client = boto3.client("s3", config=Config(signature_version=UNSIGNED), **client_kwargs)
+    else:
+        if region != None:
+            s3_client = boto3.client("s3",region=region, **client_kwargs)
+        else:
+            s3_client = boto3.client("s3", **client_kwargs)
+
+    response = s3_client.get_object(Bucket=mybucket, Key=mykey)
+    status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")
+    #print("  Success S3 unsigned",status)
+    if "Body" in response and status == 200:
+        catalog_bytes = response["Body"].read()
+        if rawbytes:
+            catalog = catalog_bytes
+        else:
+            catalog = json.loads(catalog_bytes)
+    elif status != 200:
+        print("Error, status = ",status)
+    return status, catalog
+
+def fetch_url(s3url,rawbytes=False):
+    # default is JSON, but can return raw bytes
+    httpurl = s3url_to_https(s3url)
+    response = requests.get(httpurl)
+    status = response.status_code
+    if rawbytes:
+        catalog = response.content
+    else:
+        catalog = response.json()
+    return status, catalog
+
+def fetch_S3orURL(s3url,region='us-east-1',rawbytes=False, **client_kwargs):
+    """ To get around vagualities of S3 access, this tries a cascade of:
+    straight fetch of S3 using your existing permissions
+    fetch S3 unsigned/anonymous
+    fetch S3 for a specified region only, defaulting to us-east-1
+    fetch the S3 contents via the AWS-equivalent URL
+    """
+    
+    try:
+        #print("Calling unsigned")
+        status, catalog = fetch_S3(s3url,unsigned=True,rawbytes=rawbytes, **client_kwargs)
+    except:
+        try:
+            #print("Calling signed")
+            status, catalog = fetch_S3(s3url,unsigned=False,rawbytes=rawbytes, **client_kwargs)
+        except:
+            try:
+                #print("Calling region")
+                status, catalog = fetch_S3(s3url, unsigned=True, region=region, rawbytes=rawbytes, **client_kwargs)
+            except:
+                try:
+                    #print("Calling url")
+                    status, catalog = fetch_url(s3url,rawbytes=rawbytes)
+                except:
+                    #print("Cannot fetch catalog, exiting.")
+                    return None
+    if rawbytes:
+        fr_bytes_file = BytesIO()
+        fr_bytes_file.write(catalog)
+        fr_bytes_file.seek(0)
+        return fr_bytes_file
+    else:
+        return catalog
+                        
 
 class CatalogRegistry:
     """Use to work with the the global catalog (catalog of catalogs)."""
@@ -172,6 +275,8 @@ class CloudCatalog:
 
         self.cache = cache
 
+        self.catalog = fetch_S3orURL(self.bucket_name+"/catalog.json",**client_kwargs)
+        """ # original version, added https mod
         # Create a client object with provided kwargs
         self.s3_client = boto3.client("s3", **client_kwargs)
 
@@ -189,7 +294,9 @@ class CloudCatalog:
 
         # Load the content from json
         self.catalog = json.loads(catalog_bytes)
-
+        """
+        
+        
         # Check catalog format assumptions
         if any([key not in self.catalog for key in ["status", "catalog"]]):
             raise KeyError(
@@ -231,8 +338,11 @@ class CloudCatalog:
                 os.mkdir(self.cache_folder)
 
             # Copy the content of the catalog to this file (overwrites)
-            with open(os.path.join(cache_folder, "catalog.json"), "wb") as file:
-                file.write(catalog_bytes)
+            
+            #with open(os.path.join(cache_folder, "catalog.json"), "wb") as file:
+            #    file.write(catalog_bytes)
+            with open(os.path.join(cache_folder, "catalog.json"), "w") as file:
+                json.dump(self.catalog,file, indent=4, ensure_ascii=False)
 
     def get_catalog(self) -> Dict:
         """
@@ -402,6 +512,9 @@ class CloudCatalog:
             ):
                 # May through some errors, NoSuchBucket, ClientError (file may not exists or access denied)
                 # If have ListBucket perms, no such key error will be raised instead of client error
+
+                fr_bytes_file = fetch_S3orURL(self.bucket_name+"/"+loc+filename,rawbytes=True)
+                """ # original
                 response = self.s3_client.get_object(Bucket=bucket_name, Key=loc + filename)
                 status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")
                 if "Body" in response and status == 200:
@@ -412,7 +525,8 @@ class CloudCatalog:
                     raise FailedS3Get(
                         f"Failed to get a cloud catalog object. Status: {stats}. Response: {response}"
                     )
-
+                """
+                
                 if filepath is not None:
                     with open(filepath, "wb") as file:
                         file.write(fr_bytes_file.read())
@@ -423,6 +537,13 @@ class CloudCatalog:
             else:
                 fr = pd.read_csv(filepath)
 
+            #print("Debug, version is ",self.catalog["Cloudy"])
+            if float(self.catalog["Cloudy"]) < 0.5:
+                # spec before 0.5 was start/key/filesize
+                # generate a 'maybe' stop using start time of prior entry
+                col0 = fr.columns[0]
+                fr.insert(1,"stop",fr[col0].shift(-1))
+
             # Handle # if used for the header
             if fr.columns.values[0][:2] == "# ":
                 fr.columns.values[0] = fr.columns.values[0][2:]
@@ -433,15 +554,17 @@ class CloudCatalog:
                 inplace=True,
             )
 
-            # assume first column is start, second is key, and third is filesize
+            # assume first column is start, second is stop, third is key, and fourth is filesize
             # only assuming if not found in column names
             # no error will be thrown if one of these missing, but per spec they are required
             if "start" not in fr.columns.values:
                 fr.columns.values[0] = "start"
+            if "stop" not in fr.columns.values:
+                fr.columns.values[1] = "stop"
             if "datakey" not in fr.columns.values:
-                fr.columns.values[1] = "datakey"
+                fr.columns.values[2] = "datakey"
             if "filesize" not in fr.columns.values:
-                fr.columns.values[2] = "filesize"
+                fr.columns.values[3] = "filesize"
 
             frs.append(fr)
 
@@ -449,14 +572,18 @@ class CloudCatalog:
 
         # Filter catalog dataframe to exact requested dates
         frs["start"] = pd.to_datetime(frs["start"], format="%Y-%m-%dT%H:%M:%SZ")
-        frs = frs[(start_date <= frs["start"]) & (frs["start"] < stop_date)]
+        frs["stop"] = pd.to_datetime(frs["stop"], format="%Y-%m-%dT%H:%M:%SZ")
+        # mod to add files that span a time interval longer than the requested interval
+        # was 'start date <= file_start & file_start < stop date'
+        # now 'start_date <= file_start & either file_start < stop date | file_end > start_date'
+        frs = frs[(start_date <= frs["start"]) & (frs["start"] < stop_date) | (frs["stop"] > start_date)]
 
         return frs
 
     @staticmethod
     def stream(
         cloud_catalog: pd.DataFrame,
-        process_func: Callable[[BytesIO, str, int], None],
+        process_func: Callable[[BytesIO, str, str, int], None],
         ignore_faileds3get: bool = False,
     ) -> None:
         """
@@ -465,17 +592,21 @@ class CloudCatalog:
         Parameters:
             cloud_catalog (pd.DataFrame): A pandas DataFrame containing the dataset catalog information.
             process_func (Callable): A function that takes a BytesIO object, a string representing the
-                                     start date of the file, and an integer representing the file size
-                                     as arguments.
+                                     start date of the file, a string representing the stop date of the file,
+                                     and an integer representing the file size as arguments.
             ignore_faileds3get (bool): A boolean that determines if the FailedS3Get is not thrown.
         """
-        s3_client = boto3.client("s3")
 
+        # original version, added https mod
+        #s3_client = boto3.client("s3")
+
+        
         fr_bytes_file = None
         for _, row in cloud_catalog.iterrows():
             # Get the S3 URL from the key in the dataframe
             s3_url = row["datakey"]
 
+            """ # original version, added https mod
             # Download the S3 file and read it into a BytesIO object
             response = s3_client.get_object(
                 Bucket=s3_url.split("/")[2], Key="/".join(s3_url.split("/")[3:])
@@ -489,14 +620,15 @@ class CloudCatalog:
                 raise FailedS3Get(
                     f"Failed to get a cloud catalog object. Status: {stats}. Response: {response}"
                 )
-
+            """
+            fr_bytes_file = fetch_S3orURL(s3_url,rawbytes=True)
             # Pass the BytesIO object, start date, and file size to the processing function
             # start may be a date object so making a string just in case for consistency
-            process_func(fr_bytes_file, str(row["start"]), row["filesize"])
+            process_func(fr_bytes_file, str(row["start"]), str(row["stop"]), row["filesize"])
 
     @staticmethod
     def stream_uri(
-        cloud_catalog: pd.DataFrame, process_func: Callable[[str, str, int], None]
+        cloud_catalog: pd.DataFrame, process_func: Callable[[str, str, str, int], None]
     ) -> None:
         """
         Sends S3 URLs to a processing function.
@@ -504,7 +636,8 @@ class CloudCatalog:
         Parameters:
             cloud_catalog (pd.DataFrame): A pandas DataFrame containing the dataset catalog information.
             process_func (Callable): A function that takes a string representing the S3 URL, a string
-                                     representing the start date of the file, and an integer representing
+                                     representing the start date of the file, a string representing the
+                                     stop date of the file, and an integer representing
                                      the file size as arguments.
         """
         for _, row in cloud_catalog.iterrows():
@@ -513,7 +646,7 @@ class CloudCatalog:
 
             # Pass the S3 URL, start date, and file size to the processing function
             # start may be a date object so making a string just in case for consistency
-            process_func(s3_url, str(row["start"]), row["filesize"])
+            process_func(s3_url, str(row["start"]), str(row["stop"]), row["filesize"])
 
 
 class EntireCatalogSearch:

cloudcatalog-0.5/src/quickfail.py

@@ -0,0 +1,9 @@
+import cloudcatalog
+
+cr = cloudcatalog.CatalogRegistry()
+endpoint = cr.get_endpoint('GSFC HelioCloud Public Temp')
+fr = cloudcatalog.CloudCatalog(endpoint, cache=False)
+bad="mms1_fpi_fast_des-dist"
+s='2020-01-01T00:00'
+e='2020-01-02T00:00'
+keyset = fr.request_cloud_catalog(bad,start_date=s,stop_date=e)

cloudcatalog-0.4/LICENSE

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2023 heliocloud-data
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

cloudcatalog-0.4/src/cloudcatalog.egg-info/top_level.txt

@@ -1,4 +0,0 @@
-__init__
-cloudcatalog
-data
-gov-nasa-hdrl-data1_cache