caltechdata-api 1.5.0__tar.gz → 1.6.1__tar.gz

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: caltechdata_api
-Version: 1.5.0
+Version: 1.6.1
 Summary: Python wrapper for CaltechDATA API.
 Home-page: https://github.com/caltechlibrary/caltechdata_api
 Author: Thomas E Morrell
@@ -18,38 +18,56 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 
 
-# caltechdata_api
+# CaltechDATA API Python Library
 
-Python library for using the CaltechDATA API
+The `caltechdata_api` Python library provides a convenient interface for interacting with the CaltechDATA API. It allows users to write files, create DataCite 4 standard JSON records, edit existing records, and retrieve metadata from the CaltechDATA repository.
 
-- caltechdata_write write files and a DataCite 4 standard json record to CaltechDATA repository
-- caltechdata_edit edits records in CaltechDATA
-- get_metadata gets metadata from CaltechDATA records
+## Features
 
-Requires Python 3 (Recommended via Anaconda https://www.anaconda.com/download) with reqests library.
+### Writing and Editing Records
+- `caltechdata_write`: Writes files and a DataCite 4 standard JSON record to the CaltechDATA repository.
+- `caltechdata_edit`: Edits existing records in CaltechDATA.
+
+### Metadata Operations
+- `get_metadata`: Retrieves metadata from CaltechDATA records.
+
+## Requirements
+
+- Python 3 (Anaconda is recommended)
+- `requests` library
+
+## Installation
+
+Install the library via pip:
+
+```shell
+pip install caltechdata_api
+```
 
 ## Examples
 
 There are some example python scripts in the GitHub repository.
 
-Create a record:
+###Create a record:
 
 ```shell
 python write.py example.json -fnames logo.gif
-pbkn6-m9y63
+# Output: pbkn6-m9y63 (unique identifier)
 ```
-The response will be the unique identifier for the record. You can put this at
+> The response will be the unique identifier for the record. You can put this at
 the end of a url to visit the record (e.g.
 https://data.caltechlibrary.dev/records/pbkn6-m9y63)
 
-Edit a record (make changes to the example.json file to see a change)
+###Edit a record 
+Make changes to the example.json file to see a change)
 ```
 python edit.py example.json -id pbkn6-m9y63
 10.33569/pbkn6-m9y63
 ```
-The response is the DOI for the record, which includes the unique identifier
+> The response is the DOI for the record, which includes the unique identifier
 for the record in the default configuration.
 
+## Using Custom DOIs 
 Some groups have worked with the library to create custom DOIs. These can be
 passed in the metadata like:
 
@@ -66,18 +84,16 @@ python edit.py example_custom.json -id m6zxz-p4j22
 
 This returns the custom DOI of the record if it is successful.
 
-## Setup 
 
-Install by typing 'pip install caltechdata_api'
+## Setup and Authentication
 
-## Usage
+1. Acquire a personal access token from your CaltechDATA account (found under "Applications" at the top right of your screen).
+2. Copy the token to a file named token.bash.
+3. Load the token in the command line using source token.bash.
 
-You need to acquire a personal access token from your CaltechDATA account
-(find it at the top right of your screen under "Applications").
-Then copy the token to token.bash.  Type `source token.bash` in 
-the command line to load the token.  
+## Note on Testing
 
-Only test your application on the test repository (data.caltechlibrary.dev).  Testing the API on the public 
+Only test your application on the test repository (`data.caltechlibrary.dev`).  Testing the API on the public 
 repository will generate junk records that are annoying to delete.
 
 

caltechdata_api-1.6.1/README.md

@@ -0,0 +1,78 @@
+# CaltechDATA API Python Library
+
+The `caltechdata_api` Python library provides a convenient interface for interacting with the CaltechDATA API. It allows users to write files, create DataCite 4 standard JSON records, edit existing records, and retrieve metadata from the CaltechDATA repository.
+
+## Features
+
+### Writing and Editing Records
+- `caltechdata_write`: Writes files and a DataCite 4 standard JSON record to the CaltechDATA repository.
+- `caltechdata_edit`: Edits existing records in CaltechDATA.
+
+### Metadata Operations
+- `get_metadata`: Retrieves metadata from CaltechDATA records.
+
+## Requirements
+
+- Python 3 (Anaconda is recommended)
+- `requests` library
+
+## Installation
+
+Install the library via pip:
+
+```shell
+pip install caltechdata_api
+```
+
+## Examples
+
+There are some example python scripts in the GitHub repository.
+
+###Create a record:
+
+```shell
+python write.py example.json -fnames logo.gif
+# Output: pbkn6-m9y63 (unique identifier)
+```
+> The response will be the unique identifier for the record. You can put this at
+the end of a url to visit the record (e.g.
+https://data.caltechlibrary.dev/records/pbkn6-m9y63)
+
+###Edit a record 
+Make changes to the example.json file to see a change)
+```
+python edit.py example.json -id pbkn6-m9y63
+10.33569/pbkn6-m9y63
+```
+> The response is the DOI for the record, which includes the unique identifier
+for the record in the default configuration.
+
+## Using Custom DOIs 
+Some groups have worked with the library to create custom DOIs. These can be
+passed in the metadata like:
+
+```shell
+python write.py example_custom.json -fnames logo.gif
+m6zxz-p4j22
+```
+
+And then you can edit with
+```
+python edit.py example_custom.json -id m6zxz-p4j22
+10.5281/inveniordm.1234
+```
+
+This returns the custom DOI of the record if it is successful.
+
+
+## Setup and Authentication
+
+1. Acquire a personal access token from your CaltechDATA account (found under "Applications" at the top right of your screen).
+2. Copy the token to a file named token.bash.
+3. Load the token in the command line using source token.bash.
+
+## Note on Testing
+
+Only test your application on the test repository (`data.caltechlibrary.dev`).  Testing the API on the public 
+repository will generate junk records that are annoying to delete.
+

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api/caltechdata_edit.py

@@ -33,7 +33,6 @@ def caltechdata_accept(ids, token=None, production=False):
     }
 
     for idv in ids:
-
         result = requests.get(
             url + "/api/records/" + idv + "/draft/review", headers=headers
         )

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api/caltechdata_write.py

@@ -67,9 +67,9 @@ def write_files_rdm(files, file_link, headers, f_headers, s3=None):
 def add_file_links(
     metadata, file_links, file_descriptions=[], additional_descriptions="", s3_link=None
 ):
-    # Currently configured for OSN S3 links
+    # Currently configured for S3 links, assuming all are at same endpoint
     link_string = ""
-    endpoint = "https://renc.osn.xsede.org/"
+    endpoint = "https://" + file_links[0].split("/")[2]
     s3 = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": endpoint})
     index = 0
     for link in file_links:
@@ -102,7 +102,6 @@ def add_file_links(
 
 
 def send_to_community(review_link, data, headers, publish, community, message=None):
-
     if not message:
         message = "This record is submitted automatically with the CaltechDATA API"
 

caltechdata_api-1.6.1/caltechdata_api/cli.py

@@ -0,0 +1,667 @@
+import argparse
+import requests
+import s3fs
+from caltechdata_api import caltechdata_write, caltechdata_edit
+from .md_to_json import parse_readme_to_json
+import json
+import os
+from cryptography.fernet import Fernet
+
+CALTECHDATA_API = "https://data.caltech.edu/api/names?q=identifiers.identifier:{}"
+ORCID_API = "https://orcid.org/"
+HEADERS = {"Accept": "application/json"}
+
+name = ""
+affiliationIdentifierScheme = ""
+affiliation_identifier = ""
+
+awardNumber = ""
+awardTitle = ""
+funderIdentifier = ""
+funderIdentifierType = ""
+funderName = ""
+
+home_directory = os.path.expanduser("~")
+caltechdata_directory = os.path.join(home_directory, ".caltechdata")
+
+
+if not os.path.exists(caltechdata_directory):
+    os.makedirs(caltechdata_directory)
+
+
+def generate_key():
+    return Fernet.generate_key()
+
+
+# Load the key from a file or generate a new one if not present
+def load_or_generate_key():
+    key_file = os.path.join(caltechdata_directory, "key.key")
+    if os.path.exists(key_file):
+        with open(key_file, "rb") as f:
+            return f.read()
+    else:
+        key = generate_key()
+        with open(key_file, "wb") as f:
+            f.write(key)
+        return key
+
+
+# Encrypt the token
+def encrypt_token(token, key):
+    f = Fernet(key)
+    return f.encrypt(token.encode())
+
+
+# Decrypt the token
+def decrypt_token(encrypted_token, key):
+    f = Fernet(key)
+    return f.decrypt(encrypted_token).decode()
+
+
+# Function to get or set token
+def get_or_set_token():
+    key = load_or_generate_key()
+    token_file = os.path.join(caltechdata_directory, "token.txt")
+    try:
+        with open(token_file, "rb") as f:
+            encrypted_token = f.read()
+            token = decrypt_token(encrypted_token, key)
+            return token
+    except FileNotFoundError:
+        while True:
+            token = input("Enter your CaltechDATA token: ").strip()
+            confirm_token = input("Confirm your CaltechDATA token: ").strip()
+            if token == confirm_token:
+                encrypted_token = encrypt_token(token, key)
+                with open(token_file, "wb") as f:
+                    f.write(encrypted_token)
+                return token
+            else:
+                print("Tokens do not match. Please try again.")
+
+
+def welcome_message():
+    print("Welcome to CaltechDATA CLI")
+
+
+def get_user_input(prompt, required=True):
+    while True:
+        user_input = input(prompt)
+        if required and not user_input:
+            print("This field is required. Please provide a value.")
+        else:
+            return user_input
+
+
+def confirm_upload():
+    while True:
+        user_input = input("Do you want to send this record to CaltechDATA? (y/n): ")
+        if user_input.lower() == "y":
+            return True
+        elif user_input.lower() == "n":
+            print("Upload canceled.")
+            return False
+        else:
+            print("Invalid input. Please enter 'y' or 'n'.")
+
+
+def check_award_number(award_number):
+    response = requests.get(
+        f"https://data.caltech.edu/api/awards?q=number:{award_number}"
+    )
+    data = response.json()
+    total_hits = data.get("hits", {}).get("total", 0)
+    return total_hits > 0
+
+
+def get_funding_entries():
+    while True:
+        try:
+            num_entries = int(
+                input("How many funding entries do you want to provide? ")
+            )
+            if num_entries >= 0:
+                return num_entries
+            else:
+                print("Please enter a non-negative integer.")
+        except ValueError:
+            print("Please enter a valid integer.")
+
+
+def validate_funder_identifier(funder_identifier):
+    response = requests.get(f"https://api.ror.org/organizations/{funder_identifier}")
+    if response.status_code == 200:
+        return True
+    else:
+        return False
+
+
+def get_funding_details():
+    award_number = get_user_input("Enter the award number for funding: ")
+    award_exists = check_award_number(award_number)
+    if not award_exists:
+        print(
+            f"""Error: No award with number '{award_number}' found in
+              CaltechDATA. You will need to provide more details about the
+              funding."""
+        )
+    award_title = get_user_input("Enter the award title for funding: ")
+    while True:
+        funder_identifier = get_user_input("Enter the funder ROR (https://ror.org): ")
+        if validate_funder_identifier(funder_identifier):
+            break
+        else:
+            print(
+                """This funder identifier is not a ROR. Please enter a valid
+                  ROR identifier (without the url). For example the ROR for the
+                  NSF is 021nxhr62."""
+            )
+    print("-" * 10)
+    return {
+        "awardNumber": award_number,
+        "awardTitle": award_title,
+        "funderIdentifier": funder_identifier,
+        "funderIdentifierType": "ROR",
+    }
+
+
+def parse_arguments():
+    welcome_message()
+    args = {}
+    args["title"] = get_user_input("Enter the title of the dataset: ")
+    args["description"] = get_user_input(
+        "Enter the abstract or description of the dataset: "
+    )
+    print("License options:")
+    print("1. Creative Commons Zero Waiver (cc-zero)")
+    print("2. Creative Commons Attribution (cc-by)")
+    print("3. Creative Commons Attribution Non Commercial (cc-by-nc)")
+
+    # Prompt user to select a license
+    while True:
+        license_number = input(
+            "Enter the number corresponding to the desired license: "
+        )
+        if license_number.isdigit() and 1 <= int(license_number) <= 8:
+            # Valid license number selected
+            args["license"] = {
+                "1": "cc0-1.0",
+                "2": "cc-by-4.0",
+                "3": "cc-by-nc-4.0",
+            }[license_number]
+            break
+        else:
+            print("Invalid input. Please enter a number between 1 and 8.")
+
+    while True:
+        orcid = get_user_input("Enter your ORCID identifier: ")
+        family_name, given_name = get_names(orcid)
+        if family_name is not None and given_name is not None:
+            args["orcid"] = orcid
+            break  # Break out of the loop if names are successfully retrieved
+        retry = input("Do you want to try again? (y/n): ")
+        if retry.lower() != "y":
+            print("Exiting program.")
+            return
+    # Optional arguments
+    num_funding_entries = get_funding_entries()
+    funding_references = []
+    for _ in range(num_funding_entries):
+        funding_references.append(get_funding_details())
+    args["fundingReferences"] = funding_references
+    return args
+
+
+def query_caltechdata_api(orcid):
+    response = requests.get(CALTECHDATA_API.format(orcid), headers=HEADERS)
+    return response.json()
+
+
+def query_orcid_api(orcid):
+    response = requests.get(ORCID_API + orcid, headers=HEADERS)
+    return response.json()
+
+
+def get_names(orcid):
+    caltechdata_response = query_caltechdata_api(orcid)
+    global affiliationIdentifierScheme, affiliation_identifier, name
+    if caltechdata_response.get("hits", {}).get("hits"):
+        hit = caltechdata_response["hits"]["hits"][0]
+        family_name = hit.get("family_name", "")
+        given_name = hit.get("given_name", "")
+        affiliation_identifier = "05dxps055"
+        affiliationIdentifierScheme = "ROR"
+        name = "California Institute of Technology"
+
+    else:
+        orcid_link = "https://orcid.org/"
+        headers = {"Accept": "application/json"}
+        orcid_response = requests.get(orcid_link + orcid, headers=headers)
+        try:
+            orcid_data = orcid_response.json()
+            name_info = orcid_data.get("person", {}).get("name", {})
+            family_name = name_info.get("family-name", {}).get("value", "")
+            given_name = name_info.get("given-names", {}).get("value", "")
+        except json.decoder.JSONDecodeError:
+            print(
+                f"Error: ORCID identifier not found or invalid. Please check the ORCID identifier and try again."
+            )
+            return None, None
+    return family_name, given_name
+
+
+def write_s3cmd_config(access_key, secret_key, endpoint):
+    configf = os.path.join(home_directory, ".s3cfg")
+    if not os.path.exists(key_file):
+        with open(configf, "w") as file:
+            file.write(
+                f"""[default]
+            access_key = {access_key}
+            host_base = {endpoint}
+            host_bucket = %(bucket).{endpoint}
+            secret_key = {secret_key}
+            """
+            )
+
+
+def upload_supporting_file(record_id=None):
+    filepath = ""
+    filepaths = []
+    file_links = []
+    while True:
+        choice = get_user_input(
+            "Do you want to upload or link data files? (upload/link/n): "
+        ).lower()
+        if choice == "link":
+            endpoint = "sdsc.osn.xsede.org"
+            path = "ini230004-bucket01/"
+            if not record_id:
+                access_key = get_user_input("Enter the access key: ")
+                secret_key = get_user_input("Enter the secret key: ")
+                write_s3cmd_config(access_key, secret_key, endpoint)
+                print("""S3 connection configured.""")
+                break
+            endpoint = f"https://{endpoint}/"
+            s3 = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": endpoint})
+            # Find the files
+            files = s3.glob(path + record_id + "/*")
+
+            for link in files:
+                fname = link.split("/")[-1]
+                if "." not in fname:
+                    # If there is a directory, get files
+                    folder_files = s3.glob(link + "/*")
+                    for file in folder_files:
+                        name = file.split("/")[-1]
+                        if "." not in name:
+                            level_2_files = s3.glob(file + "/*")
+                            for f in level_2_files:
+                                name = f.split("/")[-1]
+                                if "." not in name:
+                                    level_3_files = s3.glob(f + "/*")
+                                    for l3 in level_3_files:
+                                        file_links.append(endpoint + l3)
+                                else:
+                                    file_links.append(endpoint + f)
+                        else:
+                            file_links.append(endpoint + file)
+                else:
+                    file_links.append(endpoint + link)
+            return filepath, file_links
+        elif choice == "upload":
+            print("Current files in the directory:")
+            files = [
+                f for f in os.listdir() if not f.endswith(".json") and os.path.isfile(f)
+            ]
+            print("\n".join(files))
+            while True:
+                filename = get_user_input(
+                    "Enter the filename to upload as a supporting file (or 'n' to finish): "
+                )
+                if filename == "n":
+                    break
+                if filename in files:
+                    file_size = os.path.getsize(filename)
+                    if file_size > 1024 * 1024 * 1024:
+                        print(
+                            """The file is greater than 1 GB. Please upload the
+                        metadata to CaltechDATA, and you'll be provided
+                        instructions to upload the files to S3 directly."""
+                        )
+                    else:
+                        filepath = os.path.abspath(filename)
+                        filepaths.append(filepath)
+                else:
+                    print(
+                        f"Error: File '{filename}' not found. Please enter a valid filename."
+                    )
+
+            add_more = get_user_input("Do you want to add more files? (y/n): ").lower()
+            if add_more != "y":
+                break
+
+        elif choice == "n":
+            break
+        else:
+            print("Invalid input. Please enter 'link' or 'upload' or 'n'.")
+
+    return filepaths, file_links
+
+
+def upload_data_from_file():
+    while True:
+        print("Current JSON files in the directory:")
+        files = [f for f in os.listdir() if f.endswith(".json") and os.path.isfile(f)]
+        print("\n".join(files))
+
+        filename = get_user_input(
+            "Enter a README.md or JSON filename to upload to CaltechDATA (or type 'exit' to go back): "
+        )
+
+        if filename.lower() == "exit":
+            return None
+
+        if filename == "README.md":
+            data = parse_readme_to_json(filename)
+            return data
+        else:
+            try:
+                with open(filename, "r") as file:
+                    data = json.load(file)
+                return data
+
+            except json.JSONDecodeError as e:
+                print(f"Error: Invalid JSON format in the file '{filename}'. {str(e)}")
+
+
+def main():
+    choice = get_user_input(
+        "Do you want to create or edit a CaltechDATA record? (create/edit): "
+    ).lower()
+    if choice == "create":
+        create_record()
+    elif choice == "edit":
+        edit_record()
+    else:
+        print("Invalid choice. Please enter 'create' or 'edit'.")
+
+
+def create_record():
+    token = get_or_set_token()
+    print("Using CaltechDATA token:", token)
+    while True:
+        choice = get_user_input(
+            "Do you want to use metadata from an existing file or create new metadata? (existing/create): "
+        ).lower()
+        if choice == "existing":
+            existing_data = upload_data_from_file()
+            filepath, file_link = upload_supporting_file()
+            if existing_data:
+                if filepath != "":
+                    response = caltechdata_write(
+                        existing_data, token, filepath, production=False, publish=False
+                    )
+                elif file_link != "":
+                    response = caltechdata_write(
+                        existing_data,
+                        token,
+                        file_links=[file_link],
+                        s3_link=file_link,
+                        production=False,
+                        publish=False,
+                    )
+                else:
+                    response = caltechdata_write(
+                        existing_data, token, production=False, publish=False
+                    )
+                rec_id = response
+                print(
+                    f"""You can view and publish this record at https://data.caltechlibrary.dev/uploads/{rec_id}
+                    If you need to upload large files to S3, you can type
+                    `s3cmd put DATA_FILE s3://ini230004-bucket01/{rec_id}/"""
+                )
+                break
+            else:
+                print("Going back to the main menu.")
+        elif choice == "create":
+            args = parse_arguments()
+            family_name, given_name = get_names(args["orcid"])
+            metadata = {
+                "titles": [{"title": args["title"]}],
+                "descriptions": [
+                    {"description": args["description"], "descriptionType": "Abstract"}
+                ],
+                "creators": [
+                    {
+                        "affiliation": [
+                            {
+                                "affiliationIdentifier": affiliation_identifier,
+                                "affiliationIdentifierScheme": affiliationIdentifierScheme,
+                                "name": name,
+                            }
+                        ],
+                        "familyName": family_name,
+                        "givenName": given_name,
+                        "name": f"{family_name}, {given_name}",
+                        "nameIdentifiers": [
+                            {
+                                "nameIdentifier": args["orcid"],
+                                "nameIdentifierScheme": "ORCID",
+                            }
+                        ],
+                        "nameType": "Personal",
+                    }
+                ],
+                "types": {"resourceType": "", "resourceTypeGeneral": "Dataset"},
+                "rightsList": [
+                    {
+                        "rightsIdentifier": args["license"],
+                    }
+                ],
+                "fundingReferences": args["fundingReferences"],
+                "schemaVersion": "http://datacite.org/schema/kernel-4",
+            }
+            filepath, file_link = upload_supporting_file()
+            if confirm_upload():
+                if filepath != "":
+                    response = caltechdata_write(
+                        metadata, token, filepath, production=False, publish=False
+                    )
+                elif file_link != "":
+                    response = caltechdata_write(
+                        metadata,
+                        token,
+                        file_links=[file_link],
+                        production=False,
+                        publish=False,
+                    )
+                else:
+                    response = caltechdata_write(
+                        metadata, token, production=False, publish=False
+                    )
+                rec_id = response
+                print(
+                    f"""You can view and publish this record at https://data.caltechlibrary.dev/uploads/{rec_id}
+                    If you need to upload large files to S3, you can type
+                    `s3cmd put DATA_FILE s3://ini230004-bucket01/{rec_id}/"""
+                )
+                with open(response + ".json", "w") as file:
+                    json.dump(metadata, file, indent=2)
+                break
+            else:
+                break
+        else:
+            print("Invalid choice. Please enter 'existing' or 'create'.")
+
+
+def edit_record():
+    record_id = input("Enter the CaltechDATA record ID: ")
+    token = get_or_set_token()
+    file_name = download_file_by_id(record_id, token)
+    if file_name:
+        try:
+            # Read the edited metadata file
+            with open(file_name, "r") as file:
+                metadata = json.load(file)
+            response = caltechdata_edit(
+                record_id, metadata, token, production=False, publish=False
+            )
+            if response:
+                print("Metadata edited successfully.")
+            else:
+                print("Failed to edit metadata.")
+        except Exception as e:
+            print(f"An error occurred during metadata editing: {e}")
+    else:
+        print("No metadata file found.")
+    choice = get_user_input("Do you want to add files? (y/n): ").lower()
+    if choice == "y":
+        filepath, file_link = upload_supporting_file(record_id)
+        print(file_link)
+        if filepath != "":
+            response = caltechdata_edit(
+                record_id, token=token, files=filepath, production=False, publish=False
+            )
+        elif file_link != "":
+            response = caltechdata_edit(
+                record_id,
+                metadata,
+                token=token,
+                file_links=file_link,
+                production=False,
+                publish=False,
+            )
+        rec_id = response
+        print(
+            f"You can view and publish this record at https://data.caltechlibrary.dev/uploads/{rec_id}\n"
+        )
+
+
+def download_file_by_id(record_id, token=None):
+    url = f"https://data.caltechlibrary.dev/api/records/{record_id}"
+
+    headers = {
+        "accept": "application/vnd.datacite.datacite+json",
+    }
+
+    if token:
+        headers["Authorization"] = "Bearer %s" % token
+
+    try:
+        response = requests.get(url, headers=headers)
+        if response.status_code != 200:
+            # Might have a draft
+            response = requests.get(
+                url + "/draft",
+                headers=headers,
+            )
+            if response.status_code != 200:
+                raise Exception(f"Record {record_id} does not exist, cannot edit")
+        file_content = response.content
+        file_name = f"downloaded_data_{record_id}.json"
+        with open(file_name, "wb") as file:
+            file.write(file_content)
+        print(f"Metadata downloaded successfully: {file_name}")
+        with open(file_name, "r") as file:
+            metadata = json.load(file)
+            while True:
+                print("Fields:")
+                for i, field in enumerate(metadata.keys()):
+                    print(f"{i + 1}. {field}")
+
+                field_choice = int(
+                    input(
+                        "Enter the number of the field you want to edit (or 0 to skip, 'exit' to exit): "
+                    )
+                )
+
+                if field_choice == 0:
+                    break
+
+                selected_field = list(metadata.keys())[field_choice - 1]
+
+                if isinstance(metadata[selected_field], list):
+                    while True:
+                        print(f"Items in {selected_field}:")
+                        for i, item in enumerate(metadata[selected_field]):
+                            print(f"{i + 1}. {item}")
+
+                        item_choice = int(
+                            input(
+                                "Enter the number of the item you want to edit (or 0 to go back): "
+                            )
+                        )
+
+                        if item_choice == 0:
+                            break
+
+                        selected_item = metadata[selected_field][item_choice - 1]
+
+                        while True:
+                            print(f"Subfields for {selected_field}:")
+                            for i, subfield in enumerate(selected_item.keys()):
+                                print(f"{i + 1}. {subfield}")
+
+                            subfield_choice = int(
+                                input(
+                                    "Enter the number of the subfield you want to edit (or 0 to go back): "
+                                )
+                            )
+
+                            if subfield_choice == 0:
+                                break
+
+                            selected_subfield = list(selected_item.keys())[
+                                subfield_choice - 1
+                            ]
+
+                            new_value = input(
+                                f"Enter the new value for {selected_subfield}: "
+                            )
+
+                            metadata[selected_field][item_choice - 1][
+                                selected_subfield
+                            ] = new_value
+
+                            with open(file_name, "w") as file:
+                                json.dump(metadata, file, indent=2)
+
+                            print(f"File updated successfully.")
+
+                else:
+                    while True:
+                        print(f"Subfields for {selected_field}:")
+                        for i, subfield in enumerate(metadata[selected_field].keys()):
+                            print(f"{i + 1}. {subfield}")
+
+                        subfield_choice = int(
+                            input(
+                                "Enter the number of the subfield you want to edit (or 0 to go back): "
+                            )
+                        )
+
+                        if subfield_choice == 0:
+                            break
+
+                        selected_subfield = list(metadata[selected_field].keys())[
+                            subfield_choice - 1
+                        ]
+
+                        new_value = input(
+                            f"Enter the new value for {selected_subfield}: "
+                        )
+
+                        metadata[selected_field][selected_subfield] = new_value
+
+                        with open(file_name, "w") as file:
+                            json.dump(metadata, file, indent=2)
+
+                        print(f"File updated successfully.")
+
+    except Exception as e:
+        print(f"An error occurred: {e}")
+    return file_name
+
+
+if __name__ == "__main__":
+    main()

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api/customize_schema.py

@@ -57,7 +57,6 @@ def get_vocabularies():
 
 
 def customize_schema(json_record, schema="43"):
-
     if schema == "43":
         return customize_schema_rdm(json_record)
     else:
@@ -134,7 +133,6 @@ def rdm_creators_contributors(person_list, peopleroles):
 
 
 def customize_schema_rdm(json_record):
-
     # Get vocabularies used in InvenioRDM
     vocabularies = get_vocabularies()
 

caltechdata_api-1.6.1/caltechdata_api/md_to_json.py

@@ -0,0 +1,195 @@
+import re
+import json
+import requests
+
+
+class ReadmeFormatException(Exception):
+    """Custom exception for errors in the README format."""
+
+
+def camel_case(s):
+    """Converts a string to camelCase."""
+    s = re.sub(r"(\s|_|-)+", " ", s).title().replace(" ", "")
+    return s[0].lower() + s[1:] if s else ""
+
+
+def expand_special_keys(key, value):
+    """Expand special keys into their structured format (affiliation, nameIdentifiers)."""
+    if key == "affiliation":
+        if "ror.org" not in value:
+            raise ValueError("Affiliation Identifier is not a ROR")
+        ror = value.split("ror.org/")[1].split("]")[0]
+        response = requests.get(f"https://api.ror.org/organizations/{ror}").json()
+        return [
+            {
+                "affiliationIdentifier": ror,
+                "affiliationIdentifierScheme": "ROR",
+                "name": response["name"],
+            }
+        ]
+    elif key == "nameIdentifiers":
+        orcid = value.split("orcid.org/")[1].split("]")[0]
+        return [
+            {
+                "nameIdentifier": orcid,
+                "nameIdentifierScheme": "ORCID",
+                "schemeUri": f"https://orcid.org/{value}",
+            }
+        ]
+    return value
+
+
+def parse_readme_to_json(readme_path):
+    try:
+        with open(readme_path, "r") as file:
+            lines = file.read().split("\n")
+    except IOError as e:
+        raise ReadmeFormatException(f"Failed to open or read the file: {e}")
+
+    json_data = {}
+    current_section = None
+    current_object = {}
+
+    title_line = lines.pop(0)
+    if title_line.startswith("#") == False:
+        raise ValueError('README.md needs to start with "# Title"')
+    else:
+        json_data["titles"] = [{"title": title_line.replace("# ", "")}]
+
+    contributors = []
+    identifiers = []
+    item_list = []
+
+    section_pattern = re.compile(r"^##\s+(.*)$")
+    key_value_pattern = re.compile(r"^-\s+(.*?):\s+(.*)$")
+    link_pattern = re.compile(r"\[.*?\]\((.*?)\)")
+
+    for line_number, line in enumerate(lines, 1):
+        if not line.strip():
+            if item_list and current_section:
+                json_data[current_section] = item_list
+                item_list = []
+            elif current_object and current_section:
+                if current_section == "types":
+                    json_data[current_section] = current_object
+                elif len(current_object) == 1:
+                    key, value = next(iter(current_object.items()))
+                    if key in ["language", "publicationYear", "publisher", "version"]:
+                        json_data[current_section] = value
+                    else:
+                        json_data[current_section].append(current_object)
+                elif current_section in ["creators", "contributors"]:
+                    contributors.append(current_object)
+                    current_object = {}
+                elif current_section == "identifiers":
+                    identifiers.append(current_object)
+                    current_object = {}
+                else:
+                    json_data[current_section].append(current_object)
+                current_object = {}
+            continue
+
+        section_match = section_pattern.match(line)
+        if section_match:
+            if item_list:
+                json_data[current_section] = item_list
+            elif current_object:
+                if current_section in json_data:
+                    if isinstance(json_data[current_section], list):
+                        json_data[current_section].append(current_object)
+                    elif isinstance(json_data[current_section], dict):
+                        json_data[current_section].update(current_object)
+                else:
+                    json_data[current_section] = (
+                        [current_object]
+                        if current_section != "types"
+                        else current_object
+                    )
+                current_object = {}
+
+            elif contributors and current_section in ["creators", "contributors"]:
+                json_data[current_section] = contributors
+                contributors = []
+            elif identifiers and current_section == "identifiers":
+                json_data[current_section] = identifiers
+                identifiers = []
+
+            elif current_section and current_object:
+                if current_section == "types":
+                    json_data[current_section] = current_object
+                elif len(current_object) == 1:
+                    key, value = next(iter(current_object.items()))
+                    if key in ["language", "publicationYear", "publisher", "version"]:
+                        json_data[current_section].append(value)
+                    else:
+                        json_data[current_section].append(current_object)
+                else:
+                    json_data[current_section].append(current_object)
+                current_object = {}
+            current_section = camel_case(section_match.group(1))
+            json_data[current_section] = [] if current_section != "types" else {}
+            continue
+
+        key_value_match = key_value_pattern.match(line)
+        if key_value_match and current_section:
+            key, value = key_value_match.groups()
+            key = camel_case(key)
+
+            if key in ["affiliation", "nameIdentifiers"]:
+                value = expand_special_keys(key, value)
+            elif (
+                key == "nameType"
+                and current_object
+                and current_section in ["creators", "contributors"]
+            ):
+                contributors.append(current_object)
+                current_object = {}
+            elif current_section in ["subjects"]:
+                item_list.append({key: value})
+            elif current_section == "dates":
+                if key == "date":
+                    current_object["date"] = value
+                elif key == "dateType":
+                    current_object["dateType"] = value
+                    item_list.append(current_object)
+                    current_object = {}
+            else:
+                link_match = link_pattern.search(value)
+                if link_match:
+                    value = link_match.group(1)
+                current_object[key] = value
+
+        elif line.strip() and not section_match:
+            raise ReadmeFormatException(
+                f"Incorrect format detected at line {line_number}: {line}"
+            )
+
+    if contributors and current_section in ["creators", "contributors"]:
+        json_data[current_section] = contributors
+    elif identifiers and current_section == "identifiers":
+        json_data[current_section] = identifiers
+    elif current_section and current_object:
+        if current_section == "types":
+            json_data[current_section] = current_object
+        elif len(current_object) == 1:
+            key, value = next(iter(current_object.items()))
+            if key in ["language", "publicationYear", "publisher", "version"]:
+                json_data[current_section].append(value)
+            else:
+                json_data[current_section].append(current_object)
+        else:
+            json_data[current_section].append(current_object)
+
+    return json_data
+
+
+if __name__ == "__main__":
+    readme_path = "/Users/elizabethwon/downloads/exampleREADME.md"
+    try:
+        json_data = parse_readme_to_json(readme_path)
+        output_json_path = "output1.json"
+        with open(output_json_path, "w") as json_file:
+            json.dump(json_data, json_file, indent=4)
+        print(f"Converted JSON saved to {output_json_path}")
+    except ReadmeFormatException as e:
+        print(f"Error parsing README file: {e}")

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api/utils.py

@@ -54,7 +54,6 @@ def humanbytes(num: Union[int, float], metric: bool = True, precision: int = 1)
 
 
 if __name__ == "__main__":
-
     print(humanbytes(2251799813685247))  # 2 pebibytes
     print(humanbytes(2000000000000000, True))  # 2 petabytes
     print(humanbytes(1099511627776))  # 1 tebibyte

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: caltechdata-api
-Version: 1.5.0
+Version: 1.6.1
 Summary: Python wrapper for CaltechDATA API.
 Home-page: https://github.com/caltechlibrary/caltechdata_api
 Author: Thomas E Morrell
@@ -18,38 +18,56 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 
 
-# caltechdata_api
+# CaltechDATA API Python Library
 
-Python library for using the CaltechDATA API
+The `caltechdata_api` Python library provides a convenient interface for interacting with the CaltechDATA API. It allows users to write files, create DataCite 4 standard JSON records, edit existing records, and retrieve metadata from the CaltechDATA repository.
 
-- caltechdata_write write files and a DataCite 4 standard json record to CaltechDATA repository
-- caltechdata_edit edits records in CaltechDATA
-- get_metadata gets metadata from CaltechDATA records
+## Features
 
-Requires Python 3 (Recommended via Anaconda https://www.anaconda.com/download) with reqests library.
+### Writing and Editing Records
+- `caltechdata_write`: Writes files and a DataCite 4 standard JSON record to the CaltechDATA repository.
+- `caltechdata_edit`: Edits existing records in CaltechDATA.
+
+### Metadata Operations
+- `get_metadata`: Retrieves metadata from CaltechDATA records.
+
+## Requirements
+
+- Python 3 (Anaconda is recommended)
+- `requests` library
+
+## Installation
+
+Install the library via pip:
+
+```shell
+pip install caltechdata_api
+```
 
 ## Examples
 
 There are some example python scripts in the GitHub repository.
 
-Create a record:
+###Create a record:
 
 ```shell
 python write.py example.json -fnames logo.gif
-pbkn6-m9y63
+# Output: pbkn6-m9y63 (unique identifier)
 ```
-The response will be the unique identifier for the record. You can put this at
+> The response will be the unique identifier for the record. You can put this at
 the end of a url to visit the record (e.g.
 https://data.caltechlibrary.dev/records/pbkn6-m9y63)
 
-Edit a record (make changes to the example.json file to see a change)
+###Edit a record 
+Make changes to the example.json file to see a change)
 ```
 python edit.py example.json -id pbkn6-m9y63
 10.33569/pbkn6-m9y63
 ```
-The response is the DOI for the record, which includes the unique identifier
+> The response is the DOI for the record, which includes the unique identifier
 for the record in the default configuration.
 
+## Using Custom DOIs 
 Some groups have worked with the library to create custom DOIs. These can be
 passed in the metadata like:
 
@@ -66,18 +84,16 @@ python edit.py example_custom.json -id m6zxz-p4j22
 
 This returns the custom DOI of the record if it is successful.
 
-## Setup 
 
-Install by typing 'pip install caltechdata_api'
+## Setup and Authentication
 
-## Usage
+1. Acquire a personal access token from your CaltechDATA account (found under "Applications" at the top right of your screen).
+2. Copy the token to a file named token.bash.
+3. Load the token in the command line using source token.bash.
 
-You need to acquire a personal access token from your CaltechDATA account
-(find it at the top right of your screen under "Applications").
-Then copy the token to token.bash.  Type `source token.bash` in 
-the command line to load the token.  
+## Note on Testing
 
-Only test your application on the test repository (data.caltechlibrary.dev).  Testing the API on the public 
+Only test your application on the test repository (`data.caltechlibrary.dev`).  Testing the API on the public 
 repository will generate junk records that are annoying to delete.
 
 

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api.egg-info/SOURCES.txt

@@ -4,13 +4,16 @@ setup.py
 caltechdata_api/__init__.py
 caltechdata_api/caltechdata_edit.py
 caltechdata_api/caltechdata_write.py
+caltechdata_api/cli.py
 caltechdata_api/customize_schema.py
 caltechdata_api/download_file.py
 caltechdata_api/get_files.py
 caltechdata_api/get_metadata.py
+caltechdata_api/md_to_json.py
 caltechdata_api/utils.py
 caltechdata_api.egg-info/PKG-INFO
 caltechdata_api.egg-info/SOURCES.txt
 caltechdata_api.egg-info/dependency_links.txt
+caltechdata_api.egg-info/entry_points.txt
 caltechdata_api.egg-info/requires.txt
 caltechdata_api.egg-info/top_level.txt

caltechdata_api-1.6.1/caltechdata_api.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+caltechdata_api = caltechdata_api.cli:main
+

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/caltechdata_api.egg-info/requires.txt

@@ -3,3 +3,5 @@ datacite>1.1.0
 tqdm>=4.62.3
 pyyaml
 s3fs
+cryptography
+s3cmd

{caltechdata_api-1.5.0 → caltechdata_api-1.6.1}/setup.py

@@ -60,7 +60,15 @@ name = meta["name"]
 REQUIRES_PYTHON = ">=3.6.0"
 
 # What packages are required for this module to be executed?
-REQUIRED = ["requests", "datacite>1.1.0", "tqdm>=4.62.3", "pyyaml", "s3fs"]
+REQUIRED = [
+    "requests",
+    "datacite>1.1.0",
+    "tqdm>=4.62.3",
+    "pyyaml",
+    "s3fs",
+    "cryptography",
+    "s3cmd",
+]
 
 # What packages are optional?
 EXTRAS = {
@@ -163,4 +171,9 @@ setup(
     cmdclass={
         "upload": UploadCommand,
     },
+    entry_points={
+        "console_scripts": [
+            "caltechdata_api=caltechdata_api.cli:main",
+        ],
+    },
 )

caltechdata_api-1.5.0/README.md

@@ -1,62 +0,0 @@
-# caltechdata_api
-
-Python library for using the CaltechDATA API
-
-- caltechdata_write write files and a DataCite 4 standard json record to CaltechDATA repository
-- caltechdata_edit edits records in CaltechDATA
-- get_metadata gets metadata from CaltechDATA records
-
-Requires Python 3 (Recommended via Anaconda https://www.anaconda.com/download) with reqests library.
-
-## Examples
-
-There are some example python scripts in the GitHub repository.
-
-Create a record:
-
-```shell
-python write.py example.json -fnames logo.gif
-pbkn6-m9y63
-```
-The response will be the unique identifier for the record. You can put this at
-the end of a url to visit the record (e.g.
-https://data.caltechlibrary.dev/records/pbkn6-m9y63)
-
-Edit a record (make changes to the example.json file to see a change)
-```
-python edit.py example.json -id pbkn6-m9y63
-10.33569/pbkn6-m9y63
-```
-The response is the DOI for the record, which includes the unique identifier
-for the record in the default configuration.
-
-Some groups have worked with the library to create custom DOIs. These can be
-passed in the metadata like:
-
-```shell
-python write.py example_custom.json -fnames logo.gif
-m6zxz-p4j22
-```
-
-And then you can edit with
-```
-python edit.py example_custom.json -id m6zxz-p4j22
-10.5281/inveniordm.1234
-```
-
-This returns the custom DOI of the record if it is successful.
-
-## Setup 
-
-Install by typing 'pip install caltechdata_api'
-
-## Usage
-
-You need to acquire a personal access token from your CaltechDATA account
-(find it at the top right of your screen under "Applications").
-Then copy the token to token.bash.  Type `source token.bash` in 
-the command line to load the token.  
-
-Only test your application on the test repository (data.caltechlibrary.dev).  Testing the API on the public 
-repository will generate junk records that are annoying to delete.
-