pysodafair 0.1.62__tar.gz

pysodafair-0.1.62/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 FAIR Data Innovations Hub
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pysodafair-0.1.62/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: pysodafair
+Version: 0.1.62
+Summary: Pysoda package for Fairdataihub tools
+License: MIT
+License-File: LICENSE
+Author: Christopher Marroquin
+Author-email: cmarroquin@calmi2.org
+Requires-Python: >=3.6
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: openpyxl
+Project-URL: Homepage, https://github.com/fairdataihub/pysodafair
+Project-URL: Repository, https://github.com/fairdataihub/pysodafair
+Description-Content-Type: text/markdown
+
+# pysoda
+
+## Overview
+
+Pysoda is a tool for your python workflows that can help you create datasets in compliance with your favorite FAIR(Findable, Accessible, Interoperable, Reusable) data standards. At the moment, pysoda is primarily focused on neuromodulation, neurophysiology, and related data according to the SPARC guidelines that are aimed at making data FAIR. However, we are envisioning to extend the tool to support other standards such as BIDS, FHIR, etc, in the future.
+
+Pysoda stems from SODA, a desktop software that simplifies the organization and sharing of data that needs to comply to a FAIR data standard. While using the SODA app can be convenient for most investigators, others with coding proficiency may find it more convenient to implement automated workflows. Given that the backend of SODA contains many functions necessary for preparing and submitting a dataset that is compliant with the SPARC Data Structure (SDS) such as:
+
+Creating standard metadata files
+Generating manifest files
+Automatically complying with the file/folder naming conventions
+Validating against the offical SDS validator
+Uploading dataset to Pennsieve with SDS compliance (ignoring empty folders and non-allowed files, avoiding duplicate files and folders, etc.)
+And many more
+
+Pysoda makes these functions, which have been thoroughtly tested and validated, easily integratable in automated workflows such that the investigators do not have to re-write them. This will be very similar to the [pyfairdatatools](https://github.com/AI-READI/pyfairdatatools) Python package we are developing for our [AI-READI](https://aireadi.org/) project as part of the NIH Bridge2AI program.
+
+## Workflow
+
+### Import the pysoda package into your project and initialize the soda object with the supported standard of your choosing
+
+```python
+from pysoda import soda_create
+# initialize the soda_create object
+# Internal note: soda_create returns the typical sodaJSONObj with additional methods for adding data and metadata [not in version 1]
+# It is passed into the module functions just like our sodaJSONObj is passed to the backend of our api
+
+soda = soda_create(standard='sds')
+
+# add a dataset name to the soda object
+soda.set_dataset_name('my_dataset')
+
+```
+
+### Structure your data
+
+```python
+
+
+# get your base dataset files and folders structure
+dataset_structure = soda.get_dataset_structure()
+
+# fill out your dataset structure.
+# NOTE: YOu will want to reference the
+# dataset_structure key in the soda_schema.json file to understand the structure
+# and what is required.
+dataset_structure['folders'] = {
+    'data': {
+        'files': {
+            'file1': {
+                'path': '/home/user/file1.txt', 'relativePath': '/data/file1.txt', 'action': 'new'
+            }, 
+            'file2': {
+                'path': '/home/user/file2.txt', 'relativePath': '/data/file2.txt', 'action': 'new'
+            }
+        }, 
+        'folders': {
+            'primary': {
+                'files': {
+                    'file3': {
+                        'path': '/home/user/file3.txt', 'relativePath': '/data/primary/file3.txt', 'action': 'new'
+                    }
+                }
+            }
+        },
+        'relativePath': '/data'
+    },
+    'files': {},
+    'relativePath': '/'
+}
+
+
+# map your imported data files to the entity structure defined in the soda schema [here](soda_schema.py)
+entity_structure = soda.get_entity_structure()
+
+# fill out your entity structure using the schema as a reference
+# NOTE: data model not finalized
+entity = {'subjectId': 'sub-1', 'metadata': {'age': '1 year', 'sex': 'female'}, 'data-file': '/data/file1.txt'}
+entity_structure['subjects'].append(entity)
+
+
+
+```
+
+### Create your dataset metadata
+
+```python
+
+# import the metadata module from the soda_create package
+from pysoda import metadata
+
+# define your submission metadata
+submission = soda.get_submission_metadata()
+
+submission['consortium-data-standard'] = 'standard'
+submission['funding-consortium'] = 'SPARC'
+submission['award-number'] = '12345'
+submission['milestone-acheieved'] = ['one', 'two', 'three']
+submission['filepath'] = 'path/to/destination'
+
+# create the excel file for the submission metadata
+metadata.submission.create(soda, file_output_location='path/to/output')
+
+
+# repeat
+metadata.subjects.create(soda, file_output_location='path/to/output')
+metadata.samples.create(soda, file_output_location='path/to/output')
+metadata.performances.create(soda, file_output_location='path/to/output')
+metadata.sites.create(soda, file_output_location='path/to/output')
+metadata.code.create(soda, file_output_location='path/to/output')
+metadata.manifest.create(soda, file_output_location='path/to/output')
+
+```
+
+### Generate your dataset
+
+#### Generate locally
+
+```python
+
+from pysoda import generate
+
+# set the generation options
+soda.set_generate_dataset_options(destination='local', path='path/to/destination', dataset_name='my_dataset')
+
+# generate the dataset
+generate(soda)
+
+```
+
+#### Generate on Pennsieve
+
+```python
+from pysoda import generate
+
+# provide the Pennsieve API Key and secret
+soda.upload.auth(api_key='api, api_secret='api_secret)
+
+# upload new dataset
+# NOTE: You will need to download and start the Pennsieve Agent [here](https://app.pennsieve.io) to upload data to Pennsieve
+dataset_id = generate(soda) # returns dataset_id
+
+# OR upload to an existing pennsieve dataset
+# set the generate options in the soda object
+soda.set_generate_dataset_options(destination='existing-ps', if_existing="merge", if_existing_files="replace", dataset_id=dataset_id)
+update_existing(soda)
+```
+
+## Utilities
+
+### Compare a dataset on Pennsieve and a local dataset for differences
+
+```python
+from pysoda import compare
+
+# provide the Pennsieve API Key and secret
+soda.upload.auth(api_key='api, api_secret='api_secret)
+
+# import the dataset from Pennsieve
+soda.import_dataset(dataset_id='dataset_id')
+
+# compare the Pennsieve dataset with the local dataset
+results = compare(soda, local_dataset_location='path/to/local/dataset')
+```
+

pysodafair-0.1.62/README.md

@@ -0,0 +1,164 @@
+# pysoda
+
+## Overview
+
+Pysoda is a tool for your python workflows that can help you create datasets in compliance with your favorite FAIR(Findable, Accessible, Interoperable, Reusable) data standards. At the moment, pysoda is primarily focused on neuromodulation, neurophysiology, and related data according to the SPARC guidelines that are aimed at making data FAIR. However, we are envisioning to extend the tool to support other standards such as BIDS, FHIR, etc, in the future.
+
+Pysoda stems from SODA, a desktop software that simplifies the organization and sharing of data that needs to comply to a FAIR data standard. While using the SODA app can be convenient for most investigators, others with coding proficiency may find it more convenient to implement automated workflows. Given that the backend of SODA contains many functions necessary for preparing and submitting a dataset that is compliant with the SPARC Data Structure (SDS) such as:
+
+Creating standard metadata files
+Generating manifest files
+Automatically complying with the file/folder naming conventions
+Validating against the offical SDS validator
+Uploading dataset to Pennsieve with SDS compliance (ignoring empty folders and non-allowed files, avoiding duplicate files and folders, etc.)
+And many more
+
+Pysoda makes these functions, which have been thoroughtly tested and validated, easily integratable in automated workflows such that the investigators do not have to re-write them. This will be very similar to the [pyfairdatatools](https://github.com/AI-READI/pyfairdatatools) Python package we are developing for our [AI-READI](https://aireadi.org/) project as part of the NIH Bridge2AI program.
+
+## Workflow
+
+### Import the pysoda package into your project and initialize the soda object with the supported standard of your choosing
+
+```python
+from pysoda import soda_create
+# initialize the soda_create object
+# Internal note: soda_create returns the typical sodaJSONObj with additional methods for adding data and metadata [not in version 1]
+# It is passed into the module functions just like our sodaJSONObj is passed to the backend of our api
+
+soda = soda_create(standard='sds')
+
+# add a dataset name to the soda object
+soda.set_dataset_name('my_dataset')
+
+```
+
+### Structure your data
+
+```python
+
+
+# get your base dataset files and folders structure
+dataset_structure = soda.get_dataset_structure()
+
+# fill out your dataset structure.
+# NOTE: YOu will want to reference the
+# dataset_structure key in the soda_schema.json file to understand the structure
+# and what is required.
+dataset_structure['folders'] = {
+    'data': {
+        'files': {
+            'file1': {
+                'path': '/home/user/file1.txt', 'relativePath': '/data/file1.txt', 'action': 'new'
+            }, 
+            'file2': {
+                'path': '/home/user/file2.txt', 'relativePath': '/data/file2.txt', 'action': 'new'
+            }
+        }, 
+        'folders': {
+            'primary': {
+                'files': {
+                    'file3': {
+                        'path': '/home/user/file3.txt', 'relativePath': '/data/primary/file3.txt', 'action': 'new'
+                    }
+                }
+            }
+        },
+        'relativePath': '/data'
+    },
+    'files': {},
+    'relativePath': '/'
+}
+
+
+# map your imported data files to the entity structure defined in the soda schema [here](soda_schema.py)
+entity_structure = soda.get_entity_structure()
+
+# fill out your entity structure using the schema as a reference
+# NOTE: data model not finalized
+entity = {'subjectId': 'sub-1', 'metadata': {'age': '1 year', 'sex': 'female'}, 'data-file': '/data/file1.txt'}
+entity_structure['subjects'].append(entity)
+
+
+
+```
+
+### Create your dataset metadata
+
+```python
+
+# import the metadata module from the soda_create package
+from pysoda import metadata
+
+# define your submission metadata
+submission = soda.get_submission_metadata()
+
+submission['consortium-data-standard'] = 'standard'
+submission['funding-consortium'] = 'SPARC'
+submission['award-number'] = '12345'
+submission['milestone-acheieved'] = ['one', 'two', 'three']
+submission['filepath'] = 'path/to/destination'
+
+# create the excel file for the submission metadata
+metadata.submission.create(soda, file_output_location='path/to/output')
+
+
+# repeat
+metadata.subjects.create(soda, file_output_location='path/to/output')
+metadata.samples.create(soda, file_output_location='path/to/output')
+metadata.performances.create(soda, file_output_location='path/to/output')
+metadata.sites.create(soda, file_output_location='path/to/output')
+metadata.code.create(soda, file_output_location='path/to/output')
+metadata.manifest.create(soda, file_output_location='path/to/output')
+
+```
+
+### Generate your dataset
+
+#### Generate locally
+
+```python
+
+from pysoda import generate
+
+# set the generation options
+soda.set_generate_dataset_options(destination='local', path='path/to/destination', dataset_name='my_dataset')
+
+# generate the dataset
+generate(soda)
+
+```
+
+#### Generate on Pennsieve
+
+```python
+from pysoda import generate
+
+# provide the Pennsieve API Key and secret
+soda.upload.auth(api_key='api, api_secret='api_secret)
+
+# upload new dataset
+# NOTE: You will need to download and start the Pennsieve Agent [here](https://app.pennsieve.io) to upload data to Pennsieve
+dataset_id = generate(soda) # returns dataset_id
+
+# OR upload to an existing pennsieve dataset
+# set the generate options in the soda object
+soda.set_generate_dataset_options(destination='existing-ps', if_existing="merge", if_existing_files="replace", dataset_id=dataset_id)
+update_existing(soda)
+```
+
+## Utilities
+
+### Compare a dataset on Pennsieve and a local dataset for differences
+
+```python
+from pysoda import compare
+
+# provide the Pennsieve API Key and secret
+soda.upload.auth(api_key='api, api_secret='api_secret)
+
+# import the dataset from Pennsieve
+soda.import_dataset(dataset_id='dataset_id')
+
+# compare the Pennsieve dataset with the local dataset
+results = compare(soda, local_dataset_location='path/to/local/dataset')
+```

pysodafair-0.1.62/pyproject.toml

@@ -0,0 +1,26 @@
+[tool.poetry]
+name = "pysodafair"
+version = "0.1.62"
+description = "Pysoda package for Fairdataihub tools"
+authors = ["Christopher Marroquin <cmarroquin@calmi2.org>"]
+license = "MIT"
+readme = "README.md"
+homepage = "https://github.com/fairdataihub/pysodafair"
+repository = "https://github.com/fairdataihub/pysodafair"
+packages = [
+    { include = "pysoda" }
+]
+include = [
+    "pysoda/core/metadata_templates/*",
+    "pysoda/core/metadata_templates/*.xlsx",
+    "pysoda/schema/*.json"
+]
+
+[tool.poetry.dependencies]
+python = ">=3.6"
+openpyxl = "*"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

pysodafair-0.1.62/pysoda/constants.py

@@ -0,0 +1,3 @@
+
+PENNSIEVE_URL = "https://api.pennsieve.io"
+PENNSIEVE_2_URL = "https://api2.pennsieve.io"

pysodafair-0.1.62/pysoda/core/__init__.py

@@ -0,0 +1,10 @@
+import logging
+
+from .permissions import has_edit_permissions
+
+
+
+# Create a logger for the package
+logger = logging.getLogger(__name__)
+# Optional: Provide a default configuration if no handlers are set
+logger.setLevel(logging.WARNING)

pysodafair-0.1.62/pysoda/core/dataset_generation/__init__.py

@@ -0,0 +1,11 @@
+from .upload import (
+    create_folder_level_manifest,
+    check_empty_files_folders,
+    main_curate_function,
+    main_curate_function_progress,
+    generate_manifest_file_locally,
+    generate_manifest_file_data,
+    check_json_size,
+    clean_json_structure,
+    check_server_access_to_files,
+)

pysodafair-0.1.62/pysoda/core/dataset_generation/manifestSession/__init__.py

@@ -0,0 +1 @@
+from .manifest_session import UploadManifestSession

pysodafair-0.1.62/pysoda/core/dataset_generation/manifestSession/manifest_session.py

@@ -0,0 +1,146 @@
+from pennsieve2 import Pennsieve
+import re
+import math
+
+
+
+class UploadManifestSession:
+
+    # properties 
+    df_mid = None
+    ps = None
+
+    # upload values 
+    main_total_generate_dataset_size = None
+    total_files_to_upload = None 
+    elapsed_time = None
+
+    # rename values
+    renaming_files_flow = False
+    rename_total_files = None
+    list_of_files_to_rename = None
+
+    def __init__(self):
+        self.df_mid = None
+
+    def set_df_mid(self, id):
+        self.df_mid = id
+    
+    def get_df_mid(self):
+        return self.df_mid
+    
+    def set_elapsed_time(self, time):
+        self.elapsed_time = time
+
+    def get_elapsed_time(self):
+        return self.elapsed_time
+    
+    def set_main_total_generate_dataset_size(self, size):
+        self.main_total_generate_dataset_size = size
+
+    def get_main_total_generate_dataset_size(self):
+        return self.main_total_generate_dataset_size
+    
+    def set_total_files_to_upload(self, count):
+        self.total_files_to_upload = count
+
+    def get_total_files_to_upload(self):
+        return self.total_files_to_upload
+
+    def set_rename_total_files(self, count):
+        self.rename_total_files = count
+
+    def get_rename_total_files(self):
+        return self.rename_total_files
+    
+    def set_list_of_files_to_rename(self, list):
+        self.list_of_files_to_rename = list
+
+    def get_list_of_files_to_rename(self):
+        return self.list_of_files_to_rename
+    
+    def set_renaming_files_flow(self, value):
+        self.renaming_files_flow = value
+
+    def get_renaming_files_flow(self):
+        return self.renaming_files_flow
+
+    def df_mid_has_progress(self):
+        if self.ps is None:
+            self.ps = Pennsieve()
+        try: 
+            self.ps.manifest.sync(self.df_mid)
+        except Exception as e:
+            return False
+
+        try: 
+            mfs = self.ps.list_manifests()
+        except Exception as e:
+            # there are no manifests created yet
+            return False
+        return any(mf.id == self.df_mid and mf.status == "Initiated" for mf in mfs)
+    
+    def get_remaining_file_count(self, mid, total_files):
+        if self.ps is None:
+            self.ps = Pennsieve()
+        total_pages = math.ceil(total_files / 1000)
+        remaining_files = 0
+        offset = 0
+        for i in range(total_pages):
+            if i >= 1:
+                offset += 1000
+            file_page = self.ps.manifest.list_files(mid, offset , 1000)
+            # if there is no node_id then an upload hasn't started yet - all files are remaining 
+            # regular expression that searches and counts for every string that has "status: LOCAL" or "status: REGISTERED" or "status: FAILED" in the string
+            remaining_files +=  len(re.findall(r'status: REGISTERED|status: LOCAL|status: FAILED' , str(file_page)))
+        return remaining_files
+    
+    def create_obj_from_string(self,s):
+        # Split into individual objects
+        objects = re.findall(r'file {([^}]*?)}', s, re.DOTALL)
+
+        # Parse each object
+        parsed_objects = []
+        for obj in objects:
+            # Split into lines and remove empty lines
+            lines = [line.strip() for line in obj.split('\n') if line.strip()]
+            # Split each line into key and value and create a dictionary
+            parsed_object = {line.split(': ')[0]: line.split(': ')[1] for line in lines}
+            parsed_objects.append(parsed_object)
+
+        return parsed_objects
+
+    def calculate_completed_upload_size(self, mid, bytes_per_file_dict, total_files):
+        if self.ps is None:
+            self.ps = Pennsieve()
+        total_pages = math.ceil(total_files / 1000)
+        offset = 0
+        total_bytes_uploaded = 0
+        for i in range(total_pages):
+            if i >= 1:
+                offset += 1000
+            file_string = self.ps.manifest.list_files(mid, offset , 1000)
+            parsed_objects = self.create_obj_from_string(str(file_string))
+            for obj in parsed_objects:
+                if 'status' not in obj:
+                    total_bytes_uploaded += 0
+                elif obj['status'] in [
+                    'UPLOADED',
+                    'IMPORTED',
+                    'FINALIZED',
+                    'VERIFIED',
+                ]:
+                    file_path = obj['source_path']
+                    # remove the first and last characer of file_path - these are quotation marks
+                    file_path = file_path[1:-1]
+                    total_bytes_uploaded += int(bytes_per_file_dict.get(file_path, 0))
+
+        return total_bytes_uploaded
+
+    
+
+
+
+
+# ums = UploadManifestSession()
+# ums.df_mid_has_progress()