willisapi-client 1.9.2__tar.gz → 1.9.4__tar.gz

willisapi_client-1.9.4/PKG-INFO

@@ -0,0 +1,132 @@
+Metadata-Version: 2.1
+Name: willisapi_client
+Version: 1.9.4
+Summary: A Python client for willisapi
+Home-page: https://github.com/bklynhlth/willsiapi_client
+Author: bklynhlth
+Author-email: admin@brooklyn.health
+License: Apache
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+<!-- Output copied to clipboard! -->
+
+<!-----
+
+Yay, no errors, warnings, or alerts!
+
+Conversion time: 0.36 seconds.
+
+
+Using this Markdown file:
+
+1. Paste this output into your source file.
+2. See the notes and action items below regarding this conversion run.
+3. Check the rendered output (headings, lists, code blocks, tables) for proper
+   formatting and use a linkchecker before you publish this page.
+
+Conversion notes:
+
+* Docs to Markdown version 1.0β34
+* Mon Aug 21 2023 13:41:56 GMT-0700 (PDT)
+* Source doc: README.md
+----->
+
+
+
+### WillisAPI Client
+
+WillisAPI Client is the Python interface for Brooklyn Health’s WillisAPI.
+
+Official documentation for WillisAPI Client can be found in the [Github Wiki](http://www.github.com/bklynhlth/willisapi_client/wiki).
+
+To learn more about Brooklyn Health or WillisAPI, visit [brooklyn.health](https://www.brooklyn.health) or [getintouch@brooklyn.health](mailto:getintouch@brooklyn.health).
+
+---
+
+#### Installation
+
+
+```bash
+pip install willisapi_client
+```
+
+---
+
+#### Getting a Personal Access Token (PAT)
+1. Log in to the Brooklyn Health web application.
+2. Go to your profile section.
+3. Navigate to Personal Access Token (PAT) and copy it for use in the client.
+
+---
+
+#### Usage
+
+**Upload**
+To upload a CSV file:
+
+```python
+summary = willisapi.upload(key, '/path/to/data.csv')
+```
+
+How to Call the Function
+- key: Your PAT token.
+- data.csv: Path to your CSV file.
+
+How to Reupload
+
+If you need to reupload the same or updated file, simply call the upload function again:
+
+```python
+summary = willisapi.upload(key, 'data.csv', force_uploade=True)
+```
+---
+#### Understanding Returned DataFrame and Errors
+
+- The upload function returns a summary DataFrame containing the results of your upload.
+- Columns typically include status, error messages, and metadata for each row in your CSV.
+- If there are errors, they will be listed in the DataFrame under an error or message column.
+- Review the DataFrame to identify and resolve any issues before reuploading.
+
+---
+
+#### CSV Validation Details
+
+Before uploading, the client validates your CSV for:
+
+- **Required Columns**: Must include study_id, site_id, participant_id, visit_name, visit_order, coa_name, coa_item_number, coa_item_value, file_path, time_collected.
+- **Optional Columns**: rater_id, age, sex, race, language.
+- **Valid COA Name**: Only these values are allowed: MADRS, YMRS, PHQ-9, GAD-7.
+- **Valid Audio File**: The file path in each row must exist and be accessible.
+- **Valid Data Types**: visit_order, age, coa_item_number, and coa_item_value must be numeric.
+- **Language**: If present, must be in the allowed language choices.
+
+If any validation fails, errors are collected and returned for review before upload proceeds.
+
+---
+
+**Processed Data Upload**
+
+To upload a processed data CSV file:
+
+```python
+summary = willisapi.processed_upload(key, '/path/to/processed-data.csv')
+```
+
+How to Call the Function
+- key: Your PAT token.
+- processed-data.csv: Path to your processed data CSV file.
+
+---
+
+For more information on how to organize the `data.csv`, visit the [Github Wiki](http://www.github.com/bklynhlth/willisapi_client/wiki).
+
+
+If you run into trouble while using the client, please raise it in the [Issues](http://www.github.com/bklynhlth/willisapi_client/issues) tab. 
+
+***
+
+Brooklyn Health is a small team of clinicians, scientists, and engineers based in Brooklyn, NY. 
+
+We develop and maintain [OpenWillis](http://www.github.com/bklynhlth/openwillis), an open source python library for digital health measurement. 

willisapi_client-1.9.4/README.md

@@ -0,0 +1,120 @@
+<!-- Output copied to clipboard! -->
+
+<!-----
+
+Yay, no errors, warnings, or alerts!
+
+Conversion time: 0.36 seconds.
+
+
+Using this Markdown file:
+
+1. Paste this output into your source file.
+2. See the notes and action items below regarding this conversion run.
+3. Check the rendered output (headings, lists, code blocks, tables) for proper
+   formatting and use a linkchecker before you publish this page.
+
+Conversion notes:
+
+* Docs to Markdown version 1.0β34
+* Mon Aug 21 2023 13:41:56 GMT-0700 (PDT)
+* Source doc: README.md
+----->
+
+
+
+### WillisAPI Client
+
+WillisAPI Client is the Python interface for Brooklyn Health’s WillisAPI.
+
+Official documentation for WillisAPI Client can be found in the [Github Wiki](http://www.github.com/bklynhlth/willisapi_client/wiki).
+
+To learn more about Brooklyn Health or WillisAPI, visit [brooklyn.health](https://www.brooklyn.health) or [getintouch@brooklyn.health](mailto:getintouch@brooklyn.health).
+
+---
+
+#### Installation
+
+
+```bash
+pip install willisapi_client
+```
+
+---
+
+#### Getting a Personal Access Token (PAT)
+1. Log in to the Brooklyn Health web application.
+2. Go to your profile section.
+3. Navigate to Personal Access Token (PAT) and copy it for use in the client.
+
+---
+
+#### Usage
+
+**Upload**
+To upload a CSV file:
+
+```python
+summary = willisapi.upload(key, '/path/to/data.csv')
+```
+
+How to Call the Function
+- key: Your PAT token.
+- data.csv: Path to your CSV file.
+
+How to Reupload
+
+If you need to reupload the same or updated file, simply call the upload function again:
+
+```python
+summary = willisapi.upload(key, 'data.csv', force_uploade=True)
+```
+---
+#### Understanding Returned DataFrame and Errors
+
+- The upload function returns a summary DataFrame containing the results of your upload.
+- Columns typically include status, error messages, and metadata for each row in your CSV.
+- If there are errors, they will be listed in the DataFrame under an error or message column.
+- Review the DataFrame to identify and resolve any issues before reuploading.
+
+---
+
+#### CSV Validation Details
+
+Before uploading, the client validates your CSV for:
+
+- **Required Columns**: Must include study_id, site_id, participant_id, visit_name, visit_order, coa_name, coa_item_number, coa_item_value, file_path, time_collected.
+- **Optional Columns**: rater_id, age, sex, race, language.
+- **Valid COA Name**: Only these values are allowed: MADRS, YMRS, PHQ-9, GAD-7.
+- **Valid Audio File**: The file path in each row must exist and be accessible.
+- **Valid Data Types**: visit_order, age, coa_item_number, and coa_item_value must be numeric.
+- **Language**: If present, must be in the allowed language choices.
+
+If any validation fails, errors are collected and returned for review before upload proceeds.
+
+---
+
+**Processed Data Upload**
+
+To upload a processed data CSV file:
+
+```python
+summary = willisapi.processed_upload(key, '/path/to/processed-data.csv')
+```
+
+How to Call the Function
+- key: Your PAT token.
+- processed-data.csv: Path to your processed data CSV file.
+
+---
+
+For more information on how to organize the `data.csv`, visit the [Github Wiki](http://www.github.com/bklynhlth/willisapi_client/wiki).
+
+
+If you run into trouble while using the client, please raise it in the [Issues](http://www.github.com/bklynhlth/willisapi_client/issues) tab. 
+
+***
+
+Brooklyn Health is a small team of clinicians, scientists, and engineers based in Brooklyn, NY. 
+
+We develop and maintain [OpenWillis](http://www.github.com/bklynhlth/openwillis), an open source python library for digital health measurement. 

{willisapi_client-1.9.2 → willisapi_client-1.9.4}/willisapi_client/__init__.py

@@ -2,19 +2,15 @@
 
 # import the required packages
 from willisapi_client.services.api import (
-    login,
-    upload,
-    download,
     willis_diarize_call_remaining,
     willis_diarize,
-    metadata_upload,
+    upload,
+    processed_upload
 )
 
 __all__ = [
-    "login",
-    "upload",
-    "download",
     "willis_diarize_call_remaining",
     "willis_diarize",
-    "metadata_upload",
+    "upload",
+    "processed_upload"
 ]

{willisapi_client-1.9.2 → willisapi_client-1.9.4}/willisapi_client/__version__.py

@@ -1,7 +1,7 @@
 """Version details for willisapi_client"""
 
 __client__ = "willisapi_client"
-__latestVersion__ = "1.9.2"
+__latestVersion__ = "1.9.4"
 __url__ = "https://github.com/bklynhlth/willsiapi_client"
 __short_description__ = "A Python client for willisapi"
 __content_type__ = "text/markdown"

willisapi_client-1.9.4/willisapi_client/services/api.py

@@ -0,0 +1,8 @@
+# website:   https://www.brooklyn.health
+
+# import the required packages
+from willisapi_client.services.diarize import (
+    willis_diarize_call_remaining,
+    willis_diarize,
+)
+from willisapi_client.services.metadata import upload, processed_upload

willisapi_client-1.9.4/willisapi_client/services/metadata/__init__.py

@@ -0,0 +1,4 @@
+# website:   https://www.brooklyn.health
+from .upload import upload, processed_upload
+
+__all__ = ["upload", "processed_upload"]

willisapi_client-1.9.4/willisapi_client/services/metadata/upload.py

@@ -0,0 +1,194 @@
+import pandas as pd
+from tqdm import tqdm
+from datetime import datetime
+import requests
+import mimetypes
+import os
+
+from willisapi_client.timer import measure
+from willisapi_client.willisapi_client import WillisapiClient
+from willisapi_client.logging_setup import logger as logger
+from willisapi_client.services.metadata.utils import (
+    MetadataValidation,
+    ProcessedMetadataValidation,
+    UploadUtils,
+    find_files_with_pattern,
+    get_last_n_directories,
+)
+
+
+@measure
+def upload(api_key: str, csv_path: str, **kwargs):
+
+    force_upload = kwargs.get("force_upload", False)
+    csv = MetadataValidation(csv_path=csv_path, force_upload=force_upload)
+    if csv.load_and_validate():
+        logger.info(f'{datetime.now().strftime("%H:%M:%S")}: csv check passed')
+        csv.create_final_csv()
+
+        wc = WillisapiClient(env=kwargs.get("env"))
+        url = wc.get_upload_url()
+        headers = wc.get_headers()
+        headers["Authorization"] = f"token {api_key}"
+        logger.info(f'{datetime.now().strftime("%H:%M:%S")}: beginning upload')
+
+        results = []
+        for index, row in tqdm(
+            csv.transformed_df.iterrows(), total=csv.transformed_df.shape[0]
+        ):
+            u = UploadUtils(row)
+            valid, err = u.validate_row()
+            result_row = row.to_dict()
+            if valid:
+                payload = u.generate_payload()
+                res = u.post(api_key, url, headers, payload)
+                if res.get("upload_status") == "Success":
+                    result_row["upload_status"] = "Success"
+                    result_row["error"] = None
+
+                    # Handle S3 upload if presigned URL is provided
+                    presigned = res.get("response", {}).get("presigned")
+                    if presigned:
+                        try:
+                            content_type, _ = mimetypes.guess_type(
+                                payload.get("filename")
+                            )
+                            if not content_type:
+                                content_type = "application/octet-stream"
+                            with open(row.file_path, "rb") as f:
+                                response = requests.put(
+                                    presigned,
+                                    data=f,
+                                    headers={
+                                        "x-amz-checksum-sha256": payload.get(
+                                            "checksum"
+                                        ),
+                                        "x-amz-sdk-checksum-algorithm": "SHA256",
+                                        "Content-Type": content_type,
+                                    },
+                                )
+                            if response.status_code == 200:
+                                result_row["upload_status"] = "Success"
+                            else:
+                                result_row["upload_status"] = "Failed"
+                                result_row["error"] = (
+                                    f"S3 upload failed with status code {response.status_code}"
+                                )
+                        except Exception as ex:
+                            result_row["upload_status"] = "Failed"
+                            result_row["error"] = str(ex)
+                    else:
+                        result_row["upload_status"] = "Failed"
+                        result_row["error"] = (
+                            "Collect recording upload URL not received"
+                        )
+                else:
+                    result_row["upload_status"] = "Failed"
+                    result_row["error"] = res.get("error")
+            else:
+                result_row["upload_status"] = "Failed"
+                result_row["error"] = f"{err}"
+            results.append(result_row)
+
+        results_df = pd.DataFrame(results)
+        return results_df
+    else:
+        logger.error(f'{datetime.now().strftime("%H:%M:%S")}: csv check failed')
+        logger.error(csv.errors)
+        return None
+
+
+@measure
+def processed_upload(api_key: str, csv_path: str, output_path: str, **kwargs):
+
+    force_upload = kwargs.get("force_upload", False)
+    csv = ProcessedMetadataValidation(csv_path=csv_path, force_upload=force_upload)
+    if csv.load_and_validate():
+        logger.info(f'{datetime.now().strftime("%H:%M:%S")}: csv check passed')
+        csv.create_final_csv()
+
+        wc = WillisapiClient(env=kwargs.get("env"))
+        url = wc.get_processed_upload_url()
+        headers = wc.get_headers()
+        headers["Authorization"] = f"token {api_key}"
+        logger.info(f'{datetime.now().strftime("%H:%M:%S")}: beginning upload')
+
+        results = []
+        for index, row in tqdm(
+            csv.transformed_df.iterrows(), total=csv.transformed_df.shape[0]
+        ):
+            u = UploadUtils(row)
+            valid, err = u.validate_processed_data_row()
+            result_row = row.to_dict()
+            if valid:
+                filename = os.path.basename(row.recording).split(".")[0]
+                files = []
+                for file in find_files_with_pattern(output_path, filename):
+                    key, error = get_last_n_directories(file, n=2)
+                    if error:
+                        continue
+                    checksum = u.calculate_file_checksum(file)
+                    files.append(
+                        {
+                            "index": index,
+                            "recording": file,
+                            "key": key,
+                            "checksum": checksum,
+                        }
+                    )
+                payload = u.generate_processed_payload(files)
+                res = u.post(api_key, url, headers, payload)
+                if res.get("upload_status") == "Success":
+                    result_row["upload_status"] = "Success"
+                    result_row["error"] = None
+
+                    # Handle S3 upload if presigned URL is provided
+                    for file_presigned in res.get("response", []):
+                        presigned = file_presigned.get("presigned")
+                        checksum = file_presigned.get("checksum")
+                        recording = file_presigned.get("recording")
+                        index = file_presigned.get("index")
+                        try:
+                            content_type, _ = mimetypes.guess_type(recording)
+                            if not content_type:
+                                content_type = "text/csv"
+                            with open(recording, "rb") as f:
+                                response = requests.put(
+                                    presigned,
+                                    data=f,
+                                    headers={
+                                        "x-amz-checksum-sha256": checksum,
+                                        "x-amz-sdk-checksum-algorithm": "SHA256",
+                                        "Content-Type": content_type,
+                                    },
+                                )
+                            if response.status_code == 200:
+                                result_row["upload_status"] = "Success"
+                            else:
+                                result_row["upload_status"] = "Failed"
+                                if result_row["error"] is None:
+                                    result_row["error"] = ""
+                                result_row["error"] = (
+                                    result_row["error"]
+                                    + "\n"
+                                    + f"S3 upload failed with status code {response.status_code} for file {recording}"
+                                )
+                        except Exception as ex:
+                            result_row["upload_status"] = "Failed"
+                            if result_row["error"] is None:
+                                result_row["error"] = ""
+                            result_row["error"] = result_row["error"] + "\n" + str(ex)
+                else:
+                    result_row["upload_status"] = "Failed"
+                    result_row["error"] = res.get("error")
+            else:
+                result_row["upload_status"] = "Failed"
+                result_row["error"] = f"{err}"
+            results.append(result_row)
+
+        results_df = pd.DataFrame(results)
+        return results_df
+    else:
+        logger.error(f'{datetime.now().strftime("%H:%M:%S")}: csv check failed')
+        logger.error(csv.errors)
+        return None