alfrd 0.0.2__tar.gz

alfrd-0.0.2/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, Avinash Kumar
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

alfrd-0.0.2/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.1
+Name: alfrd
+Version: 0.0.2
+Summary: Automated Logical FRamework for script execution Dynamically(ALFRD)
+Home-page: https://github.com/avialxee/alfrd/
+Author: Avinash Kumar
+Author-email: avialxee@gmail.com
+Project-URL: Bug Tracker, https://github.com/avialxee/alfrd/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Intended Audience :: Science/Research
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: google-api-python-client
+Requires-Dist: google-auth-httplib2
+Requires-Dist: google-auth-oauthlib
+Requires-Dist: gspread
+Requires-Dist: gspread-formatting
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: psutil
+Requires-Dist: protobuf==3.19.6
+Provides-Extra: dev
+Requires-Dist: pytest>=3.7; extra == "dev"
+
+# ALFRD : Automated Logical FRamework for executing scripts Dynamically
+
+This program is written for the [SMILE project](smilescience.info) supported by the ERC starting grant, particularly with following purposes in mind:
+
+- Communicate with google spreadsheets and update progress periodically when required.
+- Run pipeline based on the spreadsheet progress/requirements.
+
+
+### 1. Create API credentials on Google Cloud
+
+
+A similar guide is available [here](https://www.cybrosys.com/blog/how-to-use-gspread-python-api)
+
+NOTE: In order to successfully create a google console project a billing detail is usually required. But the sheets API service is available for free, refer [here](https://developers.google.com/sheets/api/limits)
+
+- step 1:
+  Go to https://console.cloud.google.com/
+
+- step 2: 
+	click on the drop-down to create a new project 
+		- can choose organization or leave on default
+
+- step 3
+  Search in the top bar (or press / ) and type : "Google sheets api"
+  
+- step 4
+  In the search results - under Marketplace select the first result which should be the same : Google sheets api
+  
+- step 5
+  Enable the service In the product details page 
+  
+- step 6
+  Now select create credentials > Application Data
+  
+- step 7
+  Create an account name
+  Select create and continue
+  
+- step 8
+  Search and select "Editor" in role > Continue
+  
+- step 9
+  Skip next optional step
+  Select Done
+  
+- step 10
+  The Credentials are successfully created
+  Select "Credentials" on the left menu
+  
+- step 11
+  select/edit account that was just created
+  also copy the email address that is shown
+  
+- step 12
+  Select keys tab > Add keys > Create New Keys > JSON > save
+  
+- step 13
+  Go to the Google spreadsheet and "share" the sheet to the email address that was copied, as Editor.
+
+### 2. Installing
+
+- Download [ALFRD](https://github.com/avialxee/alfrd) and unzip / Or 
+   `git clone https://github.com/avialxee/alfrd`
+   
+- install alfrd
+    ```
+   cd alfrd/
+   pip install .
+    ```
+    this should install alfrd and all the dependencies automatically.
+    
+
+### 3. Using ALFRD
+
+##### Example 1 : Initializing and creating instance
+
+```python
+from alfrd.lib import GSC, LogFrame
+from alfrd.util import timeinmin, read_inputfile
+
+url='https://spreadsheet/link'
+worksheet='worksheet-name'
+
+gsc = GSC(url=url, wname=worksheet, key='path/to/json/file')      # default path for key = home/usr/.alfred
+_ = gsc.open()
+```
+
+##### Example 2: Inititalize the framework
+
+```python
+lf = LogFrame(gsc)
+lf.primary_colname          =   'FILE_NAME'
+```
+
+##### Example 3: Run - Iterate for each row
+
+```python
+from pathlib import Path
+import time, subprocess
+import glob
+
+count,failed=0,0
+allfiles=[]
+folder_for_fits ='path/to/allfits'
+
+allfiles.extend(glob.glob(f"{folder_for_fits}*/*fits"))
+
+for fitsfile in allfiles:
+    fitsfile_name = Path(fitsfile).name
+    lf.primary_value            =   fitsfile_name
+```
+
+##### Example 4: using column logic and adding values on success/failures
+
+```python
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    
+    lf.working_col          =   'Comment1'                                          # working column
+    print(lf.get_value())
+
+    result                  =   'test'
+    if result :
+        count               =   lf.put_value(result, count=count)                   
+    else:
+        failed              =   lf.put_value('failed: logfile_path', count=failed)
+    
+    print(lf.get_value())
+```
+
+##### Example 5: Script execution
+
+```python
+def run_picard(cmd):
+    t0 = time.time()
+    subprocess.run(cmd)
+    t1 = time.time()
+    td = timeinmin(t1-t0)
+    return td
+
+def scripted_picard_1(wd_ifolder, count, failed, col='fits to ms'):
+    """
+    converts fitsfile to ms file; checks if conversion was successful; skips if file exists; logs in the spreadsheet;
+    """
+    td=0
+    cmd=["picard",'-n','10',"-l","e",'--input',wd_ifolder]
+       
+    params, files, input_folder = read_inputfile(wd_ifolder, "observation.inp")
+    
+    if not Path(f"{wd_ifolder}").exists():
+        print('input folder missing..')
+    elif not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        td          =   run_picard(cmd)
+        count       =   lf.put_value(colname=col, value=td, count=count)
+    elif Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        print('skipped')
+
+    if not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        count-=1
+        failed      =   lf.put_value(colname=col, value='failed',count=failed)
+    return td, count, failed
+
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    lf.working_col          =   'Comment1'
+    ttaken, count, failed   =   scripted_picard_1(wd_ifolder='path/to/wd/input_template', count=count, failed=failed )
+
+print(lf.get_value(colname='fits to ms'))
+```
+
+##### Example 6: Update the sheet
+
+```python
+lf.update_sheet(count=count, failed=failed, csvfile='df_sheet.csv')                     # if updating the sheet fails, a copy of the dataframe is saved locally at the csvfile path.
+
+```
+
+
+##### Example 7: Conditional Formatting
+need to run only once.
+
+```python
+r1 = lf.create_rule(range='G2:G105', type='TEXT_CONTAINS' ,value='False', c='r')               # check if TSYS == False --> background color = red
+r2 = lf.create_rule(range='G2:G105', type="TEXT_CONTAINS", value='True', custom_clr=lf.create_color(0.56, 0.77, 0.49))
+r3 = lf.create_conditional_format(range='F2:F105', c='g', valtype='timeinmin')              # check if value in XXmYYs format --> background color = green
+
+r4 = lf.create_conditional_format(range='F2:F105', c='r', valtype='fail')                   # check if value contains fail --> background color = red
+r4 = lf.create_rule(range='F2:F105', type='TEXT_CONTAINS', c='r', value='fail')                # similar to above
+print(r1,r2,r3,r4)
+lf.add_conditional_format(r1, r2, r3, r4)
+```
+
+### Attribution
+
+When using ALFRD, please add a link to this repository in a footnote.
+
+### Acknowledgement
+
+ALFRD was developed within the "Search for Milli-Lenses" (SMILE) project. SMILE has received funding from the European Research Council (ERC) under the HORIZON ERC Grants 2021 programme (grant agreement No. 101040021).

alfrd-0.0.2/README.md

@@ -0,0 +1,197 @@
+# ALFRD : Automated Logical FRamework for executing scripts Dynamically
+
+This program is written for the [SMILE project](smilescience.info) supported by the ERC starting grant, particularly with following purposes in mind:
+
+- Communicate with google spreadsheets and update progress periodically when required.
+- Run pipeline based on the spreadsheet progress/requirements.
+
+
+### 1. Create API credentials on Google Cloud
+
+
+A similar guide is available [here](https://www.cybrosys.com/blog/how-to-use-gspread-python-api)
+
+NOTE: In order to successfully create a google console project a billing detail is usually required. But the sheets API service is available for free, refer [here](https://developers.google.com/sheets/api/limits)
+
+- step 1:
+  Go to https://console.cloud.google.com/
+
+- step 2: 
+	click on the drop-down to create a new project 
+		- can choose organization or leave on default
+
+- step 3
+  Search in the top bar (or press / ) and type : "Google sheets api"
+  
+- step 4
+  In the search results - under Marketplace select the first result which should be the same : Google sheets api
+  
+- step 5
+  Enable the service In the product details page 
+  
+- step 6
+  Now select create credentials > Application Data
+  
+- step 7
+  Create an account name
+  Select create and continue
+  
+- step 8
+  Search and select "Editor" in role > Continue
+  
+- step 9
+  Skip next optional step
+  Select Done
+  
+- step 10
+  The Credentials are successfully created
+  Select "Credentials" on the left menu
+  
+- step 11
+  select/edit account that was just created
+  also copy the email address that is shown
+  
+- step 12
+  Select keys tab > Add keys > Create New Keys > JSON > save
+  
+- step 13
+  Go to the Google spreadsheet and "share" the sheet to the email address that was copied, as Editor.
+
+### 2. Installing
+
+- Download [ALFRD](https://github.com/avialxee/alfrd) and unzip / Or 
+   `git clone https://github.com/avialxee/alfrd`
+   
+- install alfrd
+    ```
+   cd alfrd/
+   pip install .
+    ```
+    this should install alfrd and all the dependencies automatically.
+    
+
+### 3. Using ALFRD
+
+##### Example 1 : Initializing and creating instance
+
+```python
+from alfrd.lib import GSC, LogFrame
+from alfrd.util import timeinmin, read_inputfile
+
+url='https://spreadsheet/link'
+worksheet='worksheet-name'
+
+gsc = GSC(url=url, wname=worksheet, key='path/to/json/file')      # default path for key = home/usr/.alfred
+_ = gsc.open()
+```
+
+##### Example 2: Inititalize the framework
+
+```python
+lf = LogFrame(gsc)
+lf.primary_colname          =   'FILE_NAME'
+```
+
+##### Example 3: Run - Iterate for each row
+
+```python
+from pathlib import Path
+import time, subprocess
+import glob
+
+count,failed=0,0
+allfiles=[]
+folder_for_fits ='path/to/allfits'
+
+allfiles.extend(glob.glob(f"{folder_for_fits}*/*fits"))
+
+for fitsfile in allfiles:
+    fitsfile_name = Path(fitsfile).name
+    lf.primary_value            =   fitsfile_name
+```
+
+##### Example 4: using column logic and adding values on success/failures
+
+```python
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    
+    lf.working_col          =   'Comment1'                                          # working column
+    print(lf.get_value())
+
+    result                  =   'test'
+    if result :
+        count               =   lf.put_value(result, count=count)                   
+    else:
+        failed              =   lf.put_value('failed: logfile_path', count=failed)
+    
+    print(lf.get_value())
+```
+
+##### Example 5: Script execution
+
+```python
+def run_picard(cmd):
+    t0 = time.time()
+    subprocess.run(cmd)
+    t1 = time.time()
+    td = timeinmin(t1-t0)
+    return td
+
+def scripted_picard_1(wd_ifolder, count, failed, col='fits to ms'):
+    """
+    converts fitsfile to ms file; checks if conversion was successful; skips if file exists; logs in the spreadsheet;
+    """
+    td=0
+    cmd=["picard",'-n','10',"-l","e",'--input',wd_ifolder]
+       
+    params, files, input_folder = read_inputfile(wd_ifolder, "observation.inp")
+    
+    if not Path(f"{wd_ifolder}").exists():
+        print('input folder missing..')
+    elif not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        td          =   run_picard(cmd)
+        count       =   lf.put_value(colname=col, value=td, count=count)
+    elif Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        print('skipped')
+
+    if not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        count-=1
+        failed      =   lf.put_value(colname=col, value='failed',count=failed)
+    return td, count, failed
+
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    lf.working_col          =   'Comment1'
+    ttaken, count, failed   =   scripted_picard_1(wd_ifolder='path/to/wd/input_template', count=count, failed=failed )
+
+print(lf.get_value(colname='fits to ms'))
+```
+
+##### Example 6: Update the sheet
+
+```python
+lf.update_sheet(count=count, failed=failed, csvfile='df_sheet.csv')                     # if updating the sheet fails, a copy of the dataframe is saved locally at the csvfile path.
+
+```
+
+
+##### Example 7: Conditional Formatting
+need to run only once.
+
+```python
+r1 = lf.create_rule(range='G2:G105', type='TEXT_CONTAINS' ,value='False', c='r')               # check if TSYS == False --> background color = red
+r2 = lf.create_rule(range='G2:G105', type="TEXT_CONTAINS", value='True', custom_clr=lf.create_color(0.56, 0.77, 0.49))
+r3 = lf.create_conditional_format(range='F2:F105', c='g', valtype='timeinmin')              # check if value in XXmYYs format --> background color = green
+
+r4 = lf.create_conditional_format(range='F2:F105', c='r', valtype='fail')                   # check if value contains fail --> background color = red
+r4 = lf.create_rule(range='F2:F105', type='TEXT_CONTAINS', c='r', value='fail')                # similar to above
+print(r1,r2,r3,r4)
+lf.add_conditional_format(r1, r2, r3, r4)
+```
+
+### Attribution
+
+When using ALFRD, please add a link to this repository in a footnote.
+
+### Acknowledgement
+
+ALFRD was developed within the "Search for Milli-Lenses" (SMILE) project. SMILE has received funding from the European Research Council (ERC) under the HORIZON ERC Grants 2021 programme (grant agreement No. 101040021).

alfrd-0.0.2/setup.cfg

@@ -0,0 +1,24 @@
+[metadata]
+name = alfrd
+version = 0.0.2
+author = Avinash Kumar
+author_email = avialxee@gmail.com
+description = Automated Logical FRamework for script execution Dynamically(ALFRD)
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/avialxee/alfrd/
+project_urls = 
+	Bug Tracker = https://github.com/avialxee/alfrd/issues
+
+[options]
+package_dir = 
+	= src
+packages = find:
+
+[options.packages.find]
+where = src
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

alfrd-0.0.2/setup.py

@@ -0,0 +1,38 @@
+from setuptools import setup
+
+with open("README.md", "r") as rdme:
+    desc = rdme.read()
+
+setup(
+    name = 'alfrd',
+    version = '0.0.2',
+    url='https://github.com/avialxee/alfrd/',
+    author='Avinash Kumar',
+    author_email='avialxee@gmail.com',
+    description='Automated Logical FRamework for script execution Dynamically(ALFRD)',
+    py_modules = ["alfrd"],
+    package_dir={'':'src'},
+    classifiers=["Programming Language :: Python :: 3",
+                 "Programming Language :: Python :: 3.6",
+                 "Programming Language :: Python :: 3.7",
+                 "Programming Language :: Python :: 3.8",
+                 "License :: OSI Approved :: BSD License",
+                 "Intended Audience :: Science/Research",
+                 ],
+    long_description=desc,
+    long_description_content_type = "text/markdown",
+    install_requires=[ "google-api-python-client", "google-auth-httplib2", "google-auth-oauthlib", "gspread", "gspread-formatting",
+                        "pandas", "numpy", "psutil",
+                        "protobuf==3.19.6"
+                      ],
+    extras_require = {
+        "dev" : ["pytest>=3.7",
+        ]
+    },
+     entry_points={ 
+        "console_scripts": [
+            "alfrd=alfrd:cli"
+        ],
+    },
+
+)

alfrd-0.0.2/src/alfrd/__init__.py

@@ -0,0 +1,4 @@
+c={"x":"\033[0m","g":"\033[32m", "r":"\033[31m", "b":"\033[34m","c":"\033[36m","w":"\033[0m", "y":"\033[33m"}
+
+def cli():
+    pass

alfrd-0.0.2/src/alfrd/lib.py

@@ -0,0 +1,221 @@
+import gspread
+from google.oauth2.service_account import Credentials
+import pandas as pd
+import re
+from pathlib import Path
+import warnings
+from alfrd import c
+from gspread_formatting import ConditionalFormatRule, GridRange, BooleanCondition, BooleanRule, CellFormat, Color, get_conditional_format_rules
+
+class GSC:
+    """
+    Creates instance of google Google Spreadsheet Credential to open and update a worksheet
+    """
+    def __init__(self, sid='', url='', key=f"{Path().home()}/.alfred/credentials.json", wid=0, wname=''):
+        """
+        if sid is empty, uses url to get the spreadsheet id
+        """
+        self.sid            =   sid
+        self.url            =   url
+        self.key            =   key
+        self.wid            =   wid
+        self.wname          =   wname
+        self.authorized     =   False
+        self.scopes         =   ["https://www.googleapis.com/auth/spreadsheets"]
+        self.creds          =   Credentials.from_service_account_file(key, scopes=self.scopes)
+        
+    def auth(self):
+        self.client         =   gspread.authorize(self.creds)
+        self.authorized     =   True
+
+    def open(self):
+        if not self.authorized: self.auth()
+        if not self.sid: 
+            regex = "([\w-]){44}"
+            sid_match = re.search(regex,self.url)
+            self.sid = str(sid_match.group())
+
+        self.spreadsheet    =   self.client.open_by_key(self.sid)
+        self.sheet          =   self.spreadsheet.get_worksheet(self.w) if not self.wname else self.spreadsheet.worksheet(self.wname)
+        self.df             =   pd.DataFrame(self.sheet.get_all_records(numericise_ignore=['all']))
+        print(f"{c['g']}Success!{c['x']}")
+        return self.df
+
+    def update(self, dataframe):
+        self.sheet.update([dataframe.columns.values.tolist()] + dataframe.values.tolist())
+        print(f"{c['g']}Updated!{c['x']}")
+
+class LogFrame:
+    """
+    Input
+    ---
+
+    :gsc:               Google Spreadsheet Credentials instance
+    :primary_value:     the unique identifier of the row corrosponding to the primary_colname
+    :primary_colname:     primary column name for unique identifier
+    :registered:        keeps count of success and failed script runs in a tuple (count_success, count_failed)
+
+    """
+    def __init__(self, gsc , primary_value='',  primary_colname='FILE_NAME'):
+        self.gsc                =   gsc
+        self.df_sheet           =   self.gsc.df
+        self.primary_value      =   primary_value
+        self.primary_colname    =   primary_colname
+        self.working_col        =   ''
+        
+        self.registered         =   0,0         # (count_success, count_failed)
+        self.color ={'g': Color(red=0.56,green=0.77,blue=0.49),
+                'r': Color(red=0.8784314,green=0.4,blue=0.4),
+                'rh': Color(red=0.71,green=0.13,blue=0.0),
+                'rl': Color(red=0.98,green=0.63,blue=0.57),
+                'gl': Color(red=0.42,green=0.86,blue=0.31),
+                'gh': Color(red=0.42,green=0.60,blue=0.42)}
+
+    def col_data(self, colname='', data='', count=0, force=False, chk_colname=''):
+        """
+        program to change the column data of dataframe
+        checks if primary_value exist and then change the value for empty cell with the given data.
+        
+        set primary_colname - primary column name to find unique values corrosponding to the primary_value e.g fitsfile name
+
+        Input
+        ---
+
+        :df_sheet:          pandas dataframe to work on
+        :primary_value:     the unique identifier of the row corrosponding to the primary_colname
+        :colname:           column name to alter data
+        :data:              data to fill in the corrosponding column for the corrosponding identifier row
+        :count:             iterative count
+        :force:             to force change the data in the column even if the cell is not empty.
+        :primary_colname:     primary column name for unique identifier
+        :chk_colname:       column name to check value for, if this is given no data is changed.
+
+        Returns
+        ---
+
+        count if chk_column is empty; 
+
+        (count, column_value) corrosponding to the `chk_colname`
+        """
+        colname = self.working_col if not colname else colname
+        if not self.primary_value : print(f"{c['y']}No primary value given{c['x']}")
+        primary_col_values     = self.df_sheet[self.primary_colname].str.strip()
+        if (not force) or (primary_col_values.isin([self.primary_value]).any() and not self.df_sheet.loc[primary_col_values==self.primary_value,colname].count()):
+            
+            if chk_colname :
+                if self.df_sheet.loc[primary_col_values==self.primary_value,chk_colname].count(): 
+                    val = str(self.df_sheet.loc[primary_col_values==self.primary_value,chk_colname].values[0]).strip()
+                    count+=1
+                    return count, val 
+                else:
+                    return count, ''
+            else:
+                count+=1 
+                self.df_sheet.loc[primary_col_values==self.primary_value,colname] = data
+        else:
+            print("not updating", self.primary_value, f"{self.df_sheet.loc[primary_col_values==self.primary_value,colname].values}")
+        return count
+    
+    def isval_unique(self, colname=''):
+        """
+        checks if the colname has unique values and returns boolean
+        """
+        colname = self.working_col if not colname else colname
+        _, colv = self.col_data(colname='', data='', count=0, chk_colname=colname)
+        c = self.df_sheet[colname].value_counts()[colv]
+        r = False if int(c)!=1 else True
+        return r
+    
+    def get_value(self, colname=''):
+        """
+        gives the cell value for the colname returns string value
+        """
+        colname = self.working_col if not colname else colname
+        _, colv = self.col_data(colname='', data='', count=0, chk_colname=colname)
+        r = str(colv).strip()
+        return r
+    
+    def isvalue(self, value, colname=''):
+        """
+        Returns boolean by matching value in cell
+        """
+        v = self.get_value(colname)
+        return str(value) == str(v)
+    
+    def put_value(self, value, colname='', count=0):
+        colname = self.working_col if not colname else colname
+        count = self.col_data(colname=colname, data=value, count=count)
+        return count
+    
+    def update_sheet(self, count, failed, comment_col='Comment4', csvfile = 'df_sheet.csv'):
+        """
+        updates the google sheet if there is atleast one new count/failed count for the update
+        """
+        try:
+            if count - self.registered[0] or failed - self.registered[1]:
+                self.gsc.update(self.df_sheet)
+                self.registered = count, failed            
+            else:
+                print('skipped')
+        except Exception as e:
+            print(f'failed to update on google sheet: {e}')
+            failed  =   self.col_data(colname=comment_col, data=f'failed:{e}', count=failed)
+            self.df_sheet.to_csv(csvfile)
+
+    def create_conditional_format(self, range, c='g', valtype='timeinmin', custom_clr=None):
+        clr = self.color[c] if not custom_clr else custom_clr
+        rule ={
+                'timeinmin' : ConditionalFormatRule(
+            ranges=[GridRange.from_a1_range(f'{range}', self.gsc.sheet)],
+            booleanRule=BooleanRule(
+                condition=BooleanCondition(
+                        type='CUSTOM_FORMULA',values=[f'=AND(ISNUMBER(SEARCH("m", {range})), ISNUMBER(SEARCH("s", {range})))']),
+                format=CellFormat(backgroundColor=clr,
+                ))),
+                'True' :  ConditionalFormatRule(
+            ranges=[GridRange.from_a1_range(f'{range}', self.gsc.sheet)],
+            booleanRule=BooleanRule(
+                condition=BooleanCondition(
+                        type='TEXT_CONTAINS',values=[f'True']),
+                format=CellFormat(backgroundColor=clr,
+                ))),
+                'False' :  ConditionalFormatRule(
+            ranges=[GridRange.from_a1_range(f'{range}', self.gsc.sheet)],
+            booleanRule=BooleanRule(
+                condition=BooleanCondition(
+                        type='TEXT_CONTAINS',values=[f'False']),
+                format=CellFormat(backgroundColor=clr,
+                ))),
+                'fail' :  ConditionalFormatRule(
+            ranges=[GridRange.from_a1_range(f'{range}', self.gsc.sheet)],
+            booleanRule=BooleanRule(
+                condition=BooleanCondition(
+                        type='TEXT_CONTAINS',values=[f'fail']),
+                format=CellFormat(backgroundColor=clr,
+                ))),
+        }
+        return rule[valtype]
+
+    def create_rule(self, range, type='TEXT_CONTAINS', value='True',  c='g', custom_clr=None):
+        clr = self.color[c] if not custom_clr else custom_clr
+        return ConditionalFormatRule(
+            ranges=[GridRange.from_a1_range(f'{range}', self.gsc.sheet)],
+            booleanRule=BooleanRule(
+                condition=BooleanCondition(
+                        type=type, values=[value]),
+                format=CellFormat(backgroundColor=clr,
+                )))
+    
+    def create_color(self, r=0.56,g=0.77,b=0.49):
+        return Color(red=r,green=g,blue=b)
+    
+    def add_conditional_format(self, *new_rules):
+        rules = get_conditional_format_rules(self.gsc.sheet)
+        for rule in new_rules:
+            rules.append(rule)
+        rules.save()
+
+    def clear_conditional_format(self,):
+        rules = get_conditional_format_rules(self.gsc.sheet)
+        rules.clear()
+        rules.save()

alfrd-0.0.2/src/alfrd/util.py

@@ -0,0 +1,177 @@
+import numpy as np
+from pathlib import Path
+import os
+import subprocess, glob, shutil, time
+from collections import defaultdict
+
+def read_inputfile(folder,inputfile='.inp'):
+    """Read the input file and return a dictionary with the parameters.
+
+    Returns
+    ---
+
+    (params, files, input_folder)
+
+    """
+    params = defaultdict(list)
+    input_folder= None
+    # params['array_type']='generic'
+    
+    files=glob.glob(f'{folder}/*{inputfile}',recursive=True)
+    if not files: files=glob.glob(f'{folder}/*/*{inputfile}',recursive=False)
+    if not files: files=glob.glob(f'{folder}/*/*/*{inputfile}',recursive=False)
+    if files:
+        input_folder = str(Path(files[-1]).parent) + '/'
+        for filepath in files:
+            if '.inp' in filepath:
+                with open(filepath,'r') as f:
+                    pr=f.read().splitlines()
+                    for p in pr:
+                        if '#' in p:
+                            continue
+                        elif '=' in p:
+                            k,v=p.split('=')
+                            try:
+                                v=int(v)
+                            except:
+                                try:
+                                    v=float(v)
+                                except:
+                                    v=str(v).strip()     
+                                    v = v.lower() == 'true' if (any(boolv == v.lower() for boolv in ['true', 'false'])) else v
+                                        
+
+                                    # elif '//' in v or ';' in v: v=re.split('\\|;', str(v))
+                            params[k.strip()]=v
+    return params, files, input_folder
+
+
+def find_size(fitsfile):
+    size = np.round(Path(fitsfile).stat().st_size/(1024*1024),2)
+    if size >= 1024.0 : 
+        size = size/1024
+        size = f"{np.round(size, 2)} GB"
+    else:
+        size = f"{np.round(size, 2)} MB"
+    return size
+
+def find_project(fitsfile):
+    project     =   str(Path(fitsfile).parent.name)
+    return str(project)
+
+def build_path(filepath):
+    """
+    This builds new file path by adding _{n+1} if there exists one with similar name,
+    """
+    opt = filepath
+    if Path(filepath).exists():
+        numb = 1
+        while Path(filepath).exists():
+            filepath = "{0}_{2}{1}".format(
+                *(Path(opt).parent / Path(opt).stem, Path(opt).suffix,numb))
+            try :
+                if Path(filepath).exists():
+                    numb += 1 
+            except:
+                pass
+    return filepath
+
+def symlink_bywd(wd, fitsfile, create=True):
+    rawsymlink          =   f"{wd}/raw/{Path(fitsfile).name}" 
+    Path(f"{wd}/raw").mkdir(parents=True, exist_ok=True)
+    if create : os.symlink(fitsfile, rawsymlink)
+    return rawsymlink
+
+def dir_for_project(fitsfile, tdir='/data/avi/reductions/100test/', ifolder='/data/avi/gh/picard/src/picard/input_template/', create=True, splitted=False):
+    """
+    looks for wd using project name
+    """
+    new                 =   True
+    segment             =   find_project(fitsfile)
+    wd                  =   f"{tdir}{segment}/wd"
+    wd_ifolder          =   None
+    lookfile            =   fitsfile if not splitted else f"{str(Path(fitsfile).stem)}_split{Path(fitsfile).suffix}"
+    if create:
+        if not Path(f"{wd}/raw/{Path(lookfile).name}").exists():
+            wd                  =   build_path(wd)
+            
+        try:
+            rawsymlink = symlink_bywd(wd, fitsfile)
+               
+            wd_ifolder          =   f'{wd}/input_template/'
+            if not Path(wd_ifolder).exists():shutil.copytree(ifolder,wd_ifolder)
+        except Exception as e:
+            print(f"exists? : {segment} : {e}")
+            new             =   False
+    
+        new             =   False
+    else:
+        possible_file = glob.glob(f'{wd}*/raw/{Path(lookfile).name}')
+        
+        if len(possible_file):
+            wd_ifolder = Path(possible_file[0]).parent.parent / "input_template"
+            new = False
+    return wd_ifolder, new
+
+def del_extra_wdfolder(wd_ifolder, fitsfile):
+    """
+    useful when for a project name there are more folders created by mistakes,
+    e.g wd_1/ wd_2/ instead of just wd/
+
+    Note: assumes wd_1, wd_2 etc are created by mistake
+    """
+    if 'wd_' in str(wd_ifolder):
+        subprocess.run(['rm','-rf', str(Path(wd_ifolder).parent.parent)])
+        wd_ifolder, new= dir_for_project(fitsfile)
+        print('created',wd_ifolder)
+
+def latest_file(path: Path, pattern: str = "*"):
+    """
+    to get the last file that was generated, this can be useful for getting any new logfiles.
+    """
+    files = path.glob(pattern)
+    lastf = Path('')
+    
+    try:
+        lastf = max(files, key=lambda x: x.stat().st_ctime)
+    finally:
+        return lastf
+    
+def timeinmin(td):
+    """
+    convert timdedelta in XXmYYs format
+    """
+    tdm,tds     =   0,0
+    if td>=60:
+        tdm     =   td//60
+        tds     =   td%60
+    else:
+        tds     =   td
+    ret_time    =   f"{int(tdm)}m{np.round(tds,1)}s"
+
+    return ret_time
+
+def del_fl(ifolder, count, fl='*ms*', rm=False):
+    """
+    delete files from the input folder
+    """
+    filefound = glob.glob(f"{ifolder}/../{fl}")
+    if len(filefound):
+        cmd = ['rm','-rf']
+        cmd.extend(str(Path(ifolder.parent) / " ".join(filefound)).split(' '))
+        print(cmd)
+        count+=1
+        if rm: subprocess.run(cmd)
+    return count
+
+def build_logpath(wd_ifolder):
+    """
+    builds rpicard and casa style log files for the current time
+    """
+    thisdate = time.strftime('%F-%T', time.gmtime())
+    thisdate = thisdate.replace(':', '_')
+    
+    errlogf = Path(wd_ifolder).parent / f'mpi_and_err.out_{thisdate}'
+    casalogf = Path(wd_ifolder).parent / f'casa.log_{thisdate}'
+    
+    return str(errlogf), str(casalogf)

alfrd-0.0.2/src/alfrd.egg-info/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.1
+Name: alfrd
+Version: 0.0.2
+Summary: Automated Logical FRamework for script execution Dynamically(ALFRD)
+Home-page: https://github.com/avialxee/alfrd/
+Author: Avinash Kumar
+Author-email: avialxee@gmail.com
+Project-URL: Bug Tracker, https://github.com/avialxee/alfrd/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Intended Audience :: Science/Research
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: google-api-python-client
+Requires-Dist: google-auth-httplib2
+Requires-Dist: google-auth-oauthlib
+Requires-Dist: gspread
+Requires-Dist: gspread-formatting
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: psutil
+Requires-Dist: protobuf==3.19.6
+Provides-Extra: dev
+Requires-Dist: pytest>=3.7; extra == "dev"
+
+# ALFRD : Automated Logical FRamework for executing scripts Dynamically
+
+This program is written for the [SMILE project](smilescience.info) supported by the ERC starting grant, particularly with following purposes in mind:
+
+- Communicate with google spreadsheets and update progress periodically when required.
+- Run pipeline based on the spreadsheet progress/requirements.
+
+
+### 1. Create API credentials on Google Cloud
+
+
+A similar guide is available [here](https://www.cybrosys.com/blog/how-to-use-gspread-python-api)
+
+NOTE: In order to successfully create a google console project a billing detail is usually required. But the sheets API service is available for free, refer [here](https://developers.google.com/sheets/api/limits)
+
+- step 1:
+  Go to https://console.cloud.google.com/
+
+- step 2: 
+	click on the drop-down to create a new project 
+		- can choose organization or leave on default
+
+- step 3
+  Search in the top bar (or press / ) and type : "Google sheets api"
+  
+- step 4
+  In the search results - under Marketplace select the first result which should be the same : Google sheets api
+  
+- step 5
+  Enable the service In the product details page 
+  
+- step 6
+  Now select create credentials > Application Data
+  
+- step 7
+  Create an account name
+  Select create and continue
+  
+- step 8
+  Search and select "Editor" in role > Continue
+  
+- step 9
+  Skip next optional step
+  Select Done
+  
+- step 10
+  The Credentials are successfully created
+  Select "Credentials" on the left menu
+  
+- step 11
+  select/edit account that was just created
+  also copy the email address that is shown
+  
+- step 12
+  Select keys tab > Add keys > Create New Keys > JSON > save
+  
+- step 13
+  Go to the Google spreadsheet and "share" the sheet to the email address that was copied, as Editor.
+
+### 2. Installing
+
+- Download [ALFRD](https://github.com/avialxee/alfrd) and unzip / Or 
+   `git clone https://github.com/avialxee/alfrd`
+   
+- install alfrd
+    ```
+   cd alfrd/
+   pip install .
+    ```
+    this should install alfrd and all the dependencies automatically.
+    
+
+### 3. Using ALFRD
+
+##### Example 1 : Initializing and creating instance
+
+```python
+from alfrd.lib import GSC, LogFrame
+from alfrd.util import timeinmin, read_inputfile
+
+url='https://spreadsheet/link'
+worksheet='worksheet-name'
+
+gsc = GSC(url=url, wname=worksheet, key='path/to/json/file')      # default path for key = home/usr/.alfred
+_ = gsc.open()
+```
+
+##### Example 2: Inititalize the framework
+
+```python
+lf = LogFrame(gsc)
+lf.primary_colname          =   'FILE_NAME'
+```
+
+##### Example 3: Run - Iterate for each row
+
+```python
+from pathlib import Path
+import time, subprocess
+import glob
+
+count,failed=0,0
+allfiles=[]
+folder_for_fits ='path/to/allfits'
+
+allfiles.extend(glob.glob(f"{folder_for_fits}*/*fits"))
+
+for fitsfile in allfiles:
+    fitsfile_name = Path(fitsfile).name
+    lf.primary_value            =   fitsfile_name
+```
+
+##### Example 4: using column logic and adding values on success/failures
+
+```python
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    
+    lf.working_col          =   'Comment1'                                          # working column
+    print(lf.get_value())
+
+    result                  =   'test'
+    if result :
+        count               =   lf.put_value(result, count=count)                   
+    else:
+        failed              =   lf.put_value('failed: logfile_path', count=failed)
+    
+    print(lf.get_value())
+```
+
+##### Example 5: Script execution
+
+```python
+def run_picard(cmd):
+    t0 = time.time()
+    subprocess.run(cmd)
+    t1 = time.time()
+    td = timeinmin(t1-t0)
+    return td
+
+def scripted_picard_1(wd_ifolder, count, failed, col='fits to ms'):
+    """
+    converts fitsfile to ms file; checks if conversion was successful; skips if file exists; logs in the spreadsheet;
+    """
+    td=0
+    cmd=["picard",'-n','10',"-l","e",'--input',wd_ifolder]
+       
+    params, files, input_folder = read_inputfile(wd_ifolder, "observation.inp")
+    
+    if not Path(f"{wd_ifolder}").exists():
+        print('input folder missing..')
+    elif not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        td          =   run_picard(cmd)
+        count       =   lf.put_value(colname=col, value=td, count=count)
+    elif Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        print('skipped')
+
+    if not Path(f"{wd_ifolder}/../{params['ms_name']}").exists():
+        count-=1
+        failed      =   lf.put_value(colname=col, value='failed',count=failed)
+    return td, count, failed
+
+if lf.isvalue(value='True', colname='TSYS') and lf.isval_unique('Project'):
+    lf.working_col          =   'Comment1'
+    ttaken, count, failed   =   scripted_picard_1(wd_ifolder='path/to/wd/input_template', count=count, failed=failed )
+
+print(lf.get_value(colname='fits to ms'))
+```
+
+##### Example 6: Update the sheet
+
+```python
+lf.update_sheet(count=count, failed=failed, csvfile='df_sheet.csv')                     # if updating the sheet fails, a copy of the dataframe is saved locally at the csvfile path.
+
+```
+
+
+##### Example 7: Conditional Formatting
+need to run only once.
+
+```python
+r1 = lf.create_rule(range='G2:G105', type='TEXT_CONTAINS' ,value='False', c='r')               # check if TSYS == False --> background color = red
+r2 = lf.create_rule(range='G2:G105', type="TEXT_CONTAINS", value='True', custom_clr=lf.create_color(0.56, 0.77, 0.49))
+r3 = lf.create_conditional_format(range='F2:F105', c='g', valtype='timeinmin')              # check if value in XXmYYs format --> background color = green
+
+r4 = lf.create_conditional_format(range='F2:F105', c='r', valtype='fail')                   # check if value contains fail --> background color = red
+r4 = lf.create_rule(range='F2:F105', type='TEXT_CONTAINS', c='r', value='fail')                # similar to above
+print(r1,r2,r3,r4)
+lf.add_conditional_format(r1, r2, r3, r4)
+```
+
+### Attribution
+
+When using ALFRD, please add a link to this repository in a footnote.
+
+### Acknowledgement
+
+ALFRD was developed within the "Search for Milli-Lenses" (SMILE) project. SMILE has received funding from the European Research Council (ERC) under the HORIZON ERC Grants 2021 programme (grant agreement No. 101040021).

alfrd-0.0.2/src/alfrd.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+setup.cfg
+setup.py
+src/alfrd/__init__.py
+src/alfrd/lib.py
+src/alfrd/util.py
+src/alfrd.egg-info/PKG-INFO
+src/alfrd.egg-info/SOURCES.txt
+src/alfrd.egg-info/dependency_links.txt
+src/alfrd.egg-info/entry_points.txt
+src/alfrd.egg-info/requires.txt
+src/alfrd.egg-info/top_level.txt

alfrd-0.0.2/src/alfrd.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

alfrd-0.0.2/src/alfrd.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+alfrd = alfrd:cli

alfrd-0.0.2/src/alfrd.egg-info/requires.txt

@@ -0,0 +1,12 @@
+google-api-python-client
+google-auth-httplib2
+google-auth-oauthlib
+gspread
+gspread-formatting
+pandas
+numpy
+psutil
+protobuf==3.19.6
+
+[dev]
+pytest>=3.7

alfrd-0.0.2/src/alfrd.egg-info/top_level.txt

@@ -0,0 +1 @@
+alfrd