macsima2mc 1.2.3__tar.gz

macsima2mc-1.2.3/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, Schapiro Lab
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

macsima2mc-1.2.3/PKG-INFO

@@ -0,0 +1,64 @@
+Metadata-Version: 2.2
+Name: macsima2mc
+Version: 1.2.3
+Summary: stages MACSima tiles for registration with ASHLAR in MCMICRO.
+Author-email: Victor Perez <hhabraxas@gmail.com>, "Miguel A. Ibarra-Arellano" <c180l058j@mozmail.com>
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/SchapiroLabor/macsima2mc
+Project-URL: Issues, https://github.com/SchapiroLabor/macsima2mc/issues
+Keywords: mcmicro,multiplex imaging,proteomics,macsima,staging
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ome-types==0.5.2
+Requires-Dist: numpy
+Requires-Dist: tifffile
+Requires-Dist: pandas
+Requires-Dist: beautifulsoup4
+Requires-Dist: scikit-image
+Requires-Dist: BaSiCPy
+
+# Staging module for Miltenyi - MACSIMA to MCMICRO
+
+The macsima2mc.py script stages MACSima data sets for being registerd with ASHLAR in MCMICRO.
+
+The script takes as main input the path to the cycle folder that cointains the raw tiles generated by the Miltenyi-MACSima platform.  macsima2mc.py reads these tiles and distributes them into acquisition groups. i.e tiles with common rack, well, ROI and exposure levels will be written into an ome.tiff file with the necessary metadata for registration.  To be more precise two such ome.tiff files will be generated per cycle, one corresponds to the background signal and the other to the markers signal.
+ 
+## CLI
+### Required arguments
+| Argument|Long name|Type|Description|Default value|
+|---------|---------|----|-----------|-------------|
+| -i | string/path | --input | Absolute path to the the parent folder of the raw tiles, i.e. the cycle folder whose name follows the pattern X_Cycle_N,where N represents the cycle number | NA |
+| -o | string/path | --output | Absolute path to the directory in which the outputs will be saved. If the output directory doesn't exist it will be created. | NA | 
+
+### Optional arguments
+| Argument|Type|Long name| Description | Default value |
+|---------|----|---------|-------------|---------------|
+|-rm|string | --reference_marker | Name of the reference marker for registration|'DAPI'|
+|-osd|string | --output_subdir | String specifying the name of the subfolder in which the staged images will be saved.|'raw'|
+|-ic|boolean flag | --illumination_correction |Give this flag to apply illumination correction to all tiles, the illumination profiles are calculated with basicpy | FALSE |
+|-he|boolean flag | --hi_exposure_only |Give this flag to extract only the set of images with the highest exposure time|FALSE|
+|-rr|boolean flag | --remove_reference_marker |mark the removal of the reference marker ,e.g. DAPI, for all cycles except the first one |FALSE|
+|-qc|boolean flag | --qc_metrics | measure features of contrast, intensity and sharpness of each tile in the cycle and appends them to a table |FALSE|
+|-wt|boolean flag | --write_table | writes a table with the acquisition parameters, metadata and,if enabled, qc metrics of each tile. Table will be saved in --output/cycle_info   |FALSE|
+## Container usage
+### download container:
+- Docker
+```
+docker pull ghcr.io/schapirolabor/macsima2mc:v1.1.1
+```
+- Singularity
+```
+singularity pull docker://ghcr.io/schapirolabor/macsima2mc:v1.1.1
+```
+### Script execution
+
+- Singularity
+``` bash
+singularity exec --bind $path_to_your_local_cycle_folder:/mnt,$path_to_your_local_output_folder:/media --no-home $path_to_container python staging/macsima2mc/macsima2mc.py -i /mnt/$path_to_your_local_cycle_folder -o /media/$path_to_your_local_output_folder
+```
+

macsima2mc-1.2.3/README.md

@@ -0,0 +1,40 @@
+# Staging module for Miltenyi - MACSIMA to MCMICRO
+
+The macsima2mc.py script stages MACSima data sets for being registerd with ASHLAR in MCMICRO.
+
+The script takes as main input the path to the cycle folder that cointains the raw tiles generated by the Miltenyi-MACSima platform.  macsima2mc.py reads these tiles and distributes them into acquisition groups. i.e tiles with common rack, well, ROI and exposure levels will be written into an ome.tiff file with the necessary metadata for registration.  To be more precise two such ome.tiff files will be generated per cycle, one corresponds to the background signal and the other to the markers signal.
+ 
+## CLI
+### Required arguments
+| Argument|Long name|Type|Description|Default value|
+|---------|---------|----|-----------|-------------|
+| -i | string/path | --input | Absolute path to the the parent folder of the raw tiles, i.e. the cycle folder whose name follows the pattern X_Cycle_N,where N represents the cycle number | NA |
+| -o | string/path | --output | Absolute path to the directory in which the outputs will be saved. If the output directory doesn't exist it will be created. | NA | 
+
+### Optional arguments
+| Argument|Type|Long name| Description | Default value |
+|---------|----|---------|-------------|---------------|
+|-rm|string | --reference_marker | Name of the reference marker for registration|'DAPI'|
+|-osd|string | --output_subdir | String specifying the name of the subfolder in which the staged images will be saved.|'raw'|
+|-ic|boolean flag | --illumination_correction |Give this flag to apply illumination correction to all tiles, the illumination profiles are calculated with basicpy | FALSE |
+|-he|boolean flag | --hi_exposure_only |Give this flag to extract only the set of images with the highest exposure time|FALSE|
+|-rr|boolean flag | --remove_reference_marker |mark the removal of the reference marker ,e.g. DAPI, for all cycles except the first one |FALSE|
+|-qc|boolean flag | --qc_metrics | measure features of contrast, intensity and sharpness of each tile in the cycle and appends them to a table |FALSE|
+|-wt|boolean flag | --write_table | writes a table with the acquisition parameters, metadata and,if enabled, qc metrics of each tile. Table will be saved in --output/cycle_info   |FALSE|
+## Container usage
+### download container:
+- Docker
+```
+docker pull ghcr.io/schapirolabor/macsima2mc:v1.1.1
+```
+- Singularity
+```
+singularity pull docker://ghcr.io/schapirolabor/macsima2mc:v1.1.1
+```
+### Script execution
+
+- Singularity
+``` bash
+singularity exec --bind $path_to_your_local_cycle_folder:/mnt,$path_to_your_local_output_folder:/media --no-home $path_to_container python staging/macsima2mc/macsima2mc.py -i /mnt/$path_to_your_local_cycle_folder -o /media/$path_to_your_local_output_folder
+```
+

macsima2mc-1.2.3/macsima2mc/CLI.py

@@ -0,0 +1,84 @@
+import argparse
+import pathlib
+
+#---CLI-BLOCK---#
+def get_args():
+    """
+    This function parses the command line arguments and returns them as a namespace object.
+
+    returns: namespace object with the arguments.
+    """
+    parser=argparse.ArgumentParser()
+
+    #Mandatory arguments
+    parser.add_argument('-i',
+                    '--input',
+                    required=True,
+                    type=pathlib.Path,
+                    help='Path to the cycle folder'
+                    )
+
+    parser.add_argument('-o',
+                    '--output',
+                    required=True,
+                    type=pathlib.Path,
+                    help='Path where the stacks will be saved. If directory does not exist it will be created.'
+                    )
+    #Optional arguments
+    parser.add_argument('-rm',
+                    '--reference_marker',
+                    default='DAPI',
+                    help='string specifying the name of the reference marker'
+                    )
+    
+    parser.add_argument('-osd',
+                    '--output_subdir',
+                    default='raw',
+                    help='string specifying the name of the subfolder in which the staged images will be saved'
+                    )
+
+    parser.add_argument('-ic',
+                    '--illumination_correction',
+                    action='store_true',
+                    help='Applies illumination correction to all tiles, the illumination profiles are created with basicpy'
+                    )
+
+
+    parser.add_argument('-he',
+                    '--hi_exposure_only',
+                    action='store_true',
+                    help='Activate this flag to extract only the set of images with the highest exposure time.'
+                    )
+
+    parser.add_argument('-rr',
+                    '--remove_reference_marker',
+                    action='store_true',
+                    help='It will mark the removal of the reference markers in the markers.csv file except for the first cycle.  Use this when you \
+                        dont want to keep e.g. the DAPI images of the other cycles.'
+                    )
+    
+    parser.add_argument('-qc',
+                    '--qc_metrics',
+                    action='store_true',
+                    help='measure features of contrast, intensity and sharpness of each image'
+                    )
+    
+    parser.add_argument('-oqc',
+                    '--only_qc_file',
+                    action='store_true',
+                    help='skips the stacking of the tiles and only calculates the qc table. Still flag -wt is required to write the table. '
+                    )
+    
+    parser.add_argument('-wt',
+                    '--write_table',
+                    action='store_true',
+                    help='writes a table in --output/cycle_info. Content of table is acquisition parameters, metadata and, if enabled, qc metrics of each tile'
+                    )
+
+    args=parser.parse_args()
+
+    return args
+#---END_CLI-BLOCK---#
+
+
+

macsima2mc-1.2.3/macsima2mc/exceptions.py

@@ -0,0 +1,92 @@
+import numpy as np
+import pandas as pd
+
+
+def any_ref(mf_tuple,
+            ref_marker='DAPI'):
+    """
+    This function checks if the reference marker is present in the mf_tuple.
+    Args:
+        mf_tuple (tuple): tuple with the markers and filters
+        ref_marker (str): reference marker used for registration
+    Returns:
+        bool: True if the reference marker is present in the mf_tuple, False otherwise.
+    """
+
+    exist_ref = False
+    for m in mf_tuple:
+        if m[0] == ref_marker:
+            exist_ref = True
+            break
+    return exist_ref
+
+def miss_channel(chann_index,chans_meta,conformed_markers):
+    
+    if chans_meta[ chann_index ].size:
+        pass
+    else:
+
+        ref_index=np.argwhere([meta.size > 0 for meta in chans_meta ]).flatten()[0]
+        chann_meta=chans_meta[ ref_index ]
+        edit_feats={'marker':conformed_markers[ chann_index ][0], 
+                    'filter':conformed_markers[ chann_index ][1]}
+                    #'exposure?':}
+                    #how are conformed markers formed?
+    return True
+
+
+
+def at_roi(grouped,dimensions,ref_marker):
+    #for dimensions is expected that the last element of the list is the exposure_level
+    rows=[]
+    for index,frame in grouped:
+        marker_filter_map = list( frame.groupby(["marker","filter"]).indices.keys() )
+        ref_exists=any_ref(marker_filter_map,ref_marker)
+        subst_exp_level=index[-1]
+        rows.append( list(index) + [ ref_exists, subst_exp_level ] )
+
+    df=pd.DataFrame(rows,columns=dimensions+['ref_marker','aux_exp_level'])
+
+    frames=[]
+    for index,frame in df.groupby(dimensions[:-1]):
+        substitute=frame.loc[ frame['ref_marker']==True,'exposure_level'].max()
+        frame['aux_exp_level']=frame['aux_exp_level'].where(frame['ref_marker'],substitute)
+        frames.append(frame)
+
+    return pd.concat(frames)
+
+def at_tile(group,marker_filter_map):
+    rows=[]
+    for tile_id,frame in group.groupby(['tile']):
+        aux_array = [   frame.loc[ (frame['marker']==marker) & (frame['filter']==filt)].size
+                        for marker, filt in marker_filter_map 
+                    ]
+
+        aux_array =np.array(aux_array)==0
+        missing_channel=np.any(aux_array)
+
+        if missing_channel:
+            channel_indices=np.argwhere(aux_array).flatten()
+            #Each missing (marker,filter) pair will be separated by a colon ":", i.e. marker1,filter1:marker2,filter2
+            channel= ':'.join( [ ','.join(marker_filter_map[i]) for i in channel_indices] )
+        else:
+            channel=''
+            
+        qc_data=[tile_id[0],missing_channel,channel,tile_id[0]]
+        cols=['tile','missing_ch','channels','aux_tile']
+        rows.append(   qc_data )
+
+    df=pd.DataFrame(rows,columns=cols)
+    auxiliary_tile_index=df.loc[df['missing_ch']==False ][['tile']].values[0]
+    df.loc[ df['missing_ch']==True, ['aux_tile']]=auxiliary_tile_index
+    
+    return df 
+
+
+
+
+
+
+
+
+

macsima2mc-1.2.3/macsima2mc/illumination_corr.py

@@ -0,0 +1,64 @@
+from basicpy import BaSiC
+import numpy as np
+
+def indices_per_channel(total_imgs, no_of_channels):
+    """
+    This function creates a list of lists with the indices of the images in the stack per channel.
+    Args:
+        total_imgs (int): total number of images in the stack.
+        no_of_channels (int): number of channels.
+    Returns:
+        list: list of lists with the indices of the images in the stack per channel.
+    """
+
+    #total_imgs in the stack
+    img_indices = [list(range(ch,total_imgs,no_of_channels)) for ch in range(0, no_of_channels)]
+    return img_indices
+
+
+def extract_channel_imgs (stack,indices):
+    """
+    This function extracts the images in the stack per channel.
+    Args:
+        stack (np.array): stack of images.
+        indices (list): list of indices of the images in the stack per channel.
+    Returns:
+        np.array: stack of images per channel.
+    """
+
+    return stack[indices,:,:]
+
+def filter_out_blanks(channel_tiles):
+
+    signal_indices=[index for index in range(0,channel_tiles.shape[0]) if np.sum(channel_tiles[index,:,:])>0 ]
+    filtered_tiles=channel_tiles[signal_indices,:,:]
+    return filtered_tiles
+
+
+
+def apply_corr(uncorr_stack,no_of_channels):
+    """
+    This function applies the BaSiC algorithm to correct the illumination in the images.
+    Args:
+        uncorr_stack (np.array): stack of uncorrected images.
+        no_of_channels (int): number of channels.
+    Returns:
+        np.array: stack of corrected images.
+    """
+
+    corr_stack = np.zeros( uncorr_stack.shape,dtype=uncorr_stack.dtype )
+    total_imgs = uncorr_stack.shape[0]
+    indices = indices_per_channel(total_imgs, no_of_channels)
+    basic = BaSiC(get_darkfield=False, smoothness_flatfield=1.0, fitting_mode="approximate", sort_intensity=True)
+
+    for ind_list in indices:
+        uncorr_imgs = extract_channel_imgs(uncorr_stack, ind_list)
+        imgs_for_ffp=filter_out_blanks(uncorr_imgs)
+        #basic.fit(uncorr_imgs)
+        basic.fit(imgs_for_ffp)
+        ffp = basic.flatfield
+        corr_stack[ind_list,:,:] = np.uint16(np.clip(uncorr_imgs.astype(float) / ffp, 0, 65535))
+
+    return corr_stack
+
+

macsima2mc-1.2.3/macsima2mc/macsima2mc.py

@@ -0,0 +1,50 @@
+import tools
+from templates import macsima_pattern
+import CLI
+import mc_tools
+
+def main():
+    # Get arguments
+    args = CLI.get_args()
+
+    # Assign arguments to variables
+    input = args.input
+    output = args.output
+    ref = args.reference_marker
+    basicpy_corr = args.illumination_correction
+    out_folder_name = args.output_subdir
+
+    # Extract acquisition info of the cycle from file name,e.g. rack,well,markers,filters, etc.
+    cycle_info = tools.cycle_info(input, macsima_pattern(version=2), ref_marker= ref)
+    
+    # Extract and append ome metadata info contained in each file
+    cycle_info = tools.append_metadata( cycle_info )
+    
+    # Create stack
+    cycle_number=int(cycle_info['cycle'].unique()[0])
+    if args.only_qc_file:
+        pass
+    else:
+        output_dirs = tools.create_stack(cycle_info,
+                                     output,
+                                     ref_marker=ref,
+                                     hi_exp=args.hi_exposure_only,
+                                     ill_corr=basicpy_corr,
+                                     out_folder=out_folder_name)
+        # Save markers file in each output directory
+        for path in output_dirs:
+            mc_tools.write_markers_file(path,args.remove_reference_marker)
+
+    # Calculate and append ome metadata info contained in each file
+    if (args.qc_metrics or args.only_qc_file) :
+        import qc
+        cycle_info=qc.append_qc(cycle_info)
+
+    if args.write_table:
+        qc_output_dir=output / "cycle_info"
+        qc_output_dir.mkdir(parents=True, exist_ok=True)
+        cycle_info.to_csv( qc_output_dir / 'cycle_{c}.csv'.format( c=f'{ cycle_number:03d}' ), index=False )
+    
+
+if __name__ == '__main__':
+    main()

macsima2mc-1.2.3/macsima2mc/mc_tools.py

@@ -0,0 +1,110 @@
+from pathlib import Path
+from tools import extract_values
+import re
+import numpy as np
+from ome_types import from_tiff
+from pandas import DataFrame
+
+
+
+def flatten_list(lol):
+    """
+    This function flattens a list of lists.
+    Args:
+        lol (list): list of lists.
+    Returns:
+        list: flattened list.
+    """
+    return [x for xs in lol for x in xs]
+
+
+def markers_file():
+    """
+    This function creates a dictionary with the columns of the markers.csv file.
+    Returns:
+        dict: dictionary with the columns of the markers.csv file.
+    """
+    columns = {'channel_number': [],
+               'cycle_number': [],
+               'marker_name':[],
+               'Filter':[],
+               'background':[],
+               'exposure':[],
+               'remove':[]
+               }
+
+    return columns
+
+
+def get_patterns():
+    """
+    Dictionary of tuples where first element indicates the pattern to search and
+    second element is a boolean that indicates whether to transform the search output into a value or not
+    the keys of this dictionary should be a subset of the keys in the markers_file function.
+    Returns:
+        dict: dictionary with regular expressions to extract metadata from macsima filenames.
+    """
+
+    patterns = {
+        'cycle_number': (r"cycle-(.*?)-", True),
+        'marker_name' : (r"markers-(.*?)-filters",False),
+        'Filter'      : (r"-filters-(.*?).ome",False),
+        'background'  : (r"-src-(.*?)-" ,False)
+        }
+
+    return patterns
+
+
+def write_markers_file( data_path, rm_ref_marker,ref_marker='DAPI'):
+    """
+    This function writes the markers.csv file.
+    Args:
+        data_path (Path): path to the folder containing the images.
+        rm_ref_marker(Boolean): mark reference markers for removal except for the first one
+        ref_marker (str): reference marker.
+    Returns:
+        dict: dictionary with the columns of the markers.csv file.
+    """
+
+    img_paths = list(sorted( data_path.glob('*.tif*')))
+    mks_file = markers_file()
+    patt = get_patterns()
+
+    for img in img_paths:
+        img_name = [img.stem]
+        cycle_no = extract_values(patt['cycle_number'][0], img_name, number_cast=patt['cycle_number'][1])
+        background = extract_values(patt['background'][0], img_name, number_cast=patt['background'][1])
+        markers = extract_values(patt['marker_name'][0], img_name, number_cast=patt['marker_name'][1])[0].split('__')
+        filters = extract_values(patt['Filter'][0], img_name, number_cast=patt['Filter'][1])[0].split('__')
+        ome = from_tiff(img)
+
+        if background[0]=='B':
+            remove = len(markers)*['TRUE']
+            markers = ['bg_{c}_{m}-{f}'.format( c= f'{cycle_no[0]:03d}' , m=x , f=y ) for x,y in zip(markers,filters) ]
+            fmt_background = len(markers)*['']
+        else:
+            fmt_background = []
+            remove = len(markers)*['']
+            for x,y in zip(markers,filters):
+                if x == ref_marker:
+                    fmt_background.append('')
+                else:
+                    fmt_background.append(f'bg_{cycle_no[0]:03d}_{x}-{y}')
+
+        mks_file['cycle_number'].extend(len(markers)*cycle_no)
+        mks_file['marker_name'].extend(markers)
+        mks_file['Filter'].extend(filters)
+        mks_file['background'].extend(fmt_background)
+        mks_file['exposure'].extend([ome.images[0].pixels.planes[ch].exposure_time for ch,_ in enumerate(markers)])
+        mks_file['remove'].extend(remove)
+
+    mks_file['channel_number'] = list(range(1, 1 + len(mks_file['marker_name'])))
+    mks_file_df = DataFrame(mks_file)
+    if rm_ref_marker:
+        earliest_cycle=mks_file_df['cycle_number'].min()
+        condition=( mks_file_df['marker_name']== ref_marker ) & ( mks_file_df['cycle_number']>earliest_cycle ) 
+        mks_file_df.loc[ condition , ['remove'] ]='TRUE'
+    
+    mks_file_df.to_csv( data_path.parent.absolute() / 'markers.csv' , index=False )
+
+    return mks_file