maradoner 0.24.1__tar.gz

maradoner-0.24.1/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.4
+Name: maradoner
+Version: 0.24.1
+Summary: Variance-adjusted estimation of motif activities.
+Home-page: https://github.com/autosome-ru/maradoner
+Author: Georgy Meshcheryakov
+Author-email: iam@georgy.top
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: pip>=24.0
+Requires-Dist: typer>=0.13
+Requires-Dist: numpy>=2.1
+Requires-Dist: jax<0.5
+Requires-Dist: jaxlib<0.5
+Requires-Dist: matplotlib>=3.5
+Requires-Dist: pandas>=2.2
+Requires-Dist: scipy>=1.14
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: datatable>=1.0.0
+Requires-Dist: dill>=0.3.9
+Requires-Dist: rich>=12.6.0
+Requires-Dist: tqdm>=4.0
+Requires-Dist: scikit-learn>=1.6
+Requires-Dist: tables>=3.10
+Requires-Dist: sympy>=1.12
+Requires-Dist: seaborn>=0.12
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+
+**MARADONER**
+
+# MARADONER: Motif Activity Response Analysis Done Right
+
+MARADONER is a tool for analyzing motif activities using promoter expression data. It provides a streamlined workflow to estimate parameters, predict deviations, and export results in a tabular form.
+
+## Basic Workflow
+
+
+A typical MARADONER analysis session involves running commands sequentially for a given project:
+
+1.  **`create`**: Initialize the project. This step parses your input files (promoter expression, motif loadings, optional motif expression, and sample groupings), performs initial filtering, and sets up the project's internal data structures.
+    ```bash
+    # Example: Initialize a project named 'my_project'
+    maradoner create my_project path/to/expression.tsv path/to/loadings.tsv --sample-groups path/to/groups.json [other options...]
+    ```
+    *   Input files are typically tabular (.tsv, .csv), potentially compressed.
+    *   You only need to provide input data files at this stage.
+
+2.  **`fit`**: Estimate the model's variance parameters and mean motif activities using the data prepared by `create`.
+    ```bash
+    maradoner fit my_project [options...]
+    ```
+
+3.  **`predict`**: Estimate the *deviations* of motif activities from their means for each sample or group, based on the parameters estimated by `fit`.
+    ```bash
+    maradoner predict my_project [options...]
+    ```
+
+4.  **`export`**: Save the final results, including estimated motif activities (mean + deviations), parameter estimates, goodness-of-fit statistics, and potentially statistical test results (like ANOVA) to a specified output folder.
+    ```bash
+    maradoner export my_project path/to/output_folder [options...]
+    ```
+
+## Other Useful Commands
+
+*   **`gof`**: After `fit`, calculate Goodness-of-Fit statistics (like Fraction of Variance Explained or Correlation) to evaluate how well the model components explain the observed expression data.
+    ```bash
+    maradoner gof my_project [options...]
+    ```
+*   **`select-motifs`**: If you provided multiple loading matrices in `create` (e.g., from different databases) with unique postfixes, this command helps select the single "best" variant for each motif based on statistical criteria. The output is a list of motif names intended to be used with the `--motif-filename` option in a subsequent `create` run.
+    ```bash
+    maradoner select-motifs my_project best_motifs.txt
+    # Then, potentially re-run create using the generated list:
+    # maradoner create my_project_filtered ... --motif-filename best_motifs.txt
+    ```
+*   **`generate`**: Create a synthetic dataset with known properties for testing or demonstration purposes.
+    ```bash
+    maradoner generate path/to/synthetic_data_output [options...]
+    ```
+
+## Getting Help
+
+Each command has various options for customization. To see the full list of commands and their detailed options, use the `--help` flag:
+
+```bash
+maradoner --help
+maradoner create --help
+maradoner fit --help
+# and so on for each command

maradoner-0.24.1/README.md

@@ -0,0 +1,61 @@
+
+**MARADONER**
+
+# MARADONER: Motif Activity Response Analysis Done Right
+
+MARADONER is a tool for analyzing motif activities using promoter expression data. It provides a streamlined workflow to estimate parameters, predict deviations, and export results in a tabular form.
+
+## Basic Workflow
+
+
+A typical MARADONER analysis session involves running commands sequentially for a given project:
+
+1.  **`create`**: Initialize the project. This step parses your input files (promoter expression, motif loadings, optional motif expression, and sample groupings), performs initial filtering, and sets up the project's internal data structures.
+    ```bash
+    # Example: Initialize a project named 'my_project'
+    maradoner create my_project path/to/expression.tsv path/to/loadings.tsv --sample-groups path/to/groups.json [other options...]
+    ```
+    *   Input files are typically tabular (.tsv, .csv), potentially compressed.
+    *   You only need to provide input data files at this stage.
+
+2.  **`fit`**: Estimate the model's variance parameters and mean motif activities using the data prepared by `create`.
+    ```bash
+    maradoner fit my_project [options...]
+    ```
+
+3.  **`predict`**: Estimate the *deviations* of motif activities from their means for each sample or group, based on the parameters estimated by `fit`.
+    ```bash
+    maradoner predict my_project [options...]
+    ```
+
+4.  **`export`**: Save the final results, including estimated motif activities (mean + deviations), parameter estimates, goodness-of-fit statistics, and potentially statistical test results (like ANOVA) to a specified output folder.
+    ```bash
+    maradoner export my_project path/to/output_folder [options...]
+    ```
+
+## Other Useful Commands
+
+*   **`gof`**: After `fit`, calculate Goodness-of-Fit statistics (like Fraction of Variance Explained or Correlation) to evaluate how well the model components explain the observed expression data.
+    ```bash
+    maradoner gof my_project [options...]
+    ```
+*   **`select-motifs`**: If you provided multiple loading matrices in `create` (e.g., from different databases) with unique postfixes, this command helps select the single "best" variant for each motif based on statistical criteria. The output is a list of motif names intended to be used with the `--motif-filename` option in a subsequent `create` run.
+    ```bash
+    maradoner select-motifs my_project best_motifs.txt
+    # Then, potentially re-run create using the generated list:
+    # maradoner create my_project_filtered ... --motif-filename best_motifs.txt
+    ```
+*   **`generate`**: Create a synthetic dataset with known properties for testing or demonstration purposes.
+    ```bash
+    maradoner generate path/to/synthetic_data_output [options...]
+    ```
+
+## Getting Help
+
+Each command has various options for customization. To see the full list of commands and their detailed options, use the `--help` flag:
+
+```bash
+maradoner --help
+maradoner create --help
+maradoner fit --help
+# and so on for each command

maradoner-0.24.1/maradoner/__init__.py

@@ -0,0 +1,38 @@
+# -*- coding: utf-8 -*-
+__version__ = '0.24.1'
+import importlib
+
+
+__min_reqs__ = [
+            'pip>=24.0',
+            'typer>=0.13',
+            'numpy>=2.1',
+            'jax<0.5',
+            'jaxlib<0.5',
+            'matplotlib>=3.5',
+            'pandas>=2.2',
+            'scipy>=1.14',
+            'statsmodels>=0.14',
+            'datatable>=1.0.0' ,
+            'dill>=0.3.9',
+            'rich>=12.6.0',
+            'tqdm>=4.0',
+            'scikit-learn>=1.6',
+            'tables>=3.10',
+            'sympy>=1.12',
+            'seaborn>=0.12'
+           ]
+
+def versiontuple(v):
+    return tuple(map(int, (v.split("."))))
+
+def check_packages():
+    for req in __min_reqs__:
+        try:
+            module, ver = req.split(' @').split('>=')
+            ver = versiontuple(ver)
+            v = versiontuple(importlib.import_module(module).__version__)
+        except (AttributeError, ValueError):
+            continue
+        if v < ver:
+            raise ImportError(f'Version of the {module} package should be at least {ver} (found: {v}).')

maradoner-0.24.1/maradoner/create.py

@@ -0,0 +1,199 @@
+from .utils import logger_print, openers
+from .dataset_filter import filter_lowexp
+from .drist import DRIST
+import multiprocessing
+import scipy.stats as st
+import datatable as dt
+import pandas as pd
+import numpy as np
+import dill
+import json
+import os
+import re
+
+
+def drist_it(B: pd.DataFrame, Y: pd.DataFrame, test_chromosomes: list[str] = None,
+             share_function: bool = False, optimizer='jacobi'):
+    if test_chromosomes:
+        pattern = re.compile(r'chr([0-9XYM]+|\d+)')
+        
+        test_chromosomes = set(test_chromosomes)
+        mask = [pattern.search(p).group() in test_chromosomes for i, p in enumerate(Y.index)]
+        mask = ~np.array(mask, dtype=bool)
+    else:
+        mask = np.ones(len(B), dtype=bool)
+    Y = Y.values
+    Y = Y - Y.mean(axis=1, keepdims=True)
+    Bt = B.values[mask, :]
+    Y = Y[mask, :]
+    drist = DRIST(max_iter=1000, verbose=True, share_function=share_function,
+                  optimizer=optimizer)
+    B.values[mask, :] = drist.fit_transform(Bt, Y)
+    if not np.all(mask):
+        B.values[~mask, :] = drist.transform(B.values[~mask, :])
+        
+    B = B - B.min()
+    return B
+
+
+def transform_loadings(df, mode: str, zero_cutoff=1e-9, prom_inds=None, Y=None,
+                       test_chromosomes: list[str] = None):
+    stds = df.std()
+    drop_inds = (stds == 0) | np.isnan(stds)
+    if prom_inds is not None:
+        df = df.loc[prom_inds, ~drop_inds]
+    else:
+        df = df.loc[:, ~drop_inds]
+    # if not mode or mode == 'none':
+    #     df[df < zero_cutoff] = 0
+    #     df = (df - df.min(axis=None)) / (df.max(axis=None) - df.min(axis=None))
+    if mode == 'ecdf':
+        for j in range(len(df.columns)):
+            v = df.iloc[:, j]
+            df.iloc[:, j] = st.ecdf(v).cdf.evaluate(v)
+    elif mode in ('esf',):
+        for j in range(len(df.columns)):
+            v = df.iloc[:, j]
+            v = st.ecdf(v).sf.evaluate(v)
+            t = np.unique(v)[1]
+            v[v < t] = t
+            df.iloc[:, j] = -np.log(v)
+        # if mode == 'drist':
+        #     df = drist_it(df, Y, test_chromosomes=test_chromosomes)
+    elif mode.startswith('drist'):
+        df = drist_it(df, Y, test_chromosomes=test_chromosomes,
+                      share_function=mode.endswith('un'))
+    elif mode == 'none':
+        pass
+    elif mode:
+        raise Exception('Unknown transformation mode ' + str(mode))
+    return df
+
+def create_project(project_name: str, promoter_expression_filename: str, loading_matrix_filenames: list[str],
+                   motif_expression_filenames=None, loading_matrix_transformations=None, sample_groups=None, motif_postfixes=None,
+                   promoter_filter_lowexp_cutoff=0.95, promoter_filter_plot_filename=None, promoter_filter_max=True,
+                   motif_names_filename=None, n_jobs:float = 0.5, compression='raw', dump=True, verbose=True):
+    if not os.path.isfile(promoter_expression_filename):
+        raise FileNotFoundError(f'Promoter expression file {promoter_expression_filename} not found.')
+    if type(loading_matrix_filenames) is str:
+        loading_matrix_filenames = [loading_matrix_filenames]
+    for mx_name in loading_matrix_filenames:
+        if not os.path.isfile(mx_name):
+            raise FileNotFoundError(f'Loading matrix file {mx_name} not found.')
+    if motif_expression_filenames:
+        if type(motif_expression_filenames) is str:
+            motif_expression_filenames = [motif_expression_filenames]
+        for exp_name in motif_expression_filenames:
+            if not os.path.isfile(exp_name):
+                raise FileNotFoundError(f'Motif expresion file {exp_name} not found.')
+    if type(sample_groups) is str:
+        with open(sample_groups, 'r') as f:
+            if sample_groups.endswith('.json'):
+                sample_groups = json.load(f)
+            else:
+                sample_groups = dict()
+                for line in f:
+                    items = line.split()
+                    sample_groups[items[0]] = items[1:]
+    if motif_names_filename is not None:
+        with open(motif_names_filename, 'r') as f:
+            motif_names = list()
+            for line in f:
+                line = line.strip().split()
+                for item in line:
+                    if item:
+                        motif_names.append(item)
+    else:
+        motif_names = None
+    cpu_count = multiprocessing.cpu_count()
+    if n_jobs < 1 and n_jobs > 0:
+        n_jobs = int(n_jobs * cpu_count)
+    elif n_jobs <= 0:
+        n_jobs = cpu_count
+    logger_print('Reading dataset...', verbose)
+    promoter_expression = dt.fread(promoter_expression_filename, nthreads=n_jobs).to_pandas()
+    promoter_expression = promoter_expression.set_index(promoter_expression.columns[0])
+    
+    if sample_groups:
+        cols = set()
+        for vals in sample_groups.values():
+            cols.update(vals)
+        cols = list(cols)
+        promoter_expression = promoter_expression[cols]
+    
+    proms = promoter_expression.index
+    sample_names = promoter_expression.columns
+    loading_matrices = [dt.fread(f, nthreads=n_jobs).to_pandas() for f in loading_matrix_filenames]
+    loading_matrices = [df.set_index(df.columns[0]).loc[proms] for df in loading_matrices]
+    if loading_matrix_transformations is None or type(loading_matrix_transformations) is str:
+        loading_matrix_transformations = [loading_matrix_transformations] * len(loading_matrices)
+    else:
+        if len(loading_matrix_transformations) == 1:
+            loading_matrix_transformations = [loading_matrix_transformations[0]] * len(loading_matrices)
+        elif len(loading_matrix_transformations) != len(loading_matrices):
+            raise Exception(f'Total number of loading matrices is {len(loading_matrices)}, but the number of transformations is '
+                            f'{len(loading_matrix_transformations)}.')
+    
+    logger_print('Filtering promoters of low expression...', verbose)
+    inds, weights = filter_lowexp(promoter_expression, cutoff=promoter_filter_lowexp_cutoff, fit_plot_filename=promoter_filter_plot_filename,
+                                  max_mode=promoter_filter_max)
+    promoter_expression = promoter_expression.loc[inds]
+    proms = promoter_expression.index
+    test_chromosomes  = list() # ['chr2', 'chr15']
+    loading_matrices = [transform_loadings(df, mode, prom_inds=inds, test_chromosomes=test_chromosomes,
+                                           Y=promoter_expression) for df, mode in zip(loading_matrices, loading_matrix_transformations)]
+    if motif_postfixes is not None:
+        for mx, postfix in zip(loading_matrices, motif_postfixes):
+            mx.columns = [f'{c}_{postfix}' for c in mx.columns]
+    if motif_expression_filenames:
+        motif_expression = [dt.fread(f, nthreads=n_jobs).to_pandas() for f in motif_expression_filenames]
+        motif_expression = [df.set_index(df.columns[0]) for df in motif_expression]
+        if motif_postfixes is not None:
+            for mx, postfix in zip(motif_expression, motif_postfixes):
+                mx.index = [f'{c}_{postfix}' for c in mx.index]
+        if sample_groups:
+            if len(set(motif_expression[0].columns) & set(sample_groups)) == len(sample_groups):
+                for i in range(len(motif_expression)):
+                    mx = motif_expression[i]
+                    for group, cols in sample_groups.items():
+                        for col in cols:
+                            mx[col] = mx[group]
+                    mx = mx.drop(sorted(sample_groups), axis=1)     
+        motif_expression = [df.loc[mx.columns, sample_names] for df, mx in zip(motif_expression, loading_matrices)]
+        motif_expression = pd.concat(motif_expression, axis=0)
+    else:
+        motif_expression = None
+    loading_matrices = pd.concat(loading_matrices, axis=1)
+    if motif_names is not None:
+        motif_names = list(set(motif_names) & set(loading_matrices.columns))
+        loading_matrices = loading_matrices[motif_names]
+    proms = list(promoter_expression.index)
+    sample_names = list(promoter_expression.columns)
+    motif_names = list(loading_matrices.columns)
+    loading_matrices = loading_matrices.values
+    promoter_expression = promoter_expression.values
+    if motif_expression is not None:
+        motif_expression = motif_expression.values
+    if not sample_groups:
+        sample_groups = {n: [i] for i, n in enumerate(sample_names)}
+    else:
+        sample_groups = {n: sorted([sample_names.index(i) for i in inds]) for n, inds in sample_groups.items()}
+    res = {'expression': promoter_expression, 
+           'loadings': loading_matrices,
+           'motif_expression': motif_expression,
+           'motif_postfixes': motif_postfixes,
+           'promoter_names': proms,
+           'sample_names': sample_names,
+           'motif_names': motif_names,
+           'weights': weights,
+           'groups': sample_groups}
+    if dump:
+        folder = os.path.split(project_name)[0]
+        name = os.path.split(project_name)[-1]
+        for file in os.listdir(folder if folder else None):
+            if file.startswith(f'{name}.') and file.endswith(tuple(openers.keys())):
+                os.remove(os.path.join(folder, file))
+        logger_print('Saving project...', verbose)
+        with openers[compression](f'{project_name}.init.{compression}', 'wb') as f:
+            dill.dump(res, f)
+    return res

maradoner-0.24.1/maradoner/dataset_filter.py

@@ -0,0 +1,152 @@
+import jax.numpy as jnp
+from jax import jit, grad
+from jax.scipy.stats import norm
+from jax.scipy.special import logsumexp, logit, expit
+import pandas as pd
+import numpy as np
+from scipy.optimize import minimize
+from functools import partial
+from sklearn.mixture import GaussianMixture
+
+def compute_leftmost_probability(Y):
+    Y = Y.reshape(-1, 1)
+    gmm = GaussianMixture(n_components=2, random_state=0)
+    gmm.fit(Y)
+    
+    means = gmm.means_.flatten()
+    leftmost_component_index = np.argmin(means)
+    probas = gmm.predict_proba(Y)
+    leftmost_probs = probas[:, leftmost_component_index]
+    
+    return leftmost_probs, gmm
+
+def normax_logpdf(x: jnp.ndarray, mu: float, sigma: float, n: int):
+    x = (x - mu) / sigma
+    return jnp.log(n) - jnp.log(sigma) + norm.logpdf(x) + (n - 1) * norm.logcdf(x)
+
+def logmixture(x: jnp.ndarray, mus: jnp.ndarray, sigmas: jnp.ndarray, w: float, n: int):
+    logpdf1 = normax_logpdf(x, mus[0], sigmas[0], n)
+    logpdf2 = normax_logpdf(x, mus[1], sigmas[1], n)
+    w = jnp.array([w, 1 - w]).reshape(-1,1)
+    logpdf = jnp.array([logpdf1, logpdf2])
+    return logsumexp(logpdf, b=w, axis=0)
+
+
+def transform(params, forward=True):
+    mu = params[:2]
+    sigma = params[2:4]
+    w = params[-1:]
+    if forward:
+        sigma = sigma ** 2
+        w = expit(w)
+    else:
+        sigma = sigma ** 0.5
+        w = logit(w)
+    return jnp.concatenate([mu, sigma, w])
+
+def loglik(params: jnp.ndarray, x: jnp.ndarray, n: int):
+    params = transform(params)
+    mu = params[:2]
+    sigma = params[2:4]
+    w = params[-1]
+    return -logmixture(x, mu, sigma, w, n).sum()
+
+def filter_lowexp(expression: pd.DataFrame, cutoff=0.95, component_limit=0.6, max_mode=True,
+                  fit_plot_filename=None, plot_dpi=200):
+    expression = (expression - expression.mean()) / expression.std()
+    if not max_mode:
+        expression = expression.mean(axis=1).values
+        probs, gmm = compute_leftmost_probability(expression)
+        inds = probs < (1-cutoff)
+        if fit_plot_filename:
+            import matplotlib.pyplot as plt
+            from matplotlib.collections import LineCollection
+            import seaborn as sns
+            x = np.array(sorted(expression))
+            pdf = np.exp(gmm.score_samples(expression[:, None]))
+            points = np.array([x, pdf]).T.reshape(-1, 1, 2)
+            segments = np.concatenate([points[:-1], points[1:]], axis=1)
+            plt.figure(dpi=plot_dpi, )
+            sns.histplot(expression, stat='density', color='grey')
+            lc = LineCollection(segments, cmap='winter')
+            lc.set_array(probs)
+            lc.set_linewidth(3)
+            line = plt.gca().add_collection(lc)
+            plt.colorbar(line)
+            plt.xlabel('Standardized expression')
+            plt.tight_layout()
+            plt.savefig(fit_plot_filename)
+        return inds, probs
+        
+    expression_max = expression.max(axis=1).values
+
+    mu = [-1.0, 0.0]
+    sigmas = [1.0, 1.0]
+    w = [0.5]
+    x0 = jnp.array(mu + sigmas + w)
+    x0 = transform(x0, False)
+    fun = jit(partial(loglik, x=expression_max, n=expression.shape[1]))
+    jac = jit(grad(fun))
+    res = minimize(fun, x0, jac=jac)
+    
+    params = transform(res.x)
+    mu = params[:2]
+    sigma = params[2:4]
+    w = params[-1]
+    
+    mode1 = minimize(lambda x: -normax_logpdf(x, mu[0], sigma[0], n=expression.shape[1]), x0=[0.0]).x
+    mode2 = minimize(lambda x: -normax_logpdf(x, mu[1], sigma[1], n=expression.shape[1]), x0=[0.0]).x
+    if mode1 > mode2:
+        mu = mu[::-1]
+        sigma = sigma[::-1]
+        w = 1 - w
+    
+    inds = np.argsort(expression_max)
+    inds_inv = np.empty_like(inds, dtype=int)
+    inds_inv[inds] = np.arange(len(inds))
+    x = expression_max[inds]
+    logpdf1 = normax_logpdf(x, mu[0], sigma[0], n=expression.shape[1])
+    logpdf2 = normax_logpdf(x, mu[1], sigma[1], n=expression.shape[1])
+    pdf1 = jnp.exp(logpdf1)
+    pdf2 = jnp.exp(logpdf2)
+    ws = np.array(pdf1 / ((w * pdf1 + (1-w)*pdf2)) * w)
+    
+    if float(w) > component_limit:
+        ws[:] = 1.0
+    else:
+        ws = 1 - ws
+        if x[ws >= 0.5].mean() < x[ws < 0.5].mean():
+            ws = 1 - ws
+        j = np.argmax(ws)
+        l = np.argmin(ws)
+        ws[j:] = 1.0
+        ws[:l] = 0.0
+    
+    k = 0
+    for k in range(len(ws)):
+        if ws[k] >= 1.0-cutoff:
+            break
+    if fit_plot_filename:
+        import matplotlib.pyplot as plt
+        from matplotlib.collections import LineCollection
+        import seaborn as sns
+        pdf = jnp.exp(logmixture(x, mu, sigma, w, n=expression.shape[1]))
+        points = np.array([x, pdf]).T.reshape(-1, 1, 2)
+        segments = np.concatenate([points[:-1], points[1:]], axis=1)
+        plt.figure(dpi=plot_dpi, )
+        sns.histplot(expression_max, stat='density', color='grey')
+        lc = LineCollection(segments, cmap='winter')
+        lc.set_array(ws)
+        lc.set_linewidth(3)
+        line = plt.gca().add_collection(lc)
+        plt.colorbar(line)
+        plt.xlabel('Standardized expression')
+        plt.tight_layout()
+        plt.savefig(fit_plot_filename)
+    ws = ws[inds_inv]
+    inds = np.ones(len(expression), dtype=bool)
+    inds[:k] = False
+    # print(inds)
+    # inds[:] = 1
+    inds = inds[inds_inv]
+    return inds, ws