dietnetwork 0.3.0__tar.gz

dietnetwork-0.3.0/.flake8

@@ -0,0 +1,6 @@
+[flake8]
+# Matches black's line length, so a line black accepts is never a lint error.
+max-line-length = 100
+extend-ignore = E203,W503,E501,F841,F541
+# HussinLab's code is kept identical to upstream and never linted here.
+extend-exclude = Dietnet_train_and_evaluate,.venv

dietnetwork-0.3.0/.gitignore

@@ -0,0 +1,36 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# Private exp files
+.ipynb_checkpoints
+#*.ipynb
+*.csv
+run.sh
+make_attributions_run.sh
+Dietnet/Interpretability/experiment_scripts/*.sh
+
+# Debugging and testing
+DATA/
+EXPERIMENT_01/
+Test_dev/
+Dietnet/tests/filenames.py
+
+# Model weights
+Models_1KGP_24populations/
+
+# Local environments and tool state
+.venv/
+.claude/*
+# The project's Claude Code settings and the data-egress hook they run are
+# shared; everything else under .claude/ (local settings, worktrees) is not.
+!.claude/settings.json
+!.claude/hooks/
+.coverage
+coverage.xml
+# Written by the CI install step
+requirements-ci.txt
+
+# The documentation site (uv run mkdocs build)
+site/
+.cache/

dietnetwork-0.3.0/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
+this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.3.0] - 2026-10-26
+
+Rebuilt on HussinLab/DIETNETWORK `main` (`5481556e`). The training code is
+upstream's, unmodified. The `dietnet` command line, model packages and
+inference are re-applied around it, replacing this repository's 2022 copy of
+the core, which trained differently from the published model (its main
+dropout was fixed at 0.5; the published config uses 0.1).
+
+### Added
+
+- `dietnet build-embedding`: the auxiliary network's SNP features from several
+  sources: the published genotype frequencies, any precomputed per-SNP table
+  (oriented to the counted allele), and random or shuffled controls, with
+  per-block centring and PCA. Packages record which blocks a model used.
+- `dietnet match`: prepare a cohort on CPU; `predict` with the same `--work-dir`
+  goes straight to inference (for a GPU job).
+- `generate-embedding` and `train` refuse classes that have no training
+  samples in some fold: upstream's embedding is NaN for them (or silently
+  drops their columns).
+- `reproduction/`: the published split (seed 124, verified against Zenodo), the
+  published models' held-out accuracy (0.9214) pinned by a slow test, and the
+  SLURM jobs that retrained all 15 (0.9282).
+- `dietnet train --model mlp`: upstream's MLP baseline (no auxiliary network).
+- `dietnet alphagenome`: AlphaGenome variant effects for every SNP through the
+  hosted API, resumable, as a feature table for `build-embedding`
+  (`pip install 'dietnetwork[alphagenome]'`).
+- `dietnet create-dataset`: PLINK straight to upstream's HDF5 schema, identical
+  to upstream's `create_dataset.py` output and without the text round-trip.
+- `dietnet published`: the 15 published models (Zenodo
+  doi:10.5281/zenodo.16943452) as model packages.
+- Allele-aware SNP matching. Dosages that count the other allele are flipped,
+  the other strand is recognised, and A/T and C/G SNPs are kept only on an
+  exact match. The published D/(D−|C|) missing-SNP scale is applied.
+- Biobank-scale prediction: the cohort is matched once into a contiguous HDF5
+  under `--work-dir`, memory-mapped, and shared by every model.
+- `dietnet info`, which reports the upstream commit and whether its files are
+  unmodified.
+- Parity tests against upstream: dataset builder, `predict` vs
+  `test_independent_dataset.py`, and the published models vs
+  `Dietnet_inference/inference.py`.
+- Tooling as in manifold_genetics: uv lock, black/isort/flake8, pre-commit with
+  a sensitive-files guard and gitleaks, CI on Python 3.10–3.13 and macOS.
+- Released as `dietnetwork` on PyPI: `release.yml` publishes on a version tag
+  through trusted publishing, after checking the tag, a dated CHANGELOG entry,
+  the artefacts' contents and a clean-room install (`dietnet info` must find
+  the bundled upstream code byte for byte the pinned commit).
+- Tests of what would be published: package metadata
+  (`tests/unit/test_package_metadata.py`) and the sdist and wheel themselves
+  (`tests/integration/test_sdist_contents.py`: no untracked file, nothing the
+  sensitive-files check would flag, upstream code identical), run in a CI
+  `package` job.
+- A Claude Code data-egress hook (`.claude/hooks/egress_guard.py`, run by the
+  project's `.claude/settings.json`): agents are asked for approval before any
+  copy from a remote host, any dump of a genotype or data file over ssh, or any
+  upload of one to a host that is not an Alliance cluster, and cannot commit or
+  push with `--no-verify`.
+- `LICENSE` (BSD 3-Clause) and `NOTICE`, which keeps HussinLab's upstream code
+  outside that licence; `py.typed`.
+- A documentation site (mkdocs-material, `docs/`, the `docs` extra), built
+  strictly on every pull request and deployed to GitHub Pages from `master`.
+  Its tutorial runs the whole pipeline on the `kgp_mini` fixture on every
+  build, and a test keeps `docs/cli.md` naming exactly the CLI's commands and
+  options.
+
+### Fixed
+
+- `build-embedding` with a `class_genotype_freq` block failed when
+  `--dataset` or `--partition` was a relative path: upstream's script runs from
+  its own directory and could not find them.
+
+### Changed
+
+- `partition`, `generate-embedding`, `compute-input-stats` and `train` now run
+  upstream's scripts. Partitions are label-balanced by default, as in the paper.
+- Model packages (format 2) hold upstream's checkpoint, the fold's embedding and
+  statistics, a training header and metadata (task, normalization, seed, fold,
+  upstream commit). Weights load strictly.
+- The predictions TSV has a header: `sample`, `prediction`, `probability`,
+  `model_agreement` (regression: `prediction`, `sd_across_models`).
+- `check` takes `--packages` (held-out accuracy per model) or `--predictions`
+  with `--labels`.
+- Commands use `--plink` and `--out`. The old CLI's `--exp-path`-relative file
+  names are gone.
+- `dietnet alphagenome` sends one request per SNP and retries transient API
+  errors with backoff. About 2% of requests fail at random with gRPC
+  `INTERNAL`, and the client's batch call failed a whole 200-SNP chunk on any
+  one of them, so a run never got past its first chunk.
+
+### Removed
+
+- The 2022 core (`Dietnet/`): model, training loop and dataset code, replaced by
+  upstream's.
+- Upstream files the CLI does not run: deprecated code, experiments, upstream's
+  inference scripts and `create_dataset.py` (listed in the manifest).
+- `preprocess-plink`, `analyze-population` and the Dropbox model presets.
+  Allele-aware matching covers the first; the others return if needed.

dietnetwork-0.3.0/Dietnet_train_and_evaluate/compute_input_features_statistics.py

@@ -0,0 +1,221 @@
+import argparse
+import os
+import multiprocessing as mp
+import time
+from datetime import datetime
+
+import numpy as np
+
+import h5py
+
+
+def main():
+    start_time = time.time()
+    args = parse_args()
+
+    # Flag to compute only mean (True or False)
+    mean_only = args.mean_only
+
+    # Load indices of every fold
+    partition_data = np.load(args.partition, allow_pickle=True)
+    indices_byfold = partition_data['folds_indexes']
+
+    # Dataset
+    dataset_file = h5py.File(args.dataset, 'r')
+
+    # Compute inp feat stats by fold
+    means_by_fold = []
+    sds_by_fold = []
+    for fold in range(len(indices_byfold)):
+        print('\n---\nComputing input features statistic of fold', str(fold))
+        fold_start_time = time.time()
+
+        # Indices of the fold
+        fold_indices = indices_byfold[fold]
+
+        # Genotypes of training samples
+        train_indices = np.sort(fold_indices[0]) # sort is a hdf5 requirement
+        x_train = dataset_file['inputs'][train_indices]
+
+        # Compute statistics of every features in parallel
+        ncpus = args.ncpus
+        print('Computing input features statistics of {} SNPs for {} '
+              'samples using {} cpu(s)'.format(
+              x_train.shape[1], x_train.shape[0], ncpus))
+
+        pool = mp.Pool(processes=ncpus) #this starts ncpus nb of processes
+
+        # Results that will be returned by processes
+        results = []
+
+        # The statistics of every snp feature is computed by a different process
+        nb_snps = x_train.shape[1]
+        for i in range(nb_snps):
+            # Compute snps mean
+            if mean_only :
+                results.append(
+                        pool.apply_async(
+                            compute_feature_mean, args=(i, x_train[:,i])))
+
+            # Compute mean and standard deviation (sd)
+            else:
+                results.append(
+                        pool.apply_async(
+                            compute_feature_mean_and_sd, args=(i, x_train[:,i])))
+
+        # Get order (snp nb) and mean of features from multiprocess results
+        print('Computed statistics of all input features, getting the results')
+        results_values = np.array([np.array(p.get()) for p in results])
+
+        # Order stats based on snp nb
+        print('Ordering {} means returned by multiprocessing'.format(
+            len(results_values[:,1])))
+        idx = results_values[:,0].astype(int)
+        ordered_idx_pos = idx.argsort()
+
+        # Ordered means
+        means = results_values[:,1][ordered_idx_pos]
+
+        # Ordered sd
+        if not mean_only:
+            print('Ordering {} standard deviations returned by multiprocessing'.format(
+                len(results_values[:,2])))
+            sds = results_values[:,2][ordered_idx_pos]
+
+        # Append fold statistics
+        means_by_fold.append(means)
+        sds_by_fold.append(sds)
+
+        print('Computed fold input features stats in {} seconds\n---\n'.format(
+            time.time()-fold_start_time))
+
+    # Close access to h5py dataset file
+    dataset_file.close()
+
+    # Write input features stats to file
+    if args.out is not None:
+        stats_filename = args.out
+    else:
+        stats_filename = 'input_features_stats_' \
+                + args.dataset.split('/')[-1][0:-5] \
+                + '_' \
+                + datetime.now().strftime("%Y_%m_%d_%Hh%Mmin%Ssec")
+
+    stats_fullpath = os.path.join(args.exp_path, stats_filename)
+
+    if mean_only:
+        np.savez(stats_fullpath,
+                 means_by_fold=means_by_fold)
+
+    else:
+        np.savez(stats_fullpath,
+                 means_by_fold=means_by_fold,
+                 sd_by_fold=sds_by_fold)
+
+    print('\n---')
+    print('Input features stats were saved to {}'.format(stats_fullpath))
+    print('Execution time {} seconds\n---\n'.format(
+          time.time() - start_time))
+
+
+def compute_feature_mean(i, x):
+    """
+    i is the snp nb (from 0 to nb_snps-1)
+    x is the vector of genotypes for the snp (dim : nb_samples x 1)
+    """
+    # Set to false missing value (missing values are encoded with -1)
+    mask = (x >= 0)
+
+    # Compute mean :
+    # in numerator, missing values are 0
+    # in the denominator we divise by the nb of non missing values
+    mean = np.sum(x*mask) / np.sum(mask)
+
+    return (i,mean)
+
+
+def compute_feature_mean_and_sd(i, x):
+    """
+    i is the snp nb (from 0 to nb_snps-1)
+    x is the vector of genotypes for the snp (dim : nb_samples x 1)
+    """
+    # Compute mean
+    _,mean = compute_feature_mean(i,x)
+
+    # --- Compute sd ---
+    mask = (x >= 0) #set missing values (-1) to False
+
+    numerator = np.sum((x*mask - mean*mask)**2)
+
+    denominator = np.sum(mask) - 1
+
+    sd = np.sqrt(numerator/denominator)
+
+    sd += 1e-6
+
+    return (i,mean,sd)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+            description=('Compute features means and standard deviations '
+                         'for missing values filing and input normalization '
+                         'at training time')
+            )
+
+    # Path
+    parser.add_argument(
+            '--exp-path',
+            type=str,
+            required=True,
+            help=('Path to directory where input features statistics will '
+                  'be written')
+            )
+
+    # Files
+    parser.add_argument(
+            '--dataset',
+            type=str,
+            required=True,
+            help=('Hdf5 dataset created with create_dataset.py. '
+                  'Provide full path')
+            )
+
+    parser.add_argument(
+            '--partition',
+            type=str,
+            required=True,
+            help=('Npz dataset partition returned by partition_data.py '
+                  'Provide full path')
+            )
+
+    # Which statistic to compute (mean or mean+sd)
+    parser.add_argument(
+            '--mean-only',
+            action='store_true',
+            help=('Use this flag to compute only input features means '
+                  'and not the standard deviations')
+            )
+
+    parser.add_argument(
+            '--ncpus',
+            type=int,
+            default=1,
+            help=('Number of cpus for parallel computation of means '
+                  'and of standard deviations. Default: %(default)i')
+            )
+
+    # Output
+    parser.add_argument(
+            '--out',
+            type=str,
+            help=('Optional filename for input features statistics file. '
+                  'If not provided the file will be named '
+                  'input_features_stats_datasetFilename_date')
+            )
+
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    main()

dietnetwork-0.3.0/Dietnet_train_and_evaluate/generate_embedding.py

@@ -0,0 +1,216 @@
+import argparse
+import os
+import time
+from datetime import datetime
+import multiprocessing as mp
+
+import numpy as np
+
+import h5py
+
+import helpers.dataset_utils as du
+
+
+# Additive encoding : genotypes 0, 1 and 2
+NB_POSSIBLE_GENOTYPES = 3
+
+# Class label key in dataset file
+CLASS_LABEL = 'class_labels'
+
+
+def main():
+    start_time = time.time()
+    args = parse_args()
+
+    # Hdf5 dataset
+    dataset_file = h5py.File(args.dataset, 'r')
+
+    # Indices by fold
+    partition_data = np.load(args.partition, allow_pickle=True)
+    indices_byfold = partition_data['folds_indexes']
+
+    print('\n---')
+    print('Computing embedding by fold using {} cpu(s)'.format(args.ncpus))
+    if args.ncpus > 1:
+        pool = mp.Pool(processes=args.ncpus) #this starts ncpus nb of processes
+
+    # Results that will be returned by processes (embedding for a given fold)
+    results = []
+
+    for fold in range(len(indices_byfold)):
+        # Fold indices
+        fold_indices = indices_byfold[fold]
+        # Get fold data
+        (_,_,_,
+         x_train, y_train, _,
+         x_valid, y_valid, _,
+         x_test, y_test,_) = du.get_fold_data(fold, indices_byfold, dataset_file, label=CLASS_LABEL)
+
+        # Embedding on train+valid sets
+        if args.include_valid:
+            print('Including valid samples in emb computation')
+            x = np.concatenate((x_train, x_valid))
+            y = np.concatenate((y_train, y_valid))
+        # Embedding on valid set
+        elif args.only_valid:
+            x = x_valid
+            y = y_valid
+        # Embedding on test set
+        elif args.only_test:
+            x = x_test
+            y = y_test
+        # Embedding on training set
+        else:
+            x = x_train
+            y = y_train
+
+        # Compute embedding
+        #emb = compute_fold_embedding(x, y)
+
+        print('Computing embedding using {} samples'.format(len(y)))
+        # One cpu
+        if args.ncpus == 1:
+            results.append(compute_fold_embedding(x, y, fold))
+        else:
+            results.append(pool.apply_async(
+                compute_fold_embedding, args=(x,y,fold)))
+
+    # Close access to dataset file
+    dataset_file.close()
+
+    # Results values: [(fold0, embedding0), (fold1, embedding1) ...]
+    print('Computed embedding of every fold, getting the results')
+    if args.ncpus > 1:
+        results_values = [p.get() for p in results]
+    else:
+        results_values = results
+
+    # Order results by fold
+    print('Putting embeddings by fold order')
+    folds_order = np.array([i[0] for i in results_values])
+
+    # Order counts (nb samples) per class
+    classes_counts = np.array([i[1] for i in results_values])
+    classes_counts_ordered = classes_counts[folds_order]
+
+    # Order embeddings
+    embeddings = np.array([i[2] for i in results_values])
+    embeddings_ordered = embeddings[folds_order]
+
+    print('Computed embedding of {} folds with following '
+            'number of samples per class for each fold: {}'.format(
+              len(folds_order), classes_counts_ordered))
+    print('---')
+
+    # Write embedding to file
+    if args.out is not None:
+        emb_filename = args.out
+    else:
+        emb_filename = 'genotype_class_freq_embedding_' \
+                + args.dataset.split('/')[-1][0:-5] \
+                + '_' \
+                + datetime.now().strftime("%Y_%m_%d_%Hh%Mmin%Ssec")
+
+    emb_fullpath = os.path.join(args.exp_path, emb_filename)
+
+    print('\n---\nEmbeddings generated in {} seconds'.format(
+        time.time()-start_time))
+
+    print('Saving results to {}'.format(emb_fullpath))
+    np.savez(emb_fullpath, emb=embeddings_ordered)
+
+    print('---\n')
+
+
+def compute_fold_embedding(xs, ys, fold):
+    # Total number of classes
+    nb_class = ys.max() + 1 #class 0
+    class_to_count = {}
+
+    # Compute sum of genotypes (0-1-2) per class
+    xs = xs.transpose() # rows are snps, col are inds
+    embedding = np.zeros((xs.shape[0],nb_class*NB_POSSIBLE_GENOTYPES))
+    for c in range(nb_class):
+        # Select genotypes for samples of same class
+        class_genotypes = xs[:,ys==c]
+        nb = class_genotypes.shape[1] #nb of samples in that class
+        #print('Class:', c, 'NB:', nb)
+        class_to_count[c] = nb
+        for genotype in range(NB_POSSIBLE_GENOTYPES):
+            col = NB_POSSIBLE_GENOTYPES*c+genotype
+            embedding[:,col] = (class_genotypes == genotype).sum(axis=1)/nb
+
+    return fold, class_to_count, embedding
+
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+            description='Generate embedding'
+            )
+
+    parser.add_argument(
+            '--exp-path',
+            type=str,
+            required=True,
+            help='Path to directory where embedding will be saved. '
+            )
+
+    parser.add_argument(
+            '--dataset',
+            type=str,
+            required=True,
+            help=('Hdf5 dataset created with create_dataset.py '
+                  'Provide full path')
+            )
+
+    parser.add_argument(
+            '--partition',
+            type=str,
+            required=True,
+            help=('Npz dataset partition returned by partition_data.py '
+                  'Provide full path')
+            )
+
+    parser.add_argument(
+            '--ncpus',
+            type=int,
+            default=1,
+            help=('Number of cpus available to compute folds embedding in '
+                  'parallel. Default:%(default)i')
+            )
+
+    parser.add_argument(
+            '--include-valid',
+            action='store_true',
+            help=('Use this flag to include samples from the validation set '
+                  'in the embedding computation.')
+            )
+
+    parser.add_argument(
+            '--only-valid',
+            action='store_true',
+            help=('Use this flag to compute embedding only on samples '
+                  ' in the validation set')
+            )
+
+    parser.add_argument(
+            '--only-test',
+            action='store_true',
+            help=('Use this flag to compute embedding only on samples '
+                  ' in the test set')
+            )
+
+    parser.add_argument(
+            '--out',
+            type=str,
+            help=('Optional filename for embedding file. If not provided '
+                  'the file will be named '
+                  'genotype_class_freq_embedding_datasetFilename_date')
+            )
+
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    main()