lattice-sub 1.1.4__py3-none-any.whl → 1.3.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: lattice-sub
3
- Version: 1.1.4
3
+ Version: 1.3.0
4
4
  Summary: Lattice subtraction for cryo-EM micrographs - removes periodic crystal signals to reveal non-periodic features
5
5
  Author-email: George Stephenson <george.stephenson@colorado.edu>, Vignesh Kasinath <vignesh.kasinath@colorado.edu>
6
6
  License: MIT
@@ -94,7 +94,17 @@ lattice-sub batch input_folder/ output_folder/ --pixel-size 0.56
94
94
  lattice-sub batch input_folder/ output_folder/ --pixel-size 0.56 --vis comparisons/
95
95
  ```
96
96
 
97
- This creates side-by-side PNG images showing before/after/difference for each micrograph.
97
+ This creates 4-panel PNG comparison images for each micrograph showing:
98
+ 1. **Original** - Input micrograph
99
+ 2. **Subtracted** - Lattice-removed result
100
+ 3. **Difference** - What was removed (5x amplified)
101
+ 4. **Threshold Curve** - Threshold vs lattice removal efficacy
102
+
103
+ **Limit the number of visualizations:**
104
+ ```bash
105
+ # Generate visualizations for first 10 images only
106
+ lattice-sub batch input_folder/ output_folder/ -p 0.56 --vis comparisons/ -n 10
107
+ ```
98
108
 
99
109
  ---
100
110
 
@@ -105,6 +115,8 @@ This creates side-by-side PNG images showing before/after/difference for each mi
105
115
  | `-p, --pixel-size` | **Required.** Pixel size in Ångstroms |
106
116
  | `-o, --output` | Output file path (default: `sub_<input>`) |
107
117
  | `-t, --threshold` | Peak detection sensitivity (default: **auto** - optimized per image) |
118
+ | `--vis DIR` | Generate 4-panel comparison PNGs in DIR |
119
+ | `-n, --num-vis N` | Limit visualizations to first N images |
108
120
  | `--cpu` | Force CPU processing (GPU is used by default) |
109
121
  | `-q, --quiet` | Hide the banner and progress messages |
110
122
  | `-v, --verbose` | Show detailed processing information |
@@ -172,6 +184,76 @@ python -c "import torch; print(torch.cuda.get_device_name(0) if torch.cuda.is_av
172
184
 
173
185
  ---
174
186
 
187
+ ## Multi-GPU Support
188
+
189
+ When processing batches on systems with multiple GPUs, files are automatically distributed across all available GPUs for faster processing. No extra flags needed!
190
+
191
+ ```bash
192
+ # Automatically uses all available GPUs
193
+ lattice-sub batch input_folder/ output_folder/ -p 0.56
194
+ ```
195
+
196
+ **Example with 2 GPUs and 100 images:**
197
+ - GPU 0: processes images 1-50
198
+ - GPU 1: processes images 51-100
199
+ - Single progress bar shows combined progress
200
+
201
+ This provides near-linear speedup with additional GPUs.
202
+
203
+ ---
204
+
205
+ ## HPC Example (CU Boulder Alpine)
206
+
207
+ Using [Open OnDemand Core Desktop](https://curc.readthedocs.io/en/latest/open_ondemand/core_desktop.html) with 2× RTX 8000 GPUs:
208
+
209
+ ```bash
210
+ # Create environment
211
+ module load anaconda
212
+ conda create -n lattice_test python=3.11 -y
213
+ conda activate lattice_test
214
+ pip install lattice-sub
215
+
216
+ # Process 100 micrographs
217
+ lattice-sub batch input/ output/ -p 0.56
218
+ ```
219
+
220
+ **Output:**
221
+ ```
222
+ Phase-preserving FFT inpainting for cryo-EM | v1.3.0
223
+
224
+ Configuration
225
+ -------------
226
+ Pixel size: 0.56 A
227
+ Threshold: auto
228
+ Backend: Auto → GPU (Quadro RTX 8000)
229
+
230
+ Batch Processing
231
+ ----------------
232
+ Files: 100
233
+ Output: /projects/user/output
234
+ Workers: 1
235
+
236
+ ✓ Using 2 GPUs: GPU 0, GPU 1
237
+
238
+ ✓ GPU 0: Quadro RTX 8000
239
+ ✓ GPU 1: Quadro RTX 8000
240
+
241
+ Processing: 100%|█████████████████████████| 100/100 [05:12<00:00, 3.13s/file]
242
+
243
+ [OK] Batch complete (312.9s)
244
+ ```
245
+
246
+ **100 images processed in ~5 minutes** with automatic multi-GPU distribution.
247
+
248
+ For compute-focused workloads, use Alpine's [GPU partitions](https://curc.readthedocs.io/en/latest/clusters/alpine/alpine-hardware.html) (A100, L40, MI100):
249
+
250
+ ```bash
251
+ # Request 3 GPUs for 1 hour
252
+ sinteractive --partition=aa100 --gres=gpu:3 --ntasks=16 --nodes=1 --time=01:00:00 --qos=normal
253
+ ```
254
+
255
+ ---
256
+
175
257
  ## Python API
176
258
 
177
259
  ```python
@@ -0,0 +1,17 @@
1
+ lattice_sub-1.3.0.dist-info/licenses/LICENSE,sha256=2kPoH0cbEp0cVEGqMpyF2IQX1npxdtQmWJB__HIRSb0,1101
2
+ lattice_subtraction/__init__.py,sha256=TNaJXvSgCQdvYUYJfS5scn92YjORiGfLot9WadZ8u28,1737
3
+ lattice_subtraction/batch.py,sha256=zJzvUnr8dznvxE8jaPKDLJ7AcJg8Cbfv5nVo0FzZz1I,20891
4
+ lattice_subtraction/cli.py,sha256=W99XQClUMKaaFQxle0W-ILQ6UuYRFXZVJWD4qXpcIj4,24063
5
+ lattice_subtraction/config.py,sha256=uzwKb5Zi3phHUk2ZgoiLsQdwFdN-rTiY8n02U91SObc,8426
6
+ lattice_subtraction/core.py,sha256=VzcecSZHRuBuHUc2jHGv8LalINL75RH0aTpABI708y8,16265
7
+ lattice_subtraction/io.py,sha256=uHku6rJ0jeCph7w-gOIDJx-xpNoF6PZcLfb5TBTOiw0,4594
8
+ lattice_subtraction/masks.py,sha256=HIamrACmbQDkaCV4kXhnjMDSwIig4OtQFLig9A8PMO8,11741
9
+ lattice_subtraction/processing.py,sha256=tmnj5K4Z9HCQhRpJ-iMd9Bj_uTRuvDEWyUenh8MCWEM,8341
10
+ lattice_subtraction/threshold_optimizer.py,sha256=yEsGM_zt6YjgEulEZqtRy113xOFB69aHJIETm2xSS6k,15398
11
+ lattice_subtraction/ui.py,sha256=Sp_a-yNmBRZJxll8h9T_H5-_KsI13zGYmHcbcpVpbR8,9176
12
+ lattice_subtraction/visualization.py,sha256=hWFz49NBBrS7d6ofO0VyJ6-v8Q6hPG1dijbDtecMOQs,11890
13
+ lattice_sub-1.3.0.dist-info/METADATA,sha256=pKwt8TcftbZGm1gvWZGO1n3iQiI4JB3E_ix3InB-4D0,14901
14
+ lattice_sub-1.3.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
15
+ lattice_sub-1.3.0.dist-info/entry_points.txt,sha256=o8PzJR8kFnXlKZufoYGBIHpiosM-P4PZeKZXJjtPS6Y,61
16
+ lattice_sub-1.3.0.dist-info/top_level.txt,sha256=BOuW-sm4G-fQtsWPRdeLzWn0WS8sDYVNKIMj5I3JXew,20
17
+ lattice_sub-1.3.0.dist-info/RECORD,,
@@ -1,5 +1,5 @@
1
1
  Wheel-Version: 1.0
2
- Generator: setuptools (80.9.0)
2
+ Generator: setuptools (80.10.2)
3
3
  Root-Is-Purelib: true
4
4
  Tag: py3-none-any
5
5
 
@@ -19,7 +19,7 @@ Example:
19
19
  >>> result.save("output.mrc")
20
20
  """
21
21
 
22
- __version__ = "1.1.4"
22
+ __version__ = "1.3.0"
23
23
  __author__ = "George Stephenson & Vignesh Kasinath"
24
24
 
25
25
  from .config import Config
@@ -1,12 +1,14 @@
1
1
  """
2
2
  Batch processing for multiple micrographs.
3
3
 
4
- This module provides parallel processing capabilities for large datasets.
4
+ This module provides parallel processing capabilities for large datasets,
5
+ including automatic multi-GPU support for systems with multiple CUDA devices.
5
6
  """
6
7
 
7
8
  import os
9
+ import multiprocessing as mp
8
10
  from concurrent.futures import ProcessPoolExecutor, as_completed
9
- from dataclasses import dataclass
11
+ from dataclasses import dataclass, asdict
10
12
  from pathlib import Path
11
13
  from typing import List, Tuple, Optional, Callable
12
14
  import logging
@@ -63,6 +65,91 @@ def _process_single_file(args: tuple) -> Tuple[Path, Optional[str]]:
63
65
  return (Path(input_path), str(e))
64
66
 
65
67
 
68
+ def _gpu_worker(
69
+ gpu_id: int,
70
+ file_pairs: List[Tuple[str, str]],
71
+ config_dict: dict,
72
+ progress_queue: mp.Queue,
73
+ error_queue: mp.Queue,
74
+ ):
75
+ """
76
+ Worker function for multi-GPU processing.
77
+
78
+ Each worker processes its assigned files on a specific GPU and reports
79
+ progress through a shared queue.
80
+
81
+ Args:
82
+ gpu_id: CUDA device ID to use
83
+ file_pairs: List of (input_path, output_path) tuples
84
+ config_dict: Configuration dictionary
85
+ progress_queue: Queue to report progress (sends 1 for each completed file)
86
+ error_queue: Queue to report errors (sends (gpu_id, file_path, error_msg))
87
+ """
88
+ import torch
89
+
90
+ # Set this process to use the specific GPU
91
+ torch.cuda.set_device(gpu_id)
92
+
93
+ # Reconstruct config with the specific device_id and quiet mode
94
+ config_dict = config_dict.copy()
95
+ config_dict['device_id'] = gpu_id
96
+ config_dict['_quiet'] = True # Suppress messages - main process handles this
97
+ config = Config(**config_dict)
98
+
99
+ # Create subtractor (messages suppressed via _quiet flag)
100
+ subtractor = LatticeSubtractor(config)
101
+
102
+ for input_path, output_path in file_pairs:
103
+ try:
104
+ result = subtractor.process(input_path)
105
+ result.save(output_path, pixel_size=config.pixel_ang)
106
+ progress_queue.put(1)
107
+ except Exception as e:
108
+ error_queue.put((gpu_id, input_path, str(e)))
109
+ return # Fail-fast: exit on first error
110
+
111
+
112
+ def _check_gpu_memory(device_id: int, image_shape: Tuple[int, int]) -> Tuple[bool, str]:
113
+ """
114
+ Check if GPU has sufficient memory for processing.
115
+
116
+ Args:
117
+ device_id: CUDA device ID
118
+ image_shape: (height, width) of image
119
+
120
+ Returns:
121
+ (is_ok, message) - True if sufficient memory, False with warning message
122
+ """
123
+ try:
124
+ import torch
125
+ free_mem, total_mem = torch.cuda.mem_get_info(device_id)
126
+
127
+ # Estimate memory needed: image + FFT (complex) + masks + overhead
128
+ # Roughly 16x image size for safe margin (complex FFT, intermediate buffers)
129
+ image_bytes = image_shape[0] * image_shape[1] * 4 # float32
130
+ estimated_need = image_bytes * 16
131
+
132
+ if free_mem < estimated_need:
133
+ return False, (
134
+ f"GPU {device_id}: {free_mem / 1e9:.1f}GB free, "
135
+ f"need ~{estimated_need / 1e9:.1f}GB"
136
+ )
137
+ return True, ""
138
+ except Exception as e:
139
+ return True, "" # If we can't check, proceed anyway
140
+
141
+
142
+ def _get_available_gpus() -> List[int]:
143
+ """Get list of available CUDA GPU device IDs."""
144
+ try:
145
+ import torch
146
+ if torch.cuda.is_available():
147
+ return list(range(torch.cuda.device_count()))
148
+ return []
149
+ except ImportError:
150
+ return []
151
+
152
+
66
153
  class BatchProcessor:
67
154
  """
68
155
  Parallel batch processor for micrograph datasets.
@@ -157,6 +244,9 @@ class BatchProcessor:
157
244
  """
158
245
  Process a list of input/output file pairs.
159
246
 
247
+ Automatically uses multi-GPU processing when multiple GPUs are available.
248
+ Files are distributed evenly across GPUs in chunks.
249
+
160
250
  Args:
161
251
  file_pairs: List of (input_path, output_path) tuples
162
252
  show_progress: If True, show progress bar
@@ -168,8 +258,7 @@ class BatchProcessor:
168
258
  successful = 0
169
259
  failed_files = []
170
260
 
171
- # Check if using GPU - if so, process sequentially to avoid CUDA fork issues
172
- # With "auto" backend, check if PyTorch + CUDA is actually available
261
+ # Check if using GPU - if so, check for multi-GPU capability
173
262
  use_gpu = self.config.backend == "pytorch"
174
263
  if self.config.backend == "auto":
175
264
  try:
@@ -179,10 +268,19 @@ class BatchProcessor:
179
268
  use_gpu = False
180
269
 
181
270
  if use_gpu:
182
- # Sequential processing for GPU (CUDA doesn't support fork multiprocessing)
183
- successful, failed_files = self._process_sequential(
184
- file_pairs, show_progress
185
- )
271
+ # Check how many GPUs are available
272
+ available_gpus = _get_available_gpus()
273
+
274
+ if len(available_gpus) > 1 and total > 1:
275
+ # Multi-GPU processing
276
+ successful, failed_files = self._process_multi_gpu(
277
+ file_pairs, available_gpus, show_progress
278
+ )
279
+ else:
280
+ # Single GPU - sequential processing
281
+ successful, failed_files = self._process_sequential(
282
+ file_pairs, show_progress
283
+ )
186
284
  else:
187
285
  # Parallel processing for CPU
188
286
  successful, failed_files = self._process_parallel(
@@ -284,6 +382,159 @@ class BatchProcessor:
284
382
 
285
383
  return successful, failed_files
286
384
 
385
+ def _process_multi_gpu(
386
+ self,
387
+ file_pairs: List[Tuple[Path, Path]],
388
+ gpu_ids: List[int],
389
+ show_progress: bool = True,
390
+ ) -> Tuple[int, List[Tuple[Path, str]]]:
391
+ """
392
+ Process files in parallel across multiple GPUs.
393
+
394
+ Files are distributed evenly across GPUs in chunks.
395
+ Uses spawn-based multiprocessing to avoid CUDA fork issues.
396
+
397
+ Args:
398
+ file_pairs: List of (input_path, output_path) tuples
399
+ gpu_ids: List of CUDA device IDs to use
400
+ show_progress: If True, show unified progress bar
401
+
402
+ Returns:
403
+ (successful_count, failed_files_list)
404
+ """
405
+ import time
406
+
407
+ total = len(file_pairs)
408
+ num_gpus = len(gpu_ids)
409
+
410
+ # Print multi-GPU info with GPU names
411
+ try:
412
+ import torch
413
+ gpu_names = [torch.cuda.get_device_name(i) for i in gpu_ids]
414
+ print(f"✓ Using {num_gpus} GPUs: {', '.join(f'GPU {i}' for i in gpu_ids)}")
415
+ print("")
416
+ for i, name in zip(gpu_ids, gpu_names):
417
+ print(f" ✓ GPU {i}: {name}")
418
+ except Exception:
419
+ print(f"✓ Using {num_gpus} GPUs")
420
+
421
+ # Check GPU memory on first GPU (assume similar for all)
422
+ if file_pairs:
423
+ try:
424
+ sample_image = read_mrc(file_pairs[0][0])
425
+ is_ok, msg = _check_gpu_memory(gpu_ids[0], sample_image.shape)
426
+ if not is_ok:
427
+ print(f"⚠ Memory warning: {msg}")
428
+ except Exception:
429
+ pass # Proceed anyway
430
+
431
+ # Distribute files evenly across GPUs (chunked distribution)
432
+ chunk_size = (total + num_gpus - 1) // num_gpus # Ceiling division
433
+ gpu_file_assignments = []
434
+
435
+ for i, gpu_id in enumerate(gpu_ids):
436
+ start_idx = i * chunk_size
437
+ end_idx = min(start_idx + chunk_size, total)
438
+ if start_idx < total:
439
+ chunk = [(str(inp), str(out)) for inp, out in file_pairs[start_idx:end_idx]]
440
+ gpu_file_assignments.append((gpu_id, chunk))
441
+
442
+ # Create shared queues for progress and errors
443
+ # Use 'spawn' context to avoid CUDA fork issues
444
+ ctx = mp.get_context('spawn')
445
+ progress_queue = ctx.Queue()
446
+ error_queue = ctx.Queue()
447
+
448
+ # Create progress bar (after all GPU info printed)
449
+ if show_progress:
450
+ print() # Blank line for visual separation
451
+ pbar = tqdm(
452
+ total=total,
453
+ desc=" Processing",
454
+ unit="file",
455
+ ncols=80,
456
+ leave=True,
457
+ )
458
+ else:
459
+ pbar = None
460
+
461
+ # Start worker processes
462
+ processes = []
463
+ for gpu_id, file_chunk in gpu_file_assignments:
464
+ p = ctx.Process(
465
+ target=_gpu_worker,
466
+ args=(gpu_id, file_chunk, self._config_dict, progress_queue, error_queue),
467
+ )
468
+ p.start()
469
+ processes.append(p)
470
+
471
+ # Monitor progress and check for errors
472
+ successful = 0
473
+ failed_files = []
474
+ completed = 0
475
+
476
+ while completed < total:
477
+ # Check for progress updates (non-blocking with timeout)
478
+ try:
479
+ while True:
480
+ progress_queue.get(timeout=0.1)
481
+ successful += 1
482
+ completed += 1
483
+ if pbar:
484
+ pbar.update(1)
485
+ except:
486
+ pass # Queue empty, continue
487
+
488
+ # Check for errors (non-blocking)
489
+ try:
490
+ while True:
491
+ gpu_id, file_path, error_msg = error_queue.get_nowait()
492
+ failed_files.append((Path(file_path), error_msg))
493
+ completed += 1
494
+ if pbar:
495
+ pbar.update(1)
496
+
497
+ # Fail-fast: terminate all workers and report
498
+ print(f"\n✗ GPU {gpu_id} failed on {Path(file_path).name}: {error_msg}")
499
+ print(f"\nTip: Try a different configuration:")
500
+ print(f" lattice-sub batch <input> <output> -p {self.config.pixel_ang} --cpu -j 8")
501
+
502
+ # Terminate all processes
503
+ for p in processes:
504
+ if p.is_alive():
505
+ p.terminate()
506
+
507
+ if pbar:
508
+ pbar.close()
509
+
510
+ return successful, failed_files
511
+ except:
512
+ pass # No errors, continue
513
+
514
+ # Check if all processes have finished
515
+ all_done = all(not p.is_alive() for p in processes)
516
+ if all_done:
517
+ # Drain remaining queue items
518
+ try:
519
+ while True:
520
+ progress_queue.get_nowait()
521
+ successful += 1
522
+ completed += 1
523
+ if pbar:
524
+ pbar.update(1)
525
+ except:
526
+ pass
527
+ break
528
+
529
+ # Wait for all processes to finish
530
+ for p in processes:
531
+ p.join(timeout=1.0)
532
+
533
+ if pbar:
534
+ pbar.close()
535
+
536
+ return successful, failed_files
537
+
287
538
  def process_numbered_sequence(
288
539
  self,
289
540
  input_pattern: str,
@@ -153,9 +153,12 @@ def main():
153
153
  # Batch process directory (GPU handles parallelism)
154
154
  lattice-sub batch input_dir/ output_dir/ --pixel-size 0.56
155
155
 
156
- # Batch with visualizations
156
+ # Batch with visualizations (4-panel with threshold curve)
157
157
  lattice-sub batch input_dir/ output_dir/ -p 0.56 --vis viz_dir/
158
158
 
159
+ # Batch with limited visualizations (only first 10)
160
+ lattice-sub batch input_dir/ output_dir/ -p 0.56 --vis viz_dir/ -n 10
161
+
159
162
  # CPU batch with parallel workers (use -j only with --cpu)
160
163
  lattice-sub batch input_dir/ output_dir/ -p 0.56 --cpu -j 8
161
164
 
@@ -502,6 +505,12 @@ def process(
502
505
  default=None,
503
506
  help="Generate comparison visualizations in this directory",
504
507
  )
508
+ @click.option(
509
+ "-n", "--num-vis",
510
+ type=int,
511
+ default=None,
512
+ help="Number of visualizations to generate (default: all)",
513
+ )
505
514
  @click.option(
506
515
  "-v", "--verbose",
507
516
  is_flag=True,
@@ -528,6 +537,7 @@ def batch(
528
537
  config: Optional[str],
529
538
  recursive: bool,
530
539
  vis: Optional[str],
540
+ num_vis: Optional[int],
531
541
  verbose: bool,
532
542
  quiet: bool,
533
543
  cpu: bool,
@@ -617,7 +627,8 @@ def batch(
617
627
 
618
628
  # Generate visualizations if requested
619
629
  if vis:
620
- logger.info(f"Generating visualizations in: {vis}")
630
+ limit_msg = f" (limit: {num_vis})" if num_vis else ""
631
+ logger.info(f"Generating visualizations in: {vis}{limit_msg}")
621
632
  viz_success, viz_total = generate_visualizations(
622
633
  input_dir=input_dir,
623
634
  output_dir=output_dir,
@@ -625,6 +636,8 @@ def batch(
625
636
  prefix=prefix,
626
637
  pattern=pattern,
627
638
  show_progress=True,
639
+ limit=num_vis,
640
+ config=cfg,
628
641
  )
629
642
  logger.info(f"Visualizations: {viz_success}/{viz_total} created")
630
643
 
@@ -64,6 +64,13 @@ class Config:
64
64
  # Enabled by default when GPU is available
65
65
  use_kornia: bool = True
66
66
 
67
+ # GPU device ID for multi-GPU support. None = auto-select (GPU 0 for single-GPU mode)
68
+ # When using multi-GPU batch processing, this is set automatically per worker
69
+ device_id: Optional[int] = None
70
+
71
+ # Internal flag to suppress status messages (used by batch workers)
72
+ _quiet: bool = False
73
+
67
74
  def __post_init__(self):
68
75
  """Validate and set auto-calculated parameters."""
69
76
  if self.pixel_ang <= 0:
@@ -83,21 +83,31 @@ class LatticeSubtractor:
83
83
 
84
84
  Auto mode tries PyTorch+CUDA first, then PyTorch CPU, then NumPy.
85
85
  Prints user-friendly status message about which backend is active.
86
+
87
+ Uses config.device_id if specified for multi-GPU support.
86
88
  """
87
89
  backend = self.config.backend
88
90
  self._gpu_message_shown = getattr(self, '_gpu_message_shown', False)
89
91
 
92
+ # Check if quiet mode (suppress messages for batch workers)
93
+ quiet = getattr(self.config, '_quiet', False)
94
+ if quiet:
95
+ self._gpu_message_shown = True
96
+
97
+ # Get device ID from config (None means auto-select GPU 0)
98
+ device_id = self.config.device_id if self.config.device_id is not None else 0
99
+
90
100
  # Auto mode: try GPU first, then CPU
91
101
  if backend == "auto":
92
102
  try:
93
103
  import torch
94
104
  if torch.cuda.is_available():
95
- self.device = torch.device('cuda')
105
+ self.device = torch.device(f'cuda:{device_id}')
96
106
  self.use_gpu = True
97
107
  # Only print once per session (batch processing reuses subtractor)
98
108
  if not self._gpu_message_shown:
99
- gpu_name = torch.cuda.get_device_name(0)
100
- print(f"✓ Using GPU: {gpu_name}")
109
+ gpu_name = torch.cuda.get_device_name(device_id)
110
+ print(f"✓ Using GPU {device_id}: {gpu_name}")
101
111
  self._gpu_message_shown = True
102
112
  else:
103
113
  self.device = torch.device('cpu')
@@ -116,7 +126,7 @@ class LatticeSubtractor:
116
126
  try:
117
127
  import torch
118
128
  if torch.cuda.is_available():
119
- self.device = torch.device('cuda')
129
+ self.device = torch.device(f'cuda:{device_id}')
120
130
  self.use_gpu = True
121
131
  else:
122
132
  import warnings
@@ -2,12 +2,12 @@
2
2
  Visualization utilities for lattice subtraction results.
3
3
 
4
4
  This module provides functions to create comparison visualizations
5
- showing original, processed, and difference images.
5
+ showing original, processed, difference images, and threshold optimization curves.
6
6
  """
7
7
 
8
8
  import logging
9
9
  from pathlib import Path
10
- from typing import Optional, Tuple
10
+ from typing import Optional, Tuple, List
11
11
 
12
12
  import numpy as np
13
13
 
@@ -22,6 +22,56 @@ logging.getLogger('PIL.PngImagePlugin').setLevel(logging.WARNING)
22
22
  logger = logging.getLogger(__name__)
23
23
 
24
24
 
25
+ def compute_threshold_curve(
26
+ image: np.ndarray,
27
+ config,
28
+ n_points: int = 21,
29
+ ) -> Tuple[np.ndarray, np.ndarray, float, float]:
30
+ """
31
+ Compute quality scores across a range of thresholds.
32
+
33
+ Args:
34
+ image: Original image array
35
+ config: Config object with processing parameters
36
+ n_points: Number of threshold points to evaluate
37
+
38
+ Returns:
39
+ Tuple of (thresholds, quality_scores, optimal_threshold, optimal_quality)
40
+ """
41
+ from .threshold_optimizer import ThresholdOptimizer
42
+
43
+ optimizer = ThresholdOptimizer(config)
44
+
45
+ # Prepare FFT data once
46
+ subtracted, radial_mask, box_size = optimizer._prepare_fft_data(image)
47
+
48
+ # Evaluate across threshold range
49
+ thresholds = np.linspace(
50
+ optimizer.min_threshold,
51
+ optimizer.max_threshold,
52
+ n_points
53
+ )
54
+
55
+ # Use GPU batch if available
56
+ if optimizer.use_gpu:
57
+ qualities, peak_counts = optimizer._compute_quality_batch_gpu(
58
+ subtracted, radial_mask, thresholds
59
+ )
60
+ else:
61
+ qualities = []
62
+ for t in thresholds:
63
+ q, _ = optimizer._compute_quality(subtracted, radial_mask, t)
64
+ qualities.append(q)
65
+ qualities = np.array(qualities)
66
+
67
+ # Find optimal
68
+ best_idx = np.argmax(qualities)
69
+ optimal_threshold = thresholds[best_idx]
70
+ optimal_quality = qualities[best_idx]
71
+
72
+ return thresholds, qualities, optimal_threshold, optimal_quality
73
+
74
+
25
75
  def create_comparison_figure(
26
76
  original: np.ndarray,
27
77
  processed: np.ndarray,
@@ -81,19 +131,106 @@ def create_comparison_figure(
81
131
  return fig
82
132
 
83
133
 
134
+ def create_comparison_figure_with_threshold(
135
+ original: np.ndarray,
136
+ processed: np.ndarray,
137
+ thresholds: np.ndarray,
138
+ quality_scores: np.ndarray,
139
+ optimal_threshold: float,
140
+ optimal_quality: float,
141
+ title: str = "Lattice Subtraction Comparison",
142
+ figsize: Tuple[int, int] = (24, 6),
143
+ dpi: int = 150,
144
+ ):
145
+ """
146
+ Create a 4-panel comparison figure with threshold optimization curve.
147
+
148
+ Layout: [Original] [Subtracted] [Difference] [Threshold vs Quality]
149
+
150
+ Args:
151
+ original: Original image array
152
+ processed: Processed (lattice-subtracted) image array
153
+ thresholds: Array of threshold values tested
154
+ quality_scores: Array of quality scores for each threshold
155
+ optimal_threshold: The optimal threshold that was selected
156
+ optimal_quality: Quality score at optimal threshold
157
+ title: Figure title
158
+ figsize: Figure size in inches (width, height)
159
+ dpi: Resolution for saving
160
+
161
+ Returns:
162
+ matplotlib Figure object
163
+ """
164
+ import matplotlib.pyplot as plt
165
+
166
+ # Compute difference
167
+ difference = original - processed
168
+
169
+ # Create figure with 4 panels (1 row, 4 columns)
170
+ fig, axes = plt.subplots(1, 4, figsize=figsize)
171
+
172
+ # Contrast limits from original
173
+ vmin, vmax = np.percentile(original, [1, 99])
174
+
175
+ # Panel 1: Original
176
+ axes[0].imshow(original, cmap='gray', vmin=vmin, vmax=vmax)
177
+ axes[0].set_title(f'Original\n{original.shape}')
178
+ axes[0].axis('off')
179
+
180
+ # Panel 2: Lattice Subtracted
181
+ axes[1].imshow(processed, cmap='gray', vmin=vmin, vmax=vmax)
182
+ axes[1].set_title(f'Lattice Subtracted\n{processed.shape}')
183
+ axes[1].axis('off')
184
+
185
+ # Panel 3: Difference (removed lattice)
186
+ diff_std = np.std(difference)
187
+ axes[2].imshow(
188
+ difference,
189
+ cmap='RdBu_r',
190
+ vmin=-diff_std * 3,
191
+ vmax=diff_std * 3
192
+ )
193
+ axes[2].set_title('Difference (Removed Lattice)')
194
+ axes[2].axis('off')
195
+
196
+ # Panel 4: Threshold vs Quality Score curve
197
+ axes[3].plot(thresholds, quality_scores, 'b-', linewidth=2, label='Quality Score')
198
+ axes[3].axvline(x=optimal_threshold, color='r', linestyle='--', linewidth=2,
199
+ label=f'Optimal: {optimal_threshold:.3f}')
200
+ axes[3].scatter([optimal_threshold], [optimal_quality], color='r', s=100, zorder=5)
201
+ axes[3].set_xlabel('Threshold', fontsize=11)
202
+ axes[3].set_ylabel('Lattice Removal Efficacy', fontsize=11)
203
+ axes[3].set_title(f'Threshold Optimization\nOptimal = {optimal_threshold:.3f}')
204
+ axes[3].legend(loc='best', fontsize=9)
205
+ axes[3].grid(True, alpha=0.3)
206
+ axes[3].set_xlim(thresholds.min(), thresholds.max())
207
+
208
+ # Title
209
+ plt.suptitle(title, fontsize=14)
210
+ plt.tight_layout()
211
+
212
+ return fig
213
+
214
+
84
215
  def save_comparison_visualization(
85
216
  original_path: Path,
86
217
  processed_path: Path,
87
218
  output_path: Path,
219
+ config = None,
88
220
  dpi: int = 150,
89
221
  ) -> None:
90
222
  """
91
- Create and save a comparison visualization for a single image pair.
223
+ Create and save a 4-panel comparison visualization for a single image pair.
224
+
225
+ Includes threshold optimization curve showing how the optimal threshold
226
+ was selected based on lattice removal efficacy.
92
227
 
93
228
  Args:
94
229
  original_path: Path to original MRC file
95
230
  processed_path: Path to processed MRC file
96
231
  output_path: Path for output PNG file
232
+ config: Config object for threshold computation (optional, will use defaults)
233
+ dpi: Resolution for output images
97
234
  """
98
235
  import matplotlib.pyplot as plt
99
236
  import mrcfile
@@ -106,11 +243,32 @@ def save_comparison_visualization(
106
243
 
107
244
  # Create title
108
245
  name = original_path.name
109
- short_name = name[:60] + "..." if len(name) > 60 else name
110
- title = f"Lattice Subtraction Comparison: {short_name}"
246
+ short_name = name[:50] + "..." if len(name) > 50 else name
247
+ title = f"Lattice Subtraction: {short_name}"
248
+
249
+ # Try to compute threshold curve if config available
250
+ try:
251
+ if config is None:
252
+ # Create default config for threshold computation
253
+ from .config import Config
254
+ config = Config(pixel_ang=0.56) # Default K3 pixel size
255
+
256
+ # Compute threshold optimization curve
257
+ thresholds, quality_scores, optimal_threshold, optimal_quality = \
258
+ compute_threshold_curve(original, config)
259
+
260
+ # Create 4-panel figure with threshold curve
261
+ fig = create_comparison_figure_with_threshold(
262
+ original, processed,
263
+ thresholds, quality_scores,
264
+ optimal_threshold, optimal_quality,
265
+ title=title, dpi=dpi
266
+ )
267
+ except Exception as e:
268
+ # Fallback to 3-panel if threshold computation fails
269
+ logger.debug(f"Could not compute threshold curve: {e}, using 3-panel view")
270
+ fig = create_comparison_figure(original, processed, title=title, dpi=dpi)
111
271
 
112
- # Create and save figure
113
- fig = create_comparison_figure(original, processed, title=title, dpi=dpi)
114
272
  output_path.parent.mkdir(parents=True, exist_ok=True)
115
273
  fig.savefig(output_path, dpi=dpi, bbox_inches='tight')
116
274
  plt.close(fig)
@@ -124,9 +282,11 @@ def generate_visualizations(
124
282
  pattern: str = "*.mrc",
125
283
  dpi: int = 150,
126
284
  show_progress: bool = True,
285
+ limit: Optional[int] = None,
286
+ config = None,
127
287
  ) -> Tuple[int, int]:
128
288
  """
129
- Generate comparison visualizations for all processed images in a directory.
289
+ Generate comparison visualizations for processed images in a directory.
130
290
 
131
291
  Args:
132
292
  input_dir: Directory containing original MRC files
@@ -136,6 +296,8 @@ def generate_visualizations(
136
296
  pattern: Glob pattern for finding processed files
137
297
  dpi: Resolution for output images
138
298
  show_progress: Show progress bar
299
+ limit: Maximum number of visualizations to generate (None = all)
300
+ config: Config object for threshold computation (optional)
139
301
 
140
302
  Returns:
141
303
  Tuple of (successful_count, total_count)
@@ -152,6 +314,12 @@ def generate_visualizations(
152
314
  logger.warning(f"No processed files found matching '{prefix}{pattern}' in {output_dir}")
153
315
  return 0, 0
154
316
 
317
+ # Apply limit if specified
318
+ total_available = len(output_files)
319
+ if limit is not None and limit > 0:
320
+ output_files = output_files[:limit]
321
+ logger.info(f"Limiting to {limit} visualizations (of {total_available} available)")
322
+
155
323
  successful = 0
156
324
  total = len(output_files)
157
325
 
@@ -184,11 +352,12 @@ def generate_visualizations(
184
352
  successful += 1
185
353
  continue
186
354
 
187
- # Generate visualization
355
+ # Generate visualization with 4-panel layout
188
356
  save_comparison_visualization(
189
357
  original_path=input_path,
190
358
  processed_path=processed_path,
191
359
  output_path=viz_path,
360
+ config=config,
192
361
  dpi=dpi,
193
362
  )
194
363
  successful += 1
@@ -1,17 +0,0 @@
1
- lattice_sub-1.1.4.dist-info/licenses/LICENSE,sha256=2kPoH0cbEp0cVEGqMpyF2IQX1npxdtQmWJB__HIRSb0,1101
2
- lattice_subtraction/__init__.py,sha256=zLLC1T_bGZMDpZTcywjloIsV1vvIhsBbj_P9h7YFPqw,1737
3
- lattice_subtraction/batch.py,sha256=sTDWEL5FlEx2HFaJsTZRXyzLQoNCgUqRo900eZ6kq68,12005
4
- lattice_subtraction/cli.py,sha256=fWZ3ueahQXPQ_ApzS345AdslsMF0-6HIAIvan2IFHkk,23597
5
- lattice_subtraction/config.py,sha256=dh8EJFzJEEXwwggQ46rBMHsuVOExQWM-kCfonT94_fE,8111
6
- lattice_subtraction/core.py,sha256=QzE5CLv92XPoyuw8JcMAGIeSEVgfwSkHgG86WlHAjMo,15790
7
- lattice_subtraction/io.py,sha256=uHku6rJ0jeCph7w-gOIDJx-xpNoF6PZcLfb5TBTOiw0,4594
8
- lattice_subtraction/masks.py,sha256=HIamrACmbQDkaCV4kXhnjMDSwIig4OtQFLig9A8PMO8,11741
9
- lattice_subtraction/processing.py,sha256=tmnj5K4Z9HCQhRpJ-iMd9Bj_uTRuvDEWyUenh8MCWEM,8341
10
- lattice_subtraction/threshold_optimizer.py,sha256=yEsGM_zt6YjgEulEZqtRy113xOFB69aHJIETm2xSS6k,15398
11
- lattice_subtraction/ui.py,sha256=Sp_a-yNmBRZJxll8h9T_H5-_KsI13zGYmHcbcpVpbR8,9176
12
- lattice_subtraction/visualization.py,sha256=pMZKcz6Xgs98lLaZbvGjoMIyEYA_MLRracVxpQStC3w,5935
13
- lattice_sub-1.1.4.dist-info/METADATA,sha256=hPnPdJSHoP4DL_2R-frH5982ftSw4ggVpxiZsFcoNRM,12581
14
- lattice_sub-1.1.4.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
15
- lattice_sub-1.1.4.dist-info/entry_points.txt,sha256=o8PzJR8kFnXlKZufoYGBIHpiosM-P4PZeKZXJjtPS6Y,61
16
- lattice_sub-1.1.4.dist-info/top_level.txt,sha256=BOuW-sm4G-fQtsWPRdeLzWn0WS8sDYVNKIMj5I3JXew,20
17
- lattice_sub-1.1.4.dist-info/RECORD,,