multixem 0.2__tar.gz → 0.3.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: multixem
3
- Version: 0.2
3
+ Version: 0.3.1
4
4
  Summary: Refinement pipeline for multiple data sets in structure biology
5
5
  Author-email: Martin Maly <martin.maly.mm@email.cz>
6
6
  License: MIT
@@ -38,27 +38,33 @@ _________________
38
38
  [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
39
39
  _________________
40
40
 
41
- *This is still work in progress.*
41
+ *This is still a work in progress.*
42
42
 
43
43
 
44
44
  Installation
45
45
  ============
46
46
 
47
- Install the latest code from this GitHub repository with `pip`:
47
+ Install the latest code from this GitHub repository using:
48
48
 
49
49
  ```bash
50
50
  pip install git+https://github.com/MartinMalyMM/multixem.git
51
51
  ```
52
52
 
53
+ Or install the released package from PyPI with `pip`:
54
+
55
+ ```bash
56
+ pip install multixem
57
+ ```
58
+
53
59
  It is recommended to use a Python virtual environment:
54
60
 
55
61
  ```bash
56
62
  python -m venv .venv
57
63
  source .venv/bin/activate
58
- pip install git+https://github.com/MartinMalyMM/multixem.git
64
+ pip install multixem # or pip install git+https://github.com/MartinMalyMM/multixem.git
59
65
  ```
60
66
 
61
- You will also need [Servalcat](https://github.com/kyamashita/servalcat) installed, typically from a recent `CCP4` installation which also includes the [Monomer Library](https://github.com/MonomerLibrary/monomers).
67
+ Most of the dependencies (NumPy, pandas, [GEMMI](https://github.com/project-gemmi/gemmi), MatPlotLib Python modules) will be installed automatically. You will also need [Servalcat](https://github.com/kyamashita/servalcat) installed, typically from a recent `CCP4` installation which also includes the [Monomer Library](https://github.com/MonomerLibrary/monomers).
62
68
 
63
69
 
64
70
  Bootstrap
@@ -78,9 +84,12 @@ multixem bootstrap 10000 \
78
84
  --servalcat_confing "config.yaml" \
79
85
  --n_bins 30 \
80
86
  --n_proc 16 \
81
- --geometry_cids 1PGJ_geometry_obj.txt
87
+ --geometry_cids 1PGJ_geometry_obj.txt \
88
+ --random_weights
82
89
  ```
83
90
 
91
+ Random weights will be assigned to reflection likelihood terms in the target log-likelihood function. A fraction of reflection will be assigned zero weights based on the provided free flag in --hklin_free or --hklin, or randomly if no free flag is provided, controlled by --fraction_zero.
92
+
84
93
  The file given in `1PGJ_geometry_obj.txt` defines parameters/features of the structure model under investigation. Each lines specifies an object using atomic CIDs divided by spaces. One CID in row denotes an occupancy, two an interactomic distance, three an angle between the atoms and four a torsion angle. For instance:
85
94
 
86
95
  ```
@@ -91,12 +100,31 @@ The file given in `1PGJ_geometry_obj.txt` defines parameters/features of the str
91
100
 
92
101
  Regarding the distances and angles, the procedure is suitable for investigation non-covalently linked atoms. The environment is locally unrestrained (Van der Waals anti-bumping restraints switched off) to provide unbiased results.
93
102
 
103
+ Unrestrained refinement of a small molecule can be performed using the following setting:
104
+
105
+ ```bash
106
+ multixem bootstrap 10000 \
107
+ --hklin 1513945.cif \
108
+ --model 1513945.cif \
109
+ --prefix 1513945_bootstrap10000 \
110
+ --servalcat_args "--unre --hydrogen yes --refine_h --adp aniso --no_solvent --adpr_weight 0 --ncycle 20" \
111
+ --n_bins 20 \
112
+ --n_proc 16 \
113
+ --random_weights
114
+ ```
115
+
94
116
  All currently available options are listed using:
95
117
 
96
118
  ```bash
97
119
  multixem bootstrap --help
98
120
  ```
99
121
 
122
+ The options for Servalcat which could be provided through the `--servalcat_args` parameter to this *Multixem* pipeline can be listed using:
123
+
124
+ ```bash
125
+ servalcat refine_xtal_norefmac --help
126
+ ```
127
+
100
128
  Comparison of isomorphous data sets
101
129
  ===================================
102
130
 
@@ -112,7 +140,7 @@ multixem pipeline \
112
140
  --model insuling_cow.pdb insuling_pig.pdb insuling_people.pdb \
113
141
  --n_bins 30 \
114
142
  --n_proc 4 \
115
- --unify_cell```
143
+ --unify_cell
116
144
  ```
117
145
 
118
146
  Example: Merging and comparison of batches of unmerged diffraction data:
@@ -121,12 +149,11 @@ Example: Merging and comparison of batches of unmerged diffraction data:
121
149
  multixem pipeline \
122
150
  --hklin_unmerged insuling_people_unmerged.mtz \
123
151
  --hklin_free insuling_people.mtz \
124
- -p insulin \
152
+ -p insulin_people_600 \
125
153
  --model insuling_people.pdb \
126
154
  --n_bins 30 \
127
155
  --n_proc 4 \
128
156
  --n_batches 600
129
- ```
130
157
  ```
131
158
 
132
159
  All currently available options are listed using:
@@ -7,27 +7,33 @@ _________________
7
7
  [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
8
8
  _________________
9
9
 
10
- *This is still work in progress.*
10
+ *This is still a work in progress.*
11
11
 
12
12
 
13
13
  Installation
14
14
  ============
15
15
 
16
- Install the latest code from this GitHub repository with `pip`:
16
+ Install the latest code from this GitHub repository using:
17
17
 
18
18
  ```bash
19
19
  pip install git+https://github.com/MartinMalyMM/multixem.git
20
20
  ```
21
21
 
22
+ Or install the released package from PyPI with `pip`:
23
+
24
+ ```bash
25
+ pip install multixem
26
+ ```
27
+
22
28
  It is recommended to use a Python virtual environment:
23
29
 
24
30
  ```bash
25
31
  python -m venv .venv
26
32
  source .venv/bin/activate
27
- pip install git+https://github.com/MartinMalyMM/multixem.git
33
+ pip install multixem # or pip install git+https://github.com/MartinMalyMM/multixem.git
28
34
  ```
29
35
 
30
- You will also need [Servalcat](https://github.com/kyamashita/servalcat) installed, typically from a recent `CCP4` installation which also includes the [Monomer Library](https://github.com/MonomerLibrary/monomers).
36
+ Most of the dependencies (NumPy, pandas, [GEMMI](https://github.com/project-gemmi/gemmi), MatPlotLib Python modules) will be installed automatically. You will also need [Servalcat](https://github.com/kyamashita/servalcat) installed, typically from a recent `CCP4` installation which also includes the [Monomer Library](https://github.com/MonomerLibrary/monomers).
31
37
 
32
38
 
33
39
  Bootstrap
@@ -47,9 +53,12 @@ multixem bootstrap 10000 \
47
53
  --servalcat_confing "config.yaml" \
48
54
  --n_bins 30 \
49
55
  --n_proc 16 \
50
- --geometry_cids 1PGJ_geometry_obj.txt
56
+ --geometry_cids 1PGJ_geometry_obj.txt \
57
+ --random_weights
51
58
  ```
52
59
 
60
+ Random weights will be assigned to reflection likelihood terms in the target log-likelihood function. A fraction of reflection will be assigned zero weights based on the provided free flag in --hklin_free or --hklin, or randomly if no free flag is provided, controlled by --fraction_zero.
61
+
53
62
  The file given in `1PGJ_geometry_obj.txt` defines parameters/features of the structure model under investigation. Each lines specifies an object using atomic CIDs divided by spaces. One CID in row denotes an occupancy, two an interactomic distance, three an angle between the atoms and four a torsion angle. For instance:
54
63
 
55
64
  ```
@@ -60,12 +69,31 @@ The file given in `1PGJ_geometry_obj.txt` defines parameters/features of the str
60
69
 
61
70
  Regarding the distances and angles, the procedure is suitable for investigation non-covalently linked atoms. The environment is locally unrestrained (Van der Waals anti-bumping restraints switched off) to provide unbiased results.
62
71
 
72
+ Unrestrained refinement of a small molecule can be performed using the following setting:
73
+
74
+ ```bash
75
+ multixem bootstrap 10000 \
76
+ --hklin 1513945.cif \
77
+ --model 1513945.cif \
78
+ --prefix 1513945_bootstrap10000 \
79
+ --servalcat_args "--unre --hydrogen yes --refine_h --adp aniso --no_solvent --adpr_weight 0 --ncycle 20" \
80
+ --n_bins 20 \
81
+ --n_proc 16 \
82
+ --random_weights
83
+ ```
84
+
63
85
  All currently available options are listed using:
64
86
 
65
87
  ```bash
66
88
  multixem bootstrap --help
67
89
  ```
68
90
 
91
+ The options for Servalcat which could be provided through the `--servalcat_args` parameter to this *Multixem* pipeline can be listed using:
92
+
93
+ ```bash
94
+ servalcat refine_xtal_norefmac --help
95
+ ```
96
+
69
97
  Comparison of isomorphous data sets
70
98
  ===================================
71
99
 
@@ -81,7 +109,7 @@ multixem pipeline \
81
109
  --model insuling_cow.pdb insuling_pig.pdb insuling_people.pdb \
82
110
  --n_bins 30 \
83
111
  --n_proc 4 \
84
- --unify_cell```
112
+ --unify_cell
85
113
  ```
86
114
 
87
115
  Example: Merging and comparison of batches of unmerged diffraction data:
@@ -90,12 +118,11 @@ Example: Merging and comparison of batches of unmerged diffraction data:
90
118
  multixem pipeline \
91
119
  --hklin_unmerged insuling_people_unmerged.mtz \
92
120
  --hklin_free insuling_people.mtz \
93
- -p insulin \
121
+ -p insulin_people_600 \
94
122
  --model insuling_people.pdb \
95
123
  --n_bins 30 \
96
124
  --n_proc 4 \
97
125
  --n_batches 600
98
- ```
99
126
  ```
100
127
 
101
128
  All currently available options are listed using:
@@ -14,7 +14,8 @@ def bootstrap_dataset(
14
14
  seeds=[1001, 1002, 1003],
15
15
  labin: str = "",
16
16
  draw_factor: float = 1.0,
17
- random_resampling: bool = False,
17
+ random_weights: bool = False,
18
+ col_free: str = "",
18
19
  fraction_zero: float = 0.05,
19
20
  ):
20
21
  """
@@ -29,6 +30,12 @@ def bootstrap_dataset(
29
30
  draw_factor (float): Factor for a number of draws in resampling.
30
31
  By default, the number of draws is equal to the number of
31
32
  reflections in each bin (draw_factor==1.0).
33
+ random_weights (bool): Assign random weights instead of
34
+ resampling with replacement.
35
+ col_free (str): Label for free R flag in `mtz_file` which would be used to set
36
+ zero weight for reflections of the free set
37
+ if `fraction_zero` is zero
38
+ fraction zero (float): Sets fraction of reflections with zero weight [0.0, 1.0)
32
39
  Returns:
33
40
  list of str: List of output MTZ filenames created during bootstrapping.
34
41
  """
@@ -69,8 +76,8 @@ def bootstrap_dataset(
69
76
 
70
77
  def resample_random(
71
78
  n: int,
79
+ zero_mask: numpy.ndarray,
72
80
  seed: int = 1001,
73
- fraction_zero: float = 0.05,
74
81
  column_name: str = "llweight",
75
82
  ):
76
83
  """
@@ -80,21 +87,24 @@ def bootstrap_dataset(
80
87
  Args:
81
88
  n (int): Number of items to resample.
82
89
  seed (int): Random seed for reproducibility.
83
- fraction_zero (float): Fraction of zero weights to include in the resampling
84
- (0.0 < fraction_zero < 1.0).
90
+ zero_mask (numpy.ndarray): Optional boolean mask;
91
+ rows set to True are forced to zero.
92
+ Applied if `fraction_zero` is zero.
85
93
  column_name (str): Name of the column to create in the DataFrame.
86
94
  Returns:
87
95
  pandas.Series: Series with the weights for each reflection.
88
96
  """
89
97
  rng = numpy.random.default_rng(seed)
90
98
  df_random = pandas.DataFrame(rng.random(size=n), columns=["index_resample"])
91
- assert 0.0 < fraction_zero < 1.0
92
- n_zero = int(n * fraction_zero)
93
- n_zero = min(n_zero, n)
94
99
  df_weight = df_random["index_resample"].copy()
95
- if n_zero > 0:
96
- lowest_idx = df_random["index_resample"].nsmallest(n_zero).index
97
- df_weight.loc[lowest_idx] = 0.0
100
+ # Set a fraction of weights to zero based on the provided mask
101
+ zero_mask_array = numpy.asarray(zero_mask, dtype=bool)
102
+ if zero_mask_array.shape[0] != n:
103
+ raise ValueError(
104
+ "Length mismatch between zero_mask and bin size in "
105
+ f"random resampling: {zero_mask_array.shape[0]} != {n}"
106
+ )
107
+ df_weight.loc[zero_mask_array] = 0.0
98
108
  # Renormalize so that the total weight equals n
99
109
  weight_sum = df_weight.sum()
100
110
  assert weight_sum > 0
@@ -102,14 +112,26 @@ def bootstrap_dataset(
102
112
 
103
113
  return df_weight.rename(column_name)
104
114
 
105
- if random_resampling:
106
- logging.info(
107
- f"\nBootstrapping dataset {mtz_file} (using random resampling"
108
- f" and keeping fraction of zero weights {fraction_zero})",
109
- )
115
+ if random_weights:
116
+ if abs(fraction_zero) > 1e-6:
117
+ logging.info(
118
+ f"\nBootstrapping dataset {mtz_file} (using random resampling"
119
+ f" and keeping fraction of zero weights {fraction_zero})",
120
+ )
121
+ elif col_free:
122
+ logging.info(
123
+ f"\nBootstrapping dataset {mtz_file} (using random resampling,"
124
+ " free reflections will be excluded)."
125
+ )
126
+ else:
127
+ raise RuntimeError(
128
+ f"Free reflections were not found in {mtz_file} and --fraction_zero"
129
+ " is not set. Cannot assign reflections with zero weight. Aborting."
130
+ )
110
131
  else:
111
132
  logging.info(
112
- f"\nBootstrapping dataset {mtz_file} (using draw factor {draw_factor})",
133
+ "\n"
134
+ f"Bootstrapping dataset {mtz_file} (using a draw factor of {draw_factor})",
113
135
  )
114
136
  mtzs_out = []
115
137
  mtz = gemmi.read_mtz_file(mtz_file)
@@ -119,6 +141,7 @@ def bootstrap_dataset(
119
141
  col.label: col.type for col in mtz.columns if col.label not in ["H", "K", "L"]
120
142
  }
121
143
 
144
+ n_unique_orig = 0
122
145
  # i_col = "IMEAN" # can be just "I" after servalcat fw or sigmaa, or IMEAN?
123
146
  # dropping reflections can cause problems, let's save the filtered dataset as MTZ
124
147
  if labin and labin.split(",")[0] in df.columns:
@@ -128,11 +151,19 @@ def bootstrap_dataset(
128
151
  f"{os.path.splitext(os.path.basename(mtz_file))[0]}_filtered.mtz"
129
152
  )
130
153
  write_mtz_from_df(df, mtz, columns=columns_dict, filename=mtz_filtered_name)
154
+ n_unique_orig = df.shape[0]
131
155
  else:
132
156
  warnings.warn(
133
157
  f"Column {labin} not found in MTZ file {mtz_file}. "
134
158
  f"Using all reflections for bootstrapping."
135
159
  )
160
+ n_unique_expected = gemmi.count_reflections(
161
+ mtz.cell,
162
+ mtz.spacegroup,
163
+ mtz.resolution_high(),
164
+ mtz.resolution_low(),
165
+ unique=True,
166
+ )
136
167
 
137
168
  hkl_array = numpy.array(df[["H", "K", "L"]].values, numpy.int32)
138
169
  hkl_array = numpy.ascontiguousarray(hkl_array, dtype=numpy.int32)
@@ -141,38 +172,56 @@ def bootstrap_dataset(
141
172
  # print(df.head(10))
142
173
  # print(df.describe())
143
174
 
144
- # df_bootstrap1_weight_master = pandas.DataFrame()
175
+ bins = [bin for _, bin in df.groupby("bin")]
176
+
177
+ # Create per-bin masks once and keep them for all bootstrap samples
178
+ zero_mask_bins = []
179
+ if random_weights:
180
+ if abs(fraction_zero) > 1e-6:
181
+ # Create a random mask for reflections with zero weight
182
+ rng = numpy.random.default_rng(seeds[0])
183
+ zero_mask_bins = [rng.random(len(bin)) < fraction_zero for bin in bins]
184
+ elif col_free:
185
+ # Use free reflections to create a mask for refls with zero weight
186
+ zero_mask_bins = [bin[col_free].eq(0).to_numpy() for bin in bins]
187
+ else:
188
+ raise RuntimeError(
189
+ f"Free reflections were not found in {mtz_file} and --fraction_zero"
190
+ " is not set. Cannot assign reflections with zero weight. Aborting."
191
+ )
192
+
145
193
  completeness_list = []
146
194
  for i, seed in enumerate(seeds):
147
- if random_resampling:
148
- df_bootstrap1_weight = pandas.concat(
149
- [
150
- resample_random(len(shell), seed, fraction_zero)
151
- for _, shell in df.groupby("bin")
152
- ],
153
- ignore_index=True,
154
- )
195
+ if random_weights:
196
+ assert len(zero_mask_bins) == len(bins)
197
+ parts = []
198
+ for b, bin in enumerate(bins):
199
+ w = resample_random(
200
+ len(bin),
201
+ zero_mask_bins[b],
202
+ seed=seed,
203
+ )
204
+ parts.append(pandas.Series(w.values, index=bin.index, name="llweight"))
205
+ df_bootstrap1_weight = pandas.concat(parts).sort_index()
155
206
  else:
156
- df_bootstrap1_weight = pandas.concat(
157
- [
158
- resample(len(shell), seed, draw_factor)
159
- for _, shell in df.groupby("bin")
160
- ],
161
- ignore_index=True,
162
- )
163
- # Merge columns H, K, L from df and llweight from df_bootstrap1_weight
207
+ parts = []
208
+ for bin in bins:
209
+ w = resample(len(bin), seed, draw_factor)
210
+ parts.append(pandas.Series(w.values, index=bin.index, name="llweight"))
211
+ df_bootstrap1_weight = pandas.concat(parts).sort_index()
212
+
213
+ # Keep original reflection order to preserve HKL-to-weight mapping.
164
214
  df_bootstrap1_weight_hkl = df[["H", "K", "L"]].copy()
165
- df_bootstrap1_weight_hkl = df_bootstrap1_weight_hkl.merge(
166
- df_bootstrap1_weight, left_index=True, right_index=True
167
- )
215
+ df_bootstrap1_weight_hkl["llweight"] = df_bootstrap1_weight.reindex(
216
+ df.index
217
+ ).values
168
218
  weight_sum = df_bootstrap1_weight.sum()
169
- if weight_sum != len(df):
219
+ if abs(weight_sum - len(df)) > 1e-6:
170
220
  logging.warning(
171
221
  f"Sum of weight coefficients ({weight_sum}) from bootstrap resampling"
172
222
  f" {i} does not match the number of reflections ({len(df)})."
173
223
  )
174
224
 
175
- # TODO: FreeR_flag
176
225
  # Save the llweights in the MTZ file
177
226
  mtz_out_name = (
178
227
  f"{os.path.splitext(os.path.basename(mtz_file))[0]}_llweight{i}.mtz"
@@ -189,13 +238,6 @@ def bootstrap_dataset(
189
238
  n_unique = len(
190
239
  df_bootstrap1_weight_hkl[df_bootstrap1_weight_hkl["llweight"] > 0]
191
240
  )
192
- n_unique_expected = gemmi.count_reflections(
193
- mtz.cell,
194
- mtz.spacegroup,
195
- mtz.resolution_high(),
196
- mtz.resolution_low(),
197
- unique=True,
198
- )
199
241
  completeness = n_unique / n_unique_expected
200
242
  completeness_list.append(completeness)
201
243
 
@@ -209,7 +251,6 @@ def bootstrap_dataset(
209
251
  df_bootstrap1_weight.values
210
252
  )
211
253
 
212
- # TODO: FreeR_flag
213
254
  columns = {i_col: "J", "SIG" + i_col: "Q", "llweight": "I"}
214
255
  if "FreeR_flag" in df.columns:
215
256
  columns["FreeR_flag"] = "I"
@@ -241,18 +282,29 @@ def bootstrap_dataset(
241
282
  )"""
242
283
 
243
284
  completeness_mean = numpy.mean(completeness_list)
244
- completeness_std = numpy.std(completeness_list, ddof=1, mean=completeness_mean)
245
- if random_resampling:
246
- logging.info(
247
- f"Completeness of bootstrap datasets:"
248
- f" {completeness_mean:.2%} ± {completeness_std:.2%}"
249
- f" (using random resampling and fraction of zero weights {fraction_zero})\n"
250
- )
285
+ if random_weights:
286
+ if n_unique_orig:
287
+ logging.info(
288
+ "Completeness of bootstrap datasets"
289
+ " after exclusion of zero-weighted reflections:"
290
+ f" {completeness_mean:.2%}"
291
+ )
292
+ else:
293
+ logging.info(
294
+ "Completeness of bootstrap datasets"
295
+ " after exclusion of zero-weighted reflections:"
296
+ f" {completeness_mean:.2%}\n"
297
+ )
251
298
  else:
299
+ completeness_std = numpy.std(completeness_list, ddof=1, mean=completeness_mean)
252
300
  logging.info(
253
- f"Completeness of bootstrap datasets:"
301
+ "Completeness of bootstrap datasets"
302
+ " after exclusion of zero-weighted reflections:"
254
303
  f" {completeness_mean:.2%} ± {completeness_std:.2%}"
255
- f" (using draw factor {draw_factor})\n"
304
+ f" (using a draw factor of {draw_factor})\n"
256
305
  )
306
+ if n_unique_orig:
307
+ completeness_orig = n_unique_orig / n_unique_expected
308
+ logging.info(f"Completeness of the original dataset: {completeness_orig:.2%}\n")
257
309
 
258
310
  return mtzs_out
@@ -531,31 +531,70 @@ def scatter_plot_histogram(
531
531
 
532
532
  def bootstrap_analyse_stats(jsons, json_ref, idx=0, prefix=""):
533
533
  logging.info(f"Loading {len(jsons)} json files with statistics...")
534
+ # Find out which statistics are available
535
+ # and create related empty lists in data_overall_dict and data_overall[_init]_dict
534
536
  with open(jsons[0]) as f:
535
537
  data_first = json.load(f)
536
- stats_avail = data_first[-1]["data"]["summary"].keys()
537
- data_overall_dict = {stat: [] for stat in stats_avail}
538
- data_overall_init_dict = {stat: [] for stat in stats_avail}
538
+ stats_data_avail = data_first[-1]["data"]["summary"].keys()
539
+ data_overall_dict = {stat: [] for stat in stats_data_avail}
540
+ data_overall_init_dict = {stat: [] for stat in stats_data_avail}
541
+ # Add keys for geometry statistics from geom section
542
+ geom_stats_avail = False
543
+ geom_stats = [
544
+ "Bond distances, non H",
545
+ "Bond angles, non H",
546
+ ]
547
+ for stat in geom_stats:
548
+ if (
549
+ stat in data_first[-1]["geom"]["summary"]["r.m.s.d."].keys()
550
+ and stat in data_first[-1]["geom"]["summary"]["r.m.s.Z"].keys()
551
+ ):
552
+ geom_stats_avail = True
553
+ data_overall_dict[f"Rmsd {stat}"] = []
554
+ data_overall_dict[f"RmsZ {stat}"] = []
555
+ data_overall_init_dict[f"Rmsd {stat}"] = []
556
+ data_overall_init_dict[f"RmsZ {stat}"] = []
557
+ if "weight" in data_first[-1] and len(data_first) >= 3 and geom_stats_avail:
558
+ # First available value for weight is in the 1st cycle, not 0th cycle
559
+ data_overall_dict["weight"] = []
560
+ data_overall_init_dict["weight"] = []
539
561
  """
540
562
  stats_additional = []
541
- for stat in stats_avail:
563
+ for stat in stats_data_avail:
542
564
  if "CC" in stat:
543
565
  data_overall_dict[f"R2_{stat}"] = []
544
566
  data_overall_init_dict[f"R2_{stat}"] = []
545
567
  stats_additional.append(f"R2_{stat}")
546
568
  """
547
569
 
570
+ # Load statistics from json files and append to lists in data_overall[_init]_dict
548
571
  for json_file in jsons:
549
572
  with open(json_file) as f:
550
573
  data_loaded = json.load(f)
551
- for stat in stats_avail:
574
+ for stat in stats_data_avail:
552
575
  data_overall_init_dict[stat].append(
553
576
  data_loaded[0]["data"]["summary"].get(stat, 0)
554
577
  )
555
578
  data_overall_dict[stat].append(
556
579
  data_loaded[-1]["data"]["summary"].get(stat, 0)
557
580
  )
558
-
581
+ if geom_stats_avail:
582
+ for stat in geom_stats:
583
+ data_overall_init_dict[f"Rmsd {stat}"].append(
584
+ data_loaded[0]["geom"]["summary"]["r.m.s.d."].get(stat, 0)
585
+ )
586
+ data_overall_dict[f"Rmsd {stat}"].append(
587
+ data_loaded[-1]["geom"]["summary"]["r.m.s.d."].get(stat, 0)
588
+ )
589
+ data_overall_init_dict[f"RmsZ {stat}"].append(
590
+ data_loaded[0]["geom"]["summary"]["r.m.s.Z"].get(stat, 0)
591
+ )
592
+ data_overall_dict[f"RmsZ {stat}"].append(
593
+ data_loaded[-1]["geom"]["summary"]["r.m.s.Z"].get(stat, 0)
594
+ )
595
+ if "weight" in data_overall_dict.keys():
596
+ data_overall_init_dict["weight"].append(data_loaded[1].get("weight", 0))
597
+ data_overall_dict["weight"].append(data_loaded[-1].get("weight", 0))
559
598
  """
560
599
  if "CC" in stat:
561
600
  # also calculate R = sqrt(1 - CC^2)
@@ -614,12 +653,27 @@ def bootstrap_analyse_stats(jsons, json_ref, idx=0, prefix=""):
614
653
  "R1_llw=0": "R1free",
615
654
  "CCI_llw>0_avg": "CCIworkavg",
616
655
  "CCI_llw=0_avg": "CCIfreeavg",
656
+ "Rmsd Bond distances, non H": "Rmsd Bond distances, non H",
657
+ "Rmsd Bond angles, non H": "Rmsd Bond angles, non H",
658
+ "RmsZ Bond distances, non H": "RmsZ Bond distances, non H",
659
+ "RmsZ Bond angles, non H": "RmsZ Bond angles, non H",
660
+ "weight": "weight",
617
661
  }
618
662
  data_ref = {}
619
663
 
620
664
  with open(json_ref) as f:
621
665
  data_ref_loaded = json.load(f)
622
666
  data_ref = data_ref_loaded[-1]["data"]["summary"]
667
+ if geom_stats_avail:
668
+ for stat in geom_stats:
669
+ data_ref[f"Rmsd {stat}"] = data_ref_loaded[-1]["geom"]["summary"][
670
+ "r.m.s.d."
671
+ ].get(stat, 0)
672
+ data_ref[f"RmsZ {stat}"] = data_ref_loaded[-1]["geom"]["summary"][
673
+ "r.m.s.Z"
674
+ ].get(stat, 0)
675
+ if "weight" in data_overall_dict and len(data_ref_loaded) >= 2:
676
+ data_ref["weight"] = data_ref_loaded[-1].get("weight", 0)
623
677
  if data_ref:
624
678
  logging.info(f"Loaded reference statistics from {json_ref}")
625
679
  else:
@@ -631,8 +685,7 @@ def bootstrap_analyse_stats(jsons, json_ref, idx=0, prefix=""):
631
685
 
632
686
  distrs = {"init": {}, "final": {}}
633
687
  llweight_R1_outliers = []
634
- # for stat in list(stats_avail) + stats_additional:
635
- for stat in stats_avail:
688
+ for stat in data_overall_dict.keys():
636
689
  stat_ref = {}
637
690
  if (
638
691
  json_ref
@@ -672,6 +725,7 @@ def bootstrap_analyse_stats(jsons, json_ref, idx=0, prefix=""):
672
725
  json.dump(distrs, f, indent=2, default=json_numpy_converter)
673
726
  logging.info(f"Saved statistics to {json_filename}")
674
727
 
728
+ # Filter outliers based on R1 values and re-analyse the distributions
675
729
  if llweight_R1_outliers:
676
730
  distrs_filtered = {"init": {}, "final": {}}
677
731
  data_overall_filtered_dict = {}
@@ -681,7 +735,7 @@ def bootstrap_analyse_stats(jsons, json_ref, idx=0, prefix=""):
681
735
  " R1 value outliers which will be exluded from the following analysis\n"
682
736
  f" {llweight_R1_outliers if llweight_R1_outliers else ''}"
683
737
  )
684
- for stat in stats_avail:
738
+ for stat in data_overall_dict.keys():
685
739
  data_overall_filtered_dict[stat] = [
686
740
  val
687
741
  for i, val in enumerate(data_overall_dict[stat])
@@ -333,17 +333,26 @@ def create_parser():
333
333
  ),
334
334
  )
335
335
  common_refinement_parent.add_argument(
336
- "--random_resampling",
336
+ "--random_weights",
337
337
  action="store_true",
338
- help=("Use random resampling and keeping a fraction of zero weights."),
338
+ help=(
339
+ "Use fractional-random-weight bootstrap method, i.e. assign random weights"
340
+ " to reflection likelihood terms in the target log-likelihood function."
341
+ " A fraction of reflection will be assigned zero weights"
342
+ " based on the provided free flag in --hklin_free or --hklin,"
343
+ " or randomly if no free flag is provided, controlled by --fraction_zero."
344
+ ),
339
345
  )
340
346
  common_refinement_parent.add_argument(
341
347
  "--fraction_zero",
342
- type=positive_float,
343
- default=0.05,
348
+ type=float,
349
+ default=0.0,
344
350
  help=(
345
- "If random resampling is used, this specifies the fraction of"
351
+ "If random weighting is used, this specifies the fraction of"
346
352
  " zero weights to include. Must be in [0.0, 1.0)."
353
+ " If a value of 0.0 is set (default), zero weights will be assigned"
354
+ " according to the free flags in --hklin_free or --hklin."
355
+ " Recommended value: 0.05."
347
356
  ),
348
357
  )
349
358
 
@@ -484,10 +493,10 @@ def create_parser():
484
493
  parser.error("--unre requires --bootstrap to be provided.")
485
494
  if args.bootstrap and args.model_dir:
486
495
  validate_model_dir(args, args.bootstrap)
487
- if args.random_resampling and (
488
- args.fraction_zero <= 0.0 or args.fraction_zero >= 1.0
496
+ if args.random_weights and (
497
+ args.fraction_zero < 0.0 or args.fraction_zero >= 1.0
489
498
  ):
490
- parser.error("--fraction_zero must be in the range (0.0, 1.0).")
499
+ parser.error("--fraction_zero must be in the range [0.0, 1.0).")
491
500
 
492
501
  def validate_mean(args):
493
502
  validate_common(args)
@@ -511,10 +520,10 @@ def create_parser():
511
520
  args.models = sorted(model_files)[: args.n_samples]
512
521
  if args.model_dir:
513
522
  validate_model_dir(args, args.n_samples)
514
- if args.random_resampling and (
515
- args.fraction_zero <= 0.0 or args.fraction_zero >= 1.0
523
+ if args.random_weights and (
524
+ args.fraction_zero < 0.0 or args.fraction_zero >= 1.0
516
525
  ):
517
- parser.error("--fraction_zero must be in the range (0.0, 1.0).")
526
+ parser.error("--fraction_zero must be in the range [0.0, 1.0).")
518
527
 
519
528
  pipeline_parser.set_defaults(func=validate_pipeline)
520
529
  mean_parser.set_defaults(func=validate_mean)
@@ -534,7 +543,12 @@ def create_parser():
534
543
  return parser
535
544
 
536
545
 
537
- def check_reflection_file_columns(hklin, unmerged=False, prefer_amplitude=False):
546
+ def check_reflection_file_columns(
547
+ hklin: str,
548
+ unmerged: bool = False,
549
+ prefer_amplitude: bool = False,
550
+ search_col_free: bool = False,
551
+ ):
538
552
  """
539
553
  Check the input reflection file for the presence of intensities,
540
554
  amplitudes and Friedel pairs. Find the column labels and decide which to use.
@@ -544,6 +558,7 @@ def check_reflection_file_columns(hklin, unmerged=False, prefer_amplitude=False)
544
558
  unmerged (bool): Whether the input file is unmerged data.
545
559
  prefer_amplitude (bool): If both intensities and amplitudes are found,
546
560
  prefer amplitudes over intensities.
561
+ search_col_free (bool): Whether to search only for a free reflection column
547
562
 
548
563
  Returns:
549
564
  tuple: A tuple containing three boolean values:
@@ -570,9 +585,17 @@ def check_reflection_file_columns(hklin, unmerged=False, prefer_amplitude=False)
570
585
  col_sigfplusminus = []
571
586
  col_fmean = []
572
587
  col_sigxmean = []
588
+ col_free = ""
573
589
 
574
590
  for column in m.columns:
575
- if column.type == "K":
591
+ if not col_free and column.type == "I":
592
+ if "free" in column.label.lower() and "flag" in column.label.lower():
593
+ col_free = column.label
594
+ logging.info(f"Column with free R flag (type I) found: {column.label}")
595
+ if search_col_free:
596
+ logging.info("Free R flag column found successfully.")
597
+ return None, None, col_free
598
+ elif column.type == "K":
576
599
  logging.info(
577
600
  f"Column with intensity (type K, Friedel pairs) found: {column.label}"
578
601
  )
@@ -653,7 +676,7 @@ def check_reflection_file_columns(hklin, unmerged=False, prefer_amplitude=False)
653
676
  )
654
677
  logging.info(f"Using these columns for refinement: {labin}")
655
678
 
656
- return labin, anom
679
+ return labin, anom, col_free
657
680
 
658
681
 
659
682
  def copy_cell_mtz(mtz_input, mtz_reference):
@@ -846,7 +869,7 @@ def merge_in_groups(
846
869
  # Scan the columns of the input unmerged MTZ file
847
870
  # and check if Friedel pairs are present or not
848
871
  anom = False
849
- labin, anom = check_reflection_file_columns(m, unmerged=True)
872
+ labin, anom, col_free = check_reflection_file_columns(m, unmerged=True)
850
873
  # print(m.dataset(0).wavelength) == 0.0
851
874
  # print(m.dataset(1).wavelength) OK
852
875
  # print(m.datasets[0].wavelength) == 0.0
@@ -1748,6 +1771,7 @@ def main():
1748
1771
 
1749
1772
  mtzs_merged = []
1750
1773
  labins = []
1774
+ cols_free = []
1751
1775
  bin_stats_lists = []
1752
1776
  n_expected_list = []
1753
1777
  binner_master = None
@@ -1865,8 +1889,8 @@ def main():
1865
1889
  else:
1866
1890
  mtz = gemmi.read_mtz_file(hklin_i)
1867
1891
  mtzs_merged.append(hklin_i)
1868
- dmax = mtz.resolution_high()
1869
- dmin = mtz.resolution_low()
1892
+ dmin = mtz.resolution_high()
1893
+ dmax = mtz.resolution_low()
1870
1894
  # Check for None or nan values and recalculate if necessary
1871
1895
  if dmax is None or dmin is None or numpy.isnan(dmax) or numpy.isnan(dmin):
1872
1896
  d_array = mtz.cell.calculate_d_array(mtz.make_miller_array())
@@ -1880,11 +1904,12 @@ def main():
1880
1904
  f"Unit cell: {mtz.cell.a:.3f} {mtz.cell.b:.3f} {mtz.cell.c:.3f}"
1881
1905
  f" {mtz.cell.alpha:.3f} {mtz.cell.beta:.3f} {mtz.cell.gamma:.3f}"
1882
1906
  )
1883
- labin, anom_present = check_reflection_file_columns(
1907
+ labin, anom_present, col_free = check_reflection_file_columns(
1884
1908
  mtz, unmerged=False, prefer_amplitude=args.amplitude
1885
- )
1909
+ ) # col_free may be reset later if args.hklin_free is given
1886
1910
  # elif i_present and not f_present: TODO FW
1887
1911
  labins.append(labin)
1912
+ cols_free.append(col_free)
1888
1913
  bin_stats_lists.append([])
1889
1914
  # TODO: check and fix n_expected
1890
1915
  n_expected = gemmi.count_reflections(mtz.cell, mtz.spacegroup, dmin, dmax)
@@ -1902,6 +1927,7 @@ def main():
1902
1927
  mtz.make_1_d2_array(),
1903
1928
  mtz.get_cell(),
1904
1929
  )
1930
+ logging.info("")
1905
1931
  n_mtzs_merged = len(mtzs_merged)
1906
1932
  assert n_mtzs_merged == len(labins)
1907
1933
 
@@ -1919,6 +1945,39 @@ def main():
1919
1945
  mtzs_merged, binner_master, labins, bin_stats_matrix, n_expected_list
1920
1946
  )
1921
1947
 
1948
+ if args.hklin_free:
1949
+ logging.info(f"Free R flag data file: {args.hklin_free}")
1950
+ mtz_free = gemmi.read_mtz_file(args.hklin_free)
1951
+ mtzs_merged.append(args.hklin_free)
1952
+ dmin_free = mtz_free.resolution_high()
1953
+ dmax_free = mtz_free.resolution_low()
1954
+ # Check for None or nan values and recalculate if necessary
1955
+ if (
1956
+ dmax_free is None
1957
+ or dmin_free is None
1958
+ or numpy.isnan(dmax_free)
1959
+ or numpy.isnan(dmin_free)
1960
+ ):
1961
+ d_array_free = mtz_free.cell.calculate_d_array(mtz_free.make_miller_array())
1962
+ dmax_free = max(d_array_free)
1963
+ dmin_free = min(d_array_free)
1964
+ logging.info(f"Resolution limits: {dmax_free:.3f}" f" - {dmin_free:.3f} A")
1965
+ logging.info(
1966
+ f"Space group: {mtz_free.spacegroup.hm} (No. {mtz_free.spacegroup.number})"
1967
+ )
1968
+ logging.info(
1969
+ "Unit cell:"
1970
+ f" {mtz_free.cell.a:.3f} {mtz_free.cell.b:.3f} {mtz_free.cell.c:.3f}"
1971
+ f" {mtz_free.cell.alpha:.3f} {mtz_free.cell.beta:.3f}"
1972
+ f" {mtz_free.cell.gamma:.3f}"
1973
+ )
1974
+ # Overwrite cols_free
1975
+ _, _, col_free = check_reflection_file_columns(
1976
+ args.hklin_free, unmerged=False, search_col_free=True
1977
+ )
1978
+ cols_free = len(cols_free) * [col_free]
1979
+ logging.info("")
1980
+
1922
1981
  if args.command == "pipeline" and args.unify_cell:
1923
1982
  for i, mtz in enumerate(mtzs_merged):
1924
1983
  if i == 0:
@@ -2007,8 +2066,8 @@ def main():
2007
2066
  else args.n_samples
2008
2067
  )
2009
2068
  # TODO: some features assume only single model...
2010
- for i_mtz, (mtz_in, labin, model) in enumerate(
2011
- zip(mtzs_merged, labins, models)
2069
+ for i_mtz, (mtz_in, labin, model, col_free) in enumerate(
2070
+ zip(mtzs_merged, labins, models, cols_free)
2012
2071
  ):
2013
2072
  if args.command == "bootstrap":
2014
2073
  idx = 0
@@ -2029,7 +2088,8 @@ def main():
2029
2088
  seeds=range(1001, 1001 + n_samples),
2030
2089
  labin=labin,
2031
2090
  draw_factor=args.draw_factor,
2032
- random_resampling=args.random_resampling,
2091
+ random_weights=args.random_weights,
2092
+ col_free=col_free,
2033
2093
  fraction_zero=args.fraction_zero,
2034
2094
  )
2035
2095
  if args.model_dir and args.models:
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "multixem"
3
- version = "0.2"
3
+ version = "0.3.1"
4
4
  description = "Refinement pipeline for multiple data sets in structure biology"
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.9,<4.0"
@@ -42,8 +42,6 @@ exclude = [
42
42
  "tests*",
43
43
  "docs_backup*",
44
44
  "scripts_backup*",
45
- "multixem/scripts",
46
- "multixem/prototypes",
47
45
  ]
48
46
 
49
47
  [build-system]
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes