fpembed 0.2.2__tar.gz → 0.2.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. fpembed-0.2.2/README.md → fpembed-0.2.3/PKG-INFO +565 -596
  2. fpembed-0.2.2/PKG-INFO → fpembed-0.2.3/README.md +548 -619
  3. {fpembed-0.2.2 → fpembed-0.2.3}/pyproject.toml +7 -22
  4. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/__init__.py +1 -1
  5. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed.egg-info/PKG-INFO +17 -71
  6. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed.egg-info/SOURCES.txt +1 -19
  7. fpembed-0.2.3/src/fpembed.egg-info/requires.txt +4 -0
  8. fpembed-0.2.2/src/fpembed.egg-info/requires.txt +0 -11
  9. fpembed-0.2.2/tests/test_cache.py +0 -357
  10. fpembed-0.2.2/tests/test_compression.py +0 -297
  11. fpembed-0.2.2/tests/test_compression_blockwise.py +0 -156
  12. fpembed-0.2.2/tests/test_compression_projection.py +0 -196
  13. fpembed-0.2.2/tests/test_dtype.py +0 -285
  14. fpembed-0.2.2/tests/test_dtype_properties.py +0 -359
  15. fpembed-0.2.2/tests/test_embedding_cache.py +0 -300
  16. fpembed-0.2.2/tests/test_error_transparency.py +0 -218
  17. fpembed-0.2.2/tests/test_fingerprints.py +0 -459
  18. fpembed-0.2.2/tests/test_generator.py +0 -698
  19. fpembed-0.2.2/tests/test_hashing.py +0 -219
  20. fpembed-0.2.2/tests/test_input_validation.py +0 -280
  21. fpembed-0.2.2/tests/test_new_fp_type_matrix.py +0 -96
  22. fpembed-0.2.2/tests/test_packaging.py +0 -46
  23. fpembed-0.2.2/tests/test_properties.py +0 -392
  24. fpembed-0.2.2/tests/test_properties_050.py +0 -378
  25. fpembed-0.2.2/tests/test_resources_immutability.py +0 -448
  26. fpembed-0.2.2/tests/test_smiles_utils.py +0 -120
  27. {fpembed-0.2.2 → fpembed-0.2.3}/LICENSE +0 -0
  28. {fpembed-0.2.2 → fpembed-0.2.3}/setup.cfg +0 -0
  29. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/compression.py +0 -0
  30. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/compression_blockwise.py +0 -0
  31. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/compression_projection.py +0 -0
  32. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/compression_projection_cache.py +0 -0
  33. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/dtype_support.py +0 -0
  34. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/embedding_cache.py +0 -0
  35. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/generator.py +0 -0
  36. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/generator_smiles.py +0 -0
  37. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/hashing.py +0 -0
  38. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/input_validation.py +0 -0
  39. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/manifest.py +0 -0
  40. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/py.typed +0 -0
  41. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/skfp_factory.py +0 -0
  42. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed/smiles_utils.py +0 -0
  43. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed.egg-info/dependency_links.txt +0 -0
  44. {fpembed-0.2.2 → fpembed-0.2.3}/src/fpembed.egg-info/top_level.txt +0 -0
@@ -1,596 +1,565 @@
1
- # FPembed - Generalized Molecular Fingerprint Embeddings
2
-
3
- [![PyPI version](https://badge.fury.io/py/fpembed.svg)](https://badge.fury.io/py/fpembed)
4
- [![DOI](https://img.shields.io/badge/DOI-10.5281/zenodo.21447996-blue)](https://doi.org/10.5281/zenodo.21447996)
5
-
6
-
7
- ## Contents
8
-
9
- - [Introduction](#introduction)
10
- - [Installation](#installation)
11
- - [Conda Environment](#conda-environment)
12
- - [Quick Start](#quick-start)
13
- - [Why Use Embedded Fingerprints?](#why-use-embedded-fingerprints)
14
- - [Storage Size](#storage-size)
15
- - [ML Training and Inference Speed](#ml-training-and-inference-speed)
16
- - [Memory During ML Training](#memory-during-ml-training)
17
- - [Compression Overhead](#compression-overhead)
18
- - [Sample Efficiency](#sample-efficiency)
19
- - [Summary of Advantages](#summary-of-advantages)
20
- - [Supported Fingerprint Types](#supported-fingerprint-types)
21
- - [Compression Methods](#compression-methods)
22
- - [Method Reference](#method-reference)
23
- - [The `method` Parameter](#the-method-parameter)
24
- - [Method-Specific Parameters (`method_params`)](#method-specific-parameters-method_params)
25
- - [Code Examples](#code-examples)
26
- - [Choosing a Method](#choosing-a-method)
27
- - [Output Dtype](#output-dtype)
28
- - [Representation Manifest](#representation-manifest)
29
- - [Compression Limits](#compression-limits)
30
- - [Performance Characteristics](#performance-characteristics)
31
- - [Project Structure](#project-structure)
32
- - [Running the Demo App](#running-the-demo-app)
33
- - [Datasets](#datasets)
34
- - [License](#license)
35
- - [Legal & Trademark Notice](#legal--trademark-notice)
36
- - [Commercial & Research Services](#commercial--research-services)
37
- - [AI disclosure](#ai-disclosure)
38
- - [Support](#support)
39
-
40
- ## Introduction
41
-
42
- A Python package for generating compressed molecular fingerprint embeddings, backed by [scikit-fingerprints](https://github.com/scikit-fingerprints/scikit-fingerprints). Supports ten binary fingerprint types through a single unified class.
43
-
44
- FPembed compresses standard molecular fingerprints into compact vectors suitable for machine-learning models, using a choice of pluggable compression methods — block-wise weighting schemes (including the default weighted binary masking) or global projections (see [Compression Methods](#compression-methods)). The package accepts SMILES, SELFIES, and RDKit Mol objects as input.
45
-
46
- At a glance: a feature vector reduced from L to D = L / C dimensions (e.g. 16x fewer features at C=16), a drop-in fit for any scikit-learn model, and — in the evaluated workloads — accuracy comparable to raw fingerprints. The dimensionality reduction is exact and guaranteed; the downstream runtime, memory, and accuracy effects depend on your model, hardware, and dataset (see [Why Use Embedded Fingerprints?](#why-use-embedded-fingerprints) for what is guaranteed versus workload-dependent). If you just want to try it, jump to [Quick Start](#quick-start).
47
-
48
- The concept of compressing molecular fingerprints via weighted binary masking was originally introduced for Morgan fingerprints in the eMFP paper:
49
-
50
- > Nuñez-Andrade, E. A., Vidal-Daza, I., Gomez-Bombarelli, R., Ryan, J. W., & Martin-Martinez, F. J. (2025).
51
- > Embedded Morgan Fingerprints for more efficient molecular property predictions with machine learning.
52
- > *ChemRxiv* (preprint). https://doi.org/10.26434/chemrxiv-2025-6hfp8
53
-
54
- ```bibtex
55
- @article{nunez2025emfp,
56
- author = {Nu{\~n}ez-Andrade, Emilio A. and Vidal-Daza, Isaac and Gomez-Bombarelli, Rafael and Ryan, James W. and Martin-Martinez, Francisco J.},
57
- title = {Embedded {Morgan} Fingerprints for more efficient molecular property predictions with machine learning},
58
- journal = {ChemRxiv},
59
- year = {2025},
60
- doi = {10.26434/chemrxiv-2025-6hfp8},
61
- note = {Preprint}
62
- }
63
- ```
64
-
65
- Original concept repository: [MMLabCodes/eMFP](https://github.com/MMLabCodes/eMFP)
66
-
67
- ## Installation
68
-
69
- The package is installed from the working tree with an editable install — clone the repository, then install `fpembed` and its core dependencies (rdkit, numpy, selfies, scikit-fingerprints) in one step:
70
-
71
- ```bash
72
- git clone https://github.com/Sciencealone/fpembed.git
73
- cd fpembed
74
- pip install -e .
75
- ```
76
-
77
- Add the demo app dependencies (nicegui, optuna, pandas, scikit-learn, etc.) with the `app` extra:
78
-
79
- ```bash
80
- pip install -e ".[app]"
81
- ```
82
-
83
- > **Future consumer path.** Once a release is published to PyPI, `pip install fpembed` (or `pip install fpembed[app]`) will install the packaged wheel. That path is not the repository-setup route above and is listed here only for eventual published-package users; the working tree is always installed editable.
84
-
85
- ### Conda Environment
86
-
87
- A full conda environment is provided for reproducibility. It is authoritative for the development setup and installs `fpembed` editable (`-e .`) from the working tree:
88
-
89
- ```bash
90
- conda env create -f environment.yml
91
- conda activate fpembed
92
- ```
93
-
94
- This installs all dependencies and the `fpembed` package in editable mode.
95
-
96
- ### Dependency Files
97
-
98
- Three files describe dependencies, each with a distinct role — do not confuse the tested lock with the metadata ranges:
99
-
100
- - **`pyproject.toml`** — package metadata. Declares the version ranges of `fpembed`'s own dependencies (core and the `app` extra) that a consumer resolves against.
101
- - **`requirements.txt`** — the third-party application lock: exact pins of the app's runtime dependencies, tested together. It does **not** self-pin `fpembed`; the package itself is installed editable via `pip install -e .`.
102
- - **`environment.yml`** — the authoritative conda environment. It recreates the full development environment and installs `fpembed` editable (`-e .`). `requirements.txt` and `environment.yml` are kept in sync, with `environment.yml` as the source of truth.
103
-
104
- ## Quick Start
105
-
106
- ### Single Molecule (SMILES)
107
-
108
- ```python
109
- from fpembed import EmbeddedFingerprintGenerator
110
-
111
- gen = EmbeddedFingerprintGenerator(
112
- fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}
113
- )
114
-
115
- # Generate compressed fingerprint from SMILES
116
- emb = gen.GetFingerprintFromSmiles("CCO")
117
- print(emb.shape) # (128,)
118
- ```
119
-
120
- ### Different Fingerprint Types
121
-
122
- ```python
123
- # Atom Pair fingerprint
124
- gen_ap = EmbeddedFingerprintGenerator(
125
- fp_type="atom_pair", fp_size=2048, compression=16,
126
- fp_params={"min_distance": 1, "max_distance": 30}
127
- )
128
-
129
- # Topological Torsion fingerprint
130
- gen_tt = EmbeddedFingerprintGenerator(
131
- fp_type="topological_torsion", fp_size=2048, compression=16,
132
- fp_params={"torsion_atom_count": 4}
133
- )
134
- ```
135
-
136
- ### Single Molecule (SELFIES)
137
-
138
- ```python
139
- emb = gen.GetFingerprintFromSelfies("[C][C][O]")
140
- print(emb.shape) # (128,)
141
- ```
142
-
143
- ### Batch Processing
144
-
145
- ```python
146
- smiles_list = ["CCO", "c1ccccc1", "CC(=O)O", "invalid_smiles"]
147
-
148
- embeddings, invalid_indices = gen.GetFingerprintsFromSmiles(smiles_list)
149
- print(embeddings.shape) # (3, 128) - 3 valid molecules
150
- print(invalid_indices) # [3] - index of invalid SMILES
151
- ```
152
-
153
- ### Bounded-Memory Iteration
154
-
155
- The batch methods above collect every valid embedding before stacking them into one matrix. At scale that peak allocation dominates: one million molecules at D=1024 needs about 4.1 GB as float32 or 8.2 GB as float64 for the final matrix alone, before counting the individual arrays held while it is built. When the input does not fit in memory, iterate instead:
156
-
157
- ```python
158
- for index, emb in gen.IterFingerprintsFromSmiles(smiles_iterable):
159
- if emb is None:
160
- continue # normal invalid input at this index
161
- store(index, emb) # consume or write one row at a time
162
- ```
163
-
164
- `IterFingerprints`, `IterFingerprintsFromSmiles`, and `IterFingerprintsFromSelfies` accept any iterable, pull one input at a time, and yield `(index, embedding_or_none)`. Indices match input order and include invalid entries, which yield `None`; for `IterFingerprints`, a `None` molecule also yields `None`. Unexpected computation errors surface when the item is consumed, matching the single-molecule methods. The iterator never materializes the input or accumulates rows, so peak memory is yours to bound.
165
-
166
- **Memory contract.** These APIs are not streaming in the sense of a total process cap. The `random_projection` matrix cache is bounded by retained bytes, but a matrix referenced by a live generator stays resident, so the cache budget is not a process-wide limit. Embeddings you keep are yours to release, and `dtype` reduces the returned and stored size, not the peak transient memory during compression.
167
-
168
- ### Raw Fingerprint (No Compression)
169
-
170
- ```python
171
- gen_raw = EmbeddedFingerprintGenerator(
172
- fp_type="ecfp", fp_size=2048, compression=None, fp_params={"radius": 2}
173
- )
174
- fp = gen_raw.GetFingerprintFromSmiles("CCO")
175
- print(fp.shape) # (2048,)
176
- ```
177
-
178
- ### Standalone Compression Function
179
-
180
- ```python
181
- import numpy as np
182
- from fpembed import compress_fingerprint
183
-
184
- fp = np.random.randint(0, 2, size=2048).astype(np.float64)
185
- emb = compress_fingerprint(fp, size=16)
186
- print(emb.shape) # (1, 128)
187
- ```
188
-
189
- **Input contract.** `compress_fingerprint` accepts 1-D `(L,)` and 2-D `(N, L)` arrays of real numeric dtype: boolean, signed or unsigned integer, or floating. Non-real dtypes (complex, object, string, datetime) raise `TypeError`; floating input containing NaN or either infinity raises `ValueError`. Integer output (`dtype="uint16"`) additionally requires binary input and represents integer codes or counts, not normalized floats. Neither validation nor compression mutates the array you pass in.
190
-
191
- ### Parameter Hashing
192
-
193
- ```python
194
- from fpembed import fp_params_hash
195
-
196
- h = fp_params_hash("ecfp", {"radius": 2})
197
- print(h) # 16-char hex string, stable across sessions
198
- ```
199
-
200
- ### Caching for Repeated Lookups
201
-
202
- ```python
203
- gen = EmbeddedFingerprintGenerator(
204
- fp_type="ecfp", fp_size=2048, compression=16,
205
- fp_params={"radius": 2}, cache_size=1024
206
- )
207
-
208
- # First call computes and caches
209
- emb = gen.GetFingerprintFromSmiles("CCO")
210
-
211
- # Second call returns cached result
212
- emb2 = gen.GetFingerprintFromSmiles("CCO")
213
-
214
- print(gen.cache_info()) # CacheInfo(hits=1, misses=1, maxsize=1024, currsize=1)
215
- gen.clear_cache()
216
- ```
217
-
218
- ### SMILES Canonicalization and Chemical Standardization
219
-
220
- FPembed canonicalizes SMILES spelling before keying and before parsing, so the same molecule written two ways produces the same embedding: the pipeline works with the canonical form (atom ordering and RDKit's aromatic form). That makes results spelling-invariant for atom-order-sensitive backends, and it is why the cache key can be built from the canonical string.
221
-
222
- Canonicalization is not chemical standardization. FPembed does not remove salts, normalize tautomers, pick a protonation state, or impose a stereochemistry policy, because those decisions change scientific meaning and are the caller's to make. Two inputs that differ by a counter-ion or a tautomer stay distinct. If your workflow needs standardization, apply it before calling FPembed and describe it with `representation_manifest(preprocessing=...)` (see [Representation Manifest](#representation-manifest)).
223
-
224
- ## Why Use Embedded Fingerprints?
225
-
226
- Predictive accuracy is one axis of comparison between raw and embedded fingerprints - and the gap can be narrow, especially on large datasets where raw fingerprints have enough data to exploit all 2048 bits directly. However, accuracy is not the only metric that matters. The one guaranteed advantage is dimensionality: an embedded vector has D = L / C features instead of L, and the stored byte count follows directly from D and the chosen dtype (calculated exactly in [Storage Size](#storage-size)). The operational benefits that follow — training and inference speed, memory during training, sample efficiency — are consequences of fewer features, but their magnitude is workload-, model-, and hardware-dependent, not a fixed guarantee.
227
-
228
- The core argument is not "embedded fingerprints are always more accurate" but rather "embedded fingerprints can achieve comparable accuracy at a fraction of the feature count, and therefore of the downstream computational cost."
229
-
230
- ### Storage Size
231
-
232
- The output dimensionality (D = L / C) is deterministic and independent of dataset, model, or method; the byte counts below follow from D and the stated dtype. The size figures are per representation under the named dtype assumptions (float64 = 8 bytes/value, uint16 = 2 bytes/value):
233
-
234
- | Representation | Per-molecule (L=2048) | Per-molecule (L=4096) | 100K molecules (L=2048) |
235
- |---|---|---|---|
236
- | Raw binary FP (float64, one byte-inflated bit per element) | 16 KB | 32 KB | ~1.6 GB |
237
- | Raw binary FP (bit-packed, 1 bit per element) | 256 B | 512 B | ~25 MB |
238
- | Embedded, C=16 (float64) | 1 KB | 2 KB | ~100 MB |
239
- | Embedded, C=16 (uint16) | 256 B | 512 B | ~25 MB |
240
- | Embedded, C=32 (float64, `uniform`/projection only) | 512 B | 1 KB | ~50 MB |
241
-
242
- A word on the baseline. Comparing an embedding against a raw fingerprint *stored as float64* overstates the storage gain, because a raw fingerprint is binary and would sensibly be bit-packed. Against **bit-packed** raw fingerprints, the `geometric` embedding at C=16 in float64 is actually *larger* (1 KB vs 256 B per molecule at L=2048): geometric at C=16 is a reversible re-encoding — it packs each 16-bit block into a single wide value, carrying the same information in fewer but larger numbers. Storing that embedding as `uint16` (see [Output Dtype](#output-dtype)) brings it back to parity with bit-packed input.
243
-
244
- So raw byte savings are **not** the reason to embed. The genuine, unconditional advantage is the **downstream feature count**: the embedded vector has D = L / C features instead of L. Downstream training, inference, and the in-memory feature matrix then shrink roughly in proportion to the feature count, and that reduction is what the rest of this section quantifies. Keep the scaling directional rather than universal: similarity on bit-packed fingerprints works on packed words, tree learners subsample features and pay fixed costs, and approximate-neighbor indexes change the comparison, so none of these costs follows the feature count exactly.
245
-
246
- ### ML Training and Inference Speed
247
-
248
- The downstream ML model operates on the feature vector. Fewer features means faster training and prediction:
249
-
250
- - **Tree-based models (Random Forest, XGBoost)**: Feature splitting cost is proportional to the number of features. Going from 2048 to 128 features means each tree split considers ~16x fewer candidates. For hyperparameter searches (e.g., Optuna with hundreds of trials), this compounds into significant wall-clock savings.
251
- - **Neural networks**: The first dense layer's weight matrix shrinks from `(2048 x hidden)` to `(128 x hidden)` - 16x fewer parameters and 16x fewer multiply-adds per forward pass.
252
- - **Distance-based methods (k-NN, similarity search)**: Pairwise distance computation is O(N² x D). Reducing D from 2048 to 128 shrinks the distance computation itself by 16x; index build and query overhead, memory traffic, and candidate re-ranking sit on top of that and reduce the realized speedup.
253
-
254
- ### Memory During ML Training
255
-
256
- During model training, the feature matrix for N=100K molecules occupies `(100000, 2048)` float64 = ~1.6 GB for raw fingerprints, versus `(100000, 128)` = ~100 MB for embedded. Tree-based models create internal copies and histograms proportional to feature count. GPU-based models benefit from smaller input tensors that allow larger batch sizes and better hardware utilization.
257
-
258
- ### Compression Overhead
259
-
260
- The compression step itself is small for block-wise methods, but its cost is not universal. Under warm-cache conditions, and on the machine where the project measured it, block-wise compression added on the order of 1 ms per 1000 molecules. Treat that as an order-of-magnitude indication with `L`, dtype, and hardware attached, not as a constant. Cold-cache cost is higher: the projection methods build or fetch a projection artifact on first use, and `random_projection` additionally pays to draw and store its `(L, D)` matrix.
261
-
262
- The total pipeline cost is:
263
-
264
- - **Raw**: `skfp generation time`
265
- - **Embedded**: `skfp generation time + compression time`
266
-
267
- Measure both phases on the target hardware before relying on the overhead being negligible. The downstream ML saving from 128 versus 2048 features is usually larger, but that too is workload-dependent.
268
-
269
- ### Sample Efficiency
270
-
271
- High-dimensional spaces (2048 binary features) suffer from the curse of dimensionality - distances become less meaningful and models need more data to fill the space. Compressing to 128 dense features can act as a form of regularization. In practice, embedded fingerprints have reached good predictive performance with fewer training samples than raw fingerprints on the datasets we examined; this is a workload- and model-dependent effect, not a guarantee, and it is most likely to matter when labeled molecular data is scarce or expensive to obtain.
272
-
273
- ### Summary of Advantages
274
-
275
- | Metric | Raw FP (L=2048) | Embedded FP (D=128) | Advantage |
276
- |---|-------------------------|---------------------------------|-----------------------|
277
- | Feature matrix memory (100K mols) | ~1.6 GB | ~100 MB | 16x smaller |
278
- | Per-molecule storage vs float64 raw | 16 KB | 1 KB (256 B as uint16) | 16x smaller (64x) |
279
- | Per-molecule storage vs bit-packed raw | 256 B | 256 B as uint16 (1 KB float64) | Parity, not a win (see [Storage Size](#storage-size)) |
280
- | Tree model training speed | Baseline | ~16x fewer split candidates | Workload-dependent |
281
- | Neural net first-layer params | 2048 x H | 128 x H | 16x fewer |
282
- | Pairwise distance computation | O(N² x 2048) | O(N² x 128) | ~16x fewer ops |
283
- | Small-dataset accuracy | Baseline | Can be superior (regularization) | Model-dependent |
284
- | Large-dataset accuracy | Slightly higher ceiling | Comparable | Marginal tradeoff |
285
-
286
- The guaranteed columns are the feature count and its exact consequences (parameter counts, per-molecule byte counts under a named dtype). The speed and accuracy columns are directional: they follow from fewer features but their realized magnitude depends on the model, hardware, and dataset. The choice between raw and embedded fingerprints is a classic accuracy-vs-efficiency tradeoff — embedded fingerprints trade a small amount of information for a smaller, denser feature vector, which is a practical default for many molecular ML workflows.
287
-
288
- ## Supported Fingerprint Types
289
-
290
- | Type | `fp_type` | Type-specific params |
291
- |------|-----------|---------------------|
292
- | Extended Connectivity (ECFP) | `ecfp` | `radius` (default 2) |
293
- | Atom Pair | `atom_pair` | `min_distance` (1), `max_distance` (30) |
294
- | Topological Torsion | `topological_torsion` | `torsion_atom_count` (4) |
295
- | RDKit | `rdkit` | `min_path` (1), `max_path` (7) |
296
- | Layered | `layered` | `min_path` (1), `max_path` (7) |
297
- | Pattern | `pattern` | (none) |
298
- | Avalon | `avalon` | (none) |
299
- | SECFP | `secfp` | `radius` (3), `min_radius` (1) |
300
- | MHFP | `mhfp` | `radius` (3), `min_radius` (1) |
301
- | MAP4 | `map` | `radius` (default 2) |
302
-
303
- ## Compression Methods
304
-
305
- FPembed supports six compression methods, selectable via the `method` parameter on `EmbeddedFingerprintGenerator`. The default is `geometric`.
306
-
307
- ### Method Reference
308
-
309
- | Method (`method` value) | Category | `method_params` | Dynamic Range / Distance Preservation | Complexity |
310
- |-------------------------|----------|-----------------|---------------------------------------|------------|
311
- | `geometric` | block-wise | `interleave` (bool) | (2**C − 1):1 dynamic range (e.g. 65,535:1 at C=16) | O(L) |
312
- | `linear` | block-wise | `interleave` (bool) | C:1 dynamic range | O(L) |
313
- | `log` | block-wise | `interleave` (bool) | ~log2(C+1):1 (~4.1:1 at C=16) | O(L) |
314
- | `uniform` | block-wise | `interleave` (bool) | 1:1 (mean pooling) | O(L) |
315
- | `hadamard` | global | `seed` (int) | seeded, truncated Hadamard projection (expected squared norm D/L) | O(L log L) |
316
- | `random_projection` | global | `seed` (int) + `sparse` (bool) | JL-style distance preservation for the projection itself | O(L D) |
317
-
318
- ### The `method` Parameter
319
-
320
- Pass `method` to the Generator constructor to select a compression strategy.
321
-
322
- ### Method-Specific Parameters (`method_params`)
323
-
324
- - **Block-wise methods** (`geometric`, `linear`, `log`, `uniform`): accept `interleave` (bool, default `False`). When `True`, bits are assigned to blocks by stride (`bit[i] -> block[i % n_blocks]`) instead of contiguous partitioning, breaking hash clustering artifacts.
325
- - **`hadamard`**: accepts `seed` (int, default `42`). Controls the random sign flips applied before the Fast Walsh-Hadamard Transform.
326
- - **`random_projection`**: accepts `seed` (int, default `42`) and `sparse` (bool, default `False`). `sparse=True` draws the Achlioptas distribution, whose entries are about 2/3 zero, but the matrix is stored densely, so it saves no memory. On the binary-input fast path the work is set by the number of set bits in the input row, not by the zeros in the matrix, so `sparse` does not automatically make that path faster; any arithmetic saving applies to the dense-product fallback used for non-binary input. It also changes the projection values, so compare the two settings on your task.
327
-
328
- Seed-based methods (`hadamard`, `random_projection`) are fully deterministic given the same seed and NumPy version. The default seed is `42`.
329
-
330
- ### Code Examples
331
-
332
- ```python
333
- from fpembed import EmbeddedFingerprintGenerator
334
-
335
- # Geometric (default)
336
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2})
337
-
338
- # Linear weights
339
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="linear")
340
-
341
- # Logarithmic weights
342
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="log")
343
-
344
- # Uniform weights (mean pooling)
345
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="uniform")
346
-
347
- # Hadamard (SRHT)
348
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="hadamard")
349
-
350
- # Random projection
351
- gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="random_projection")
352
- ```
353
-
354
- Bit-interleaving with a block-wise method:
355
-
356
- ```python
357
- gen = EmbeddedFingerprintGenerator(
358
- fp_type="ecfp", fp_size=2048, compression=16,
359
- fp_params={"radius": 2}, method="linear",
360
- method_params={"interleave": True}
361
- )
362
- ```
363
-
364
- Standalone `compress_fingerprint` with a non-default method:
365
-
366
- ```python
367
- from fpembed import compress_fingerprint
368
- import numpy as np
369
-
370
- fp = np.random.randint(0, 2, size=2048).astype(np.float64)
371
- emb = compress_fingerprint(fp, size=16, method="hadamard", method_params={"seed": 42})
372
- print(emb.shape) # (1, 128)
373
- ```
374
-
375
- ### Choosing a Method
376
-
377
- Block-wise methods (`geometric`, `linear`, `log`, `uniform`) are fast (O(L)) and simple, so reach for them when speed or modest compression ratios matter most. `geometric` keeps the widest code range per block, while `uniform` treats every bit equally (mean pooling). Global projection methods (`hadamard`, `random_projection`) mix information across all input bits, which tends to retain more information at high compression ratios. `hadamard` is efficient (O(L log L)) and requires power-of-2 fingerprint sizes; `random_projection` carries the strongest theoretical distance-preservation statement (Johnson-Lindenstrauss) at the cost of O(L D) work.
378
-
379
- Method rankings depend on the downstream model, so treat any single recommendation as a starting point rather than an optimum. The table below records the direction found in the project's own experiments (see `dev_docs/investigation_foundation_model_embedding_compression.md`); those measurements were taken on specific datasets and models and do not transfer as a guarantee. The recorded configuration ECFP with `fp_size=16384` and `C=16` is a starting candidate, not a proven optimum, and it does not change the default method.
380
-
381
- | Application | Starting point | What to check before relying on it |
382
- |---|---|---|
383
- | Tree-based molecular property prediction | `geometric`, `C=16` | Compare against raw fingerprints and other block-wise methods; tree feature subsampling changes the realized speedup |
384
- | Linear prediction | `random_projection` | Re-tune regularization after changing the representation |
385
- | Approximate similarity, clustering, or retrieval | `random_projection` | Measure neighborhood preservation and `recall@k` under the intended metric; cosine correlation with a reference embedding is not enough |
386
- | Exact binary similarity (Tanimoto) | Bit-packed raw fingerprints | Compressed-space Euclidean distance is not binary Tanimoto |
387
- | Dense continuous embeddings | Projection methods | The geometric binary-code argument no longer applies |
388
-
389
- Retrieval and similarity use cases need task-specific retrieval evaluation. A high correlation with a reference embedding does not establish that nearest neighbors are preserved.
390
-
391
- ### Output Dtype
392
-
393
- Both `EmbeddedFingerprintGenerator` (constructor argument `dtype`, applied to every returned array) and the standalone `compress_fingerprint()` (keyword-only `dtype`) accept an output dtype. Accepted values are given either as strings or NumPy types:
394
-
395
- | `dtype` | Bytes/value | Semantics |
396
- |---|---|---|
397
- | `float64` (default) | 8 | The values existing callers already get, computed exactly as before. For `geometric`, the value equals the integer code divided by `2**C - 1`, apart from the float64 round-off of the accumulation path. |
398
- | `float32` | 4 | **Information-preserving** for `geometric` at `C <= 16` (see below). ~1e-7 relative error for the projection methods. Permitted for every method. |
399
- | `uint16` | 2 | The **exact integer code**. Permitted only for `geometric` and `uniform`, and only for binary input. |
400
-
401
- `dtype` reduces the **returned and stored** size of the embedding. It does **not** reduce peak transient memory during compression: the float paths still compute internally in `float64` and cast on return, so the working set is unchanged — only the array you keep is smaller.
402
-
403
- `dtype` is a representation of identical information, not a different fingerprint, so it does not enter `fp_params_hash()` and does not affect cache keys.
404
-
405
- #### `float32` — information-preserving, not bit-equal
406
-
407
- For `geometric` at `C <= 16`, the `float32` output is *information-preserving*: the underlying block integer `n` is recovered **exactly** by
408
-
409
- ```
410
- n = rint(value * (2**C - 1))
411
- ```
412
-
413
- The stored `float32` value is **not** bit-equal to the `float64` output — it differs by about 0.002 of one code step at `C = 16`. That difference is far below the 0.5 needed to misround, so recovery of `n` is reliable, but you must not compare `float32` and `float64` outputs for equality. This is why the dtype is described as "information-preserving" rather than "lossless".
414
-
415
- #### `uint16` — the exact integer code
416
-
417
- `uint16` carries the raw integer code directly, not a rescaled `[0, 1]` value, and is computed exactly (it does not inherit the tiny residual the float64 einsum carries):
418
-
419
- | Method | Stored value | Range | Recover the float64 embedding by |
420
- |---|---|---|---|
421
- | `geometric` | block integer `n = Σ bᵢ·2ⁱ` (first bit least significant) | `0 .. 2**C - 1` | `value / (2**C - 1)` |
422
- | `uniform` | block popcount | `0 .. C` | `value / C` |
423
-
424
- `interleave=True` is supported identically — interleaving changes which bits form a block, not the recovery formula.
425
-
426
- #### Reversibility is not distance preservation
427
-
428
- At supported precision, `geometric` recovers the original bits exactly, but that does not mean the decoded values preserve the geometry of those bits, or that a model can reach them equally easily. A two-bit counterexample, with `C=2` so `2**C - 1 = 3`:
429
-
430
- - Codes `1` and `2` differ in both bits, yet their decoded values differ by only `1/3`.
431
- - Codes `0` and `2` differ in one bit, yet their decoded values differ by `2/3`.
432
-
433
- Ranking pairs by Euclidean distance in the decoded space therefore does not reproduce ranking by Hamming distance. Higher-order bits also map to larger positional values than lower-order bits, which makes them easier for a threshold-based model to reach; injectivity alone does not make all bits equally learnable. Exact binary similarity (Tanimoto on the original bits) stays a separate operation from comparing compressed vectors.
434
-
435
- Pooling methods (`linear`, `log`, `uniform`) additionally collide distinct blocks, so their suitability has to be measured on the task rather than inferred from the weight range.
436
-
437
- Keep the sources of loss separate: fingerprint hashing collisions come from the fingerprint itself, block pooling from the compression step, float rounding from the stored dtype, and any accuracy change from the downstream model. Only pooling and rounding are compression concerns.
438
-
439
- **Which combinations raise `ValueError`:**
440
-
441
- - `uint16` with `linear`, `log`, `hadamard`, or `random_projection` — these have no exact integer form.
442
- - `uint16` + `geometric` with `C > 16` — the block integer would exceed `uint16` range. (`uint16` + `uniform` requires `C <= 65535`, which is unreachable in practice but still enforced.)
443
- - `uint16` with **non-binary** input — the integer identity requires every element to be 0 or 1; count fingerprints or arbitrary float input would be silently corrupted, so this is rejected rather than accepted.
444
- - Any dtype outside `{float64, float32, uint16}` (including lossy types such as `int8`, `uint8`, `float16`) — rejected with a message naming the accepted set.
445
-
446
- ```python
447
- import numpy as np
448
- from fpembed import compress_fingerprint
449
-
450
- fp = np.random.randint(0, 2, size=2048).astype(np.uint8)
451
-
452
- # Exact integer code, 4x smaller than float64
453
- codes = compress_fingerprint(fp, size=16, dtype="uint16")
454
-
455
- # Recover the geometric float64 value (the code / (2**16 - 1), up to float64 round-off)
456
- recovered = codes / (2**16 - 1)
457
- ```
458
-
459
- ### Representation Manifest
460
-
461
- `fp_params_hash()` is a cache key: it covers the fingerprint type, its parameters, and the compression method. It is deliberately narrow and must not change, so it is not a complete identity for an array you persist. `EmbeddedFingerprintGenerator.representation_manifest()` returns a fresh JSON-compatible mapping for that purpose:
462
-
463
- ```python
464
- import json
465
-
466
- manifest = gen.representation_manifest()
467
- print(json.dumps(manifest, indent=2))
468
- ```
469
-
470
- It records the resolved fingerprint type, width, and parameters (omitted and explicitly-default values resolve to the same settings), whether compression is active together with its factor, method and effective method parameters including `seed` and `interleave`, the output dimension, dtype, and physical encoding, the built-in canonicalization policy, an optional declarative `preprocessing` mapping you supply, and the installed `fpembed`, `numpy`, `rdkit`, `scikit-fingerprints`, and `selfies` versions. `schema_version` describes the manifest format and moves independently of the package version.
471
-
472
- The `output.encoding` field distinguishes what the values physically are: `raw_binary` for an uncompressed fingerprint, `normalized_block_value` and `mean_block_value` for geometric and uniform float output, `integer_code` and `integer_count` for the exact `uint16` codes, `weighted_block_value` for linear and log, and `projection_value` for Hadamard and random projection. A `uint16` geometric array and its `float64` counterpart are therefore not interchangeable: the recovery formula relates them, but only the integer form is exact.
473
-
474
- Built-in canonicalization and external preprocessing are reported separately. The manifest stores a caller-supplied `preprocessing` mapping verbatim under `preprocessing.external` and never executes it; supplying one does not make the pipeline standardized. The manifest is additive: it does not change `fp_params_hash`, cache keys, or returned arrays. Implementations may change the layout between versions, so re-derive a manifest rather than comparing its bytes across `fpembed` releases.
475
-
476
- ### Compression Limits
477
-
478
- `geometric` compression is capped at `C <= 32`; requesting `compression > 32` with `geometric` (at either entry point) raises `ValueError`. Geometric block weights span `2**-C`, so the block integer survives only while it fits the float64 mantissa. The einsum accumulates roughly `C · 2**-52` relative error against a code spacing of `2**-C`, so recovery starts failing well before the naive 53-bit limit — at `C = 64` the low ~11 bits of every block are silently absorbed. `C <= 32` is the largest power of two comfortably inside the safe range (verified exact for `C <= 16`). A `ValueError`, not a warning, is used, because warnings can be suppressed in evaluation runs — precisely where silent bit loss does the most damage.
479
-
480
- The cap applies only to `geometric` and only when compression is active (`compression=None` or `0` is exempt). The other methods (`linear`, `log`, `uniform`, `hadamard`, `random_projection`) have bounded weight ratios or use no block weights, so none approaches the mantissa limit even at large `C` and none is capped.
481
-
482
- ### Performance Characteristics
483
-
484
- All methods produce the same output dimensionality (D = L / compression) but differ in speed and memory:
485
-
486
- | Method | Speed | Precomputed Memory | Best For |
487
- |--------|------------------------------------------------------------------------------------|--------------------------------------|----------|
488
- | Block-wise (all four) | Fastest - single vectorized einsum, O(L) | Negligible (C-length weight vector) | Default choice; large batches |
489
- | Hadamard (seeded, truncated) | Fast: pruned fold-then-FWHT, where the input is folded down to length D *before* transforming, so the discarded outputs are never computed. The retained first-D outputs are bit-identical to truncating the full transform for binary input. Correctness against that reference does not by itself prove neighborhood or distance preservation. | L-length sign vector (~16 KB) | High compression ratios |
490
- | Random projection | Fast - for binary input, gathers and sums only the matrix rows selected by the set bits (exploiting fingerprint sparsity) instead of a full dense matmul; non-binary input falls back to the dense product. The gather result is mathematically equivalent to the dense product **within float64 round-off** (the two sum the same terms in a different order), not bit-identical. One canonical C-contiguous `(L, D)` matrix is cached per parameter set. | (L, D) matrix (~2 MB for L=2048, D=128) | High compression ratios; JL-style guarantee for the projection itself |
491
-
492
- Both projection methods gained fast paths — the Hadamard fold is bit-identical to the full transform for binary input, and the random-projection gather is equivalent to the dense product within float64 round-off. Measured wall-time improvements (L=16384, D=1024, on one machine):
493
-
494
- - **Hadamard** (pruned fold-then-FWHT vs full-length transform then truncate): roughly **3x** at N=1 and **6x** at N=16.
495
- - **Random projection** (binary row-gather vs dense matmul, at ECFP-like sparsity): roughly **28x** at N=1 and **2.5x** at N=16.
496
-
497
- These are measured wall times on one machine, not a theoretical claim: the pruned FWHT reduces butterfly *operation count* by about 9x, but the code is already vectorized and largely bandwidth-bound, so the wall-time gain is materially less than the operation-count reduction. Block-wise methods remain the fastest overall choice. Random projection's matrix memory still grows with `L × D`. Note that `sparse=True` (the Achlioptas variant) stores its matrix **densely** and therefore saves no memory despite the name; it trades roughly 2/3 zero entries for a denser arithmetic pattern in the dense-product path, not a smaller footprint, and the binary-input gather selects matrix rows by the input's set bits rather than by the matrix nonzeros.
498
-
499
- **Hadamard specifics.** The implementation applies seeded random sign flips and keeps the first `D` outputs in Sylvester order. It does not randomly choose which `D` of the `L` coordinates to keep, so it is a seeded, truncated Hadamard projection rather than conventional randomly subsampled SRHT. With the retained `1/sqrt(L)` normalization, averaging over the random signs gives an expected squared norm of `D / L` times the input squared norm; a distance-scale correction would multiply the outputs by `sqrt(L / D)`, but that does not remove the fixed-coordinate selection. Read the fold's bit-identity with the full transform as correctness of this transform, not as evidence of neighborhood preservation.
500
-
501
- **Projection matrix cache.** The random-projection matrix is cached to avoid redrawing it on every call. Only a **single canonical orientation** is retained per parameter set — one C-contiguous `(L, D)` matrix; the `(D, L)` orientation is served as a non-owning `.T` view of the same buffer, so no second owning copy exists. The cache is bounded by **total retained bytes**, not entry count: the budget defaults to 512 MiB (`RP_CACHE_BYTE_BUDGET` in `compression_projection_cache.py`), with a small entry-count floor so a single oversized matrix cannot evict itself. Least-recently-used matrices are dropped once the accumulated bytes exceed the budget.
502
-
503
- ## Project Structure
504
-
505
- ```
506
- fpembed/
507
- ├── src/fpembed/ # importable package (installed editable)
508
- │ ├── __init__.py # public API and __version__
509
- │ ├── generator.py # EmbeddedFingerprintGenerator
510
- │ ├── generator_smiles.py # SMILES/SELFIES/Mol input mixin
511
- │ ├── compression.py # compress_fingerprint (orchestrator)
512
- │ ├── compression_blockwise.py # block-wise weight schemes
513
- │ ├── compression_projection.py # seeded truncated Hadamard + random projection
514
- │ ├── compression_projection_cache.py # byte-bounded projection-matrix cache
515
- │ ├── dtype_support.py # output dtype validation and casting
516
- │ ├── embedding_cache.py # BoundedCache for embedding lookups
517
- │ ├── hashing.py # fp_params_hash
518
- │ ├── input_validation.py # real/finite input boundary guard
519
- │ ├── manifest.py # representation manifest builder
520
- │ ├── skfp_factory.py # scikit-fingerprints backend factory
521
- │ ├── smiles_utils.py # parse_smiles, canonicalize_smiles
522
- │ └── py.typed # PEP 561 marker
523
- ├── examples/
524
- │ ├── quickstart.ipynb # usage notebook
525
- │ ├── datasets/ # molecular datasets (RedDB, NFA, QM9)
526
- │ └── nicegui_app/ # NiceGUI demo application
527
- ├── pyproject.toml # package metadata and dependency ranges
528
- ├── requirements.txt # third-party app lock (no FPembed self-pin)
529
- ├── environment.yml # authoritative conda env (installs -e .)
530
- └── README.md
531
- ```
532
-
533
- ## Running the Demo App
534
-
535
- The NiceGUI demo app provides an interactive UI for optimizing fingerprint embeddings. The examples live in the repository working tree, not in the installed package - clone the repository to access them.
536
-
537
- ```bash
538
- git clone https://github.com/Sciencealone/fpembed.git
539
- cd fpembed
540
-
541
- # Install FPembed (editable) plus the app dependencies from the working tree
542
- pip install -e ".[app]"
543
-
544
- # Alternatively, recreate the full pinned environment (installs -e . itself)
545
- conda env create -f environment.yml
546
- conda activate fpembed
547
-
548
- # Run the NiceGUI app
549
- cd examples/nicegui_app
550
- python app.py
551
- ```
552
-
553
- The `-e ".[app]"` install and `environment.yml` both provide `fpembed` from the working tree; neither pulls a `fpembed` wheel from PyPI. `requirements.txt` pins only the third-party app dependencies and can be installed on top of the editable package (`pip install -r requirements.txt`) if you want the exact tested versions.
554
-
555
- ### Demo App Cache Lifecycle
556
-
557
- The demo app writes a cache database at `examples/nicegui_app/cache.db` to avoid recomputing fingerprints and embeddings across runs.
558
-
559
- - **Purpose & growth.** The cache accumulates computed results as you evaluate molecules. It can grow large during extended optimization runs — provide at least **100 GB of free disk space** before a large evaluation.
560
- - **Safe deletion.** `cache.db` is a pure regenerable artifact. You may delete it at any time while the app is **not running**; the app recreates an empty cache on the next launch. Stop the app before deleting the file to avoid removing a database with open handles.
561
- - **Data-loss consequences.** Deleting the cache loses only saved computation, never source data or results you have exported — every entry can be recomputed from the input molecules. The trade-off is a slower first run afterward while the cache refills. Deleting it while the app is running is unsafe and can leave a partially written file; if that happens, stop the app and delete the leftover file before restarting.
562
-
563
- A Jupyter notebook with quick-start examples is also available at `examples/quickstart.ipynb`.
564
-
565
- ## Datasets
566
-
567
- The following datasets are included in `examples/datasets/` (obtained from their original sources):
568
-
569
- | Dataset | DOI |
570
- |---|---|
571
- | RedDB Database | https://doi.org/10.1038/s41597-022-01832-2 |
572
- | Non-Fullerene Acceptors Database | https://doi.org/10.1016/j.joule.2017.10.006 |
573
- | QM9 Database | https://doi.org/10.1038/sdata.2014.22 |
574
-
575
- ## License
576
-
577
- This project is licensed under the terms of the MIT open source license. Please refer to the [LICENSE](LICENSE) for the full terms.
578
-
579
- ## Legal & Trademark Notice
580
-
581
- The name **"FPembed"**, its branding, and identifiers are the intellectual property of the project author (@Sciencealone).
582
- * **Commercial Branding:** The unauthorized use of the name "FPembed" to brand, market, or promote commercial software, corporate AI engines, or proprietary services within the fields of bioinformatics, chemoinformatics, and AI drug discovery is strictly prohibited.
583
- * **Community Protection:** This notice is established to prevent public confusion and to protect the open-source community from misleading corporate misrepresentations. For licensing queries regarding the project name, please open an issue or contact the author directly.
584
-
585
- ## Commercial & Research Services
586
-
587
- For the project onboarding, corporate deployments, high-performance compute scaling (HPC / Databricks / Ray), or custom model fine-tuning for pharma R&D and research grants (Horizon Europe / Art. 60 LOSU / AEI) as well as other related tasks, official technical support and contract services are available.
588
-
589
- **[View Full Service Offerings & Contact Details](COMMERCIAL_SUPPORT.md)**
590
-
591
- ## AI disclosure
592
-
593
- AI usage during project development is declared in [aidecl.yaml](aidecl.yaml) following the [AI Declaration Format](https://ai-declaration.org/).
594
-
595
- ## Support
596
- This project is provided as-is, and may be updated over time. If you have questions, please open an issue.
1
+ Metadata-Version: 2.4
2
+ Name: fpembed
3
+ Version: 0.2.3
4
+ Summary: FPembed - Generalized fingerprint embedding library
5
+ License-Expression: MIT
6
+ Project-URL: Homepage, https://github.com/Sciencealone/FPembed
7
+ Project-URL: Commercial & Research Support, https://github.com/Sciencealone/FPembed/blob/main/COMMERCIAL_SUPPORT.md
8
+ Project-URL: Issue Tracker, https://github.com/Sciencealone/FPembed/issues
9
+ Requires-Python: <3.14,>=3.10
10
+ Description-Content-Type: text/markdown
11
+ License-File: LICENSE
12
+ Requires-Dist: numpy<3,>=1.26.4
13
+ Requires-Dist: rdkit>=2023.9.6
14
+ Requires-Dist: scikit-fingerprints<3,>=2.1
15
+ Requires-Dist: selfies>=2.2
16
+ Dynamic: license-file
17
+
18
+ # FPembed - Generalized Molecular Fingerprint Embeddings
19
+
20
+ [![PyPI version](https://badge.fury.io/py/fpembed.svg)](https://badge.fury.io/py/fpembed)
21
+ [![DOI](https://img.shields.io/badge/DOI-10.5281/zenodo.21447996-blue)](https://doi.org/10.5281/zenodo.21447996)
22
+
23
+
24
+ ## Contents
25
+
26
+ - [Introduction](#introduction)
27
+ - [Installation](#installation)
28
+ - [Conda Environment](#conda-environment)
29
+ - [Dependency Files](#dependency-files)
30
+ - [Quick Start](#quick-start)
31
+ - [Why Use Embedded Fingerprints?](#why-use-embedded-fingerprints)
32
+ - [Storage Size](#storage-size)
33
+ - [ML Training and Inference Speed](#ml-training-and-inference-speed)
34
+ - [Memory During ML Training](#memory-during-ml-training)
35
+ - [Compression Overhead](#compression-overhead)
36
+ - [Sample Efficiency](#sample-efficiency)
37
+ - [Summary of Advantages](#summary-of-advantages)
38
+ - [Supported Fingerprint Types](#supported-fingerprint-types)
39
+ - [Compression Methods](#compression-methods)
40
+ - [Method Reference](#method-reference)
41
+ - [The `method` Parameter](#the-method-parameter)
42
+ - [Method-Specific Parameters (`method_params`)](#method-specific-parameters-method_params)
43
+ - [Code Examples](#code-examples)
44
+ - [Choosing a Method](#choosing-a-method)
45
+ - [Output Dtype](#output-dtype)
46
+ - [Representation Manifest](#representation-manifest)
47
+ - [Compression Limits](#compression-limits)
48
+ - [Performance Characteristics](#performance-characteristics)
49
+ - [Project Structure](#project-structure)
50
+ - [Examples](#examples)
51
+ - [License](#license)
52
+ - [Legal & Trademark Notice](#legal--trademark-notice)
53
+ - [Commercial & Research Services](#commercial--research-services)
54
+ - [AI disclosure](#ai-disclosure)
55
+ - [Support](#support)
56
+
57
+ ## Introduction
58
+
59
+ A Python package for generating compressed molecular fingerprint embeddings, backed by [scikit-fingerprints](https://github.com/scikit-fingerprints/scikit-fingerprints). Supports ten binary fingerprint types through a single unified class.
60
+
61
+ FPembed compresses standard molecular fingerprints into compact vectors suitable for machine-learning models, using a choice of pluggable compression methods — block-wise weighting schemes (including the default weighted binary masking) or global projections (see [Compression Methods](#compression-methods)). The package accepts SMILES, SELFIES, and RDKit Mol objects as input.
62
+
63
+ At a glance: a feature vector reduced from L to D = L / C dimensions (e.g. 16x fewer features at C=16), a drop-in fit for any scikit-learn model, and — in the evaluated workloads — accuracy comparable to raw fingerprints. The dimensionality reduction is exact and guaranteed; the downstream runtime, memory, and accuracy effects depend on your model, hardware, and dataset (see [Why Use Embedded Fingerprints?](#why-use-embedded-fingerprints) for what is guaranteed versus workload-dependent). If you just want to try it, jump to [Quick Start](#quick-start).
64
+
65
+ The concept of compressing molecular fingerprints via weighted binary masking was originally introduced for Morgan fingerprints in the eMFP paper:
66
+
67
+ > Nuñez-Andrade, E. A., Vidal-Daza, I., Gomez-Bombarelli, R., Ryan, J. W., & Martin-Martinez, F. J. (2025).
68
+ > Embedded Morgan Fingerprints for more efficient molecular property predictions with machine learning.
69
+ > *ChemRxiv* (preprint). https://doi.org/10.26434/chemrxiv-2025-6hfp8
70
+
71
+ ```bibtex
72
+ @article{nunez2025emfp,
73
+ author = {Nu{\~n}ez-Andrade, Emilio A. and Vidal-Daza, Isaac and Gomez-Bombarelli, Rafael and Ryan, James W. and Martin-Martinez, Francisco J.},
74
+ title = {Embedded {Morgan} Fingerprints for more efficient molecular property predictions with machine learning},
75
+ journal = {ChemRxiv},
76
+ year = {2025},
77
+ doi = {10.26434/chemrxiv-2025-6hfp8},
78
+ note = {Preprint}
79
+ }
80
+ ```
81
+
82
+ Original concept repository: [MMLabCodes/eMFP](https://github.com/MMLabCodes/eMFP)
83
+
84
+ ## Installation
85
+
86
+ The package is installed from the working tree with an editable install — clone the repository, then install `fpembed` and its core dependencies (rdkit, numpy, selfies, scikit-fingerprints) in one step:
87
+
88
+ ```bash
89
+ git clone https://github.com/Sciencealone/fpembed.git
90
+ cd fpembed
91
+ pip install -e .
92
+ ```
93
+
94
+ The installed version is reported by `fpembed.__version__` and by the package metadata (`pip show fpembed`).
95
+
96
+ ### Conda Environment
97
+
98
+ A full conda environment is provided for reproducibility. It is authoritative for the development setup and installs `fpembed` editable (`-e .`) from the working tree:
99
+
100
+ ```bash
101
+ conda env create -f environment.yml
102
+ conda activate fpembed
103
+ ```
104
+
105
+ This installs all dependencies and the `fpembed` package in editable mode.
106
+
107
+ ### Dependency Files
108
+
109
+ Two files describe dependencies, each with a distinct role - do not confuse the tested lock with the metadata ranges:
110
+
111
+ - **`pyproject.toml`** - package metadata. Declares the compatible version ranges of `fpembed`'s dependencies that a consumer resolves against.
112
+ - **`environment.yml`** - the authoritative exact-pin lock. It recreates the full development and test environment and installs `fpembed` editable (`-e .`).
113
+
114
+ ## Quick Start
115
+
116
+ ### Single Molecule (SMILES)
117
+
118
+ ```python
119
+ from fpembed import EmbeddedFingerprintGenerator
120
+
121
+ gen = EmbeddedFingerprintGenerator(
122
+ fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}
123
+ )
124
+
125
+ # Generate compressed fingerprint from SMILES
126
+ emb = gen.GetFingerprintFromSmiles("CCO")
127
+ print(emb.shape) # (128,)
128
+ ```
129
+
130
+ ### Different Fingerprint Types
131
+
132
+ ```python
133
+ # Atom Pair fingerprint
134
+ gen_ap = EmbeddedFingerprintGenerator(
135
+ fp_type="atom_pair", fp_size=2048, compression=16,
136
+ fp_params={"min_distance": 1, "max_distance": 30}
137
+ )
138
+
139
+ # Topological Torsion fingerprint
140
+ gen_tt = EmbeddedFingerprintGenerator(
141
+ fp_type="topological_torsion", fp_size=2048, compression=16,
142
+ fp_params={"torsion_atom_count": 4}
143
+ )
144
+ ```
145
+
146
+ ### Single Molecule (SELFIES)
147
+
148
+ ```python
149
+ emb = gen.GetFingerprintFromSelfies("[C][C][O]")
150
+ print(emb.shape) # (128,)
151
+ ```
152
+
153
+ ### Batch Processing
154
+
155
+ ```python
156
+ smiles_list = ["CCO", "c1ccccc1", "CC(=O)O", "invalid_smiles"]
157
+
158
+ embeddings, invalid_indices = gen.GetFingerprintsFromSmiles(smiles_list)
159
+ print(embeddings.shape) # (3, 128) - 3 valid molecules
160
+ print(invalid_indices) # [3] - index of invalid SMILES
161
+ ```
162
+
163
+ ### Bounded-Memory Iteration
164
+
165
+ The batch methods above collect every valid embedding before stacking them into one matrix. At scale that peak allocation dominates: one million molecules at D=1024 needs about 4.1 GB as float32 or 8.2 GB as float64 for the final matrix alone, before counting the individual arrays held while it is built. When the input does not fit in memory, iterate instead:
166
+
167
+ ```python
168
+ for index, emb in gen.IterFingerprintsFromSmiles(smiles_iterable):
169
+ if emb is None:
170
+ continue # normal invalid input at this index
171
+ store(index, emb) # consume or write one row at a time
172
+ ```
173
+
174
+ `IterFingerprints`, `IterFingerprintsFromSmiles`, and `IterFingerprintsFromSelfies` accept any iterable, pull one input at a time, and yield `(index, embedding_or_none)`. Indices match input order and include invalid entries, which yield `None`; for `IterFingerprints`, a `None` molecule also yields `None`. Unexpected computation errors surface when the item is consumed, matching the single-molecule methods. The iterator never materializes the input or accumulates rows, so peak memory is yours to bound.
175
+
176
+ **Memory contract.** These APIs are not streaming in the sense of a total process cap. The `random_projection` matrix cache is bounded by retained bytes, but a matrix referenced by a live generator stays resident, so the cache budget is not a process-wide limit. Embeddings you keep are yours to release, and `dtype` reduces the returned and stored size, not the peak transient memory during compression.
177
+
178
+ ### Raw Fingerprint (No Compression)
179
+
180
+ ```python
181
+ gen_raw = EmbeddedFingerprintGenerator(
182
+ fp_type="ecfp", fp_size=2048, compression=None, fp_params={"radius": 2}
183
+ )
184
+ fp = gen_raw.GetFingerprintFromSmiles("CCO")
185
+ print(fp.shape) # (2048,)
186
+ ```
187
+
188
+ ### Standalone Compression Function
189
+
190
+ ```python
191
+ import numpy as np
192
+ from fpembed import compress_fingerprint
193
+
194
+ fp = np.random.randint(0, 2, size=2048).astype(np.float64)
195
+ emb = compress_fingerprint(fp, size=16)
196
+ print(emb.shape) # (1, 128)
197
+ ```
198
+
199
+ **Input contract.** `compress_fingerprint` accepts 1-D `(L,)` and 2-D `(N, L)` arrays of real numeric dtype: boolean, signed or unsigned integer, or floating. Non-real dtypes (complex, object, string, datetime) raise `TypeError`; floating input containing NaN or either infinity raises `ValueError`. Integer output (`dtype="uint16"`) additionally requires binary input and represents integer codes or counts, not normalized floats. Neither validation nor compression mutates the array you pass in.
200
+
201
+ ### Parameter Hashing
202
+
203
+ ```python
204
+ from fpembed import fp_params_hash
205
+
206
+ h = fp_params_hash("ecfp", {"radius": 2})
207
+ print(h) # 16-char hex string, stable across sessions
208
+ ```
209
+
210
+ ### Caching for Repeated Lookups
211
+
212
+ ```python
213
+ gen = EmbeddedFingerprintGenerator(
214
+ fp_type="ecfp", fp_size=2048, compression=16,
215
+ fp_params={"radius": 2}, cache_size=1024
216
+ )
217
+
218
+ # First call computes and caches
219
+ emb = gen.GetFingerprintFromSmiles("CCO")
220
+
221
+ # Second call returns cached result
222
+ emb2 = gen.GetFingerprintFromSmiles("CCO")
223
+
224
+ print(gen.cache_info()) # CacheInfo(hits=1, misses=1, maxsize=1024, currsize=1)
225
+ gen.clear_cache()
226
+ ```
227
+
228
+ ### SMILES Canonicalization and Chemical Standardization
229
+
230
+ FPembed canonicalizes SMILES spelling before keying and before parsing, so the same molecule written two ways produces the same embedding: the pipeline works with the canonical form (atom ordering and RDKit's aromatic form). That makes results spelling-invariant for atom-order-sensitive backends, and it is why the cache key can be built from the canonical string.
231
+
232
+ Canonicalization is not chemical standardization. FPembed does not remove salts, normalize tautomers, pick a protonation state, or impose a stereochemistry policy, because those decisions change scientific meaning and are the caller's to make. Two inputs that differ by a counter-ion or a tautomer stay distinct. If your workflow needs standardization, apply it before calling FPembed and describe it with `representation_manifest(preprocessing=...)` (see [Representation Manifest](#representation-manifest)).
233
+
234
+ ## Why Use Embedded Fingerprints?
235
+
236
+ Predictive accuracy is one axis of comparison between raw and embedded fingerprints - and the gap can be narrow, especially on large datasets where raw fingerprints have enough data to exploit all 2048 bits directly. However, accuracy is not the only metric that matters. The one guaranteed advantage is dimensionality: an embedded vector has D = L / C features instead of L, and the stored byte count follows directly from D and the chosen dtype (calculated exactly in [Storage Size](#storage-size)). The operational benefits that follow — training and inference speed, memory during training, sample efficiency — are consequences of fewer features, but their magnitude is workload-, model-, and hardware-dependent, not a fixed guarantee.
237
+
238
+ The core argument is not "embedded fingerprints are always more accurate" but rather "embedded fingerprints can achieve comparable accuracy at a fraction of the feature count, and therefore of the downstream computational cost."
239
+
240
+ ### Storage Size
241
+
242
+ The output dimensionality (D = L / C) is deterministic and independent of dataset, model, or method; the byte counts below follow from D and the stated dtype. The size figures are per representation under the named dtype assumptions (float64 = 8 bytes/value, uint16 = 2 bytes/value):
243
+
244
+ | Representation | Per-molecule (L=2048) | Per-molecule (L=4096) | 100K molecules (L=2048) |
245
+ |---|---|---|---|
246
+ | Raw binary FP (float64, one byte-inflated bit per element) | 16 KB | 32 KB | ~1.6 GB |
247
+ | Raw binary FP (bit-packed, 1 bit per element) | 256 B | 512 B | ~25 MB |
248
+ | Embedded, C=16 (float64) | 1 KB | 2 KB | ~100 MB |
249
+ | Embedded, C=16 (uint16) | 256 B | 512 B | ~25 MB |
250
+ | Embedded, C=32 (float64, `uniform`/projection only) | 512 B | 1 KB | ~50 MB |
251
+
252
+ A word on the baseline. Comparing an embedding against a raw fingerprint *stored as float64* overstates the storage gain, because a raw fingerprint is binary and would sensibly be bit-packed. Against **bit-packed** raw fingerprints, the `geometric` embedding at C=16 in float64 is actually *larger* (1 KB vs 256 B per molecule at L=2048): geometric at C=16 is a reversible re-encoding — it packs each 16-bit block into a single wide value, carrying the same information in fewer but larger numbers. Storing that embedding as `uint16` (see [Output Dtype](#output-dtype)) brings it back to parity with bit-packed input.
253
+
254
+ So raw byte savings are **not** the reason to embed. The genuine, unconditional advantage is the **downstream feature count**: the embedded vector has D = L / C features instead of L. Downstream training, inference, and the in-memory feature matrix then shrink roughly in proportion to the feature count, and that reduction is what the rest of this section quantifies. Keep the scaling directional rather than universal: similarity on bit-packed fingerprints works on packed words, tree learners subsample features and pay fixed costs, and approximate-neighbor indexes change the comparison, so none of these costs follows the feature count exactly.
255
+
256
+ ### ML Training and Inference Speed
257
+
258
+ The downstream ML model operates on the feature vector. Fewer features means faster training and prediction:
259
+
260
+ - **Tree-based models (Random Forest, XGBoost)**: Feature splitting cost is proportional to the number of features. Going from 2048 to 128 features means each tree split considers ~16x fewer candidates. For hyperparameter searches (e.g., Optuna with hundreds of trials), this compounds into significant wall-clock savings.
261
+ - **Neural networks**: The first dense layer's weight matrix shrinks from `(2048 x hidden)` to `(128 x hidden)` - 16x fewer parameters and 16x fewer multiply-adds per forward pass.
262
+ - **Distance-based methods (k-NN, similarity search)**: Pairwise distance computation is O(N² x D). Reducing D from 2048 to 128 shrinks the distance computation itself by 16x; index build and query overhead, memory traffic, and candidate re-ranking sit on top of that and reduce the realized speedup.
263
+
264
+ ### Memory During ML Training
265
+
266
+ During model training, the feature matrix for N=100K molecules occupies `(100000, 2048)` float64 = ~1.6 GB for raw fingerprints, versus `(100000, 128)` = ~100 MB for embedded. Tree-based models create internal copies and histograms proportional to feature count. GPU-based models benefit from smaller input tensors that allow larger batch sizes and better hardware utilization.
267
+
268
+ ### Compression Overhead
269
+
270
+ The compression step itself is small for block-wise methods, but its cost is not universal. Under warm-cache conditions, and on the machine where the project measured it, block-wise compression added on the order of 1 ms per 1000 molecules. Treat that as an order-of-magnitude indication with `L`, dtype, and hardware attached, not as a constant. Cold-cache cost is higher: the projection methods build or fetch a projection artifact on first use, and `random_projection` additionally pays to draw and store its `(L, D)` matrix.
271
+
272
+ The total pipeline cost is:
273
+
274
+ - **Raw**: `skfp generation time`
275
+ - **Embedded**: `skfp generation time + compression time`
276
+
277
+ Measure both phases on the target hardware before relying on the overhead being negligible. The downstream ML saving from 128 versus 2048 features is usually larger, but that too is workload-dependent.
278
+
279
+ ### Sample Efficiency
280
+
281
+ High-dimensional spaces (2048 binary features) suffer from the curse of dimensionality - distances become less meaningful and models need more data to fill the space. Compressing to 128 dense features can act as a form of regularization. In practice, embedded fingerprints have reached good predictive performance with fewer training samples than raw fingerprints on the datasets we examined; this is a workload- and model-dependent effect, not a guarantee, and it is most likely to matter when labeled molecular data is scarce or expensive to obtain.
282
+
283
+ ### Summary of Advantages
284
+
285
+ | Metric | Raw FP (L=2048) | Embedded FP (D=128) | Advantage |
286
+ |---|-------------------------|---------------------------------|-----------------------|
287
+ | Feature matrix memory (100K mols) | ~1.6 GB | ~100 MB | 16x smaller |
288
+ | Per-molecule storage vs float64 raw | 16 KB | 1 KB (256 B as uint16) | 16x smaller (64x) |
289
+ | Per-molecule storage vs bit-packed raw | 256 B | 256 B as uint16 (1 KB float64) | Parity, not a win (see [Storage Size](#storage-size)) |
290
+ | Tree model training speed | Baseline | ~16x fewer split candidates | Workload-dependent |
291
+ | Neural net first-layer params | 2048 x H | 128 x H | 16x fewer |
292
+ | Pairwise distance computation | O(N² x 2048) | O(N² x 128) | ~16x fewer ops |
293
+ | Small-dataset accuracy | Baseline | Can be superior (regularization) | Model-dependent |
294
+ | Large-dataset accuracy | Slightly higher ceiling | Comparable | Marginal tradeoff |
295
+
296
+ The guaranteed columns are the feature count and its exact consequences (parameter counts, per-molecule byte counts under a named dtype). The speed and accuracy columns are directional: they follow from fewer features but their realized magnitude depends on the model, hardware, and dataset. The choice between raw and embedded fingerprints is a classic accuracy-vs-efficiency tradeoff — embedded fingerprints trade a small amount of information for a smaller, denser feature vector, which is a practical default for many molecular ML workflows.
297
+
298
+ ## Supported Fingerprint Types
299
+
300
+ | Type | `fp_type` | Type-specific params |
301
+ |------|-----------|---------------------|
302
+ | Extended Connectivity (ECFP) | `ecfp` | `radius` (default 2) |
303
+ | Atom Pair | `atom_pair` | `min_distance` (1), `max_distance` (30) |
304
+ | Topological Torsion | `topological_torsion` | `torsion_atom_count` (4) |
305
+ | RDKit | `rdkit` | `min_path` (1), `max_path` (7) |
306
+ | Layered | `layered` | `min_path` (1), `max_path` (7) |
307
+ | Pattern | `pattern` | (none) |
308
+ | Avalon | `avalon` | (none) |
309
+ | SECFP | `secfp` | `radius` (3), `min_radius` (1) |
310
+ | MHFP | `mhfp` | `radius` (3), `min_radius` (1) |
311
+ | MAP4 | `map` | `radius` (default 2) |
312
+
313
+ ## Compression Methods
314
+
315
+ FPembed supports six compression methods, selectable via the `method` parameter on `EmbeddedFingerprintGenerator`. The default is `geometric`.
316
+
317
+ ### Method Reference
318
+
319
+ | Method (`method` value) | Category | `method_params` | Dynamic Range / Distance Preservation | Complexity |
320
+ |-------------------------|----------|-----------------|---------------------------------------|------------|
321
+ | `geometric` | block-wise | `interleave` (bool) | (2**C − 1):1 dynamic range (e.g. 65,535:1 at C=16) | O(L) |
322
+ | `linear` | block-wise | `interleave` (bool) | C:1 dynamic range | O(L) |
323
+ | `log` | block-wise | `interleave` (bool) | ~log2(C+1):1 (~4.1:1 at C=16) | O(L) |
324
+ | `uniform` | block-wise | `interleave` (bool) | 1:1 (mean pooling) | O(L) |
325
+ | `hadamard` | global | `seed` (int) | seeded, truncated Hadamard projection (expected squared norm D/L) | O(L log L) |
326
+ | `random_projection` | global | `seed` (int) + `sparse` (bool) | JL-style distance preservation for the projection itself | O(L D) |
327
+
328
+ ### The `method` Parameter
329
+
330
+ Pass `method` to the Generator constructor to select a compression strategy.
331
+
332
+ ### Method-Specific Parameters (`method_params`)
333
+
334
+ - **Block-wise methods** (`geometric`, `linear`, `log`, `uniform`): accept `interleave` (bool, default `False`). When `True`, bits are assigned to blocks by stride (`bit[i] -> block[i % n_blocks]`) instead of contiguous partitioning, breaking hash clustering artifacts.
335
+ - **`hadamard`**: accepts `seed` (int, default `42`). Controls the random sign flips applied before the Fast Walsh-Hadamard Transform.
336
+ - **`random_projection`**: accepts `seed` (int, default `42`) and `sparse` (bool, default `False`). `sparse=True` draws the Achlioptas distribution, whose entries are about 2/3 zero, but the matrix is stored densely, so it saves no memory. On the binary-input fast path the work is set by the number of set bits in the input row, not by the zeros in the matrix, so `sparse` does not automatically make that path faster; any arithmetic saving applies to the dense-product fallback used for non-binary input. It also changes the projection values, so compare the two settings on your task.
337
+
338
+ Seed-based methods (`hadamard`, `random_projection`) are fully deterministic given the same seed and NumPy version. The default seed is `42`.
339
+
340
+ ### Code Examples
341
+
342
+ ```python
343
+ from fpembed import EmbeddedFingerprintGenerator
344
+
345
+ # Geometric (default)
346
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2})
347
+
348
+ # Linear weights
349
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="linear")
350
+
351
+ # Logarithmic weights
352
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="log")
353
+
354
+ # Uniform weights (mean pooling)
355
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="uniform")
356
+
357
+ # Hadamard (SRHT)
358
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="hadamard")
359
+
360
+ # Random projection
361
+ gen = EmbeddedFingerprintGenerator(fp_type="ecfp", fp_size=2048, compression=16, fp_params={"radius": 2}, method="random_projection")
362
+ ```
363
+
364
+ Bit-interleaving with a block-wise method:
365
+
366
+ ```python
367
+ gen = EmbeddedFingerprintGenerator(
368
+ fp_type="ecfp", fp_size=2048, compression=16,
369
+ fp_params={"radius": 2}, method="linear",
370
+ method_params={"interleave": True}
371
+ )
372
+ ```
373
+
374
+ Standalone `compress_fingerprint` with a non-default method:
375
+
376
+ ```python
377
+ from fpembed import compress_fingerprint
378
+ import numpy as np
379
+
380
+ fp = np.random.randint(0, 2, size=2048).astype(np.float64)
381
+ emb = compress_fingerprint(fp, size=16, method="hadamard", method_params={"seed": 42})
382
+ print(emb.shape) # (1, 128)
383
+ ```
384
+
385
+ ### Choosing a Method
386
+
387
+ Block-wise methods (`geometric`, `linear`, `log`, `uniform`) are fast (O(L)) and simple, so reach for them when speed or modest compression ratios matter most. `geometric` keeps the widest code range per block, while `uniform` treats every bit equally (mean pooling). Global projection methods (`hadamard`, `random_projection`) mix information across all input bits, which tends to retain more information at high compression ratios. `hadamard` is efficient (O(L log L)) and requires power-of-2 fingerprint sizes; `random_projection` carries the strongest theoretical distance-preservation statement (Johnson-Lindenstrauss) at the cost of O(L D) work.
388
+
389
+ Method rankings depend on the downstream model, so treat any single recommendation as a starting point rather than an optimum. The table below records the direction found in the project's own experiments (see `dev_docs/investigation_foundation_model_embedding_compression.md`); those measurements were taken on specific datasets and models and do not transfer as a guarantee. The recorded configuration ECFP with `fp_size=16384` and `C=16` is a starting candidate, not a proven optimum, and it does not change the default method.
390
+
391
+ | Application | Starting point | What to check before relying on it |
392
+ |---|---|---|
393
+ | Tree-based molecular property prediction | `geometric`, `C=16` | Compare against raw fingerprints and other block-wise methods; tree feature subsampling changes the realized speedup |
394
+ | Linear prediction | `random_projection` | Re-tune regularization after changing the representation |
395
+ | Approximate similarity, clustering, or retrieval | `random_projection` | Measure neighborhood preservation and `recall@k` under the intended metric; cosine correlation with a reference embedding is not enough |
396
+ | Exact binary similarity (Tanimoto) | Bit-packed raw fingerprints | Compressed-space Euclidean distance is not binary Tanimoto |
397
+ | Dense continuous embeddings | Projection methods | The geometric binary-code argument no longer applies |
398
+
399
+ Retrieval and similarity use cases need task-specific retrieval evaluation. A high correlation with a reference embedding does not establish that nearest neighbors are preserved.
400
+
401
+ ### Output Dtype
402
+
403
+ Both `EmbeddedFingerprintGenerator` (constructor argument `dtype`, applied to every returned array) and the standalone `compress_fingerprint()` (keyword-only `dtype`) accept an output dtype. Accepted values are given either as strings or NumPy types:
404
+
405
+ | `dtype` | Bytes/value | Semantics |
406
+ |---|---|---|
407
+ | `float64` (default) | 8 | The values existing callers already get, computed exactly as before. For `geometric`, the value equals the integer code divided by `2**C - 1`, apart from the float64 round-off of the accumulation path. |
408
+ | `float32` | 4 | **Information-preserving** for `geometric` at `C <= 16` (see below). ~1e-7 relative error for the projection methods. Permitted for every method. |
409
+ | `uint16` | 2 | The **exact integer code**. Permitted only for `geometric` and `uniform`, and only for binary input. |
410
+
411
+ `dtype` reduces the **returned and stored** size of the embedding. It does **not** reduce peak transient memory during compression: the float paths still compute internally in `float64` and cast on return, so the working set is unchanged — only the array you keep is smaller.
412
+
413
+ `dtype` is a representation of identical information, not a different fingerprint, so it does not enter `fp_params_hash()` and does not affect cache keys.
414
+
415
+ #### `float32` — information-preserving, not bit-equal
416
+
417
+ For `geometric` at `C <= 16`, the `float32` output is *information-preserving*: the underlying block integer `n` is recovered **exactly** by
418
+
419
+ ```
420
+ n = rint(value * (2**C - 1))
421
+ ```
422
+
423
+ The stored `float32` value is **not** bit-equal to the `float64` output — it differs by about 0.002 of one code step at `C = 16`. That difference is far below the 0.5 needed to misround, so recovery of `n` is reliable, but you must not compare `float32` and `float64` outputs for equality. This is why the dtype is described as "information-preserving" rather than "lossless".
424
+
425
+ #### `uint16` — the exact integer code
426
+
427
+ `uint16` carries the raw integer code directly, not a rescaled `[0, 1]` value, and is computed exactly (it does not inherit the tiny residual the float64 einsum carries):
428
+
429
+ | Method | Stored value | Range | Recover the float64 embedding by |
430
+ |---|---|---|---|
431
+ | `geometric` | block integer `n = Σ bᵢ·2ⁱ` (first bit least significant) | `0 .. 2**C - 1` | `value / (2**C - 1)` |
432
+ | `uniform` | block popcount | `0 .. C` | `value / C` |
433
+
434
+ `interleave=True` is supported identically — interleaving changes which bits form a block, not the recovery formula.
435
+
436
+ #### Reversibility is not distance preservation
437
+
438
+ At supported precision, `geometric` recovers the original bits exactly, but that does not mean the decoded values preserve the geometry of those bits, or that a model can reach them equally easily. A two-bit counterexample, with `C=2` so `2**C - 1 = 3`:
439
+
440
+ - Codes `1` and `2` differ in both bits, yet their decoded values differ by only `1/3`.
441
+ - Codes `0` and `2` differ in one bit, yet their decoded values differ by `2/3`.
442
+
443
+ Ranking pairs by Euclidean distance in the decoded space therefore does not reproduce ranking by Hamming distance. Higher-order bits also map to larger positional values than lower-order bits, which makes them easier for a threshold-based model to reach; injectivity alone does not make all bits equally learnable. Exact binary similarity (Tanimoto on the original bits) stays a separate operation from comparing compressed vectors.
444
+
445
+ Pooling methods (`linear`, `log`, `uniform`) additionally collide distinct blocks, so their suitability has to be measured on the task rather than inferred from the weight range.
446
+
447
+ Keep the sources of loss separate: fingerprint hashing collisions come from the fingerprint itself, block pooling from the compression step, float rounding from the stored dtype, and any accuracy change from the downstream model. Only pooling and rounding are compression concerns.
448
+
449
+ **Which combinations raise `ValueError`:**
450
+
451
+ - `uint16` with `linear`, `log`, `hadamard`, or `random_projection` — these have no exact integer form.
452
+ - `uint16` + `geometric` with `C > 16` — the block integer would exceed `uint16` range. (`uint16` + `uniform` requires `C <= 65535`, which is unreachable in practice but still enforced.)
453
+ - `uint16` with **non-binary** input — the integer identity requires every element to be 0 or 1; count fingerprints or arbitrary float input would be silently corrupted, so this is rejected rather than accepted.
454
+ - Any dtype outside `{float64, float32, uint16}` (including lossy types such as `int8`, `uint8`, `float16`) — rejected with a message naming the accepted set.
455
+
456
+ ```python
457
+ import numpy as np
458
+ from fpembed import compress_fingerprint
459
+
460
+ fp = np.random.randint(0, 2, size=2048).astype(np.uint8)
461
+
462
+ # Exact integer code, 4x smaller than float64
463
+ codes = compress_fingerprint(fp, size=16, dtype="uint16")
464
+
465
+ # Recover the geometric float64 value (the code / (2**16 - 1), up to float64 round-off)
466
+ recovered = codes / (2**16 - 1)
467
+ ```
468
+
469
+ ### Representation Manifest
470
+
471
+ `fp_params_hash()` is a cache key: it covers the fingerprint type, its parameters, and the compression method. It is deliberately narrow and must not change, so it is not a complete identity for an array you persist. `EmbeddedFingerprintGenerator.representation_manifest()` returns a fresh JSON-compatible mapping for that purpose:
472
+
473
+ ```python
474
+ import json
475
+
476
+ manifest = gen.representation_manifest()
477
+ print(json.dumps(manifest, indent=2))
478
+ ```
479
+
480
+ It records the resolved fingerprint type, width, and parameters (omitted and explicitly-default values resolve to the same settings), whether compression is active together with its factor, method and effective method parameters including `seed` and `interleave`, the output dimension, dtype, and physical encoding, the built-in canonicalization policy, an optional declarative `preprocessing` mapping you supply, and the installed `fpembed`, `numpy`, `rdkit`, `scikit-fingerprints`, and `selfies` versions. `schema_version` describes the manifest format and moves independently of the package version.
481
+
482
+ The `output.encoding` field distinguishes what the values physically are: `raw_binary` for an uncompressed fingerprint, `normalized_block_value` and `mean_block_value` for geometric and uniform float output, `integer_code` and `integer_count` for the exact `uint16` codes, `weighted_block_value` for linear and log, and `projection_value` for Hadamard and random projection. A `uint16` geometric array and its `float64` counterpart are therefore not interchangeable: the recovery formula relates them, but only the integer form is exact.
483
+
484
+ Built-in canonicalization and external preprocessing are reported separately. The manifest stores a caller-supplied `preprocessing` mapping verbatim under `preprocessing.external` and never executes it; supplying one does not make the pipeline standardized. The manifest is additive: it does not change `fp_params_hash`, cache keys, or returned arrays. Implementations may change the layout between versions, so re-derive a manifest rather than comparing its bytes across `fpembed` releases.
485
+
486
+ ### Compression Limits
487
+
488
+ `geometric` compression is capped at `C <= 32`; requesting `compression > 32` with `geometric` (at either entry point) raises `ValueError`. Geometric block weights span `2**-C`, so the block integer survives only while it fits the float64 mantissa. The einsum accumulates roughly `C · 2**-52` relative error against a code spacing of `2**-C`, so recovery starts failing well before the naive 53-bit limit — at `C = 64` the low ~11 bits of every block are silently absorbed. `C <= 32` is the largest power of two comfortably inside the safe range (verified exact for `C <= 16`). A `ValueError`, not a warning, is used, because warnings can be suppressed in evaluation runs — precisely where silent bit loss does the most damage.
489
+
490
+ The cap applies only to `geometric` and only when compression is active (`compression=None` or `0` is exempt). The other methods (`linear`, `log`, `uniform`, `hadamard`, `random_projection`) have bounded weight ratios or use no block weights, so none approaches the mantissa limit even at large `C` and none is capped.
491
+
492
+ ### Performance Characteristics
493
+
494
+ All methods produce the same output dimensionality (D = L / compression) but differ in speed and memory:
495
+
496
+ | Method | Speed | Precomputed Memory | Best For |
497
+ |--------|------------------------------------------------------------------------------------|--------------------------------------|----------|
498
+ | Block-wise (all four) | Fastest - single vectorized einsum, O(L) | Negligible (C-length weight vector) | Default choice; large batches |
499
+ | Hadamard (seeded, truncated) | Fast: pruned fold-then-FWHT, where the input is folded down to length D *before* transforming, so the discarded outputs are never computed. The retained first-D outputs are bit-identical to truncating the full transform for binary input. Correctness against that reference does not by itself prove neighborhood or distance preservation. | L-length sign vector (~16 KB) | High compression ratios |
500
+ | Random projection | Fast - for binary input, gathers and sums only the matrix rows selected by the set bits (exploiting fingerprint sparsity) instead of a full dense matmul; non-binary input falls back to the dense product. The gather result is mathematically equivalent to the dense product **within float64 round-off** (the two sum the same terms in a different order), not bit-identical. One canonical C-contiguous `(L, D)` matrix is cached per parameter set. | (L, D) matrix (~2 MB for L=2048, D=128) | High compression ratios; JL-style guarantee for the projection itself |
501
+
502
+ Both projection methods gained fast paths — the Hadamard fold is bit-identical to the full transform for binary input, and the random-projection gather is equivalent to the dense product within float64 round-off. Measured wall-time improvements (L=16384, D=1024, on one machine):
503
+
504
+ - **Hadamard** (pruned fold-then-FWHT vs full-length transform then truncate): roughly **3x** at N=1 and **6x** at N=16.
505
+ - **Random projection** (binary row-gather vs dense matmul, at ECFP-like sparsity): roughly **28x** at N=1 and **2.5x** at N=16.
506
+
507
+ These are measured wall times on one machine, not a theoretical claim: the pruned FWHT reduces butterfly *operation count* by about 9x, but the code is already vectorized and largely bandwidth-bound, so the wall-time gain is materially less than the operation-count reduction. Block-wise methods remain the fastest overall choice. Random projection's matrix memory still grows with `L × D`. Note that `sparse=True` (the Achlioptas variant) stores its matrix **densely** and therefore saves no memory despite the name; it trades roughly 2/3 zero entries for a denser arithmetic pattern in the dense-product path, not a smaller footprint, and the binary-input gather selects matrix rows by the input's set bits rather than by the matrix nonzeros.
508
+
509
+ **Hadamard specifics.** The implementation applies seeded random sign flips and keeps the first `D` outputs in Sylvester order. It does not randomly choose which `D` of the `L` coordinates to keep, so it is a seeded, truncated Hadamard projection rather than conventional randomly subsampled SRHT. With the retained `1/sqrt(L)` normalization, averaging over the random signs gives an expected squared norm of `D / L` times the input squared norm; a distance-scale correction would multiply the outputs by `sqrt(L / D)`, but that does not remove the fixed-coordinate selection. Read the fold's bit-identity with the full transform as correctness of this transform, not as evidence of neighborhood preservation.
510
+
511
+ **Projection matrix cache.** The random-projection matrix is cached to avoid redrawing it on every call. Only a **single canonical orientation** is retained per parameter set — one C-contiguous `(L, D)` matrix; the `(D, L)` orientation is served as a non-owning `.T` view of the same buffer, so no second owning copy exists. The cache is bounded by **total retained bytes**, not entry count: the budget defaults to 512 MiB (`RP_CACHE_BYTE_BUDGET` in `compression_projection_cache.py`), with a small entry-count floor so a single oversized matrix cannot evict itself. Least-recently-used matrices are dropped once the accumulated bytes exceed the budget.
512
+
513
+ ## Project Structure
514
+
515
+ ```
516
+ fpembed/
517
+ ├── src/fpembed/ # importable package (installed editable)
518
+ │ ├── __init__.py # public API and __version__
519
+ │ ├── generator.py # EmbeddedFingerprintGenerator
520
+ │ ├── generator_smiles.py # SMILES/SELFIES/Mol input mixin
521
+ │ ├── compression.py # compress_fingerprint (orchestrator)
522
+ │ ├── compression_blockwise.py # block-wise weight schemes
523
+ │ ├── compression_projection.py # seeded truncated Hadamard + random projection
524
+ │ ├── compression_projection_cache.py # byte-bounded projection-matrix cache
525
+ │ ├── dtype_support.py # output dtype validation and casting
526
+ │ ├── embedding_cache.py # BoundedCache for embedding lookups
527
+ │ ├── hashing.py # fp_params_hash
528
+ │ ├── input_validation.py # real/finite input boundary guard
529
+ │ ├── manifest.py # representation manifest builder
530
+ │ ├── skfp_factory.py # scikit-fingerprints backend factory
531
+ │ ├── smiles_utils.py # parse_smiles, canonicalize_smiles
532
+ │ └── py.typed # PEP 561 marker
533
+ ├── examples/
534
+ │ └── quickstart.ipynb # usage notebook
535
+ ├── pyproject.toml # package metadata and dependency ranges
536
+ ├── environment.yml # authoritative exact-pin conda env (installs -e .)
537
+ └── README.md
538
+ ```
539
+
540
+ ## Examples
541
+
542
+ A Jupyter notebook with quick-start examples is available at `examples/quickstart.ipynb`.
543
+
544
+ ## License
545
+
546
+ This project is licensed under the terms of the MIT open source license. Please refer to the [LICENSE](LICENSE) for the full terms.
547
+
548
+ ## Legal & Trademark Notice
549
+
550
+ The name **"FPembed"**, its branding, and identifiers are the intellectual property of the project author (@Sciencealone).
551
+ * **Commercial Branding:** The unauthorized use of the name "FPembed" to brand, market, or promote commercial software, corporate AI engines, or proprietary services within the fields of bioinformatics, chemoinformatics, and AI drug discovery is strictly prohibited.
552
+ * **Community Protection:** This notice is established to prevent public confusion and to protect the open-source community from misleading corporate misrepresentations. For licensing queries regarding the project name, please open an issue or contact the author directly.
553
+
554
+ ## Commercial & Research Services
555
+
556
+ For the project onboarding, corporate deployments, high-performance compute scaling (HPC / Databricks / Ray), or custom model fine-tuning for pharma R&D and research grants (Horizon Europe / Art. 60 LOSU / AEI) as well as other related tasks, official technical support and contract services are available.
557
+
558
+ **[View Full Service Offerings & Contact Details](COMMERCIAL_SUPPORT.md)**
559
+
560
+ ## AI disclosure
561
+
562
+ AI usage during project development is declared in [aidecl.yaml](aidecl.yaml) following the [AI Declaration Format](https://ai-declaration.org/).
563
+
564
+ ## Support
565
+ This project is provided as-is, and may be updated over time. If you have questions, please open an issue.