fpembed 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fpembed-0.2.0 → fpembed-0.2.1}/PKG-INFO +104 -22
- fpembed-0.2.0/src/fpembed.egg-info/PKG-INFO → fpembed-0.2.1/README.md +596 -537
- {fpembed-0.2.0 → fpembed-0.2.1}/pyproject.toml +5 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/__init__.py +1 -1
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/compression.py +9 -2
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/compression_projection.py +21 -8
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/embedding_cache.py +6 -4
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/generator.py +19 -1
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/generator_smiles.py +21 -1
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/hashing.py +6 -0
- fpembed-0.2.1/src/fpembed/input_validation.py +74 -0
- fpembed-0.2.1/src/fpembed/manifest.py +170 -0
- fpembed-0.2.0/README.md → fpembed-0.2.1/src/fpembed.egg-info/PKG-INFO +619 -517
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed.egg-info/SOURCES.txt +2 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/LICENSE +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/setup.cfg +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/compression_blockwise.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/compression_projection_cache.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/dtype_support.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/py.typed +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/skfp_factory.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed/smiles_utils.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed.egg-info/dependency_links.txt +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed.egg-info/requires.txt +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/src/fpembed.egg-info/top_level.txt +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_cache.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_compression.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_compression_blockwise.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_compression_projection.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_dtype.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_dtype_properties.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_embedding_cache.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_error_transparency.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_fingerprints.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_generator.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_hashing.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_input_validation.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_new_fp_type_matrix.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_packaging.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_properties.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_properties_050.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_resources_immutability.py +0 -0
- {fpembed-0.2.0 → fpembed-0.2.1}/tests/test_smiles_utils.py +0 -0
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: fpembed
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.1
|
|
4
4
|
Summary: Generalized fingerprint embedding library
|
|
5
5
|
License-Expression: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/Sciencealone/FPembed
|
|
7
|
+
Project-URL: Commercial & Research Support, https://github.com/Sciencealone/FPembed/blob/main/COMMERCIAL_SUPPORT.md
|
|
8
|
+
Project-URL: Issue Tracker, https://github.com/Sciencealone/FPembed/issues
|
|
6
9
|
Requires-Python: >=3.10
|
|
7
10
|
Description-Content-Type: text/markdown
|
|
8
11
|
License-File: LICENSE
|
|
@@ -45,6 +48,7 @@ Dynamic: license-file
|
|
|
45
48
|
- [Code Examples](#code-examples)
|
|
46
49
|
- [Choosing a Method](#choosing-a-method)
|
|
47
50
|
- [Output Dtype](#output-dtype)
|
|
51
|
+
- [Representation Manifest](#representation-manifest)
|
|
48
52
|
- [Compression Limits](#compression-limits)
|
|
49
53
|
- [Performance Characteristics](#performance-characteristics)
|
|
50
54
|
- [Project Structure](#project-structure)
|
|
@@ -52,6 +56,7 @@ Dynamic: license-file
|
|
|
52
56
|
- [Datasets](#datasets)
|
|
53
57
|
- [License](#license)
|
|
54
58
|
- [Legal & Trademark Notice](#legal--trademark-notice)
|
|
59
|
+
- [Commercial & Research Services](#commercial--research-services)
|
|
55
60
|
- [AI disclosure](#ai-disclosure)
|
|
56
61
|
- [Support](#support)
|
|
57
62
|
|
|
@@ -168,6 +173,21 @@ print(embeddings.shape) # (3, 128) - 3 valid molecules
|
|
|
168
173
|
print(invalid_indices) # [3] - index of invalid SMILES
|
|
169
174
|
```
|
|
170
175
|
|
|
176
|
+
### Bounded-Memory Iteration
|
|
177
|
+
|
|
178
|
+
The batch methods above collect every valid embedding before stacking them into one matrix. At scale that peak allocation dominates: one million molecules at D=1024 needs about 4.1 GB as float32 or 8.2 GB as float64 for the final matrix alone, before counting the individual arrays held while it is built. When the input does not fit in memory, iterate instead:
|
|
179
|
+
|
|
180
|
+
```python
|
|
181
|
+
for index, emb in gen.IterFingerprintsFromSmiles(smiles_iterable):
|
|
182
|
+
if emb is None:
|
|
183
|
+
continue # normal invalid input at this index
|
|
184
|
+
store(index, emb) # consume or write one row at a time
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
`IterFingerprints`, `IterFingerprintsFromSmiles`, and `IterFingerprintsFromSelfies` accept any iterable, pull one input at a time, and yield `(index, embedding_or_none)`. Indices match input order and include invalid entries, which yield `None`; for `IterFingerprints`, a `None` molecule also yields `None`. Unexpected computation errors surface when the item is consumed, matching the single-molecule methods. The iterator never materializes the input or accumulates rows, so peak memory is yours to bound.
|
|
188
|
+
|
|
189
|
+
**Memory contract.** These APIs are not streaming in the sense of a total process cap. The `random_projection` matrix cache is bounded by retained bytes, but a matrix referenced by a live generator stays resident, so the cache budget is not a process-wide limit. Embeddings you keep are yours to release, and `dtype` reduces the returned and stored size, not the peak transient memory during compression.
|
|
190
|
+
|
|
171
191
|
### Raw Fingerprint (No Compression)
|
|
172
192
|
|
|
173
193
|
```python
|
|
@@ -189,6 +209,8 @@ emb = compress_fingerprint(fp, size=16)
|
|
|
189
209
|
print(emb.shape) # (1, 128)
|
|
190
210
|
```
|
|
191
211
|
|
|
212
|
+
**Input contract.** `compress_fingerprint` accepts 1-D `(L,)` and 2-D `(N, L)` arrays of real numeric dtype: boolean, signed or unsigned integer, or floating. Non-real dtypes (complex, object, string, datetime) raise `TypeError`; floating input containing NaN or either infinity raises `ValueError`. Integer output (`dtype="uint16"`) additionally requires binary input and represents integer codes or counts, not normalized floats. Neither validation nor compression mutates the array you pass in.
|
|
213
|
+
|
|
192
214
|
### Parameter Hashing
|
|
193
215
|
|
|
194
216
|
```python
|
|
@@ -216,6 +238,12 @@ print(gen.cache_info()) # CacheInfo(hits=1, misses=1, maxsize=1024, currsize=1)
|
|
|
216
238
|
gen.clear_cache()
|
|
217
239
|
```
|
|
218
240
|
|
|
241
|
+
### SMILES Canonicalization and Chemical Standardization
|
|
242
|
+
|
|
243
|
+
FPembed canonicalizes SMILES spelling before keying and before parsing, so the same molecule written two ways produces the same embedding: the pipeline works with the canonical form (atom ordering and RDKit's aromatic form). That makes results spelling-invariant for atom-order-sensitive backends, and it is why the cache key can be built from the canonical string.
|
|
244
|
+
|
|
245
|
+
Canonicalization is not chemical standardization. FPembed does not remove salts, normalize tautomers, pick a protonation state, or impose a stereochemistry policy, because those decisions change scientific meaning and are the caller's to make. Two inputs that differ by a counter-ion or a tautomer stay distinct. If your workflow needs standardization, apply it before calling FPembed and describe it with `representation_manifest(preprocessing=...)` (see [Representation Manifest](#representation-manifest)).
|
|
246
|
+
|
|
219
247
|
## Why Use Embedded Fingerprints?
|
|
220
248
|
|
|
221
249
|
Predictive accuracy is one axis of comparison between raw and embedded fingerprints - and the gap can be narrow, especially on large datasets where raw fingerprints have enough data to exploit all 2048 bits directly. However, accuracy is not the only metric that matters. The one guaranteed advantage is dimensionality: an embedded vector has D = L / C features instead of L, and the stored byte count follows directly from D and the chosen dtype (calculated exactly in [Storage Size](#storage-size)). The operational benefits that follow — training and inference speed, memory during training, sample efficiency — are consequences of fewer features, but their magnitude is workload-, model-, and hardware-dependent, not a fixed guarantee.
|
|
@@ -236,7 +264,7 @@ The output dimensionality (D = L / C) is deterministic and independent of datase
|
|
|
236
264
|
|
|
237
265
|
A word on the baseline. Comparing an embedding against a raw fingerprint *stored as float64* overstates the storage gain, because a raw fingerprint is binary and would sensibly be bit-packed. Against **bit-packed** raw fingerprints, the `geometric` embedding at C=16 in float64 is actually *larger* (1 KB vs 256 B per molecule at L=2048): geometric at C=16 is a reversible re-encoding — it packs each 16-bit block into a single wide value, carrying the same information in fewer but larger numbers. Storing that embedding as `uint16` (see [Output Dtype](#output-dtype)) brings it back to parity with bit-packed input.
|
|
238
266
|
|
|
239
|
-
So raw byte savings are **not** the reason to embed. The genuine, unconditional advantage is the **downstream feature count**: the embedded vector has D = L / C features instead of L, and
|
|
267
|
+
So raw byte savings are **not** the reason to embed. The genuine, unconditional advantage is the **downstream feature count**: the embedded vector has D = L / C features instead of L. Downstream training, inference, and the in-memory feature matrix then shrink roughly in proportion to the feature count, and that reduction is what the rest of this section quantifies. Keep the scaling directional rather than universal: similarity on bit-packed fingerprints works on packed words, tree learners subsample features and pay fixed costs, and approximate-neighbor indexes change the comparison, so none of these costs follows the feature count exactly.
|
|
240
268
|
|
|
241
269
|
### ML Training and Inference Speed
|
|
242
270
|
|
|
@@ -244,7 +272,7 @@ The downstream ML model operates on the feature vector. Fewer features means fas
|
|
|
244
272
|
|
|
245
273
|
- **Tree-based models (Random Forest, XGBoost)**: Feature splitting cost is proportional to the number of features. Going from 2048 to 128 features means each tree split considers ~16x fewer candidates. For hyperparameter searches (e.g., Optuna with hundreds of trials), this compounds into significant wall-clock savings.
|
|
246
274
|
- **Neural networks**: The first dense layer's weight matrix shrinks from `(2048 x hidden)` to `(128 x hidden)` - 16x fewer parameters and 16x fewer multiply-adds per forward pass.
|
|
247
|
-
- **Distance-based methods (k-NN, similarity search)**: Pairwise distance computation is O(N² x D). Reducing D from 2048 to 128
|
|
275
|
+
- **Distance-based methods (k-NN, similarity search)**: Pairwise distance computation is O(N² x D). Reducing D from 2048 to 128 shrinks the distance computation itself by 16x; index build and query overhead, memory traffic, and candidate re-ranking sit on top of that and reduce the realized speedup.
|
|
248
276
|
|
|
249
277
|
### Memory During ML Training
|
|
250
278
|
|
|
@@ -252,12 +280,14 @@ During model training, the feature matrix for N=100K molecules occupies `(100000
|
|
|
252
280
|
|
|
253
281
|
### Compression Overhead
|
|
254
282
|
|
|
255
|
-
The compression step itself is
|
|
283
|
+
The compression step itself is small for block-wise methods, but its cost is not universal. Under warm-cache conditions, and on the machine where the project measured it, block-wise compression added on the order of 1 ms per 1000 molecules. Treat that as an order-of-magnitude indication with `L`, dtype, and hardware attached, not as a constant. Cold-cache cost is higher: the projection methods build or fetch a projection artifact on first use, and `random_projection` additionally pays to draw and store its `(L, D)` matrix.
|
|
284
|
+
|
|
285
|
+
The total pipeline cost is:
|
|
256
286
|
|
|
257
287
|
- **Raw**: `skfp generation time`
|
|
258
|
-
- **Embedded**: `skfp generation time +
|
|
288
|
+
- **Embedded**: `skfp generation time + compression time`
|
|
259
289
|
|
|
260
|
-
The downstream ML
|
|
290
|
+
Measure both phases on the target hardware before relying on the overhead being negligible. The downstream ML saving from 128 versus 2048 features is usually larger, but that too is workload-dependent.
|
|
261
291
|
|
|
262
292
|
### Sample Efficiency
|
|
263
293
|
|
|
@@ -289,8 +319,8 @@ The guaranteed columns are the feature count and its exact consequences (paramet
|
|
|
289
319
|
| Layered | `layered` | `min_path` (1), `max_path` (7) |
|
|
290
320
|
| Pattern | `pattern` | (none) |
|
|
291
321
|
| Avalon | `avalon` | (none) |
|
|
292
|
-
| SECFP | `secfp` | `radius` (
|
|
293
|
-
| MHFP | `mhfp` | `radius` (
|
|
322
|
+
| SECFP | `secfp` | `radius` (3), `min_radius` (1) |
|
|
323
|
+
| MHFP | `mhfp` | `radius` (3), `min_radius` (1) |
|
|
294
324
|
| MAP4 | `map` | `radius` (default 2) |
|
|
295
325
|
|
|
296
326
|
## Compression Methods
|
|
@@ -303,10 +333,10 @@ FPembed supports six compression methods, selectable via the `method` parameter
|
|
|
303
333
|
|-------------------------|----------|-----------------|---------------------------------------|------------|
|
|
304
334
|
| `geometric` | block-wise | `interleave` (bool) | (2**C − 1):1 dynamic range (e.g. 65,535:1 at C=16) | O(L) |
|
|
305
335
|
| `linear` | block-wise | `interleave` (bool) | C:1 dynamic range | O(L) |
|
|
306
|
-
| `log` | block-wise | `interleave` (bool) | ~4.1:1
|
|
336
|
+
| `log` | block-wise | `interleave` (bool) | ~log2(C+1):1 (~4.1:1 at C=16) | O(L) |
|
|
307
337
|
| `uniform` | block-wise | `interleave` (bool) | 1:1 (mean pooling) | O(L) |
|
|
308
|
-
| `hadamard` | global | `seed` (int) |
|
|
309
|
-
| `random_projection` | global | `seed` (int) + `sparse` (bool) | JL distance preservation | O(L D) |
|
|
338
|
+
| `hadamard` | global | `seed` (int) | seeded, truncated Hadamard projection (expected squared norm D/L) | O(L log L) |
|
|
339
|
+
| `random_projection` | global | `seed` (int) + `sparse` (bool) | JL-style distance preservation for the projection itself | O(L D) |
|
|
310
340
|
|
|
311
341
|
### The `method` Parameter
|
|
312
342
|
|
|
@@ -316,7 +346,7 @@ Pass `method` to the Generator constructor to select a compression strategy.
|
|
|
316
346
|
|
|
317
347
|
- **Block-wise methods** (`geometric`, `linear`, `log`, `uniform`): accept `interleave` (bool, default `False`). When `True`, bits are assigned to blocks by stride (`bit[i] -> block[i % n_blocks]`) instead of contiguous partitioning, breaking hash clustering artifacts.
|
|
318
348
|
- **`hadamard`**: accepts `seed` (int, default `42`). Controls the random sign flips applied before the Fast Walsh-Hadamard Transform.
|
|
319
|
-
- **`random_projection`**: accepts `seed` (int, default `42`) and `sparse` (bool, default `False`).
|
|
349
|
+
- **`random_projection`**: accepts `seed` (int, default `42`) and `sparse` (bool, default `False`). `sparse=True` draws the Achlioptas distribution, whose entries are about 2/3 zero, but the matrix is stored densely, so it saves no memory. On the binary-input fast path the work is set by the number of set bits in the input row, not by the zeros in the matrix, so `sparse` does not automatically make that path faster; any arithmetic saving applies to the dense-product fallback used for non-binary input. It also changes the projection values, so compare the two settings on your task.
|
|
320
350
|
|
|
321
351
|
Seed-based methods (`hadamard`, `random_projection`) are fully deterministic given the same seed and NumPy version. The default seed is `42`.
|
|
322
352
|
|
|
@@ -367,7 +397,19 @@ print(emb.shape) # (1, 128)
|
|
|
367
397
|
|
|
368
398
|
### Choosing a Method
|
|
369
399
|
|
|
370
|
-
Block-wise methods (`geometric`, `linear`, `log`, `uniform`) are fast (O(L)) and simple
|
|
400
|
+
Block-wise methods (`geometric`, `linear`, `log`, `uniform`) are fast (O(L)) and simple, so reach for them when speed or modest compression ratios matter most. `geometric` keeps the widest code range per block, while `uniform` treats every bit equally (mean pooling). Global projection methods (`hadamard`, `random_projection`) mix information across all input bits, which tends to retain more information at high compression ratios. `hadamard` is efficient (O(L log L)) and requires power-of-2 fingerprint sizes; `random_projection` carries the strongest theoretical distance-preservation statement (Johnson-Lindenstrauss) at the cost of O(L D) work.
|
|
401
|
+
|
|
402
|
+
Method rankings depend on the downstream model, so treat any single recommendation as a starting point rather than an optimum. The table below records the direction found in the project's own experiments (see `dev_docs/investigation_foundation_model_embedding_compression.md`); those measurements were taken on specific datasets and models and do not transfer as a guarantee. The recorded configuration ECFP with `fp_size=16384` and `C=16` is a starting candidate, not a proven optimum, and it does not change the default method.
|
|
403
|
+
|
|
404
|
+
| Application | Starting point | What to check before relying on it |
|
|
405
|
+
|---|---|---|
|
|
406
|
+
| Tree-based molecular property prediction | `geometric`, `C=16` | Compare against raw fingerprints and other block-wise methods; tree feature subsampling changes the realized speedup |
|
|
407
|
+
| Linear prediction | `random_projection` | Re-tune regularization after changing the representation |
|
|
408
|
+
| Approximate similarity, clustering, or retrieval | `random_projection` | Measure neighborhood preservation and `recall@k` under the intended metric; cosine correlation with a reference embedding is not enough |
|
|
409
|
+
| Exact binary similarity (Tanimoto) | Bit-packed raw fingerprints | Compressed-space Euclidean distance is not binary Tanimoto |
|
|
410
|
+
| Dense continuous embeddings | Projection methods | The geometric binary-code argument no longer applies |
|
|
411
|
+
|
|
412
|
+
Retrieval and similarity use cases need task-specific retrieval evaluation. A high correlation with a reference embedding does not establish that nearest neighbors are preserved.
|
|
371
413
|
|
|
372
414
|
### Output Dtype
|
|
373
415
|
|
|
@@ -375,7 +417,7 @@ Both `EmbeddedFingerprintGenerator` (constructor argument `dtype`, applied to ev
|
|
|
375
417
|
|
|
376
418
|
| `dtype` | Bytes/value | Semantics |
|
|
377
419
|
|---|---|---|
|
|
378
|
-
| `float64` (default) | 8 |
|
|
420
|
+
| `float64` (default) | 8 | The values existing callers already get, computed exactly as before. For `geometric`, the value equals the integer code divided by `2**C - 1`, apart from the float64 round-off of the accumulation path. |
|
|
379
421
|
| `float32` | 4 | **Information-preserving** for `geometric` at `C <= 16` (see below). ~1e-7 relative error for the projection methods. Permitted for every method. |
|
|
380
422
|
| `uint16` | 2 | The **exact integer code**. Permitted only for `geometric` and `uniform`, and only for binary input. |
|
|
381
423
|
|
|
@@ -404,6 +446,19 @@ The stored `float32` value is **not** bit-equal to the `float64` output — it d
|
|
|
404
446
|
|
|
405
447
|
`interleave=True` is supported identically — interleaving changes which bits form a block, not the recovery formula.
|
|
406
448
|
|
|
449
|
+
#### Reversibility is not distance preservation
|
|
450
|
+
|
|
451
|
+
At supported precision, `geometric` recovers the original bits exactly, but that does not mean the decoded values preserve the geometry of those bits, or that a model can reach them equally easily. A two-bit counterexample, with `C=2` so `2**C - 1 = 3`:
|
|
452
|
+
|
|
453
|
+
- Codes `1` and `2` differ in both bits, yet their decoded values differ by only `1/3`.
|
|
454
|
+
- Codes `0` and `2` differ in one bit, yet their decoded values differ by `2/3`.
|
|
455
|
+
|
|
456
|
+
Ranking pairs by Euclidean distance in the decoded space therefore does not reproduce ranking by Hamming distance. Higher-order bits also map to larger positional values than lower-order bits, which makes them easier for a threshold-based model to reach; injectivity alone does not make all bits equally learnable. Exact binary similarity (Tanimoto on the original bits) stays a separate operation from comparing compressed vectors.
|
|
457
|
+
|
|
458
|
+
Pooling methods (`linear`, `log`, `uniform`) additionally collide distinct blocks, so their suitability has to be measured on the task rather than inferred from the weight range.
|
|
459
|
+
|
|
460
|
+
Keep the sources of loss separate: fingerprint hashing collisions come from the fingerprint itself, block pooling from the compression step, float rounding from the stored dtype, and any accuracy change from the downstream model. Only pooling and rounding are compression concerns.
|
|
461
|
+
|
|
407
462
|
**Which combinations raise `ValueError`:**
|
|
408
463
|
|
|
409
464
|
- `uint16` with `linear`, `log`, `hadamard`, or `random_projection` — these have no exact integer form.
|
|
@@ -420,13 +475,30 @@ fp = np.random.randint(0, 2, size=2048).astype(np.uint8)
|
|
|
420
475
|
# Exact integer code, 4x smaller than float64
|
|
421
476
|
codes = compress_fingerprint(fp, size=16, dtype="uint16")
|
|
422
477
|
|
|
423
|
-
# Recover the float64
|
|
424
|
-
recovered = codes / (2**16 - 1)
|
|
478
|
+
# Recover the geometric float64 value (the code / (2**16 - 1), up to float64 round-off)
|
|
479
|
+
recovered = codes / (2**16 - 1)
|
|
425
480
|
```
|
|
426
481
|
|
|
482
|
+
### Representation Manifest
|
|
483
|
+
|
|
484
|
+
`fp_params_hash()` is a cache key: it covers the fingerprint type, its parameters, and the compression method. It is deliberately narrow and must not change, so it is not a complete identity for an array you persist. `EmbeddedFingerprintGenerator.representation_manifest()` returns a fresh JSON-compatible mapping for that purpose:
|
|
485
|
+
|
|
486
|
+
```python
|
|
487
|
+
import json
|
|
488
|
+
|
|
489
|
+
manifest = gen.representation_manifest()
|
|
490
|
+
print(json.dumps(manifest, indent=2))
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
It records the resolved fingerprint type, width, and parameters (omitted and explicitly-default values resolve to the same settings), whether compression is active together with its factor, method and effective method parameters including `seed` and `interleave`, the output dimension, dtype, and physical encoding, the built-in canonicalization policy, an optional declarative `preprocessing` mapping you supply, and the installed `fpembed`, `numpy`, `rdkit`, `scikit-fingerprints`, and `selfies` versions. `schema_version` describes the manifest format and moves independently of the package version.
|
|
494
|
+
|
|
495
|
+
The `output.encoding` field distinguishes what the values physically are: `raw_binary` for an uncompressed fingerprint, `normalized_block_value` and `mean_block_value` for geometric and uniform float output, `integer_code` and `integer_count` for the exact `uint16` codes, `weighted_block_value` for linear and log, and `projection_value` for Hadamard and random projection. A `uint16` geometric array and its `float64` counterpart are therefore not interchangeable: the recovery formula relates them, but only the integer form is exact.
|
|
496
|
+
|
|
497
|
+
Built-in canonicalization and external preprocessing are reported separately. The manifest stores a caller-supplied `preprocessing` mapping verbatim under `preprocessing.external` and never executes it; supplying one does not make the pipeline standardized. The manifest is additive: it does not change `fp_params_hash`, cache keys, or returned arrays. Implementations may change the layout between versions, so re-derive a manifest rather than comparing its bytes across `fpembed` releases.
|
|
498
|
+
|
|
427
499
|
### Compression Limits
|
|
428
500
|
|
|
429
|
-
`geometric` compression is capped at `C <= 32`; requesting `compression > 32` with `geometric` (at either entry point) raises `ValueError`. Geometric block weights span `2**-C`, so the block integer survives only while it fits the float64 mantissa. The einsum accumulates roughly `C
|
|
501
|
+
`geometric` compression is capped at `C <= 32`; requesting `compression > 32` with `geometric` (at either entry point) raises `ValueError`. Geometric block weights span `2**-C`, so the block integer survives only while it fits the float64 mantissa. The einsum accumulates roughly `C · 2**-52` relative error against a code spacing of `2**-C`, so recovery starts failing well before the naive 53-bit limit — at `C = 64` the low ~11 bits of every block are silently absorbed. `C <= 32` is the largest power of two comfortably inside the safe range (verified exact for `C <= 16`). A `ValueError`, not a warning, is used, because warnings can be suppressed in evaluation runs — precisely where silent bit loss does the most damage.
|
|
430
502
|
|
|
431
503
|
The cap applies only to `geometric` and only when compression is active (`compression=None` or `0` is exempt). The other methods (`linear`, `log`, `uniform`, `hadamard`, `random_projection`) have bounded weight ratios or use no block weights, so none approaches the mantissa limit even at large `C` and none is capped.
|
|
432
504
|
|
|
@@ -437,15 +509,17 @@ All methods produce the same output dimensionality (D = L / compression) but dif
|
|
|
437
509
|
| Method | Speed | Precomputed Memory | Best For |
|
|
438
510
|
|--------|------------------------------------------------------------------------------------|--------------------------------------|----------|
|
|
439
511
|
| Block-wise (all four) | Fastest - single vectorized einsum, O(L) | Negligible (C-length weight vector) | Default choice; large batches |
|
|
440
|
-
| Hadamard (
|
|
441
|
-
| Random projection | Fast - for binary input, gathers and sums only the matrix rows selected by the set bits (exploiting fingerprint sparsity) instead of a full dense matmul; non-binary input falls back to the dense product. The gather result is mathematically equivalent to the dense product **within float64 round-off** (the two sum the same terms in a different order), not bit-identical. One canonical C-contiguous `(L, D)` matrix is cached per parameter set. | (L, D) matrix (~2 MB for L=2048, D=128) |
|
|
512
|
+
| Hadamard (seeded, truncated) | Fast: pruned fold-then-FWHT, where the input is folded down to length D *before* transforming, so the discarded outputs are never computed. The retained first-D outputs are bit-identical to truncating the full transform for binary input. Correctness against that reference does not by itself prove neighborhood or distance preservation. | L-length sign vector (~16 KB) | High compression ratios |
|
|
513
|
+
| Random projection | Fast - for binary input, gathers and sums only the matrix rows selected by the set bits (exploiting fingerprint sparsity) instead of a full dense matmul; non-binary input falls back to the dense product. The gather result is mathematically equivalent to the dense product **within float64 round-off** (the two sum the same terms in a different order), not bit-identical. One canonical C-contiguous `(L, D)` matrix is cached per parameter set. | (L, D) matrix (~2 MB for L=2048, D=128) | High compression ratios; JL-style guarantee for the projection itself |
|
|
442
514
|
|
|
443
515
|
Both projection methods gained fast paths — the Hadamard fold is bit-identical to the full transform for binary input, and the random-projection gather is equivalent to the dense product within float64 round-off. Measured wall-time improvements (L=16384, D=1024, on one machine):
|
|
444
516
|
|
|
445
517
|
- **Hadamard** (pruned fold-then-FWHT vs full-length transform then truncate): roughly **3x** at N=1 and **6x** at N=16.
|
|
446
518
|
- **Random projection** (binary row-gather vs dense matmul, at ECFP-like sparsity): roughly **28x** at N=1 and **2.5x** at N=16.
|
|
447
519
|
|
|
448
|
-
These are measured wall times on one machine, not a theoretical claim
|
|
520
|
+
These are measured wall times on one machine, not a theoretical claim: the pruned FWHT reduces butterfly *operation count* by about 9x, but the code is already vectorized and largely bandwidth-bound, so the wall-time gain is materially less than the operation-count reduction. Block-wise methods remain the fastest overall choice. Random projection's matrix memory still grows with `L × D`. Note that `sparse=True` (the Achlioptas variant) stores its matrix **densely** and therefore saves no memory despite the name; it trades roughly 2/3 zero entries for a denser arithmetic pattern in the dense-product path, not a smaller footprint, and the binary-input gather selects matrix rows by the input's set bits rather than by the matrix nonzeros.
|
|
521
|
+
|
|
522
|
+
**Hadamard specifics.** The implementation applies seeded random sign flips and keeps the first `D` outputs in Sylvester order. It does not randomly choose which `D` of the `L` coordinates to keep, so it is a seeded, truncated Hadamard projection rather than conventional randomly subsampled SRHT. With the retained `1/sqrt(L)` normalization, averaging over the random signs gives an expected squared norm of `D / L` times the input squared norm; a distance-scale correction would multiply the outputs by `sqrt(L / D)`, but that does not remove the fixed-coordinate selection. Read the fold's bit-identity with the full transform as correctness of this transform, not as evidence of neighborhood preservation.
|
|
449
523
|
|
|
450
524
|
**Projection matrix cache.** The random-projection matrix is cached to avoid redrawing it on every call. Only a **single canonical orientation** is retained per parameter set — one C-contiguous `(L, D)` matrix; the `(D, L)` orientation is served as a non-owning `.T` view of the same buffer, so no second owning copy exists. The cache is bounded by **total retained bytes**, not entry count: the budget defaults to 512 MiB (`RP_CACHE_BYTE_BUDGET` in `compression_projection_cache.py`), with a small entry-count floor so a single oversized matrix cannot evict itself. Least-recently-used matrices are dropped once the accumulated bytes exceed the budget.
|
|
451
525
|
|
|
@@ -459,11 +533,13 @@ fpembed/
|
|
|
459
533
|
│ ├── generator_smiles.py # SMILES/SELFIES/Mol input mixin
|
|
460
534
|
│ ├── compression.py # compress_fingerprint (orchestrator)
|
|
461
535
|
│ ├── compression_blockwise.py # block-wise weight schemes
|
|
462
|
-
│ ├── compression_projection.py # Hadamard
|
|
536
|
+
│ ├── compression_projection.py # seeded truncated Hadamard + random projection
|
|
463
537
|
│ ├── compression_projection_cache.py # byte-bounded projection-matrix cache
|
|
464
538
|
│ ├── dtype_support.py # output dtype validation and casting
|
|
465
539
|
│ ├── embedding_cache.py # BoundedCache for embedding lookups
|
|
466
540
|
│ ├── hashing.py # fp_params_hash
|
|
541
|
+
│ ├── input_validation.py # real/finite input boundary guard
|
|
542
|
+
│ ├── manifest.py # representation manifest builder
|
|
467
543
|
│ ├── skfp_factory.py # scikit-fingerprints backend factory
|
|
468
544
|
│ ├── smiles_utils.py # parse_smiles, canonicalize_smiles
|
|
469
545
|
│ └── py.typed # PEP 561 marker
|
|
@@ -472,7 +548,7 @@ fpembed/
|
|
|
472
548
|
│ ├── datasets/ # molecular datasets (RedDB, NFA, QM9)
|
|
473
549
|
│ └── nicegui_app/ # NiceGUI demo application
|
|
474
550
|
├── pyproject.toml # package metadata and dependency ranges
|
|
475
|
-
├── requirements.txt # third-party app lock (no
|
|
551
|
+
├── requirements.txt # third-party app lock (no FPembed self-pin)
|
|
476
552
|
├── environment.yml # authoritative conda env (installs -e .)
|
|
477
553
|
└── README.md
|
|
478
554
|
```
|
|
@@ -529,6 +605,12 @@ The name **"FPembed"**, its branding, and identifiers are the intellectual prope
|
|
|
529
605
|
* **Commercial Branding:** The unauthorized use of the name "FPembed" to brand, market, or promote commercial software, corporate AI engines, or proprietary services within the fields of bioinformatics, chemoinformatics, and AI drug discovery is strictly prohibited.
|
|
530
606
|
* **Community Protection:** This notice is established to prevent public confusion and to protect the open-source community from misleading corporate misrepresentations. For licensing queries regarding the project name, please open an issue or contact the author directly.
|
|
531
607
|
|
|
608
|
+
## Commercial & Research Services
|
|
609
|
+
|
|
610
|
+
For the project onboarding, corporate deployments, high-performance compute scaling (HPC / Databricks / Ray), or custom model fine-tuning for pharma R&D and research grants (Horizon Europe / Art. 60 LOSU / AEI) as well as other related tasks, official technical support and contract services are available.
|
|
611
|
+
|
|
612
|
+
**[View Full Service Offerings & Contact Details](COMMERCIAL_SUPPORT.md)**
|
|
613
|
+
|
|
532
614
|
## AI disclosure
|
|
533
615
|
|
|
534
616
|
AI usage during project development is declared in [aidecl.yaml](aidecl.yaml) following the [AI Declaration Format](https://ai-declaration.org/).
|