superintervals 0.2.1__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {superintervals-0.2.1/src/superintervals.egg-info → superintervals-0.2.2}/PKG-INFO +1 -1
- superintervals-0.2.2/README.md +282 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/pyproject.toml +2 -2
- {superintervals-0.2.1 → superintervals-0.2.2/src/superintervals.egg-info}/PKG-INFO +1 -1
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.hpp +9 -86
- superintervals-0.2.1/README.md +0 -279
- {superintervals-0.2.1 → superintervals-0.2.2}/LICENSE +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/MANIFEST.in +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/setup.cfg +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/setup.py +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals/__init__.py +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals/intervalset.cpp +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals/intervalset.pxd +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals/intervalset.pyx +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.egg-info/SOURCES.txt +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.egg-info/dependency_links.txt +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.egg-info/requires.txt +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.egg-info/top_level.txt +0 -0
- {superintervals-0.2.1 → superintervals-0.2.2}/test/tests.py +0 -0
|
@@ -0,0 +1,282 @@
|
|
|
1
|
+
SuperIntervals
|
|
2
|
+
==============
|
|
3
|
+
|
|
4
|
+
A fast, memory-efficient data structure for interval intersection queries.
|
|
5
|
+
SuperIntervals uses a novel superset-index approach that maintains
|
|
6
|
+
intervals in position-sorted order, enabling cache-friendly searches and SIMD-optimized counting.
|
|
7
|
+
|
|
8
|
+
### Features:
|
|
9
|
+
|
|
10
|
+
- Linear-time index construction from sorted intervals
|
|
11
|
+
- Cache-friendly querying
|
|
12
|
+
- SIMD acceleration (AVX2/Neon) for counting operations
|
|
13
|
+
- Minimal memory overhead (one size_t per interval)
|
|
14
|
+
- Available for C++, Rust, Python, and C
|
|
15
|
+
- Optional Eytzinger memory layout for slightly faster queries (C++/Rust only)
|
|
16
|
+
- No dependencies, header only
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
## Quick Start
|
|
20
|
+
|
|
21
|
+
- Intervals are considered end-inclusive
|
|
22
|
+
- The index() function must be called before any queries
|
|
23
|
+
- Found intervals are returned in reverse position-sorted order
|
|
24
|
+
|
|
25
|
+
### 🐍 Python
|
|
26
|
+
|
|
27
|
+
```python
|
|
28
|
+
from superintervals import IntervalSet
|
|
29
|
+
|
|
30
|
+
iset = IntervalSet()
|
|
31
|
+
iset.add(10, 20, 'A')
|
|
32
|
+
iset.index()
|
|
33
|
+
overlaps = iset.find_overlaps(8, 20)
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### ⚙️ C++
|
|
37
|
+
```cpp
|
|
38
|
+
#include "SuperIntervals.hpp"
|
|
39
|
+
|
|
40
|
+
SuperIntervals<int, std::string> intervals;
|
|
41
|
+
intervals.add(1, 5, "A");
|
|
42
|
+
intervals.index();
|
|
43
|
+
std::vector<std::string> results;
|
|
44
|
+
intervals.findOverlaps(4, 9, results);
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
### 🦀 Rust
|
|
48
|
+
|
|
49
|
+
```rust
|
|
50
|
+
use super_intervals::SuperIntervals;
|
|
51
|
+
|
|
52
|
+
let mut intervals = SuperIntervals::new();
|
|
53
|
+
intervals.add(1, 5, "A");
|
|
54
|
+
intervals.index();
|
|
55
|
+
let mut results = Vec::new();
|
|
56
|
+
intervals.find_overlaps(4, 11, &mut results);
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
## Test programs
|
|
61
|
+
Test programs expect plain text BED files and only assess chr1 records - other chromosomes are ignored.
|
|
62
|
+
|
|
63
|
+
C++ program compares SuperIntervals, ImplicitIntervalTree, IntervalTree and NCLS:
|
|
64
|
+
```
|
|
65
|
+
cd test; make
|
|
66
|
+
./run-cpp-libs a.bed b.bed
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Rust program:
|
|
70
|
+
```
|
|
71
|
+
RUSTFLAGS="-Ctarget-cpu=native" cargo run --release --example bed-intersect-si
|
|
72
|
+
cargo run --release --example bed-intersect-si a.bed b.bed
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Benchmark
|
|
76
|
+
|
|
77
|
+
Benchmark
|
|
78
|
+
|
|
79
|
+
SuperIntervals (SI) was compared with:
|
|
80
|
+
|
|
81
|
+
Coitrees (Rust: https://github.com/dcjones/coitrees)
|
|
82
|
+
Implicit Interval Tree (C++: https://github.com/lh3/cgranges)
|
|
83
|
+
Interval Tree (C++: https://github.com/ekg/intervaltree)
|
|
84
|
+
Nested Containment List (C: https://github.com/pyranges/ncls/tree/master/ncls/src)
|
|
85
|
+
|
|
86
|
+
Main results:
|
|
87
|
+
|
|
88
|
+
- Finding interval intersections is roughly ~1.5-3x faster than the next best library (Coitrees for Rust, Implicit Interval Tree for C++), with some
|
|
89
|
+
exceptions. Coitrees-s was faster for one test (ONT reads, sorted DB53 reads).
|
|
90
|
+
- The SIMD counting performance of coitrees and superintervals is similar.
|
|
91
|
+
|
|
92
|
+
Datasets:
|
|
93
|
+
|
|
94
|
+
1. `rna / anno` RNA-seq reads and annotations from cgranges repository
|
|
95
|
+
2. `ONT reads` nanopore alignments from sample PAO33946 chr1, converted to bed format
|
|
96
|
+
3. `DB53 reads` paired-end reads from sample DB53, NCBI BioProject PRJNA417592, chr1, converted to bed format
|
|
97
|
+
4. `mito-b, mito-a` paired-end reads from sample DB53 chrM, converted to bed format (mito-b and mito-a are the same)
|
|
98
|
+
5. `genes` UCSC genes from hg19
|
|
99
|
+
|
|
100
|
+
Test programs use internal timers and print data to stdout, measuring the index time, and time to find all intersections. Other steps such as file IO are ignored. Test programs also only assess chr1 bed records - other chromosomes are ignored. For 'chrM' records, the M was replaced with 1 using sed. Data were assessed in position sorted and random order. Datasets can be found on the Releases page, and the test/run_tools.sh script has instructions for how to repeat the benchmark.
|
|
101
|
+
|
|
102
|
+
Timings were in microseconds using an i9-11900K, 64 GB, 2TB NVMe machine.
|
|
103
|
+
## Finding interval intersections
|
|
104
|
+
|
|
105
|
+
Coitrees-s uses the SortedQuerent version of coitrees
|
|
106
|
+
SI = superintervals. Eytz refers to the eytzinger layout. -rs is the Rust implementation.
|
|
107
|
+
|
|
108
|
+
### Intervals in sorted order
|
|
109
|
+
|
|
110
|
+
| | Coitrees | Coitrees-s | SuperIntervals-rs | SuperIntervalsEytz-rs | ImplicitITree-C++ | IntervalTree-C++ | NCLS-C | SuperIntervals-C++ | SuperIntervalsEytz-C++ |
|
|
111
|
+
| --------------------- | -------- | ---------- |-------------------| --------------------- | ----------------- | ---------------- | ------ | ------------------ | ---------------------- |
|
|
112
|
+
| DB53 reads, ONT reads | 1668 | 3179 | **757** | **757** | 3831 | 44404 | 10642 | **1315** | 1358 |
|
|
113
|
+
| DB53 reads, genes | 55 | 84 | **21** | **21** | 122 | 109 | 291 | 42 | **40** |
|
|
114
|
+
| ONT reads, DB53 reads | 6504 | **3354** | 3859 | 3854 | 17949 | 12280 | 30772 | 5290 | **4462** |
|
|
115
|
+
| anno, rna | 50 | 35 | **18** | **18** | 127 | 90 | 208 | 29 | **22** |
|
|
116
|
+
| genes, DB53 reads | 1171 | 1018 | 301 | **296** | 3129 | 1315 | 1780 | 442 | **323** |
|
|
117
|
+
| mito-b, mito-a | 34769 | 34594 | 16971 | **16952** | 93900 | 107660 | 251707 | 33177 | **32985** |
|
|
118
|
+
| rna, anno | 31 | 23 | 21 | **20** | 70 | 55 | 233 | 28 | **27** |
|
|
119
|
+
|
|
120
|
+
### Intervals in random order
|
|
121
|
+
|
|
122
|
+
| | Coitrees | Coitrees-s | SuperIntervals-rs | SuperIntervalsEytz-rs | ImplicitITree-C++ | IntervalTree-C++ | NCLS-C | SuperIntervals-C++ | SuperIntervalsEytz-C++ |
|
|
123
|
+
| --------------------- | -------- | ---------- | ----------------- | --------------------- | ----------------- | ---------------- | ------ | ------------------ | ---------------------- |
|
|
124
|
+
| DB53 reads, ONT reads | 2943 | 4663 | 1356 | **1355** | 6505 | 46743 | 11947 | 2491 | **2169** |
|
|
125
|
+
| DB53 reads, genes | 78 | 130 | 27 | **26** | 170 | 125 | 305 | 58 | **51** |
|
|
126
|
+
| ONT reads, DB53 reads | 16650 | 18931 | 16116 | **16037** | 38677 | 27832 | 53452 | **23003** | 23232 |
|
|
127
|
+
| anno, rna | 89 | 105 | **54** | **54** | 188 | 143 | 294 | **58** | 60 |
|
|
128
|
+
| genes, DB53 reads | 2222 | 2424 | 1693 | **1684** | 4490 | 2701 | 3605 | **1251** | 1749 |
|
|
129
|
+
| mito-b, mito-a | 38030 | 86309 | **18326** | 18368 | 125336 | 118321 | 256293 | 42195 | **41695** |
|
|
130
|
+
| rna, anno | 53 | 73 | **45** | **45** | 137 | 83 | 311 | **52** | **52** |
|
|
131
|
+
|
|
132
|
+
## Counting interval intersections
|
|
133
|
+
|
|
134
|
+
### Intervals in sorted order
|
|
135
|
+
|
|
136
|
+
| | Coitrees | SuperIntervals-rs | SuperIntervalsEytz-rs | SuperIntervals-C++ | SuperIntervalsEytz-C++ |
|
|
137
|
+
| --------------------- | -------- | ----------------- | --------------------- | ------------------ | ---------------------- |
|
|
138
|
+
| DB53 reads, ONT reads | 551 | 370 | 371 | **241** | 263 |
|
|
139
|
+
| DB53 reads, genes | 28 | 12 | 12 | 8 | **7** |
|
|
140
|
+
| ONT reads, DB53 reads | 2478 | 1909 | 1890 | 2209 | **1312** |
|
|
141
|
+
| anno, rna | 26 | 14 | 14 | 22 | **11** |
|
|
142
|
+
| genes, DB53 reads | 747 | 321 | 336 | 446 | **290** |
|
|
143
|
+
| mito-b, mito-a | 6894 | 6727 | 6746 | 3088 | **2966** |
|
|
144
|
+
| rna, anno | **9** | 13 | 13 | 12 | 10 |
|
|
145
|
+
|
|
146
|
+
### Intervals in random order
|
|
147
|
+
|
|
148
|
+
| | Coitrees | SuperIntervals-rs | SuperIntervalsEytz-rs | SuperIntervals-C++ | SuperIntervalsEytz-C++ |
|
|
149
|
+
| --------------------- | -------- | ----------------- | --------------------- | ------------------ | ---------------------- |
|
|
150
|
+
| DB53 reads, ONT reads | 1988 | 972 | 969 | 1016 | **778** |
|
|
151
|
+
| DB53 reads, genes | 53 | 20 | 20 | 16 | **13** |
|
|
152
|
+
| ONT reads, DB53 reads | 6692 | 8864 | 8733 | **8182** | 9523 |
|
|
153
|
+
| anno, rna | 52 | 49 | 48 | **47** | 50 |
|
|
154
|
+
| genes, DB53 reads | 1503 | 1628 | 1592 | **1120** | 1623 |
|
|
155
|
+
| mito-b, mito-a | 14354 | 7579 | 7600 | 4442 | **4383** |
|
|
156
|
+
| rna, anno | 22 | 30 | 29 | **25** | **25** |
|
|
157
|
+
|
|
158
|
+
## Python
|
|
159
|
+
|
|
160
|
+
Install using `pip install superintervals`
|
|
161
|
+
|
|
162
|
+
```
|
|
163
|
+
from superintervals import IntervalSet
|
|
164
|
+
|
|
165
|
+
iset = IntervalSet()
|
|
166
|
+
|
|
167
|
+
# Add interval start, end, identifier. Integer values are supported
|
|
168
|
+
iset.add(10, 20, 0)
|
|
169
|
+
iset.add(19, 18, 1)
|
|
170
|
+
iset.add(8, 11, 2)
|
|
171
|
+
|
|
172
|
+
# Index method must be called before queries
|
|
173
|
+
iset.index()
|
|
174
|
+
|
|
175
|
+
iset.any_overlaps(8, 20)
|
|
176
|
+
# >>> True
|
|
177
|
+
|
|
178
|
+
iset.count_overlaps(8, 20)
|
|
179
|
+
# >>> 3
|
|
180
|
+
|
|
181
|
+
iset.find_overlaps(8, 20)
|
|
182
|
+
# >>> [1, 0, 2]
|
|
183
|
+
|
|
184
|
+
iset.set_search_interval(8, 20)
|
|
185
|
+
for itv in iset:
|
|
186
|
+
print(itv)
|
|
187
|
+
|
|
188
|
+
# >>> (19, 18, 1)
|
|
189
|
+
# >>> (10, 20, 0)
|
|
190
|
+
# >>> (8, 11, 2)
|
|
191
|
+
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
## Cpp
|
|
195
|
+
|
|
196
|
+
```cpp
|
|
197
|
+
#include <iostream>
|
|
198
|
+
#include <vector>
|
|
199
|
+
#include "SuperIntervals.hpp"
|
|
200
|
+
|
|
201
|
+
int main() {
|
|
202
|
+
// Create a SuperIntervals instance for integer intervals with string data
|
|
203
|
+
// Specify with S, T template types
|
|
204
|
+
SuperIntervals<int, std::string> intervals;
|
|
205
|
+
|
|
206
|
+
// Add some intervals
|
|
207
|
+
intervals.add(1, 5, "Interval A");
|
|
208
|
+
intervals.add(3, 7, "Interval B");
|
|
209
|
+
intervals.add(6, 10, "Interval C");
|
|
210
|
+
intervals.add(8, 12, "Interval D");
|
|
211
|
+
|
|
212
|
+
// Index the intervals (must be called before querying)
|
|
213
|
+
intervals.index();
|
|
214
|
+
|
|
215
|
+
// Find overlaps for the range [4, 9]
|
|
216
|
+
std::vector<std::string> overlaps;
|
|
217
|
+
intervals.findOverlaps(4, 9, overlaps);
|
|
218
|
+
|
|
219
|
+
// Print the overlapping intervals
|
|
220
|
+
for (const auto& interval : overlaps) {
|
|
221
|
+
std::cout << interval << std::endl;
|
|
222
|
+
}
|
|
223
|
+
|
|
224
|
+
// Count the intervals instead
|
|
225
|
+
std::cout << "Count: " << intervals.countOverlaps(4, 9) << std::endl;
|
|
226
|
+
|
|
227
|
+
// Count stabbed intervals at point 7
|
|
228
|
+
std::cout << "Number of intervals containing point 7: " << intervals.countStabbed(7) << std::endl;
|
|
229
|
+
|
|
230
|
+
return 0;
|
|
231
|
+
}
|
|
232
|
+
```
|
|
233
|
+
There is also a `SuperIntervalsEytz` subclasses that can be used. `SuperIntervalsEytz`
|
|
234
|
+
uses an Eytzinger memory layout that can sometimes offer faster query times at the cost of higher memory
|
|
235
|
+
usage and slower indexing time.
|
|
236
|
+
|
|
237
|
+
## Rust
|
|
238
|
+
|
|
239
|
+
Fetch using cargo add.
|
|
240
|
+
|
|
241
|
+
```
|
|
242
|
+
use super_intervals::SuperIntervals;
|
|
243
|
+
|
|
244
|
+
fn main() {
|
|
245
|
+
// Create a new instance of SuperIntervals
|
|
246
|
+
let mut intervals = SuperIntervals::new();
|
|
247
|
+
|
|
248
|
+
// Add some intervals with associated data of type T
|
|
249
|
+
intervals.add(1, 5, "Interval A");
|
|
250
|
+
intervals.add(10, 15, "Interval B");
|
|
251
|
+
intervals.add(7, 12, "Interval C");
|
|
252
|
+
|
|
253
|
+
// Call index() to prepare the intervals for queries
|
|
254
|
+
intervals.index();
|
|
255
|
+
|
|
256
|
+
// Query for overlapping intervals with a range (4, 11)
|
|
257
|
+
let mut found_intervals = Vec::new();
|
|
258
|
+
intervals.find_overlaps(4, 11, &mut found_intervals);
|
|
259
|
+
|
|
260
|
+
// Display found intervals
|
|
261
|
+
for interval in found_intervals {
|
|
262
|
+
println!("Found overlapping interval: {}", interval);
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
// Count overlaps with a range (4, 11)
|
|
266
|
+
let overlap_count = intervals.count_overlaps(4, 11);
|
|
267
|
+
println!("Number of overlapping intervals: {}", overlap_count);
|
|
268
|
+
}
|
|
269
|
+
```
|
|
270
|
+
There is also `SuperIntervalsEytz` implementation. `SuperIntervalsEytz`
|
|
271
|
+
uses an Eytzinger memory layout that can sometimes offer faster query times at the cost of higher memory
|
|
272
|
+
usage and slower indexing time.
|
|
273
|
+
|
|
274
|
+
## Acknowledgements
|
|
275
|
+
|
|
276
|
+
- The rust test program borrows heavily from the coitrees package
|
|
277
|
+
- The superset-index implemented here exploits a similar interval ordering as described in
|
|
278
|
+
Schmidt 2009 "Interval Stabbing Problems in Small Integer Ranges". However, the superset-index has several advantages including
|
|
279
|
+
1. An implicit memory layout
|
|
280
|
+
1. General purpose implementation (not just small integer ranges)
|
|
281
|
+
1. SIMD counting algorithm
|
|
282
|
+
- The Eytzinger layout was adapted from Sergey Slotin, Algorithmica
|
|
@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
|
|
|
8
8
|
|
|
9
9
|
[project]
|
|
10
10
|
name = "superintervals"
|
|
11
|
-
version = "0.2.
|
|
11
|
+
version = "0.2.2"
|
|
12
12
|
description = "Rapid interval intersections"
|
|
13
13
|
dependencies = ['Cython']
|
|
14
|
-
authors = [{name = "Kez Cleal", email = "clealk@cardiff.ac.uk"}]
|
|
14
|
+
authors = [{name = "Kez Cleal", email = "clealk@cardiff.ac.uk"}]
|
|
@@ -209,15 +209,15 @@ class SuperIntervals {
|
|
|
209
209
|
virtual inline void upperBound(const S value) noexcept { // https://github.com/mh-dm/sb_lower_bound/blob/master/sbpm_lower_bound.h
|
|
210
210
|
size_t length = starts.size() - 1;
|
|
211
211
|
idx = 0;
|
|
212
|
-
constexpr int num_per_cache_line = 3 * hardware_constructive_interference_size;
|
|
213
|
-
while (length >= num_per_cache_line) {
|
|
214
|
-
size_t half = length / 2;
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
idx += (starts[idx + half] <= value) * (length - half);
|
|
219
|
-
length = half;
|
|
220
|
-
}
|
|
212
|
+
// constexpr int num_per_cache_line = 3 * hardware_constructive_interference_size;
|
|
213
|
+
// while (length >= num_per_cache_line) {
|
|
214
|
+
// size_t half = length / 2;
|
|
215
|
+
//// __builtin_prefetch(&starts[idx + half / 2]);
|
|
216
|
+
//// size_t first_half1 = idx + (length - half);
|
|
217
|
+
//// __builtin_prefetch(&starts[first_half1 + half / 2]);
|
|
218
|
+
// idx += (starts[idx + half] <= value) * (length - half);
|
|
219
|
+
// length = half;
|
|
220
|
+
// }
|
|
221
221
|
|
|
222
222
|
while (length > 0) {
|
|
223
223
|
size_t half = length / 2;
|
|
@@ -623,80 +623,3 @@ private:
|
|
|
623
623
|
return eytzinger_helper(arr, n, 0, 0);
|
|
624
624
|
}
|
|
625
625
|
};
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
template<typename S, typename T>
|
|
629
|
-
class SuperIntervalsDense : public SuperIntervals<S, T> {
|
|
630
|
-
public:
|
|
631
|
-
|
|
632
|
-
void index() override {
|
|
633
|
-
if (this->starts.size() == 0) {
|
|
634
|
-
return;
|
|
635
|
-
}
|
|
636
|
-
if (this->starts.size() == 1) {
|
|
637
|
-
dense.resize(1, 0);
|
|
638
|
-
}
|
|
639
|
-
this->starts.shrink_to_fit();
|
|
640
|
-
this->ends.shrink_to_fit();
|
|
641
|
-
this->data.shrink_to_fit();
|
|
642
|
-
this->sortIntervals();
|
|
643
|
-
|
|
644
|
-
// Could probably use a queue here to make dense vector in O(n) time
|
|
645
|
-
min_value = this->starts.front();
|
|
646
|
-
max_value = *std::max_element(this->ends.begin(), this->ends.end());
|
|
647
|
-
S max_size = max_value - min_value;
|
|
648
|
-
dense.resize((size_t)max_size, INT_MAX);
|
|
649
|
-
size_t index, end_index;
|
|
650
|
-
for (int i = this->starts.size() - 1; i >= 0; --i) {
|
|
651
|
-
index = (size_t)((this->starts[i] - min_value));
|
|
652
|
-
end_index = (size_t)(this->ends[i] - min_value);
|
|
653
|
-
for (size_t j=index; j < end_index + 1; ++j) {
|
|
654
|
-
if (dense[j] == INT_MAX) {
|
|
655
|
-
dense[j] = i;
|
|
656
|
-
}
|
|
657
|
-
}
|
|
658
|
-
++end_index;
|
|
659
|
-
while (end_index < dense.size() && dense[end_index] == INT_MAX) {
|
|
660
|
-
dense[end_index] = i;
|
|
661
|
-
++end_index;
|
|
662
|
-
}
|
|
663
|
-
}
|
|
664
|
-
|
|
665
|
-
this->branch.resize(this->starts.size(), SIZE_MAX);
|
|
666
|
-
std::vector<std::pair<S, size_t>> br;
|
|
667
|
-
br.reserve(1000);
|
|
668
|
-
br.emplace_back() = {this->ends[0], 0};
|
|
669
|
-
for (size_t i=1; i < this->ends.size(); ++i) {
|
|
670
|
-
while (!br.empty() && br.back().first < this->ends[i]) {
|
|
671
|
-
br.pop_back();
|
|
672
|
-
}
|
|
673
|
-
if (!br.empty()) {
|
|
674
|
-
this->branch[i] = br.back().second;
|
|
675
|
-
}
|
|
676
|
-
br.emplace_back() = {this->ends[i], i};
|
|
677
|
-
}
|
|
678
|
-
this->idx = 0;
|
|
679
|
-
}
|
|
680
|
-
|
|
681
|
-
inline void upperBound(const S x) noexcept override {
|
|
682
|
-
size_t i = 0;
|
|
683
|
-
if (min_value > x) {
|
|
684
|
-
this->idx = 0;
|
|
685
|
-
return;
|
|
686
|
-
} else if (x > max_value) {
|
|
687
|
-
this->idx = this->starts.size() - 1;
|
|
688
|
-
return;
|
|
689
|
-
}
|
|
690
|
-
size_t target_idx = (size_t)(x - min_value);
|
|
691
|
-
|
|
692
|
-
this->idx = (size_t)dense[target_idx];
|
|
693
|
-
if (this->idx > this->starts.size()) {
|
|
694
|
-
this->idx = 0;
|
|
695
|
-
}
|
|
696
|
-
}
|
|
697
|
-
|
|
698
|
-
private:
|
|
699
|
-
std::vector<uint32_t> dense;
|
|
700
|
-
|
|
701
|
-
S min_value, max_value;
|
|
702
|
-
};
|
superintervals-0.2.1/README.md
DELETED
|
@@ -1,279 +0,0 @@
|
|
|
1
|
-
SuperIntervals
|
|
2
|
-
==============
|
|
3
|
-
|
|
4
|
-
A fast, memory-efficient data structure for interval intersection queries.
|
|
5
|
-
SuperIntervals uses a novel superset-index approach that maintains
|
|
6
|
-
intervals in position-sorted order, enabling cache-friendly searches and SIMD-optimized counting.
|
|
7
|
-
|
|
8
|
-
### Features:
|
|
9
|
-
|
|
10
|
-
- Linear-time index construction from sorted intervals
|
|
11
|
-
- Cache-friendly querying
|
|
12
|
-
- SIMD acceleration (AVX2/Neon) for counting operations
|
|
13
|
-
- Minimal memory overhead (one size_t per interval)
|
|
14
|
-
- Available for C++, Rust, Python, and C
|
|
15
|
-
- Optional Eytzinger memory layout for slightly faster queries (C++/Rust only)
|
|
16
|
-
- No dependencies, header only
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
## Quick Start
|
|
20
|
-
|
|
21
|
-
- Intervals are considered end-inclusive
|
|
22
|
-
- The index() function must be called before any queries
|
|
23
|
-
- Found intervals are returned in reverse position-sorted order
|
|
24
|
-
|
|
25
|
-
### 🐍 Python
|
|
26
|
-
|
|
27
|
-
```python
|
|
28
|
-
from superintervals import IntervalSet
|
|
29
|
-
|
|
30
|
-
iset = IntervalSet()
|
|
31
|
-
iset.add(10, 20, 'A')
|
|
32
|
-
iset.index()
|
|
33
|
-
overlaps = iset.find_overlaps(8, 20)
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
### ⚙️ C++
|
|
37
|
-
```cpp
|
|
38
|
-
#include "SuperIntervals.hpp"
|
|
39
|
-
|
|
40
|
-
SuperIntervals<int, std::string> intervals;
|
|
41
|
-
intervals.add(1, 5, "A");
|
|
42
|
-
intervals.index();
|
|
43
|
-
std::vector<std::string> results;
|
|
44
|
-
intervals.findOverlaps(4, 9, results);
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
### 🦀 Rust
|
|
48
|
-
|
|
49
|
-
```rust
|
|
50
|
-
use super_intervals::SuperIntervals;
|
|
51
|
-
|
|
52
|
-
let mut intervals = SuperIntervals::new();
|
|
53
|
-
intervals.add(1, 5, "A");
|
|
54
|
-
intervals.index();
|
|
55
|
-
let mut results = Vec::new();
|
|
56
|
-
intervals.find_overlaps(4, 11, &mut results);
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
## Test programs
|
|
61
|
-
Test programs expect plain text BED files and only assess chr1 records - other chromosomes are ignored.
|
|
62
|
-
|
|
63
|
-
C++ program compares SuperIntervals, ImplicitIntervalTree, IntervalTree and NCLS:
|
|
64
|
-
```
|
|
65
|
-
cd test; make
|
|
66
|
-
./run-cpp-libs a.bed b.bed
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
Rust program:
|
|
70
|
-
```
|
|
71
|
-
RUSTFLAGS="-Ctarget-cpu=native" cargo run --release --example bed-intersect-si
|
|
72
|
-
cargo run --release --example bed-intersect-si a.bed b.bed
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
## Benchmark
|
|
76
|
-
|
|
77
|
-
SuperIntervals (SI) was compared with:
|
|
78
|
-
- Coitrees (Rust: https://github.com/dcjones/coitrees)
|
|
79
|
-
- Implicit Interval Tree (C++: https://github.com/lh3/cgranges)
|
|
80
|
-
- Interval Tree (C++: https://github.com/ekg/intervaltree)
|
|
81
|
-
- Nested Containment List (C: https://github.com/pyranges/ncls/tree/master/ncls/src)
|
|
82
|
-
|
|
83
|
-
Main results:
|
|
84
|
-
- Roughly ~2-3x faster than the next best library (Coitrees for Rust, Implicit Interval Tree for C++)
|
|
85
|
-
|
|
86
|
-
### Datasets:
|
|
87
|
-
1. Random regions generated using bedtools
|
|
88
|
-
2. RNA-seq reads and annotations from cgranges repository
|
|
89
|
-
3. ONT reads from sample PAO33946 (chr1, chrM)
|
|
90
|
-
4. Paired-end reads from sample DB53, NCBI BioProject PRJNA417592, (chr1, chrM)
|
|
91
|
-
5. UCSC genes from hg19
|
|
92
|
-
|
|
93
|
-
Test programs use internal timers and print data to stdout, measuring the
|
|
94
|
-
index time, and time to find all intersections. Other steps such as file IO are ignored. Test programs also
|
|
95
|
-
only assess chr1 bed records - other chromosomes are ignored. For 'chrM' records,
|
|
96
|
-
the M was replaced with 1 using sed. Data were assessed in position sorted and random order.
|
|
97
|
-
Datasets can be found on the Releases page, and the `test/run_tools.sh` script has instructions
|
|
98
|
-
for how to repeat the benchmark.
|
|
99
|
-
|
|
100
|
-
Timings were in microseconds using an i9-11900K, 64 GB, 2TB NVMe machine.
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
### 1. Finding interval intersections
|
|
104
|
-
|
|
105
|
-
- Coitrees-s uses the `SortedQuerent` version of coitrees
|
|
106
|
-
- SI = superintervals. Eytz refers to the eytzinger layout. `-rs` is the Rust implementation.
|
|
107
|
-
|
|
108
|
-
#### Intervals in sorted order
|
|
109
|
-
|
|
110
|
-
| | Coitrees | Coitrees-s | SI-rs | SI-rs | ImplicitITree-C++ | IntervalTree-C++ | NCLS-C | SI-C++ | SI-Eytz-C++ |
|
|
111
|
-
| --------------------- | -------- | ---------- |-------------|-----------| ----------------- | ---------------- | -------- |---------|-------------|
|
|
112
|
-
| DB53 reads, ONT reads | 1649.6 | 3169 | 732 | **729** | 3802.6 | 46393.8 | 10833.6 | 1391.6 | **1365.6** |
|
|
113
|
-
| DB53 reads, genes | 54.2 | 82.8 | **21** | **21** | 121.6 | 108 | 292.8 | 43 | **40.2** |
|
|
114
|
-
| ONT reads, DB53 reads | 6487.2 | 3437.2 | 534.6 | **533.6** | 18067.4 | 12448 | 31466.2 | 5333.2 | **4545.2** |
|
|
115
|
-
| anno, rna | 49.6 | 33.6 | 17.2 | **17** | 127.2 | 91.2 | 210.6 | 31.2 | **21.2** |
|
|
116
|
-
| genes, DB53 reads | 1171 | 992.8 | 270 | **269.2** | 3141 | 1339.8 | 1768 | 441.8 | **315** |
|
|
117
|
-
| mito-b, mito-a | 35046.2 | 35134 | **13115.2** | 13117.2 | 95137.4 | 108567.8 | 250671.8 | 33703.8 | **33298.6** |
|
|
118
|
-
| rna, anno | 31.8 | 22.6 | **4** | **4** | 71.2 | 54 | 238.8 | 29.4 | **27.2** |
|
|
119
|
-
|
|
120
|
-
#### Intervals in random order
|
|
121
|
-
|
|
122
|
-
| | Coitrees | Coitrees-s | SI-rs | SI-Eytz-rs | ImplicitITree-C++ | IntervalTree-C++ | NCLS-C | SI-C++ | SI-Eytz-C++ |
|
|
123
|
-
| --------------------- | -------- | ---------- |-----------|------------| ----------------- | ---------------- | -------- |------------|-------------|
|
|
124
|
-
| DB53 reads, ONT reads | 2939.6 | 4746.6 | 1323 | **1273** | 6654.6 | 46771.8 | 12082.4 | 2544.4 | **2180.2** |
|
|
125
|
-
| DB53 reads, genes | 75.2 | 131 | 26.6 | **26** | 168.2 | 122.8 | 308.2 | 56.4 | **51.4** |
|
|
126
|
-
| ONT reads, DB53 reads | 17100.6 | 19309.2 | 3815 | **3714.6** | 40490.8 | 28633.2 | 55317.6 | 24047 | **23664** |
|
|
127
|
-
| anno, rna | 89.6 | 110 | 42.2 | **41.8** | 188.8 | 150.2 | 299.4 | **58** | **58** |
|
|
128
|
-
| genes, DB53 reads | 2217.6 | 2448.8 | 1343.8 | **1331.6** | 4495.8 | 2747.2 | 3632.2 | **1265.2** | 1730.8 |
|
|
129
|
-
| mito-b, mito-a | 39002.8 | 88901.8 | **13540** | 13541.8 | 128507.2 | 120712 | 261409.2 | 43682 | **42576.8** |
|
|
130
|
-
| rna, anno | 51 | 69.2 | 12 | **11.8** | 140.4 | 84.4 | 323.8 | 54.2 | **53** |
|
|
131
|
-
|
|
132
|
-
### 2. Counting interval intersections
|
|
133
|
-
|
|
134
|
-
#### Intervals in sorted order
|
|
135
|
-
|
|
136
|
-
| | Coitrees | SI-rs | SI-Eytz-rs | SI-C++ | SI-Eytz-C++ |
|
|
137
|
-
| --------------------- | -------- |-----------|------------|-----------|-------------|
|
|
138
|
-
| DB53 reads, ONT reads | 551.4 | 337.6 | 338 | **239.4** | 265 |
|
|
139
|
-
| DB53 reads, genes | 26 | 10.6 | 10.8 | 8 | **7** |
|
|
140
|
-
| ONT reads, DB53 reads | 2517.2 | **795.4** | 796.6 | 2234.2 | 1414.2 |
|
|
141
|
-
| anno, rna | 26.8 | 13.4 | 13.2 | 22.6 | **12** |
|
|
142
|
-
| genes, DB53 reads | 737.4 | **292.6** | 294.6 | 459.6 | 338.2 |
|
|
143
|
-
| mito-b, mito-a | 7030 | 6634.6 | 6633.4 | 3065.6 | **2991.8** |
|
|
144
|
-
| rna, anno | 9 | **4** | **4** | 12 | 10 |
|
|
145
|
-
|
|
146
|
-
#### Intervals in random order
|
|
147
|
-
|
|
148
|
-
| | Coitrees | SI-rs | SI-Eytz-rs | SI-C++ | SI-Eytz-C++ |
|
|
149
|
-
| --------------------- | -------- | ------ | ---------- | ------ | ----------- |
|
|
150
|
-
| DB53 reads, ONT reads | 1990 | 937.2 | 883.4 | 1018.8 | **789.6** |
|
|
151
|
-
| DB53 reads, genes | 49.2 | 16 | 15 | 15.2 | **13.4** |
|
|
152
|
-
| ONT reads, DB53 reads | 6835 | 4037.8 | **3964.4** | 8547.8 | 10153.8 |
|
|
153
|
-
| anno, rna | 52 | 39 | **38.6** | 47 | 46 |
|
|
154
|
-
| genes, DB53 reads | 1523.6 | 1261 | 1269 | **1119.4** | 1519.6 |
|
|
155
|
-
| mito-b, mito-a | 15001.2 | 7290.6 | 7298.4 | 4493.6 | **4452.4** |
|
|
156
|
-
| rna, anno | 22 | **12** | **12** | 25.2 | 25.4 |
|
|
157
|
-
|
|
158
|
-
## Python
|
|
159
|
-
|
|
160
|
-
Install using `pip install .`
|
|
161
|
-
|
|
162
|
-
```
|
|
163
|
-
from superintervals import IntervalSet
|
|
164
|
-
|
|
165
|
-
iset = IntervalSet()
|
|
166
|
-
|
|
167
|
-
# Add interval start, end, identifier. Integer values are supported
|
|
168
|
-
iset.add(10, 20, 0)
|
|
169
|
-
iset.add(19, 18, 1)
|
|
170
|
-
iset.add(8, 11, 2)
|
|
171
|
-
|
|
172
|
-
# Index method must be called before queries
|
|
173
|
-
iset.index()
|
|
174
|
-
|
|
175
|
-
iset.any_overlaps(8, 20)
|
|
176
|
-
# >>> True
|
|
177
|
-
|
|
178
|
-
iset.count_overlaps(8, 20)
|
|
179
|
-
# >>> 3
|
|
180
|
-
|
|
181
|
-
iset.find_overlaps(8, 20)
|
|
182
|
-
# >>> [1, 0, 2]
|
|
183
|
-
|
|
184
|
-
iset.set_search_interval(8, 20)
|
|
185
|
-
for itv in iset:
|
|
186
|
-
print(itv)
|
|
187
|
-
|
|
188
|
-
# >>> (19, 18, 1)
|
|
189
|
-
# >>> (10, 20, 0)
|
|
190
|
-
# >>> (8, 11, 2)
|
|
191
|
-
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
## Cpp
|
|
195
|
-
|
|
196
|
-
```cpp
|
|
197
|
-
#include <iostream>
|
|
198
|
-
#include <vector>
|
|
199
|
-
#include "SuperIntervals.hpp"
|
|
200
|
-
|
|
201
|
-
int main() {
|
|
202
|
-
// Create a SuperIntervals instance for integer intervals with string data
|
|
203
|
-
// Specify with S, T template types
|
|
204
|
-
SuperIntervals<int, std::string> intervals;
|
|
205
|
-
|
|
206
|
-
// Add some intervals
|
|
207
|
-
intervals.add(1, 5, "Interval A");
|
|
208
|
-
intervals.add(3, 7, "Interval B");
|
|
209
|
-
intervals.add(6, 10, "Interval C");
|
|
210
|
-
intervals.add(8, 12, "Interval D");
|
|
211
|
-
|
|
212
|
-
// Index the intervals (must be called before querying)
|
|
213
|
-
intervals.index();
|
|
214
|
-
|
|
215
|
-
// Find overlaps for the range [4, 9]
|
|
216
|
-
std::vector<std::string> overlaps;
|
|
217
|
-
intervals.findOverlaps(4, 9, overlaps);
|
|
218
|
-
|
|
219
|
-
// Print the overlapping intervals
|
|
220
|
-
for (const auto& interval : overlaps) {
|
|
221
|
-
std::cout << interval << std::endl;
|
|
222
|
-
}
|
|
223
|
-
|
|
224
|
-
// Count the intervals instead
|
|
225
|
-
std::cout << "Count: " << intervals.countOverlaps(4, 9) << std::endl;
|
|
226
|
-
|
|
227
|
-
// Count stabbed intervals at point 7
|
|
228
|
-
std::cout << "Number of intervals containing point 7: " << intervals.countStabbed(7) << std::endl;
|
|
229
|
-
|
|
230
|
-
return 0;
|
|
231
|
-
}
|
|
232
|
-
```
|
|
233
|
-
There is also a `SuperIntervalsEytz` subclasses that can be used. `SuperIntervalsEytz`
|
|
234
|
-
uses an Eytzinger memory layout that can sometimes offer faster query times at the cost of higher memory
|
|
235
|
-
usage and slower indexing time.
|
|
236
|
-
|
|
237
|
-
## Rust
|
|
238
|
-
|
|
239
|
-
```
|
|
240
|
-
use super_intervals::SuperIntervals;
|
|
241
|
-
|
|
242
|
-
fn main() {
|
|
243
|
-
// Create a new instance of SuperIntervals
|
|
244
|
-
let mut intervals = SuperIntervals::new();
|
|
245
|
-
|
|
246
|
-
// Add some intervals with associated data of type T
|
|
247
|
-
intervals.add(1, 5, "Interval A");
|
|
248
|
-
intervals.add(10, 15, "Interval B");
|
|
249
|
-
intervals.add(7, 12, "Interval C");
|
|
250
|
-
|
|
251
|
-
// Call index() to prepare the intervals for queries
|
|
252
|
-
intervals.index();
|
|
253
|
-
|
|
254
|
-
// Query for overlapping intervals with a range (4, 11)
|
|
255
|
-
let mut found_intervals = Vec::new();
|
|
256
|
-
intervals.find_overlaps(4, 11, &mut found_intervals);
|
|
257
|
-
|
|
258
|
-
// Display found intervals
|
|
259
|
-
for interval in found_intervals {
|
|
260
|
-
println!("Found overlapping interval: {}", interval);
|
|
261
|
-
}
|
|
262
|
-
|
|
263
|
-
// Count overlaps with a range (4, 11)
|
|
264
|
-
let overlap_count = intervals.count_overlaps(4, 11);
|
|
265
|
-
println!("Number of overlapping intervals: {}", overlap_count);
|
|
266
|
-
}
|
|
267
|
-
```
|
|
268
|
-
There is also `SuperIntervalsEytz` implementation. `SuperIntervalsEytz`
|
|
269
|
-
uses an Eytzinger memory layout that can sometimes offer faster query times at the cost of higher memory
|
|
270
|
-
usage and slower indexing time.
|
|
271
|
-
|
|
272
|
-
## Acknowledgements
|
|
273
|
-
|
|
274
|
-
- The rust test program borrows heavily from the coitrees package
|
|
275
|
-
- The superset-index implemented here exploits a similar interval ordering as described in
|
|
276
|
-
Schmidt 2009 "Interval Stabbing Problems in Small Integer Ranges". However, the superset-index has several advantages including
|
|
277
|
-
1. An implicit memory layout
|
|
278
|
-
1. General purpose implementation (not just small integer ranges)
|
|
279
|
-
1. SIMD counting algorithm
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{superintervals-0.2.1 → superintervals-0.2.2}/src/superintervals.egg-info/dependency_links.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|