usearch 2.4.0 → 2.5.0

package/docs/format.md

@@ -1,48 +1,71 @@
-# USearch Format Specification
+# USearch File Format Specification
 
-> ⚠️ **Note**: The format is expected to change in the 3.0 release to support sparse indexes and offer even faster startup times when `view`-ing large indexes from disk.
+> ⚠️ __Important Note__: The file format is slated for updates in the upcoming 3.0 release. These changes aim to support sparse indexes and enhance startup performance when loading large indexes from disk using the `view` command.
 
-## Table of Contents
+## Upcoming Version: v3
 
-- [Dense Indexes](#dense-indexes)
-- [Core](#core)
-- [Fragmented Storage](#fragmented-storage)
+The v3 file format is designed for compatibility with Apache Arrow arrays and consists of an array of variable-length binary strings. The structure for a file with `N` entries is as follows:
 
-## Dense Indexes
+### File Structure
+
+1. __File Header__: Metadata and other essential information.
+2. __Offset Array__: An array of `N+1` eight-byte unsigned integers (`uint64_t`).
+    - The entry at index `0 <= i < N` indicates the byte-level offset of the `i-th` vector in the file.
+    - The entry at index `N` specifies the total length of the binary file.
+3. __Data Chunks__: `N` binary data chunks, each starting with the vector itself, followed by index system data.
+
+### Data Retrieval
+
+To retrieve the Binary Large Object (BLOB) corresponding to a specific vector, subtract the consecutive offsets indicated in the Offset Array.
+
+### Advantages of the v3 Format
+
+The v3 format offers several benefits, making it more future-proof:
+
+- __Flexibility__: Easily accommodates variable-length strings.
+- __Performance__: Co-locates vectors with proximity graph entries for optimized data access.
+- __Compatibility__: Simplifies memory-mapping and casting to an Apache Arrow array of vectors or strings.
+- __Extensibility__: Enables external storage systems to reuse the underlying `index_gt` more extensively, particularly in database systems.
+
+---
+
+## Current: v2
+
+### Dense Indexes
 
 Dense index files consist of two main parts: [a binary matrix of all vectors](#matrix-blob) and the [index](#index-blob) itself.
 
-### Matrix BLOB
+#### Matrix BLOB
 
 The matrix BLOB is optionally prepended to the index file. It starts with the number of `rows` and `columns`, which can be either 32-bit or 64-bit unsigned integers.
 
 - The number of `rows` corresponds to the number of vectors in the index.
 - The number of `columns` corresponds to the number of bytes in a single vector.
 
-### Index BLOB
+#### Index BLOB
 
 The index BLOB is composed of three main sections:
 
-1. **Metadata**: Occupies 64 bytes and contains the following fields:
+1. __Metadata__: Occupies 64 bytes and contains the following fields:
     1. 7-byte magic string `usearch` to infer the MIME type of the file.
     2. 3-byte version number, containing the `major`, `minor`, and `patch` release numbers.
     3. 1-byte enums for `metric`, `scalar`, `key`, and `compressed_slot` types.
     4. 8-byte integers for the number of present vectors, number of deleted vectors, and number of dimensions.
     5. 1-byte flags for multi-vector support.
 
-2. **Levels**: Contains 1-byte integers representing the level each node occupies. This information is used to estimate memory requirements and to enable memory-mapping of the entire file.
+2. __Levels__: Contains 1-byte integers representing the level each node occupies. This information is used to estimate memory requirements and to enable memory-mapping of the entire file.
 
-3. **Core**: Detailed in the [Core section](#core). This format is consistent across all types of indexes.
+3. __Core__: Detailed in the [Core section](#core). This format is consistent across all types of indexes.
 
-## Fragmented Storage
+### Fragmented Storage
 
 The matrix BLOB can be stored separately and is not required to successfully open the file.
 
-## Core
+### Core
 
 The core section of the index is serialized as a sequence of nodes, each representing a point in the index. This section immediately follows the `levels` block in the serialized format.
 
-### Header
+#### Header
 
 The core section begins with a header containing the following fields:
 
@@ -54,10 +77,10 @@ The core section begins with a header containing the following fields:
 
 These fields are serialized in a contiguous block of bytes, in the order listed.
 
-### Levels
+#### Levels
 
 Following the header, the `levels` block contains a sequence of 1-byte integers. Each integer represents the level of a node in the index. The number of levels is equal to the `size` field in the header.
 
-### Nodes
+#### Nodes
 
 After the `levels` block, the nodes are serialized. Each node is represented as a contiguous block of bytes. The size and format of these blocks are implementation-specific. Nodes are serialized in the same order as their corresponding levels in the `levels` block.

package/include/usearch/index.hpp

@@ -10,7 +10,7 @@
 #define UNUM_USEARCH_HPP
 
 #define USEARCH_VERSION_MAJOR 2
-#define USEARCH_VERSION_MINOR 4
+#define USEARCH_VERSION_MINOR 5
 #define USEARCH_VERSION_PATCH 0
 
 // Inferring C++ version

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "usearch",
-    "version": "2.4.0",
+    "version": "2.5.0",
     "description": "Smaller & Faster Single-File Vector Search Engine from Unum",
     "author": "Ash Vardanian",
     "license": "Apache 2.0",
@@ -25,7 +25,7 @@
         "@semantic-release/exec": "^6.0.3",
         "@semantic-release/git": "^10.0.1",
         "conventional-changelog-eslint": "^3.0.9",
-        "semantic-release": "^21.1.1",
+        "semantic-release": "^21.1.2",
         "typescript": "^5.1.6"
     }
 }

package/python/lib.cpp

@@ -608,9 +608,9 @@ static py::tuple search_many_brute_force(    //
     if (dataset_kind != queries_kind)
         throw std::invalid_argument("The types of vectors don't match!");
 
-    py::array_t<dense_key_t> keys_py({dataset_count, static_cast<Py_ssize_t>(wanted)});
-    py::array_t<distance_t> distances_py({dataset_count, static_cast<Py_ssize_t>(wanted)});
-    py::array_t<Py_ssize_t> counts_py(dataset_count);
+    py::array_t<dense_key_t> keys_py({queries_count, static_cast<Py_ssize_t>(wanted)});
+    py::array_t<distance_t> distances_py({queries_count, static_cast<Py_ssize_t>(wanted)});
+    py::array_t<Py_ssize_t> counts_py(queries_count);
 
     std::size_t dimensions = static_cast<std::size_t>(queries_dimensions);
     metric_t metric =  //

package/python/usearch/index.py

@@ -1200,7 +1200,7 @@ class Index:
     def __repr__(self) -> str:
         if not hasattr(self, "_compiled"):
             return "usearch.Index(failed)"
-        f = "usearch.Index({} x {}, {}, connectivity: {}, expansion: {} & {}, {:,} vectors in {} levels, {} hardware accerlation)"
+        f = "usearch.Index({} x {}, {}, connectivity: {}, expansion: {} & {}, {:,} vectors in {} levels, {} hardware acceleration)"
         return f.format(
             self.dtype,
             self.ndim,

package/rust/lib.cpp

@@ -23,65 +23,71 @@ template <typename scalar_at> Matches search_(index_dense_t& index, scalar_at co
     return matches;
 }
 
-Index::Index(std::unique_ptr<index_t> index) : index_(std::move(index)) {}
+NativeIndex::NativeIndex(std::unique_ptr<index_t> index) : index_(std::move(index)) {}
 
 // clang-format off
-void Index::add(key_t key, rust::Slice<float const> vec) const { index_->add(key, vec.data()).error.raise(); }
-void Index::add_i8(key_t key, rust::Slice<int8_t const> vec) const { index_->add(key, vec.data()).error.raise(); }
-void Index::add_f16(key_t key, rust::Slice<uint16_t const> vec) const { index_->add(key, (f16_t const*)vec.data()).error.raise(); }
-void Index::add_f32(key_t key, rust::Slice<float const> vec) const { index_->add(key, vec.data()).error.raise(); }
-void Index::add_f64(key_t key, rust::Slice<double const> vec) const { index_->add(key, vec.data()).error.raise(); }
-
-Matches Index::search(rust::Slice<float const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
-Matches Index::search_i8(rust::Slice<int8_t const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
-Matches Index::search_f16(rust::Slice<uint16_t const> vec, size_t count) const { return search_(*index_, (f16_t const*)vec.data(), count); }
-Matches Index::search_f32(rust::Slice<float const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
-Matches Index::search_f64(rust::Slice<double const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
+void NativeIndex::add_i8(key_t key, rust::Slice<int8_t const> vec) const { index_->add(key, vec.data()).error.raise(); }
+void NativeIndex::add_f16(key_t key, rust::Slice<uint16_t const> vec) const { index_->add(key, (f16_t const*)vec.data()).error.raise(); }
+void NativeIndex::add_f32(key_t key, rust::Slice<float const> vec) const { index_->add(key, vec.data()).error.raise(); }
+void NativeIndex::add_f64(key_t key, rust::Slice<double const> vec) const { index_->add(key, vec.data()).error.raise(); }
+
+Matches NativeIndex::search_i8(rust::Slice<int8_t const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
+Matches NativeIndex::search_f16(rust::Slice<uint16_t const> vec, size_t count) const { return search_(*index_, (f16_t const*)vec.data(), count); }
+Matches NativeIndex::search_f32(rust::Slice<float const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
+Matches NativeIndex::search_f64(rust::Slice<double const> vec, size_t count) const { return search_(*index_, vec.data(), count); }
+
+size_t NativeIndex::get_i8(key_t key, rust::Slice<int8_t> vec) const { return index_->get(key, vec.data(), vec.size() / dimensions()); }
+size_t NativeIndex::get_f16(key_t key, rust::Slice<uint16_t> vec) const { return index_->get(key, (f16_t*)vec.data(), vec.size() / dimensions()); }
+size_t NativeIndex::get_f32(key_t key, rust::Slice<float> vec) const { return index_->get(key, vec.data(), vec.size() / dimensions()); }
+size_t NativeIndex::get_f64(key_t key, rust::Slice<double> vec) const { return index_->get(key, vec.data(), vec.size() / dimensions()); }
 // clang-format on
 
-bool Index::remove(key_t key) const {
+size_t NativeIndex::remove(key_t key) const {
     labeling_result_t result = index_->remove(key);
     result.error.raise();
     return result.completed;
 }
 
-bool Index::rename(key_t from, key_t to) const {
+size_t NativeIndex::rename(key_t from, key_t to) const {
     labeling_result_t result = index_->rename(from, to);
     result.error.raise();
     return result.completed;
 }
 
-bool Index::contains(key_t key) const { return index_->contains(key); }
+size_t NativeIndex::count(key_t key) const { return index_->count(key); }
+bool NativeIndex::contains(key_t key) const { return index_->contains(key); }
 
-void Index::reserve(size_t capacity) const { index_->reserve(capacity); }
+void NativeIndex::reserve(size_t capacity) const { index_->reserve(capacity); }
 
-size_t Index::dimensions() const { return index_->dimensions(); }
-size_t Index::connectivity() const { return index_->connectivity(); }
-size_t Index::size() const { return index_->size(); }
-size_t Index::capacity() const { return index_->capacity(); }
-size_t Index::serialized_length() const { return index_->serialized_length(); }
+size_t NativeIndex::dimensions() const { return index_->dimensions(); }
+size_t NativeIndex::connectivity() const { return index_->connectivity(); }
+size_t NativeIndex::size() const { return index_->size(); }
+size_t NativeIndex::capacity() const { return index_->capacity(); }
+size_t NativeIndex::serialized_length() const { return index_->serialized_length(); }
 
-void Index::save(rust::Str path) const { index_->save(output_file_t(std::string(path).c_str())).error.raise(); }
-void Index::load(rust::Str path) const { index_->load(input_file_t(std::string(path).c_str())).error.raise(); }
-void Index::view(rust::Str path) const { index_->view(memory_mapped_file_t(std::string(path).c_str())).error.raise(); }
+void NativeIndex::save(rust::Str path) const { index_->save(output_file_t(std::string(path).c_str())).error.raise(); }
+void NativeIndex::load(rust::Str path) const { index_->load(input_file_t(std::string(path).c_str())).error.raise(); }
+void NativeIndex::view(rust::Str path) const {
+    index_->view(memory_mapped_file_t(std::string(path).c_str())).error.raise();
+}
 
-void Index::save_to_buffer(rust::Slice<uint8_t> buffer) const {
+void NativeIndex::save_to_buffer(rust::Slice<uint8_t> buffer) const {
     index_->save(memory_mapped_file_t((byte_t*)buffer.data(), buffer.size())).error.raise();
 }
 
-void Index::load_from_buffer(rust::Slice<uint8_t const> buffer) const {
+void NativeIndex::load_from_buffer(rust::Slice<uint8_t const> buffer) const {
     index_->load(memory_mapped_file_t((byte_t*)buffer.data(), buffer.size())).error.raise();
 }
 
-void Index::view_from_buffer(rust::Slice<uint8_t const> buffer) const {
+void NativeIndex::view_from_buffer(rust::Slice<uint8_t const> buffer) const {
     index_->view(memory_mapped_file_t((byte_t*)buffer.data(), buffer.size())).error.raise();
 }
 
-std::unique_ptr<Index> wrap(index_t&& index) {
+std::unique_ptr<NativeIndex> wrap(index_t&& index) {
     std::unique_ptr<index_t> punned_ptr;
     punned_ptr.reset(new index_t(std::move(index)));
-    std::unique_ptr<Index> result;
-    result.reset(new Index(std::move(punned_ptr)));
+    std::unique_ptr<NativeIndex> result;
+    result.reset(new NativeIndex(std::move(punned_ptr)));
     return result;
 }
 
@@ -110,10 +116,11 @@ scalar_kind_t rust_to_cpp_scalar(ScalarKind value) {
     }
 }
 
-std::unique_ptr<Index> new_index(IndexOptions const& options) {
+std::unique_ptr<NativeIndex> new_native_index(IndexOptions const& options) {
     metric_kind_t metric_kind = rust_to_cpp_metric(options.metric);
     scalar_kind_t scalar_kind = rust_to_cpp_scalar(options.quantization);
     metric_punned_t metric(options.dimensions, metric_kind, scalar_kind);
     index_dense_config_t config(options.connectivity, options.expansion_add, options.expansion_search);
+    config.multi = options.multi;
     return wrap(index_t::make(metric, config));
 }

package/rust/lib.hpp

@@ -8,7 +8,7 @@
 struct Matches;
 struct IndexOptions;
 
-class Index {
+class NativeIndex {
   public:
     using metric_t = unum::usearch::metric_punned_t;
     using distance_t = unum::usearch::distance_punned_t;
@@ -17,25 +17,29 @@ class Index {
     using search_result_t = typename index_dense_t::search_result_t;
     using key_t = typename index_dense_t::key_t;
 
-    Index(std::unique_ptr<index_dense_t> index);
+    NativeIndex(std::unique_ptr<index_dense_t> index);
 
     void reserve(size_t) const;
 
-    void add(key_t key, rust::Slice<float const> vector) const;
     void add_i8(key_t key, rust::Slice<int8_t const> vector) const;
     void add_f16(key_t key, rust::Slice<uint16_t const> vector) const;
     void add_f32(key_t key, rust::Slice<float const> vector) const;
     void add_f64(key_t key, rust::Slice<double const> vector) const;
 
-    Matches search(rust::Slice<float const> vector, size_t count) const;
     Matches search_i8(rust::Slice<int8_t const> vector, size_t count) const;
     Matches search_f16(rust::Slice<uint16_t const> vector, size_t count) const;
     Matches search_f32(rust::Slice<float const> vector, size_t count) const;
     Matches search_f64(rust::Slice<double const> vector, size_t count) const;
 
-    bool remove(key_t key) const;
+    size_t get_i8(key_t key, rust::Slice<int8_t> vector) const;
+    size_t get_f16(key_t key, rust::Slice<uint16_t> vector) const;
+    size_t get_f32(key_t key, rust::Slice<float> vector) const;
+    size_t get_f64(key_t key, rust::Slice<double> vector) const;
+
+    size_t count(key_t key) const;
+    size_t remove(key_t key) const;
+    size_t rename(key_t from, key_t to) const;
     bool contains(key_t key) const;
-    bool rename(key_t from, key_t to) const;
 
     size_t dimensions() const;
     size_t connectivity() const;
@@ -55,4 +59,4 @@ class Index {
     std::unique_ptr<index_dense_t> index_;
 };
 
-std::unique_ptr<Index> new_index(IndexOptions const& options);
+std::unique_ptr<NativeIndex> new_native_index(IndexOptions const& options);