usearch 2.4.0 → 2.4.1

package/docs/format.md

@@ -1,48 +1,71 @@
-# USearch Format Specification
+# USearch File Format Specification
 
-> ⚠️ **Note**: The format is expected to change in the 3.0 release to support sparse indexes and offer even faster startup times when `view`-ing large indexes from disk.
+> ⚠️ __Important Note__: The file format is slated for updates in the upcoming 3.0 release. These changes aim to support sparse indexes and enhance startup performance when loading large indexes from disk using the `view` command.
 
-## Table of Contents
+## Upcoming Version: v3
 
-- [Dense Indexes](#dense-indexes)
-- [Core](#core)
-- [Fragmented Storage](#fragmented-storage)
+The v3 file format is designed for compatibility with Apache Arrow arrays and consists of an array of variable-length binary strings. The structure for a file with `N` entries is as follows:
 
-## Dense Indexes
+### File Structure
+
+1. __File Header__: Metadata and other essential information.
+2. __Offset Array__: An array of `N+1` eight-byte unsigned integers (`uint64_t`).
+    - The entry at index `0 <= i < N` indicates the byte-level offset of the `i-th` vector in the file.
+    - The entry at index `N` specifies the total length of the binary file.
+3. __Data Chunks__: `N` binary data chunks, each starting with the vector itself, followed by index system data.
+
+### Data Retrieval
+
+To retrieve the Binary Large Object (BLOB) corresponding to a specific vector, subtract the consecutive offsets indicated in the Offset Array.
+
+### Advantages of the v3 Format
+
+The v3 format offers several benefits, making it more future-proof:
+
+- __Flexibility__: Easily accommodates variable-length strings.
+- __Performance__: Co-locates vectors with proximity graph entries for optimized data access.
+- __Compatibility__: Simplifies memory-mapping and casting to an Apache Arrow array of vectors or strings.
+- __Extensibility__: Enables external storage systems to reuse the underlying `index_gt` more extensively, particularly in database systems.
+
+---
+
+## Current: v2
+
+### Dense Indexes
 
 Dense index files consist of two main parts: [a binary matrix of all vectors](#matrix-blob) and the [index](#index-blob) itself.
 
-### Matrix BLOB
+#### Matrix BLOB
 
 The matrix BLOB is optionally prepended to the index file. It starts with the number of `rows` and `columns`, which can be either 32-bit or 64-bit unsigned integers.
 
 - The number of `rows` corresponds to the number of vectors in the index.
 - The number of `columns` corresponds to the number of bytes in a single vector.
 
-### Index BLOB
+#### Index BLOB
 
 The index BLOB is composed of three main sections:
 
-1. **Metadata**: Occupies 64 bytes and contains the following fields:
+1. __Metadata__: Occupies 64 bytes and contains the following fields:
     1. 7-byte magic string `usearch` to infer the MIME type of the file.
     2. 3-byte version number, containing the `major`, `minor`, and `patch` release numbers.
     3. 1-byte enums for `metric`, `scalar`, `key`, and `compressed_slot` types.
     4. 8-byte integers for the number of present vectors, number of deleted vectors, and number of dimensions.
     5. 1-byte flags for multi-vector support.
 
-2. **Levels**: Contains 1-byte integers representing the level each node occupies. This information is used to estimate memory requirements and to enable memory-mapping of the entire file.
+2. __Levels__: Contains 1-byte integers representing the level each node occupies. This information is used to estimate memory requirements and to enable memory-mapping of the entire file.
 
-3. **Core**: Detailed in the [Core section](#core). This format is consistent across all types of indexes.
+3. __Core__: Detailed in the [Core section](#core). This format is consistent across all types of indexes.
 
-## Fragmented Storage
+### Fragmented Storage
 
 The matrix BLOB can be stored separately and is not required to successfully open the file.
 
-## Core
+### Core
 
 The core section of the index is serialized as a sequence of nodes, each representing a point in the index. This section immediately follows the `levels` block in the serialized format.
 
-### Header
+#### Header
 
 The core section begins with a header containing the following fields:
 
@@ -54,10 +77,10 @@ The core section begins with a header containing the following fields:
 
 These fields are serialized in a contiguous block of bytes, in the order listed.
 
-### Levels
+#### Levels
 
 Following the header, the `levels` block contains a sequence of 1-byte integers. Each integer represents the level of a node in the index. The number of levels is equal to the `size` field in the header.
 
-### Nodes
+#### Nodes
 
 After the `levels` block, the nodes are serialized. Each node is represented as a contiguous block of bytes. The size and format of these blocks are implementation-specific. Nodes are serialized in the same order as their corresponding levels in the `levels` block.

package/include/usearch/index.hpp

@@ -11,7 +11,7 @@
 
 #define USEARCH_VERSION_MAJOR 2
 #define USEARCH_VERSION_MINOR 4
-#define USEARCH_VERSION_PATCH 0
+#define USEARCH_VERSION_PATCH 1
 
 // Inferring C++ version
 // https://stackoverflow.com/a/61552074

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "usearch",
-    "version": "2.4.0",
+    "version": "2.4.1",
     "description": "Smaller & Faster Single-File Vector Search Engine from Unum",
     "author": "Ash Vardanian",
     "license": "Apache 2.0",
@@ -25,7 +25,7 @@
         "@semantic-release/exec": "^6.0.3",
         "@semantic-release/git": "^10.0.1",
         "conventional-changelog-eslint": "^3.0.9",
-        "semantic-release": "^21.1.1",
+        "semantic-release": "^21.1.2",
         "typescript": "^5.1.6"
     }
 }

package/python/lib.cpp

@@ -608,9 +608,9 @@ static py::tuple search_many_brute_force(    //
     if (dataset_kind != queries_kind)
         throw std::invalid_argument("The types of vectors don't match!");
 
-    py::array_t<dense_key_t> keys_py({dataset_count, static_cast<Py_ssize_t>(wanted)});
-    py::array_t<distance_t> distances_py({dataset_count, static_cast<Py_ssize_t>(wanted)});
-    py::array_t<Py_ssize_t> counts_py(dataset_count);
+    py::array_t<dense_key_t> keys_py({queries_count, static_cast<Py_ssize_t>(wanted)});
+    py::array_t<distance_t> distances_py({queries_count, static_cast<Py_ssize_t>(wanted)});
+    py::array_t<Py_ssize_t> counts_py(queries_count);
 
     std::size_t dimensions = static_cast<std::size_t>(queries_dimensions);
     metric_t metric =  //

package/python/usearch/index.py

@@ -1200,7 +1200,7 @@ class Index:
     def __repr__(self) -> str:
         if not hasattr(self, "_compiled"):
             return "usearch.Index(failed)"
-        f = "usearch.Index({} x {}, {}, connectivity: {}, expansion: {} & {}, {:,} vectors in {} levels, {} hardware accerlation)"
+        f = "usearch.Index({} x {}, {}, connectivity: {}, expansion: {} & {}, {:,} vectors in {} levels, {} hardware acceleration)"
         return f.format(
             self.dtype,
             self.ndim,

package/wasmer.toml

@@ -1,6 +1,6 @@
 [package]
 name="unum/usearch"
-version="2.4.0"
+version="2.4.1"
 description="Smaller & Faster Single-File Vector Search Engine from Unum"
 license="Apache-2.0"
 readme="README.md"

package/csharp/src/Cloud.Unum.USearch.Tests/InitTests.cs

@@ -1,306 +0,0 @@
-using Cloud.Unum.USearch;
-
-namespace Cloud.Unum.USearch.Tests;
-
-public class InitTests
-{
-    // Epsilon for real values comparisons
-    private const float Epsilon = 0.000001f;
-
-    [Fact]
-    public void InitializedAndDisposedSucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3))
-        {
-            Assert.NotNull(index);
-            Assert.Equal((uint)3, index.Dimensions());
-        }
-    }
-
-    [Fact]
-    public void AddDoubleVectorSucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float64, 3))
-        {
-            index.Add(1, new double[] { 1.0, 2.0, 3.0 });
-            Assert.True(index.Contains(1));
-            Assert.False(index.Contains(2));
-            Assert.Equal((uint)1, index.Size());
-        }
-    }
-
-    [Fact]
-    public void AddDoubleVectorsUnderSameKeySucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float64, 3, multi: true))
-        {
-            index.Add(1, new double[] { 1.0, 2.0, 3.0 });
-            index.Add(2, new double[] { 1.0, 2.0, 3.0 });
-            index.Add(3, new double[] { 1.0, 2.0, 3.0 });
-            index.Add(1, new double[] { 1.0, 2.0, 3.0 });
-            index.Add(2, new double[] { 1.0, 2.0, 3.0 });
-            Assert.True(index.Contains(1));
-            Assert.True(index.Contains(2));
-            Assert.True(index.Contains(3));
-            Assert.Equal((uint)5, index.Size());
-        }
-    }
-
-    [Fact]
-    public void ItCanGetMultipleFloatVectorsForSingleKey()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3, multi: true))
-        {
-            var inputVectors = new float[][] {
-                    new float[] { 1.0f, 6.0f, 11.0f },
-                    new float[] { 2.0f, 7.0f, 12.0f },
-                    new float[] { 3.0f, 8.0f, 13.0f },
-                    new float[] { 5.0f, 10.0f, 15.0f }
-            };
-            index.Add(
-                new ulong[] { 2, 2, 2, 2 },
-                inputVectors
-            );
-            Assert.True(index.Contains(2));
-            Assert.Equal((uint)4, index.Size());
-            Assert.Equal(4, index.Count(2));
-
-            int foundVectorsCount = index.Get(2, 3, out float[][] retrievedVectors);
-            Assert.Equal(3, foundVectorsCount);
-        }
-    }
-
-    [Fact]
-    public void ItCanGetMultipleDoubleVectorsForSingleKey()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float64, 3, multi: true))
-        {
-            var inputVectors = new double[][] {
-                new double[] { 1.0, 6.0, 11.0 },
-                new double[] { 2.0, 7.0, 12.0 },
-                new double[] { 3.0, 8.0, 13.0 },
-                new double[] { 5.0, 10.0, 15.0 }
-            };
-            index.Add(
-                new ulong[] { 2, 2, 2, 2 },
-                inputVectors
-            );
-            Assert.True(index.Contains(2));
-            Assert.Equal((uint)4, index.Size());
-            Assert.Equal(4, index.Count(2));
-
-            int foundVectorsCount = index.Get(2, 3, out double[][] retrievedVectors);
-            Assert.Equal(3, foundVectorsCount);
-        }
-    }
-
-    [Fact]
-    public void AddFloatVectorSucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3))
-        {
-            index.Add(1, new float[] { 1.0f, 2.0f, 3.0f });
-            Assert.Equal((uint)1, index.Size());
-            Assert.True(index.Contains(1));
-            Assert.False(index.Contains(2));
-        }
-    }
-
-    [Fact]
-    public void AddFloatVectorsUnderSameKeySucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3, multi: true))
-        {
-            index.Add(1, new float[] { 1.0f, 2.0f, 3.0f });
-            index.Add(2, new float[] { 1.0f, 2.0f, 3.0f });
-            index.Add(3, new float[] { 1.0f, 2.0f, 3.0f });
-            index.Add(1, new float[] { 1.0f, 2.0f, 3.0f });
-            index.Add(2, new float[] { 1.0f, 2.0f, 3.0f });
-            Assert.True(index.Contains(1));
-            Assert.True(index.Contains(2));
-            Assert.True(index.Contains(3));
-            Assert.Equal((uint)5, index.Size());
-        }
-    }
-
-    [Fact]
-    public void AddGetFloatVectorSucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3))
-        {
-            var inputVector = new float[] { 1.0f, 2.0f, 3.0f };
-            index.Add(1, inputVector);
-            Assert.Equal((uint)1, index.Size());
-            Assert.True(index.Contains(1));
-            Assert.False(index.Contains(2));
-            Assert.True(index.Get(1, out float[] retrievedVector) == 1);
-            Assert.NotNull(retrievedVector);
-            bool areEqual = inputVector
-                .Zip(retrievedVector, (a, b) => (a, b))
-                .All(pair => Math.Abs(pair.a - pair.b) <= Epsilon);
-
-            Assert.True(areEqual);
-        }
-    }
-
-    [Fact]
-    public void AddGetDoubleVectorSucceeds()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3))
-        {
-            var inputVector = new double[] { 1.0, 2.0, 3.33 };
-            index.Add(1, inputVector);
-            Assert.Equal((uint)1, index.Size());
-            Assert.True(index.Contains(1));
-            Assert.False(index.Contains(2));
-            Assert.True(index.Get(1, out double[] retrievedVector) == 1);
-            Assert.NotNull(retrievedVector);
-            bool areEqual = inputVector
-                .Zip(retrievedVector, (a, b) => (a, b))
-                .All(pair => Math.Abs(pair.a - pair.b) <= Epsilon);
-
-            Assert.True(areEqual);
-        }
-    }
-
-    [Fact]
-    public void ItThrowsException_AddAddWithSameKeyAndGetVector()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 3))
-        {
-            ulong key = 1;
-            float[] vector1 = { 1.0f, 2.0f, 33.3f };
-            index.Add(key, vector1);
-            float[] vector2 = { 3.0f, 4.0f, 34.2f };
-            Assert.Throws<USearchException>(() => index.Add(key, vector2));
-        }
-    }
-
-    [Fact]
-    public void TestAddRemoveAddAndGet()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 2))
-        {
-            ulong key = 1;
-            float[] vector = { 1.0f, 2.0f };
-
-            index.Add(key, vector);
-            Assert.True(index.Contains(key));
-            index.Remove(key);
-            Assert.False(index.Contains(key));
-            index.Add(key, vector);
-            Assert.True(index.Contains(key));
-            Assert.True(index.Get(key, out float[] retrievedVector) == 1);
-            Assert.Equal(vector, retrievedVector);
-        }
-    }
-
-    [Fact]
-    public void TestAddRemoveAddWithSameKeyVectorAndGet()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 2))
-        {
-            ulong key = 1;
-            float[] vector = { 1.0f, 2.0f };
-
-            index.Add(key, vector);
-            Assert.True(index.Contains(key));
-            index.Remove(key);
-            Assert.False(index.Contains(key));
-            index.Add(key, vector);
-            Assert.True(index.Contains(key));
-            Assert.True(index.Get(key, out float[] retrievedVector) == 1);
-            Assert.Equal(vector, retrievedVector);
-        }
-    }
-
-    [Fact]
-    public void TestGetKeyThatNotExist()
-    {
-        using (var index = new USearchIndex(MetricKind.Cos, ScalarKind.Float32, 2))
-        {
-            Assert.True(index.Get(1, out float[] retrievedVector) == 0);
-            Assert.False(index.Contains(1));
-        }
-    }
-
-    [Fact]
-    public void Search_FindsExpectedResults()
-    {
-        using (var index = new USearchIndex(MetricKind.L2sq, ScalarKind.Float32, 2))
-        {
-            index.Add(
-                Enumerable.Range(1, 5).Select(i => (ulong)i).ToArray(),
-                new float[][]
-                {
-                    new float[] { -0.5f, -0.6f },
-                    new float[] { -0.1f, -0.1f },
-                    new float[] { 0.2f, 0.2f },
-                    new float[] { 0.7f, 0.8f },
-                    new float[] { 0.9f, 1.0f }
-                }
-            );
-            var queryVector = new float[] { 0.0f, 0.0f };
-
-            var matches = index.Search(queryVector, 2, out ulong[] keys, out float[] distances);
-
-            Assert.Equal(2, matches);
-            Assert.Contains(2UL, keys);
-            Assert.Contains(3UL, keys);
-            Assert.Equal(0.02, distances[0], Epsilon);
-            Assert.Equal(0.08, distances[1], Epsilon);
-        }
-    }
-
-    [Fact]
-    public void Search_AfterRemovingKey_FindsExpectedResults()
-    {
-        using (var index = new USearchIndex(MetricKind.L2sq, ScalarKind.Float32, 2))
-        {
-            index.Add(1, new float[] { 0.1f, 0.2f });
-            index.Add(2, new float[] { 0.2f, 0.3f });
-            index.Add(3, new float[] { 0.3f, 0.4f });
-            index.Remove(2);
-            var queryVector = new float[] { 0.2f, 0.3f };
-
-            var matches = index.Search(queryVector, 2, out ulong[] keys, out float[] distances);
-
-            Assert.Equal(2, matches);
-            Assert.Contains(1UL, keys);
-            Assert.Contains(3UL, keys);
-            Assert.DoesNotContain(2UL, keys);
-        }
-    }
-
-    [Fact]
-    public void Search_WithVectorsLessThanResultsLimit()
-    {
-        using (var index = new USearchIndex(MetricKind.L2sq, ScalarKind.Float32, 2))
-        {
-            index.Add(1, new float[] { 0.9f, 0.9f });
-            index.Add(2, new float[] { 0.8f, 0.8f });
-            var queryVector = new float[] { 0.1f, 0.2f };
-
-            var matches = index.Search(queryVector, 10, out ulong[] keys, out float[] distances);
-
-            Assert.Equal(2, matches);
-            Assert.Contains(1UL, keys);
-            Assert.Contains(2UL, keys);
-        }
-    }
-
-    [Fact]
-    public void ItCanRenameKey()
-    {
-        using (var index = new USearchIndex(MetricKind.L2sq, ScalarKind.Float32, 2))
-        {
-            index.Add(1, new float[] { 0.9f, 0.9f });
-            index.Add(2, new float[] { 0.8f, 0.8f });
-            index.Rename(1, 3);
-            Assert.False(index.Contains(1));
-            Assert.True(index.Contains(2));
-            Assert.True(index.Contains(3));
-        }
-    }
-
-}