usearch 0.2.4 → 0.8.0

package/CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+info@unum.cloud.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

package/CONTRIBUTING.md

@@ -0,0 +1,12 @@
+# Contribution Guide
+
+Thank you for even opening this page!
+It's always nice to have third-party contributors!
+
+---
+
+To keep the quality of the code high, we have a set of guidelines common to all Unum projects.
+
+- Whats the procedure? [answered](https://github.com/unum-cloud/awesome/blob/main/Workflow.md#organizing-software-development)
+- How to organize branches? [answered](https://github.com/unum-cloud/awesome/blob/main/Workflow.md#branches)
+- How to style commits? [answered](https://github.com/unum-cloud/awesome/blob/main/Workflow.md#commits)

package/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "usearch"
-version = "0.2.4"
+version = "0.8.0"
 authors = ["Ashot Vardanian <1983160+ashvardanian@users.noreply.github.com>"]
 description = "Smaller & Faster Single-File Vector Search Engine from Unum"
 edition = "2021"

package/Dockerfile

@@ -0,0 +1,14 @@
+# syntax=docker/dockerfile:1
+
+FROM ubuntu:23.04
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends python3.11 python3-pip && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+RUN pip3 install --no-cache-dir --break-system-packages ucall usearch
+
+WORKDIR /usearch
+COPY python/usearch/server.py server.py
+
+CMD ["python3", "./server.py"]
+EXPOSE 8545

package/README.md

@@ -82,7 +82,7 @@ USearch is designed to be compact and broadly compatible without sacrificing per
 > Jump to the [Performance Tuning][benchmarking] section to read about the effects of those hyper-parameters.
 
 [sloc]: https://en.wikipedia.org/wiki/Source_lines_of_code
-[benchmarking]: https://github.com/unum-cloud/usearch
+[benchmarking]: https://github.com/unum-cloud/usearch/blob/main/docs/benchmarks.md
 
 ## User-Defined Functions
 
@@ -92,7 +92,7 @@ A good example would be the rare [Haversine][haversine] distance, used to comput
 Another example would be designing a custom metric for **composite embeddings** concatenated from multiple AI models in real-world applications. 
 USearch supports that: [Python](#user-defined-functions-in-python) and [C++](#user-defined-functions-in-c) examples.
 
-![USearch: Vector Search Approaches](assets/usearch-approaches-white.png)
+![USearch: Vector Search Approaches](https://github.com/unum-cloud/usearch/blob/main/assets/usearch-approaches-white.png?raw=true)
 
 Unlike older approaches indexing high-dimensional spaces, like KD-Trees and Locality Sensitive Hashing, HNSW doesn't require vectors to be identical in length.
 They only have to be comparable.
@@ -106,7 +106,7 @@ So you can apply it in [obscure][obscure] applications, like searching for simil
 Training a quantization model and dimension-reduction is a common approach to accelerate vector search.
 Those, however, are only sometimes reliable, can significantly affect the statistical properties of your data, and require regular adjustments if your distribution shifts.
 
-![USearch uint40_t support](assets/usearch-neighbor-types.png)
+![USearch uint40_t support](https://github.com/unum-cloud/usearch/blob/main/assets/usearch-neighbor-types.png?raw=true)
 
 Instead, we have focused on high-precision arithmetic over low-precision downcasted vectors.
 The same index, and `add` and `search` operations will automatically down-cast or up-cast between `f32_t`, `f16_t`, `f64_t`, and `f8_t` representations, even if the hardware doesn't natively support it.
@@ -157,10 +157,11 @@ index_gt<cos_gt<float>> index;
 float vec[3] = {0.1, 0.3, 0.2};
 
 index.reserve(10);
-index.add(/* label: */ 42, /* vector: */ {&vec, 3});
-index.search(
-  /* query: */ {&vec, 3}, /* top */ 5 /* results */,
-  /* with callback: */ [](std::size_t label, float distance) { });
+index.add(/* label: */ 42, /* vector: */ {&vec[0], 3});
+auto results = index.search(/* query: */ {&vec[0], 3}, 5 /* neighbors */);
+
+for (std::size_t i = 0; i != results.size(); ++i)
+    results[i].member.label, results[i].member.vector, results[i].distance;
 ```
 
 The `add` is thread-safe for concurrent index construction.
@@ -205,7 +206,7 @@ The following distances are pre-packaged:
 - `pearson_correlation_gt<scalar_t>` for "Pearson" correlation between probability distributions.
 - `haversine_gt<scalar_t>` for "Haversine" or "Great Circle" distance between coordinates.
 
-#### Bring your Threads
+#### Multi-Threading
 
 Most AI, HPC, or Big Data packages use some form of a thread pool.
 Instead of spawning additional threads within USearch, we focus on the thread safety of `add()` function, simplifying resource management.
@@ -213,7 +214,7 @@ Instead of spawning additional threads within USearch, we focus on the thread sa
 ```cpp
 #pragma omp parallel for
     for (std::size_t i = 0; i < n; ++i)
-        native.add(label, span_t{vector, dims}, omp_get_thread_num());
+        native.add(label, span_t{vector, dims}, add_config_t { .thread = omp_get_thread_num() });
 ```
 
 During initialization, we allocate enough temporary memory for all the cores on the machine.
@@ -235,7 +236,7 @@ from usearch.index import Index
 
 index = Index(
     ndim=3, # Define the number of dimensions in input vectors
-    metric='cos', # Choose 'l2', 'haversine' or other metric, default = 'ip'
+    metric='cos', # Choose 'l2sq', 'haversine' or other metric, default = 'ip'
     dtype='f32', # Quantize to 'f16' or 'f8' if needed, default = 'f32'
     connectivity=16, # How frequent should the connections in the graph be, optional
     expansion_add=128, # Control the recall of indexing, optional
@@ -383,6 +384,14 @@ let results = index.search(&first, 10).unwrap();
 assert_eq!(results.count, 2);
 ```
 
+#### Multi-Threading
+
+```rust
+assert!(index.add_in_thread(42, &first, 0).is_ok());
+assert!(index.add_in_thread(43, &second, 0).is_ok());
+let results = index.search_in_thread(&first, 10, 0).unwrap();
+```
+
 Being a systems-programming language, Rust has better control over memory management and concurrency but lacks function overloading.
 Aside from the `add` and `search`, USearch Rust binding also provides `add_in_thread` and `search_in_thread`, which let users identify the calling thread to use underlying temporary memory more efficiently.
 
@@ -397,7 +406,7 @@ assert!(index.view("index.usearch").is_ok());
 #### Metrics
 
 ```rust
-assert!(new_l2(3, &quant, 0, 0, 0).is_ok());
+assert!(new_l2sq(3, &quant, 0, 0, 0).is_ok());
 assert!(new_cos(3, &quant, 0, 0, 0).is_ok());
 assert!(new_haversine(&quant, 0, 0, 0).is_ok());
 ```
@@ -436,7 +445,7 @@ https://github.com/unum-cloud/usearch
 #### Quickstart
 
 ```swift
-let index = Index.l2(dimensions: 3, connectivity: 8)
+let index = Index.l2sq(dimensions: 3, connectivity: 8)
 let vectorA: [Float32] = [0.3, 0.5, 1.2]
 let vectorB: [Float32] = [0.4, 0.2, 1.2]
 index.add(label: 42, vector: vectorA[...])

package/VERSION

@@ -1 +1 @@
-0.2.4
+0.8.0

package/binding.gyp

@@ -15,15 +15,19 @@
             "dependencies": [
                 "<!(node -p \"require('node-addon-api').gyp\")"
             ],
-            "cflags!": ["-fno-exceptions", "-Wunknown-pragmas"],
-            "cflags_cc!": ["-fno-exceptions", "-Wunknown-pragmas"],
-            "conditions": [
-                ["OS==\"mac\"", {
-                    "xcode_settings": {
-                        "GCC_ENABLE_CPP_EXCEPTIONS": "YES"
-                    }
-                }]
-            ]
+            "cflags": ["-fexceptions", "-Wno-unknown-pragmas"],
+            "cflags_cc": ["-fexceptions", "-Wno-unknown-pragmas", "-std=c++11"],
+            "xcode_settings": {
+                "GCC_ENABLE_CPP_EXCEPTIONS": "YES",
+                "CLANG_CXX_LIBRARY": "libc++",
+                "MACOSX_DEPLOYMENT_TARGET": "10.15"
+            },
+            "msvs_settings": {
+                "VCCLCompilerTool": { 
+                    "ExceptionHandling": 1, 
+                    "AdditionalOptions": [ "-std:c++11" ]
+                }
+            }
         }
     ]
 }

package/build.rs

@@ -2,6 +2,7 @@ fn main() {
     cxx_build::bridge("rust/lib.rs")
         .file("rust/lib.cpp")
         .flag_if_supported("-std=c++11")
+        .flag_if_supported("-Wno-unknown-pragmas")
         .include("include")
         .include("rust")
         .include("src")
@@ -12,6 +13,6 @@ fn main() {
     println!("cargo:rerun-if-changed=rust/lib.rs");
     println!("cargo:rerun-if-changed=rust/lib.cpp");
     println!("cargo:rerun-if-changed=rust/lib.hpp");
-    println!("cargo:rerun-if-changed=src/advanced.hpp");
+    println!("cargo:rerun-if-changed=src/punned.hpp");
     println!("cargo:rerun-if-changed=include/usearch/usearch.hpp");
 }

package/cmake/config.cmake.in

@@ -0,0 +1,15 @@
+include(FindPackageHandleStandardArgs)
+set(${CMAKE_FIND_PACKAGE_NAME}_CONFIG ${CMAKE_CURRENT_LIST_FILE})
+find_package_handle_standard_args(@PROJECT_NAME@ CONFIG_MODE)
+
+if(NOT TARGET @PROJECT_NAME@::@USEARCH_TARGET_NAME@)
+    include("${CMAKE_CURRENT_LIST_DIR}/@USEARCH_TARGETS_EXPORT_NAME@.cmake")
+    if((NOT TARGET @USEARCH_TARGET_NAME@) AND
+       (NOT @PROJECT_NAME@_FIND_VERSION OR
+        @PROJECT_NAME@_FIND_VERSION VERSION_LESS 3.2.0))
+        add_library(@USEARCH_TARGET_NAME@ INTERFACE IMPORTED)
+        set_target_properties(@USEARCH_TARGET_NAME@ PROPERTIES
+            INTERFACE_LINK_LIBRARIES @PROJECT_NAME@::@USEARCH_TARGET_NAME@
+        )
+    endif()
+endif()

package/cmake/pkg-config.pc.in

@@ -0,0 +1,4 @@
+Name: ${PROJECT_NAME}
+Description: Smaller & Faster Single-File Vector Search Engine from Unum
+Version: ${PROJECT_VERSION}
+Cflags: -I${CMAKE_INSTALL_FULL_INCLUDEDIR}

package/cmake/usearchConfigVersion.cmake.in

@@ -0,0 +1,20 @@
+# This is essentially cmake's BasicConfigVersion-SameMajorVersion.cmake.in but
+# without the 32/64-bit check.  Since USearch is a header-only library, it doesn't
+# matter if it was built on a different platform than what it is used on (see
+# https://github.com/nlohmann/json/issues/1697).
+set(PACKAGE_VERSION "@PROJECT_VERSION@")
+
+if(PACKAGE_VERSION VERSION_LESS PACKAGE_FIND_VERSION)
+  set(PACKAGE_VERSION_COMPATIBLE FALSE)
+else()
+
+  if(PACKAGE_FIND_VERSION_MAJOR STREQUAL "@PROJECT_VERSION_MAJOR@")
+    set(PACKAGE_VERSION_COMPATIBLE TRUE)
+  else()
+    set(PACKAGE_VERSION_COMPATIBLE FALSE)
+  endif()
+
+  if(PACKAGE_FIND_VERSION STREQUAL PACKAGE_VERSION)
+      set(PACKAGE_VERSION_EXACT TRUE)
+  endif()
+endif()

package/conanfile.py

@@ -4,7 +4,7 @@ from conans import ConanFile
 class USearchConan(ConanFile):
 
     name = 'USearch'
-    version = '0.2.4'
+    version = '0.8.0'
     license = 'Apache 2.0'
     url = 'https://github.com/unum-cloud/usearch'
     description = 'Smaller & Faster Single-File Vector Search Engine from Unum'

package/docs/{Benchmarking.md → benchmarks.md}

@@ -77,7 +77,7 @@ SYNOPSIS
         ./build_release/bench [--vectors <path>] [--queries <path>] [--neighbors <path>] [-b] [-j
                               <integer>] [-c <integer>] [--expansion-add <integer>]
                               [--expansion-search <integer>] [--native|--f16quant|--i8quant]
-                              [--ip|--l2|--cos|--haversine] [-h]
+                              [--ip|--l2sq|--cos|--haversine] [-h]
 
 OPTIONS
         --vectors <path>
@@ -106,7 +106,7 @@ OPTIONS
         --f16quant  Enable `f16_t` quantization
         --i8quant   Enable `int8_t` quantization
         --ip        Choose Inner Product metric
-        --l2        Choose L2 Euclidean metric
+        --l2sq        Choose L2 Euclidean metric
         --cos       Choose Angular metric
         --haversine Choose Haversine metric
         -h, --help  Print this help information on this tool and exit
@@ -170,6 +170,15 @@ mkdir -p datasets/deep_1B/ && \
 
 ## Profiling
 
+Enabling Huge Pages:
+
+```sh
+sudo cat /proc/sys/vm/nr_hugepages
+sudo sysctl -w vm.nr_hugepages=2048
+sudo reboot
+sudo cat /proc/sys/vm/nr_hugepages
+```
+
 With `perf`:
 
 ```sh

package/docs/benchmarks.rst

@@ -0,0 +1,5 @@
+========================
+Benchmarks
+========================
+
+.. mdinclude:: benchmarks.md

package/docs/{Compilation.md → compilation.md}

@@ -1,65 +1,86 @@
 # Compiling USearch
 
-## Build
-
 Before building the first time, please pull submodules.
 
 ```sh
 git submodule update --init --recursive
 ```
 
-### Linux
+## C++
+
+Linux:
 
 ```sh
 cmake -B ./build_release \
     -DCMAKE_CXX_COMPILER="g++-12" \
     -DCMAKE_BUILD_TYPE=Release \
+    -DUSEARCH_USE_JEMALLOC=1 \
     -DUSEARCH_USE_OPENMP=1 \
-    -DUSEARCH_USE_JEMALLOC=1 && \
+    -DUSEARCH_USE_SIMSIMD=1 \
+    -DUSEARCH_BUILD_BENCHMARK=1 \
+    -DUSEARCH_BUILD_TEST=1 \
+    && \
     make -C ./build_release -j
+```
+
+MacOS:
+
+```sh
+brew install libomp llvm
+cmake -B ./build_release \
+    -DCMAKE_C_COMPILER="/opt/homebrew/opt/llvm/bin/clang" \
+    -DCMAKE_CXX_COMPILER="/opt/homebrew/opt/llvm/bin/clang++" \
+    -DUSEARCH_USE_OPENMP=1 \
+    -DUSEARCH_USE_SIMSIMD=1 \
+    -DUSEARCH_BUILD_BENCHMARK=1 \
+    -DUSEARCH_BUILD_TEST=1 \
+    && \
+    make -C ./build_release -j 
+```
+
+Benchmarking:
 
-# To benchmark in C++:
+```sh
 ./build_release/bench \
     --vectors datasets/wiki_1M/base.1M.fbin \
     --queries datasets/wiki_1M/query.public.100K.fbin \
     --neighbors datasets/wiki_1M/groundtruth.public.100K.ibin
 
-# Large benchmark on a 1TB dataset:
 ./build_release/bench \
     --vectors datasets/t2i_1B/base.1B.fbin \
     --queries datasets/t2i_1B/query.public.100K.fbin \
     --neighbors datasets/t2i_1B/groundtruth.public.100K.ibin \
     --output datasets/t2i_1B/index.usearch \
     --cos
+```
 
-# To benchmark in Python:
-python python/bench.py \
-    --vectors datasets/wiki_1M/base.1M.fbin \
-    --queries datasets/wiki_1M/query.public.100K.fbin \
-    --neighbors datasets/wiki_1M/groundtruth.public.100K.ibin
+## Python
+
+```sh
+pip install -e .
+pytest python/scripts/test.py
 ```
 
-### MacOS
+Benchmarking:
 
 ```sh
-brew install libomp llvm
-cmake \
-    -DCMAKE_C_COMPILER="/opt/homebrew/opt/llvm/bin/clang" \
-    -DCMAKE_CXX_COMPILER="/opt/homebrew/opt/llvm/bin/clang++" \
-    -DUSEARCH_USE_SIMD=1 \
-    -DUSEARCH_USE_OPENMP=1 \
-     -B ./build_release && \
-    make -C ./build_release -j 
+pip install faiss-cpu
+python python/scripts/bench.py \
+    --vectors datasets/wiki_1M/base.1M.fbin \
+    --queries datasets/wiki_1M/query.public.100K.fbin \
+    --neighbors datasets/wiki_1M/groundtruth.public.100K.ibin
 ```
 
-### Python
+> Optional parameters include `connectivity`, `expansion_add`, `expansion_search`.
+
+Checking the effect of different embedding dimensions on construction speed:
 
 ```sh
-pip install -e .
-pytest python/test.py
+python python/scripts/bench_params.py dimensions
+python python/scripts/bench_params.py connectivity
 ```
 
-### JavaScript
+## JavaScript
 
 ```sh
 npm install
@@ -67,14 +88,14 @@ node javascript/test.js
 npm publish
 ```
 
-### Rust
+## Rust
 
 ```sh
 cargo test -p usearch
 cargo publish
 ```
 
-### Java
+## Java
 
 ```sh
 gradle clean build
@@ -105,8 +126,14 @@ java -cp . -Djava.library.path=/Users/av/github/usearch/java/cloud/unum/usearch/
 java -cp . -Djava.library.path=/Users/av/github/usearch/java cloud.unum.usearch.Index
 ```
 
-### Wolfram
+## Wolfram
 
 ```sh
 brew install --cask wolfram-engine
 ```
+
+## Docker
+
+```sh
+docker build -t unum/usearch . && docker run unum/usearch
+```

package/docs/compilation.rst

@@ -0,0 +1,5 @@
+========================
+Compilation
+========================
+
+.. mdinclude:: compilation.md