usearch 2.9.0 → 2.9.2

package/stringzilla/README.md

@@ -1,5 +1,16 @@
 # StringZilla 🦖
 
+_The world wastes a minimum of $100M annually due to inefficient string operations.
+A typical codebase processes strings character by character, resulting in too many branches and data-dependencies, neglecting 90% of modern CPU's potential.
+LibC is different.
+It attempts to leverage SIMD instructions to boost some operations, and is often used by higher-level languages, runtimes, and databases.
+But it isn't perfect.
+1️⃣ First, even on common hardware, including over a billion 64-bit ARM CPUs, common functions like `strstr` and `memmem` only achieve 1/3 of the CPU's thoughput.
+2️⃣ Second, SIMD coverage is inconsistent: acceleration in forward scans does not guarantee speed in the reverse-order search.
+3️⃣ At last, most high-level languages can't always use LibC, as the strings are often not NULL-terminated or may contain the Unicode "Zero" character in the middle of the string.
+That's why StringZilla was created.
+To provide predictably high performance, portable to any modern platform, operating system, and programming language._
+
 [![StringZilla Python installs](https://static.pepy.tech/personalized-badge/stringzilla?period=total&units=abbreviation&left_color=black&right_color=blue&left_text=StringZilla%20Python%20installs)](https://github.com/ashvardanian/stringzilla)
 [![StringZilla Rust installs](https://img.shields.io/crates/d/stringzilla?logo=rust&label=Rust%20installs)](https://crates.io/crates/stringzilla)
 [![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/ashvardanian/StringZilla/release.yml?branch=main&label=Ubuntu)](https://github.com/ashvardanian/StringZilla/actions/workflows/release.yml)
@@ -20,9 +31,14 @@ It __accelerates exact and fuzzy string matching, edit distance computations, so
 - 🐍 __[Python](#quick-start-python-🐍):__ Upgrade your `str` to faster `Str`
 - 🍎 __[Swift](#quick-start-swift-🍏):__ Use the `String+StringZilla` extension
 - 🦀 __[Rust](#quick-start-rust-🦀):__ Use the `StringZilla` traits crate
+- 🐚 __[Shell][faq-shell]__: Accelerate common CLI tools with `sz_` prefix
 - 📚 Researcher? Jump to [Algorithms & Design Decisions](#algorithms--design-decisions-📚)
-- 🤝 Want to benchmark or contribute? Jump to [Contributing](CONTRIBUTING.md)
-- Code in other languages? Let [me](https://github.com/ashvardanian) know!
+- 💡 Thinking to contribute? Look for ["good first issues"][first-issues]
+- 🤝 And check the [guide](CONTRIBUTING.md) to setup the environment
+- Want more bindings or features? Let [me](https://github.com/ashvardanian) know!
+
+[faq-shell]: https://github.com/ashvardanian/StringZilla/blob/main/cli/README.md
+[first-issues]: https://github.com/ashvardanian/StringZilla/issues
 
 __Who is this for?__
 
@@ -293,14 +309,14 @@ Consider contributing, if you need a feature that's not yet implemented.
 
 ### Basic Usage
 
-If you've ever used the Python `str` or `bytes` class, you'll know what to expect.
+If you've ever used the Python `str`, `bytes`, `bytearray`, `memoryview` class, you'll know what to expect.
 StringZilla's `Str` class is a hybrid of those two, providing `str`-like interface to byte-arrays.
 
 ```python
 from stringzilla import Str, File
 
-text_from_str = Str('some-string')
-text_from_file = Str(File('some-file.txt'))
+text_from_str = Str('some-string') # no copies, just a view
+text_from_file = Str(File('some-file.txt')) # memory-mapped file
 ```
 
 The `File` class memory-maps a file from persistent memory without loading its copy into RAM.
@@ -312,43 +328,94 @@ A standard dataset pre-processing use case would be to map a sizeable textual da
 - Length: `len(text) -> int`
 - Indexing: `text[42] -> str`
 - Slicing: `text[42:46] -> Str`
-- String conversion: `str(text) -> str`
 - Substring check: `'substring' in text -> bool`
 - Hashing: `hash(text) -> int`
+- String conversion: `str(text) -> str`
 
 ### Advanced Operations
 
-- `text.contains('substring', start=0, end=9223372036854775807) -> bool`
-- `text.find('substring', start=0, end=9223372036854775807) -> int`
-- `text.count('substring', start=0, end=9223372036854775807, allowoverlap=False) -> int`
-- `text.splitlines(keeplinebreaks=False, separator='\n') -> Strs`
-- `text.split(separator=' ', maxsplit=9223372036854775807, keepseparator=False) -> Strs`
+```py
+import sys
+
+x: bool = text.contains('substring', start=0, end=sys.maxsize)
+x: int = text.find('substring', start=0, end=sys.maxsize)
+x: int = text.count('substring', start=0, end=sys.maxsize, allowoverlap=False)
+x: str = text.decode(encoding='utf-8', errors='strict')
+x: Strs = text.split(separator=' ', maxsplit=sys.maxsize, keepseparator=False)
+x: Strs = text.rsplit(separator=' ', maxsplit=sys.maxsize, keepseparator=False)
+x: Strs = text.splitlines(keeplinebreaks=False, maxsplit=sys.maxsize)
+```
+
+It's important to note, that the last function behavior is slightly different from Python's `str.splitlines`.
+The [native version][faq-splitlines] matches `\n`, `\r`, `\v` or `\x0b`, `\f` or `\x0c`, `\x1c`, `\x1d`, `\x1e`, `\x85`, `\r\n`, `\u2028`, `\u2029`, including 3x two-bytes-long runes.
+The StringZilla version matches only `\n` and is practically a shortcut for `text.split('\n')`.
+
+[faq-splitlines]: https://docs.python.org/3/library/stdtypes.html#str.splitlines
+
+### Character Set Operations
+
+Python strings don't natively support character set operations.
+This forces people to use regular expressions, which are slow and hard to read.
+To avoid the need for `re.finditer`, StringZilla provides the following interfaces:
+
+```py
+x: int = text.find_first_of('chars', start=0, end=sys.maxsize)
+x: int = text.find_last_of('chars', start=0, end=sys.maxsize)
+x: int = text.find_first_not_of('chars', start=0, end=sys.maxsize)
+x: int = text.find_last_not_of('chars', start=0, end=sys.maxsize)
+x: Strs = text.split_charset(separator='chars', maxsplit=sys.maxsize, keepseparator=False)
+x: Strs = text.rsplit_charset(separator='chars', maxsplit=sys.maxsize, keepseparator=False)
+```
 
 ### Collection-Level Operations
 
-Once split into a `Strs` object, you can sort, shuffle, and reorganize the slices.
+Once split into a `Strs` object, you can sort, shuffle, and reorganize the slices, with minimum memory footprint.
+If all the chunks are located in consecutive memory regions, the memory overhead can be as low as 4 bytes per chunk.
 
 ```python
 lines: Strs = text.split(separator='\n') # 4 bytes per line overhead for under 4 GB of text
-lines.sort() # explodes to 16 bytes per line overhead for any length text
-lines.shuffle(seed=42) # reproducing dataset shuffling with a seed
+batch: Strs = lines.sample(seed=42) # 10x faster than `random.choices`
+lines.shuffle(seed=42) # or shuffle all lines in place and shard with slices
+# WIP: lines.sort() # explodes to 16 bytes per line overhead for any length text
+# WIP: sorted_order: tuple = lines.argsort() # similar to `numpy.argsort`
 ```
 
-Assuming superior search speed splitting should also work 3x faster than with native Python strings.
-Need copies?
+Working on [RedPajama][redpajama], addressing 20 Billion annotated english documents, one will need only 160 GB of RAM instead of Terabytes.
+Once loaded, the data will be memory-mapped, and can be reused between multiple Python processes without copies.
+And of course, you can use slices to navigate the dataset and shard it between multiple workers.
 
 ```python
-sorted_copy: Strs = lines.sorted()
-shuffled_copy: Strs = lines.shuffled(seed=42)
+lines[::3] # every third line
+lines[1::1] # every odd line
+lines[:-100:-1] # last 100 lines in reverse order
 ```
 
-Those collections of `Strs` are designed to keep the memory consumption low.
-If all the chunks are located in consecutive memory regions, the memory overhead can be as low as 4 bytes per chunk.
-That's designed to handle very large datasets, like [RedPajama][redpajama].
-To address all 20 Billion annotated english documents in it, one will need only 160 GB of RAM instead of Terabytes.
-
 [redpajama]: https://github.com/togethercomputer/RedPajama-Data
 
+### Iterators and Memory Efficiency
+
+Python's operations like `split()` and `readlines()` immediately materialize a `list` of copied parts.
+This can be very memory-inefficient for large datasets.
+StringZilla saves a lot of memory by viewing existing memory regions as substrings, but even more memory can be saved by using lazily evaluated iterators.
+
+- `text.split_iter(separator=' ', keepseparator=False) -> SplitIterator[Str]`
+- `text.rsplit_iter(separator=' ', keepseparator=False) -> SplitIterator[Str]`
+- `text.split_charset_iter(separator='chars', keepseparator=False) -> SplitIterator[Str]`
+- `text.rsplit_charset_iter(separator='chars', keepseparator=False) -> SplitIterator[Str]`
+
+StringZilla can easily be 10x more memory efficient than native Python classes for tokenization.
+With lazy operations, it practically becomes free.
+
+```py
+import stringzilla as sz
+%load_ext memory_profiler
+
+text = open("enwik9.txt", "r").read() # 1 GB, mean word length 7.73 bytes
+%memit text.split() # increment: 8670.12 MiB (152 ms)
+%memit sz.split(text) # increment: 530.75 MiB (25 ms)
+%memit sum(1 for _ in sz.split_iter(text)) # increment: 0.00 MiB
+```
+
 ### Low-Level Python API
 
 Aside from calling the methods on the `Str` and `Strs` classes, you can also call the global functions directly on `str` and `bytes` instances.
@@ -357,9 +424,9 @@ Assuming StringZilla CPython bindings are implemented [without any intermediate
 ```py
 import stringzilla as sz
 
-contains: bool = sz.contains("haystack", "needle", start=0, end=9223372036854775807)
-offset: int = sz.find("haystack", "needle", start=0, end=9223372036854775807)
-count: int = sz.count("haystack", "needle", start=0, end=9223372036854775807, allowoverlap=False)
+contains: bool = sz.contains("haystack", "needle", start=0, end=sys.maxsize)
+offset: int = sz.find("haystack", "needle", start=0, end=sys.maxsize)
+count: int = sz.count("haystack", "needle", start=0, end=sys.maxsize, allowoverlap=False)
 ```
 
 ### Edit Distances
@@ -438,6 +505,35 @@ assert sz.alignment_score(
 
 </details>
 
+### Serialization
+
+#### Filesystem
+
+Similar to how `File` can be used to read a large file, other interfaces can be used to dump strings to disk faster.
+The `Str` class has `write_to` to write the string to a file, and `offset_within` to obtain integer offsets of substring view in larger string for navigation.
+
+```py
+web_archieve = Str("<html>...</html><html>...</html>")
+_, end_tag, next_doc = web_archieve.partition("</html>") # or use `find`
+next_doc_offset = next_doc.offset_within(web_archieve)
+web_archieve.write_to("next_doc.html")
+```
+
+#### PyArrow
+
+A `Str` is easy to cast to [PyArrow](https://arrow.apache.org/docs/python/arrays.html#string-and-binary-types) buffers.
+
+```py
+from pyarrow as foreign_buffer
+from stringzilla import Str
+
+original = "hello"
+view = Str(native)
+arrow = foreign_buffer(view.address, view.nbytes, view)
+```
+
+That means you can convert `Str` to `pyarrow.Buffer` and `Strs` to `pyarrow.Array` without extra copies.
+
 ## Quick Start: C/C++ 🛠️
 
 The C library is header-only, so you can just copy the `stringzilla.h` header into your project.
@@ -638,8 +734,8 @@ sz_string_is_on_stack(&string); // == sz_true_k
 sz_string_grow(&string, 100, &allocator); // == sz_true_k
 
 // Append, erase, insert into the string.
-sz_string_append(&string, "_Hello_", 7, &allocator); // == sz_true_k
-sz_string_append(&string, "world", 5, &allocator); // == sz_true_k
+sz_string_expand(&string, 0, "_Hello_", 7, &allocator); // == sz_true_k
+sz_string_expand(&string, SZ_SIZE_MAX, "world", 5, &allocator); // == sz_true_k
 sz_string_erase(&string, 0, 1);
 
 // Unpacking & introspection.
@@ -1015,9 +1111,27 @@ __`STRINGZILLA_BUILD_SHARED`, `STRINGZILLA_BUILD_TEST`, `STRINGZILLA_BUILD_BENCH
 ## Quick Start: Rust 🦀
 
 StringZilla is available as a Rust crate.
-It currently covers only the most basic functionality, but is planned to be extended to cover the full C++ API.
+Some of the interfaces will look familiar to the users of the `memchr` crate.
+
+```rust
+use stringzilla::sz;
+
+// Identical to `memchr::memmem::find` and `memchr::memmem::rfind` functions
+sz::find("Hello, world!", "world") // 7
+sz::rfind("Hello, world!", "world") // 7
+
+// Generalizations of `memchr::memrchr[123]`
+sz::find_char_from("Hello, world!", "world") // 2
+sz::rfind_char_from("Hello, world!", "world") // 11
+```
+
+Unlike `memchr`, the throughput of `stringzilla` is [high in both normal and reverse-order searches][memchr-benchmarks].
+It also provides no constraints on the size of the character set, while `memchr` allows only 1, 2, or 3 characters.
+In addition to global functions, `stringzilla` provides a `StringZilla` extension trait:
 
 ```rust
+use stringzilla::StringZilla;
+
 let my_string: String = String::from("Hello, world!");
 let my_str = my_string.as_str();
 let my_cow_str = Cow::from(&my_string);
@@ -1035,6 +1149,23 @@ assert_eq!(my_str.sz_find("world"), Some(7));
 assert_eq!(my_cow_str.as_ref().sz_find("world"), Some(7));
 ```
 
+The library also exposes Levenshtein and Hamming edit-distances for byte-arrays and UTF-8 strings, as well as Needleman-Wunch alignment scores.
+
+```rust
+use stringzilla::sz;
+
+// Handling arbitrary byte arrays:
+sz::edit_distance("Hello, world!", "Hello, world?"); // 1
+sz::hamming_distance("Hello, world!", "Hello, world?"); // 1
+sz::alignment_score("Hello, world!", "Hello, world?", sz::unary_substitution_costs(), -1); // -1
+
+// Handling UTF-8 strings:
+sz::hamming_distance_utf8("αβγδ", "αγγδ") // 1
+sz::edit_distance_utf8("façade", "facade") // 1
+```
+
+[memchr-benchmarks]: https://github.com/ashvardanian/memchr_vs_stringzilla
+
 ## Quick Start: Swift 🍏
 
 StringZilla is available as a Swift package.
@@ -1256,13 +1387,16 @@ Another one is the [Fibonacci hash trick](https://probablydance.com/2018/06/16/f
 
 ### Unicode, UTF-8, and Wide Characters
 
-StringZilla does not __yet__ implement any Unicode-specific algorithms.
-The content is addressed at byte-level, and the string is assumed to be encoded in UTF-8 or extended ASCII.
-Refer to [simdutf](https://github.com/simdutf/simdutf) for fast conversions and [icu](https://github.com/unicode-org/icu) for character metadata.
+Most StringZilla operations are byte-level, so they work well with ASCII and UTF8 content out of the box.
+In some cases, like edit-distance computation, the result of byte-level evaluation and character-level evaluation may differ.
+So StringZilla provides following functions to work with Unicode:
 
-This may introduce frictions, when binding to some programming languages.
-Namely, Java, JavaScript, Python 2, C#, and Objective-C use wide characters (`wchar`) - two byte long codes.
+- `sz_edit_distance_utf8` - computes the Levenshtein distance between two UTF-8 strings.
+- `sz_hamming_distance_utf8` - computes the Hamming distance between two UTF-8 strings.
+
+Java, JavaScript, Python 2, C#, and Objective-C, however, use wide characters (`wchar`) - two byte long codes, instead of the more reasonable fixed-length UTF32 or variable-length UTF8.
 This leads [to all kinds of offset-counting issues][wide-char-offsets] when facing four-byte long Unicode characters.
+So consider transcoding with [simdutf](https://github.com/simdutf/simdutf), if you are coming from such environments.
 
 [wide-char-offsets]: https://josephg.com/blog/string-length-lies/
 
@@ -1276,6 +1410,13 @@ If you like this project, you may also enjoy [USearch][usearch], [UCall][ucall],
 [uform]: https://github.com/unum-cloud/uform
 [simsimd]: https://github.com/ashvardanian/simsimd
 
+If you like strings and value efficiency, you may also enjoy the following projects:
+
+- [simdutf](https://github.com/simdutf/simdutf) - transcoding UTF8, UTF16, and UTF32 LE and BE.
+- [hyperscan](https://github.com/intel/hyperscan) - regular expressions with SIMD acceleration.
+- [pyahocorasick](https://github.com/WojciechMula/pyahocorasick) - Aho-Corasick algorithm in Python.
+- [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) - fast string matching in C++ and Python.
+
 ## License 📜
 
 Feel free to use the project under Apache 2.0 or the Three-clause BSD license at your preference.

package/stringzilla/VERSION

@@ -1 +1 @@
-3.2.0
+3.5.0

package/stringzilla/build.rs

@@ -1,12 +1,85 @@
+use std::env;
+
 fn main() {
-    cc::Build::new()
+    let mut build = cc::Build::new();
+    build
         .file("c/lib.c")
         .include("include")
         .warnings(false)
         .flag_if_supported("-std=c99")
-        .flag_if_supported("-fcolor-diagnostics")
-        .flag_if_supported("-fPIC")
-        .compile("stringzilla");
+        .flag_if_supported("-fPIC");
+
+    // Cargo will set different environment variables that we can use to properly configure the build.
+    // https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts
+    let target_arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default();
+    let target_endian = env::var("CARGO_CFG_TARGET_ENDIAN").unwrap_or_default();
+
+    // To get the operating system we can use the TARGET environment variable.
+    // To check the list of available targets, run `rustc --print target-list`.
+    let target = env::var("TARGET").unwrap_or_default();
+
+    if target.contains("linux") {
+        build.flag_if_supported("-fdiagnostics-color=always");
+        build.flag_if_supported("-O3");
+        build.flag_if_supported("-pedantic");
+
+        // Set architecture-specific flags and macros
+        if target_arch == "x86_64" {
+            build.define("SZ_USE_X86_AVX512", "1");
+            build.define("SZ_USE_X86_AVX2", "1");
+        } else {
+            build.define("SZ_USE_X86_AVX512", "0");
+            build.define("SZ_USE_X86_AVX2", "0");
+        }
+
+        if target_arch == "aarch64" {
+            build.flag_if_supported("-march=armv8-a+simd");
+            build.define("SZ_USE_ARM_SVE", "1");
+            build.define("SZ_USE_ARM_NEON", "1");
+        } else {
+            build.define("SZ_USE_ARM_SVE", "0");
+            build.define("SZ_USE_ARM_NEON", "0");
+        }
+    } else if target.contains("darwin") {
+        build.flag_if_supported("-fcolor-diagnostics");
+        build.flag_if_supported("-O3");
+        build.flag_if_supported("-pedantic");
+
+        if target_arch == "x86_64" {
+            // Assuming no AVX-512 support for Darwin as per setup.py logic
+            build.define("SZ_USE_X86_AVX512", "0");
+            build.define("SZ_USE_X86_AVX2", "1");
+        } else {
+            build.define("SZ_USE_X86_AVX512", "0");
+            build.define("SZ_USE_X86_AVX2", "0");
+        }
+
+        if target_arch == "aarch64" {
+            build.define("SZ_USE_ARM_SVE", "0"); // Assuming no SVE support for Darwin
+            build.define("SZ_USE_ARM_NEON", "1");
+        } else {
+            build.define("SZ_USE_ARM_SVE", "0");
+            build.define("SZ_USE_ARM_NEON", "0");
+        }
+    } else if target.contains("windows") {
+        // Set architecture-specific flags and macros
+        if target_arch == "x86_64" {
+            build.define("SZ_USE_X86_AVX512", "1");
+            build.define("SZ_USE_X86_AVX2", "1");
+        } else {
+            build.define("SZ_USE_X86_AVX512", "0");
+            build.define("SZ_USE_X86_AVX2", "0");
+        }
+    }
+
+    // Set endian-specific macro
+    if target_endian == "big" {
+        build.define("SZ_DETECT_BIG_ENDIAN", "1");
+    } else {
+        build.define("SZ_DETECT_BIG_ENDIAN", "0");
+    }
+
+    build.compile("stringzilla");
 
     println!("cargo:rerun-if-changed=c/lib.c");
     println!("cargo:rerun-if-changed=rust/lib.rs");

package/stringzilla/c/lib.c

@@ -23,6 +23,7 @@ typedef sz_size_t size_t; // Reuse the type definition we've inferred from `stri
 #else
 typedef __SIZE_TYPE__ size_t; // For GCC/Clang
 #endif
+int rand(void) { return 0; }
 void free(void *start) { sz_unused(start); }
 void *malloc(size_t length) {
     sz_unused(length);
@@ -255,6 +256,20 @@ SZ_DYNAMIC sz_cptr_t sz_rfind_charset(sz_cptr_t text, sz_size_t length, sz_chars
     return sz_dispatch_table.rfind_from_set(text, length, set);
 }
 
+SZ_DYNAMIC sz_size_t sz_hamming_distance( //
+    sz_cptr_t a, sz_size_t a_length,      //
+    sz_cptr_t b, sz_size_t b_length,      //
+    sz_size_t bound) {
+    return sz_hamming_distance_serial(a, a_length, b, b_length, bound);
+}
+
+SZ_DYNAMIC sz_size_t sz_hamming_distance_utf8( //
+    sz_cptr_t a, sz_size_t a_length,           //
+    sz_cptr_t b, sz_size_t b_length,           //
+    sz_size_t bound) {
+    return sz_hamming_distance_utf8_serial(a, a_length, b, b_length, bound);
+}
+
 SZ_DYNAMIC sz_size_t sz_edit_distance( //
     sz_cptr_t a, sz_size_t a_length,   //
     sz_cptr_t b, sz_size_t b_length,   //
@@ -262,6 +277,13 @@ SZ_DYNAMIC sz_size_t sz_edit_distance( //
     return sz_dispatch_table.edit_distance(a, a_length, b, b_length, bound, alloc);
 }
 
+SZ_DYNAMIC sz_size_t sz_edit_distance_utf8( //
+    sz_cptr_t a, sz_size_t a_length,        //
+    sz_cptr_t b, sz_size_t b_length,        //
+    sz_size_t bound, sz_memory_allocator_t *alloc) {
+    return _sz_edit_distance_wagner_fisher_serial(a, a_length, b, b_length, bound, sz_true_k, alloc);
+}
+
 SZ_DYNAMIC sz_ssize_t sz_alignment_score(sz_cptr_t a, sz_size_t a_length, sz_cptr_t b, sz_size_t b_length,
                                          sz_error_cost_t const *subs, sz_error_cost_t gap,
                                          sz_memory_allocator_t *alloc) {
@@ -302,3 +324,14 @@ SZ_DYNAMIC sz_cptr_t sz_rfind_char_not_from(sz_cptr_t h, sz_size_t h_length, sz_
     sz_charset_invert(&set);
     return sz_rfind_charset(h, h_length, &set);
 }
+
+sz_u64_t _sz_random_generator(void *empty_state) {
+    sz_unused(empty_state);
+    return (sz_u64_t)rand();
+}
+
+SZ_DYNAMIC void sz_generate(sz_cptr_t alphabet, sz_size_t alphabet_size, sz_ptr_t result, sz_size_t result_length,
+                            sz_random_generator_t generator, void *generator_user_data) {
+    if (!generator) generator = _sz_random_generator;
+    sz_generate_serial(alphabet, alphabet_size, result, result_length, generator, generator_user_data);
+}

package/stringzilla/cli/README.md

@@ -1,4 +1,28 @@
-# SIMD-accelerate CLI utilities based on StringZilla
+# SIMD-accelerated CLI utilities based on StringZilla
+
+This section of the project is pretty much a work in progress.
+The goal is to provide a set of command-line utilities that:
+
+- ✅ that benefit the most from SIMD instructions, 
+- ✅ rely solely on core StringZilla functionality,
+- ✅ works the same on Linux, MacOS, and Windows.
+
+Other utilities are, of course, welcome to use StringZilla but may not be good candidates for this repository.
+To install, pull the Python package from PyPi:
+
+```bash
+pip install stringzilla
+```
+
+Currently implemented:
+
+- `sz_wc`: 3x faster `wc` word count.
+- `sz_split`: 4x faster `splt` file splitting.
+
+What other interfaces should be added?
+Levenshtein distances?
+Fuzzy search?
+Are there common alternatives to mimic?
 
 ## `wc`: Word Count
 
@@ -6,15 +30,15 @@ The `wc` utility on Linux can be used to count the number of lines, words, and b
 Using SIMD-accelerated character and character-set search, StringZilla, even with slow SSDs, it can be noticeably faster.
 
 ```bash
-$ time wc enwik9.txt 
-  13147025  129348346 1000000000 enwik9.txt
+$ time wc enwik9.txt
+  13147025 129348346 1000000000 enwik9.txt
 
 real    0m3.562s
 user    0m3.470s
 sys     0m0.092s
 
-$ time cli/wc.py enwik9.txt
-13147025 139132610 1000000000 enwik9.txt
+$ time sz_wc enwik9.txt
+  13147025 139132610 1000000000 enwik9.txt # Note: different word count, WIP
 
 real    0m1.165s
 user    0m1.121s
@@ -33,16 +57,9 @@ real    0m6.424s
 user    0m0.179s
 sys     0m0.663s
 
-$ time cli/split.py 100000 enwik9.txt ...
+$ time sz_split -l 100000 enwik9.txt ...
 
 real    0m1.482s
 user    0m1.020s
 sys     0m0.460s
 ```
-
----
-
-What other interfaces should be added?
-
-- Levenshtein distances?
-- Fuzzy search?