vicinus 0.0.1__tar.gz

vicinus-0.0.1/.gitignore

@@ -0,0 +1,9 @@
+venv/
+.vscode/
+__pycache__/
+**/__pycache__/
+dist/
+build/
+temp/
+**/debug.py
+**.sw?

vicinus-0.0.1/PKG-INFO

@@ -0,0 +1,272 @@
+Metadata-Version: 2.4
+Name: vicinus
+Version: 0.0.1
+Summary: Fuzzy search library from scratch implementing novel NGF/NGF-IDF system.
+Project-URL: Homepage, https://pycelium.com
+Project-URL: Repository, https://github.com/k98kurz/vicinus
+Project-URL: Bug Tracker, https://github.com/k98kurz/vicinus/issues
+Author-email: k98kurz <k98kurz@gmail.com>
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: ISC License (ISCL)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Provides-Extra: docs
+Requires-Dist: autodox>=0.1.16; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# Vicinus
+
+Vicinus is a library for building fuzzy search tools. It includes multiple
+similarity/distance algorithms, an in-memory vector DB, and an optional
+persistent vector DB using my sqloquent ORM package (sqlite3). Made from
+all-organic, artisanal, 100% hand-crafted code with no gen AI (except to write
+some text samples for testing) and no external dependencies.
+
+This library includes several standard algorithms as well as a few novel (to my
+knowledge) constructions: NGF and NGF-IDF, adapted from TF and TF-IDF but using
+N-Grams to construct a comprehensive lexicon (26 letters + 10 numerals). The goal
+of these novel constructions is to provide a lightweight alternative to text
+embedding models and full TF-IDF while preserving utility for fuzzy searching.
+
+("Vicinus" means "neighboring/nearby" in Latin.)
+
+## Status
+
+- [x] Hamming Distance/Similarity
+- [x] Levenshtein Distance/Similarity
+- [x] N-Gram Extraction
+- [x] Jaccard Index
+- [x] Sparse Vector Cosine Similarity
+- [x] NGF (N-Gram Frequency)
+- [x] NGF-IDF (N-Gram Frequency * Inverse Document Frequecy)
+- [ ] Planned Optimizations: Levenshtein, NGF-IDF
+- [ ] In-Memory VectorDB using NGF or NGF-IDF
+- [ ] Persistent VectorDB using NGF or NGF-IDF
+
+Open issues can be tracked [here](https://github.com/k98kurz/vicinus/issues).
+Historical changes can be found in the
+[changelog](https://github.com/k98kurz/vicinus/blob/master/changelog.md).
+
+## Usage
+
+### Installation
+
+```bash
+pip install vicinus
+```
+
+### CLI
+
+This package includes a CLI tool for exporting an agent skill to help clankers
+use this package correctly. (It also serves as decent documentation in general,
+so it might also be worth scanning if you aren't coding with a clanker.)
+
+```bash
+vicinus skill               # prints to stdout
+vicinus skill -o path       # exports to path/vicinus/
+vicinus skill --agent       # exports to .agent/skills/vicinus/
+vicinus skill --claude      # exports to .claude/skills/vicinus/
+vicinus skill --codex       # exports to .agent/skills/vicinus/
+vicinus skill --cursor      # exports to .cursor/skills/vicinus/
+vicinus skill --opencode    # exports to .opencode/skills/vicinus/
+```
+
+The CLI can also be used to export the bundled text samples for experimentation.
+
+```bash
+vicinus samples             # prints all to stdout with filenames
+vicinus samples -l          # prints list of all filenames
+vicinus samples -n name     # prints a specific sample to stdout
+vicinus samples -o path     # exports all files to path/
+```
+
+### Distance/Similarity
+
+There are two distance metrics with corresponding similarity scores Hamming and
+Levenshtein. 
+
+There are additional similarity scores:
+- N-Grams w/ Jaccard Index: 
+- NGF w/ Cosine Similarity
+- NGF-IDF w/ Cosine Similarity
+
+There are also two helper functions:
+- `rank(score_fn: Callable, candidates: list) -> list[tuple[float, int]]`
+- `select(rankings: list[tuple[float, int]], k: int = 4) -> list[tuple[float, int]]`
+
+#### Hamming
+
+Relatively simple and cheap, but not very robust. Not suitable for large texts.
+
+```python
+from vicinus import hamming_distance, hamming_similarity, rank, select
+
+# distance between two texts
+distance = hamming_distance(text1, text2)
+
+# rank by similarity
+corpus = [text1, text2, ...]
+query = "something"
+# rank all by query similarity
+rankings = rank(lambda t: hamming_similarity(query, t), corpus)
+# get top k=2 rankings
+candidates = select(rankings, 2)
+```
+
+#### Levenshtein
+
+More complex and robust than Hamming, but entails recursive computation. Optimized
+by a limited `lru_cache(maxsize=1024)`. Not suitable for large texts.
+
+```python
+from vicinus import levenshtein_distance, levenshtein_similarity, rank, select
+
+# distance between two texts
+distance = levenshtein_distance(text1, text2)
+
+# rank by similarity
+corpus = [text1, text2, ...]
+query = "something"
+# rank all by query similarity
+rankings = rank(lambda t: levenshtein_similarity(query, t), corpus)
+# get top k=2 rankings
+candidates = select(rankings, 2)
+```
+
+#### N-Grams w/ Jaccard Index
+
+Normalize and break text into N-Grams, then use Jaccard Index on the N-Gram sets.
+Scales well but loses cardinality of N-Gram frequencies -- a "bag of words"
+technique. N-Grams should be at least 3 long (trigrams) but can be made longer for
+larger texts. Jaccard Index is a measure of similarity based upon set inclusion of
+N-Grams: shared/total (|intersection| / |union|). Only aphanumeric chars (standard
+Latin lowercase + Arabic numerals) are supported.
+
+```python
+from vicinus import n_grams, jaccard_index
+
+# calculate similarity
+ng1 = n_grams(text1, N=3) # default N
+ng2 = n_grams(text2, N=3)
+similarity = jaccard_index(ng1, ng2) # between 0 and 1.0
+
+# rank by similarity
+corpus = [text1, text2, ...]
+query = "something"
+corpus_ngrams = [n_grams(t) for t in corpus]
+query_ngrams = n_grams(query)
+# rank all by query similarity
+rankings = rank(lambda t: jaccard_index(query_ngrams, t), corpus_ngrams)
+# get top k=2 rankings
+candidates = select(rankings, k=2)
+```
+
+#### NGF w/ Cosine Similarity
+
+Like TF but with N-Grams instead of terms. Includes cardinality of N-Gram
+frequencies within texts but not between texts. Does not require recalculating
+all vectors when a text in the corpus changes. Uses `SparseVector`s for efficiency.
+
+```python
+from vicinus import ngf, ngf_rank, ngf_select
+
+corpus = ["Text 1 is some text...", "Text 2 is another thing...", ...]
+vecs = [ngf(t) for t in corpus]
+query = "text about something"
+# to sort all texts
+rankings = ngf_rank(query, vecs)
+# or select top k=2
+candidates = ngf_select(query, vecs, k=2) # or candidates = select(rankings, 2)
+```
+
+#### NGF-IDF w/ Cosine Similarity
+
+Like TF-IDF but with N-Grams instead of terms. Considers N-Gram frequencies within
+texts and between texts. Requires setup phase that recalculates the IDF for a
+corpus whenever a text changes.
+
+```python
+from vicinus import ngf_idf_setup, ngf_idf_rank, ngf_idf_select
+
+corpus = ["Text 1 is some text...", "Text 2 is another thing...", ...]
+corpus_idf, vecs = ngf_idf_setup(corpus)
+query = "text about something"
+# to sort all texts by cosine similarity
+all_ranked = ngf_idf_rank(query, corpus_idf, vecs)
+# or select the top k=4 (default)
+candidates = ngf_idf_select(query, corpus_idf, vecs)
+```
+
+Note: N=5 resulted in far more accurate rankings for the test vectors than the
+default N=3 for Jaccard Index, NGF, and NGF-IDF. This may require some tuning.
+Higher `N` values lead to sparser vectors and lower cosine similarity/Jaccard
+Index scores, but this appears to be primarily a culling of noise.
+
+### Docs
+
+Documentation generated by [autodox](https://pypi.org/project/autodox) (with one
+fix due to a bug/missing feature in autodox) can be found
+[here](https://github.com/k98kurz/vicinus/blob/master/docs.md).
+
+## Note on Generative AI Use
+
+All code and documentation was written by hand, including the test suite. I used
+mistral and gemma4:e4b (running on my own machine via ollama with some fun bashfu)
+to write the text samples for testing. I also used gemma4:31b via Ollama cloud in
+OpenCode for code review (it caught a few loose ends and documentation issues).
+
+The inspiration for this library came during development of a yet unreleased new
+agentic harness system that I intend to optimize for small, local models: as I
+drafted a spec for a `tool_search` tool as an option to avoid context bloat, I
+initially thought to copy a Hamming distance implementation I wrote a few months
+ago in a different project for fuzzy search. I reviewed the draft spec with
+gemma4:e4b, and it suggested I look at Levenshtein distance and Jaccard Index.
+I then read some formulas on Wikipedia, implemented Levenshtein distance, N-grams,
+and Jaccard index, and then derived the concept of NGF/NGF-IDF independently while
+reflecting upon the code I had just written and the inherent difficulties with
+TF/TF-IDF (i.e. the need to construct an indexed lexicon or use a text embedding
+model).
+
+Code projects like this are too fun to give to clankers. Gotta maintain some joy
+in life.
+
+## Testing
+
+First, clone the repo, set up a virtual env, and install requirements from
+requirements.txt. Then run the following command to run the test suite.
+
+```bash
+pip install -e . # or use uv
+python -m unittest -s discover tests
+```
+
+There are currently 38 tests covering all functionality except the CLI, which is
+tested manually.
+
+## Contributing
+
+Check out the [Pycelium discord server](https://discord.gg/b2QFEJDX69). If you
+experience a problem, please discuss it on the Discord server. All suggestions
+for improvement are also welcome, and the best place for that is also Discord.
+If you experience a bug and do not use Discord, open an issue or discussion on
+Github.
+
+## ISC License
+
+Copyright (c) 2026 Jonathan Voss (k98kurz)
+
+Permission to use, copy, modify, and/or distribute this software
+for any purpose with or without fee is hereby granted, provided
+that the above copyright notice and this permission notice appear in
+all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
+WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
+AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
+CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

vicinus-0.0.1/changelog.md

@@ -0,0 +1,8 @@
+## 0.0.1
+
+- Initial version:
+  - Hamming & Levenshtein distance and similarity
+  - N-Grams w/ Jaccard Index
+  - NGF w/ Cosine Similarity (and `SparseVector`s)
+  - NGF-IDF w/ Cosine Similarity
+  - CLI: export skill and bundled samples

vicinus-0.0.1/docs.md

@@ -0,0 +1,166 @@
+# vicinus
+
+## Classes
+
+### `SparseVector`
+
+Class representing a sparse vector, i.e. omitting every element that equals 0.
+Useful for high dimensionality vectors with mostly 0s.
+
+#### Methods
+
+##### `__init__(data: dict[int, int | float] = None):`
+
+Initialize. Raises `TypeError` for invalid data.
+
+##### `get(index: int = 0) -> int | float | None:`
+
+Get the element at the specified index.
+
+##### `keys() -> set[int]:`
+
+Return the set of indices for which elements exist.
+
+##### `values() -> ValuesView[int | float]:`
+
+Return a view of all elements.
+
+##### `items() -> ItemsView[tuple[int, int | float]]:`
+
+Return an ItemsView for `for i, e in vec.items()` syntax.
+
+##### `norm() -> float:`
+
+Calculate the Euclidean norm of a SparseVector, defined as the square root of
+the dot product of itself (i.e. the sqrt of the sum of the squares of the
+elements). Raises `TypeError` for invalid input.
+
+##### `dot_product(other: SparseVector) -> float:`
+
+Calculate the dot product between two `SparseVector`s. Raises `TypeError` for
+invalid other. Treats unset indices as 0 values mathematically.
+
+##### `cosine_similarity(other: SparseVector) -> float:`
+
+Calculate the cosine similarity with another SparseVector. Raises `TypeError`.
+
+## Functions
+
+### `hamming_distance(str1: str, str2: str, normalize: bool = False) -> int | float:`
+
+Calculate Hamming distance between two strs. If `normalize` is `False`
+(default), distance is returned directly as an int. If `normalize` is set to
+`True`, the distance is normalized to a float proportional to the length of the
+longest string.
+
+### `hamming_similarity(str1: str, str2: str) -> float:`
+
+Hamming similarity: 1 - normalized Hamming distance.
+
+### `rank(score_fn: Callable, candidates: list) -> list[tuple[float, int]]:`
+
+Ranks `candidates` using the `score_fn` callable. Returns a list of tuples in
+form `(score: float, index: int)`, sorted by score descending.
+
+### `select(rankings: list, k: int = 4) -> list[tuple[float, int]]:`
+
+Select the top `k` from the rankings. Sorts beforehand. Assumes similarity
+scores and not distance scores. Raises `TypeError` or `ValueError` for invalid
+inputs.
+
+### `levenshtein_distance(a: str, b: str, normalize: bool = False) -> int|float:`
+
+Calculate Levenshtein distance between two strs. If `normalize` is `False`
+(default), distance is returned directly as an int. If `normalize` is set to
+`True`, the distance is normalized to a float proportional to the average
+length of the strings, clamped to a max of 1.0. Optimized by `@lru_cache`
+decorator -- be sure to use `.lower()` or `' '.join(tokenize())` on strings
+before passing them in to avoid cache fragmentation.
+
+### `levenshtein_similarity(a: str, b: str) -> float:`
+
+Turns the normalized Levenshtein difference metric into a similarity metric by
+substracting it from 1.0.
+
+### `tokenize(text: str) -> list[str]:`
+
+Preprocess some text by casting to lower, removing punctuation, and splitting on
+word boundaries. Returns only alphanumeric tokens.
+
+### `ngf_index(ng: str) -> int:`
+
+Calculate the NGF vector index for a given alphanumeric N-Gram.
+
+### `ng_count(text: str, N: int = 3, ngs: set = None) -> SparseVector:`
+
+Calculate the N-Gram Count of some text. First step in NGF or NGF-IDF analysis,
+where NGF is like TF but with N-Grams instead of whole words. The `N` parameter
+controls N-Gram size. Produces a SparseVector with max length `36^N`, where
+element index is ordered by N-Gram alphanumeric order, e.g. 000, 001, ..., 0a0,
+0a1, ..., zzz.
+
+### `ngf(text: str, N: int = 3, vec: SparseVector = None) -> SparseVector:`
+
+Calculate the N-Gram Frequency of some text. Analogous to TF, but with N-Grams
+instead of whole words. The `N` parameter controls N-Gram size. Produces a
+SparseVector max length `36^N`, where element index is ordered by N-Gram
+alphanumeric order, e.g. 000, 001, ..., 0a0, 0a1, ..., zzz. Pass `vec` result of
+`ng_count` to bypass retokenization of the text and recalculation of N-Grams.
+
+### `ngf_rank(query: str, text_vecs: list, N: int = 3) -> list[tuple[float, int]]:`
+
+Runs NGF cosine similarity between the query and the text vectors. Returns a
+sorted list of form `[(similarity, index)]`.
+
+### `ngf_select(query: str, text_vecs: list, k: int = 4, N: int = 3) -> list[tuple[float, int]]:`
+
+Runs NGF cosine similarity between the query and the text vectors. Returns the
+top `k` candidates in a sorted list of form `[(similarity, index)]`.
+
+### `ngf_idf_setup(corpus: list, N: int = 3) -> tuple[vicinus.sparse_vector.SparseVector, list[vicinus.sparse_vector.SparseVector]]:`
+
+Processes a corpus, creating the vectors for each text in a pipeline: 1) extract
+N-Grams and create NG count vectors; 2) aggregate corpus NG count vector; 3)
+create NGF vectors; 4) calculate the IDF vector; 5) modify NGF vectors by
+multiplying by the IDF vector. Returns the corpus IDF vector as well as the
+NGF-IDF vectors to enable fuzzy search on queries.
+
+### `ngf_idf_query(query: str, corpus_idf: SparseVector, N: int = 3) -> SparseVector:`
+
+Processes a query, calculating the NGF-IDF vector for it using the corpus IDF
+vector.
+
+### `ngf_idf_rank(query: str, corpus_idf: SparseVector, text_vecs: list, N: int = 3) -> list[tuple[float, int]]:`
+
+Runs NGF-IDF cosine similarity between the query and the text vectors. Returns a
+sorted list of form `[(similarity, index)]`.
+
+### `ngf_idf_select(query: str, corpus_idf: SparseVector, text_vecs: list, k: int = 4, N: int = 3) -> list[tuple[float, int]]:`
+
+Runs NGF-IDF cosine similarity between the query and the text vectors. Returns
+the top `k` candidates in a sorted list of form `[(similarity, index)]`.
+
+### `n_grams(text: str, N: int = 3) -> set[str]:`
+
+Extract the N-Grams of a text, defined as the set of unique substrings of the
+given N.
+
+### `jaccard_index(a: set, b: set) -> float:`
+
+Calculate the Jaccard similarity index between two sets, e.g. of N-Grams.
+Defined as the cardinality of the intersection divided by the cardinality of the
+union.
+
+### `list_samples() -> list[str]:`
+
+Returns the list of bundled sample file names.
+
+### `get_sample(name: str) -> str:`
+
+Reads and returns the content of the named sample. Raises FileNotFoundError for
+an invalid name.
+
+### `version() -> str:`
+
+Return the current library version.
+

vicinus-0.0.1/license

@@ -0,0 +1,17 @@
+ISC License
+
+Copyright (c) 2026 Jonathan Voss (k98kurz)
+
+Permission to use, copy, modify, and/or distribute this software
+for any purpose with or without fee is hereby granted, provided
+that the above copyright notice and this permission notice appear in
+all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
+WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
+AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
+CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.