rankops 0.1.20__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (48) hide show
  1. rankops-0.1.20/.cursor/rules/integration-priorities.mdc +104 -0
  2. rankops-0.1.20/.cursor/rules/super-workspace-context.mdc +22 -0
  3. rankops-0.1.20/.cursor/rules/tekne-core.mdc +212 -0
  4. rankops-0.1.20/.cursor/rules/tekne-module-taxonomy.mdc +30 -0
  5. rankops-0.1.20/.github/settings.yml +21 -0
  6. rankops-0.1.20/.github/workflows/ci.yml +36 -0
  7. rankops-0.1.20/.github/workflows/publish-pypi.yml +101 -0
  8. rankops-0.1.20/.gitignore +6 -0
  9. rankops-0.1.20/Cargo.toml +40 -0
  10. rankops-0.1.20/LICENSE-APACHE +85 -0
  11. rankops-0.1.20/LICENSE-MIT +21 -0
  12. rankops-0.1.20/PKG-INFO +342 -0
  13. rankops-0.1.20/README.md +204 -0
  14. rankops-0.1.20/SECURITY.md +22 -0
  15. rankops-0.1.20/examples/diversity.rs +72 -0
  16. rankops-0.1.20/examples/evaluate.rs +368 -0
  17. rankops-0.1.20/examples/fusion.rs +92 -0
  18. rankops-0.1.20/examples/hybrid_search.rs +241 -0
  19. rankops-0.1.20/examples/pareto_rerank.rs +118 -0
  20. rankops-0.1.20/examples/rerank_maxsim.rs +67 -0
  21. rankops-0.1.20/proptest-regressions/rerank/colbert.txt +7 -0
  22. rankops-0.1.20/pyproject.toml +55 -0
  23. rankops-0.1.20/rankops-python/.python-version +2 -0
  24. rankops-0.1.20/rankops-python/Cargo.lock +147 -0
  25. rankops-0.1.20/rankops-python/Cargo.toml +24 -0
  26. rankops-0.1.20/rankops-python/README.md +312 -0
  27. rankops-0.1.20/rankops-python/pytest.ini +10 -0
  28. rankops-0.1.20/rankops-python/rankops/__init__.py +3 -0
  29. rankops-0.1.20/rankops-python/rankops/py.typed +3 -0
  30. rankops-0.1.20/rankops-python/rankops.pyi +397 -0
  31. rankops-0.1.20/rankops-python/src/lib.rs +1234 -0
  32. rankops-0.1.20/rankops-python/tests/__init__.py +2 -0
  33. rankops-0.1.20/rankops-python/tests/test_fusion.py +462 -0
  34. rankops-0.1.20/src/adapt.rs +223 -0
  35. rankops-0.1.20/src/diagnostics.rs +717 -0
  36. rankops-0.1.20/src/lib.rs +5291 -0
  37. rankops-0.1.20/src/pipeline.rs +531 -0
  38. rankops-0.1.20/src/proptests.rs +375 -0
  39. rankops-0.1.20/src/rerank/colbert.rs +2514 -0
  40. rankops-0.1.20/src/rerank/diversity.rs +1380 -0
  41. rankops-0.1.20/src/rerank/embedding.rs +406 -0
  42. rankops-0.1.20/src/rerank/explain.rs +742 -0
  43. rankops-0.1.20/src/rerank/matryoshka.rs +538 -0
  44. rankops-0.1.20/src/rerank/mod.rs +147 -0
  45. rankops-0.1.20/src/rerank/quantization.rs +376 -0
  46. rankops-0.1.20/src/rerank/scoring.rs +881 -0
  47. rankops-0.1.20/src/rerank/simd.rs +3222 -0
  48. rankops-0.1.20/src/validate.rs +300 -0
@@ -0,0 +1,104 @@
1
+ ---
2
+ title: "Integration Priorities"
3
+ id: integration-priorities
4
+ description: "Guidelines for how Tekne repositories interact and point to canonical overview"
5
+ globs: ["Cargo.toml", "evals/arch/dev_repos_overview.html", "evals/arch/**/*.html", "docs/**/*.md", "README.md"]
6
+ alwaysApply: true
7
+ ---
8
+
9
+ # Integration Priorities: What Actually Matters
10
+
11
+ ## Canonical integration view (single HTML artifact)
12
+
13
+ **Canonical file**: `evals/arch/dev_repos_overview.html`
14
+
15
+ **Rule**: If you need to understand or explain how repos interact, update that one HTML page instead of creating a new report page.
16
+
17
+ **It should contain**:
18
+ - **Levels (L0–L6)**: group repos by the Tekne stack layers (conceptual).
19
+ - **Repo-level graph**: integration edges (git deps, `[patch]` overrides, and the local “super-workspace”).
20
+ - **Crate-level graph**: the dev workspace view (“what Cargo sees”) plus a small conceptual edge set, and a mismatch table.
21
+ - **Tekne facade table**: feature ↔ crate mapping.
22
+ - **Recent activity + size**: the few biggest disk drivers + a short “what changed recently” list.
23
+ - **Axes view**: call out which repos are Tekne (L0–L6) vs Agents (A0–A3) vs Governance (G0–G5), since these are different partial orders.
24
+
25
+ **Links**:
26
+ - `tekne-core.mdc` defines the *meaning* of levels (L0–L6).
27
+ - `tekne-module-taxonomy.mdc` assesses doc quality by module; it is not a dependency graph.
28
+
29
+ ## The Ecosystem at a Glance
30
+
31
+ ```
32
+ PRODUCTION SCALE MATH PRIMITIVES
33
+ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
34
+ anno 371.3K LOC 7531 tests equi 17 LOC 0 tests
35
+ cerno 90.5K LOC 2627 tests fynch 1.5K LOC 53 tests
36
+ hop 117.8K LOC 2410 tests walk 234 LOC 0 tests
37
+ innr 4.5K LOC 153 tests helm 21 LOC 0 tests
38
+ jin 29.8K LOC 367 tests hyp 2K LOC 51 tests
39
+ subsume 14.5K LOC 282 tests kuji 738 LOC 15 tests
40
+ tier 4.7K LOC 76 tests lapl 968 LOC 13 tests
41
+ webs 13.9K LOC 253 tests law 38 LOC 0 tests
42
+ weigh 4 LOC 0 tests liga 12 LOC 0 tests
43
+ logp 567 LOC 10 tests
44
+ pare 403 LOC 3 tests
45
+ propago 4.7K LOC 0 tests
46
+ qig 148 LOC 4 tests
47
+ ritu 49 LOC 0 tests
48
+ rkhs 1.3K LOC 23 tests
49
+ rmt 230 LOC 7 tests
50
+ textprep 197 LOC 3 tests
51
+ sbits 736 LOC 8 tests
52
+ sign 15 LOC 0 tests
53
+ slabs 2.3K LOC 82 tests
54
+ telic 31 LOC 0 tests
55
+ wass 691 LOC 13 tests
56
+ ```
57
+
58
+ ## The Real Problems
59
+
60
+ ### Problem 1: Differentiable ranking should stay centralized
61
+ `fynch` is the single canonical crate for differentiable sort/rank/top-k.
62
+
63
+ ### Problem 2: fynch/metrics vs cerno-eval Overlap
64
+ Keep **core rank-based** metrics in one place (e.g. `fynch::metrics`), and keep domain adapters (IR / KGE / NED) in their domain crates.
65
+
66
+ ### Problem 3: New Math Crates Are Islands
67
+ The “islands” problem is missing usage/tests, not missing dependency edges. Ensure integration tests verify that combining components improves outcomes versus using them individually.
68
+
69
+ ### Problem 4: Hypha integration
70
+ `hypha` provides the P2P substrate. `bop` (Agents) should eventually run *over* `hypha` for distributed coordination. `hypha-knowledge` should leverage `jin` (L3) for vector storage to avoid duplication. `recal` provides the agent's long-term memory via FTS5/vector retrieval.
71
+
72
+ ### Problem 5: Graph Layer Split
73
+ Keep `lattix` as the low-level graph core (L3/L4) and `webs` as the high-level knowledge graph layer (L5). `webs` consumes `lattix` types.
74
+
75
+ ### Problem 6: Cross-axis coupling creep
76
+ The most common structural regression is accidental coupling where:
77
+ - Tekne crates start depending on Agents or Governance crates, making the substrate “application-shaped”
78
+ - Governance becomes a compile-time cross-cutting dependency instead of a gate/eval layer
79
+
80
+ Prefer governance as *inputs/outputs* (policy decisions, eval reports) over ubiquitous imports.
81
+
82
+ ### Problem 7: Sparse lexical retrieval has no clear “low-level home”
83
+ Right now, **BM25/TF‑IDF + (learned) sparse retrieval** lives in `cerno-retrieve` (L5 Domain), but the underlying **inverted-index engine** shape belongs lower:
84
+
85
+ - **L1/L1.5**: scoring math + tokenization/normalization (`textprep`)
86
+ - **L3**: inverted index / segments / deletes / compaction / persistence format
87
+ - **L5**: pipelines (BM25 → rerank), hybrid orchestration, adapters
88
+
89
+ For a publishable, reusable L3 “sparse index” crate, borrow the *constraints-first* boundary from BurntSushi’s Nakala plan:
90
+ - keep tokenization pluggable (don’t bake in language-specific processing)
91
+ - keep query model small (boolean term queries are enough to start)
92
+ - avoid “fields/schema” early (multi-set of terms is the core)
93
+ - index-only (don’t store document content)
94
+ - handle deletes + segment merges as owned complexity
95
+
96
+ Also keep the candidate-generation contract explicit:
97
+ - **No false negatives** for the index/planner path: it may return false positives, but must not exclude a true match.
98
+ - **Planner bailouts are required**: if the query is too broad, it must be legal to fall back to exhaustive scanning.
99
+
100
+ Evidence/anchors:
101
+ - ripgrep indexing RFC (`BurntSushi/ripgrep#1497`) and implementation discussion (`BurntSushi/ripgrep#1518`) are a concrete example of these constraints applied to regex candidate generation (n-gram index + verification).
102
+ - Russ Cox’s trigram-index writeup is the canonical mental model for regex → boolean n-gram query → postings → verify.
103
+
104
+ Reference: `https://raw.githubusercontent.com/BurntSushi/nakala/master/doc/PLAN.md`
@@ -0,0 +1,22 @@
1
+ ---
2
+ description: Super-workspace context discipline (repo truth vs editor hints)
3
+ globs: ["**/*"]
4
+ alwaysApply: true
5
+ ---
6
+
7
+ ## Invariant: repo path is truth
8
+
9
+ This workspace is a super-workspace containing many independent repos.
10
+
11
+ - Treat `open_and_recently_viewed_files` as a UI hint only.
12
+ - Never assume the currently open file indicates the repo/task being worked on.
13
+
14
+ ## Practical workflow
15
+
16
+ - When a task is repo-scoped, first name the repo directory explicitly (e.g. `anno/`, `cerno/`, `wass/`, `devdev/tooling/dev-top/`).
17
+ - Prefer pinning the repo root with `@Folders` before making multi-file changes.
18
+ - Scope searches to the chosen repo directory (avoid whole-workspace scans unless explicitly needed).
19
+
20
+ ## Drift check (cheap)
21
+
22
+ If the only open file(s) are outside the named repo, call that out and ask to pin the correct folder/file(s) before proceeding.
@@ -0,0 +1,212 @@
1
+ ---
2
+ description: PRIMARY INVARIANTS for the Tekne Stack (L0-L6). This is the single source of truth for architecture and safety.
3
+ globs: ["**/*.rs", "**/Cargo.toml"]
4
+ alwaysApply: true
5
+ ---
6
+
7
+ # Tekne Core Invariants (L0-L6)
8
+
9
+ ## 0.5 Dev Workspace Reality (this folder is not a repo)
10
+
11
+ This Cursor workspace (the folder you opened in Cursor) is a **super-workspace** directory:
12
+
13
+ - **Not a git repo**: do not assume `git` operations make sense at the workspace root.
14
+ - **Many independent projects live side-by-side**: most subdirectories are their own repos (or archives), with their own history and tooling.
15
+ - **One root Cargo workspace exists**: `Cargo.toml` at the root defines a *virtual* Rust workspace that includes a curated subset of crates across subdirectories.
16
+ - **Cargo.lock is not present at the root**: dependency stability comes from per-crate `Cargo.lock` files (where used) and pinned workspace dependency versions in the root `Cargo.toml`.
17
+
18
+ Implication: when reasoning about “the project”, be explicit about *which* layer you mean:
19
+ \[
20
+ \text{root dev folder} \;\supset\; \text{root Cargo workspace members} \;\supset\; \text{Tekne stack subset (one organizing view)}.
21
+ \]
22
+
23
+ ## 0. The Principle of Economy of Thought
24
+ **Goal**: Reduce the reasoning required to understand the system.
25
+ * **Framing over Computation**: Time spent finding the right mathematical framing (e.g. Invariants) beats time spent debugging executions.
26
+ * **Separation of Concerns**:
27
+ * **Efficiency** is settled in L0 (Iron).
28
+ * **Correctness** is settled in L1 (Logic).
29
+ * **Orchestration** is settled in L6 (Mesh).
30
+ * **The Five-Line Rule**: If a core logic argument or function body exceeds ~5 lines, reconsider the framing.
31
+
32
+ ## 1. The Layering Invariant (DAG Hard-Stop)
33
+ The Tekne Stack is a Directed Acyclic Graph (DAG). **No lower layer may depend on a higher layer.**
34
+
35
+ | Layer | Name | Domain | Responsibility |
36
+ | :--- | :--- | :--- | :--- |
37
+ | **L0** | **Iron** | Hardware/SIMD | Raw performance, vector similarity ($Cosine$, $L_2$). |
38
+ | **L1** | **Logic** | Math Primitives | Scalar measures: Entropy, Divergences ($KL$, $JS$). |
39
+ | **L2** | **Algos** | Composite | Differentiable sorting, optimal transport, kernels. |
40
+ | **L3** | **Structures** | Aggregators | ANN Indices (`jin`), Hierarchy/aggregation (`tier`), Graph core (`lattix`). |
41
+ | **L4** | **Embeddings** | Representations | Geometric boxes (`subsume`), Statistical mirror (`weigh`). |
42
+ | **L5** | **Domain** | Systems/NLP/IR | NER/Coref (`anno`), Retrieval (`cerno`), Knowledge Graphs (`webs`). |
43
+ | **L6** | **Orch** | Multi-Agent/Mesh | Retrieval/pipeline (`hop`), coordination substrate (`hypha`). |
44
+
45
+ * **Private Gravity**: If a higher-level system implements reusable logic, push it *down* to L0-L2.
46
+ * **Enforcement**: Prefer automated checks, but don’t assume a verifier exists.
47
+ If enforcement tooling is missing, treat the root `Cargo.toml` membership list as the source of truth.
48
+
49
+ ## 1.5 Multiple “Virtual Workspaces” Can Coexist
50
+
51
+ `tekne` is one coherent naming/stack view used by the root Cargo workspace, but it is not the only possible organizing principle.
52
+
53
+ - It’s acceptable to have other coherent sub-ecosystems (e.g., “Scholar stack”, “Infra ops”, editor plugins) living in the same dev directory.
54
+ - Do not force everything into Tekne if the dependency DAG / semantics don’t fit.
55
+
56
+ ## 1.25 Three Parallel DAGs (Math/System vs Agents vs Governance)
57
+
58
+ The “Tekne stack” (L0–L6) is a **math/system DAG**. Agent/tooling and governance are **different axes**.
59
+
60
+ - **Math/System DAG**: operators, representations, and systems (e.g. `innr → logp → fynch → ... → anno/cerno/webs`).
61
+ - **Agents/Tooling DAG**: agent contract engine + orchestration apps (e.g. `axi → bop`, which then *uses* the math/system crates).
62
+ - **Governance DAG**: constraints/policy/oversight that can apply to both agents and systems, without being “higher math”.
63
+
64
+ This avoids category errors like “`axi` should be L1”: agent contracts are not math primitives.
65
+
66
+ ### Why Tekne is “special”
67
+
68
+ Tekne is special in one sense: it is the **reusable substrate DAG**.
69
+
70
+ - If something is in Tekne, it should be *plausibly reusable* outside a single application.
71
+ - Therefore Tekne is the place where we are strictest about:
72
+ - dependency direction (no higher → lower)
73
+ - Unicode correctness at boundaries (character offsets)
74
+ - stability of core types and invariants
75
+
76
+ Agents and Governance can be equally rigorous, but they are *different* partial orders.
77
+
78
+ ### Agent/Tooling Tree (A0–A3)
79
+
80
+ Use this when you want a *different* partial order than Tekne’s L0–L6:
81
+
82
+ | Level | Name | Responsibility | Examples |
83
+ | :--- | :--- | :--- | :--- |
84
+ | **A0** | **Contracts** | Typed tool contracts, schema/repair, run traces | `axi` |
85
+ | **A1** | **Adapters** | Providers/transports (LLM, MCP, storage) | (often lives inside `axi`/apps initially) |
86
+ | **A2** | **Orchestration Apps** | The runnable agent application layer | `bop` |
87
+ | **A3** | **Operations** | Deployment, cost, monitoring | (repo-specific) |
88
+
89
+ ### Governance Tree (G0–G5)
90
+
91
+ Governance is its own axis (not “L7–L12 of Tekne”):
92
+
93
+ | Level | Name | Responsibility | Repo |
94
+ | :--- | :--- | :--- | :--- |
95
+ | **G0** | **Adjudication** | deciding / evaluating / ruling | `law` |
96
+ | **G1** | **Steering** | policy + strategy control | `helm` |
97
+ | **G2** | **Equilibrium** | constraints / balance / alignment | `equi` |
98
+ | **G3** | **Interface** | representation boundaries / signaling | `sign` |
99
+ | **G4** | **Federation** | multi-party coordination contracts | `liga` |
100
+ | **G5** | **Telos** | goals/purpose/intention | `telic` |
101
+
102
+ ### Cross-axis dependency rules (compile-time vs “gates”)
103
+
104
+ **Compile-time dependencies (Cargo `dependencies`)**:
105
+
106
+ - **Allowed**:
107
+ - Agents → Tekne (agents can use substrate)
108
+ - Governance → Tekne (governance can use substrate for scoring/metrics/types)
109
+ - Apps (A2) → Governance (apps may call adjudicators/steering code)
110
+ - **Forbidden by default**:
111
+ - Tekne → Agents
112
+ - Tekne → Governance
113
+ - Governance → Agents (especially Governance → A2 apps)
114
+
115
+ **Why**: if Tekne depends on Agents/Governance, the substrate becomes “application-shaped” and stops being substrate.
116
+
117
+ **Governance should often be a gate, not a dependency**:
118
+
119
+ - Prefer governance as *inputs/outputs* and pipeline checks:
120
+ - plan/request → policy decision
121
+ - run trace → evaluation/audit
122
+ - Avoid “sprinkling governance imports everywhere” as a cross-cutting compile-time dependency, which tends to reintroduce cycles and coupling.
123
+
124
+ ## 2. The Naming Invariant (Structure > Style)
125
+
126
+ **Naming Strategy**: Names should make the *structure* easier to see.
127
+
128
+ \[
129
+ \text{good name} \;\approx\; \text{a cheap index into the right invariants.}
130
+ \]
131
+
132
+ ### Primary Preference: Structural Resonance
133
+
134
+ - Prefer names that align with the *true symmetry* of the object:
135
+ - distance / similarity / norm / kernel / transport / rank / conformal / graph / lattice / contract / covenant / ledger
136
+ - Prefer names that let readers guess the constraints:
137
+ - e.g. “rank fusion” vs “rerank” vs “retrieve” are distinct operators in a pipeline; names should preserve that.
138
+ - Cross-language is encouraged when it improves precision (sometimes the original term is the most exact handle).
139
+
140
+ ### Secondary Preference: Etymology (cross-cultural only when helpful)
141
+
142
+ - Culture is not the goal; *precision* is.
143
+ - Use Sanskrit/Arabic/Old Norse/etc. when it carries the exact nuance for governance or agency concepts.
144
+ - Use Latin/Greek/English when it better matches the mathematical framing.
145
+
146
+ ### Ecosystem/Prefix Coherence (important)
147
+
148
+ - If a set of crates is meant to be used together, prefer a shared prefix.
149
+ - IR stack: prefer `cerno-*` as the canonical prefix (retrieve/fuse/rerank/eval/train + facade `cerno`).
150
+ - Text chunking: prefer `slabs` as canonical (avoid dual naming like `slabs` vs `slabs`).
151
+
152
+ ### crates.io Policy (mandatory, explicit)
153
+
154
+ Always check crates.io **before** selecting a name (don’t guess).
155
+
156
+ - If a name is **taken** on crates.io (HTTP 200), treat it as **unavailable** unless:
157
+ - the crate is permanently `publish = false`, and
158
+ - we accept that the name is *internal-only* (no future publish path).
159
+
160
+ This prevents “pretty internal names” that block publication later.
161
+
162
+ ### Consistency Rule: one concept, one name
163
+
164
+ - Avoid “ghost prior names” lingering in code/docs (e.g., `colligo` remnants inside `hop`).
165
+ - Avoid having directory name, package name, and crate name drift.
166
+ - If we keep an old directory during migration, document it and fix it quickly.
167
+
168
+ ## 3. The Meta-ND Invariant (Multi-Dimensional Metadata)
169
+ **Goal**: Unified handling of high-dimensional metadata and structures.
170
+ * **Dimensional Rigor**: Systems dealing with N-dimensional data (e.g., `subsume` boxes, `innr` vectors) must explicitly track and leverage metadata dimensions.
171
+ * **Subsumption Theory**: Use formal subsumption/entailment theory to manage metadata hierarchies across the stack.
172
+ * **Representation**: Metadata should be treated as a first-class N-dimensional object, not a flat bag of tags.
173
+
174
+ ## 4. Type Hierarchy & Invariants
175
+ * **Canonical Flows**: Signal (Detection) → Track (Coreference) → Identity (Linking).
176
+ * **Numerical Stability**: Use `NORM_EPSILON` ($10^{-9}$) for zero-norm handling.
177
+ * **Algebraic Closure**: Intersection operations (e.g. in L4) must return the same type.
178
+
179
+ ## 3. Engineering Rigor
180
+ * **Unicode**: ALWAYS use character offsets, NEVER byte offsets for text slicing.
181
+ * **SIMD**: Prefer `innr` for vector operations with automatic runtime dispatch.
182
+ * **Fast Math**: Use reciprocal square root ($rsqrt$) + Newton-Raphson for throughput.
183
+ * **Math**: Cite foundational papers (TAOCP style).
184
+
185
+ ## 4. Documentation vs. Process Logs
186
+ * **Chat for Process**: Do not write temporary markdown files for progress. Reply in chat.
187
+ * **Lecture Notes**: Review `docs/lectures/*.md` to pick up stack-wide habits.
188
+ * **Visual Rigor**: Visualizations must use real data from actual execution. fit distributions.
189
+
190
+ ## 4.5 Memory Recall Discipline (Cursor history is data)
191
+
192
+ When intent or prior decisions matter (renames, merges, “publish vs private”, repo cleanup), **do not guess**.
193
+ Treat prior chat + rule history as a dataset and re-derive intent from it.
194
+
195
+ - **Preferred**: use `iksh` to inspect Cursor’s DB-backed chat history and rule memory:
196
+ - list conversation ids: `cargo run -p iksh -- chats --limit N`
197
+ - list with previews (recommended): `cargo run -p iksh -- chats --limit N --meta --format text --preview-chars 160`
198
+ - list as JSON (for tooling): `cargo run -p iksh -- chats --limit N --format json --preview-chars 160`
199
+ - reconstruct a conversation: `cargo run -p iksh -- convo <conversation-uuid>`
200
+ - search for keywords: `cargo run -p iksh -- db-search-values <needle> --limit N`
201
+ - **Fallback**: consult the saved agent transcript file(s) when the DB doesn’t contain what we need.
202
+
203
+ Goal: minimize “phantom intent” errors by making recall a first-class step.
204
+
205
+ ## 5. Repository & Tool Safety
206
+ * **Isolation**: Never create a new repo inside another repo.
207
+ * **Timeout Handling**: If a tool call fails/times out, narrow the scope. Do not repeat.
208
+ * **Local Resolution**: Use `[patch.crates-io]` in root for development speed.
209
+
210
+ ## 6. Git Discipline (Staging)
211
+
212
+ Default to explicit staging (e.g. `git add <paths>`). Avoid `git add -A` unless the user explicitly requests “stage everything”.
@@ -0,0 +1,30 @@
1
+ ---
2
+ title: "Tekne Module Taxonomy"
3
+ id: tekne-module-taxonomy
4
+ description: "Assessment of documentation quality and purpose by module"
5
+ alwaysApply: false
6
+ ---
7
+
8
+ # Tekne Module Taxonomy
9
+
10
+ Assessment of documentation quality and pedagogical readiness across the stack.
11
+
12
+ ## Grade A: Pedagogical Gold Standard (TAOCP / Mackay Style)
13
+
14
+ | Module | Lines | Additions |
15
+ |:---|:---|:---|
16
+ | `jin/hnsw` | ~210 | Mathematical invariant check, probability distribution |
17
+ | `tier/kmeans` | ~110 | Hard vs soft, GMM math, linkage table |
18
+ | `hyp` (lib.rs) | ~80 | **Model selection table**, when NOT to use |
19
+ | `innr` (lib.rs) | ~60 | **Function selection table**, SIMD dispatch |
20
+
21
+ ## Verification Checklist
22
+
23
+ For each module, verify:
24
+ - [ ] Problem statement (what does this solve?)
25
+ - [ ] Why this approach (what's the insight?)
26
+ - [ ] Comparison to alternatives (when to use/not use)
27
+ - [ ] Parameters with effects (not just names)
28
+ - [ ] Complexity (time/space)
29
+ - [ ] Working code example
30
+ - [ ] References (papers, not just URLs)
@@ -0,0 +1,21 @@
1
+ # Managed by repository-settings/app
2
+ # https://github.com/repository-settings/app
3
+
4
+ repository:
5
+ description: "Operations on ranked lists: fusion (RRF, CombMNZ, Borda, DBSF), reranking (MaxSim/ColBERT, MMR, DPP, Matryoshka). Pairs with rankfns (scoring kernels)."
6
+ homepage: "https://docs.rs/rankops"
7
+ private: false
8
+ has_issues: true
9
+ has_wiki: false
10
+ has_downloads: true
11
+ default_branch: main
12
+ allow_squash_merge: true
13
+ allow_merge_commit: true
14
+ allow_rebase_merge: true
15
+ delete_branch_on_merge: true
16
+
17
+ topics:
18
+ - information-retrieval
19
+ - reranking
20
+ - rank-fusion
21
+ - rust
@@ -0,0 +1,36 @@
1
+ name: ci
2
+
3
+ on:
4
+ push:
5
+ branches: ["main", "master"]
6
+ pull_request:
7
+
8
+ permissions:
9
+ contents: read
10
+
11
+ env:
12
+ CARGO_TERM_COLOR: always
13
+
14
+ jobs:
15
+ rust:
16
+ runs-on: ubuntu-latest
17
+ steps:
18
+ - uses: actions/checkout@v4
19
+ - uses: dtolnay/rust-toolchain@stable
20
+ with:
21
+ components: rustfmt, clippy
22
+ - uses: Swatinem/rust-cache@v2
23
+
24
+ - name: fmt
25
+ run: cargo fmt --check
26
+
27
+ - name: clippy
28
+ run: cargo clippy --all-targets --all-features -- -D warnings
29
+
30
+ - name: test
31
+ run: cargo test --all-targets --all-features
32
+
33
+ - name: build rankops-wasm (wasm32)
34
+ run: |
35
+ rustup target add wasm32-unknown-unknown
36
+ cargo build --manifest-path crates/rankops-wasm/Cargo.toml --target wasm32-unknown-unknown
@@ -0,0 +1,101 @@
1
+ name: publish-pypi
2
+
3
+ on:
4
+ push:
5
+ tags:
6
+ - "python-v*"
7
+
8
+ permissions:
9
+ contents: read
10
+
11
+ jobs:
12
+ linux:
13
+ runs-on: ubuntu-latest
14
+ strategy:
15
+ matrix:
16
+ target: [x86_64, aarch64]
17
+ steps:
18
+ - uses: actions/checkout@v4
19
+ - uses: actions/setup-python@v5
20
+ with:
21
+ python-version: "3.12"
22
+ - name: Build wheels
23
+ uses: PyO3/maturin-action@v1
24
+ with:
25
+ target: ${{ matrix.target }}
26
+ args: --release --out dist --find-interpreter
27
+ manylinux: auto
28
+ working-directory: rankops-python
29
+ - uses: actions/upload-artifact@v4
30
+ with:
31
+ name: wheel-linux-${{ matrix.target }}
32
+ path: rankops-python/dist/*.whl
33
+
34
+ macos:
35
+ runs-on: macos-latest
36
+ strategy:
37
+ matrix:
38
+ target: [x86_64, aarch64]
39
+ steps:
40
+ - uses: actions/checkout@v4
41
+ - uses: actions/setup-python@v5
42
+ with:
43
+ python-version: "3.12"
44
+ - name: Build wheels
45
+ uses: PyO3/maturin-action@v1
46
+ with:
47
+ target: ${{ matrix.target }}-apple-darwin
48
+ args: --release --out dist --find-interpreter
49
+ working-directory: rankops-python
50
+ - uses: actions/upload-artifact@v4
51
+ with:
52
+ name: wheel-macos-${{ matrix.target }}
53
+ path: rankops-python/dist/*.whl
54
+
55
+ windows:
56
+ runs-on: windows-latest
57
+ steps:
58
+ - uses: actions/checkout@v4
59
+ - uses: actions/setup-python@v5
60
+ with:
61
+ python-version: "3.12"
62
+ architecture: x64
63
+ - name: Build wheels
64
+ uses: PyO3/maturin-action@v1
65
+ with:
66
+ args: --release --out dist --find-interpreter
67
+ working-directory: rankops-python
68
+ - uses: actions/upload-artifact@v4
69
+ with:
70
+ name: wheel-windows-x64
71
+ path: rankops-python/dist/*.whl
72
+
73
+ sdist:
74
+ runs-on: ubuntu-latest
75
+ steps:
76
+ - uses: actions/checkout@v4
77
+ - name: Build sdist
78
+ uses: PyO3/maturin-action@v1
79
+ with:
80
+ command: sdist
81
+ args: --out dist
82
+ working-directory: rankops-python
83
+ - uses: actions/upload-artifact@v4
84
+ with:
85
+ name: sdist
86
+ path: rankops-python/dist/*.tar.gz
87
+
88
+ publish:
89
+ needs: [linux, macos, windows, sdist]
90
+ runs-on: ubuntu-latest
91
+ environment: pypi
92
+ permissions:
93
+ id-token: write
94
+ steps:
95
+ - uses: actions/download-artifact@v4
96
+ with:
97
+ path: dist
98
+ merge-multiple: true
99
+ - uses: pypa/gh-action-pypi-publish@release/v1
100
+ with:
101
+ packages-dir: dist/
@@ -0,0 +1,6 @@
1
+ target/
2
+ **/target/
3
+ .DS_Store
4
+ Cargo 2.toml
5
+ rankops-python/Cargo.lock.fastembed_cache/
6
+ .fastembed_cache/
@@ -0,0 +1,40 @@
1
+ [package]
2
+ name = "rankops"
3
+ version = "0.1.4"
4
+ edition = "2021"
5
+ license = "MIT OR Apache-2.0"
6
+ description = "Operations on ranked lists: fusion (RRF, Copeland, CombMNZ, DBSF, 14 methods), reranking (MaxSim/ColBERT, MMR, DPP), evaluation (NDCG, MAP, MRR), diagnostics, pipeline builder."
7
+ repository = "https://github.com/arclabs561/rankops"
8
+ readme = "README.md"
9
+ keywords = ["rerank", "rrf", "retrieval", "rankops", "fusion"]
10
+ categories = ["algorithms", "mathematics", "science"]
11
+
12
+ [dependencies]
13
+ serde = { version = "1.0", optional = true, features = ["derive"] }
14
+ # SIMD primitives for reranking (MaxSim/ColBERT)
15
+ innr = { version = "0.1.9", optional = true }
16
+ # Hierarchical clustering for ColBERT (optional)
17
+ kodama = { version = "0.3", optional = true }
18
+ # Pareto frontier for multi-objective reranking (optional)
19
+ pare = { version = "0.2.0", optional = true }
20
+
21
+ [features]
22
+ default = ["rerank"]
23
+ serde = ["dep:serde"]
24
+ # Reranking: MaxSim (ColBERT), diversity, matryoshka — requires innr for SIMD
25
+ rerank = ["dep:innr"]
26
+ # Alias so #[cfg(feature = "innr")] in rerank/simd.rs matches (rerank enables innr)
27
+ innr = ["rerank"]
28
+ # Hierarchical ColBERT; enables kodama for dendrogram clustering
29
+ hierarchical = ["dep:kodama"]
30
+ # Pareto frontier for multi-objective result selection
31
+ pareto = ["dep:pare"]
32
+
33
+ [dev-dependencies]
34
+ criterion = { version = "0.7", features = ["html_reports"] }
35
+ proptest = "1.9"
36
+
37
+
38
+ [package.metadata.docs.rs]
39
+ all-features = true
40
+ rustdoc-args = ["--cfg", "docsrs"]
@@ -0,0 +1,85 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity.
18
+
19
+ "You" (or "Your") shall mean an individual or Legal Entity
20
+ exercising permissions granted by this License.
21
+
22
+ "Source" form shall mean the preferred form for making modifications.
23
+
24
+ "Object" form shall mean any form resulting from mechanical
25
+ transformation or translation of a Source form.
26
+
27
+ "Work" shall mean the work of authorship made available under the License.
28
+
29
+ "Derivative Works" shall mean any work that is based on the Work.
30
+
31
+ "Contribution" shall mean any work of authorship submitted to the Licensor.
32
+
33
+ "Contributor" shall mean Licensor and any Legal Entity on behalf of whom
34
+ a Contribution has been received by Licensor.
35
+
36
+ 2. Grant of Copyright License. Subject to the terms of this License, each
37
+ Contributor grants to You a perpetual, worldwide, non-exclusive, no-charge,
38
+ royalty-free, irrevocable copyright license to reproduce, prepare Derivative
39
+ Works of, publicly display, publicly perform, sublicense, and distribute the
40
+ Work and such Derivative Works in Source or Object form.
41
+
42
+ 3. Grant of Patent License. Subject to the terms of this License, each
43
+ Contributor grants to You a perpetual, worldwide, non-exclusive, no-charge,
44
+ royalty-free, irrevocable patent license to make, have made, use, offer to
45
+ sell, sell, import, and otherwise transfer the Work.
46
+
47
+ 4. Redistribution. You may reproduce and distribute copies of the Work
48
+ provided that You meet the following conditions:
49
+ (a) You must give recipients a copy of this License; and
50
+ (b) You must cause modified files to carry prominent notices; and
51
+ (c) You must retain all copyright, patent, trademark notices; and
52
+ (d) If the Work includes a "NOTICE" file, You must include a readable copy.
53
+
54
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
55
+ any Contribution submitted for inclusion in the Work shall be under the
56
+ terms of this License, without any additional terms or conditions.
57
+
58
+ 6. Trademarks. This License does not grant permission to use the trade
59
+ names, trademarks, service marks, or product names of the Licensor.
60
+
61
+ 7. Disclaimer of Warranty. The Work is provided on an "AS IS" BASIS,
62
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
63
+
64
+ 8. Limitation of Liability. In no event shall any Contributor be liable
65
+ to You for damages, including any direct, indirect, special, incidental,
66
+ or consequential damages.
67
+
68
+ 9. Accepting Warranty or Additional Liability. You may choose to offer
69
+ support, warranty, indemnity, or other liability obligations.
70
+
71
+ END OF TERMS AND CONDITIONS
72
+
73
+ Copyright 2024 arclabs561
74
+
75
+ Licensed under the Apache License, Version 2.0 (the "License");
76
+ you may not use this file except in compliance with the License.
77
+ You may obtain a copy of the License at
78
+
79
+ http://www.apache.org/licenses/LICENSE-2.0
80
+
81
+ Unless required by applicable law or agreed to in writing, software
82
+ distributed under the License is distributed on an "AS IS" BASIS,
83
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
84
+ See the License for the specific language governing permissions and
85
+ limitations under the License.