rankops 0.1.20__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- rankops-0.1.20/.cursor/rules/integration-priorities.mdc +104 -0
- rankops-0.1.20/.cursor/rules/super-workspace-context.mdc +22 -0
- rankops-0.1.20/.cursor/rules/tekne-core.mdc +212 -0
- rankops-0.1.20/.cursor/rules/tekne-module-taxonomy.mdc +30 -0
- rankops-0.1.20/.github/settings.yml +21 -0
- rankops-0.1.20/.github/workflows/ci.yml +36 -0
- rankops-0.1.20/.github/workflows/publish-pypi.yml +101 -0
- rankops-0.1.20/.gitignore +6 -0
- rankops-0.1.20/Cargo.toml +40 -0
- rankops-0.1.20/LICENSE-APACHE +85 -0
- rankops-0.1.20/LICENSE-MIT +21 -0
- rankops-0.1.20/PKG-INFO +342 -0
- rankops-0.1.20/README.md +204 -0
- rankops-0.1.20/SECURITY.md +22 -0
- rankops-0.1.20/examples/diversity.rs +72 -0
- rankops-0.1.20/examples/evaluate.rs +368 -0
- rankops-0.1.20/examples/fusion.rs +92 -0
- rankops-0.1.20/examples/hybrid_search.rs +241 -0
- rankops-0.1.20/examples/pareto_rerank.rs +118 -0
- rankops-0.1.20/examples/rerank_maxsim.rs +67 -0
- rankops-0.1.20/proptest-regressions/rerank/colbert.txt +7 -0
- rankops-0.1.20/pyproject.toml +55 -0
- rankops-0.1.20/rankops-python/.python-version +2 -0
- rankops-0.1.20/rankops-python/Cargo.lock +147 -0
- rankops-0.1.20/rankops-python/Cargo.toml +24 -0
- rankops-0.1.20/rankops-python/README.md +312 -0
- rankops-0.1.20/rankops-python/pytest.ini +10 -0
- rankops-0.1.20/rankops-python/rankops/__init__.py +3 -0
- rankops-0.1.20/rankops-python/rankops/py.typed +3 -0
- rankops-0.1.20/rankops-python/rankops.pyi +397 -0
- rankops-0.1.20/rankops-python/src/lib.rs +1234 -0
- rankops-0.1.20/rankops-python/tests/__init__.py +2 -0
- rankops-0.1.20/rankops-python/tests/test_fusion.py +462 -0
- rankops-0.1.20/src/adapt.rs +223 -0
- rankops-0.1.20/src/diagnostics.rs +717 -0
- rankops-0.1.20/src/lib.rs +5291 -0
- rankops-0.1.20/src/pipeline.rs +531 -0
- rankops-0.1.20/src/proptests.rs +375 -0
- rankops-0.1.20/src/rerank/colbert.rs +2514 -0
- rankops-0.1.20/src/rerank/diversity.rs +1380 -0
- rankops-0.1.20/src/rerank/embedding.rs +406 -0
- rankops-0.1.20/src/rerank/explain.rs +742 -0
- rankops-0.1.20/src/rerank/matryoshka.rs +538 -0
- rankops-0.1.20/src/rerank/mod.rs +147 -0
- rankops-0.1.20/src/rerank/quantization.rs +376 -0
- rankops-0.1.20/src/rerank/scoring.rs +881 -0
- rankops-0.1.20/src/rerank/simd.rs +3222 -0
- rankops-0.1.20/src/validate.rs +300 -0
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Integration Priorities"
|
|
3
|
+
id: integration-priorities
|
|
4
|
+
description: "Guidelines for how Tekne repositories interact and point to canonical overview"
|
|
5
|
+
globs: ["Cargo.toml", "evals/arch/dev_repos_overview.html", "evals/arch/**/*.html", "docs/**/*.md", "README.md"]
|
|
6
|
+
alwaysApply: true
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Integration Priorities: What Actually Matters
|
|
10
|
+
|
|
11
|
+
## Canonical integration view (single HTML artifact)
|
|
12
|
+
|
|
13
|
+
**Canonical file**: `evals/arch/dev_repos_overview.html`
|
|
14
|
+
|
|
15
|
+
**Rule**: If you need to understand or explain how repos interact, update that one HTML page instead of creating a new report page.
|
|
16
|
+
|
|
17
|
+
**It should contain**:
|
|
18
|
+
- **Levels (L0–L6)**: group repos by the Tekne stack layers (conceptual).
|
|
19
|
+
- **Repo-level graph**: integration edges (git deps, `[patch]` overrides, and the local “super-workspace”).
|
|
20
|
+
- **Crate-level graph**: the dev workspace view (“what Cargo sees”) plus a small conceptual edge set, and a mismatch table.
|
|
21
|
+
- **Tekne facade table**: feature ↔ crate mapping.
|
|
22
|
+
- **Recent activity + size**: the few biggest disk drivers + a short “what changed recently” list.
|
|
23
|
+
- **Axes view**: call out which repos are Tekne (L0–L6) vs Agents (A0–A3) vs Governance (G0–G5), since these are different partial orders.
|
|
24
|
+
|
|
25
|
+
**Links**:
|
|
26
|
+
- `tekne-core.mdc` defines the *meaning* of levels (L0–L6).
|
|
27
|
+
- `tekne-module-taxonomy.mdc` assesses doc quality by module; it is not a dependency graph.
|
|
28
|
+
|
|
29
|
+
## The Ecosystem at a Glance
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
PRODUCTION SCALE MATH PRIMITIVES
|
|
33
|
+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
34
|
+
anno 371.3K LOC 7531 tests equi 17 LOC 0 tests
|
|
35
|
+
cerno 90.5K LOC 2627 tests fynch 1.5K LOC 53 tests
|
|
36
|
+
hop 117.8K LOC 2410 tests walk 234 LOC 0 tests
|
|
37
|
+
innr 4.5K LOC 153 tests helm 21 LOC 0 tests
|
|
38
|
+
jin 29.8K LOC 367 tests hyp 2K LOC 51 tests
|
|
39
|
+
subsume 14.5K LOC 282 tests kuji 738 LOC 15 tests
|
|
40
|
+
tier 4.7K LOC 76 tests lapl 968 LOC 13 tests
|
|
41
|
+
webs 13.9K LOC 253 tests law 38 LOC 0 tests
|
|
42
|
+
weigh 4 LOC 0 tests liga 12 LOC 0 tests
|
|
43
|
+
logp 567 LOC 10 tests
|
|
44
|
+
pare 403 LOC 3 tests
|
|
45
|
+
propago 4.7K LOC 0 tests
|
|
46
|
+
qig 148 LOC 4 tests
|
|
47
|
+
ritu 49 LOC 0 tests
|
|
48
|
+
rkhs 1.3K LOC 23 tests
|
|
49
|
+
rmt 230 LOC 7 tests
|
|
50
|
+
textprep 197 LOC 3 tests
|
|
51
|
+
sbits 736 LOC 8 tests
|
|
52
|
+
sign 15 LOC 0 tests
|
|
53
|
+
slabs 2.3K LOC 82 tests
|
|
54
|
+
telic 31 LOC 0 tests
|
|
55
|
+
wass 691 LOC 13 tests
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## The Real Problems
|
|
59
|
+
|
|
60
|
+
### Problem 1: Differentiable ranking should stay centralized
|
|
61
|
+
`fynch` is the single canonical crate for differentiable sort/rank/top-k.
|
|
62
|
+
|
|
63
|
+
### Problem 2: fynch/metrics vs cerno-eval Overlap
|
|
64
|
+
Keep **core rank-based** metrics in one place (e.g. `fynch::metrics`), and keep domain adapters (IR / KGE / NED) in their domain crates.
|
|
65
|
+
|
|
66
|
+
### Problem 3: New Math Crates Are Islands
|
|
67
|
+
The “islands” problem is missing usage/tests, not missing dependency edges. Ensure integration tests verify that combining components improves outcomes versus using them individually.
|
|
68
|
+
|
|
69
|
+
### Problem 4: Hypha integration
|
|
70
|
+
`hypha` provides the P2P substrate. `bop` (Agents) should eventually run *over* `hypha` for distributed coordination. `hypha-knowledge` should leverage `jin` (L3) for vector storage to avoid duplication. `recal` provides the agent's long-term memory via FTS5/vector retrieval.
|
|
71
|
+
|
|
72
|
+
### Problem 5: Graph Layer Split
|
|
73
|
+
Keep `lattix` as the low-level graph core (L3/L4) and `webs` as the high-level knowledge graph layer (L5). `webs` consumes `lattix` types.
|
|
74
|
+
|
|
75
|
+
### Problem 6: Cross-axis coupling creep
|
|
76
|
+
The most common structural regression is accidental coupling where:
|
|
77
|
+
- Tekne crates start depending on Agents or Governance crates, making the substrate “application-shaped”
|
|
78
|
+
- Governance becomes a compile-time cross-cutting dependency instead of a gate/eval layer
|
|
79
|
+
|
|
80
|
+
Prefer governance as *inputs/outputs* (policy decisions, eval reports) over ubiquitous imports.
|
|
81
|
+
|
|
82
|
+
### Problem 7: Sparse lexical retrieval has no clear “low-level home”
|
|
83
|
+
Right now, **BM25/TF‑IDF + (learned) sparse retrieval** lives in `cerno-retrieve` (L5 Domain), but the underlying **inverted-index engine** shape belongs lower:
|
|
84
|
+
|
|
85
|
+
- **L1/L1.5**: scoring math + tokenization/normalization (`textprep`)
|
|
86
|
+
- **L3**: inverted index / segments / deletes / compaction / persistence format
|
|
87
|
+
- **L5**: pipelines (BM25 → rerank), hybrid orchestration, adapters
|
|
88
|
+
|
|
89
|
+
For a publishable, reusable L3 “sparse index” crate, borrow the *constraints-first* boundary from BurntSushi’s Nakala plan:
|
|
90
|
+
- keep tokenization pluggable (don’t bake in language-specific processing)
|
|
91
|
+
- keep query model small (boolean term queries are enough to start)
|
|
92
|
+
- avoid “fields/schema” early (multi-set of terms is the core)
|
|
93
|
+
- index-only (don’t store document content)
|
|
94
|
+
- handle deletes + segment merges as owned complexity
|
|
95
|
+
|
|
96
|
+
Also keep the candidate-generation contract explicit:
|
|
97
|
+
- **No false negatives** for the index/planner path: it may return false positives, but must not exclude a true match.
|
|
98
|
+
- **Planner bailouts are required**: if the query is too broad, it must be legal to fall back to exhaustive scanning.
|
|
99
|
+
|
|
100
|
+
Evidence/anchors:
|
|
101
|
+
- ripgrep indexing RFC (`BurntSushi/ripgrep#1497`) and implementation discussion (`BurntSushi/ripgrep#1518`) are a concrete example of these constraints applied to regex candidate generation (n-gram index + verification).
|
|
102
|
+
- Russ Cox’s trigram-index writeup is the canonical mental model for regex → boolean n-gram query → postings → verify.
|
|
103
|
+
|
|
104
|
+
Reference: `https://raw.githubusercontent.com/BurntSushi/nakala/master/doc/PLAN.md`
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Super-workspace context discipline (repo truth vs editor hints)
|
|
3
|
+
globs: ["**/*"]
|
|
4
|
+
alwaysApply: true
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Invariant: repo path is truth
|
|
8
|
+
|
|
9
|
+
This workspace is a super-workspace containing many independent repos.
|
|
10
|
+
|
|
11
|
+
- Treat `open_and_recently_viewed_files` as a UI hint only.
|
|
12
|
+
- Never assume the currently open file indicates the repo/task being worked on.
|
|
13
|
+
|
|
14
|
+
## Practical workflow
|
|
15
|
+
|
|
16
|
+
- When a task is repo-scoped, first name the repo directory explicitly (e.g. `anno/`, `cerno/`, `wass/`, `devdev/tooling/dev-top/`).
|
|
17
|
+
- Prefer pinning the repo root with `@Folders` before making multi-file changes.
|
|
18
|
+
- Scope searches to the chosen repo directory (avoid whole-workspace scans unless explicitly needed).
|
|
19
|
+
|
|
20
|
+
## Drift check (cheap)
|
|
21
|
+
|
|
22
|
+
If the only open file(s) are outside the named repo, call that out and ask to pin the correct folder/file(s) before proceeding.
|
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: PRIMARY INVARIANTS for the Tekne Stack (L0-L6). This is the single source of truth for architecture and safety.
|
|
3
|
+
globs: ["**/*.rs", "**/Cargo.toml"]
|
|
4
|
+
alwaysApply: true
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Tekne Core Invariants (L0-L6)
|
|
8
|
+
|
|
9
|
+
## 0.5 Dev Workspace Reality (this folder is not a repo)
|
|
10
|
+
|
|
11
|
+
This Cursor workspace (the folder you opened in Cursor) is a **super-workspace** directory:
|
|
12
|
+
|
|
13
|
+
- **Not a git repo**: do not assume `git` operations make sense at the workspace root.
|
|
14
|
+
- **Many independent projects live side-by-side**: most subdirectories are their own repos (or archives), with their own history and tooling.
|
|
15
|
+
- **One root Cargo workspace exists**: `Cargo.toml` at the root defines a *virtual* Rust workspace that includes a curated subset of crates across subdirectories.
|
|
16
|
+
- **Cargo.lock is not present at the root**: dependency stability comes from per-crate `Cargo.lock` files (where used) and pinned workspace dependency versions in the root `Cargo.toml`.
|
|
17
|
+
|
|
18
|
+
Implication: when reasoning about “the project”, be explicit about *which* layer you mean:
|
|
19
|
+
\[
|
|
20
|
+
\text{root dev folder} \;\supset\; \text{root Cargo workspace members} \;\supset\; \text{Tekne stack subset (one organizing view)}.
|
|
21
|
+
\]
|
|
22
|
+
|
|
23
|
+
## 0. The Principle of Economy of Thought
|
|
24
|
+
**Goal**: Reduce the reasoning required to understand the system.
|
|
25
|
+
* **Framing over Computation**: Time spent finding the right mathematical framing (e.g. Invariants) beats time spent debugging executions.
|
|
26
|
+
* **Separation of Concerns**:
|
|
27
|
+
* **Efficiency** is settled in L0 (Iron).
|
|
28
|
+
* **Correctness** is settled in L1 (Logic).
|
|
29
|
+
* **Orchestration** is settled in L6 (Mesh).
|
|
30
|
+
* **The Five-Line Rule**: If a core logic argument or function body exceeds ~5 lines, reconsider the framing.
|
|
31
|
+
|
|
32
|
+
## 1. The Layering Invariant (DAG Hard-Stop)
|
|
33
|
+
The Tekne Stack is a Directed Acyclic Graph (DAG). **No lower layer may depend on a higher layer.**
|
|
34
|
+
|
|
35
|
+
| Layer | Name | Domain | Responsibility |
|
|
36
|
+
| :--- | :--- | :--- | :--- |
|
|
37
|
+
| **L0** | **Iron** | Hardware/SIMD | Raw performance, vector similarity ($Cosine$, $L_2$). |
|
|
38
|
+
| **L1** | **Logic** | Math Primitives | Scalar measures: Entropy, Divergences ($KL$, $JS$). |
|
|
39
|
+
| **L2** | **Algos** | Composite | Differentiable sorting, optimal transport, kernels. |
|
|
40
|
+
| **L3** | **Structures** | Aggregators | ANN Indices (`jin`), Hierarchy/aggregation (`tier`), Graph core (`lattix`). |
|
|
41
|
+
| **L4** | **Embeddings** | Representations | Geometric boxes (`subsume`), Statistical mirror (`weigh`). |
|
|
42
|
+
| **L5** | **Domain** | Systems/NLP/IR | NER/Coref (`anno`), Retrieval (`cerno`), Knowledge Graphs (`webs`). |
|
|
43
|
+
| **L6** | **Orch** | Multi-Agent/Mesh | Retrieval/pipeline (`hop`), coordination substrate (`hypha`). |
|
|
44
|
+
|
|
45
|
+
* **Private Gravity**: If a higher-level system implements reusable logic, push it *down* to L0-L2.
|
|
46
|
+
* **Enforcement**: Prefer automated checks, but don’t assume a verifier exists.
|
|
47
|
+
If enforcement tooling is missing, treat the root `Cargo.toml` membership list as the source of truth.
|
|
48
|
+
|
|
49
|
+
## 1.5 Multiple “Virtual Workspaces” Can Coexist
|
|
50
|
+
|
|
51
|
+
`tekne` is one coherent naming/stack view used by the root Cargo workspace, but it is not the only possible organizing principle.
|
|
52
|
+
|
|
53
|
+
- It’s acceptable to have other coherent sub-ecosystems (e.g., “Scholar stack”, “Infra ops”, editor plugins) living in the same dev directory.
|
|
54
|
+
- Do not force everything into Tekne if the dependency DAG / semantics don’t fit.
|
|
55
|
+
|
|
56
|
+
## 1.25 Three Parallel DAGs (Math/System vs Agents vs Governance)
|
|
57
|
+
|
|
58
|
+
The “Tekne stack” (L0–L6) is a **math/system DAG**. Agent/tooling and governance are **different axes**.
|
|
59
|
+
|
|
60
|
+
- **Math/System DAG**: operators, representations, and systems (e.g. `innr → logp → fynch → ... → anno/cerno/webs`).
|
|
61
|
+
- **Agents/Tooling DAG**: agent contract engine + orchestration apps (e.g. `axi → bop`, which then *uses* the math/system crates).
|
|
62
|
+
- **Governance DAG**: constraints/policy/oversight that can apply to both agents and systems, without being “higher math”.
|
|
63
|
+
|
|
64
|
+
This avoids category errors like “`axi` should be L1”: agent contracts are not math primitives.
|
|
65
|
+
|
|
66
|
+
### Why Tekne is “special”
|
|
67
|
+
|
|
68
|
+
Tekne is special in one sense: it is the **reusable substrate DAG**.
|
|
69
|
+
|
|
70
|
+
- If something is in Tekne, it should be *plausibly reusable* outside a single application.
|
|
71
|
+
- Therefore Tekne is the place where we are strictest about:
|
|
72
|
+
- dependency direction (no higher → lower)
|
|
73
|
+
- Unicode correctness at boundaries (character offsets)
|
|
74
|
+
- stability of core types and invariants
|
|
75
|
+
|
|
76
|
+
Agents and Governance can be equally rigorous, but they are *different* partial orders.
|
|
77
|
+
|
|
78
|
+
### Agent/Tooling Tree (A0–A3)
|
|
79
|
+
|
|
80
|
+
Use this when you want a *different* partial order than Tekne’s L0–L6:
|
|
81
|
+
|
|
82
|
+
| Level | Name | Responsibility | Examples |
|
|
83
|
+
| :--- | :--- | :--- | :--- |
|
|
84
|
+
| **A0** | **Contracts** | Typed tool contracts, schema/repair, run traces | `axi` |
|
|
85
|
+
| **A1** | **Adapters** | Providers/transports (LLM, MCP, storage) | (often lives inside `axi`/apps initially) |
|
|
86
|
+
| **A2** | **Orchestration Apps** | The runnable agent application layer | `bop` |
|
|
87
|
+
| **A3** | **Operations** | Deployment, cost, monitoring | (repo-specific) |
|
|
88
|
+
|
|
89
|
+
### Governance Tree (G0–G5)
|
|
90
|
+
|
|
91
|
+
Governance is its own axis (not “L7–L12 of Tekne”):
|
|
92
|
+
|
|
93
|
+
| Level | Name | Responsibility | Repo |
|
|
94
|
+
| :--- | :--- | :--- | :--- |
|
|
95
|
+
| **G0** | **Adjudication** | deciding / evaluating / ruling | `law` |
|
|
96
|
+
| **G1** | **Steering** | policy + strategy control | `helm` |
|
|
97
|
+
| **G2** | **Equilibrium** | constraints / balance / alignment | `equi` |
|
|
98
|
+
| **G3** | **Interface** | representation boundaries / signaling | `sign` |
|
|
99
|
+
| **G4** | **Federation** | multi-party coordination contracts | `liga` |
|
|
100
|
+
| **G5** | **Telos** | goals/purpose/intention | `telic` |
|
|
101
|
+
|
|
102
|
+
### Cross-axis dependency rules (compile-time vs “gates”)
|
|
103
|
+
|
|
104
|
+
**Compile-time dependencies (Cargo `dependencies`)**:
|
|
105
|
+
|
|
106
|
+
- **Allowed**:
|
|
107
|
+
- Agents → Tekne (agents can use substrate)
|
|
108
|
+
- Governance → Tekne (governance can use substrate for scoring/metrics/types)
|
|
109
|
+
- Apps (A2) → Governance (apps may call adjudicators/steering code)
|
|
110
|
+
- **Forbidden by default**:
|
|
111
|
+
- Tekne → Agents
|
|
112
|
+
- Tekne → Governance
|
|
113
|
+
- Governance → Agents (especially Governance → A2 apps)
|
|
114
|
+
|
|
115
|
+
**Why**: if Tekne depends on Agents/Governance, the substrate becomes “application-shaped” and stops being substrate.
|
|
116
|
+
|
|
117
|
+
**Governance should often be a gate, not a dependency**:
|
|
118
|
+
|
|
119
|
+
- Prefer governance as *inputs/outputs* and pipeline checks:
|
|
120
|
+
- plan/request → policy decision
|
|
121
|
+
- run trace → evaluation/audit
|
|
122
|
+
- Avoid “sprinkling governance imports everywhere” as a cross-cutting compile-time dependency, which tends to reintroduce cycles and coupling.
|
|
123
|
+
|
|
124
|
+
## 2. The Naming Invariant (Structure > Style)
|
|
125
|
+
|
|
126
|
+
**Naming Strategy**: Names should make the *structure* easier to see.
|
|
127
|
+
|
|
128
|
+
\[
|
|
129
|
+
\text{good name} \;\approx\; \text{a cheap index into the right invariants.}
|
|
130
|
+
\]
|
|
131
|
+
|
|
132
|
+
### Primary Preference: Structural Resonance
|
|
133
|
+
|
|
134
|
+
- Prefer names that align with the *true symmetry* of the object:
|
|
135
|
+
- distance / similarity / norm / kernel / transport / rank / conformal / graph / lattice / contract / covenant / ledger
|
|
136
|
+
- Prefer names that let readers guess the constraints:
|
|
137
|
+
- e.g. “rank fusion” vs “rerank” vs “retrieve” are distinct operators in a pipeline; names should preserve that.
|
|
138
|
+
- Cross-language is encouraged when it improves precision (sometimes the original term is the most exact handle).
|
|
139
|
+
|
|
140
|
+
### Secondary Preference: Etymology (cross-cultural only when helpful)
|
|
141
|
+
|
|
142
|
+
- Culture is not the goal; *precision* is.
|
|
143
|
+
- Use Sanskrit/Arabic/Old Norse/etc. when it carries the exact nuance for governance or agency concepts.
|
|
144
|
+
- Use Latin/Greek/English when it better matches the mathematical framing.
|
|
145
|
+
|
|
146
|
+
### Ecosystem/Prefix Coherence (important)
|
|
147
|
+
|
|
148
|
+
- If a set of crates is meant to be used together, prefer a shared prefix.
|
|
149
|
+
- IR stack: prefer `cerno-*` as the canonical prefix (retrieve/fuse/rerank/eval/train + facade `cerno`).
|
|
150
|
+
- Text chunking: prefer `slabs` as canonical (avoid dual naming like `slabs` vs `slabs`).
|
|
151
|
+
|
|
152
|
+
### crates.io Policy (mandatory, explicit)
|
|
153
|
+
|
|
154
|
+
Always check crates.io **before** selecting a name (don’t guess).
|
|
155
|
+
|
|
156
|
+
- If a name is **taken** on crates.io (HTTP 200), treat it as **unavailable** unless:
|
|
157
|
+
- the crate is permanently `publish = false`, and
|
|
158
|
+
- we accept that the name is *internal-only* (no future publish path).
|
|
159
|
+
|
|
160
|
+
This prevents “pretty internal names” that block publication later.
|
|
161
|
+
|
|
162
|
+
### Consistency Rule: one concept, one name
|
|
163
|
+
|
|
164
|
+
- Avoid “ghost prior names” lingering in code/docs (e.g., `colligo` remnants inside `hop`).
|
|
165
|
+
- Avoid having directory name, package name, and crate name drift.
|
|
166
|
+
- If we keep an old directory during migration, document it and fix it quickly.
|
|
167
|
+
|
|
168
|
+
## 3. The Meta-ND Invariant (Multi-Dimensional Metadata)
|
|
169
|
+
**Goal**: Unified handling of high-dimensional metadata and structures.
|
|
170
|
+
* **Dimensional Rigor**: Systems dealing with N-dimensional data (e.g., `subsume` boxes, `innr` vectors) must explicitly track and leverage metadata dimensions.
|
|
171
|
+
* **Subsumption Theory**: Use formal subsumption/entailment theory to manage metadata hierarchies across the stack.
|
|
172
|
+
* **Representation**: Metadata should be treated as a first-class N-dimensional object, not a flat bag of tags.
|
|
173
|
+
|
|
174
|
+
## 4. Type Hierarchy & Invariants
|
|
175
|
+
* **Canonical Flows**: Signal (Detection) → Track (Coreference) → Identity (Linking).
|
|
176
|
+
* **Numerical Stability**: Use `NORM_EPSILON` ($10^{-9}$) for zero-norm handling.
|
|
177
|
+
* **Algebraic Closure**: Intersection operations (e.g. in L4) must return the same type.
|
|
178
|
+
|
|
179
|
+
## 3. Engineering Rigor
|
|
180
|
+
* **Unicode**: ALWAYS use character offsets, NEVER byte offsets for text slicing.
|
|
181
|
+
* **SIMD**: Prefer `innr` for vector operations with automatic runtime dispatch.
|
|
182
|
+
* **Fast Math**: Use reciprocal square root ($rsqrt$) + Newton-Raphson for throughput.
|
|
183
|
+
* **Math**: Cite foundational papers (TAOCP style).
|
|
184
|
+
|
|
185
|
+
## 4. Documentation vs. Process Logs
|
|
186
|
+
* **Chat for Process**: Do not write temporary markdown files for progress. Reply in chat.
|
|
187
|
+
* **Lecture Notes**: Review `docs/lectures/*.md` to pick up stack-wide habits.
|
|
188
|
+
* **Visual Rigor**: Visualizations must use real data from actual execution. fit distributions.
|
|
189
|
+
|
|
190
|
+
## 4.5 Memory Recall Discipline (Cursor history is data)
|
|
191
|
+
|
|
192
|
+
When intent or prior decisions matter (renames, merges, “publish vs private”, repo cleanup), **do not guess**.
|
|
193
|
+
Treat prior chat + rule history as a dataset and re-derive intent from it.
|
|
194
|
+
|
|
195
|
+
- **Preferred**: use `iksh` to inspect Cursor’s DB-backed chat history and rule memory:
|
|
196
|
+
- list conversation ids: `cargo run -p iksh -- chats --limit N`
|
|
197
|
+
- list with previews (recommended): `cargo run -p iksh -- chats --limit N --meta --format text --preview-chars 160`
|
|
198
|
+
- list as JSON (for tooling): `cargo run -p iksh -- chats --limit N --format json --preview-chars 160`
|
|
199
|
+
- reconstruct a conversation: `cargo run -p iksh -- convo <conversation-uuid>`
|
|
200
|
+
- search for keywords: `cargo run -p iksh -- db-search-values <needle> --limit N`
|
|
201
|
+
- **Fallback**: consult the saved agent transcript file(s) when the DB doesn’t contain what we need.
|
|
202
|
+
|
|
203
|
+
Goal: minimize “phantom intent” errors by making recall a first-class step.
|
|
204
|
+
|
|
205
|
+
## 5. Repository & Tool Safety
|
|
206
|
+
* **Isolation**: Never create a new repo inside another repo.
|
|
207
|
+
* **Timeout Handling**: If a tool call fails/times out, narrow the scope. Do not repeat.
|
|
208
|
+
* **Local Resolution**: Use `[patch.crates-io]` in root for development speed.
|
|
209
|
+
|
|
210
|
+
## 6. Git Discipline (Staging)
|
|
211
|
+
|
|
212
|
+
Default to explicit staging (e.g. `git add <paths>`). Avoid `git add -A` unless the user explicitly requests “stage everything”.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Tekne Module Taxonomy"
|
|
3
|
+
id: tekne-module-taxonomy
|
|
4
|
+
description: "Assessment of documentation quality and purpose by module"
|
|
5
|
+
alwaysApply: false
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Tekne Module Taxonomy
|
|
9
|
+
|
|
10
|
+
Assessment of documentation quality and pedagogical readiness across the stack.
|
|
11
|
+
|
|
12
|
+
## Grade A: Pedagogical Gold Standard (TAOCP / Mackay Style)
|
|
13
|
+
|
|
14
|
+
| Module | Lines | Additions |
|
|
15
|
+
|:---|:---|:---|
|
|
16
|
+
| `jin/hnsw` | ~210 | Mathematical invariant check, probability distribution |
|
|
17
|
+
| `tier/kmeans` | ~110 | Hard vs soft, GMM math, linkage table |
|
|
18
|
+
| `hyp` (lib.rs) | ~80 | **Model selection table**, when NOT to use |
|
|
19
|
+
| `innr` (lib.rs) | ~60 | **Function selection table**, SIMD dispatch |
|
|
20
|
+
|
|
21
|
+
## Verification Checklist
|
|
22
|
+
|
|
23
|
+
For each module, verify:
|
|
24
|
+
- [ ] Problem statement (what does this solve?)
|
|
25
|
+
- [ ] Why this approach (what's the insight?)
|
|
26
|
+
- [ ] Comparison to alternatives (when to use/not use)
|
|
27
|
+
- [ ] Parameters with effects (not just names)
|
|
28
|
+
- [ ] Complexity (time/space)
|
|
29
|
+
- [ ] Working code example
|
|
30
|
+
- [ ] References (papers, not just URLs)
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Managed by repository-settings/app
|
|
2
|
+
# https://github.com/repository-settings/app
|
|
3
|
+
|
|
4
|
+
repository:
|
|
5
|
+
description: "Operations on ranked lists: fusion (RRF, CombMNZ, Borda, DBSF), reranking (MaxSim/ColBERT, MMR, DPP, Matryoshka). Pairs with rankfns (scoring kernels)."
|
|
6
|
+
homepage: "https://docs.rs/rankops"
|
|
7
|
+
private: false
|
|
8
|
+
has_issues: true
|
|
9
|
+
has_wiki: false
|
|
10
|
+
has_downloads: true
|
|
11
|
+
default_branch: main
|
|
12
|
+
allow_squash_merge: true
|
|
13
|
+
allow_merge_commit: true
|
|
14
|
+
allow_rebase_merge: true
|
|
15
|
+
delete_branch_on_merge: true
|
|
16
|
+
|
|
17
|
+
topics:
|
|
18
|
+
- information-retrieval
|
|
19
|
+
- reranking
|
|
20
|
+
- rank-fusion
|
|
21
|
+
- rust
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: ["main", "master"]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
|
|
11
|
+
env:
|
|
12
|
+
CARGO_TERM_COLOR: always
|
|
13
|
+
|
|
14
|
+
jobs:
|
|
15
|
+
rust:
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
steps:
|
|
18
|
+
- uses: actions/checkout@v4
|
|
19
|
+
- uses: dtolnay/rust-toolchain@stable
|
|
20
|
+
with:
|
|
21
|
+
components: rustfmt, clippy
|
|
22
|
+
- uses: Swatinem/rust-cache@v2
|
|
23
|
+
|
|
24
|
+
- name: fmt
|
|
25
|
+
run: cargo fmt --check
|
|
26
|
+
|
|
27
|
+
- name: clippy
|
|
28
|
+
run: cargo clippy --all-targets --all-features -- -D warnings
|
|
29
|
+
|
|
30
|
+
- name: test
|
|
31
|
+
run: cargo test --all-targets --all-features
|
|
32
|
+
|
|
33
|
+
- name: build rankops-wasm (wasm32)
|
|
34
|
+
run: |
|
|
35
|
+
rustup target add wasm32-unknown-unknown
|
|
36
|
+
cargo build --manifest-path crates/rankops-wasm/Cargo.toml --target wasm32-unknown-unknown
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
name: publish-pypi
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags:
|
|
6
|
+
- "python-v*"
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
linux:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
strategy:
|
|
15
|
+
matrix:
|
|
16
|
+
target: [x86_64, aarch64]
|
|
17
|
+
steps:
|
|
18
|
+
- uses: actions/checkout@v4
|
|
19
|
+
- uses: actions/setup-python@v5
|
|
20
|
+
with:
|
|
21
|
+
python-version: "3.12"
|
|
22
|
+
- name: Build wheels
|
|
23
|
+
uses: PyO3/maturin-action@v1
|
|
24
|
+
with:
|
|
25
|
+
target: ${{ matrix.target }}
|
|
26
|
+
args: --release --out dist --find-interpreter
|
|
27
|
+
manylinux: auto
|
|
28
|
+
working-directory: rankops-python
|
|
29
|
+
- uses: actions/upload-artifact@v4
|
|
30
|
+
with:
|
|
31
|
+
name: wheel-linux-${{ matrix.target }}
|
|
32
|
+
path: rankops-python/dist/*.whl
|
|
33
|
+
|
|
34
|
+
macos:
|
|
35
|
+
runs-on: macos-latest
|
|
36
|
+
strategy:
|
|
37
|
+
matrix:
|
|
38
|
+
target: [x86_64, aarch64]
|
|
39
|
+
steps:
|
|
40
|
+
- uses: actions/checkout@v4
|
|
41
|
+
- uses: actions/setup-python@v5
|
|
42
|
+
with:
|
|
43
|
+
python-version: "3.12"
|
|
44
|
+
- name: Build wheels
|
|
45
|
+
uses: PyO3/maturin-action@v1
|
|
46
|
+
with:
|
|
47
|
+
target: ${{ matrix.target }}-apple-darwin
|
|
48
|
+
args: --release --out dist --find-interpreter
|
|
49
|
+
working-directory: rankops-python
|
|
50
|
+
- uses: actions/upload-artifact@v4
|
|
51
|
+
with:
|
|
52
|
+
name: wheel-macos-${{ matrix.target }}
|
|
53
|
+
path: rankops-python/dist/*.whl
|
|
54
|
+
|
|
55
|
+
windows:
|
|
56
|
+
runs-on: windows-latest
|
|
57
|
+
steps:
|
|
58
|
+
- uses: actions/checkout@v4
|
|
59
|
+
- uses: actions/setup-python@v5
|
|
60
|
+
with:
|
|
61
|
+
python-version: "3.12"
|
|
62
|
+
architecture: x64
|
|
63
|
+
- name: Build wheels
|
|
64
|
+
uses: PyO3/maturin-action@v1
|
|
65
|
+
with:
|
|
66
|
+
args: --release --out dist --find-interpreter
|
|
67
|
+
working-directory: rankops-python
|
|
68
|
+
- uses: actions/upload-artifact@v4
|
|
69
|
+
with:
|
|
70
|
+
name: wheel-windows-x64
|
|
71
|
+
path: rankops-python/dist/*.whl
|
|
72
|
+
|
|
73
|
+
sdist:
|
|
74
|
+
runs-on: ubuntu-latest
|
|
75
|
+
steps:
|
|
76
|
+
- uses: actions/checkout@v4
|
|
77
|
+
- name: Build sdist
|
|
78
|
+
uses: PyO3/maturin-action@v1
|
|
79
|
+
with:
|
|
80
|
+
command: sdist
|
|
81
|
+
args: --out dist
|
|
82
|
+
working-directory: rankops-python
|
|
83
|
+
- uses: actions/upload-artifact@v4
|
|
84
|
+
with:
|
|
85
|
+
name: sdist
|
|
86
|
+
path: rankops-python/dist/*.tar.gz
|
|
87
|
+
|
|
88
|
+
publish:
|
|
89
|
+
needs: [linux, macos, windows, sdist]
|
|
90
|
+
runs-on: ubuntu-latest
|
|
91
|
+
environment: pypi
|
|
92
|
+
permissions:
|
|
93
|
+
id-token: write
|
|
94
|
+
steps:
|
|
95
|
+
- uses: actions/download-artifact@v4
|
|
96
|
+
with:
|
|
97
|
+
path: dist
|
|
98
|
+
merge-multiple: true
|
|
99
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
100
|
+
with:
|
|
101
|
+
packages-dir: dist/
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "rankops"
|
|
3
|
+
version = "0.1.4"
|
|
4
|
+
edition = "2021"
|
|
5
|
+
license = "MIT OR Apache-2.0"
|
|
6
|
+
description = "Operations on ranked lists: fusion (RRF, Copeland, CombMNZ, DBSF, 14 methods), reranking (MaxSim/ColBERT, MMR, DPP), evaluation (NDCG, MAP, MRR), diagnostics, pipeline builder."
|
|
7
|
+
repository = "https://github.com/arclabs561/rankops"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
keywords = ["rerank", "rrf", "retrieval", "rankops", "fusion"]
|
|
10
|
+
categories = ["algorithms", "mathematics", "science"]
|
|
11
|
+
|
|
12
|
+
[dependencies]
|
|
13
|
+
serde = { version = "1.0", optional = true, features = ["derive"] }
|
|
14
|
+
# SIMD primitives for reranking (MaxSim/ColBERT)
|
|
15
|
+
innr = { version = "0.1.9", optional = true }
|
|
16
|
+
# Hierarchical clustering for ColBERT (optional)
|
|
17
|
+
kodama = { version = "0.3", optional = true }
|
|
18
|
+
# Pareto frontier for multi-objective reranking (optional)
|
|
19
|
+
pare = { version = "0.2.0", optional = true }
|
|
20
|
+
|
|
21
|
+
[features]
|
|
22
|
+
default = ["rerank"]
|
|
23
|
+
serde = ["dep:serde"]
|
|
24
|
+
# Reranking: MaxSim (ColBERT), diversity, matryoshka — requires innr for SIMD
|
|
25
|
+
rerank = ["dep:innr"]
|
|
26
|
+
# Alias so #[cfg(feature = "innr")] in rerank/simd.rs matches (rerank enables innr)
|
|
27
|
+
innr = ["rerank"]
|
|
28
|
+
# Hierarchical ColBERT; enables kodama for dendrogram clustering
|
|
29
|
+
hierarchical = ["dep:kodama"]
|
|
30
|
+
# Pareto frontier for multi-objective result selection
|
|
31
|
+
pareto = ["dep:pare"]
|
|
32
|
+
|
|
33
|
+
[dev-dependencies]
|
|
34
|
+
criterion = { version = "0.7", features = ["html_reports"] }
|
|
35
|
+
proptest = "1.9"
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
[package.metadata.docs.rs]
|
|
39
|
+
all-features = true
|
|
40
|
+
rustdoc-args = ["--cfg", "docsrs"]
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity.
|
|
18
|
+
|
|
19
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
20
|
+
exercising permissions granted by this License.
|
|
21
|
+
|
|
22
|
+
"Source" form shall mean the preferred form for making modifications.
|
|
23
|
+
|
|
24
|
+
"Object" form shall mean any form resulting from mechanical
|
|
25
|
+
transformation or translation of a Source form.
|
|
26
|
+
|
|
27
|
+
"Work" shall mean the work of authorship made available under the License.
|
|
28
|
+
|
|
29
|
+
"Derivative Works" shall mean any work that is based on the Work.
|
|
30
|
+
|
|
31
|
+
"Contribution" shall mean any work of authorship submitted to the Licensor.
|
|
32
|
+
|
|
33
|
+
"Contributor" shall mean Licensor and any Legal Entity on behalf of whom
|
|
34
|
+
a Contribution has been received by Licensor.
|
|
35
|
+
|
|
36
|
+
2. Grant of Copyright License. Subject to the terms of this License, each
|
|
37
|
+
Contributor grants to You a perpetual, worldwide, non-exclusive, no-charge,
|
|
38
|
+
royalty-free, irrevocable copyright license to reproduce, prepare Derivative
|
|
39
|
+
Works of, publicly display, publicly perform, sublicense, and distribute the
|
|
40
|
+
Work and such Derivative Works in Source or Object form.
|
|
41
|
+
|
|
42
|
+
3. Grant of Patent License. Subject to the terms of this License, each
|
|
43
|
+
Contributor grants to You a perpetual, worldwide, non-exclusive, no-charge,
|
|
44
|
+
royalty-free, irrevocable patent license to make, have made, use, offer to
|
|
45
|
+
sell, sell, import, and otherwise transfer the Work.
|
|
46
|
+
|
|
47
|
+
4. Redistribution. You may reproduce and distribute copies of the Work
|
|
48
|
+
provided that You meet the following conditions:
|
|
49
|
+
(a) You must give recipients a copy of this License; and
|
|
50
|
+
(b) You must cause modified files to carry prominent notices; and
|
|
51
|
+
(c) You must retain all copyright, patent, trademark notices; and
|
|
52
|
+
(d) If the Work includes a "NOTICE" file, You must include a readable copy.
|
|
53
|
+
|
|
54
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
55
|
+
any Contribution submitted for inclusion in the Work shall be under the
|
|
56
|
+
terms of this License, without any additional terms or conditions.
|
|
57
|
+
|
|
58
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
59
|
+
names, trademarks, service marks, or product names of the Licensor.
|
|
60
|
+
|
|
61
|
+
7. Disclaimer of Warranty. The Work is provided on an "AS IS" BASIS,
|
|
62
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
|
|
63
|
+
|
|
64
|
+
8. Limitation of Liability. In no event shall any Contributor be liable
|
|
65
|
+
to You for damages, including any direct, indirect, special, incidental,
|
|
66
|
+
or consequential damages.
|
|
67
|
+
|
|
68
|
+
9. Accepting Warranty or Additional Liability. You may choose to offer
|
|
69
|
+
support, warranty, indemnity, or other liability obligations.
|
|
70
|
+
|
|
71
|
+
END OF TERMS AND CONDITIONS
|
|
72
|
+
|
|
73
|
+
Copyright 2024 arclabs561
|
|
74
|
+
|
|
75
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
76
|
+
you may not use this file except in compliance with the License.
|
|
77
|
+
You may obtain a copy of the License at
|
|
78
|
+
|
|
79
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
80
|
+
|
|
81
|
+
Unless required by applicable law or agreed to in writing, software
|
|
82
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
83
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
84
|
+
See the License for the specific language governing permissions and
|
|
85
|
+
limitations under the License.
|