rankweave 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,26 @@
1
+ name: ci
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ permissions:
9
+ contents: read
10
+
11
+ jobs:
12
+ test:
13
+ runs-on: ubuntu-latest
14
+ strategy:
15
+ fail-fast: false
16
+ matrix:
17
+ python-version: ["3.10", "3.11", "3.12", "3.13"]
18
+ steps:
19
+ - uses: actions/checkout@v4
20
+ - uses: actions/setup-python@v5
21
+ with:
22
+ python-version: ${{ matrix.python-version }}
23
+ - run: python -m pip install --upgrade pip
24
+ - run: pip install -e ".[dev]"
25
+ - run: python -m ruff check .
26
+ - run: python -m pytest -q
@@ -0,0 +1,26 @@
1
+ name: publish
2
+
3
+ on:
4
+ push:
5
+ branches: ["publish-release"]
6
+ workflow_dispatch: {}
7
+
8
+ permissions:
9
+ contents: read
10
+
11
+ jobs:
12
+ pypi:
13
+ runs-on: ubuntu-latest
14
+ steps:
15
+ - uses: actions/checkout@v4
16
+ - uses: actions/setup-python@v5
17
+ with:
18
+ python-version: "3.12"
19
+ - run: python -m pip install --upgrade build twine
20
+ - run: python -m build
21
+ - run: twine check dist/*
22
+ - name: Upload to PyPI
23
+ env:
24
+ TWINE_USERNAME: __token__
25
+ TWINE_PASSWORD: ${{ secrets.PIPY_TOKEN }}
26
+ run: twine upload --non-interactive --disable-progress-bar dist/*
@@ -0,0 +1,12 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ *.egg-info/
4
+ .eggs/
5
+ build/
6
+ dist/
7
+ .pytest_cache/
8
+ .ruff_cache/
9
+ .venv/
10
+ venv/
11
+ .coverage
12
+ htmlcov/
@@ -0,0 +1,45 @@
1
+ # AGENTS.md — rankweave
2
+
3
+ Operating guide for automated agents working in this repo.
4
+
5
+ ## What this is
6
+
7
+ `rankweave` is a **pure-Python, stdlib-only** library for
8
+ language-agnostic hybrid-retrieval score fusion, extracted unchanged
9
+ in behavior from
10
+ [ContextualWisdomLab/naruon](https://github.com/ContextualWisdomLab/naruon)
11
+ Context Search under the lab's ONE SOURCE MULTI USE convention
12
+ (standalone product *and* submodule-importable).
13
+
14
+ ## Hard rules
15
+
16
+ - **No dependencies.** The library imports only the Python standard
17
+ library. Do not add a runtime dependency; if you think you need one,
18
+ the feature probably belongs in the consumer, not here.
19
+ - **Store-agnostic.** rankweave never talks to a database, an
20
+ embedding provider, or a search index. It fuses scores and normalizes
21
+ query text. Keep SQL, HTTP, and ORM concerns out.
22
+ - **Behavior parity with naruon.** This is an extraction, not a fork.
23
+ A behavior change here must be mirrored in naruon's
24
+ `services/hybrid_retrieval` (and vice versa) until naruon consumes
25
+ this package directly. Prefer additive, backward-compatible changes.
26
+ - **Permissive license only** (Apache-2.0). Any added code or asset must
27
+ be compatible.
28
+ - **Research-grounded defaults.** Numeric defaults (alpha=0.7, eta=60,
29
+ the theoretical bounds) trace to the papers in `docs/research/`.
30
+ Changing a default requires citing the evidence.
31
+
32
+ ## Develop
33
+
34
+ ```bash
35
+ pip install -e ".[dev]"
36
+ python -m pytest -q # no external services required
37
+ python -m ruff check .
38
+ ```
39
+
40
+ ## Layout
41
+
42
+ - `src/rankweave/score_fusion.py` — TM2C2 + RRF fusion primitives.
43
+ - `src/rankweave/query_normalization.py` — NFC query normalization.
44
+ - `tests/` — behavior tests (hand-computed expected values).
45
+ - `docs/research/` — paper PDFs + citation manifest.
@@ -0,0 +1,25 @@
1
+ # Changelog
2
+
3
+ All notable changes to rankweave are documented here. The format
4
+ follows [Keep a Changelog](https://keepachangelog.com/), and the
5
+ project adheres to [Semantic Versioning](https://semver.org/).
6
+
7
+ ## [0.1.0] — 2026-07-11
8
+
9
+ Initial extraction from
10
+ [ContextualWisdomLab/naruon](https://github.com/ContextualWisdomLab/naruon)
11
+ Context Search, unchanged in behavior (ONE SOURCE MULTI USE).
12
+
13
+ ### Added
14
+ - `fuse_channel_scores` — fuse one candidate's lexical + dense channel
15
+ evidence into a single bounded score under the selected strategy.
16
+ - `FusionSettings` — immutable strategy + parameters (`convex_combination`
17
+ default with `semantic_weight_alpha=0.7`; `reciprocal_rank_fusion`
18
+ with `rank_constant_eta=60`).
19
+ - `convex_combination_score`, `reciprocal_rank_fusion_score`,
20
+ `theoretical_min_max_normalize` — the underlying fusion primitives.
21
+ - `normalize_search_text` — NFC compose + whitespace-collapse +
22
+ length-cap for the query side of a language-agnostic lexical channel.
23
+ - `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS`.
24
+ - 27 unit tests; no dependencies (stdlib only); typed (`py.typed`).
25
+ - Research grounding + paper manifest under `docs/research/`.
@@ -0,0 +1,201 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright (c) 2026 Contextual Wisdom Lab / Seongho Bae
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
@@ -0,0 +1,128 @@
1
+ Metadata-Version: 2.4
2
+ Name: rankweave
3
+ Version: 0.1.0
4
+ Summary: Language-agnostic hybrid-retrieval score fusion (TM2C2 + RRF) and Unicode query normalization — pure-Python, store-agnostic.
5
+ Project-URL: Homepage, https://github.com/ContextualWisdomLab/rankweave
6
+ Project-URL: Source, https://github.com/ContextualWisdomLab/rankweave
7
+ Project-URL: Issues, https://github.com/ContextualWisdomLab/rankweave/issues
8
+ Author: Contextual Wisdom Lab
9
+ License-Expression: Apache-2.0
10
+ License-File: LICENSE
11
+ Keywords: dense-retrieval,hybrid-search,information-retrieval,lexical-retrieval,pg_trgm,pgvector,rank-fusion,reciprocal-rank-fusion,rrf
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: License :: OSI Approved :: Apache Software License
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3 :: Only
17
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
18
+ Classifier: Topic :: Text Processing :: Indexing
19
+ Classifier: Typing :: Typed
20
+ Requires-Python: >=3.10
21
+ Provides-Extra: dev
22
+ Requires-Dist: pytest>=8; extra == 'dev'
23
+ Requires-Dist: ruff>=0.6; extra == 'dev'
24
+ Description-Content-Type: text/markdown
25
+
26
+ # rankweave
27
+
28
+ **Language-agnostic hybrid-retrieval score fusion — pure-Python, store-agnostic.**
29
+
30
+ `rankweave` decides *how to combine* the scores from a lexical channel
31
+ (character-trigram / BM25 / learned-sparse) and a semantic channel
32
+ (dense embeddings) into one ranking. It ships two research-grounded
33
+ fusion strategies and the query-side Unicode normalization that makes
34
+ character-level lexical matching language-agnostic. It has **no
35
+ dependencies** (stdlib only) and **no opinion about your store** — bring
36
+ your own channels; rankweave fuses their scores.
37
+
38
+ It is extracted, unchanged in behavior, from the Context Search engine of
39
+ [naruon](https://github.com/ContextualWisdomLab/naruon), following the
40
+ lab's ONE SOURCE MULTI USE convention: standalone product *and*
41
+ submodule-importable.
42
+
43
+ ## Why
44
+
45
+ A convex combination of **theoretically** min-max normalized scores
46
+ (TM2C2) beats Reciprocal Rank Fusion in- and out-of-domain, is robust
47
+ for `alpha ∈ [0.6, 0.8]` with no training data, and — unlike rank
48
+ fusion — preserves the score distribution (Bruch, Gai & Ingber 2023).
49
+ RRF remains available for channels that expose only ranks. See
50
+ [`docs/research/`](docs/research/) for the grounding.
51
+
52
+ ## Install
53
+
54
+ ```bash
55
+ pip install rankweave
56
+ ```
57
+
58
+ ## Quickstart
59
+
60
+ ```python
61
+ from rankweave import FusionSettings, fuse_channel_scores, normalize_search_text
62
+
63
+ # 1) Normalize the query the same way you normalize indexed documents
64
+ # (NFC compose; do accent-folding + lowercasing on the store side too).
65
+ query = normalize_search_text(" Trần Hưng Đạo 회의 ") # -> "Trần Hưng Đạo 회의"
66
+
67
+ # 2) Run your own lexical + dense channels, then fuse per candidate.
68
+ settings = FusionSettings() # TM2C2, semantic weight alpha = 0.7
69
+ score = fuse_channel_scores(
70
+ word_similarity_score=0.62, # lexical channel score in [0, 1]
71
+ cosine_distance=0.30, # dense channel distance in [0, 2]
72
+ channel_ranks={"lexical": 1, "dense": 1},
73
+ settings=settings,
74
+ ) # -> bounded [0, 1] fused score
75
+ ```
76
+
77
+ A channel that did not return a candidate contributes its theoretical
78
+ minimum (absent evidence is the infimum, not an imputed value). Pass
79
+ `FusionSettings(strategy_name="reciprocal_rank_fusion")` to fuse by rank
80
+ instead — then only `channel_ranks` matters.
81
+
82
+ ## API
83
+
84
+ | Symbol | Purpose |
85
+ |---|---|
86
+ | `FusionSettings` | Immutable strategy + parameters (`strategy_name`, `semantic_weight_alpha`, `rank_constant_eta`). |
87
+ | `fuse_channel_scores(...)` | Fuse one candidate's channel evidence into a single score under the chosen strategy. |
88
+ | `convex_combination_score(...)` | TM2C2 over already-normalized `[0, 1]` scores. |
89
+ | `reciprocal_rank_fusion_score(ranks, eta=60)` | RRF over 1-based per-channel ranks. |
90
+ | `theoretical_min_max_normalize(score, bounds)` | Scale a score to `[0, 1]` using a scoring function's theoretical bounds. |
91
+ | `normalize_search_text(text)` | NFC-compose + whitespace-collapse + length-cap a query. |
92
+ | `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS` | `(lower, upper)` tuples for the common lexical/dense pairing. |
93
+
94
+ ## The normalization contract
95
+
96
+ Character-trigram lexical retrieval is language-agnostic only if query
97
+ and documents fold **identically**. `normalize_search_text` owns the
98
+ query side (NFC). Do accent-folding + lowercasing on the **store** side,
99
+ in one place, and call it from both — e.g. a PostgreSQL `IMMUTABLE`
100
+ wrapper `lower(unaccent(normalize(text, NFC)))` used in a `pg_trgm` GiST
101
+ expression index. rankweave stays out of your store so the two sides
102
+ cannot silently diverge.
103
+
104
+ ## Research grounding
105
+
106
+ - **Bruch, Gai & Ingber (2023).** *An Analysis of Fusion Functions for
107
+ Hybrid Retrieval.* ACM TOIS 42(1). arXiv:2210.11934. — TM2C2 > RRF;
108
+ theoretical-normalization stability; the fusion desiderata
109
+ (monotonicity, homogeneity, boundedness, Lipschitz continuity,
110
+ sample efficiency) this library's defaults satisfy.
111
+ - **Cormack, Clarke & Büttcher (2009).** *Reciprocal Rank Fusion
112
+ outperforms Condorcet and individual Rank Learning Methods.* SIGIR
113
+ 2009. — RRF definition, η = 60.
114
+ - **UAX #15**, Unicode Normalization Forms — NFC composition.
115
+
116
+ PDFs and a citation manifest live in [`docs/research/`](docs/research/).
117
+
118
+ ## Development
119
+
120
+ ```bash
121
+ pip install -e ".[dev]"
122
+ python -m pytest -q # 27 tests, no external services
123
+ python -m ruff check .
124
+ ```
125
+
126
+ ## License
127
+
128
+ Apache-2.0 — see [LICENSE](LICENSE).
@@ -0,0 +1,103 @@
1
+ # rankweave
2
+
3
+ **Language-agnostic hybrid-retrieval score fusion — pure-Python, store-agnostic.**
4
+
5
+ `rankweave` decides *how to combine* the scores from a lexical channel
6
+ (character-trigram / BM25 / learned-sparse) and a semantic channel
7
+ (dense embeddings) into one ranking. It ships two research-grounded
8
+ fusion strategies and the query-side Unicode normalization that makes
9
+ character-level lexical matching language-agnostic. It has **no
10
+ dependencies** (stdlib only) and **no opinion about your store** — bring
11
+ your own channels; rankweave fuses their scores.
12
+
13
+ It is extracted, unchanged in behavior, from the Context Search engine of
14
+ [naruon](https://github.com/ContextualWisdomLab/naruon), following the
15
+ lab's ONE SOURCE MULTI USE convention: standalone product *and*
16
+ submodule-importable.
17
+
18
+ ## Why
19
+
20
+ A convex combination of **theoretically** min-max normalized scores
21
+ (TM2C2) beats Reciprocal Rank Fusion in- and out-of-domain, is robust
22
+ for `alpha ∈ [0.6, 0.8]` with no training data, and — unlike rank
23
+ fusion — preserves the score distribution (Bruch, Gai & Ingber 2023).
24
+ RRF remains available for channels that expose only ranks. See
25
+ [`docs/research/`](docs/research/) for the grounding.
26
+
27
+ ## Install
28
+
29
+ ```bash
30
+ pip install rankweave
31
+ ```
32
+
33
+ ## Quickstart
34
+
35
+ ```python
36
+ from rankweave import FusionSettings, fuse_channel_scores, normalize_search_text
37
+
38
+ # 1) Normalize the query the same way you normalize indexed documents
39
+ # (NFC compose; do accent-folding + lowercasing on the store side too).
40
+ query = normalize_search_text(" Trần Hưng Đạo 회의 ") # -> "Trần Hưng Đạo 회의"
41
+
42
+ # 2) Run your own lexical + dense channels, then fuse per candidate.
43
+ settings = FusionSettings() # TM2C2, semantic weight alpha = 0.7
44
+ score = fuse_channel_scores(
45
+ word_similarity_score=0.62, # lexical channel score in [0, 1]
46
+ cosine_distance=0.30, # dense channel distance in [0, 2]
47
+ channel_ranks={"lexical": 1, "dense": 1},
48
+ settings=settings,
49
+ ) # -> bounded [0, 1] fused score
50
+ ```
51
+
52
+ A channel that did not return a candidate contributes its theoretical
53
+ minimum (absent evidence is the infimum, not an imputed value). Pass
54
+ `FusionSettings(strategy_name="reciprocal_rank_fusion")` to fuse by rank
55
+ instead — then only `channel_ranks` matters.
56
+
57
+ ## API
58
+
59
+ | Symbol | Purpose |
60
+ |---|---|
61
+ | `FusionSettings` | Immutable strategy + parameters (`strategy_name`, `semantic_weight_alpha`, `rank_constant_eta`). |
62
+ | `fuse_channel_scores(...)` | Fuse one candidate's channel evidence into a single score under the chosen strategy. |
63
+ | `convex_combination_score(...)` | TM2C2 over already-normalized `[0, 1]` scores. |
64
+ | `reciprocal_rank_fusion_score(ranks, eta=60)` | RRF over 1-based per-channel ranks. |
65
+ | `theoretical_min_max_normalize(score, bounds)` | Scale a score to `[0, 1]` using a scoring function's theoretical bounds. |
66
+ | `normalize_search_text(text)` | NFC-compose + whitespace-collapse + length-cap a query. |
67
+ | `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS` | `(lower, upper)` tuples for the common lexical/dense pairing. |
68
+
69
+ ## The normalization contract
70
+
71
+ Character-trigram lexical retrieval is language-agnostic only if query
72
+ and documents fold **identically**. `normalize_search_text` owns the
73
+ query side (NFC). Do accent-folding + lowercasing on the **store** side,
74
+ in one place, and call it from both — e.g. a PostgreSQL `IMMUTABLE`
75
+ wrapper `lower(unaccent(normalize(text, NFC)))` used in a `pg_trgm` GiST
76
+ expression index. rankweave stays out of your store so the two sides
77
+ cannot silently diverge.
78
+
79
+ ## Research grounding
80
+
81
+ - **Bruch, Gai & Ingber (2023).** *An Analysis of Fusion Functions for
82
+ Hybrid Retrieval.* ACM TOIS 42(1). arXiv:2210.11934. — TM2C2 > RRF;
83
+ theoretical-normalization stability; the fusion desiderata
84
+ (monotonicity, homogeneity, boundedness, Lipschitz continuity,
85
+ sample efficiency) this library's defaults satisfy.
86
+ - **Cormack, Clarke & Büttcher (2009).** *Reciprocal Rank Fusion
87
+ outperforms Condorcet and individual Rank Learning Methods.* SIGIR
88
+ 2009. — RRF definition, η = 60.
89
+ - **UAX #15**, Unicode Normalization Forms — NFC composition.
90
+
91
+ PDFs and a citation manifest live in [`docs/research/`](docs/research/).
92
+
93
+ ## Development
94
+
95
+ ```bash
96
+ pip install -e ".[dev]"
97
+ python -m pytest -q # 27 tests, no external services
98
+ python -m ruff check .
99
+ ```
100
+
101
+ ## License
102
+
103
+ Apache-2.0 — see [LICENSE](LICENSE).
@@ -0,0 +1,41 @@
1
+ # Research grounding — rankweave
2
+
3
+ rankweave's defaults are not arbitrary; each is the published,
4
+ empirically-supported choice. This directory preserves the source
5
+ material so the grounding travels with the code.
6
+
7
+ ## Papers
8
+
9
+ | File | Citation | License / redistribution |
10
+ |---|---|---|
11
+ | `pdfs/bruch-gai-ingber-2023-analysis-fusion-functions-hybrid-retrieval.pdf` | Bruch, S., Gai, S., & Ingber, A. (2023). *An Analysis of Fusion Functions for Hybrid Retrieval.* ACM Transactions on Information Systems 42(1). arXiv:2210.11934. | cite-only pending license confirmation |
12
+ | `pdfs/cormack-clarke-buettcher-2009-reciprocal-rank-fusion.pdf` | Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). *Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.* SIGIR 2009. | cite-only pending license confirmation |
13
+
14
+ Standards: **UAX #15 — Unicode Normalization Forms** (Unicode
15
+ Consortium), the basis for `normalize_search_text`'s NFC step.
16
+
17
+ ## What each grounds
18
+
19
+ - **Bruch, Gai & Ingber 2023** → the **default strategy**. TM2C2 (a
20
+ convex combination of *theoretically* min-max normalized scores)
21
+ outperforms Reciprocal Rank Fusion in- and out-of-domain (their
22
+ Tables 2–4); the choice of normalization is immaterial for a convex
23
+ combination (§4.2); `alpha ∈ [0.6, 0.8]` is a robust range needing no
24
+ training data (we default to 0.7). Their five desiderata —
25
+ monotonicity, homogeneity, boundedness, Lipschitz continuity, sample
26
+ efficiency — are exactly the properties `convex_combination_score`
27
+ provides and `reciprocal_rank_fusion_score` (a function of ranks,
28
+ not scores) does not.
29
+ - **Cormack, Clarke & Büttcher 2009** → the **RRF alternative** and its
30
+ `eta = 60` default.
31
+ - **UAX #15** → NFC composition, so decomposed Vietnamese/Korean input
32
+ matches composed indexed text.
33
+
34
+ ## PDF preservation note
35
+
36
+ Git LFS is intentionally **not** used; PDFs are committed as regular
37
+ binaries. Where a PDF is absent, it is because the authoring
38
+ environment's network policy blocked the source host (e.g. arxiv.org);
39
+ the citation + arXiv id above make the drop mechanical from a
40
+ network-allowed session. Only permissively-redistributable PDFs are
41
+ committed; others stay cite-only until their license is confirmed.
@@ -0,0 +1,56 @@
1
+ [build-system]
2
+ requires = ["hatchling"]
3
+ build-backend = "hatchling.build"
4
+
5
+ [project]
6
+ name = "rankweave"
7
+ version = "0.1.0"
8
+ description = "Language-agnostic hybrid-retrieval score fusion (TM2C2 + RRF) and Unicode query normalization — pure-Python, store-agnostic."
9
+ readme = "README.md"
10
+ requires-python = ">=3.10"
11
+ license = "Apache-2.0"
12
+ license-files = ["LICENSE"]
13
+ authors = [{ name = "Contextual Wisdom Lab" }]
14
+ keywords = [
15
+ "information-retrieval",
16
+ "hybrid-search",
17
+ "rank-fusion",
18
+ "reciprocal-rank-fusion",
19
+ "rrf",
20
+ "dense-retrieval",
21
+ "lexical-retrieval",
22
+ "pgvector",
23
+ "pg_trgm",
24
+ ]
25
+ classifiers = [
26
+ "Development Status :: 4 - Beta",
27
+ "Intended Audience :: Developers",
28
+ "License :: OSI Approved :: Apache Software License",
29
+ "Programming Language :: Python :: 3",
30
+ "Programming Language :: Python :: 3 :: Only",
31
+ "Topic :: Text Processing :: Indexing",
32
+ "Topic :: Scientific/Engineering :: Information Analysis",
33
+ "Typing :: Typed",
34
+ ]
35
+ dependencies = []
36
+
37
+ [project.urls]
38
+ Homepage = "https://github.com/ContextualWisdomLab/rankweave"
39
+ Source = "https://github.com/ContextualWisdomLab/rankweave"
40
+ Issues = "https://github.com/ContextualWisdomLab/rankweave/issues"
41
+
42
+ [project.optional-dependencies]
43
+ dev = ["pytest>=8", "ruff>=0.6"]
44
+
45
+ [tool.hatch.build.targets.wheel]
46
+ packages = ["src/rankweave"]
47
+
48
+ [tool.pytest.ini_options]
49
+ testpaths = ["tests"]
50
+
51
+ [tool.ruff]
52
+ line-length = 88
53
+ target-version = "py310"
54
+
55
+ [tool.ruff.lint]
56
+ select = ["E", "F", "I", "UP", "B"]
@@ -0,0 +1,62 @@
1
+ """rankweave — language-agnostic hybrid-retrieval score fusion.
2
+
3
+ Pure-Python (stdlib-only) fusion of lexical and semantic retrieval
4
+ channels, plus Unicode NFC query normalization. Store-agnostic: bring
5
+ your own dense (embedding) and lexical (character-trigram / BM25 /
6
+ learned-sparse) channels; rankweave decides how to combine their
7
+ scores.
8
+
9
+ Two fusion strategies, research-grounded (see ``docs/research/``):
10
+
11
+ - ``convex_combination`` (default, "TM2C2") — Bruch, Gai & Ingber
12
+ 2023 (arXiv:2210.11934): a convex combination of theoretically
13
+ min-max normalized scores; robust, distribution-preserving, no
14
+ training data needed.
15
+ - ``reciprocal_rank_fusion`` — Cormack, Clarke & Büttcher 2009: the
16
+ non-parametric rank-only alternative.
17
+
18
+ Quickstart::
19
+
20
+ from rankweave import FusionSettings, fuse_channel_scores
21
+
22
+ settings = FusionSettings() # TM2C2, alpha=0.7
23
+ score = fuse_channel_scores(
24
+ word_similarity_score=0.62, # lexical channel, [0, 1]
25
+ cosine_distance=0.30, # dense channel, [0, 2]
26
+ channel_ranks={"lexical": 1, "dense": 1},
27
+ settings=settings,
28
+ )
29
+ """
30
+
31
+ from rankweave.query_normalization import (
32
+ DEFAULT_MAX_QUERY_CHARACTER_LENGTH,
33
+ normalize_search_text,
34
+ )
35
+ from rankweave.score_fusion import (
36
+ CONVEX_COMBINATION_STRATEGY,
37
+ COSINE_DISTANCE_THEORETICAL_BOUNDS,
38
+ RECIPROCAL_RANK_STRATEGY,
39
+ WORD_SIMILARITY_THEORETICAL_BOUNDS,
40
+ FusionSettings,
41
+ convex_combination_score,
42
+ fuse_channel_scores,
43
+ reciprocal_rank_fusion_score,
44
+ theoretical_min_max_normalize,
45
+ )
46
+
47
+ __version__ = "0.1.0"
48
+
49
+ __all__ = [
50
+ "CONVEX_COMBINATION_STRATEGY",
51
+ "COSINE_DISTANCE_THEORETICAL_BOUNDS",
52
+ "DEFAULT_MAX_QUERY_CHARACTER_LENGTH",
53
+ "RECIPROCAL_RANK_STRATEGY",
54
+ "WORD_SIMILARITY_THEORETICAL_BOUNDS",
55
+ "FusionSettings",
56
+ "convex_combination_score",
57
+ "fuse_channel_scores",
58
+ "normalize_search_text",
59
+ "reciprocal_rank_fusion_score",
60
+ "theoretical_min_max_normalize",
61
+ "__version__",
62
+ ]
File without changes
@@ -0,0 +1,44 @@
1
+ """Query-side text normalization for language-agnostic search.
2
+
3
+ Character-trigram lexical retrieval is language-agnostic only if the
4
+ query and the indexed documents fold *identically*. This module owns
5
+ the query-side half of that contract; the store side must mirror it.
6
+
7
+ 1. **Unicode NFC composition (UAX #15).** Vietnamese and Korean text
8
+ arrives in mixed composed/decomposed forms depending on the source
9
+ platform (macOS filenames and some webmail clients emit NFD), so
10
+ both sides must compose before comparison.
11
+ 2. **Accent folding + lowercasing belong on the store side.** Do them
12
+ where the documents are indexed so the indexed expression and the
13
+ bound query parameter go through the identical path — for example a
14
+ PostgreSQL ``IMMUTABLE`` wrapper
15
+ ``lower(unaccent(normalize(text, NFC)))`` used in a ``pg_trgm``
16
+ GiST expression index. Keep that transform in one place and call it
17
+ from both sides.
18
+
19
+ Only whitespace shaping and NFC are done here; anything that depends
20
+ on the store's runtime (accent dictionaries, collations) stays on the
21
+ store side so it cannot silently diverge.
22
+ """
23
+
24
+ import unicodedata
25
+
26
+ DEFAULT_MAX_QUERY_CHARACTER_LENGTH = 1000
27
+
28
+
29
+ def normalize_search_text(
30
+ raw_text: str,
31
+ *,
32
+ max_characters: int = DEFAULT_MAX_QUERY_CHARACTER_LENGTH,
33
+ ) -> str:
34
+ """Compose the query to NFC and collapse insignificant whitespace.
35
+
36
+ ``max_characters`` caps pathological queries; set it to match the
37
+ store-side bound. Returns the composed, whitespace-collapsed,
38
+ length-capped query text.
39
+ """
40
+ if max_characters < 1:
41
+ raise ValueError("max_characters must be >= 1")
42
+ composed_text = unicodedata.normalize("NFC", raw_text)
43
+ collapsed_text = " ".join(composed_text.split())
44
+ return collapsed_text[:max_characters]
@@ -0,0 +1,160 @@
1
+ """Fusion functions for hybrid (lexical + semantic) retrieval.
2
+
3
+ Default strategy: a convex combination of theoretically min-max
4
+ normalized channel scores ("TM2C2"; Bruch, Gai & Ingber 2023,
5
+ *An Analysis of Fusion Functions for Hybrid Retrieval*, ACM TOIS
6
+ 42(1), arXiv:2210.11934). Their analysis shows TM2C2 outperforms
7
+ Reciprocal Rank Fusion in- and out-of-domain, is robust for alpha in
8
+ [0.6, 0.8] without training data, and — unlike RRF — preserves the
9
+ score distribution (Lipschitz continuity).
10
+
11
+ Reciprocal Rank Fusion (Cormack, Clarke & Büttcher 2009,
12
+ *Reciprocal Rank Fusion outperforms Condorcet and individual Rank
13
+ Learning Methods*, SIGIR) is the non-parametric alternative for
14
+ channels that expose only ranks (learned-sparse or external
15
+ channels), selected via ``FusionSettings.strategy_name``.
16
+
17
+ The convex strategy assumes each channel score has *theoretical*
18
+ bounds, so no per-query data-dependent normalization is needed. Two
19
+ bound constants ship for the common lexical+dense pairing, but they
20
+ are just ``(lower, upper)`` tuples — pass your own to
21
+ ``theoretical_min_max_normalize`` for any bounded scoring function:
22
+
23
+ - ``WORD_SIMILARITY_THEORETICAL_BOUNDS`` = (0.0, 1.0) — e.g. a
24
+ character-trigram word-similarity such as PostgreSQL ``pg_trgm``.
25
+ - ``COSINE_DISTANCE_THEORETICAL_BOUNDS`` = (0.0, 2.0) — cosine
26
+ distance for unit-norm vectors, e.g. a pgvector ``<=>`` channel;
27
+ fuse_channel_scores inverts it so smaller distance scores higher.
28
+ """
29
+
30
+ from dataclasses import dataclass
31
+
32
+ WORD_SIMILARITY_THEORETICAL_BOUNDS = (0.0, 1.0)
33
+ COSINE_DISTANCE_THEORETICAL_BOUNDS = (0.0, 2.0)
34
+
35
+ CONVEX_COMBINATION_STRATEGY = "convex_combination"
36
+ RECIPROCAL_RANK_STRATEGY = "reciprocal_rank_fusion"
37
+
38
+ _SUPPORTED_STRATEGY_NAMES = frozenset(
39
+ {CONVEX_COMBINATION_STRATEGY, RECIPROCAL_RANK_STRATEGY}
40
+ )
41
+
42
+
43
+ @dataclass(frozen=True)
44
+ class FusionSettings:
45
+ """Tunable fusion parameters (immutable; construct one per query set)."""
46
+
47
+ strategy_name: str = CONVEX_COMBINATION_STRATEGY
48
+ # Weight of the semantic channel; 0.7 is the midpoint of the
49
+ # robust [0.6, 0.8] range reported by Bruch et al. (2023).
50
+ semantic_weight_alpha: float = 0.7
51
+ # RRF eta; 60 per Cormack et al. (2009).
52
+ rank_constant_eta: int = 60
53
+
54
+ def __post_init__(self) -> None:
55
+ if self.strategy_name not in _SUPPORTED_STRATEGY_NAMES:
56
+ raise ValueError(
57
+ "strategy_name must be one of "
58
+ f"{sorted(_SUPPORTED_STRATEGY_NAMES)}, got {self.strategy_name!r}"
59
+ )
60
+ if not 0.0 <= self.semantic_weight_alpha <= 1.0:
61
+ raise ValueError("semantic_weight_alpha must be within [0, 1]")
62
+ if self.rank_constant_eta < 1:
63
+ raise ValueError("rank_constant_eta must be >= 1")
64
+
65
+
66
+ def theoretical_min_max_normalize(
67
+ score: float, bounds: tuple[float, float]
68
+ ) -> float:
69
+ """Scale a score to [0, 1] using the scoring function's theoretical bounds.
70
+
71
+ Using theoretical rather than observed bounds keeps the transform
72
+ stable across queries and candidate sets (Bruch et al. 2023, §4.2).
73
+ Out-of-range inputs (floating-point drift) are clamped.
74
+ """
75
+ lower_bound, upper_bound = bounds
76
+ if upper_bound <= lower_bound:
77
+ raise ValueError("bounds must satisfy upper > lower")
78
+ normalized = (score - lower_bound) / (upper_bound - lower_bound)
79
+ return min(1.0, max(0.0, normalized))
80
+
81
+
82
+ def convex_combination_score(
83
+ semantic_score: float | None,
84
+ lexical_score: float | None,
85
+ semantic_weight_alpha: float,
86
+ ) -> float:
87
+ """TM2C2 fusion over already-normalized [0, 1] channel scores.
88
+
89
+ A channel absent for a candidate (e.g. no embedding stored yet)
90
+ contributes its theoretical minimum, 0 — absent evidence is the
91
+ infimum, not a missing value to impute.
92
+ """
93
+ semantic_component = semantic_score if semantic_score is not None else 0.0
94
+ lexical_component = lexical_score if lexical_score is not None else 0.0
95
+ return (
96
+ semantic_weight_alpha * semantic_component
97
+ + (1.0 - semantic_weight_alpha) * lexical_component
98
+ )
99
+
100
+
101
+ def reciprocal_rank_fusion_score(
102
+ channel_ranks: dict[str, int], rank_constant_eta: int = 60
103
+ ) -> float:
104
+ """RRF over 1-based per-channel ranks: sum of 1 / (eta + rank)."""
105
+ if rank_constant_eta < 1:
106
+ raise ValueError("rank_constant_eta must be >= 1")
107
+ fused_score = 0.0
108
+ for channel_name, one_based_rank in channel_ranks.items():
109
+ if one_based_rank < 1:
110
+ raise ValueError(
111
+ f"rank for channel {channel_name!r} must be >= 1,"
112
+ f" got {one_based_rank}"
113
+ )
114
+ fused_score += 1.0 / (rank_constant_eta + one_based_rank)
115
+ return fused_score
116
+
117
+
118
+ def fuse_channel_scores(
119
+ *,
120
+ word_similarity_score: float | None,
121
+ cosine_distance: float | None,
122
+ channel_ranks: dict[str, int],
123
+ settings: FusionSettings,
124
+ ) -> float:
125
+ """Fuse one candidate's channel evidence into a single score.
126
+
127
+ ``word_similarity_score`` and ``cosine_distance`` are the raw
128
+ channel outputs (None when the channel did not produce this
129
+ candidate); ``channel_ranks`` are the candidate's 1-based ranks in
130
+ the channels that returned it, used by the RRF strategy.
131
+ """
132
+ if settings.strategy_name == RECIPROCAL_RANK_STRATEGY:
133
+ if not channel_ranks:
134
+ return 0.0
135
+ return reciprocal_rank_fusion_score(
136
+ channel_ranks, settings.rank_constant_eta
137
+ )
138
+
139
+ normalized_lexical_score = (
140
+ theoretical_min_max_normalize(
141
+ word_similarity_score, WORD_SIMILARITY_THEORETICAL_BOUNDS
142
+ )
143
+ if word_similarity_score is not None
144
+ else None
145
+ )
146
+ # Cosine *distance* decreases as relevance increases; invert inside
147
+ # the theoretical [0, 2] range so 1.0 means identical direction.
148
+ normalized_semantic_score = (
149
+ 1.0
150
+ - theoretical_min_max_normalize(
151
+ cosine_distance, COSINE_DISTANCE_THEORETICAL_BOUNDS
152
+ )
153
+ if cosine_distance is not None
154
+ else None
155
+ )
156
+ return convex_combination_score(
157
+ normalized_semantic_score,
158
+ normalized_lexical_score,
159
+ settings.semantic_weight_alpha,
160
+ )
@@ -0,0 +1,204 @@
1
+ import math
2
+
3
+ import pytest
4
+
5
+ from rankweave import (
6
+ COSINE_DISTANCE_THEORETICAL_BOUNDS,
7
+ WORD_SIMILARITY_THEORETICAL_BOUNDS,
8
+ FusionSettings,
9
+ convex_combination_score,
10
+ fuse_channel_scores,
11
+ normalize_search_text,
12
+ reciprocal_rank_fusion_score,
13
+ theoretical_min_max_normalize,
14
+ )
15
+ from rankweave.score_fusion import (
16
+ CONVEX_COMBINATION_STRATEGY,
17
+ RECIPROCAL_RANK_STRATEGY,
18
+ )
19
+
20
+
21
+ class TestNormalizeSearchText:
22
+ def test_composes_decomposed_hangul_to_nfc(self):
23
+ # U+1112 U+1161 U+11AB (decomposed jamo) -> U+D55C
24
+ decomposed_hangul = "\u1112\u1161\u11ab"
25
+ assert normalize_search_text(decomposed_hangul) == "\ud55c"
26
+
27
+ def test_composes_decomposed_vietnamese_diacritics(self):
28
+ # "a" + combining circumflex + combining grave -> U+1EA7
29
+ decomposed_vietnamese = "Tra\u0302\u0300n"
30
+ assert normalize_search_text(decomposed_vietnamese) == "Tr\u1ea7n"
31
+
32
+ def test_collapses_whitespace_and_strips(self):
33
+ assert normalize_search_text(" hello \t world \n") == "hello world"
34
+
35
+ def test_caps_pathological_query_length(self):
36
+ assert len(normalize_search_text("가" * 5000)) == 1000
37
+
38
+ def test_keeps_cjk_text_intact_without_tokenization(self):
39
+ korean_query = "다음주 회의 일정"
40
+ assert normalize_search_text(korean_query) == korean_query
41
+
42
+ def test_max_characters_override(self):
43
+ assert len(normalize_search_text("a" * 50, max_characters=10)) == 10
44
+
45
+ def test_rejects_non_positive_max_characters(self):
46
+ with pytest.raises(ValueError):
47
+ normalize_search_text("hello", max_characters=0)
48
+
49
+
50
+ class TestTheoreticalMinMaxNormalize:
51
+ def test_word_similarity_bounds_map_to_unit_interval(self):
52
+ assert theoretical_min_max_normalize(
53
+ 0.0, WORD_SIMILARITY_THEORETICAL_BOUNDS
54
+ ) == 0.0
55
+ assert theoretical_min_max_normalize(
56
+ 1.0, WORD_SIMILARITY_THEORETICAL_BOUNDS
57
+ ) == 1.0
58
+ assert theoretical_min_max_normalize(
59
+ 0.25, WORD_SIMILARITY_THEORETICAL_BOUNDS
60
+ ) == pytest.approx(0.25)
61
+
62
+ def test_cosine_distance_bounds_map_to_unit_interval(self):
63
+ assert theoretical_min_max_normalize(
64
+ 0.0, COSINE_DISTANCE_THEORETICAL_BOUNDS
65
+ ) == 0.0
66
+ assert theoretical_min_max_normalize(
67
+ 2.0, COSINE_DISTANCE_THEORETICAL_BOUNDS
68
+ ) == 1.0
69
+ assert theoretical_min_max_normalize(
70
+ 0.5, COSINE_DISTANCE_THEORETICAL_BOUNDS
71
+ ) == pytest.approx(0.25)
72
+
73
+ def test_clamps_floating_point_drift(self):
74
+ assert theoretical_min_max_normalize(
75
+ 1.0000001, WORD_SIMILARITY_THEORETICAL_BOUNDS
76
+ ) == 1.0
77
+ assert theoretical_min_max_normalize(
78
+ -0.0000001, WORD_SIMILARITY_THEORETICAL_BOUNDS
79
+ ) == 0.0
80
+
81
+ def test_rejects_inverted_bounds(self):
82
+ with pytest.raises(ValueError):
83
+ theoretical_min_max_normalize(0.5, (1.0, 0.0))
84
+
85
+
86
+ class TestConvexCombinationScore:
87
+ def test_hand_computed_fusion(self):
88
+ # alpha * semantic + (1 - alpha) * lexical
89
+ fused_score = convex_combination_score(0.8, 0.5, 0.7)
90
+ assert fused_score == pytest.approx(0.7 * 0.8 + 0.3 * 0.5)
91
+
92
+ def test_missing_channel_contributes_theoretical_minimum(self):
93
+ assert convex_combination_score(None, 0.5, 0.7) == pytest.approx(0.15)
94
+ assert convex_combination_score(0.8, None, 0.7) == pytest.approx(0.56)
95
+ assert convex_combination_score(None, None, 0.7) == 0.0
96
+
97
+ def test_alpha_extremes_select_single_channel(self):
98
+ assert convex_combination_score(0.9, 0.4, 1.0) == pytest.approx(0.9)
99
+ assert convex_combination_score(0.9, 0.4, 0.0) == pytest.approx(0.4)
100
+
101
+ def test_monotone_in_each_channel(self):
102
+ base_score = convex_combination_score(0.5, 0.5, 0.7)
103
+ assert convex_combination_score(0.6, 0.5, 0.7) > base_score
104
+ assert convex_combination_score(0.5, 0.6, 0.7) > base_score
105
+
106
+ def test_bounded_in_unit_interval(self):
107
+ assert 0.0 <= convex_combination_score(1.0, 1.0, 0.7) <= 1.0
108
+ assert 0.0 <= convex_combination_score(0.0, 0.0, 0.7) <= 1.0
109
+
110
+
111
+ class TestReciprocalRankFusionScore:
112
+ def test_hand_computed_rrf(self):
113
+ fused_score = reciprocal_rank_fusion_score(
114
+ {"lexical_email": 1, "dense_email": 3}, rank_constant_eta=60
115
+ )
116
+ assert fused_score == pytest.approx(1.0 / 61.0 + 1.0 / 63.0)
117
+
118
+ def test_lower_rank_scores_higher(self):
119
+ better_ranked = reciprocal_rank_fusion_score({"channel": 1})
120
+ worse_ranked = reciprocal_rank_fusion_score({"channel": 10})
121
+ assert better_ranked > worse_ranked
122
+
123
+ def test_more_channels_score_higher(self):
124
+ single_channel = reciprocal_rank_fusion_score({"a": 5})
125
+ two_channels = reciprocal_rank_fusion_score({"a": 5, "b": 5})
126
+ assert two_channels > single_channel
127
+
128
+ def test_rejects_invalid_rank_and_eta(self):
129
+ with pytest.raises(ValueError):
130
+ reciprocal_rank_fusion_score({"a": 0})
131
+ with pytest.raises(ValueError):
132
+ reciprocal_rank_fusion_score({"a": 1}, rank_constant_eta=0)
133
+
134
+
135
+ class TestFusionSettings:
136
+ def test_defaults_follow_research_grounding(self):
137
+ settings = FusionSettings()
138
+ assert settings.strategy_name == CONVEX_COMBINATION_STRATEGY
139
+ assert settings.semantic_weight_alpha == 0.7
140
+ assert settings.rank_constant_eta == 60
141
+
142
+ def test_rejects_unknown_strategy(self):
143
+ with pytest.raises(ValueError):
144
+ FusionSettings(strategy_name="borda_count")
145
+
146
+ def test_rejects_out_of_range_alpha(self):
147
+ with pytest.raises(ValueError):
148
+ FusionSettings(semantic_weight_alpha=1.5)
149
+
150
+
151
+ class TestFuseChannelScores:
152
+ def test_convex_strategy_normalizes_and_inverts_distance(self):
153
+ settings = FusionSettings()
154
+ fused_score = fuse_channel_scores(
155
+ word_similarity_score=0.5,
156
+ cosine_distance=0.4,
157
+ channel_ranks={"lexical_email": 1, "dense_email": 1},
158
+ settings=settings,
159
+ )
160
+ expected_semantic = 1.0 - 0.4 / 2.0 # 0.8
161
+ assert fused_score == pytest.approx(0.7 * expected_semantic + 0.3 * 0.5)
162
+
163
+ def test_identical_vectors_and_exact_word_match_score_one(self):
164
+ settings = FusionSettings()
165
+ fused_score = fuse_channel_scores(
166
+ word_similarity_score=1.0,
167
+ cosine_distance=0.0,
168
+ channel_ranks={"lexical_email": 1, "dense_email": 1},
169
+ settings=settings,
170
+ )
171
+ assert fused_score == pytest.approx(1.0)
172
+
173
+ def test_rrf_strategy_uses_ranks_only(self):
174
+ settings = FusionSettings(strategy_name=RECIPROCAL_RANK_STRATEGY)
175
+ fused_score = fuse_channel_scores(
176
+ word_similarity_score=0.99,
177
+ cosine_distance=0.01,
178
+ channel_ranks={"lexical_email": 2, "dense_email": 4},
179
+ settings=settings,
180
+ )
181
+ assert fused_score == pytest.approx(1.0 / 62.0 + 1.0 / 64.0)
182
+
183
+ def test_rrf_strategy_with_no_ranks_scores_zero(self):
184
+ settings = FusionSettings(strategy_name=RECIPROCAL_RANK_STRATEGY)
185
+ assert (
186
+ fuse_channel_scores(
187
+ word_similarity_score=None,
188
+ cosine_distance=None,
189
+ channel_ranks={},
190
+ settings=settings,
191
+ )
192
+ == 0.0
193
+ )
194
+
195
+ def test_lexical_only_candidate_is_finite_and_positive(self):
196
+ settings = FusionSettings()
197
+ fused_score = fuse_channel_scores(
198
+ word_similarity_score=0.6,
199
+ cosine_distance=None,
200
+ channel_ranks={"lexical_email": 1},
201
+ settings=settings,
202
+ )
203
+ assert math.isfinite(fused_score)
204
+ assert fused_score == pytest.approx(0.3 * 0.6)