rankweave 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- rankweave-0.1.0/.github/workflows/ci.yml +26 -0
- rankweave-0.1.0/.github/workflows/publish.yml +26 -0
- rankweave-0.1.0/.gitignore +12 -0
- rankweave-0.1.0/AGENTS.md +45 -0
- rankweave-0.1.0/CHANGELOG.md +25 -0
- rankweave-0.1.0/LICENSE +201 -0
- rankweave-0.1.0/PKG-INFO +128 -0
- rankweave-0.1.0/README.md +103 -0
- rankweave-0.1.0/docs/research/README.md +41 -0
- rankweave-0.1.0/docs/research/pdfs/bruch-gai-ingber-2023-analysis-fusion-functions-hybrid-retrieval.pdf +0 -0
- rankweave-0.1.0/docs/research/pdfs/cormack-clarke-buettcher-2009-reciprocal-rank-fusion.pdf +0 -0
- rankweave-0.1.0/pyproject.toml +56 -0
- rankweave-0.1.0/src/rankweave/__init__.py +62 -0
- rankweave-0.1.0/src/rankweave/py.typed +0 -0
- rankweave-0.1.0/src/rankweave/query_normalization.py +44 -0
- rankweave-0.1.0/src/rankweave/score_fusion.py +160 -0
- rankweave-0.1.0/tests/test_fusion.py +204 -0
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
test:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
strategy:
|
|
15
|
+
fail-fast: false
|
|
16
|
+
matrix:
|
|
17
|
+
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v4
|
|
20
|
+
- uses: actions/setup-python@v5
|
|
21
|
+
with:
|
|
22
|
+
python-version: ${{ matrix.python-version }}
|
|
23
|
+
- run: python -m pip install --upgrade pip
|
|
24
|
+
- run: pip install -e ".[dev]"
|
|
25
|
+
- run: python -m ruff check .
|
|
26
|
+
- run: python -m pytest -q
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
name: publish
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: ["publish-release"]
|
|
6
|
+
workflow_dispatch: {}
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
pypi:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v4
|
|
16
|
+
- uses: actions/setup-python@v5
|
|
17
|
+
with:
|
|
18
|
+
python-version: "3.12"
|
|
19
|
+
- run: python -m pip install --upgrade build twine
|
|
20
|
+
- run: python -m build
|
|
21
|
+
- run: twine check dist/*
|
|
22
|
+
- name: Upload to PyPI
|
|
23
|
+
env:
|
|
24
|
+
TWINE_USERNAME: __token__
|
|
25
|
+
TWINE_PASSWORD: ${{ secrets.PIPY_TOKEN }}
|
|
26
|
+
run: twine upload --non-interactive --disable-progress-bar dist/*
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# AGENTS.md — rankweave
|
|
2
|
+
|
|
3
|
+
Operating guide for automated agents working in this repo.
|
|
4
|
+
|
|
5
|
+
## What this is
|
|
6
|
+
|
|
7
|
+
`rankweave` is a **pure-Python, stdlib-only** library for
|
|
8
|
+
language-agnostic hybrid-retrieval score fusion, extracted unchanged
|
|
9
|
+
in behavior from
|
|
10
|
+
[ContextualWisdomLab/naruon](https://github.com/ContextualWisdomLab/naruon)
|
|
11
|
+
Context Search under the lab's ONE SOURCE MULTI USE convention
|
|
12
|
+
(standalone product *and* submodule-importable).
|
|
13
|
+
|
|
14
|
+
## Hard rules
|
|
15
|
+
|
|
16
|
+
- **No dependencies.** The library imports only the Python standard
|
|
17
|
+
library. Do not add a runtime dependency; if you think you need one,
|
|
18
|
+
the feature probably belongs in the consumer, not here.
|
|
19
|
+
- **Store-agnostic.** rankweave never talks to a database, an
|
|
20
|
+
embedding provider, or a search index. It fuses scores and normalizes
|
|
21
|
+
query text. Keep SQL, HTTP, and ORM concerns out.
|
|
22
|
+
- **Behavior parity with naruon.** This is an extraction, not a fork.
|
|
23
|
+
A behavior change here must be mirrored in naruon's
|
|
24
|
+
`services/hybrid_retrieval` (and vice versa) until naruon consumes
|
|
25
|
+
this package directly. Prefer additive, backward-compatible changes.
|
|
26
|
+
- **Permissive license only** (Apache-2.0). Any added code or asset must
|
|
27
|
+
be compatible.
|
|
28
|
+
- **Research-grounded defaults.** Numeric defaults (alpha=0.7, eta=60,
|
|
29
|
+
the theoretical bounds) trace to the papers in `docs/research/`.
|
|
30
|
+
Changing a default requires citing the evidence.
|
|
31
|
+
|
|
32
|
+
## Develop
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
pip install -e ".[dev]"
|
|
36
|
+
python -m pytest -q # no external services required
|
|
37
|
+
python -m ruff check .
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Layout
|
|
41
|
+
|
|
42
|
+
- `src/rankweave/score_fusion.py` — TM2C2 + RRF fusion primitives.
|
|
43
|
+
- `src/rankweave/query_normalization.py` — NFC query normalization.
|
|
44
|
+
- `tests/` — behavior tests (hand-computed expected values).
|
|
45
|
+
- `docs/research/` — paper PDFs + citation manifest.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to rankweave are documented here. The format
|
|
4
|
+
follows [Keep a Changelog](https://keepachangelog.com/), and the
|
|
5
|
+
project adheres to [Semantic Versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [0.1.0] — 2026-07-11
|
|
8
|
+
|
|
9
|
+
Initial extraction from
|
|
10
|
+
[ContextualWisdomLab/naruon](https://github.com/ContextualWisdomLab/naruon)
|
|
11
|
+
Context Search, unchanged in behavior (ONE SOURCE MULTI USE).
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
- `fuse_channel_scores` — fuse one candidate's lexical + dense channel
|
|
15
|
+
evidence into a single bounded score under the selected strategy.
|
|
16
|
+
- `FusionSettings` — immutable strategy + parameters (`convex_combination`
|
|
17
|
+
default with `semantic_weight_alpha=0.7`; `reciprocal_rank_fusion`
|
|
18
|
+
with `rank_constant_eta=60`).
|
|
19
|
+
- `convex_combination_score`, `reciprocal_rank_fusion_score`,
|
|
20
|
+
`theoretical_min_max_normalize` — the underlying fusion primitives.
|
|
21
|
+
- `normalize_search_text` — NFC compose + whitespace-collapse +
|
|
22
|
+
length-cap for the query side of a language-agnostic lexical channel.
|
|
23
|
+
- `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS`.
|
|
24
|
+
- 27 unit tests; no dependencies (stdlib only); typed (`py.typed`).
|
|
25
|
+
- Research grounding + paper manifest under `docs/research/`.
|
rankweave-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright (c) 2026 Contextual Wisdom Lab / Seongho Bae
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|
rankweave-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: rankweave
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Language-agnostic hybrid-retrieval score fusion (TM2C2 + RRF) and Unicode query normalization — pure-Python, store-agnostic.
|
|
5
|
+
Project-URL: Homepage, https://github.com/ContextualWisdomLab/rankweave
|
|
6
|
+
Project-URL: Source, https://github.com/ContextualWisdomLab/rankweave
|
|
7
|
+
Project-URL: Issues, https://github.com/ContextualWisdomLab/rankweave/issues
|
|
8
|
+
Author: Contextual Wisdom Lab
|
|
9
|
+
License-Expression: Apache-2.0
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: dense-retrieval,hybrid-search,information-retrieval,lexical-retrieval,pg_trgm,pgvector,rank-fusion,reciprocal-rank-fusion,rrf
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
18
|
+
Classifier: Topic :: Text Processing :: Indexing
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Python: >=3.10
|
|
21
|
+
Provides-Extra: dev
|
|
22
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
23
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
# rankweave
|
|
27
|
+
|
|
28
|
+
**Language-agnostic hybrid-retrieval score fusion — pure-Python, store-agnostic.**
|
|
29
|
+
|
|
30
|
+
`rankweave` decides *how to combine* the scores from a lexical channel
|
|
31
|
+
(character-trigram / BM25 / learned-sparse) and a semantic channel
|
|
32
|
+
(dense embeddings) into one ranking. It ships two research-grounded
|
|
33
|
+
fusion strategies and the query-side Unicode normalization that makes
|
|
34
|
+
character-level lexical matching language-agnostic. It has **no
|
|
35
|
+
dependencies** (stdlib only) and **no opinion about your store** — bring
|
|
36
|
+
your own channels; rankweave fuses their scores.
|
|
37
|
+
|
|
38
|
+
It is extracted, unchanged in behavior, from the Context Search engine of
|
|
39
|
+
[naruon](https://github.com/ContextualWisdomLab/naruon), following the
|
|
40
|
+
lab's ONE SOURCE MULTI USE convention: standalone product *and*
|
|
41
|
+
submodule-importable.
|
|
42
|
+
|
|
43
|
+
## Why
|
|
44
|
+
|
|
45
|
+
A convex combination of **theoretically** min-max normalized scores
|
|
46
|
+
(TM2C2) beats Reciprocal Rank Fusion in- and out-of-domain, is robust
|
|
47
|
+
for `alpha ∈ [0.6, 0.8]` with no training data, and — unlike rank
|
|
48
|
+
fusion — preserves the score distribution (Bruch, Gai & Ingber 2023).
|
|
49
|
+
RRF remains available for channels that expose only ranks. See
|
|
50
|
+
[`docs/research/`](docs/research/) for the grounding.
|
|
51
|
+
|
|
52
|
+
## Install
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
pip install rankweave
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Quickstart
|
|
59
|
+
|
|
60
|
+
```python
|
|
61
|
+
from rankweave import FusionSettings, fuse_channel_scores, normalize_search_text
|
|
62
|
+
|
|
63
|
+
# 1) Normalize the query the same way you normalize indexed documents
|
|
64
|
+
# (NFC compose; do accent-folding + lowercasing on the store side too).
|
|
65
|
+
query = normalize_search_text(" Trần Hưng Đạo 회의 ") # -> "Trần Hưng Đạo 회의"
|
|
66
|
+
|
|
67
|
+
# 2) Run your own lexical + dense channels, then fuse per candidate.
|
|
68
|
+
settings = FusionSettings() # TM2C2, semantic weight alpha = 0.7
|
|
69
|
+
score = fuse_channel_scores(
|
|
70
|
+
word_similarity_score=0.62, # lexical channel score in [0, 1]
|
|
71
|
+
cosine_distance=0.30, # dense channel distance in [0, 2]
|
|
72
|
+
channel_ranks={"lexical": 1, "dense": 1},
|
|
73
|
+
settings=settings,
|
|
74
|
+
) # -> bounded [0, 1] fused score
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
A channel that did not return a candidate contributes its theoretical
|
|
78
|
+
minimum (absent evidence is the infimum, not an imputed value). Pass
|
|
79
|
+
`FusionSettings(strategy_name="reciprocal_rank_fusion")` to fuse by rank
|
|
80
|
+
instead — then only `channel_ranks` matters.
|
|
81
|
+
|
|
82
|
+
## API
|
|
83
|
+
|
|
84
|
+
| Symbol | Purpose |
|
|
85
|
+
|---|---|
|
|
86
|
+
| `FusionSettings` | Immutable strategy + parameters (`strategy_name`, `semantic_weight_alpha`, `rank_constant_eta`). |
|
|
87
|
+
| `fuse_channel_scores(...)` | Fuse one candidate's channel evidence into a single score under the chosen strategy. |
|
|
88
|
+
| `convex_combination_score(...)` | TM2C2 over already-normalized `[0, 1]` scores. |
|
|
89
|
+
| `reciprocal_rank_fusion_score(ranks, eta=60)` | RRF over 1-based per-channel ranks. |
|
|
90
|
+
| `theoretical_min_max_normalize(score, bounds)` | Scale a score to `[0, 1]` using a scoring function's theoretical bounds. |
|
|
91
|
+
| `normalize_search_text(text)` | NFC-compose + whitespace-collapse + length-cap a query. |
|
|
92
|
+
| `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS` | `(lower, upper)` tuples for the common lexical/dense pairing. |
|
|
93
|
+
|
|
94
|
+
## The normalization contract
|
|
95
|
+
|
|
96
|
+
Character-trigram lexical retrieval is language-agnostic only if query
|
|
97
|
+
and documents fold **identically**. `normalize_search_text` owns the
|
|
98
|
+
query side (NFC). Do accent-folding + lowercasing on the **store** side,
|
|
99
|
+
in one place, and call it from both — e.g. a PostgreSQL `IMMUTABLE`
|
|
100
|
+
wrapper `lower(unaccent(normalize(text, NFC)))` used in a `pg_trgm` GiST
|
|
101
|
+
expression index. rankweave stays out of your store so the two sides
|
|
102
|
+
cannot silently diverge.
|
|
103
|
+
|
|
104
|
+
## Research grounding
|
|
105
|
+
|
|
106
|
+
- **Bruch, Gai & Ingber (2023).** *An Analysis of Fusion Functions for
|
|
107
|
+
Hybrid Retrieval.* ACM TOIS 42(1). arXiv:2210.11934. — TM2C2 > RRF;
|
|
108
|
+
theoretical-normalization stability; the fusion desiderata
|
|
109
|
+
(monotonicity, homogeneity, boundedness, Lipschitz continuity,
|
|
110
|
+
sample efficiency) this library's defaults satisfy.
|
|
111
|
+
- **Cormack, Clarke & Büttcher (2009).** *Reciprocal Rank Fusion
|
|
112
|
+
outperforms Condorcet and individual Rank Learning Methods.* SIGIR
|
|
113
|
+
2009. — RRF definition, η = 60.
|
|
114
|
+
- **UAX #15**, Unicode Normalization Forms — NFC composition.
|
|
115
|
+
|
|
116
|
+
PDFs and a citation manifest live in [`docs/research/`](docs/research/).
|
|
117
|
+
|
|
118
|
+
## Development
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
pip install -e ".[dev]"
|
|
122
|
+
python -m pytest -q # 27 tests, no external services
|
|
123
|
+
python -m ruff check .
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## License
|
|
127
|
+
|
|
128
|
+
Apache-2.0 — see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# rankweave
|
|
2
|
+
|
|
3
|
+
**Language-agnostic hybrid-retrieval score fusion — pure-Python, store-agnostic.**
|
|
4
|
+
|
|
5
|
+
`rankweave` decides *how to combine* the scores from a lexical channel
|
|
6
|
+
(character-trigram / BM25 / learned-sparse) and a semantic channel
|
|
7
|
+
(dense embeddings) into one ranking. It ships two research-grounded
|
|
8
|
+
fusion strategies and the query-side Unicode normalization that makes
|
|
9
|
+
character-level lexical matching language-agnostic. It has **no
|
|
10
|
+
dependencies** (stdlib only) and **no opinion about your store** — bring
|
|
11
|
+
your own channels; rankweave fuses their scores.
|
|
12
|
+
|
|
13
|
+
It is extracted, unchanged in behavior, from the Context Search engine of
|
|
14
|
+
[naruon](https://github.com/ContextualWisdomLab/naruon), following the
|
|
15
|
+
lab's ONE SOURCE MULTI USE convention: standalone product *and*
|
|
16
|
+
submodule-importable.
|
|
17
|
+
|
|
18
|
+
## Why
|
|
19
|
+
|
|
20
|
+
A convex combination of **theoretically** min-max normalized scores
|
|
21
|
+
(TM2C2) beats Reciprocal Rank Fusion in- and out-of-domain, is robust
|
|
22
|
+
for `alpha ∈ [0.6, 0.8]` with no training data, and — unlike rank
|
|
23
|
+
fusion — preserves the score distribution (Bruch, Gai & Ingber 2023).
|
|
24
|
+
RRF remains available for channels that expose only ranks. See
|
|
25
|
+
[`docs/research/`](docs/research/) for the grounding.
|
|
26
|
+
|
|
27
|
+
## Install
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
pip install rankweave
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Quickstart
|
|
34
|
+
|
|
35
|
+
```python
|
|
36
|
+
from rankweave import FusionSettings, fuse_channel_scores, normalize_search_text
|
|
37
|
+
|
|
38
|
+
# 1) Normalize the query the same way you normalize indexed documents
|
|
39
|
+
# (NFC compose; do accent-folding + lowercasing on the store side too).
|
|
40
|
+
query = normalize_search_text(" Trần Hưng Đạo 회의 ") # -> "Trần Hưng Đạo 회의"
|
|
41
|
+
|
|
42
|
+
# 2) Run your own lexical + dense channels, then fuse per candidate.
|
|
43
|
+
settings = FusionSettings() # TM2C2, semantic weight alpha = 0.7
|
|
44
|
+
score = fuse_channel_scores(
|
|
45
|
+
word_similarity_score=0.62, # lexical channel score in [0, 1]
|
|
46
|
+
cosine_distance=0.30, # dense channel distance in [0, 2]
|
|
47
|
+
channel_ranks={"lexical": 1, "dense": 1},
|
|
48
|
+
settings=settings,
|
|
49
|
+
) # -> bounded [0, 1] fused score
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
A channel that did not return a candidate contributes its theoretical
|
|
53
|
+
minimum (absent evidence is the infimum, not an imputed value). Pass
|
|
54
|
+
`FusionSettings(strategy_name="reciprocal_rank_fusion")` to fuse by rank
|
|
55
|
+
instead — then only `channel_ranks` matters.
|
|
56
|
+
|
|
57
|
+
## API
|
|
58
|
+
|
|
59
|
+
| Symbol | Purpose |
|
|
60
|
+
|---|---|
|
|
61
|
+
| `FusionSettings` | Immutable strategy + parameters (`strategy_name`, `semantic_weight_alpha`, `rank_constant_eta`). |
|
|
62
|
+
| `fuse_channel_scores(...)` | Fuse one candidate's channel evidence into a single score under the chosen strategy. |
|
|
63
|
+
| `convex_combination_score(...)` | TM2C2 over already-normalized `[0, 1]` scores. |
|
|
64
|
+
| `reciprocal_rank_fusion_score(ranks, eta=60)` | RRF over 1-based per-channel ranks. |
|
|
65
|
+
| `theoretical_min_max_normalize(score, bounds)` | Scale a score to `[0, 1]` using a scoring function's theoretical bounds. |
|
|
66
|
+
| `normalize_search_text(text)` | NFC-compose + whitespace-collapse + length-cap a query. |
|
|
67
|
+
| `WORD_SIMILARITY_THEORETICAL_BOUNDS`, `COSINE_DISTANCE_THEORETICAL_BOUNDS` | `(lower, upper)` tuples for the common lexical/dense pairing. |
|
|
68
|
+
|
|
69
|
+
## The normalization contract
|
|
70
|
+
|
|
71
|
+
Character-trigram lexical retrieval is language-agnostic only if query
|
|
72
|
+
and documents fold **identically**. `normalize_search_text` owns the
|
|
73
|
+
query side (NFC). Do accent-folding + lowercasing on the **store** side,
|
|
74
|
+
in one place, and call it from both — e.g. a PostgreSQL `IMMUTABLE`
|
|
75
|
+
wrapper `lower(unaccent(normalize(text, NFC)))` used in a `pg_trgm` GiST
|
|
76
|
+
expression index. rankweave stays out of your store so the two sides
|
|
77
|
+
cannot silently diverge.
|
|
78
|
+
|
|
79
|
+
## Research grounding
|
|
80
|
+
|
|
81
|
+
- **Bruch, Gai & Ingber (2023).** *An Analysis of Fusion Functions for
|
|
82
|
+
Hybrid Retrieval.* ACM TOIS 42(1). arXiv:2210.11934. — TM2C2 > RRF;
|
|
83
|
+
theoretical-normalization stability; the fusion desiderata
|
|
84
|
+
(monotonicity, homogeneity, boundedness, Lipschitz continuity,
|
|
85
|
+
sample efficiency) this library's defaults satisfy.
|
|
86
|
+
- **Cormack, Clarke & Büttcher (2009).** *Reciprocal Rank Fusion
|
|
87
|
+
outperforms Condorcet and individual Rank Learning Methods.* SIGIR
|
|
88
|
+
2009. — RRF definition, η = 60.
|
|
89
|
+
- **UAX #15**, Unicode Normalization Forms — NFC composition.
|
|
90
|
+
|
|
91
|
+
PDFs and a citation manifest live in [`docs/research/`](docs/research/).
|
|
92
|
+
|
|
93
|
+
## Development
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
pip install -e ".[dev]"
|
|
97
|
+
python -m pytest -q # 27 tests, no external services
|
|
98
|
+
python -m ruff check .
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## License
|
|
102
|
+
|
|
103
|
+
Apache-2.0 — see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Research grounding — rankweave
|
|
2
|
+
|
|
3
|
+
rankweave's defaults are not arbitrary; each is the published,
|
|
4
|
+
empirically-supported choice. This directory preserves the source
|
|
5
|
+
material so the grounding travels with the code.
|
|
6
|
+
|
|
7
|
+
## Papers
|
|
8
|
+
|
|
9
|
+
| File | Citation | License / redistribution |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| `pdfs/bruch-gai-ingber-2023-analysis-fusion-functions-hybrid-retrieval.pdf` | Bruch, S., Gai, S., & Ingber, A. (2023). *An Analysis of Fusion Functions for Hybrid Retrieval.* ACM Transactions on Information Systems 42(1). arXiv:2210.11934. | cite-only pending license confirmation |
|
|
12
|
+
| `pdfs/cormack-clarke-buettcher-2009-reciprocal-rank-fusion.pdf` | Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). *Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.* SIGIR 2009. | cite-only pending license confirmation |
|
|
13
|
+
|
|
14
|
+
Standards: **UAX #15 — Unicode Normalization Forms** (Unicode
|
|
15
|
+
Consortium), the basis for `normalize_search_text`'s NFC step.
|
|
16
|
+
|
|
17
|
+
## What each grounds
|
|
18
|
+
|
|
19
|
+
- **Bruch, Gai & Ingber 2023** → the **default strategy**. TM2C2 (a
|
|
20
|
+
convex combination of *theoretically* min-max normalized scores)
|
|
21
|
+
outperforms Reciprocal Rank Fusion in- and out-of-domain (their
|
|
22
|
+
Tables 2–4); the choice of normalization is immaterial for a convex
|
|
23
|
+
combination (§4.2); `alpha ∈ [0.6, 0.8]` is a robust range needing no
|
|
24
|
+
training data (we default to 0.7). Their five desiderata —
|
|
25
|
+
monotonicity, homogeneity, boundedness, Lipschitz continuity, sample
|
|
26
|
+
efficiency — are exactly the properties `convex_combination_score`
|
|
27
|
+
provides and `reciprocal_rank_fusion_score` (a function of ranks,
|
|
28
|
+
not scores) does not.
|
|
29
|
+
- **Cormack, Clarke & Büttcher 2009** → the **RRF alternative** and its
|
|
30
|
+
`eta = 60` default.
|
|
31
|
+
- **UAX #15** → NFC composition, so decomposed Vietnamese/Korean input
|
|
32
|
+
matches composed indexed text.
|
|
33
|
+
|
|
34
|
+
## PDF preservation note
|
|
35
|
+
|
|
36
|
+
Git LFS is intentionally **not** used; PDFs are committed as regular
|
|
37
|
+
binaries. Where a PDF is absent, it is because the authoring
|
|
38
|
+
environment's network policy blocked the source host (e.g. arxiv.org);
|
|
39
|
+
the citation + arXiv id above make the drop mechanical from a
|
|
40
|
+
network-allowed session. Only permissively-redistributable PDFs are
|
|
41
|
+
committed; others stay cite-only until their license is confirmed.
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["hatchling"]
|
|
3
|
+
build-backend = "hatchling.build"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "rankweave"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Language-agnostic hybrid-retrieval score fusion (TM2C2 + RRF) and Unicode query normalization — pure-Python, store-agnostic."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.10"
|
|
11
|
+
license = "Apache-2.0"
|
|
12
|
+
license-files = ["LICENSE"]
|
|
13
|
+
authors = [{ name = "Contextual Wisdom Lab" }]
|
|
14
|
+
keywords = [
|
|
15
|
+
"information-retrieval",
|
|
16
|
+
"hybrid-search",
|
|
17
|
+
"rank-fusion",
|
|
18
|
+
"reciprocal-rank-fusion",
|
|
19
|
+
"rrf",
|
|
20
|
+
"dense-retrieval",
|
|
21
|
+
"lexical-retrieval",
|
|
22
|
+
"pgvector",
|
|
23
|
+
"pg_trgm",
|
|
24
|
+
]
|
|
25
|
+
classifiers = [
|
|
26
|
+
"Development Status :: 4 - Beta",
|
|
27
|
+
"Intended Audience :: Developers",
|
|
28
|
+
"License :: OSI Approved :: Apache Software License",
|
|
29
|
+
"Programming Language :: Python :: 3",
|
|
30
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
31
|
+
"Topic :: Text Processing :: Indexing",
|
|
32
|
+
"Topic :: Scientific/Engineering :: Information Analysis",
|
|
33
|
+
"Typing :: Typed",
|
|
34
|
+
]
|
|
35
|
+
dependencies = []
|
|
36
|
+
|
|
37
|
+
[project.urls]
|
|
38
|
+
Homepage = "https://github.com/ContextualWisdomLab/rankweave"
|
|
39
|
+
Source = "https://github.com/ContextualWisdomLab/rankweave"
|
|
40
|
+
Issues = "https://github.com/ContextualWisdomLab/rankweave/issues"
|
|
41
|
+
|
|
42
|
+
[project.optional-dependencies]
|
|
43
|
+
dev = ["pytest>=8", "ruff>=0.6"]
|
|
44
|
+
|
|
45
|
+
[tool.hatch.build.targets.wheel]
|
|
46
|
+
packages = ["src/rankweave"]
|
|
47
|
+
|
|
48
|
+
[tool.pytest.ini_options]
|
|
49
|
+
testpaths = ["tests"]
|
|
50
|
+
|
|
51
|
+
[tool.ruff]
|
|
52
|
+
line-length = 88
|
|
53
|
+
target-version = "py310"
|
|
54
|
+
|
|
55
|
+
[tool.ruff.lint]
|
|
56
|
+
select = ["E", "F", "I", "UP", "B"]
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
"""rankweave — language-agnostic hybrid-retrieval score fusion.
|
|
2
|
+
|
|
3
|
+
Pure-Python (stdlib-only) fusion of lexical and semantic retrieval
|
|
4
|
+
channels, plus Unicode NFC query normalization. Store-agnostic: bring
|
|
5
|
+
your own dense (embedding) and lexical (character-trigram / BM25 /
|
|
6
|
+
learned-sparse) channels; rankweave decides how to combine their
|
|
7
|
+
scores.
|
|
8
|
+
|
|
9
|
+
Two fusion strategies, research-grounded (see ``docs/research/``):
|
|
10
|
+
|
|
11
|
+
- ``convex_combination`` (default, "TM2C2") — Bruch, Gai & Ingber
|
|
12
|
+
2023 (arXiv:2210.11934): a convex combination of theoretically
|
|
13
|
+
min-max normalized scores; robust, distribution-preserving, no
|
|
14
|
+
training data needed.
|
|
15
|
+
- ``reciprocal_rank_fusion`` — Cormack, Clarke & Büttcher 2009: the
|
|
16
|
+
non-parametric rank-only alternative.
|
|
17
|
+
|
|
18
|
+
Quickstart::
|
|
19
|
+
|
|
20
|
+
from rankweave import FusionSettings, fuse_channel_scores
|
|
21
|
+
|
|
22
|
+
settings = FusionSettings() # TM2C2, alpha=0.7
|
|
23
|
+
score = fuse_channel_scores(
|
|
24
|
+
word_similarity_score=0.62, # lexical channel, [0, 1]
|
|
25
|
+
cosine_distance=0.30, # dense channel, [0, 2]
|
|
26
|
+
channel_ranks={"lexical": 1, "dense": 1},
|
|
27
|
+
settings=settings,
|
|
28
|
+
)
|
|
29
|
+
"""
|
|
30
|
+
|
|
31
|
+
from rankweave.query_normalization import (
|
|
32
|
+
DEFAULT_MAX_QUERY_CHARACTER_LENGTH,
|
|
33
|
+
normalize_search_text,
|
|
34
|
+
)
|
|
35
|
+
from rankweave.score_fusion import (
|
|
36
|
+
CONVEX_COMBINATION_STRATEGY,
|
|
37
|
+
COSINE_DISTANCE_THEORETICAL_BOUNDS,
|
|
38
|
+
RECIPROCAL_RANK_STRATEGY,
|
|
39
|
+
WORD_SIMILARITY_THEORETICAL_BOUNDS,
|
|
40
|
+
FusionSettings,
|
|
41
|
+
convex_combination_score,
|
|
42
|
+
fuse_channel_scores,
|
|
43
|
+
reciprocal_rank_fusion_score,
|
|
44
|
+
theoretical_min_max_normalize,
|
|
45
|
+
)
|
|
46
|
+
|
|
47
|
+
__version__ = "0.1.0"
|
|
48
|
+
|
|
49
|
+
__all__ = [
|
|
50
|
+
"CONVEX_COMBINATION_STRATEGY",
|
|
51
|
+
"COSINE_DISTANCE_THEORETICAL_BOUNDS",
|
|
52
|
+
"DEFAULT_MAX_QUERY_CHARACTER_LENGTH",
|
|
53
|
+
"RECIPROCAL_RANK_STRATEGY",
|
|
54
|
+
"WORD_SIMILARITY_THEORETICAL_BOUNDS",
|
|
55
|
+
"FusionSettings",
|
|
56
|
+
"convex_combination_score",
|
|
57
|
+
"fuse_channel_scores",
|
|
58
|
+
"normalize_search_text",
|
|
59
|
+
"reciprocal_rank_fusion_score",
|
|
60
|
+
"theoretical_min_max_normalize",
|
|
61
|
+
"__version__",
|
|
62
|
+
]
|
|
File without changes
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
"""Query-side text normalization for language-agnostic search.
|
|
2
|
+
|
|
3
|
+
Character-trigram lexical retrieval is language-agnostic only if the
|
|
4
|
+
query and the indexed documents fold *identically*. This module owns
|
|
5
|
+
the query-side half of that contract; the store side must mirror it.
|
|
6
|
+
|
|
7
|
+
1. **Unicode NFC composition (UAX #15).** Vietnamese and Korean text
|
|
8
|
+
arrives in mixed composed/decomposed forms depending on the source
|
|
9
|
+
platform (macOS filenames and some webmail clients emit NFD), so
|
|
10
|
+
both sides must compose before comparison.
|
|
11
|
+
2. **Accent folding + lowercasing belong on the store side.** Do them
|
|
12
|
+
where the documents are indexed so the indexed expression and the
|
|
13
|
+
bound query parameter go through the identical path — for example a
|
|
14
|
+
PostgreSQL ``IMMUTABLE`` wrapper
|
|
15
|
+
``lower(unaccent(normalize(text, NFC)))`` used in a ``pg_trgm``
|
|
16
|
+
GiST expression index. Keep that transform in one place and call it
|
|
17
|
+
from both sides.
|
|
18
|
+
|
|
19
|
+
Only whitespace shaping and NFC are done here; anything that depends
|
|
20
|
+
on the store's runtime (accent dictionaries, collations) stays on the
|
|
21
|
+
store side so it cannot silently diverge.
|
|
22
|
+
"""
|
|
23
|
+
|
|
24
|
+
import unicodedata
|
|
25
|
+
|
|
26
|
+
DEFAULT_MAX_QUERY_CHARACTER_LENGTH = 1000
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
def normalize_search_text(
|
|
30
|
+
raw_text: str,
|
|
31
|
+
*,
|
|
32
|
+
max_characters: int = DEFAULT_MAX_QUERY_CHARACTER_LENGTH,
|
|
33
|
+
) -> str:
|
|
34
|
+
"""Compose the query to NFC and collapse insignificant whitespace.
|
|
35
|
+
|
|
36
|
+
``max_characters`` caps pathological queries; set it to match the
|
|
37
|
+
store-side bound. Returns the composed, whitespace-collapsed,
|
|
38
|
+
length-capped query text.
|
|
39
|
+
"""
|
|
40
|
+
if max_characters < 1:
|
|
41
|
+
raise ValueError("max_characters must be >= 1")
|
|
42
|
+
composed_text = unicodedata.normalize("NFC", raw_text)
|
|
43
|
+
collapsed_text = " ".join(composed_text.split())
|
|
44
|
+
return collapsed_text[:max_characters]
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
"""Fusion functions for hybrid (lexical + semantic) retrieval.
|
|
2
|
+
|
|
3
|
+
Default strategy: a convex combination of theoretically min-max
|
|
4
|
+
normalized channel scores ("TM2C2"; Bruch, Gai & Ingber 2023,
|
|
5
|
+
*An Analysis of Fusion Functions for Hybrid Retrieval*, ACM TOIS
|
|
6
|
+
42(1), arXiv:2210.11934). Their analysis shows TM2C2 outperforms
|
|
7
|
+
Reciprocal Rank Fusion in- and out-of-domain, is robust for alpha in
|
|
8
|
+
[0.6, 0.8] without training data, and — unlike RRF — preserves the
|
|
9
|
+
score distribution (Lipschitz continuity).
|
|
10
|
+
|
|
11
|
+
Reciprocal Rank Fusion (Cormack, Clarke & Büttcher 2009,
|
|
12
|
+
*Reciprocal Rank Fusion outperforms Condorcet and individual Rank
|
|
13
|
+
Learning Methods*, SIGIR) is the non-parametric alternative for
|
|
14
|
+
channels that expose only ranks (learned-sparse or external
|
|
15
|
+
channels), selected via ``FusionSettings.strategy_name``.
|
|
16
|
+
|
|
17
|
+
The convex strategy assumes each channel score has *theoretical*
|
|
18
|
+
bounds, so no per-query data-dependent normalization is needed. Two
|
|
19
|
+
bound constants ship for the common lexical+dense pairing, but they
|
|
20
|
+
are just ``(lower, upper)`` tuples — pass your own to
|
|
21
|
+
``theoretical_min_max_normalize`` for any bounded scoring function:
|
|
22
|
+
|
|
23
|
+
- ``WORD_SIMILARITY_THEORETICAL_BOUNDS`` = (0.0, 1.0) — e.g. a
|
|
24
|
+
character-trigram word-similarity such as PostgreSQL ``pg_trgm``.
|
|
25
|
+
- ``COSINE_DISTANCE_THEORETICAL_BOUNDS`` = (0.0, 2.0) — cosine
|
|
26
|
+
distance for unit-norm vectors, e.g. a pgvector ``<=>`` channel;
|
|
27
|
+
fuse_channel_scores inverts it so smaller distance scores higher.
|
|
28
|
+
"""
|
|
29
|
+
|
|
30
|
+
from dataclasses import dataclass
|
|
31
|
+
|
|
32
|
+
WORD_SIMILARITY_THEORETICAL_BOUNDS = (0.0, 1.0)
|
|
33
|
+
COSINE_DISTANCE_THEORETICAL_BOUNDS = (0.0, 2.0)
|
|
34
|
+
|
|
35
|
+
CONVEX_COMBINATION_STRATEGY = "convex_combination"
|
|
36
|
+
RECIPROCAL_RANK_STRATEGY = "reciprocal_rank_fusion"
|
|
37
|
+
|
|
38
|
+
_SUPPORTED_STRATEGY_NAMES = frozenset(
|
|
39
|
+
{CONVEX_COMBINATION_STRATEGY, RECIPROCAL_RANK_STRATEGY}
|
|
40
|
+
)
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
@dataclass(frozen=True)
|
|
44
|
+
class FusionSettings:
|
|
45
|
+
"""Tunable fusion parameters (immutable; construct one per query set)."""
|
|
46
|
+
|
|
47
|
+
strategy_name: str = CONVEX_COMBINATION_STRATEGY
|
|
48
|
+
# Weight of the semantic channel; 0.7 is the midpoint of the
|
|
49
|
+
# robust [0.6, 0.8] range reported by Bruch et al. (2023).
|
|
50
|
+
semantic_weight_alpha: float = 0.7
|
|
51
|
+
# RRF eta; 60 per Cormack et al. (2009).
|
|
52
|
+
rank_constant_eta: int = 60
|
|
53
|
+
|
|
54
|
+
def __post_init__(self) -> None:
|
|
55
|
+
if self.strategy_name not in _SUPPORTED_STRATEGY_NAMES:
|
|
56
|
+
raise ValueError(
|
|
57
|
+
"strategy_name must be one of "
|
|
58
|
+
f"{sorted(_SUPPORTED_STRATEGY_NAMES)}, got {self.strategy_name!r}"
|
|
59
|
+
)
|
|
60
|
+
if not 0.0 <= self.semantic_weight_alpha <= 1.0:
|
|
61
|
+
raise ValueError("semantic_weight_alpha must be within [0, 1]")
|
|
62
|
+
if self.rank_constant_eta < 1:
|
|
63
|
+
raise ValueError("rank_constant_eta must be >= 1")
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def theoretical_min_max_normalize(
|
|
67
|
+
score: float, bounds: tuple[float, float]
|
|
68
|
+
) -> float:
|
|
69
|
+
"""Scale a score to [0, 1] using the scoring function's theoretical bounds.
|
|
70
|
+
|
|
71
|
+
Using theoretical rather than observed bounds keeps the transform
|
|
72
|
+
stable across queries and candidate sets (Bruch et al. 2023, §4.2).
|
|
73
|
+
Out-of-range inputs (floating-point drift) are clamped.
|
|
74
|
+
"""
|
|
75
|
+
lower_bound, upper_bound = bounds
|
|
76
|
+
if upper_bound <= lower_bound:
|
|
77
|
+
raise ValueError("bounds must satisfy upper > lower")
|
|
78
|
+
normalized = (score - lower_bound) / (upper_bound - lower_bound)
|
|
79
|
+
return min(1.0, max(0.0, normalized))
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def convex_combination_score(
|
|
83
|
+
semantic_score: float | None,
|
|
84
|
+
lexical_score: float | None,
|
|
85
|
+
semantic_weight_alpha: float,
|
|
86
|
+
) -> float:
|
|
87
|
+
"""TM2C2 fusion over already-normalized [0, 1] channel scores.
|
|
88
|
+
|
|
89
|
+
A channel absent for a candidate (e.g. no embedding stored yet)
|
|
90
|
+
contributes its theoretical minimum, 0 — absent evidence is the
|
|
91
|
+
infimum, not a missing value to impute.
|
|
92
|
+
"""
|
|
93
|
+
semantic_component = semantic_score if semantic_score is not None else 0.0
|
|
94
|
+
lexical_component = lexical_score if lexical_score is not None else 0.0
|
|
95
|
+
return (
|
|
96
|
+
semantic_weight_alpha * semantic_component
|
|
97
|
+
+ (1.0 - semantic_weight_alpha) * lexical_component
|
|
98
|
+
)
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
def reciprocal_rank_fusion_score(
|
|
102
|
+
channel_ranks: dict[str, int], rank_constant_eta: int = 60
|
|
103
|
+
) -> float:
|
|
104
|
+
"""RRF over 1-based per-channel ranks: sum of 1 / (eta + rank)."""
|
|
105
|
+
if rank_constant_eta < 1:
|
|
106
|
+
raise ValueError("rank_constant_eta must be >= 1")
|
|
107
|
+
fused_score = 0.0
|
|
108
|
+
for channel_name, one_based_rank in channel_ranks.items():
|
|
109
|
+
if one_based_rank < 1:
|
|
110
|
+
raise ValueError(
|
|
111
|
+
f"rank for channel {channel_name!r} must be >= 1,"
|
|
112
|
+
f" got {one_based_rank}"
|
|
113
|
+
)
|
|
114
|
+
fused_score += 1.0 / (rank_constant_eta + one_based_rank)
|
|
115
|
+
return fused_score
|
|
116
|
+
|
|
117
|
+
|
|
118
|
+
def fuse_channel_scores(
|
|
119
|
+
*,
|
|
120
|
+
word_similarity_score: float | None,
|
|
121
|
+
cosine_distance: float | None,
|
|
122
|
+
channel_ranks: dict[str, int],
|
|
123
|
+
settings: FusionSettings,
|
|
124
|
+
) -> float:
|
|
125
|
+
"""Fuse one candidate's channel evidence into a single score.
|
|
126
|
+
|
|
127
|
+
``word_similarity_score`` and ``cosine_distance`` are the raw
|
|
128
|
+
channel outputs (None when the channel did not produce this
|
|
129
|
+
candidate); ``channel_ranks`` are the candidate's 1-based ranks in
|
|
130
|
+
the channels that returned it, used by the RRF strategy.
|
|
131
|
+
"""
|
|
132
|
+
if settings.strategy_name == RECIPROCAL_RANK_STRATEGY:
|
|
133
|
+
if not channel_ranks:
|
|
134
|
+
return 0.0
|
|
135
|
+
return reciprocal_rank_fusion_score(
|
|
136
|
+
channel_ranks, settings.rank_constant_eta
|
|
137
|
+
)
|
|
138
|
+
|
|
139
|
+
normalized_lexical_score = (
|
|
140
|
+
theoretical_min_max_normalize(
|
|
141
|
+
word_similarity_score, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
142
|
+
)
|
|
143
|
+
if word_similarity_score is not None
|
|
144
|
+
else None
|
|
145
|
+
)
|
|
146
|
+
# Cosine *distance* decreases as relevance increases; invert inside
|
|
147
|
+
# the theoretical [0, 2] range so 1.0 means identical direction.
|
|
148
|
+
normalized_semantic_score = (
|
|
149
|
+
1.0
|
|
150
|
+
- theoretical_min_max_normalize(
|
|
151
|
+
cosine_distance, COSINE_DISTANCE_THEORETICAL_BOUNDS
|
|
152
|
+
)
|
|
153
|
+
if cosine_distance is not None
|
|
154
|
+
else None
|
|
155
|
+
)
|
|
156
|
+
return convex_combination_score(
|
|
157
|
+
normalized_semantic_score,
|
|
158
|
+
normalized_lexical_score,
|
|
159
|
+
settings.semantic_weight_alpha,
|
|
160
|
+
)
|
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
import math
|
|
2
|
+
|
|
3
|
+
import pytest
|
|
4
|
+
|
|
5
|
+
from rankweave import (
|
|
6
|
+
COSINE_DISTANCE_THEORETICAL_BOUNDS,
|
|
7
|
+
WORD_SIMILARITY_THEORETICAL_BOUNDS,
|
|
8
|
+
FusionSettings,
|
|
9
|
+
convex_combination_score,
|
|
10
|
+
fuse_channel_scores,
|
|
11
|
+
normalize_search_text,
|
|
12
|
+
reciprocal_rank_fusion_score,
|
|
13
|
+
theoretical_min_max_normalize,
|
|
14
|
+
)
|
|
15
|
+
from rankweave.score_fusion import (
|
|
16
|
+
CONVEX_COMBINATION_STRATEGY,
|
|
17
|
+
RECIPROCAL_RANK_STRATEGY,
|
|
18
|
+
)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
class TestNormalizeSearchText:
|
|
22
|
+
def test_composes_decomposed_hangul_to_nfc(self):
|
|
23
|
+
# U+1112 U+1161 U+11AB (decomposed jamo) -> U+D55C
|
|
24
|
+
decomposed_hangul = "\u1112\u1161\u11ab"
|
|
25
|
+
assert normalize_search_text(decomposed_hangul) == "\ud55c"
|
|
26
|
+
|
|
27
|
+
def test_composes_decomposed_vietnamese_diacritics(self):
|
|
28
|
+
# "a" + combining circumflex + combining grave -> U+1EA7
|
|
29
|
+
decomposed_vietnamese = "Tra\u0302\u0300n"
|
|
30
|
+
assert normalize_search_text(decomposed_vietnamese) == "Tr\u1ea7n"
|
|
31
|
+
|
|
32
|
+
def test_collapses_whitespace_and_strips(self):
|
|
33
|
+
assert normalize_search_text(" hello \t world \n") == "hello world"
|
|
34
|
+
|
|
35
|
+
def test_caps_pathological_query_length(self):
|
|
36
|
+
assert len(normalize_search_text("가" * 5000)) == 1000
|
|
37
|
+
|
|
38
|
+
def test_keeps_cjk_text_intact_without_tokenization(self):
|
|
39
|
+
korean_query = "다음주 회의 일정"
|
|
40
|
+
assert normalize_search_text(korean_query) == korean_query
|
|
41
|
+
|
|
42
|
+
def test_max_characters_override(self):
|
|
43
|
+
assert len(normalize_search_text("a" * 50, max_characters=10)) == 10
|
|
44
|
+
|
|
45
|
+
def test_rejects_non_positive_max_characters(self):
|
|
46
|
+
with pytest.raises(ValueError):
|
|
47
|
+
normalize_search_text("hello", max_characters=0)
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
class TestTheoreticalMinMaxNormalize:
|
|
51
|
+
def test_word_similarity_bounds_map_to_unit_interval(self):
|
|
52
|
+
assert theoretical_min_max_normalize(
|
|
53
|
+
0.0, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
54
|
+
) == 0.0
|
|
55
|
+
assert theoretical_min_max_normalize(
|
|
56
|
+
1.0, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
57
|
+
) == 1.0
|
|
58
|
+
assert theoretical_min_max_normalize(
|
|
59
|
+
0.25, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
60
|
+
) == pytest.approx(0.25)
|
|
61
|
+
|
|
62
|
+
def test_cosine_distance_bounds_map_to_unit_interval(self):
|
|
63
|
+
assert theoretical_min_max_normalize(
|
|
64
|
+
0.0, COSINE_DISTANCE_THEORETICAL_BOUNDS
|
|
65
|
+
) == 0.0
|
|
66
|
+
assert theoretical_min_max_normalize(
|
|
67
|
+
2.0, COSINE_DISTANCE_THEORETICAL_BOUNDS
|
|
68
|
+
) == 1.0
|
|
69
|
+
assert theoretical_min_max_normalize(
|
|
70
|
+
0.5, COSINE_DISTANCE_THEORETICAL_BOUNDS
|
|
71
|
+
) == pytest.approx(0.25)
|
|
72
|
+
|
|
73
|
+
def test_clamps_floating_point_drift(self):
|
|
74
|
+
assert theoretical_min_max_normalize(
|
|
75
|
+
1.0000001, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
76
|
+
) == 1.0
|
|
77
|
+
assert theoretical_min_max_normalize(
|
|
78
|
+
-0.0000001, WORD_SIMILARITY_THEORETICAL_BOUNDS
|
|
79
|
+
) == 0.0
|
|
80
|
+
|
|
81
|
+
def test_rejects_inverted_bounds(self):
|
|
82
|
+
with pytest.raises(ValueError):
|
|
83
|
+
theoretical_min_max_normalize(0.5, (1.0, 0.0))
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
class TestConvexCombinationScore:
|
|
87
|
+
def test_hand_computed_fusion(self):
|
|
88
|
+
# alpha * semantic + (1 - alpha) * lexical
|
|
89
|
+
fused_score = convex_combination_score(0.8, 0.5, 0.7)
|
|
90
|
+
assert fused_score == pytest.approx(0.7 * 0.8 + 0.3 * 0.5)
|
|
91
|
+
|
|
92
|
+
def test_missing_channel_contributes_theoretical_minimum(self):
|
|
93
|
+
assert convex_combination_score(None, 0.5, 0.7) == pytest.approx(0.15)
|
|
94
|
+
assert convex_combination_score(0.8, None, 0.7) == pytest.approx(0.56)
|
|
95
|
+
assert convex_combination_score(None, None, 0.7) == 0.0
|
|
96
|
+
|
|
97
|
+
def test_alpha_extremes_select_single_channel(self):
|
|
98
|
+
assert convex_combination_score(0.9, 0.4, 1.0) == pytest.approx(0.9)
|
|
99
|
+
assert convex_combination_score(0.9, 0.4, 0.0) == pytest.approx(0.4)
|
|
100
|
+
|
|
101
|
+
def test_monotone_in_each_channel(self):
|
|
102
|
+
base_score = convex_combination_score(0.5, 0.5, 0.7)
|
|
103
|
+
assert convex_combination_score(0.6, 0.5, 0.7) > base_score
|
|
104
|
+
assert convex_combination_score(0.5, 0.6, 0.7) > base_score
|
|
105
|
+
|
|
106
|
+
def test_bounded_in_unit_interval(self):
|
|
107
|
+
assert 0.0 <= convex_combination_score(1.0, 1.0, 0.7) <= 1.0
|
|
108
|
+
assert 0.0 <= convex_combination_score(0.0, 0.0, 0.7) <= 1.0
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
class TestReciprocalRankFusionScore:
|
|
112
|
+
def test_hand_computed_rrf(self):
|
|
113
|
+
fused_score = reciprocal_rank_fusion_score(
|
|
114
|
+
{"lexical_email": 1, "dense_email": 3}, rank_constant_eta=60
|
|
115
|
+
)
|
|
116
|
+
assert fused_score == pytest.approx(1.0 / 61.0 + 1.0 / 63.0)
|
|
117
|
+
|
|
118
|
+
def test_lower_rank_scores_higher(self):
|
|
119
|
+
better_ranked = reciprocal_rank_fusion_score({"channel": 1})
|
|
120
|
+
worse_ranked = reciprocal_rank_fusion_score({"channel": 10})
|
|
121
|
+
assert better_ranked > worse_ranked
|
|
122
|
+
|
|
123
|
+
def test_more_channels_score_higher(self):
|
|
124
|
+
single_channel = reciprocal_rank_fusion_score({"a": 5})
|
|
125
|
+
two_channels = reciprocal_rank_fusion_score({"a": 5, "b": 5})
|
|
126
|
+
assert two_channels > single_channel
|
|
127
|
+
|
|
128
|
+
def test_rejects_invalid_rank_and_eta(self):
|
|
129
|
+
with pytest.raises(ValueError):
|
|
130
|
+
reciprocal_rank_fusion_score({"a": 0})
|
|
131
|
+
with pytest.raises(ValueError):
|
|
132
|
+
reciprocal_rank_fusion_score({"a": 1}, rank_constant_eta=0)
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
class TestFusionSettings:
|
|
136
|
+
def test_defaults_follow_research_grounding(self):
|
|
137
|
+
settings = FusionSettings()
|
|
138
|
+
assert settings.strategy_name == CONVEX_COMBINATION_STRATEGY
|
|
139
|
+
assert settings.semantic_weight_alpha == 0.7
|
|
140
|
+
assert settings.rank_constant_eta == 60
|
|
141
|
+
|
|
142
|
+
def test_rejects_unknown_strategy(self):
|
|
143
|
+
with pytest.raises(ValueError):
|
|
144
|
+
FusionSettings(strategy_name="borda_count")
|
|
145
|
+
|
|
146
|
+
def test_rejects_out_of_range_alpha(self):
|
|
147
|
+
with pytest.raises(ValueError):
|
|
148
|
+
FusionSettings(semantic_weight_alpha=1.5)
|
|
149
|
+
|
|
150
|
+
|
|
151
|
+
class TestFuseChannelScores:
|
|
152
|
+
def test_convex_strategy_normalizes_and_inverts_distance(self):
|
|
153
|
+
settings = FusionSettings()
|
|
154
|
+
fused_score = fuse_channel_scores(
|
|
155
|
+
word_similarity_score=0.5,
|
|
156
|
+
cosine_distance=0.4,
|
|
157
|
+
channel_ranks={"lexical_email": 1, "dense_email": 1},
|
|
158
|
+
settings=settings,
|
|
159
|
+
)
|
|
160
|
+
expected_semantic = 1.0 - 0.4 / 2.0 # 0.8
|
|
161
|
+
assert fused_score == pytest.approx(0.7 * expected_semantic + 0.3 * 0.5)
|
|
162
|
+
|
|
163
|
+
def test_identical_vectors_and_exact_word_match_score_one(self):
|
|
164
|
+
settings = FusionSettings()
|
|
165
|
+
fused_score = fuse_channel_scores(
|
|
166
|
+
word_similarity_score=1.0,
|
|
167
|
+
cosine_distance=0.0,
|
|
168
|
+
channel_ranks={"lexical_email": 1, "dense_email": 1},
|
|
169
|
+
settings=settings,
|
|
170
|
+
)
|
|
171
|
+
assert fused_score == pytest.approx(1.0)
|
|
172
|
+
|
|
173
|
+
def test_rrf_strategy_uses_ranks_only(self):
|
|
174
|
+
settings = FusionSettings(strategy_name=RECIPROCAL_RANK_STRATEGY)
|
|
175
|
+
fused_score = fuse_channel_scores(
|
|
176
|
+
word_similarity_score=0.99,
|
|
177
|
+
cosine_distance=0.01,
|
|
178
|
+
channel_ranks={"lexical_email": 2, "dense_email": 4},
|
|
179
|
+
settings=settings,
|
|
180
|
+
)
|
|
181
|
+
assert fused_score == pytest.approx(1.0 / 62.0 + 1.0 / 64.0)
|
|
182
|
+
|
|
183
|
+
def test_rrf_strategy_with_no_ranks_scores_zero(self):
|
|
184
|
+
settings = FusionSettings(strategy_name=RECIPROCAL_RANK_STRATEGY)
|
|
185
|
+
assert (
|
|
186
|
+
fuse_channel_scores(
|
|
187
|
+
word_similarity_score=None,
|
|
188
|
+
cosine_distance=None,
|
|
189
|
+
channel_ranks={},
|
|
190
|
+
settings=settings,
|
|
191
|
+
)
|
|
192
|
+
== 0.0
|
|
193
|
+
)
|
|
194
|
+
|
|
195
|
+
def test_lexical_only_candidate_is_finite_and_positive(self):
|
|
196
|
+
settings = FusionSettings()
|
|
197
|
+
fused_score = fuse_channel_scores(
|
|
198
|
+
word_similarity_score=0.6,
|
|
199
|
+
cosine_distance=None,
|
|
200
|
+
channel_ranks={"lexical_email": 1},
|
|
201
|
+
settings=settings,
|
|
202
|
+
)
|
|
203
|
+
assert math.isfinite(fused_score)
|
|
204
|
+
assert fused_score == pytest.approx(0.3 * 0.6)
|