sparpartner 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,4 @@
1
+ Copyright (c) 2026 Your Henry. All Rights Reserved.
2
+
3
+ Contact the author at osas2henry@gmail.com with questions or requests
4
+ regarding use of this software.
@@ -0,0 +1,227 @@
1
+ Metadata-Version: 2.4
2
+ Name: sparpartner
3
+ Version: 0.1.0
4
+ Summary: Deterministic, benchmark-driven stratified sampler for train/test prep.
5
+ Author: Henry
6
+ Author-email: Henry <osas2henry@gmail.com>
7
+ License: All Rights Reserved
8
+ Classifier: License :: Other/Proprietary License
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.8
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: pandas>=1.3
15
+ Dynamic: author
16
+ Dynamic: license-file
17
+ Dynamic: requires-python
18
+
19
+ # sparpartner
20
+
21
+ A deterministic stratified sampler for train/test prep. It doesn't
22
+ split your data randomly, it deliberately finds the rows that most
23
+ resemble a benchmark case, so you can hold those out as a genuine
24
+ test set and train on everything else.
25
+
26
+ ## Why this exists
27
+
28
+ A random train/test split assumes the test set should look
29
+ statistically like the train set. That's the wrong question if
30
+ what you actually want to know is: **does the model generalize past
31
+ one specific profile, or did it just memorize the neighborhood
32
+ around it?**
33
+
34
+ `sparpartner` answers that by ranking every row in your data by how
35
+ closely it resembles a benchmark ("bench_marks") you define, then
36
+ sorting on that resemblance. You then slice the sorted frame
37
+ yourself:
38
+
39
+ - **Lookalikes as test, the rest as train**: train on rows *unlike*
40
+ the benchmark, test on the rows that *are* like it. This is the
41
+ harder, more honest check: it tells you whether the model actually
42
+ learned something general, or only performs well near cases it's
43
+ already seen a lot of.
44
+ - **Lookalikes as train, the rest as test**: the reverse, if you
45
+ want to check the opposite direction.
46
+
47
+ `sparpartner` only produces the ranking. The actual train/test cut
48
+ is a plain slice on your side (see [Usage](#usage) below).
49
+
50
+ ## Where the idea comes from
51
+
52
+ Two unrelated places, both about comparing something against a
53
+ reference on purpose rather than at random:
54
+
55
+ **A boxer preparing for a match.** A fighter in camp doesn't spar
56
+ with whoever's free in the gym, they specifically look for a
57
+ sparring partner who moves, reaches, and hits like the opponent
58
+ they're about to face. Training against a random partner tells you
59
+ nothing about how you'll actually do; training against someone who
60
+ resembles the real threat does. `sparpartner` applies that same
61
+ logic to a model: instead of a random holdout, it finds the rows
62
+ that resemble the toughest, most relevant "opponent" profile and
63
+ holds those back as the real test, so what's left to train on is
64
+ everything *unlike* that opponent, and the test genuinely checks
65
+ whether the model can handle the match it's actually walking into.
66
+
67
+ **Astrology's approach to comparing a chart to a reference.** Reading
68
+ a chart against a benchmark isn't a single yes/no match, it's
69
+ several weighted dimensions (sun sign, moon sign, rising, houses,
70
+ and so on) each compared individually, then combined into one
71
+ overall resemblance reading. No single dimension decides the
72
+ outcome alone; the composite is what matters. That's the same shape
73
+ `sparpartner` uses: several weighted signals, each scored
74
+ independently against a benchmark, summed into one composite
75
+ `_score` rather than a single all-or-nothing match.
76
+
77
+ ## How the scoring works
78
+
79
+ You give it:
80
+
81
+ - `df`: your data
82
+ - `bench_marks`: a dict of `{column_name: benchmark_value}`, one
83
+ entry per signal you care about
84
+ - `custom_weights`: a list of `[column_name, weight]` pairs, saying
85
+ which columns matter and how much
86
+
87
+ For each weighted column, `sparpartner` auto-detects the column's
88
+ type and scores every row's distance to the benchmark on a 0–1
89
+ scale (1.0 = exact match, 0.0 = as far as possible):
90
+
91
+ | Detected type | How distance is measured |
92
+ |---|---|
93
+ | **numeric** | `abs(value - bench)`, capped by the column's own max observed distance from bench |
94
+ | **date** | both sides converted to "age in days" relative to the benchmark date, capped by the column's own max observed age |
95
+ | **string** | exact match = 1, anything else = 0 |
96
+
97
+ Date detection is automatic, it only treats a column as a date if
98
+ its values look date-shaped (contain a separator like `-`, `/`, `.`
99
+ or a recognizable month name) **and** parse successfully at least
100
+ 98% of the time. Bare numeric-looking strings (e.g. `"12345"`) are
101
+ never mistaken for dates, and object-dtype columns of digit strings
102
+ are treated as exact-match strings, not numbers. Only a real
103
+ numeric dtype gets the numeric path.
104
+
105
+ Each column's 0–1 score is multiplied by its weight and summed into
106
+ one raw score per row, then divided by the total weight so the
107
+ final `_score` always lands in the 0–1 range, however many signals
108
+ or weights you used.
109
+
110
+ ### Tie-breaking
111
+
112
+ Rows that land on the exact same `_score` aren't left to random or
113
+ arbitrary order. Ties are broken by the per-signal score of the
114
+ **highest-weight** signal first, then the next-highest, cascading
115
+ down the weighted signal list until the tie resolves. Signals that
116
+ share the same weight are compared in the order they appear in
117
+ `custom_weights`. The tie-break always sorts in the same direction
118
+ as the main score (see `look_alike` below). Only if every signal is
119
+ exhausted and rows are still tied does it fall back to original row
120
+ order.
121
+
122
+ ## `look_alike`
123
+
124
+ Controls sort direction:
125
+
126
+ - `look_alike=True` (default): highest `_score` (closest to
127
+ benchmark) sorted to the **top**.
128
+ - `look_alike=False`: highest `_score` sorted to the **bottom**,
129
+ farthest-from-benchmark rows come first.
130
+
131
+ ## Usage
132
+
133
+ ### Sample usage
134
+
135
+ ```python
136
+ import pandas as pd
137
+ from sparpartner import sample
138
+
139
+ df = pd.DataFrame({
140
+ "id": [1, 2, 3, 4, 5],
141
+ "age": [25, 30, 47, 52, 33],
142
+ "signup_date": ["2023-01-15", "2023-03-02", "2022-11-20", "2023-01-10", "2023-06-01"],
143
+ "country": ["US", "US", "CA", "US", "MX"],
144
+ })
145
+
146
+ bench_marks = {
147
+ "age": 30,
148
+ "signup_date": "2023-01-01",
149
+ "country": "US",
150
+ }
151
+
152
+ custom_weights = [
153
+ ["age", 2],
154
+ ["signup_date", 1],
155
+ ["country", 1],
156
+ ]
157
+
158
+ result = sample(
159
+ df,
160
+ bench_marks,
161
+ custom_weights,
162
+ look_alike=True, # True = most similar rows first
163
+ show_progress=True, # prints the full scoring breakdown
164
+ return_score=True, # keep _score columns in the output
165
+ drop_nan=True,
166
+ )
167
+
168
+ print(result)
169
+ ```
170
+
171
+ `age=30`, `signup_date="2023-01-01"`, and `country="US"` closely
172
+ match row `id=1` (age 25, close date, US), so with
173
+ `look_alike=True` that row lands at or near the top of the sorted
174
+ output. Flip `look_alike=False` to instead surface the rows that
175
+ look *least* like the benchmark (e.g. the Mexico row with the
176
+ furthest signup date).
177
+
178
+ ```python
179
+ # --- post-sample: turn the ranking into an actual train/test split ---
180
+ # Re-run with look_alike=False so lookalikes sort to the BOTTOM of `result`.
181
+ ranked = sample(df, bench_marks, custom_weights, look_alike=False)
182
+
183
+ # Slice however large you want the test set to be, e.g. the bottom 30%:
184
+ cut = int(len(ranked) * 0.7)
185
+ train = ranked.iloc[:cut] # non-lookalikes
186
+ test = ranked.iloc[cut:] # lookalikes, the harder, honest test set
187
+ ```
188
+
189
+ ### Parameters
190
+
191
+ | Name | Type | Default | What it does |
192
+ |---|---|---|---|
193
+ | `df` | DataFrame | required | Must contain a column for every name in `custom_weights` |
194
+ | `bench_marks` | dict | required | `{column_name: benchmark_value}` |
195
+ | `custom_weights` | list of `[name, weight]` | required | Which columns to score and how much each contributes |
196
+ | `look_alike` | bool | `True` | `True` = lookalikes sorted to top; `False` = sorted to bottom |
197
+ | `show_progress` | bool | `False` | Prints a full readout: header, per-column type/cap/sample scores, sort direction, top/bottom ranked rows with weighted contributions, and a health check flagging any signal that isn't adding useful separation |
198
+ | `return_score` | bool | `False` | If `True`, keeps `_score` and `_score_<name>` columns in the result instead of dropping them |
199
+ | `drop_nan` | bool | `True` | If `True`, drops any row with a NaN component score (e.g. missing value or unparseable date) after printing a sanity check of what was dropped and why |
200
+
201
+ ### Validation
202
+
203
+ `custom_weights` is validated upfront, before any scoring starts, and
204
+ raises `ValueError` if:
205
+
206
+ - a column name isn't a string, or doesn't exist in `df`
207
+ - a column name has no matching entry in `bench_marks`
208
+ - the same column name appears more than once
209
+ - a weight isn't numeric (bools are rejected too, a `bool` is
210
+ technically an `int` in Python but was never meant as a weight)
211
+ - a weight is `NaN`
212
+
213
+ ## A couple of things worth knowing
214
+
215
+ - **Zero-variance columns**: if every row (including the benchmark)
216
+ has the same value for a signal, there's nothing to measure
217
+ distance against, that signal scores every row 1.0 rather than
218
+ dividing by zero.
219
+ - **Negative total weight**: if your weights sum to ≤ 0, the
220
+ normalization step is skipped and `_score` is left as the raw
221
+ weighted sum instead of being silently divided by a non-positive
222
+ number.
223
+ - **Object-dtype numeric strings**: a column of strings like
224
+ `"100"`, `"200"` (object dtype, no separator) is scored as an
225
+ exact-match string column, *not* auto-converted to numeric. Only
226
+ genuine numeric dtypes (`int`, `float`) get the numeric distance
227
+ path.
@@ -0,0 +1,209 @@
1
+ # sparpartner
2
+
3
+ A deterministic stratified sampler for train/test prep. It doesn't
4
+ split your data randomly, it deliberately finds the rows that most
5
+ resemble a benchmark case, so you can hold those out as a genuine
6
+ test set and train on everything else.
7
+
8
+ ## Why this exists
9
+
10
+ A random train/test split assumes the test set should look
11
+ statistically like the train set. That's the wrong question if
12
+ what you actually want to know is: **does the model generalize past
13
+ one specific profile, or did it just memorize the neighborhood
14
+ around it?**
15
+
16
+ `sparpartner` answers that by ranking every row in your data by how
17
+ closely it resembles a benchmark ("bench_marks") you define, then
18
+ sorting on that resemblance. You then slice the sorted frame
19
+ yourself:
20
+
21
+ - **Lookalikes as test, the rest as train**: train on rows *unlike*
22
+ the benchmark, test on the rows that *are* like it. This is the
23
+ harder, more honest check: it tells you whether the model actually
24
+ learned something general, or only performs well near cases it's
25
+ already seen a lot of.
26
+ - **Lookalikes as train, the rest as test**: the reverse, if you
27
+ want to check the opposite direction.
28
+
29
+ `sparpartner` only produces the ranking. The actual train/test cut
30
+ is a plain slice on your side (see [Usage](#usage) below).
31
+
32
+ ## Where the idea comes from
33
+
34
+ Two unrelated places, both about comparing something against a
35
+ reference on purpose rather than at random:
36
+
37
+ **A boxer preparing for a match.** A fighter in camp doesn't spar
38
+ with whoever's free in the gym, they specifically look for a
39
+ sparring partner who moves, reaches, and hits like the opponent
40
+ they're about to face. Training against a random partner tells you
41
+ nothing about how you'll actually do; training against someone who
42
+ resembles the real threat does. `sparpartner` applies that same
43
+ logic to a model: instead of a random holdout, it finds the rows
44
+ that resemble the toughest, most relevant "opponent" profile and
45
+ holds those back as the real test, so what's left to train on is
46
+ everything *unlike* that opponent, and the test genuinely checks
47
+ whether the model can handle the match it's actually walking into.
48
+
49
+ **Astrology's approach to comparing a chart to a reference.** Reading
50
+ a chart against a benchmark isn't a single yes/no match, it's
51
+ several weighted dimensions (sun sign, moon sign, rising, houses,
52
+ and so on) each compared individually, then combined into one
53
+ overall resemblance reading. No single dimension decides the
54
+ outcome alone; the composite is what matters. That's the same shape
55
+ `sparpartner` uses: several weighted signals, each scored
56
+ independently against a benchmark, summed into one composite
57
+ `_score` rather than a single all-or-nothing match.
58
+
59
+ ## How the scoring works
60
+
61
+ You give it:
62
+
63
+ - `df`: your data
64
+ - `bench_marks`: a dict of `{column_name: benchmark_value}`, one
65
+ entry per signal you care about
66
+ - `custom_weights`: a list of `[column_name, weight]` pairs, saying
67
+ which columns matter and how much
68
+
69
+ For each weighted column, `sparpartner` auto-detects the column's
70
+ type and scores every row's distance to the benchmark on a 0–1
71
+ scale (1.0 = exact match, 0.0 = as far as possible):
72
+
73
+ | Detected type | How distance is measured |
74
+ |---|---|
75
+ | **numeric** | `abs(value - bench)`, capped by the column's own max observed distance from bench |
76
+ | **date** | both sides converted to "age in days" relative to the benchmark date, capped by the column's own max observed age |
77
+ | **string** | exact match = 1, anything else = 0 |
78
+
79
+ Date detection is automatic, it only treats a column as a date if
80
+ its values look date-shaped (contain a separator like `-`, `/`, `.`
81
+ or a recognizable month name) **and** parse successfully at least
82
+ 98% of the time. Bare numeric-looking strings (e.g. `"12345"`) are
83
+ never mistaken for dates, and object-dtype columns of digit strings
84
+ are treated as exact-match strings, not numbers. Only a real
85
+ numeric dtype gets the numeric path.
86
+
87
+ Each column's 0–1 score is multiplied by its weight and summed into
88
+ one raw score per row, then divided by the total weight so the
89
+ final `_score` always lands in the 0–1 range, however many signals
90
+ or weights you used.
91
+
92
+ ### Tie-breaking
93
+
94
+ Rows that land on the exact same `_score` aren't left to random or
95
+ arbitrary order. Ties are broken by the per-signal score of the
96
+ **highest-weight** signal first, then the next-highest, cascading
97
+ down the weighted signal list until the tie resolves. Signals that
98
+ share the same weight are compared in the order they appear in
99
+ `custom_weights`. The tie-break always sorts in the same direction
100
+ as the main score (see `look_alike` below). Only if every signal is
101
+ exhausted and rows are still tied does it fall back to original row
102
+ order.
103
+
104
+ ## `look_alike`
105
+
106
+ Controls sort direction:
107
+
108
+ - `look_alike=True` (default): highest `_score` (closest to
109
+ benchmark) sorted to the **top**.
110
+ - `look_alike=False`: highest `_score` sorted to the **bottom**,
111
+ farthest-from-benchmark rows come first.
112
+
113
+ ## Usage
114
+
115
+ ### Sample usage
116
+
117
+ ```python
118
+ import pandas as pd
119
+ from sparpartner import sample
120
+
121
+ df = pd.DataFrame({
122
+ "id": [1, 2, 3, 4, 5],
123
+ "age": [25, 30, 47, 52, 33],
124
+ "signup_date": ["2023-01-15", "2023-03-02", "2022-11-20", "2023-01-10", "2023-06-01"],
125
+ "country": ["US", "US", "CA", "US", "MX"],
126
+ })
127
+
128
+ bench_marks = {
129
+ "age": 30,
130
+ "signup_date": "2023-01-01",
131
+ "country": "US",
132
+ }
133
+
134
+ custom_weights = [
135
+ ["age", 2],
136
+ ["signup_date", 1],
137
+ ["country", 1],
138
+ ]
139
+
140
+ result = sample(
141
+ df,
142
+ bench_marks,
143
+ custom_weights,
144
+ look_alike=True, # True = most similar rows first
145
+ show_progress=True, # prints the full scoring breakdown
146
+ return_score=True, # keep _score columns in the output
147
+ drop_nan=True,
148
+ )
149
+
150
+ print(result)
151
+ ```
152
+
153
+ `age=30`, `signup_date="2023-01-01"`, and `country="US"` closely
154
+ match row `id=1` (age 25, close date, US), so with
155
+ `look_alike=True` that row lands at or near the top of the sorted
156
+ output. Flip `look_alike=False` to instead surface the rows that
157
+ look *least* like the benchmark (e.g. the Mexico row with the
158
+ furthest signup date).
159
+
160
+ ```python
161
+ # --- post-sample: turn the ranking into an actual train/test split ---
162
+ # Re-run with look_alike=False so lookalikes sort to the BOTTOM of `result`.
163
+ ranked = sample(df, bench_marks, custom_weights, look_alike=False)
164
+
165
+ # Slice however large you want the test set to be, e.g. the bottom 30%:
166
+ cut = int(len(ranked) * 0.7)
167
+ train = ranked.iloc[:cut] # non-lookalikes
168
+ test = ranked.iloc[cut:] # lookalikes, the harder, honest test set
169
+ ```
170
+
171
+ ### Parameters
172
+
173
+ | Name | Type | Default | What it does |
174
+ |---|---|---|---|
175
+ | `df` | DataFrame | required | Must contain a column for every name in `custom_weights` |
176
+ | `bench_marks` | dict | required | `{column_name: benchmark_value}` |
177
+ | `custom_weights` | list of `[name, weight]` | required | Which columns to score and how much each contributes |
178
+ | `look_alike` | bool | `True` | `True` = lookalikes sorted to top; `False` = sorted to bottom |
179
+ | `show_progress` | bool | `False` | Prints a full readout: header, per-column type/cap/sample scores, sort direction, top/bottom ranked rows with weighted contributions, and a health check flagging any signal that isn't adding useful separation |
180
+ | `return_score` | bool | `False` | If `True`, keeps `_score` and `_score_<name>` columns in the result instead of dropping them |
181
+ | `drop_nan` | bool | `True` | If `True`, drops any row with a NaN component score (e.g. missing value or unparseable date) after printing a sanity check of what was dropped and why |
182
+
183
+ ### Validation
184
+
185
+ `custom_weights` is validated upfront, before any scoring starts, and
186
+ raises `ValueError` if:
187
+
188
+ - a column name isn't a string, or doesn't exist in `df`
189
+ - a column name has no matching entry in `bench_marks`
190
+ - the same column name appears more than once
191
+ - a weight isn't numeric (bools are rejected too, a `bool` is
192
+ technically an `int` in Python but was never meant as a weight)
193
+ - a weight is `NaN`
194
+
195
+ ## A couple of things worth knowing
196
+
197
+ - **Zero-variance columns**: if every row (including the benchmark)
198
+ has the same value for a signal, there's nothing to measure
199
+ distance against, that signal scores every row 1.0 rather than
200
+ dividing by zero.
201
+ - **Negative total weight**: if your weights sum to ≤ 0, the
202
+ normalization step is skipped and `_score` is left as the raw
203
+ weighted sum instead of being silently divided by a non-positive
204
+ number.
205
+ - **Object-dtype numeric strings**: a column of strings like
206
+ `"100"`, `"200"` (object dtype, no separator) is scored as an
207
+ exact-match string column, *not* auto-converted to numeric. Only
208
+ genuine numeric dtypes (`int`, `float`) get the numeric distance
209
+ path.
@@ -0,0 +1,25 @@
1
+ [build-system]
2
+ requires = ["setuptools>=61.0"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "sparpartner"
7
+ version = "0.1.0"
8
+ description = "Deterministic, benchmark-driven stratified sampler for train/test prep."
9
+ readme = "README.md"
10
+ requires-python = ">=3.8"
11
+ license = { text = "All Rights Reserved" }
12
+ authors = [
13
+ { name = "Henry", email = "osas2henry@gmail.com" }
14
+ ]
15
+ dependencies = [
16
+ "pandas>=1.3"
17
+ ]
18
+ classifiers = [
19
+ "License :: Other/Proprietary License",
20
+ "Programming Language :: Python :: 3",
21
+ "Operating System :: OS Independent",
22
+ ]
23
+
24
+ [tool.setuptools.packages.find]
25
+ where = ["."]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,22 @@
1
+ from setuptools import setup, find_packages
2
+
3
+ setup(
4
+ name="sparpartner",
5
+ version="0.1.0",
6
+ description="Deterministic, benchmark-driven stratified sampler for train/test prep.",
7
+ long_description=open("README.md", encoding="utf-8").read(),
8
+ long_description_content_type="text/markdown",
9
+ author="Henry",
10
+ author_email="osas2henry@gmail.com",
11
+ license="Proprietary",
12
+ packages=find_packages(),
13
+ install_requires=[
14
+ "pandas>=1.3",
15
+ ],
16
+ python_requires=">=3.8",
17
+ classifiers=[
18
+ "License :: Other/Proprietary License",
19
+ "Programming Language :: Python :: 3",
20
+ "Operating System :: OS Independent",
21
+ ],
22
+ )
@@ -0,0 +1,22 @@
1
+ """
2
+ sparpartner
3
+ ============
4
+
5
+ Type-aware benchmark scoring / stratified sampler for train-test prep.
6
+
7
+ Guards against overfitting by letting you deliberately hold out the
8
+ rows that most resemble a benchmark case (rather than a random split),
9
+ so a model trained on the rest can be tested against its true
10
+ "opponent" instead of an easy, self-similar sample.
11
+
12
+ Usage
13
+ -----
14
+ from sparpartner import sample
15
+
16
+ result = sample(df, bench_marks, custom_weights, look_alike=True)
17
+ """
18
+
19
+ from .main import sample
20
+
21
+ __all__ = ["sample"]
22
+ __version__ = "0.1.0"
@@ -0,0 +1,474 @@
1
+ import re
2
+ import warnings
3
+ import pandas as pd
4
+
5
+
6
+ # Values must look date-shaped (contain a date separator or month name)
7
+ # before we even attempt pd.to_datetime — this stops bare digit strings
8
+ # like "2023", "12345" (zip codes), or plain IDs from being mistaken
9
+ # for dates just because to_datetime happens to be able to parse them.
10
+ _MONTH_NAMES = (
11
+ r"jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|"
12
+ r"jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|"
13
+ r"nov(?:ember)?|dec(?:ember)?"
14
+ )
15
+ _DATE_LIKE_PATTERN = re.compile(
16
+ rf"[-/.]|\b(?:{_MONTH_NAMES})\b", re.IGNORECASE
17
+ )
18
+
19
+
20
+ def _looks_date_shaped(series, min_fraction=0.9):
21
+ """
22
+ True only if at least `min_fraction` of the column's non-null
23
+ values contain a date separator (-, /, .) or a recognizable month
24
+ name/abbreviation. Bare digit strings ("2023", "12345", plain IDs)
25
+ don't match this and are never even sent to pd.to_datetime, so
26
+ they can't be misdetected as dates just because to_datetime is
27
+ lenient enough to parse them.
28
+ """
29
+ non_null = series.dropna().astype(str)
30
+ if len(non_null) == 0:
31
+ return False
32
+ return non_null.str.contains(_DATE_LIKE_PATTERN).mean() >= min_fraction
33
+
34
+
35
+ # =============================================================================
36
+ # INPUT VALIDATION
37
+ # =============================================================================
38
+
39
+ def _validate_custom_weights(df, bench_marks, custom_weights):
40
+ """
41
+ Validates custom_weights, a list of [name, weight] pairs, against
42
+ df and bench_marks — run once, upfront, before any scoring loop
43
+ starts, so bad input fails fast instead of partway through.
44
+
45
+ Raises ValueError if:
46
+ - a name (index 0 of the pair) is not a string
47
+ - a name doesn't match a column in df
48
+ - a name has no matching entry in bench_marks
49
+ - the same name appears more than once (duplicate signal —
50
+ ambiguous which weight/bench should apply, and it would
51
+ silently double-count that column's contribution to _score)
52
+ - a weight (index 1 of the pair) is not numeric — bools are
53
+ rejected too (bool is technically an int subclass in Python
54
+ but was never intended as a weight), and NaN floats are
55
+ rejected since they'd silently poison every row's _score
56
+ """
57
+ seen = set()
58
+ duplicates = set()
59
+ for pair in custom_weights:
60
+ name, weight = pair[0], pair[1]
61
+
62
+ if not isinstance(name, str):
63
+ raise ValueError(
64
+ f"sparpartner: signal name must be a string, got {type(name).__name__}: {name!r}"
65
+ )
66
+
67
+ if name not in df.columns:
68
+ raise ValueError(f"sparpartner: column {name!r} not found in df")
69
+
70
+ if name not in bench_marks:
71
+ raise ValueError(f"sparpartner: no bench_marks entry for {name!r}")
72
+
73
+ if name in seen:
74
+ duplicates.add(name)
75
+ seen.add(name)
76
+
77
+ is_numeric = isinstance(weight, (int, float)) and not isinstance(weight, bool)
78
+ if not is_numeric:
79
+ raise ValueError(
80
+ f"sparpartner: weight for {name!r} must be numeric (int or float), "
81
+ f"got {type(weight).__name__}: {weight!r}"
82
+ )
83
+ if isinstance(weight, float) and pd.isna(weight):
84
+ raise ValueError(f"sparpartner: weight for {name!r} is NaN — must be a real number")
85
+
86
+ if duplicates:
87
+ raise ValueError(
88
+ f"sparpartner: duplicate signal name(s) in custom_weights: {sorted(duplicates)}"
89
+ )
90
+
91
+
92
+ # =============================================================================
93
+ # TYPE DETECTION + NUMERIC CONVERSION
94
+ # =============================================================================
95
+
96
+ def _to_numeric_series(series, bench_value):
97
+ """
98
+ Given a column and its benchmark value, returns (col_numeric,
99
+ bench_numeric, kind) — both values reduced to plain numbers so the
100
+ same distance formula works regardless of dtype, plus `kind` so
101
+ the caller knows exactly which path was taken (needed to pick the
102
+ right cap — re-guessing dtype after this point is unreliable for
103
+ date-strings stored as object dtype).
104
+
105
+ - numeric dtype -> used as-is; bench_numeric = float(bench_value)
106
+ kind = "numeric"
107
+ - date/datetime -> both column and bench converted to "age in
108
+ days" relative to the benchmark date, so the
109
+ benchmark itself always lands on 0. col dtype
110
+ is auto-detected either from actual dtype, or
111
+ by first requiring the column's string values
112
+ look date-shaped (contain a separator like
113
+ -/. or a month name — bare digit strings like
114
+ "2023" or a zip code never qualify) and then
115
+ successfully parsing with pd.to_datetime for
116
+ essentially every value (>=98%).
117
+ kind = "date"
118
+ - string/object -> 1 if the cell equals bench_value exactly,
119
+ else 0; bench_numeric = 1 (a perfect match
120
+ scores as touching the benchmark exactly).
121
+ kind = "string"
122
+ """
123
+ # --- date/datetime path ---
124
+ is_datetime_dtype = pd.api.types.is_datetime64_any_dtype(series)
125
+ parsed_as_date = None
126
+ if not is_datetime_dtype and not pd.api.types.is_numeric_dtype(series):
127
+ if _looks_date_shaped(series):
128
+ with warnings.catch_warnings():
129
+ warnings.simplefilter("ignore", UserWarning)
130
+ parsed_as_date = pd.to_datetime(series, errors="coerce")
131
+ # only treat as a date column if parsing worked for
132
+ # essentially every value — otherwise this is a real
133
+ # string column that happened to partially parse
134
+ if parsed_as_date.notna().mean() < 0.98:
135
+ parsed_as_date = None
136
+
137
+ if is_datetime_dtype or parsed_as_date is not None:
138
+ col_dt = series if is_datetime_dtype else parsed_as_date
139
+ col_dt = pd.to_datetime(col_dt, errors="coerce")
140
+ bench_dt = pd.to_datetime(bench_value)
141
+
142
+ col_numeric = (col_dt - bench_dt).dt.days.abs().astype(float)
143
+ bench_numeric = 0.0
144
+ return col_numeric, bench_numeric, "date"
145
+
146
+ # --- numeric path ---
147
+ if pd.api.types.is_numeric_dtype(series):
148
+ return series.astype(float), float(bench_value), "numeric"
149
+
150
+ # --- string/exact-match path ---
151
+ col_numeric = (series == bench_value).astype(float)
152
+ bench_numeric = 1.0
153
+ return col_numeric, bench_numeric, "string"
154
+
155
+
156
+ # =============================================================================
157
+ # PROGRESS DISPLAY (pure print — no logic, no return values)
158
+ # =============================================================================
159
+
160
+ def _progress_header(bench_marks, custom_weights, look_alike, row_count):
161
+ total_weight = sum(w for _, w in custom_weights)
162
+ title = "SPARPARTNER, type-aware benchmark scoring"
163
+ print(f"\n{title}")
164
+ print("-" * len(title))
165
+ look_alike_note = ("highest score sorted UP, lookalikes on top" if look_alike
166
+ else "highest score sorted DOWN, lookalikes pushed to bottom")
167
+ print(f" {'rows in pool':<16} = {row_count}")
168
+ print(f" {'look_alike':<16} = {look_alike} ({look_alike_note})")
169
+
170
+ signals_title = "SIGNALS"
171
+ print(f"\n {signals_title}")
172
+ print(f" {'-' * len(signals_title)}")
173
+ for name, weight in custom_weights:
174
+ bench = bench_marks.get(name, "<!! MISSING !!>")
175
+ print(f" {name:<20} weight={weight:<6} bench={bench!r}")
176
+ print(f"\n {'total weight':<16} = {total_weight}")
177
+
178
+
179
+ def _progress_column(name, weight, kind, bench_value, cap, distance, score):
180
+ title = f"[{name}] detected type = {kind}"
181
+ print(f"\n {title}")
182
+ print(f" {'-' * len(title)}")
183
+ print(f" {'bench':<20} = {bench_value!r}")
184
+ print(f" {'weight':<20} = {weight}")
185
+ print(f" {'cap':<20} = {cap}"
186
+ + (" [!] cap <= 0, every row will score 1.0 for this signal (no variance to score against)"
187
+ if cap is not None and not pd.isna(cap) and cap <= 0 else ""))
188
+ n_perfect = int((distance == 0).sum())
189
+ print(f" {'exact matches':<20} = {n_perfect}/{len(distance)} rows")
190
+ print(f" {'sample distances':<20} : {distance.head(3).round(4).tolist()}")
191
+ print(f" {'sample scores':<20} : {score.head(3).round(4).tolist()}")
192
+
193
+
194
+ def _progress_sort_apply(df_before, look_alike, sort_cols, n=5):
195
+ ascending = not look_alike
196
+ direction = ("descending, closest-to-benchmark scores first"
197
+ if look_alike else
198
+ "ascending, farthest-from-benchmark scores first")
199
+ title = f"APPLYING look_alike SORT (look_alike={look_alike})"
200
+ print(f"\n {title}")
201
+ print(f" {'-' * len(title)}")
202
+ print(f" {'sort call':<20} : df.sort_values(_score + {len(sort_cols) - 1} tie-break signal(s), ascending={ascending})")
203
+ print(f" {'direction':<20} : {direction}")
204
+ tie_break_names = sort_cols[1:]
205
+ if not tie_break_names:
206
+ tie_break_display = "(none, single signal)"
207
+ else:
208
+ shown = tie_break_names[:3]
209
+ remaining = len(tie_break_names) - len(shown)
210
+ tie_break_display = ", ".join(shown) + (f", ... (+{remaining} more)" if remaining > 0 else "")
211
+ print(f" {'tie-break order':<20} : {tie_break_display}")
212
+ before = df_before["_score"].round(4).tolist()
213
+ print(f" {'before sort':<20} : {before[:n]}")
214
+
215
+
216
+ def _progress_sort_glimpse(df, look_alike, n=5):
217
+ scores = df["_score"].round(4).tolist()
218
+ head = scores[:n]
219
+ tail = scores[-n:]
220
+ print(f" {'after sort (head)':<20} : {head}")
221
+ print(f" {'after sort (tail)':<20} : {tail}")
222
+
223
+
224
+ def _progress_top_rows(df, custom_weights, look_alike, n=5):
225
+ label = "TOP" if look_alike else "BOTTOM (farthest from benchmark)"
226
+ title = f"{label} {n} ROWS AFTER SCORING"
227
+ print(f"\n {title}")
228
+ print(f" {'-' * len(title)}")
229
+ display_cols = [c for c in df.columns if not c.startswith("_score")]
230
+ id_col = "id" if "id" in df.columns else display_cols[0]
231
+ for i in range(min(n, len(df))):
232
+ row = df.iloc[i]
233
+ print(f"\n #{i+1} [{row.get(id_col, '?')}] total _score = {row['_score']:.4f}")
234
+ for name, weight in custom_weights:
235
+ comp_key = f"_score_{name}"
236
+ if comp_key in df.columns:
237
+ comp_score = row[comp_key]
238
+ contribution = comp_score * weight
239
+ print(f" {name:<20} score={comp_score:.4f} x weight={weight:<6} = {contribution:.4f}")
240
+
241
+
242
+ def _progress_health_check(df, custom_weights, bench_marks):
243
+ title = "HEALTH CHECK"
244
+ print(f"\n {title}")
245
+ print(f" {'-' * len(title)}")
246
+ for name, weight in custom_weights:
247
+ comp_key = f"_score_{name}"
248
+ if comp_key not in df.columns:
249
+ continue
250
+ avg_score = df[comp_key].mean()
251
+ quality = "high quality" if avg_score > 0.7 else "moderate" if avg_score > 0.4 else "low quality"
252
+ print(f" {name:<20} avg score={avg_score:.4f} ({quality})")
253
+ if avg_score == 0.0:
254
+ print(f" [!] ZERO rows are close to bench={bench_marks[name]!r} for {name!r}, "
255
+ f"this signal is contributing nothing useful to ranking.")
256
+
257
+ dist_title = "OVERALL SCORE DISTRIBUTION"
258
+ print(f"\n {dist_title}")
259
+ print(f" {'-' * len(dist_title)}")
260
+ print(f" {'min _score':<20} = {df['_score'].min():.4f}")
261
+ print(f" {'max _score':<20} = {df['_score'].max():.4f}")
262
+ print(f" {'mean _score':<20} = {df['_score'].mean():.4f}")
263
+
264
+
265
+ def _progress_drop_nan(df, score_cols, nan_mask):
266
+ title = "DROP_NAN CHECK"
267
+ n_dropped = int(nan_mask.sum())
268
+ print(f"\n {title}")
269
+ print(f" {'-' * len(title)}")
270
+ print(f" {'rows scanned':<20} = {len(df)}")
271
+ print(f" {'rows with NaN':<20} = {n_dropped}")
272
+ print(f" {'rows kept':<20} = {len(df) - n_dropped}")
273
+ if n_dropped:
274
+ offending = [c for c in score_cols if df[c].isna().any()]
275
+ print(f" {'NaN found in':<20} = {offending}")
276
+
277
+
278
+ def _progress_normalize(raw_scores, normalized_scores, total_weight):
279
+ title = "NORMALIZE CHECK"
280
+ print(f"\n {title}")
281
+ print(f" {'-' * len(title)}")
282
+ print(f" {'total weight':<20} = {total_weight}")
283
+ print(f" {'formula':<20} : _score = raw_score / total_weight")
284
+ n = min(5, len(raw_scores))
285
+ raw_label = f"raw (first {n})"
286
+ norm_label = f"normalized (first {n})"
287
+ print(f" {raw_label:<20} = {raw_scores.head(n).round(4).tolist()}")
288
+ print(f" {norm_label:<20} = {normalized_scores.head(n).round(4).tolist()}")
289
+ print(f" {'min normalized':<20} = {normalized_scores.min():.4f}")
290
+ print(f" {'max normalized':<20} = {normalized_scores.max():.4f}")
291
+
292
+
293
+ def _progress_normalize_skip(total_weight):
294
+ title = "NORMALIZE CHECK"
295
+ print(f"\n {title}")
296
+ print(f" {'-' * len(title)}")
297
+ print(f" {'total weight':<20} = {total_weight}")
298
+ print(f" [!] total weight <= 0. skipping normalization, _score left as raw sum")
299
+
300
+
301
+ # =============================================================================
302
+ # SPARPARTNER
303
+ # =============================================================================
304
+
305
+ def sample(df, bench_marks, custom_weights, look_alike=True, show_progress=False, return_score=False, drop_nan=True):
306
+ """
307
+ Stratified sampler for train/test prep — guards against overfitting
308
+ by letting you deliberately hold out the rows that most resemble a
309
+ benchmark case (rather than a random split), so a model trained on
310
+ the rest can be tested against its true "opponent" instead of an
311
+ easy, self-similar sample.
312
+
313
+ Type-aware scoring: mirrors the reference file's
314
+ fixed_distance_score, closest to the benchmark scores highest.
315
+
316
+ For each [name, weight] in custom_weights, looks up bench_marks[name]
317
+ and detects the column's type automatically:
318
+ - numeric column -> distance = abs(value - bench), cap = the
319
+ column's own max observed distance from
320
+ bench (so a bench of 0 or a negative
321
+ number still scores correctly — the
322
+ farthest row from bench scores 0, bench
323
+ itself scores 1, regardless of bench's
324
+ own magnitude)
325
+ - date column -> value and bench both converted to "age in
326
+ days" relative to the benchmark date
327
+ (benchmark itself = 0), cap = the column's
328
+ own max observed age (so scores stay in
329
+ a sane 0-1 range regardless of the actual
330
+ day-count magnitude)
331
+ - string column -> 1 if exact match to bench else 0, cap = 1
332
+
333
+ score = clip(1 - distance / cap, 0, 1) — same formula as the
334
+ reference file's fixed_distance_score.
335
+
336
+ Each column's score is multiplied by its weight and summed into
337
+ one raw score per row, then divided by the total weight so the
338
+ final "_score" always lands in 0-1 regardless of how many signals
339
+ or what weights were used (1.0 = perfect lookalike across every
340
+ signal, 0.0 = as far as possible on every signal). A "NORMALIZE
341
+ CHECK" print always runs — regardless of show_progress — showing
342
+ the total weight and a few raw-vs-normalized score examples so
343
+ this step is easy to sanity-check.
344
+
345
+ look_alike : bool, default True.
346
+ True -> highest _score sorted to the TOP — rows that look most
347
+ like the benchmark (lookalikes) come first.
348
+ False -> highest _score sorted to the BOTTOM instead — rows
349
+ farthest from the benchmark come first.
350
+
351
+ Tie-breaking: rows that land on the exact same "_score" are not
352
+ left to pandas' stable sort alone. They're broken next by the
353
+ per-signal score (_score_<name>, the 0-1 value before weighting)
354
+ of the highest-weight signal in custom_weights; if still tied,
355
+ the next-highest-weight signal is checked, cascading down the
356
+ weight-sorted signal list until the tie breaks. Signals sharing
357
+ the same weight are compared in the order they appear in
358
+ custom_weights. Every tie-break column sorts in the same
359
+ direction as "_score" itself — a higher per-signal score wins
360
+ under look_alike=True, a lower one wins under look_alike=False.
361
+ Only if every signal is exhausted and rows are still tied does
362
+ the order fall back to pandas' stable sort (original row order).
363
+
364
+ Parameters
365
+ ----------
366
+ df : DataFrame
367
+ Must contain a column for every name in custom_weights.
368
+ bench_marks : dict
369
+ {column_name: benchmark_value} — one entry per weighted signal.
370
+ custom_weights : list[[name, weight]]
371
+ Which columns to score and how much each contributes.
372
+ look_alike : bool, default True
373
+ See above.
374
+ show_progress : bool, default False
375
+ If True, prints a client-centric readout: a header naming every
376
+ signal/weight/benchmark, a per-column breakdown of detected
377
+ type + cap + sample distances/scores as it scores, a sort-apply
378
+ readout showing the sort_values call being made plus scores
379
+ before and after, so the look_alike direction is visible as
380
+ it's applied, the top (or bottom) N ranked
381
+ rows with each signal's value x weight contribution spelled
382
+ out, and a final health check flagging any signal that isn't
383
+ contributing useful separation.
384
+ return_score : bool, default False
385
+ If True, keeps the "_score" and per-signal "_score_<name>"
386
+ columns in the returned DataFrame. If False (default), those
387
+ columns are dropped before returning — they're still used
388
+ internally for sorting and (if show_progress) for the
389
+ progress readout, just not included in the result.
390
+ drop_nan : bool, default True
391
+ If True, after scoring, any row with a NaN in any per-signal
392
+ "_score_<name>" column is dropped (e.g. from a missing value
393
+ in a signal column, or a date that failed to parse). A sanity
394
+ print always runs first — regardless of show_progress —
395
+ reporting rows scanned, rows with NaN, rows kept, and which
396
+ signal(s) had the NaN, before the drop happens. If False, rows
397
+ with NaN scores are kept and will carry a NaN "_score".
398
+
399
+ Returns
400
+ -------
401
+ df — a copy of the input, sorted by _score (direction depends on
402
+ look_alike), with the "_score" (and per-signal "_score_<name>")
403
+ columns included only if return_score is True.
404
+ """
405
+ _validate_custom_weights(df, bench_marks, custom_weights)
406
+
407
+ df = df.copy()
408
+ df["_score"] = 0.0
409
+
410
+ if show_progress:
411
+ _progress_header(bench_marks, custom_weights, look_alike, len(df))
412
+
413
+ for name, weight in custom_weights:
414
+ bench_value = bench_marks[name]
415
+ col_numeric, bench_numeric, kind = _to_numeric_series(df[name], bench_value)
416
+
417
+ distance = (col_numeric - bench_numeric).abs()
418
+ cap = distance.max() if kind in ("date", "numeric") else bench_numeric
419
+
420
+ if cap is None or pd.isna(cap) or cap <= 0:
421
+ score = pd.Series(1.0, index=df.index)
422
+ else:
423
+ score = (1.0 - (distance / cap)).clip(0.0, 1.0)
424
+
425
+ df[f"_score_{name}"] = score
426
+ df["_score"] += score * weight
427
+
428
+ if show_progress:
429
+ _progress_column(name, weight, kind, bench_value, cap, distance, score)
430
+
431
+ if drop_nan:
432
+ score_cols = [f"_score_{name}" for name, _ in custom_weights]
433
+ nan_mask = df[score_cols].isna().any(axis=1)
434
+ _progress_drop_nan(df, score_cols, nan_mask)
435
+ if nan_mask.any():
436
+ df = df[~nan_mask].reset_index(drop=True)
437
+
438
+ total_weight = sum(w for _, w in custom_weights)
439
+ if total_weight > 0:
440
+ raw_scores = df["_score"].copy()
441
+ df["_score"] = df["_score"] / total_weight
442
+ _progress_normalize(raw_scores, df["_score"], total_weight)
443
+ else:
444
+ _progress_normalize_skip(total_weight)
445
+
446
+ # Tie-break: rows with an identical "_score" are ordered next by the
447
+ # per-signal score (_score_<name>, the 0-1 value before weighting) of
448
+ # the highest-weight signal, then the next-highest weight, and so on,
449
+ # cascading through custom_weights sorted by descending weight until
450
+ # the tie breaks or the signals run out (any remaining tie then falls
451
+ # back to pandas' stable sort, i.e. original row order). Signals that
452
+ # share the same weight are compared in the order they appear in
453
+ # custom_weights. Every tie-break column sorts in the same direction
454
+ # as "_score" itself, so a higher per-signal score wins under
455
+ # look_alike=True and a lower one wins under look_alike=False.
456
+ tie_break_signals = sorted(custom_weights, key=lambda pair: -pair[1])
457
+ sort_cols = ["_score"] + [f"_score_{name}" for name, _ in tie_break_signals]
458
+ ascending = not look_alike
459
+
460
+ if show_progress:
461
+ _progress_sort_apply(df, look_alike, sort_cols)
462
+
463
+ df = df.sort_values(sort_cols, ascending=ascending).reset_index(drop=True)
464
+
465
+ if show_progress:
466
+ _progress_sort_glimpse(df, look_alike)
467
+ _progress_top_rows(df, custom_weights, look_alike)
468
+ _progress_health_check(df, custom_weights, bench_marks)
469
+
470
+ if not return_score:
471
+ score_cols = [c for c in df.columns if c == "_score" or c.startswith("_score_")]
472
+ df = df.drop(columns=score_cols)
473
+
474
+ return df
@@ -0,0 +1,227 @@
1
+ Metadata-Version: 2.4
2
+ Name: sparpartner
3
+ Version: 0.1.0
4
+ Summary: Deterministic, benchmark-driven stratified sampler for train/test prep.
5
+ Author: Henry
6
+ Author-email: Henry <osas2henry@gmail.com>
7
+ License: All Rights Reserved
8
+ Classifier: License :: Other/Proprietary License
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.8
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: pandas>=1.3
15
+ Dynamic: author
16
+ Dynamic: license-file
17
+ Dynamic: requires-python
18
+
19
+ # sparpartner
20
+
21
+ A deterministic stratified sampler for train/test prep. It doesn't
22
+ split your data randomly, it deliberately finds the rows that most
23
+ resemble a benchmark case, so you can hold those out as a genuine
24
+ test set and train on everything else.
25
+
26
+ ## Why this exists
27
+
28
+ A random train/test split assumes the test set should look
29
+ statistically like the train set. That's the wrong question if
30
+ what you actually want to know is: **does the model generalize past
31
+ one specific profile, or did it just memorize the neighborhood
32
+ around it?**
33
+
34
+ `sparpartner` answers that by ranking every row in your data by how
35
+ closely it resembles a benchmark ("bench_marks") you define, then
36
+ sorting on that resemblance. You then slice the sorted frame
37
+ yourself:
38
+
39
+ - **Lookalikes as test, the rest as train**: train on rows *unlike*
40
+ the benchmark, test on the rows that *are* like it. This is the
41
+ harder, more honest check: it tells you whether the model actually
42
+ learned something general, or only performs well near cases it's
43
+ already seen a lot of.
44
+ - **Lookalikes as train, the rest as test**: the reverse, if you
45
+ want to check the opposite direction.
46
+
47
+ `sparpartner` only produces the ranking. The actual train/test cut
48
+ is a plain slice on your side (see [Usage](#usage) below).
49
+
50
+ ## Where the idea comes from
51
+
52
+ Two unrelated places, both about comparing something against a
53
+ reference on purpose rather than at random:
54
+
55
+ **A boxer preparing for a match.** A fighter in camp doesn't spar
56
+ with whoever's free in the gym, they specifically look for a
57
+ sparring partner who moves, reaches, and hits like the opponent
58
+ they're about to face. Training against a random partner tells you
59
+ nothing about how you'll actually do; training against someone who
60
+ resembles the real threat does. `sparpartner` applies that same
61
+ logic to a model: instead of a random holdout, it finds the rows
62
+ that resemble the toughest, most relevant "opponent" profile and
63
+ holds those back as the real test, so what's left to train on is
64
+ everything *unlike* that opponent, and the test genuinely checks
65
+ whether the model can handle the match it's actually walking into.
66
+
67
+ **Astrology's approach to comparing a chart to a reference.** Reading
68
+ a chart against a benchmark isn't a single yes/no match, it's
69
+ several weighted dimensions (sun sign, moon sign, rising, houses,
70
+ and so on) each compared individually, then combined into one
71
+ overall resemblance reading. No single dimension decides the
72
+ outcome alone; the composite is what matters. That's the same shape
73
+ `sparpartner` uses: several weighted signals, each scored
74
+ independently against a benchmark, summed into one composite
75
+ `_score` rather than a single all-or-nothing match.
76
+
77
+ ## How the scoring works
78
+
79
+ You give it:
80
+
81
+ - `df`: your data
82
+ - `bench_marks`: a dict of `{column_name: benchmark_value}`, one
83
+ entry per signal you care about
84
+ - `custom_weights`: a list of `[column_name, weight]` pairs, saying
85
+ which columns matter and how much
86
+
87
+ For each weighted column, `sparpartner` auto-detects the column's
88
+ type and scores every row's distance to the benchmark on a 0–1
89
+ scale (1.0 = exact match, 0.0 = as far as possible):
90
+
91
+ | Detected type | How distance is measured |
92
+ |---|---|
93
+ | **numeric** | `abs(value - bench)`, capped by the column's own max observed distance from bench |
94
+ | **date** | both sides converted to "age in days" relative to the benchmark date, capped by the column's own max observed age |
95
+ | **string** | exact match = 1, anything else = 0 |
96
+
97
+ Date detection is automatic, it only treats a column as a date if
98
+ its values look date-shaped (contain a separator like `-`, `/`, `.`
99
+ or a recognizable month name) **and** parse successfully at least
100
+ 98% of the time. Bare numeric-looking strings (e.g. `"12345"`) are
101
+ never mistaken for dates, and object-dtype columns of digit strings
102
+ are treated as exact-match strings, not numbers. Only a real
103
+ numeric dtype gets the numeric path.
104
+
105
+ Each column's 0–1 score is multiplied by its weight and summed into
106
+ one raw score per row, then divided by the total weight so the
107
+ final `_score` always lands in the 0–1 range, however many signals
108
+ or weights you used.
109
+
110
+ ### Tie-breaking
111
+
112
+ Rows that land on the exact same `_score` aren't left to random or
113
+ arbitrary order. Ties are broken by the per-signal score of the
114
+ **highest-weight** signal first, then the next-highest, cascading
115
+ down the weighted signal list until the tie resolves. Signals that
116
+ share the same weight are compared in the order they appear in
117
+ `custom_weights`. The tie-break always sorts in the same direction
118
+ as the main score (see `look_alike` below). Only if every signal is
119
+ exhausted and rows are still tied does it fall back to original row
120
+ order.
121
+
122
+ ## `look_alike`
123
+
124
+ Controls sort direction:
125
+
126
+ - `look_alike=True` (default): highest `_score` (closest to
127
+ benchmark) sorted to the **top**.
128
+ - `look_alike=False`: highest `_score` sorted to the **bottom**,
129
+ farthest-from-benchmark rows come first.
130
+
131
+ ## Usage
132
+
133
+ ### Sample usage
134
+
135
+ ```python
136
+ import pandas as pd
137
+ from sparpartner import sample
138
+
139
+ df = pd.DataFrame({
140
+ "id": [1, 2, 3, 4, 5],
141
+ "age": [25, 30, 47, 52, 33],
142
+ "signup_date": ["2023-01-15", "2023-03-02", "2022-11-20", "2023-01-10", "2023-06-01"],
143
+ "country": ["US", "US", "CA", "US", "MX"],
144
+ })
145
+
146
+ bench_marks = {
147
+ "age": 30,
148
+ "signup_date": "2023-01-01",
149
+ "country": "US",
150
+ }
151
+
152
+ custom_weights = [
153
+ ["age", 2],
154
+ ["signup_date", 1],
155
+ ["country", 1],
156
+ ]
157
+
158
+ result = sample(
159
+ df,
160
+ bench_marks,
161
+ custom_weights,
162
+ look_alike=True, # True = most similar rows first
163
+ show_progress=True, # prints the full scoring breakdown
164
+ return_score=True, # keep _score columns in the output
165
+ drop_nan=True,
166
+ )
167
+
168
+ print(result)
169
+ ```
170
+
171
+ `age=30`, `signup_date="2023-01-01"`, and `country="US"` closely
172
+ match row `id=1` (age 25, close date, US), so with
173
+ `look_alike=True` that row lands at or near the top of the sorted
174
+ output. Flip `look_alike=False` to instead surface the rows that
175
+ look *least* like the benchmark (e.g. the Mexico row with the
176
+ furthest signup date).
177
+
178
+ ```python
179
+ # --- post-sample: turn the ranking into an actual train/test split ---
180
+ # Re-run with look_alike=False so lookalikes sort to the BOTTOM of `result`.
181
+ ranked = sample(df, bench_marks, custom_weights, look_alike=False)
182
+
183
+ # Slice however large you want the test set to be, e.g. the bottom 30%:
184
+ cut = int(len(ranked) * 0.7)
185
+ train = ranked.iloc[:cut] # non-lookalikes
186
+ test = ranked.iloc[cut:] # lookalikes, the harder, honest test set
187
+ ```
188
+
189
+ ### Parameters
190
+
191
+ | Name | Type | Default | What it does |
192
+ |---|---|---|---|
193
+ | `df` | DataFrame | required | Must contain a column for every name in `custom_weights` |
194
+ | `bench_marks` | dict | required | `{column_name: benchmark_value}` |
195
+ | `custom_weights` | list of `[name, weight]` | required | Which columns to score and how much each contributes |
196
+ | `look_alike` | bool | `True` | `True` = lookalikes sorted to top; `False` = sorted to bottom |
197
+ | `show_progress` | bool | `False` | Prints a full readout: header, per-column type/cap/sample scores, sort direction, top/bottom ranked rows with weighted contributions, and a health check flagging any signal that isn't adding useful separation |
198
+ | `return_score` | bool | `False` | If `True`, keeps `_score` and `_score_<name>` columns in the result instead of dropping them |
199
+ | `drop_nan` | bool | `True` | If `True`, drops any row with a NaN component score (e.g. missing value or unparseable date) after printing a sanity check of what was dropped and why |
200
+
201
+ ### Validation
202
+
203
+ `custom_weights` is validated upfront, before any scoring starts, and
204
+ raises `ValueError` if:
205
+
206
+ - a column name isn't a string, or doesn't exist in `df`
207
+ - a column name has no matching entry in `bench_marks`
208
+ - the same column name appears more than once
209
+ - a weight isn't numeric (bools are rejected too, a `bool` is
210
+ technically an `int` in Python but was never meant as a weight)
211
+ - a weight is `NaN`
212
+
213
+ ## A couple of things worth knowing
214
+
215
+ - **Zero-variance columns**: if every row (including the benchmark)
216
+ has the same value for a signal, there's nothing to measure
217
+ distance against, that signal scores every row 1.0 rather than
218
+ dividing by zero.
219
+ - **Negative total weight**: if your weights sum to ≤ 0, the
220
+ normalization step is skipped and `_score` is left as the raw
221
+ weighted sum instead of being silently divided by a non-positive
222
+ number.
223
+ - **Object-dtype numeric strings**: a column of strings like
224
+ `"100"`, `"200"` (object dtype, no separator) is scored as an
225
+ exact-match string column, *not* auto-converted to numeric. Only
226
+ genuine numeric dtypes (`int`, `float`) get the numeric distance
227
+ path.
@@ -0,0 +1,11 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ setup.py
5
+ sparpartner/__init__.py
6
+ sparpartner/main.py
7
+ sparpartner.egg-info/PKG-INFO
8
+ sparpartner.egg-info/SOURCES.txt
9
+ sparpartner.egg-info/dependency_links.txt
10
+ sparpartner.egg-info/requires.txt
11
+ sparpartner.egg-info/top_level.txt
@@ -0,0 +1 @@
1
+ pandas>=1.3
@@ -0,0 +1,2 @@
1
+ dist
2
+ sparpartner