workbench 0.8.332__tar.gz → 0.8.334__tar.gz

{workbench-0.8.332 → workbench-0.8.334}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: workbench
-Version: 0.8.332
+Version: 0.8.334
 Summary: Workbench: A Dashboard and Python API for creating and deploying AWS SageMaker Model Pipelines
 Author-email: SuperCowPowers LLC <support@supercowpowers.com>
 License: MIT License

{workbench-0.8.332 → workbench-0.8.334}/applications/aws_dashboard/Dockerfile

@@ -31,7 +31,7 @@ open('/tmp/deps.txt', 'w').write('\n'.join(deps))" && \
     rm /tmp/pyproject.toml /tmp/deps.txt
 
 # Layer 3: Workbench only (changes often, ~20MB)
-ARG WORKBENCH_VERSION=0.8.309
+ARG WORKBENCH_VERSION=0.8.333
 RUN pip install --no-cache-dir --no-deps "workbench[ui]==${WORKBENCH_VERSION}"
 
 # Copy app code and configuration (after pip installs to avoid cache invalidation)

{workbench-0.8.332 → workbench-0.8.334}/aws_setup/workbench_dashboard_full/app.py

@@ -12,7 +12,7 @@ print(f"Account: {aws_account}")
 print(f"Region: {aws_region}")
 
 # When you want a different docker image change this line
-dashboard_image = "public.ecr.aws/m6i5k1r2/workbench_dashboard:v0_8_309_amd64"
+dashboard_image = "public.ecr.aws/m6i5k1r2/workbench_dashboard:v0_8_333_amd64"
 
 # Workbench Configuration
 try:

{workbench-0.8.332 → workbench-0.8.334}/docs/blogs/model_confidence.md

@@ -16,7 +16,7 @@ Every Workbench model — whether XGBoost, PyTorch, or ChemProp — is actually
 The idea behind using ensemble disagreement as an uncertainty signal is well-established in the ML literature (see [Lakshminarayanan et al., 2017](https://arxiv.org/abs/1612.01474)): **when the models disagree, the prediction is less reliable.** If all 5 models predict log CLint = 2.4 ± 0.02, we have reason to be confident. If they predict 2.4 ± 0.71, something about that compound is tricky and we should be cautious.
 
 <figure style="margin: 20px auto; text-align: center;">
-<img src="../../images/ensemble_disagreement.svg" alt="Ensemble disagreement drives confidence" style="max-width: 800px; width: 100%; min-height: 250px;">
+<img src="../../images/ensemble_disagreement.svg" alt="Ensemble disagreement drives confidence" style="width: 100%; height: auto; min-height: 400px;">
 </figure>
 
 This ensemble standard deviation (`prediction_std`) is the raw uncertainty signal. It comes directly from the model itself — not from an external surrogate or statistical assumption. In our testing, it correlates strongly with actual prediction error (Spearman r > 0.85 for ChemProp on MLM CLint from the [OpenADMET Blind Challenge](https://openadmet.org/)), though your mileage will vary depending on the dataset and model type.
@@ -52,35 +52,33 @@ In practice, this gives us intervals that inherit the ensemble's discrimination
 
 ## Confidence Scores
 
-With calibrated intervals in hand, we compute a **confidence score** between 0 and 1 for every prediction. We explored several approaches (exponential decay, z-score normalization) and settled on a simple percentile-rank method inspired by the nonparametric statistics literature:
+With calibrated intervals in hand, we compute a **confidence score** between 0 and 1 for every prediction. The naïve approach is to use ensemble std directly: rank a prediction's std against the cal-set distribution. That works most of the time but has a known failure mode — when the ensemble unanimously agrees on a prediction that's nonetheless wrong, std-based confidence has no way to surface the problem.
 
-<figure style="margin: 20px auto; text-align: center;">
-<img src="../../images/confidence_percentile.svg" alt="Confidence bands showing prediction uncertainty" style="max-width: 800px; width: 100%; min-height: 300px;">
-</figure>
+This happens most often near **censoring boundaries** or in dense regions of target space. Solubility is a textbook example: kinetic-sol assays cap at ~-3.5 LogS, producing a large training cluster at -3.5 to -3.7. When the model encounters a chemically similar compound whose true LogS is much lower (say -5.5), all 5 ensemble members tend to converge on the attractor and predict -3.6 anyway — including the one that didn't see the compound during training. The ensemble agreement is genuine but uninformative; the prediction is confidently wrong.
 
-Specifically, confidence is the **percentile rank** of the prediction's `prediction_std` within the training set's std distribution:
+To address this, we use a **residual-aware confidence** metric. Instead of ranking std directly, we fit a locally adaptive calibrator that maps `(prediction, prediction_std) → expected |residual|`, then rank the *expected residual* against the cal-set distribution:
 
 ```
-confidence = 1 - percentile_rank(prediction_std)
+expected_residual = calibrator(prediction, prediction_std)
+confidence = 1 - percentile_rank(expected_residual)
 ```
 
-A confidence of **0.7** means this prediction's ensemble disagreement is lower than 70% of the training set — it's a relatively tight prediction. A confidence of **0.1** means 90% of training predictions had lower uncertainty — this compound is an outlier in some way.
+The calibrator is fit on the cross-fold validation data, where every compound's residual comes from a model that didn't see it during training. We bin predictions into quantile bins along the prediction axis and fit an `IsotonicRegression(std → |residual|)` within each bin. At inference this becomes a fast `np.interp` lookup — no sklearn dependency in production.
 
-We like this approach for a few reasons:
+This is the standard **locally adaptive conformal prediction** approach from [Lei et al. (2018)](https://www.tandfonline.com/doi/abs/10.1080/01621459.2017.1307116) applied to the scalar confidence score: instead of treating ensemble disagreement as a constant indicator of uncertainty, we let the data tell us how std actually relates to error in different regions of prediction space.
 
-- **Full range**: Confidence scores spread across the entire 0–1 range, rather than clustering near zero
-- **Directly interpretable**: "confidence 0.7" means "tighter than 70% of training predictions"
-- **No arbitrary parameters**: No decay rates or normalization constants to tune
-- **Grounded in the calibration data**: Derived from the same distribution used for interval calibration
+<figure style="margin: 20px auto; text-align: center;">
+<img src="../../images/confidence_percentile.svg" alt="Residual-aware confidence: same std means different expected error depending on prediction band" style="max-width: 800px; width: 100%; min-height: 300px;">
+</figure>
 
-That said, percentile-rank confidence is **relative, not absolute**. A confidence of 0.7 means "tighter than 70% of training predictions," not "70% probability of being correct" — two different models trained on the same data will produce similar confidence *distributions* even if one is far more accurate. The calibrated conformal intervals (`q_025`, `q_05`, … `q_975`) are where the absolute-coverage guarantee lives; the scalar confidence is for ranking and visualization. We think this is an acceptable trade-off for now, but adding an error-calibrated absolute confidence (analogous to what classification does with isotonic P(correct)) is something we're considering.
+**Interpretation:** confidence of 0.7 means "this prediction's expected error is lower than 70% of cal-set predictions." Unlike the naïve std-percentile approach, this is now a *probabilistically meaningful* statement: among compounds with similar `(prediction, std)` signatures, roughly 70% will have lower error than this one. Two compounds with identical std but predictions in different regions of target space can now get different confidence scores — which is the right behavior, because std means different things in different regions.
 
 <figure style="margin: 20px auto; text-align: center;">
 <img src="../../images/uq_conf_residual.png" alt="Confidence vs residual plot" style="max-width: 800px; width: 100%;">
 <figcaption><em>Confidence vs. prediction residual — high-confidence predictions (right) cluster near zero error, while low-confidence predictions (left) show the largest residuals.</em></figcaption>
 </figure>
 
-You'll notice the outlier around confidence ~0.55 with a residual near 1.0 — the model is moderately confident on that compound but clearly getting it wrong. We're not going to pretend this doesn't happen. The value of this plot is that it gives us **visibility** into exactly these cases, so we can investigate individual compounds where the model's confidence doesn't match reality.
+A residual-aware metric isn't magic. When good and bad predictions share the same `(prediction, std)` signature — same prediction, same ensemble agreement — no post-hoc calibrator can distinguish them. In our solubility example, ~96% of compounds with `prediction ≈ -3.6` and tight std are correctly predicted; the remaining 4% have true values far from -3.6. The calibrator will assign them all roughly the same moderate-high confidence, because that's what the population-level evidence supports. Distinguishing the unlucky 4% requires either fixing the upstream model bias (e.g., censored-aware training with `bounded_loss=True`) or adding chemistry-aware applicability-domain features — both of which are orthogonal to the UQ pipeline itself.
 
 ## Classification Confidence
 
@@ -170,7 +168,8 @@ We want to be upfront about the limitations. Confidence reflects how much the en
 - **Novel chemistry may get falsely high confidence** if it happens to fall in a region where the models extrapolate consistently.
 - **Confidence is relative to the training set.** A confidence of 0.9 on a kinase solubility model doesn't transfer to a PROTAC dataset.
 - **Conformal coverage assumes exchangeability.** The guarantee holds when test data comes from the same distribution as calibration data. For out-of-distribution compounds, coverage may degrade.
-- **Training-exposure bias in calibration.** Calibration `prediction_std` is computed by running all 5 ensemble members on the full training set, so every row was seen by 4 of the 5 models during training. Truly novel molecules (seen by 0 of 5) will tend to produce larger stds than the calibration distribution captures, which can make confidence skew optimistic-then-low on out-of-distribution chemistry. Workbench now defaults to **scaffold-based cross-validation splits** (Bemis-Murcko) for any dataset with a SMILES column — this makes confidence calibration reflect scaffold-hopping performance rather than same-scaffold interpolation. For stricter "novel chemistry" evaluation, set `split_strategy="butina"` (Morgan-fingerprint clustering).
+- **Training-exposure bias in calibration.** Calibration `prediction_std` is computed by running all 5 ensemble members on the full training set, so every row was seen by 4 of the 5 models during training. Truly novel molecules (seen by 0 of 5) will tend to produce larger stds than the calibration distribution captures, which can make confidence skew optimistic-then-low on out-of-distribution chemistry. The residual-aware calibrator helps here by anchoring confidence to historical *errors* rather than raw std — but it doesn't eliminate the bias. Workbench now defaults to **scaffold-based cross-validation splits** (Bemis-Murcko) for any dataset with a SMILES column, which makes confidence calibration reflect scaffold-hopping performance rather than same-scaffold interpolation. For stricter "novel chemistry" evaluation, set `split_strategy="butina"` (Morgan-fingerprint clustering).
+- **Indistinguishable populations within a calibration bin.** When a chunk of compounds shares the same `(prediction, std)` signature but a subset are wrong (e.g. censored-data attractors in solubility), the residual-aware metric assigns them all roughly the same confidence — population-correct, but unable to flag individual unlucky misses. Addressing this requires either upstream model fixes (censored-aware training) or chemistry-aware applicability-domain features, neither of which is part of the confidence pipeline itself.
 
 For truly out-of-distribution detection, we'd recommend pairing confidence with applicability domain analysis (e.g., feature-space proximity to training data). This is something we're actively exploring for future Workbench releases.
 
@@ -182,7 +181,7 @@ Here's how Workbench approaches model confidence today:
 
 1. **Ensemble disagreement** — Building on [Lakshminarayanan et al.](https://arxiv.org/abs/1612.01474), the 5-fold CV ensemble provides `prediction_std` as the raw uncertainty signal
 2. **Conformal calibration** — Following [Angelopoulos & Bates](https://arxiv.org/abs/2107.07511), we scale std into prediction intervals with target coverage (80% CI → ~80% coverage)
-3. **Percentile-rank confidence** — Ranks each prediction's std against the training distribution (0.0 – 1.0)
+3. **Residual-aware confidence** — Following [Lei et al. (2018)](https://www.tandfonline.com/doi/abs/10.1080/01621459.2017.1307116), a locally adaptive calibrator maps `(prediction, std) → expected |residual|`, then ranks the expected residual against the cal-set distribution. This surfaces the failure mode where ensemble disagreement is artificially low in dense regions of target space (e.g. near censoring boundaries)
 
 **Classification models:**
 
@@ -197,6 +196,7 @@ Both approaches share the same philosophy: leverage the ensemble's own disagreem
 - [Lakshminarayanan et al., "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles" (2017)](https://arxiv.org/abs/1612.01474) — Foundational work on using ensemble disagreement for uncertainty
 - [Vovk et al., "Algorithmic Learning in a Random World"](https://link.springer.com/book/10.1007/978-3-031-06649-8) — Foundational text on conformal prediction
 - [Angelopoulos & Bates, "Conformal Prediction: A Gentle Introduction" (2021)](https://arxiv.org/abs/2107.07511) — Accessible introduction to conformal methods
+- [Lei et al., "Distribution-Free Predictive Inference for Regression" (2018)](https://www.tandfonline.com/doi/abs/10.1080/01621459.2017.1307116) — Locally adaptive conformal prediction; basis for the residual-aware confidence calibrator
 - [Gneiting et al., "Probabilistic Forecasts, Calibration and Sharpness" (2007)](https://doi.org/10.1111/j.1467-9868.2007.00587.x) — Calibration vs. discrimination framework
 - [Ovadia et al., "Can You Trust Your Model's Uncertainty?" (2019)](https://arxiv.org/abs/1906.02530) — Analysis of ensemble UQ under dataset shift
 - [Gillis et al., "Variance-Gated Ensembles: An Epistemic-Aware Framework" (2025)](https://arxiv.org/abs/2602.08142) — VGMU approach for combining margin and ensemble variance in classification

workbench-0.8.334/docs/images/ensemble_disagreement.svg

@@ -0,0 +1,118 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 520" width="100%" height="auto">
+  <defs>
+    <linearGradient id="highConf" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0%" stop-color="#00d4aa"/>
+      <stop offset="100%" stop-color="#00a080"/>
+    </linearGradient>
+    <linearGradient id="midConf" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0%" stop-color="#ff9f43"/>
+      <stop offset="100%" stop-color="#cc7a30"/>
+    </linearGradient>
+    <linearGradient id="lowConf" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0%" stop-color="#ff6b6b"/>
+      <stop offset="100%" stop-color="#c44040"/>
+    </linearGradient>
+    <linearGradient id="modelBox" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0%" stop-color="#3a86ff"/>
+      <stop offset="100%" stop-color="#2a66cc"/>
+    </linearGradient>
+    <linearGradient id="calibBox" x1="0" y1="0" x2="0" y2="1">
+      <stop offset="0%" stop-color="#9b59b6"/>
+      <stop offset="100%" stop-color="#7d3f96"/>
+    </linearGradient>
+  </defs>
+
+  <!-- Title -->
+  <text x="450" y="38" text-anchor="middle" fill="#ccc" font-family="Helvetica, Arial" font-size="24" font-weight="bold">Ensemble Std + Region → Confidence</text>
+  <text x="450" y="64" text-anchor="middle" fill="#999" font-family="Helvetica, Arial" font-size="16">via the residual-aware calibrator</text>
+
+  <!-- ============ PANEL 1: Models agree, sparse region ============ -->
+  <text x="155" y="100" text-anchor="middle" fill="#00d4aa" font-family="Helvetica, Arial" font-size="18" font-weight="bold">Models Agree</text>
+  <text x="155" y="120" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="14">in a typical region</text>
+
+  <rect x="35" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="75" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.41</text>
+  <rect x="120" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="160" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.38</text>
+  <rect x="205" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="245" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.43</text>
+  <rect x="80" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="120" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.39</text>
+  <rect x="165" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="205" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.40</text>
+
+  <line x1="155" y1="215" x2="155" y2="240" stroke="#999" stroke-width="2"/>
+  <polygon points="155,248 149,236 161,236" fill="#999"/>
+  <text x="155" y="270" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="16">pred=2.40, std=0.02</text>
+
+  <rect x="40" y="285" width="230" height="60" rx="10" fill="url(#calibBox)" opacity="0.25" stroke="#9b59b6" stroke-width="1.5"/>
+  <text x="155" y="307" text-anchor="middle" fill="#c89bd9" font-family="Helvetica, Arial" font-size="15" font-weight="bold">Residual Calibrator</text>
+  <text x="155" y="328" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="13">historical |resid| ≈ 0.05</text>
+
+  <line x1="155" y1="355" x2="155" y2="385" stroke="#00d4aa" stroke-width="3"/>
+  <polygon points="155,392 148,378 162,378" fill="#00d4aa"/>
+
+  <rect x="40" y="400" width="230" height="80" rx="10" fill="url(#highConf)" opacity="0.2" stroke="#00d4aa" stroke-width="2"/>
+  <text x="155" y="430" text-anchor="middle" fill="#ccc" font-family="Helvetica, Arial" font-size="15">confidence = 0.92</text>
+  <text x="155" y="462" text-anchor="middle" fill="#00d4aa" font-family="Helvetica, Arial" font-size="22" font-weight="bold">HIGH</text>
+
+  <!-- ============ PANEL 2: Models agree, dense region (NEW CASE) ============ -->
+  <text x="450" y="100" text-anchor="middle" fill="#ff9f43" font-family="Helvetica, Arial" font-size="18" font-weight="bold">Models Agree</text>
+  <text x="450" y="120" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="14">in a dense/censored region</text>
+
+  <rect x="330" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="370" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">-3.56</text>
+  <rect x="415" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="455" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">-3.55</text>
+  <rect x="500" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="540" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">-3.58</text>
+  <rect x="375" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="415" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">-3.57</text>
+  <rect x="460" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="500" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">-3.56</text>
+
+  <line x1="450" y1="215" x2="450" y2="240" stroke="#999" stroke-width="2"/>
+  <polygon points="450,248 444,236 456,236" fill="#999"/>
+  <text x="450" y="270" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="16">pred=-3.56, std=0.02</text>
+
+  <rect x="335" y="285" width="230" height="60" rx="10" fill="url(#calibBox)" opacity="0.25" stroke="#9b59b6" stroke-width="1.5"/>
+  <text x="450" y="307" text-anchor="middle" fill="#c89bd9" font-family="Helvetica, Arial" font-size="15" font-weight="bold">Residual Calibrator</text>
+  <text x="450" y="328" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="13">historical |resid| ≈ 0.50</text>
+
+  <line x1="450" y1="355" x2="450" y2="385" stroke="#ff9f43" stroke-width="3"/>
+  <polygon points="450,392 443,378 457,378" fill="#ff9f43"/>
+
+  <rect x="335" y="400" width="230" height="80" rx="10" fill="url(#midConf)" opacity="0.2" stroke="#ff9f43" stroke-width="2"/>
+  <text x="450" y="430" text-anchor="middle" fill="#ccc" font-family="Helvetica, Arial" font-size="15">confidence = 0.55</text>
+  <text x="450" y="462" text-anchor="middle" fill="#ff9f43" font-family="Helvetica, Arial" font-size="22" font-weight="bold">MODERATE</text>
+
+  <!-- ============ PANEL 3: Models disagree ============ -->
+  <text x="745" y="100" text-anchor="middle" fill="#ff6b6b" font-family="Helvetica, Arial" font-size="18" font-weight="bold">Models Disagree</text>
+  <text x="745" y="120" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="14">wide ensemble spread</text>
+
+  <rect x="625" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="665" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">1.82</text>
+  <rect x="710" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="750" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.95</text>
+  <rect x="795" y="135" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="835" y="156" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">2.31</text>
+  <rect x="670" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="710" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">3.47</text>
+  <rect x="755" y="175" width="80" height="32" rx="6" fill="url(#modelBox)" opacity="0.9"/>
+  <text x="795" y="196" text-anchor="middle" fill="white" font-family="Helvetica, Arial" font-size="16">1.58</text>
+
+  <line x1="745" y1="215" x2="745" y2="240" stroke="#999" stroke-width="2"/>
+  <polygon points="745,248 739,236 751,236" fill="#999"/>
+  <text x="745" y="270" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="16">pred=2.43, std=0.71</text>
+
+  <rect x="630" y="285" width="230" height="60" rx="10" fill="url(#calibBox)" opacity="0.25" stroke="#9b59b6" stroke-width="1.5"/>
+  <text x="745" y="307" text-anchor="middle" fill="#c89bd9" font-family="Helvetica, Arial" font-size="15" font-weight="bold">Residual Calibrator</text>
+  <text x="745" y="328" text-anchor="middle" fill="#aaa" font-family="Helvetica, Arial" font-size="13">historical |resid| ≈ 0.85</text>
+
+  <line x1="745" y1="355" x2="745" y2="385" stroke="#ff6b6b" stroke-width="3"/>
+  <polygon points="745,392 738,378 752,378" fill="#ff6b6b"/>
+
+  <rect x="630" y="400" width="230" height="80" rx="10" fill="url(#lowConf)" opacity="0.2" stroke="#ff6b6b" stroke-width="2"/>
+  <text x="745" y="430" text-anchor="middle" fill="#ccc" font-family="Helvetica, Arial" font-size="15">confidence = 0.15</text>
+  <text x="745" y="462" text-anchor="middle" fill="#ff6b6b" font-family="Helvetica, Arial" font-size="22" font-weight="bold">LOW</text>
+</svg>

workbench-0.8.334/examples/models/chemprop_bounded_loss_test.py

@@ -0,0 +1,175 @@
+"""Bounded-Loss Sanity Check (ChemProp)
+
+This script verifies that ChemProp's bounded-loss path (BoundedMAE / BoundedMSE)
+fires correctly when a FeatureSet provides per-target `_gt` / `_lt` censor
+columns and the model is trained with hyperparameters={"bounded_loss": True}.
+
+It builds a small synthetic FeatureSet from public LogP data with artificial
+right-censoring, then trains two ChemProp variants against the same FeatureSet:
+
+    1. logp-bounded-off  -- hyperparameters omitted (bounded_loss defaults to False)
+    2. logp-bounded-on   -- hyperparameters={"bounded_loss": True}
+
+Predictions on the *artificially censored* rows should look very different:
+
+    bounded_loss=False:  predictions cluster near the censor value T because the
+                         labels are exactly T and MAE punishes deviations in
+                         either direction.
+    bounded_loss=True:   predictions are free to go above T without penalty, and
+                         since the uncensored half of the high-LogP chemistry
+                         remains at its true (>T) value, the model can learn
+                         that high-LogP chemistry actually exists.
+
+Test design notes
+-----------------
+- Use a real LogP dataset so the model has actual chemistry signal to learn.
+- Pick T at a percentile that leaves a meaningful population above the cap.
+- Censor ONLY half of the rows above T -- the other half stays at its true
+  value so the bounded model has uncensored examples to learn from. Without
+  any uncensored examples above T, even bounded_loss=True can't pull
+  predictions above T (no information to push them there).
+
+Verifying the bounded-loss effect after training
+------------------------------------------------
+Note: workbench stores boolean FS columns as nullable Int64, so cast logp_gt to
+bool before boolean indexing (or compare with == 1):
+
+    fs = FeatureSet("bounded_loss_test_fs").pull_dataframe()
+    fs["logp_gt"] = fs["logp_gt"].fillna(0).astype(bool)
+    off = Model("logp-bounded-off").get_inference_predictions("full_cross_fold")
+    on  = Model("logp-bounded-on" ).get_inference_predictions("full_cross_fold")
+    off_m = off.merge(fs[["id", "logp_gt", "logp_true"]], on="id")
+    on_m  = on .merge(fs[["id", "logp_gt", "logp_true"]], on="id")
+    for name, m in [("off", off_m), ("on", on_m)]:
+        c = m[m["logp_gt"]]
+        err = (c["prediction"] - c["logp_true"]).abs()
+        print(f"{name}: MAE on censored vs true = {err.mean():.3f}")
+"""
+
+import numpy as np
+
+from workbench.api import Endpoint, FeatureSet, Model, ModelFramework, ModelType, PublicData
+from workbench.core.transforms.pandas_transforms.pandas_to_features import PandasToFeatures
+
+# Recreate Flag in case you want to recreate the artifacts
+recreate = False
+
+# Synthetic dataset tunables -- these define how visible the bounded-loss effect is
+FS_NAME = "bounded_loss_test_fs"
+PUBLIC_KEY = "comp_chem/logp/logp_all"
+N_ROWS = 3000  # subsample size (keeps training fast)
+CENSOR_THRESHOLD = 3.0  # T: cap above which rows can be censored (top ~25% of LogP)
+CENSOR_FRACTION = 0.5  # fraction of >T rows to actually censor
+RANDOM_SEED = 42
+
+
+# =============================================================================
+# Build Synthetic FeatureSet (LogP with artificial right-censoring)
+# =============================================================================
+# Pulls real LogP data from public storage, censors half of the rows above T
+# (recorded logp clipped to T, logp_gt flag set), and keeps the other half at
+# their true values. logp_true is preserved for post-training analysis.
+
+if recreate or not FeatureSet(FS_NAME).exists():
+    df = PublicData().get(PUBLIC_KEY)
+    if df is None:
+        raise RuntimeError(f"Could not load {PUBLIC_KEY} from public data")
+
+    df = df[["smiles", "logp"]].dropna().reset_index(drop=True)
+    df = df.sample(n=min(N_ROWS, len(df)), random_state=RANDOM_SEED).reset_index(drop=True)
+    df["id"] = df.index.astype(int)
+
+    # Preserve the true value so we can analyze post-training
+    df["logp_true"] = df["logp"]
+
+    # Censor half of the rows with logp > T: set recorded logp = T and mark _gt
+    rng = np.random.default_rng(RANDOM_SEED)
+    above = df.index[df["logp_true"] > CENSOR_THRESHOLD]
+    n_to_censor = int(len(above) * CENSOR_FRACTION)
+    censored_idx = rng.choice(above, size=n_to_censor, replace=False)
+
+    df["logp_gt"] = False
+    df.loc[censored_idx, "logp_gt"] = True
+    df.loc[censored_idx, "logp"] = CENSOR_THRESHOLD  # clip the label to the cap
+
+    df = df[["id", "smiles", "logp", "logp_gt", "logp_true"]]
+    n_above = int((df["logp_true"] > CENSOR_THRESHOLD).sum())
+    n_censored = int(df["logp_gt"].sum())
+    print(
+        f"FeatureSet '{FS_NAME}': {len(df)} rows | "
+        f"true logp > {CENSOR_THRESHOLD}: {n_above} | censored (logp_gt=True): {n_censored} | "
+        f"uncensored above cap: {n_above - n_censored}"
+    )
+
+    to_features = PandasToFeatures(FS_NAME)
+    to_features.set_input(df, id_column="id")
+    to_features.set_output_tags(["chemprop", "bounded-loss", "synthetic", "logp"])
+    to_features.transform()
+
+
+# =============================================================================
+# Control Model (bounded_loss=False -- default MAE)
+# =============================================================================
+# Trained without the bounded_loss hyperparameter so the censor labels are
+# treated as exact targets. Predictions on censored rows should cluster near
+# the cap value (T=3.0) because MAE punishes deviations in either direction.
+
+if recreate or not Model("logp-bounded-off").exists():
+    feature_set = FeatureSet(FS_NAME)
+    m = feature_set.to_model(
+        name="logp-bounded-off",
+        model_type=ModelType.UQ_REGRESSOR,
+        model_framework=ModelFramework.CHEMPROP,
+        target_column="logp",
+        feature_list=["smiles"],
+        description=(
+            "Bounded-loss sanity check (control): standard MAE on artificially censored "
+            f"LogP. Censored rows have logp = {CENSOR_THRESHOLD} and logp_gt = True; "
+            "trained without bounded_loss hyperparameter so censor labels are treated as exact."
+        ),
+        tags=["chemprop", "bounded-loss", "synthetic", "logp", "off"],
+    )
+    m.set_owner("BW")
+
+# Create an Endpoint
+if recreate or not Endpoint("logp-bounded-off").exists():
+    m = Model("logp-bounded-off")
+    end = m.to_endpoint(tags=["chemprop", "bounded-loss", "synthetic", "logp", "off"])
+    end.set_owner("BW")
+    end.auto_inference()
+    end.cross_fold_inference()
+
+
+# =============================================================================
+# Treatment Model (bounded_loss=True -- BoundedMAE)
+# =============================================================================
+# Trained with hyperparameters={"bounded_loss": True} so the logp_gt column is
+# read by the template and censor labels become lower bounds. Predictions on
+# censored rows should spread above the cap, using the chemistry signal from
+# the uncensored half of the high-LogP rows.
+
+if recreate or not Model("logp-bounded-on").exists():
+    feature_set = FeatureSet(FS_NAME)
+    m = feature_set.to_model(
+        name="logp-bounded-on",
+        model_type=ModelType.UQ_REGRESSOR,
+        model_framework=ModelFramework.CHEMPROP,
+        target_column="logp",
+        feature_list=["smiles"],
+        description=(
+            "Bounded-loss sanity check (treatment): BoundedMAE on artificially censored "
+            f"LogP. Censored rows have logp = {CENSOR_THRESHOLD} and logp_gt = True; "
+            "trained with bounded_loss=True so censor labels are treated as lower bounds."
+        ),
+        tags=["chemprop", "bounded-loss", "synthetic", "logp", "on"],
+        hyperparameters={"bounded_loss": True},
+    )
+    m.set_owner("BW")
+
+# Create an Endpoint
+if recreate or not Endpoint("logp-bounded-on").exists():
+    m = Model("logp-bounded-on")
+    end = m.to_endpoint(tags=["chemprop", "bounded-loss", "synthetic", "logp", "on"])
+    end.set_owner("BW")
+    end.auto_inference()
+    end.cross_fold_inference()

{workbench-0.8.332 → workbench-0.8.334}/sagemaker_images/ml_pipelines/Dockerfile

@@ -39,7 +39,7 @@ COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/pytho
 COPY --from=builder /usr/local/bin /usr/local/bin
 
 # Layer 2: Workbench + Bridges (changes often, small layer)
-ARG WORKBENCH_VERSION=0.8.331
+ARG WORKBENCH_VERSION=0.8.332
 RUN pip install --no-cache-dir --no-deps "workbench==${WORKBENCH_VERSION}" && \
     pip install --no-cache-dir --no-deps workbench-bridges
 

{workbench-0.8.332 → workbench-0.8.334}/src/workbench/algorithms/dataframe/smart_aggregator.py

@@ -10,7 +10,7 @@ import logging
 log = logging.getLogger("workbench")
 
 
-def smart_aggregator(df: pd.DataFrame, target_rows: int = 5000, outlier_column: str = "residual") -> pd.DataFrame:
+def smart_aggregator(df: pd.DataFrame, target_rows: int = 50000, outlier_column: str = "residual") -> pd.DataFrame:
     """
     Reduce DataFrame rows by aggregating similar rows based on numeric column similarity.
 
@@ -36,11 +36,9 @@ def smart_aggregator(df: pd.DataFrame, target_rows: int = 5000, outlier_column:
     # Preserve original column order
     original_columns = df.columns.tolist()
 
-    # If already at or below target, just add the count column and return
+    # If already at or below target, just return the dataframe
     if n_rows <= target_rows:
-        result = df.copy()
-        result["aggregation_count"] = 1
-        return result
+        return df
 
     log.info(f"smart_aggregator: Reducing {n_rows} rows to ~{target_rows} rows")
 
@@ -103,7 +101,7 @@ if __name__ == "__main__":
 
     # Create test data with clusters
     np.random.seed(42)
-    n_samples = 10000
+    n_samples = 100000
 
     # Create 3 distinct clusters
     cluster_1 = np.random.randn(n_samples // 3, 3) + np.array([0, 0, 0])

{workbench-0.8.332 → workbench-0.8.334}/src/workbench/cached/cached_artifact_mixin.py

@@ -49,6 +49,24 @@ class CachedArtifactMixin:
 
         return wrapper
 
+    def refresh(self) -> int:
+        """Invalidate all cached method results for this artifact.
+
+        Forces subsequent method calls to bypass the cache and re-fetch fresh
+        data. Useful when post-processing logic has changed but the underlying
+        artifact's Modified timestamp has not.
+
+        Returns:
+            int: Number of cache entries deleted.
+        """
+        class_name = self.__class__.__name__.lower()
+        prefix = f"{class_name}_{self.name}_"
+        keys = self.artifact_cache.list_subkeys(prefix)
+        for key in keys:
+            self.artifact_cache.delete(key)
+        self.log.info(f"Refresh: Cleared {len(keys)} cached entries for {self.name}")
+        return len(keys)
+
 
 # Example usage of CachedModel
 if __name__ == "__main__":

{workbench-0.8.332 → workbench-0.8.334}/src/workbench/cached/cached_model.py

@@ -86,7 +86,7 @@ class CachedModel(CachedArtifactMixin, ModelCore):
 
     @CachedArtifactMixin.cache_result
     def get_inference_predictions(
-        self, capture_name: str = "full_cross_fold", target_rows: int = 5000
+        self, capture_name: str = "full_cross_fold", target_rows: int = 50000
     ) -> Union[pd.DataFrame, None]:
         """Retrieve the captured prediction results for this model
 

{workbench-0.8.332 → workbench-0.8.334}/src/workbench/core/artifacts/endpoint_core.py

@@ -609,12 +609,17 @@ class EndpointCore(Artifact):
         target_list = targets if isinstance(targets, list) else [targets]
         primary_target = target_list[0]
 
-        # If we don't have a smiles column, try to merge it from the FeatureSet
-        if "smiles" not in out_of_fold_df.columns and "smiles" in fs.columns:
-            fs_df = fs.query(f'SELECT {fs.id_column}, "smiles" FROM "{fs.athena_table}"')
-            if "smiles" in fs_df.columns:
-                self.log.info("Merging 'smiles' column from FeatureSet into out-of-fold predictions.")
-                out_of_fold_df = out_of_fold_df.merge(fs_df, on=fs.id_column, how="left")
+        # Templates don't save smiles or user-specified meta columns to
+        # validation_predictions.csv — merge them in from the FeatureSet so
+        # cross_fold captures match the live-inference column set.
+        meta_cols = self.workbench_meta().get("inference_meta", []) or []
+        merge_candidates = ["smiles"] + [c for c in meta_cols if c != "smiles"]
+        to_merge = [c for c in merge_candidates if c not in out_of_fold_df.columns and c in fs.columns]
+        if to_merge:
+            quoted = ", ".join(f'"{c}"' for c in to_merge)
+            fs_df = fs.query(f'SELECT {fs.id_column}, {quoted} FROM "{fs.athena_table}"')
+            self.log.info(f"Merging columns from FeatureSet into out-of-fold predictions: {to_merge}")
+            out_of_fold_df = out_of_fold_df.merge(fs_df, on=fs.id_column, how="left")
 
         # Collect UQ columns (q_*, confidence) for additional tracking (used for hashing)
         additional_columns = [col for col in out_of_fold_df.columns if col.startswith("q_") or col == "confidence"]
@@ -1100,6 +1105,12 @@ class EndpointCore(Artifact):
         if "smiles" in cols:
             output_columns.append("smiles")
 
+        # Add user-specified passthrough columns (workbench_meta["inference_meta"])
+        meta_cols = self.workbench_meta().get("inference_meta", []) or []
+        for c in meta_cols:
+            if c in cols and c not in output_columns:
+                output_columns.append(c)
+
         # Write the predictions to S3
         output_file = f"{inference_capture_path}/inference_predictions.csv"
         self.log.info(f"Writing predictions to {output_file}")